Benchmarks and fault/soak measurements

benchmarks/compare.py measures real LPUSH/LINDEX/LPOP list workloads against owned ffkv/Redis processes. It compares list remove-on-pop semantics, not native lease/ACK semantics. benchmarks/runtime.py supplies the standard-library RESP client, process ownership, histogram and current-RSS measurement. No performance numbers are generated by this documentation.

Reproduce the repository workload

Build a release server. Install/provide redis-server if including the owned Redis reference:

cargo build --release --bins
python3 benchmarks/compare.py --binary target/release/ffkv \
  --items 4096 --hot-operations 512 \
  --json benchmark.json --csv benchmark.csv

The harness owns disposable processes and temporary directories; it does not require a pre-existing ffkv server. If Redis is intentionally unavailable, explicitly omit it rather than treating absence as a result:

python3 benchmarks/compare.py --binary target/release/ffkv --skip-redis \
  --items 4096 --hot-operations 512 --json benchmark-ffkv.json

Do not point external options at a production dataset. Even though external cases use a unique ffkv:benchmark:* namespace and are not reconfigured, they still create/read/drain real lists and consume resources.

Workload and tunable controls

The default matrix includes memory/strict/everysec durability with 256 B, 4 KiB and 64 KiB bodies. ffkv hybrid cases require persistence, so volatile mode does not invent a hybrid-without-WAL case. Owned persistence settings are aligned/verified; this matters more than comparing unrelated defaults.

Option	Current default / use
--binary, --redis-binary	target/release/ffkv, redis-server
--skip-redis	Explicitly omit Redis
--modes	memory strict everysec
--payloads	256 4096 65536 bytes; values 1 byte–1 MiB
--items	512 total queued bodies per case, not per worker; accepts suffixes such as 1m/5m/10m
--concurrency / --jobs	4; 1–min(items, 128)
--pipeline	8; 1–128, with combined payload/framing safety validation
--hot-items	4
--hot-operations	256 per worker when duration is zero
--duration	0; positive seconds select a timed hot phase per case
--timeout, --maintenance-timeout	15 / 120 seconds
--inline-threshold, --cache-bytes	1024 / 1048576 bytes for hybrid benchmark cases
--segment-bytes, --disk-budget	8388608 / 0 bytes
--json, --csv	JSON defaults to stdout; CSV optional

The benchmark cache/segment settings differ from server defaults. Record the complete invocation and report settings rather than assuming they match your deployment. Larger million-item or timed runs are available but were not automatically performed by publishing a small baseline.

Interpret phases correctly

The harness records enqueue/backlog RSS, first reads, hot repeats, drain and maintenance. Cold means the first application LINDEX pass over the backlog, not eviction of OS page cache. Do not label this a physical-disk cold-cache test.

Latency is measured from batch send-start to each reply and includes pipeline queueing. It is not divided by pipeline depth to invent per-command latency. Percentiles are measured histogram bucket bounds with their method/semantics recorded. Throughput is completed operations divided by elapsed phase time; attempted requests, errors, unknown outcomes and retries are separate fields.

Current RSS is sampled from owned child processes at named phases, not max-RSS from an unrelated process or a logical allocation estimate. External endpoint RSS is unavailable. Compare backlog RSS together with payload bytes, cache size and throughput: lower RSS can come with worse small-body overhead, lower strict-write throughput or cold-read cost.

JSON/CSV include environment, versions, settings, command lines, per-phase counts/latency, RSS, cleanup errors and case/run failures. Inspect the report’s ok and errors, not only a selected fast row. A missing engine, failed case or unknown command outcome must remain visible; do not discard failures to improve a headline.

External Valkey/Dragonfly or Redis endpoints

--external accepts redis=HOST:PORT, valkey=HOST:PORT or dragonfly=HOST:PORT, with --external-durability memory|strict|everysec describing the intended policy. The harness does not reconfigure external persistence, and labels these results external/unverified. Verify their actual policy yourself and report the verification evidence. A redis_version signature does not uniquely identify every compatible engine.

This narrow benchmark client does not supply the protected native CLI’s complete TLS/ACL interface. Do not weaken production security to accommodate it; use disposable owned/reference endpoints. Do not claim an external engine was tested unless the actual report contains a completed case for it.

Fault/soak workflow

python3 tests/soak_smoke.py --binary target/release/ffkv \
  --duration 60 --json soak.json

This is a finite workload with owned data/processes. It exercises concurrent queue operations, SIGKILL/recovery cycles, checkpoint/compaction, stale receipts, idempotency and restoration into a real new process. It is not seven-day endurance, disk-loss simulation or proof of honest device caches. Fsync under real power loss is a different experiment from killing a process.

Existing evidence and honest claims

Repository artifacts are benchmarks/results/macos-arm64.json and benchmarks/results/soak-macos-arm64.json; the README records their original environment and selected rows. They are historical single-workstation evidence, not checks rerun by this site. Read the full report before quoting any result and include engine/version, OS/architecture, concurrency/pipeline, payload/backlog, storage/durability, cache and latency semantics.

The published baseline demonstrates a memory/throughput tradeoff, including worse small-body overhead and strict-write throughput below Redis in that run. It does not establish a universal speed advantage, an 80% throughput target, performance-per-dollar, million-job/beyond-RAM capability, Valkey/Dragonfly results or production HA. Those require their own executed workloads and evidence. Keep behavioral tests and performance measurements separate: a fast incorrect or unverified case is not a useful benchmark.

MCP transport overhead

benchmarks/mcp.py compares persistent native RESP and persistent MCP stdio sessions against the same disposable strict-durability ffkv server:

cargo build --release --bins
python3 benchmarks/mcp.py \
  --binary target/release/ffkv --mcp-binary target/release/ffkv-mcp \
  --rounds 100 --batch-sizes 1 16 128 --output mcp-benchmark.json

PING and 64-byte SET/GET workloads alternate transport order after warmup. SET/GET uses two commands per workload item. Both paths decode complete replies; MCP includes JSON request encoding while direct RESP requests are pre-encoded. The direct socket drains replies concurrently with a reused writer thread, so large bidirectional batches do not deadlock. Worker scheduling is included in direct timings; no thread is created per measured round. Assertions run outside the timed interval. Results include completed command counts, commands/second, median and p95 round-trip time, and amortized command cost. The amortized figure is not individual-command latency.

Batching amortizes JSON-RPC and process-boundary overhead. This local sequential microbenchmark does not measure concurrent clients, remote networks, agent model latency, or production durability hardware. See MCP for access semantics and limits; do not substitute speed measurements for behavioral tests.

The large-payload regression that previously stalled the direct reference path:

python3 benchmarks/mcp.py \
  --binary target/release/ffkv --mcp-binary target/release/ffkv-mcp \
  --rounds 1 --warmup 1 --batch-sizes 12 --value-bytes 1048576

This is a correctness regression, not the published small-value baseline. Large MCP results require artifact paging; that retrieval time remains included.

Observed MCP baseline

benchmarks/results/mcp-macos-arm64.json records the executed release-binary baseline on macOS 27.0.1 arm64 with Python 3.12.10: 100 measured rounds per case after ten excluded warmup rounds, one sequential client, persistent connections, 64-byte values, and strict durability on the same owned server. Each SET/GET item contributes two commands. The direct path uses the duplex writer described above.

Workload	Commands per round	MCP commands/s	Direct RESP commands/s	MCP median round (ms)	Direct median round (ms)
PING	1	2,050	4,451	0.265	0.154
PING	16	11,508	13,028	1.310	1.043
PING	128	15,320	19,753	6.922	5.986
SET/GET	2	301	318	5.786	5.196
SET/GET	32	403	397	74.213	73.240
SET/GET	256	534	548	469.901	470.471

The full report retains elapsed time, exact command counts and p95 values. Strict writes synchronize before acknowledgement; pipelining did not turn this workload into one shared fsync. The observed small write-throughput difference does not imply zero MCP overhead, and PING throughput is not write throughput. These are workstation observations, not latency or throughput guarantees.