# Write-Ahead Logs on S3: Compatibility Is Not Performance > We ran an identical CAS write-ahead-log workload against AWS S3, S3 Express One Zone, Tigris, and Cloudflare R2. S3 API compatibility does not imply WAL-grade performance. Published: 2026-08-24 Author: Max Tags: wal, s3, object-storage, benchmarks, tigris, cloudflare-r2
S3 Express One
## Introduction Running a write-ahead log directly on object storage has become one of the hottest architectural ideas of the past week, thanks to [Cursor's "Git at Any Scale"](https://cursor.com/blog/git-at-any-scale) blog post. The recipe is seductive: take S3's conditional writes (ETag-based compare-and-swap), serialize your commits through a CAS chain on a log object, and you get a durable, leaderless, infinitely-storable WAL without running a single disk or consensus node yourself. The catch — and the reason we ran this benchmark — is that **S3 API compatibility does not come with the underlying performance characteristics**. Every provider we tested speaks the same wire protocol and passes the same correctness gates, but a WAL is a pathological workload for an object store: it is dominated by small, latency-bound, strictly serialized conditional writes. That is precisely the axis on which "S3-compatible" services differ the most, and the axis their marketing pages talk about the least. One of the first inspired implementations was [Waltier](https://x.com/Dan_The_Goodman/status/2090649153315750112) by Dan Goodman. We took it further by putting a real benchmarking harness around it, fixing some bugs, and running really heavy stuff for many hours. Our fork and [benchmark methodology](https://trylle.com/trylle/waltier/blob/main/docs/benchmarking.md) are live on Trylle. We ran an identical durability-first WAL workload against four providers: **AWS S3**, **AWS S3 Express One Zone**, **Tigris**, and **Cloudflare R2**. Each run pushed the same ~200 GiB payload matrix (4 KiB / 1 MiB / 16 MiB objects) from a single host over a 3 Gbit/s uplink, on the same benchmark commit, with 15 s warmup and 60 s measurement windows per phase. All four providers produced 12,044 acknowledged payloads per run — 36,132 verified payloads total — and every one of them passed correctness fully: exactly one CAS-create winner, exactly one CAS-update winner, converged followers, zero orphan objects. **The ranking below is therefore about the latency and cost of meeting the contract, not about whether the contract is met.** ### TL;DR ![Relative performance scorecard](/blog/wal-s3/performance-scorecard.svg) *Figure 1 — Relative performance index across durable writes, WAL pressure, cold reads, and follower behavior. A reading aid, not a universal vendor grade; cost and host-local cache rates are excluded.* 1. **AWS S3 Express One Zone** (overall index 92.5) is the uncontested performance leader: ~98 op/s on a single serialized WAL, sub-millisecond-class control path (p50 ≈ 10 ms), and the fastest cold reads and follower propagation. Its asterisk is structural: it is explicitly **zonal** — a single Availability Zone. 2. **AWS S3** (37.2) is the best **multi-AZ all-rounder**. Slower per operation, but it keeps scaling through 64 WAL shards (1,109 op/s) with remarkably tight tails. 3. **Tigris** (13.3) is the "premium-economy" seat: roughly 60–70 % of S3's throughput with longer tails, but correct, workable, and with zero-egress-fee economics. 4. **Cloudflare R2** (10.0) passes every correctness gate but its WAL serialization (~5 op/s single-shard), slow small durable writes, and ~1 s follower tail latency make it a poor fit for this CAS-heavy workload today. --- ## WAL Sharding Explained A single WAL is a single CAS chain: every commit must read-modify-write one object, and the next commit cannot begin until the previous ETag is known. Throughput is therefore bounded by `1 / round-trip-latency` — the provider's control-path latency *is* the ceiling. If a provider serves conditional PUTs slowly, no amount of client-side parallelism helps. The single-WAL numbers make this brutally visible: | Provider | Single-WAL CAS (op/s) | p50 (ms) | p99 (ms) | |---|---:|---:|---:| | AWS S3 Express One Zone | **98** | 10 | 12 | | AWS S3 | 17 | 55 | 127 | | Tigris | 12 | 73 | 306 | | Cloudflare R2 | 5 | 175 | 358 | ![Single-WAL CAS over time](/blog/wal-s3/wal-history-single.svg) *Figure 2 — Per-second operations for one serialized WAL across all four providers. The gap between S3 Express and everyone else is the gap in conditional-write latency, nothing more.* ### Sharding the log **Multi-WAL** means sharding the log into 16, 32, or 64 dedicated WAL objects, each carrying its own independent CAS chain. Aggregate op/s rises almost linearly — but the crucial caveat is that **ordering is preserved only within a shard**. This is acceptable exactly when your domain has a natural unit of serialization. For a Git-hosting exercise it isolates cleanly at the repository level: all refs updates for one repo flow through one shard, and cross-repo ordering was never a requirement anyway. The flip side is equally important: **per-repository throughput remains bounded by its WAL shard** — sharding raises the aggregate ceiling, not the ceiling of any single hot repo. | Provider | 1 shard | 16 shards | 32 shards | 64 shards | |---|---:|---:|---:|---:| | AWS S3 Express One Zone | 98 | 1,043 | 1,356 | **1,449** | | AWS S3 | 17 | 275 | 549 | **1,109** | | Tigris | 12 | 184 | 381 | **764** | | Cloudflare R2 | 5 | 91 | 189 | **375** | ![WAL sharding throughput](/blog/wal-s3/wal-scaling-ops.svg) *Figure 3 — Aggregate CAS throughput for the 1/16/32/64-shard sweep. Independent per-object chains; ordering remains shard-local.* ![WAL sharding speedup](/blog/wal-s3/comparison-wal-speedup.svg) *Figure 4 — Throughput relative to each provider's own single-WAL baseline. S3, Tigris, and R2 scale near-linearly; S3 Express hits its knee early.* Two distinct regimes emerge. S3, Tigris, and R2 are **latency-bound**: each shard adds nearly its full single-shard rate (S3 achieves a ~65× speedup at 64 shards), and their per-operation latency stays flat as shards increase. S3 Express is **already near saturation**: it jumps to 1,043 op/s at 16 shards, then shows clearly diminishing returns — only +39 % going from 16 to 64 shards, while p99 degrades from 34 ms to 252 ms. In the sharded time series its aggregate rate visibly converges toward the same low-thousands region that plain S3 reaches and holds steadily at 64 shards (with Tigris following the same scaling shape one tier below). Whether that knee is an S3 Express service-side limit or an artifact of driving everything from one benchmark host is not distinguishable from a single run — it may partly be a measurement issue — but the practical takeaway stands: **beyond ~16 shards, the S3 Express advantage narrows sharply, while standard S3 is still scaling at 64.** ![Sharded WAL CAS over time](/blog/wal-s3/wal-history-sharded.svg) *Figure 5 — Twelve lines: 16/32/64 independent WAL chains across all four providers, per second.* --- ## Comparing Providers Aggregate throughput hides the metric that actually pages you at 3 a.m.: **tail latency**. Here the providers separate far more than their medians suggest. ### Durable small writes The 4 KiB durable-commit phase (payload PUT plus accepted WAL reference) is a pure control-path test: | Provider | 4 KiB commits (op/s) | p50 (ms) | p99 (ms) | |---|---:|---:|---:| | AWS S3 Express One Zone | 835 | 18 | **54** | | AWS S3 | 192 | 79 | **147** | | Tigris | 108 | 122 | **634** | | Cloudflare R2 | 41 | 328 | **1,187** | ![4 KiB durable p99 latency](/blog/wal-s3/durable-4kib-p99.svg) *Figure 6 — p99 latency for durable 4 KiB commits. At 16 MiB all providers converge (transfer time dominates); at 4 KiB the control path is everything.* Note the shape, not just the magnitude: S3's p99 is under 2× its p50 — a tight, predictable distribution. Tigris's p99 is 5× its p50, and R2's is both high *and* wide. At 16 MiB the picture flattens completely — S3, S3 Express, and even Tigris converge on ~331–336 MiB/s of logical bandwidth, because transfer time swamps the control path. ![Durable throughput relative to the leader](/blog/wal-s3/comparison-durable-relative.svg) *Figure 7 — Durable throughput relative to the per-size leader. The provider gap is a small-object phenomenon.* ### Cold reads: median versus tail Cold reads expose an interesting asymmetry between the two zero-egress challengers: | Provider | 4 KiB p50 / p99 (ms) | 1 MiB p50 / p99 (ms) | 16 MiB p50 / p99 (ms) | |---|---:|---:|---:| | AWS S3 Express One Zone | 6.5 / 7.2 | 15 / 26 | 233 / 359 | | AWS S3 | 25 / 45 | 39 / 123 | 245 / 471 | | Tigris | 25 / **216** | 48 / **335** | 238 / 935 | | Cloudflare R2 | 46 / 102 | 56 / 150 | 258 / 985 | The nuance worth highlighting: **Tigris's cold-read latency degrades much more sharply than Cloudflare's** — its medians match AWS S3, but its p99 blows out to 3–5× R2's at small and mid sizes (216 ms vs 102 ms at 4 KiB), consistent with a cache-miss path that occasionally goes a long way to fetch the object. **Cloudflare, however, has the worse tail latency overall** once you look past isolated cold reads: its WAL CAS p50 alone (175 ms) exceeds Tigris's p99 cold penalty in cost, and its follower propagation (next section) is an order of magnitude behind everyone. ![4 KiB cold-read p99 latency](/blog/wal-s3/cold-4kib-p99.svg) *Figure 8 — 4 KiB cold-read p99, averaged across the four isolated process roles.* ![16 MiB cold-read p99 latency](/blog/wal-s3/cold-16mib-p99.svg) *Figure 9 — At 16 MiB the tail gap narrows but Tigris and R2 still trail ~2× behind AWS.* ### Follower propagation For a replicated system the number that matters is commit-acknowledgment-to-first-apply on a follower. With hint-assisted notification (250 ms poll fallback), steady state: | Provider | best follower p99 (ms) | worst follower p99 (ms) | |---|---:|---:| | AWS S3 Express One Zone | 9.7 | 10.8 | | AWS S3 | 44 | 56 | | Tigris | 211 | 2,216 | | Cloudflare R2 | 925 | 1,612 | ![Single WAL follower tail, hint + poll fallback](/blog/wal-s3/follower-tail-single-hint-steady.svg) *Figure 10 — Steady-state follower lag, hint-assisted (log scale). Every Cloudflare follower sits near the one-second mark; AWS followers sit near the network RTT.* ![Single WAL follower tail, poll only](/blog/wal-s3/follower-tail-single-poll-steady.svg) *Figure 11 — Poll-only mode: AWS providers degrade gracefully to the poll interval; R2's lag is dominated by read-after-CAS latency rather than polling cadence.* R2 followers hover around a full second of steady-state lag even with hints — meaning the tail is not a polling artifact but a property of its conditional-read path under CAS pressure. --- ## Pricing Considerations ### Methodology The pricing model deliberately projects from *observed* behavior rather than list-price arithmetic alone. The modeled deployment: **30 logical commits/s sustained for a 30-day month** (77.76 M durable commits), **50 TB stored**, **250 TB public egress**. Request counts are not guessed — the benchmark recorded the actual Class A / Class B operations each provider needed per acknowledged commit (~2 Class A per commit: payload PUT + WAL CAS), then scaled them to the monthly commit volume. Prices are public list prices as of 2026-08-23 (US East for AWS), before tax, support, or negotiated discounts. ### Results | Monthly component (USD) | Tigris | AWS S3 | S3 Express One Zone | Cloudflare R2 | |---|---:|---:|---:|---:| | Storage (50 TB) | $1,000 | $1,150 | $5,500 | $750 | | Egress (250 TB) | **$0** | $16,295 | $16,295 | **$0** | | Class A requests (~156 M) | $778 | $778 | **$176** | $698 | | Class B requests | $0 | ~$0 | ~$0 | $0 | | Upload/retrieval processing | — | — | $1,679 | — | | **Total** | **$1,778** | **$18,223** | **$23,649** | **$1,447** | ![Modeled monthly total](/blog/wal-s3/pricing-total.svg) *Figure 12 — Modeled monthly totals. The chasm is egress, not requests.* ![Public egress cost](/blog/wal-s3/pricing-egress.svg) *Figure 13 — 250 TB of public egress: AWS internet-transfer tiers versus the zero-egress-fee services.* ![Durable-commit request cost](/blog/wal-s3/pricing-write-requests.svg) *Figure 14 — Request cost for 77.76 M commits. Note S3 Express: Class A operations are ~4.4× cheaper than standard S3.* ### Reading the table Our general position: **Tigris might simply be enough.** It delivers the same access pattern at a speed and throughput only modestly below original S3 — same medians on cold reads, ~65 % of S3's WAL throughput at every shard count — while erasing the single largest line item on the AWS bill entirely. Because **AWS's major disadvantage is not requests or storage — it is traffic.** $16,295 of the $18,223 S3 total is internet egress. This penalty is also highly architecture-dependent: if the Git service is operated mostly *from* internal AWS machines, that traffic is free, and the externally-served remainder can be cached once on a handful of edge machines and fanned out to the rest of the world from there. Under that topology, S3's effective bill collapses toward ~$1,930/month — right back into Tigris/R2 territory, with strictly better tails. And **egress aside, S3 Express One Zone is a superior-value surprise.** Its headline storage price ($0.11/GB-month, 4.8× standard S3) reads as the premium tier, but its **Class A request pricing is actually cheaper** — $0.00113 per thousand versus $0.005, a 4.4× reduction on precisely the operation class a WAL hammers hardest. For a commit-heavy, storage-light, internally-consumed log (aggressive compaction, short retention), the request-side economics combined with 6–10× the performance make it astonishing value — provided you can live with a single Availability Zone, or pay for the log twice and run two. ![Storage cost](/blog/wal-s3/pricing-storage.svg) *Figure 15 — 50 TB standard/hot storage after free allowances. The spread here is small compared to egress.* --- ## Resume There is no single winner — **choose against your project's needs**, not against a leaderboard: - **Latency-critical, commit-heavy, AWS-internal consumers** → S3 Express One Zone, ideally with dual-zone redundancy for the log. - **Multi-AZ durability with predictable tails, traffic mostly inside AWS** → standard S3, sharded to your natural serialization unit. - **Cost-driven, internet-facing, moderate throughput** → Tigris; you trade tail latency for a bill an order of magnitude smaller. - **R2** passes the correctness gates and reads large objects competently, but its conditional-write path is not yet ready to carry a CAS-serialized WAL. One closing distinction deserves more weight than it usually gets: **separate availability from durability.** Every provider here made every acknowledged byte durable — zero orphans, full convergence, all 36,132 payloads verified. But durability is a statement about *eventually*; availability is a statement about *now*. A WAL object that technically cannot be lost, yet sits behind a one-second follower tail, a 20-second recovery window, or a single unavailable zone, will happily block your entire commit pipeline while remaining perfectly "durable" the whole time. Eleven nines on the object do not transfer to the system built on top of it. Design for the outage, not just for the fsync. --- *Benchmark: Waltier, commit [01dedff](https://trylle.com/trylle/waltier/tree/01dedffe19f4f320a09db5d992128c7e3dc82983), one run per provider (run-to-run variance not quantified), single host, 3 Gbit/s uplink, 16 dataset writers, 15 s warmup / 60 s measure per phase. Full methodology, environment details, and raw CSV/SVG downloads: [wal-report.trylle.dev](https://wal-report.trylle.dev/).* --- Canonical: https://blog.trylle.com/wal-s3