Initial experiments focused on micro-optimizations, later experiments matured to larger algorithmic rewrites. But the product/feature question is the most interesting -> Can zk-autoresearch develop something from scratch?
During a leanMultisig performance optimization run we have found that switching to mimalloc from glibc resulted in a steady ~25% speedup on production during the review we have realized that it was hardware dependent. Using AWS c7a.2xlarge the memory pressure was real at 16GB. Running mimalloc on 64GB Hetzner bare metal resulted in a regression. Instead of shipping the adjustment under a feature flag we have investigated for an agnostic option.
Examples from other provers showed a diverse picture:
-
Jolt regularly sweeps for allocs, with agentic loops to keep it compute bound, rather than letting it become memory bound
-
Binius attempted using bumpalo, but after a chain of PRs deprecated the feature in favor of solving the underlying alloc issue with algorithmic changes
And while this provided a clear experiment roadmap that significantly improved the e2e latency, it couldn’t match simply changing to mimalloc for a memory pressure environment.
The allocations after each iteration got more diffused in the codebase and while an agent is very efficient at eliminating hot path bottlenecks, once its diffused it would need to rewrite the whole codebase to optimize allocations further.
Thesis:
Allocators have been finetuned for decades, but there is no allocator directly targeted at zero-knowledge provers.
This is a clear gap, and the observed reasoning level on allocations were genuinely impressive. We have decided it is a reasonable target to attempt to develop a custom allocator for LeanMultisig.
Spec design vision:
A custom allocator developed from a basic sketched Rust scaffolding, only using zk-autoresearch experiment loops. There were big initial ideas, such as:
-
Automatically detecting proving phases and switching allocator setups —> no API changes required
-
Runtime memory-pressure adaptation (solving the mimalloc 16GB-vs-64GB divergence by reading /proc/meminfo and adjusting retention policy)
Experiment designs were written in advance, but these were obviously heavily modified on the fly:
-
1st round - Bug fixing and correctness checks
-
2nd round - Matching glibc performance
-
3rd round - Speed up compared to glibc
-
4th round - Generalizing over multiple provers
-
5th round - Second performance optimization round
What actually happened:
-
1st round - Bug fixing and correctness checks
-
2nd round - Matching glibc performance - with a valid solution
Intermediate bug fixing with external agent - finding that we are actually already 27% faster than glibc
- 3rd round - Tying up loose ends with a few directions, determining that we are at local optimum with high confidence. Overengineering is not beneficial.
Interesting cheating behavior observed:
Round 2 final gate was matching glibc with zk-alloc. Agent rewrote the implementation to pass everything to glibc, and make the allocator a thin wrapper. Classic human error and classic Son of Anton behavior (the quickest way to match glibc, is to simply use glibc)
zk-alloc setup by the end:
A simple big arena with bump allocator.
zk-alloc is a ~130-line Rust crate implementing a bump+reset arena allocator. It mmaps 128GB virtual (8GB per thread, 16 threads) with MAP_NORESERVE, bumps a pointer for every allocation (no metadata, no free-list), and resets the pointer between proofs. dealloc is a no-op for arena-owned memory. Physical pages are reused across proofs, eliminating demand-paging and cache pollution from glibc’s memory management.
Cross-prover validation
| Prover | Architecture | zk-alloc vs glibc | Mechanism |
|---|---|---|---|
| leanMultisig | FFT-based · Plonky3 / WHIR, KoalaBear | −27% warm proof | Page reuse eliminates demand-paging; contiguous layout improves prefetch |
| Plonky3 | FFT-based · BabyBear / FRI | −12% to −17% | Same mechanism; validated on Poseidon1/2 and Keccak benchmarks |
| Jolt | Sumcheck-based · Dory / BN254 | +1% to +4% (null) | Compute-bound; allocator overhead is less than 1% of runtime |
Original figure note: FFT-based provers are memory-bound (large NTT/DFT buffers, parallel Merkle hashing) and benefit significantly. Sumcheck-based provers are compute-bound by design and are unaffected — validating zk-alloc as an architectural fit for FFT-heavy proving systems, not a universal speedup.
Hardware sensitivity
During the review process we noticed differences between two equivalent Hetzner AX42-U (AMD Ryzen 7 PRO 8700GE, 8C/16T, 61GB DDR5) machines. The observed delta was significantly different (8% vs 27%).
After validating all potential setups are equivalent a longer debugging session finally pinpointed the issue to a silent failure:
Root Cause: madvise(MADV_HUGEPAGE) silently failed on the development machine, masking this bug. Fix -> Switch to MADV_NOHUGEPAGE
This brought down the delta to 4%, and it was stable over multiple measurements. This clearly signalled that this is not simple hardware noise.
After another 30 minute of Claude CLI debugging (and multiple redirects that this is not just hardware noise) we have found the explanation:
The identical machines had different DIMM timing grades (EB0/OL vs DB0/OD). Same manufacturers, identical rated speeds -> 4% difference.
This means in practice is that the solution is very much hardware sensitive.
What this means
zk-alloc isn’t an allocator breakthrough, it is a 1990s bump allocator applied to a domain where nobody questioned malloc. The novelty is the application, not the technique.
The bump+reset arena pattern was formalized by Hanson in 1990. The technique is 35 years old. What’s new is applying it as a drop-in GlobalAlloc for ZK provers.
Reference: Hanson, D.R. (1990). “Fast allocation and deallocation of memory based on object lifetimes.” Software: Practice and Experience, 20(1), 5-12.
The counterargument is straightforward: a compute-bound prover doesn’t need this simple allocator. Jolt’s null result proves this. But most FFT-based provers need to do that engineering, and the arena gives them a free -22% to -27% without rewriting their memory model.
The deeper insight is architectural: there are two local optima for ZK proving. Compute-bound provers (sumcheck) eliminate allocation by design. Memory-bound provers (FFT) can accept large allocations and make them free.
Potentially both are valid and the arena makes the FFT path viable at production scales where it would not be.
Conclusion
The original idea, based on observed gaps, was to implement a zk-alloc as a standalone library.
What zk-autoresearch found was that there is likely no need to implement anything more than a feature.
This in itself is a strong validation of the technique.
And the simplicity of the design is beautiful.
References
Link to PR: https://github.com/leanEthereum/leanMultisig/pull/205
Link to standalone zk-alloc implementation: https://github.com/Barnadrot/zk-alloc
Link to zk-autoresearch repo experiment logs: https://github.com/Barnadrot/zk-autoresearch/tree/main/experiment_logs/zk-alloc