Over the last few weeks I became the #5 contributor on leanMultisig repo, the Lean Ethereum zkVM.
The PRs we have contributed had an estimated 40% e2e speedup on the repo on AVX512.
Initial approach
We have initially ported the Plonky3 approach of microoptimizations to leanMultisig, but we had to target e2e benchmarks instead of micro ones. This in the end turned out to be a trigger for scaling.
The biggest problem was nudging the agent to be ambitious, and it is still a recurring problem. Claude gravitates towards quick wins, easy micro optimizations very easily. Wins exist, but the fact that it is afraid (for a lack of a better word) is a ceiling issue.
Why is this the case? Depth doesn’t come from agent, the human in the loop is a critical piece. This was described as taste by others, but it is also key to understand the architectural pieces that exist in the codebase and make a clean decision.
The ability to make these decisions degrades quickly with the rapidity of context switching. I can recognize this more quickly today, if we are circling its not because of agent performance, BUT because I am too tired. The pitfall is the eagerness to move quickly and find solutions for everything. You can have an illusion of progress.
The quick(er) wins
The first win came from a simple force inline on quintic extension field arithmetic. 10 lines changed, 3 files touched, 3.6% performance improvement. Having a win was great, but this was a more simple replication of what happened on Plonky3.
The second win was the degree-split AIR constraint evaluation. 330 lines changed in 5 iterations, 7.64% speedup on AVX512. This was a very clean experiment, with solid results. The idea was actually tried previously, but resulted in no improvement (back in December 25’ - different codebase back then). This was the first PR that saw additional commits on top, the idea was flagged that it could be expanded. While that didn’t yield results, this was great to see, agents are great at noticing something, but experienced engineers might find the extra squeeze on top.
The depth - allocations
When I say that depth of understanding is key, the storyline of these 3 PRs explains it best.
One of the agents most impactful potential finding was replacing glibc with mimalloc. This resulted in a local speedup of 27-36% depending on benchmark (prod vs our quicker custom criterion gate). During testing it turned out to be hardware specific, with later research also showing signs that it could have been cloud specific.
The review conclusion was A) this is worth investigating further B) the solution needs to be hardware agnostic
We initially checked bumpalo and Binius’s attempt to integrate it. Because they decided to fix the underlying problem and deprecated it, we had a clean signal that bumpalo is not the clean way.
The first step was depth, and during research I specifically started focusing on Jolt’s approach. I was aware that Jolt uses agentic loops during development, but after diving in I think I understand their recipe even better:
-
Jolt focuses on being compute-bound and not memory bound
-
During development they inherently introduce alloc overhead
-
The agentic loops come in handy to reduce this
The next experiment specifically focused only on this. Agent profiles, finds allocation hotpath, removes it from codebase. 10.3% speedup and agent went very deep. At last step it tried removing 1.6m allocs, but then change was insignificant at that point. The alloc overhead gets diffused in codebase and you need to rewrite it completely, which was outside of scope. We were thinking about follow up experiments to tackle this, but then I came up with an idea.
The next step was clearly taste, plus some reasonable risk taking. At Thursday midnight I started thinking, do we have a zk targeted allocator in the space? If we don’t have one, why don’t we attempt to build one? Quick research showed validity behind the idea. And while during the research loops the agents are risk averse, they are really good at supporting your idea and saying its feasible. Just have to be careful not to end up in an agent psychosis. The risk was attempting this, allocators can contribute to soundness issues. But its a weekend of agentic iterations, time is less valuable than trying things.
This matured into the first attempted zk product build, with the caveat that it is on the verge of feature/product from a technical standpoint. Warm proof speedup at 27% from a 200 LoC Rust build. I wont go into details, the previous blog post already shared the story. There is also active development going on to make this production ready, and I am planning to submit a PR to Plonky3 with an improved build.
It is clear that the methodology is a blue print for the future. The industry is moving in the direction.
The next steps for zk-autoresearch are also clear:
-
I have built an orchestrator called brain with a harness to be able to reset it for clean context quickly.
-
The brain manages the bandwidth for experiments with an SSH session to machines capable of proving
-
A new bug hunter agent - This ties back to the feature/product builder. zk-alloc needs this specifically and we mixed up proper sequencing by not running one post implementation
We will continue to focus on leanMultisig and Plonky3 for the next month and by now I am quite confident we can replicate the initial success. And these are compounding on top of each other as the new PRs are merged one by one.