Plonky3 Monty-31 Optimization — Model Scaling Experiment - zk-autoresearch

Research question: Does model scaling help in autonomous performance optimization research?

The Results

The Sonnet CLI cell is intentionally empty. Once CLI Opus results were in, testing Sonnet CLI became unnecessary — the architecture difference dominates the model difference at this task. The cost through the CLI is below Sonnet API with a Claude subscription and it’s difficult to hit daily limits.

Part 1 — Opus vs Sonnet on Loop.py

Hit Rate: Identical

Both models found exactly one improvement in iter 1. Both circled for the remaining 10–11 iterations. Neither kept anything after iter 1. Opus cost 3.6× more per iteration and produced the same result.

What Each Model Found

Sonnet iter 1: Replaced vpminud with vpcmpgeud/vpcmpltud in Add/Sub — port 0 pressure relief hypothesis. +0.86% in-session. Later confirmed as a regression on the correct hardware (Zen 4, AVX-512 active). The Intel port 0/5 reasoning does not apply to AMD.

Opus iter 1: Fused butterfly layers 0 and 1 in first_half_general into a single memory pass. +0.94% in-session. Not cross-session validated. Not submitted upstream.

The Sonnet finding was later confirmed as a regression when re-tested on the correct hardware. The Opus finding was superseded by CLI Opus results and not pursued further.

Circular Thinking

After iter 1, both models spent remaining iterations re-deriving structurally identical ideas.

Sonnet (iters 2–11):

• Iter 8: remove backwards from dit_layer_rev — −3.88%
• Iter 9: remove backwards from dit_layer — −1.72%
• Iters 4–7, 11: API failures / forbidden-pattern gates

Opus (iters 2–12):

• Iter 2: extend the same 2-layer fusion — −0.38%
• Iter 3: same fusion in second_half — −1.89%
• Iter 4: same fusion in first_half — −1.84%
• Iters 5, 11, 12: broader OOP batching variants — all regressions

Opus re-derived the same layer fusion idea three times across iters 2–4. Sonnet targeted backwards removal twice. Neither model escaped its iter 1 hypothesis.

Why loop.py Context Management Fails

loop.py manages agent context via a Python script: an eliminated_ideas.md file is injected into every prompt listing analytically dismissed ideas, confirmed dead ends, and regressions. By iter 8, this file contained 31 entries.

The failure mode: agents read the eliminated list, acknowledge the constraints, and then — within the same response — re-derive an idea structurally equivalent to one already on the list. The model applies the same pattern to an adjacent function, gives it a different name, and proceeds. The eliminated list records specific implementations, not conceptual categories. Loop.py cannot enforce categorical exclusion.

More fundamentally: loop.py has no memory of why previous ideas failed. Each iteration starts with the same injected context. The agent cannot build suspicion, update its world model, or notice that 8 consecutive iterations have all regressed — because it has no access to the trajectory, only the snapshot.

Part 2 — The CLI Pivot

The loop.py results raised a hypothesis: the bottleneck is not model capability but agent architecture. Opus and Sonnet reached the same dead end because both ran as stateless loop.py agents, not because they lacked the reasoning ability to do better.

Prior conviction from a different domain: While this Plonky3 experiment was running, we won the Paradigm hackathon with the same CLI Opus approach — a GPU kernel optimization challenge (H100, block-sparse attention) solved in 83 agent iterations, by a 40% margin over second place. CLI Opus navigated a complex optimization landscape that stateless agents could not. That result, arriving mid-experiment, reinforced the conviction to commit fully to CLI Opus on Plonky3 and test whether the same architectural advantage transfers to CPU/ZK optimization.

The Sonnet CLI cell was left empty by design. The loop.py results showed that reasoning quality — not model size — was the bottleneck: both models hit the same ceiling because the architecture prevented cross-iteration learning. The fix is better reasoning with persistent context, not a cheaper model. Opus CLI addresses the root cause directly.

Part 3 — CLI Opus: Early Results

This section is a preview. Full results will be published separately.

The CLI Opus run is ongoing. Results so far are qualitatively different from the loop.py runs in two ways.

The agent diagnosed a broken benchmark. After multiple iterations of null results, the agent did not conclude “local optimum” — it accumulated suspicion, ran perf stat on the benchmark process, traced the anomaly to a specific location in the measurement code, and proposed a fix. The fix revealed that 54% of measured benchmark time was not FFT computation at all. Every loop.py iteration had been evaluated against a confounded baseline.

With clean measurement, previously discarded compute-side changes showed real signal. The agent systematically retested its own prior null results and found that most of them were real improvements masked by noise. Hit rate on retested ideas: 6 of 9 kept.

The agent falsified its own hypothesis. Before committing to a large algorithmic rewrite, the agent ran CPU profiling to test whether the workload was memory-bound or compute-bound. It found IPC=1.91 and 4.62% cache miss rate — firmly compute-bound. The memory-bound hypothesis was abandoned. The agent then proceeded with arithmetic optimizations instead of cache restructuring.

Neither behavior is possible in a stateless loop.py agent. Both required reasoning across multiple prior iterations.

Full results, final breakdown, and upstream PR details will be published when the experiment concludes.

Conclusion

Model scaling (Sonnet → Opus) produced no measurable benefit on loop.py: identical hit rate, identical circling behavior, 3.6× higher cost. The bottleneck was architecture, not capability.

CLI Opus broke out of the pattern that trapped both loop.py runs. The key capability is not reasoning quality within a single iteration — it is the ability to reason across iterations: to notice patterns, build suspicion, diagnose root causes, and update strategy. That capability is structurally unavailable to stateless agents regardless of model size.

The answer to the research question: model scaling does not help on loop.py. Architecture scaling does.

Open Question: Security as Part of the Autoresearch Loop

A question raised during this experiment: should security verification be part of the autoresearch loop itself, not just a gate before human review?

loop.py addressed this by running a correctness check after every iteration — if tests fail, the change is reverted. But correctness and security are not the same thing. A change can pass all tests and be mathematically incorrect in ways that only manifest at the proof system level: subtle field arithmetic errors, broken invariants in the permutation structure, or changes that preserve test vectors while silently altering the reduction semantics.

The current flow closes this gap manually — human review before any upstream PR, with a post-experiment verification script as a pre-review gate. But this is a bottleneck: it requires a domain expert to check every experiment output, and it does not scale as the autoresearch loop runs faster and across more targets.

The clear priority is closing this gap further— moving security verification earlier in the loop so that the agent itself can distinguish “passes tests” from “cryptographically sound.” This is a hard problem: it requires either formal verification integrated into the loop, or a separate security-focused agent that reviews changes before they are committed.

We will expand on this in a standalone article.

Acknowledgements

The Opus 4.6 API run was sponsored by zkproof.org and QEDIT. We thank them for supporting open research into AI-assisted ZK performance optimization.

What's Next

CLI Opus on Plonky3 is ongoing. When the experiment concludes, a full breakdown of every kept improvement, the bench fix methodology, and upstream PR will be published.