Chip design agents can close a layout, but Build-A-Chip-Bench puts the best overall score at 16.6%
ChipAgents and NVIDIA's new benchmark scores five models from specification to GDS. GPT-6 Astra passes the most gates, while Claude Fable 5.1 scores highest overall.
GPT-6 Astra passes the most checks on Build-A-Chip-Bench, but Claude Fable 5.1 has the best score. Astra clears both the functional gate and the clean layout gate on 85% of tasks, yet scores 8.7%. Fable clears fewer gates and scores 16.6%, the best result among the five models.
The gap is PPA, short for power, performance and area. A legal layout can still be too slow, too power hungry, or too large to make. The gates do not measure that directly.
ChipAgents introduced the benchmark on 14 October, built with NVIDIA. It tests AI agents on a full digital design job. An agent starts from a written specification and goes through physical layout to a verified GDS file, the layout file a chip factory uses.
Across the five models, agents passed functional checks on about 40% of tasks and met PPA targets on only about 10%. The 10% is not a hit rate. It is the average closeness to the constraint on designs that already passed.
The task is long and the gate is hard. Each design must be functionally correct against hidden verification IP, which is a hidden test suite. It must also be routable, meet hold timing, and pass design rule checks. The work runs on open source tools and an open source process design kit, the file set that describes the factory process.
Only designs that pass are scored on PPA. In the research page, the final score is the mean over all tasks of three results multiplied together: functional, GDS, and PPA. A design that fails either gate contributes zero.
The authors say older tests no longer separate models. CVDP has plateaued at about 65 to 70% pass@3, meaning success within three attempts. ChipBench rose from 30.7% at release to 90.6%. Point tests climb until they stop separating models, so the field moves to a longer task.
The example task shows how tight the limits are. The agent builds an inference accelerator for Qwen3-0.6B INT4, a small language model in 4 bit form, on the Nangate45 platform at 400 MHz. Die area is capped at 6 mm², and decode throughput must reach 15 tokens per second.
The sample reference design lands at 5.76 mm² and 16.35 tokens per second, leaving little headroom on either limit. Verification runs in a hidden container that the agent can query but not inspect. A third container grades the submission, preventing an agent from tuning against the checker.
On the longer task the ranking flips. GPT-6 Astra leads on both gates, yet its PPA column reads 10.2%, against 25.5% for Fable. Multiply each PPA figure by the share of tasks that pass both gates and the scores follow: 85% times 10.2% gives 8.7%, while 65% times 25.5% gives 16.6%. The gate leader has 20 points more passing but only about 0.4 times the PPA quality of the score leader.
Below the top two, the PPA column falls away. Gemini 3.8 Flash scores 4.1%, Kimi K3 scores 1.3%, and GLM-5.3 scores 0.0%. Functional competence and physical quality come apart in this table. The economic prize sits in the second.
A layout that routes but burns die area costs more per wafer. Judging an agent by gate passes alone would pick GPT-6 Astra and give the weaker physical result. The research page reports the same weakness: agents underestimate slow tool runs and often break working behaviour while optimising, and the gates never test that directly.
The traces show why. When a physical step fails, agents fix it at the physical level first. In one run GPT-6 Astra documented a change that would nearly halve its SRAM macro count, which is the number of memory blocks. It deferred the change and spent six hours on routing configurations while detailed routing reported 373,623 violations.
In another run, a Kimi K3 push to raise throughput introduced a functional bug and a hang. The run finished about 3% slower than where it started. The page says backend tool use is reasonably reliable, so agents can usually configure and run the layout tools. The weak point is judgement about what to change. RTL is the hardware description code that comes before layout, and these traces show agents patching layout instead of going back to it.
A single place and route run can take more than four hours under a fixed time budget, so each wrong turn is expensive. The page says agents rarely treat existing behaviour as a constraint to keep intact while they optimise. They repair damage after the fact, and the slow loop punishes them.
The benchmark's authorship also matters. ChipAgents wrote the suite, and NVIDIA contributed to the suite and scoring rubric. The research page thanks Analog Devices, MediaTek, and NVIDIA for technical advice. According to the company's funding announcements, MediaTek and Micron are also investors in ChipAgents. NVIDIA's own announcement says ChipAgents is working with it on the benchmark and reports no results.
A benchmark where the best generic agent scores 16.6% makes a specialised agent platform look necessary, and the test's author sells one. All five models ran through OpenCode, a generic agent shell, so ChipAgents' own product does not appear in the table. The ranking is a test of generic models.
The constraints are calibrated against a reference design built by ChipAgents' lead digital designer with AI assistance, and that design meets every one of them. The constraints are therefore known to be reachable, and 16.6% is the distance still to cover. The flip between GPT-6 Astra and Fable can be read straight from the published table.
As of 14 October 2026, the table lists five models and says more results are coming soon. A live leaderboard for industry partners and an expanded task set will follow. When they arrive, the first check is whether the model that passes 65% of tasks keeps its PPA lead over the model that passes 85%.
The traces point to late RTL revisions and slow tool loops as agent habits rather than gaps in the data. Those habits are unlikely to disappear each time the benchmark is refreshed, so the PPA column should remain low on the expanded set.