AutoCAD Bench

What can you do with it?

Evaluate agents that reconstruct dimensioned references through visible mouse and keyboard input, including AutoCAD's in-app command bar. Inspect 21 two-dimensional and 29 three-dimensional tasks; 3D deliverables require model-space geometry and the specified Layout1 presentation.

What should you know before using it?

Operators must supply a licensed AutoCAD Windows worker; no ready-to-use hosted desktop is provided. The pinned contract specifies AutoCAD 2019, Windows Server 2022, broker v10 and evaluator v4; broker v11 remains a candidate. Gold files are scorer-side and model-hidden, not an access-gated public dataset. The pinned scorer contract makes deterministic final-DWG scoring authoritative, while the publisher report describes a vision judge as ground truth; these protocols must not be merged. The pinned contract still requires manual unit and AutoCAD-open audits of gold drawings before release-grade scoring; complete evaluator-cache coverage is a separate claim. No repository code/data license or verified publication date for the report was established. No runtime was tested.

What is verified here?

Runtime: Python 3.12; pinned worker AutoCAD 2019 R23.0.46.0.0 on Windows Server 2022 build 20348. The AWS reference backend requires the operator's image, infrastructure and storage; other backends implement the published plugin contract. Vision auditing has separate model/API requirements.

License note: No repository-wide code or corpus license was present at the inspected revision. AutoCAD is separately licensed software.

This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.

Resource type

Environment + benchmark

Provenance

Development provenance not established

Reviewed

2026-10-16

Platforms

Windows