41.3% of design items clear the safety threshold, yet complex engineering projects remain bogged down by tightly coupled constraints.
Authors | Zheng Jiamei Cen Feng
Editor | Cen Feng
In engineering, 91% correctness can sometimes be indistinguishable from zero ability to deliver.
A cable tray crossing fire-protection piping at the top of a server room can render dozens of pages of correct calculations worthless as a deliverable. The current-carrying capacity was calculated correctly, the voltage drop met the requirements, and the equipment selection had a sound basis, but once the design entered the three-dimensional space, several systems competed for the same clearance, leaving the engineers with no choice but to send the design back.
AI agents can rapidly complete local calculations, but engineering projects do not operate within the boundaries of calculation problems: one change ripples through another, and results that have just been verified may be invalidated by subsequent revisions.
Google and Mercor’s recently released Physical-Engineering’s Last Exam (PLE) quantified this gap through 85 tasks in mechanical, electrical, civil, and structural engineering.
Across 56 hidden tests, Claude Opus 5.5 paired with Claude Code scored 48.3%; in co-creation tasks, it met an average of 91% of individual constraints, but only 41.3% of delivered design items met all mandatory safety and feasibility requirements. More troubling still, the remaining 9% often cannot be solved simply by further improving the model’s computational capabilities.Because the requirements in engineering design are interrelated, solving one problem may give rise to a new one. The technical design of PLE reveals precisely this dilemma: once local capabilities are strong enough, what exactly is still missing from engineering agents?
01
Engineering Agents Are Encountering a Feasible-Region Bottleneck
Many earlier agent evaluations used item-by-item cumulative scoring, awarding higher ratings whenever more requirements were completed. This approach is suitable for measuring knowledge comprehension and isolated execution ability, but it obscures a key type of failure in engineering design: a model may satisfy the vast majority of the requirements, yet the one remaining requirement may be exactly what determines whether the solution can be put into use.
Across 41 human–agent co-creation tasks, PLE places engineering constraints and design optimization within a single evaluation framework. Of these tasks, 37 use constraint scoring. Under a unified scoring mechanism, 40% of the scoring space depends on the proportion of individual checks passed, while the remaining 60% relates to cost or performance optimization. The optimization component, however, has one prerequisite: all hard safety and feasibility constraints must be passed.
The research team calls this mechanism Hard-Constraint Gating.
There is a critical design feature in this mechanism. Even if a proposal can substantially reduce material costs, it earns no points in the cost-optimization component if it violates even one hard constraint. The system still records the checks it has completed, but it does not allow economic benefits to offset engineering shortcomings.
PLE also provides a compliant but not fully optimized baseline design, as well as a lower bound for compliant costs. The former is used to determine whether the agent has achieved genuine optimization gains, while the latter provides a benchmark for measuring cost improvements. Thus, a model can neither easily satisfy the specifications by using excess equipment nor produce attractive cost figures by sacrificing feasibility.
This scoring approach makes a long-standing issue clear: the optimization space in engineering design is bounded by constraints, and those constraints are often interrelated.
For example, in cooling-system design, pipe diameter determines hydraulic resistance, hydraulic resistance affects the pump operating point, and the actual flow rate then determines heat-transfer capacity. If an agent reduces the pipe diameter based solely on material costs, it may subsequently require a pump with a higher head, shifting the original savings instead to equipment and operating energy consumption.
In engineering, the range of designs in which all constraints hold simultaneously is called the feasible region. It contains continuously varying parameters and is also constrained by discrete conditions such as equipment models and standard sizes. Whenever the model changes one variable, the range available to the other parameters may change.
This imposes on agents a form of reasoning different from ordinary reasoning. After generating candidate solutions, they must also determine which current conditions are already close to their permissible boundaries and which constraints the next modification will affect. In engineering optimization, constraints that further restrict the design space in this way are typically called active constraints.
PLE does not directly provide agents with a feasible-region search algorithm, but Hard-Constraint Gating makes this capability gap measurable. A 91% individual-constraint satisfaction rate indicates that the model can already handle most local requirements; the remaining difficulty lies in making multiple conditions hold simultaneously and continuing to optimize once they are satisfied.
But when design moves from a one-off solve into a long-running, multi-round revision process, the problem expands further: constraints that were previously satisfied may no longer hold after the next operation.
02
Long-running workflows are becoming a state consistency problem
PLE's task environment does not organize the design conditions into a complete specification. The 85 tasks contain 5,170 files in total, averaging about 61 files per task, including PDFs, spreadsheets, emails, CAD files, and BIM models, with obsolete design versions mixed in.
The Agent needs to identify the currently effective parameters and specifications in these materials, and then use them for calculations and deliverables. The difficulty is that engineering files have numerous cross-references. If the capacity in an equipment schedule is changed, a calculation document may continue to use the old value; if a design drawing updates the layout, the bill of materials may not be adjusted in sync. Even if the model can accurately read every file, it may combine information from different versions.
For a long-running Agent, this means the task state has gone beyond what can be managed in an ordinary conversational context.
In software development, after a module is modified, dependencies can be used to determine which code needs to be rebuilt. Engineering design needs a similar capability, but its objects are more complex. It involves not only references among files, but also physical parameters, spatial relationships, and the conditions under which codes and standards apply.
One 3D MEP coordination task in PLE requires the Agent to lay out cable trays, busbars, chilled-water piping, and fire sprinklers in a machine room containing 128 equipment cabinets. The system needs to obtain locations from floor plans, read dimensions from the equipment schedule, determine elevations from section information, and ultimately output a 3D model that passes collision and clearance checks.
Here, the technical difficulty arises during continuous revision. Suppose the Agent adjusts the cable tray route to eliminate a spatial conflict. As the cable length increases, the previous voltage-drop calculation may no longer be valid. If the equipment position is then adjusted, the cable tray route and adjacent pipelines must be revised again.
Each operation changes the design conditions used in the next step, and checks already completed may also need to be rerun. Therefore, engineering Agents need to maintain design state with dependencies.
Under this architecture, equipment, piping, components, and calculation results have explicit object identities, and parameters are linked to source files, revision versions, and applicable operating conditions. When a design variable changes, the system can identify affected calculations and deliverable documents and recheck only the relevant portions.
This is backed by incremental computation and state invalidation propagation. For example, after equipment capacity changes, related power distribution circuits need to be recalculated, while unaffected structural calculations can be retained. For mutually interacting parameters such as pipe-network flow and resistance, the system also needs to invoke a solver to iterate until the relevant physical conditions are valid again.
Design modifications also cannot immediately overwrite existing results. A more sensible approach is to first save a candidate version and accept the modification only after the related checks pass. If a check fails, preserve the problem location and relevant parameters, then readjust or roll back.
Such state management can reduce redundant computation and prevent outdated design conclusions from carrying over into new deliverables. PLE’s experiments showed that when the same model ran the same task three times under identical conditions, the average spread of per-task scores reached 15.4 percentage points.
The study did not attribute the variability to any single type of error, but the results showed that the execution path can significantly affect delivery quality. Merely expanding the context or allowing more tool calls does not automatically resolve synchronization issues between design versions and computation results.
Long-horizon capabilities therefore need to evolve from remembering the sequence of operations to maintaining a valid design state. To determine whether a state modification can be accepted, a validation mechanism capable of independently checking engineering results is needed.
03
Verifiers Will Become the Core Infrastructure for Engineering Agents
PLE uses two task-construction pathways for its validation mechanism. Of the 85 tasks, 44 were directly authored by professional engineers, with deliverables assessed against expert scoring criteria, while the other 41 used the Creator–Solver mechanism, allowing Agents to participate in constructing the task environment, reference design, and executable validation programs.
In the latter mechanism, engineers first specify real-world work requirements. Based on those requirements, the Creator Agent searches for engineering materials and generates the task files, reference deliverable, and verifier. An independent Solver Agent then attempts the task in an isolated environment without access to the reference answer or scoring program.
The research team typically gives the Solver three attempts and then has the Creator analyze the execution traces. If the Solver finds a shortcut that bypasses engineering calculations, or if the task itself lacks necessary constraints, the Creator must revise the task and validation rules. After completing these steps, professional engineers also check whether the task requirements, reference design, and verifier are consistent.
This mechanism addresses an easily overlooked problem in engineering evaluation: the validation program itself may contain loopholes.
If only easily extracted numerical values are checked, an Agent may produce a deliverable that appears to meet the requirements on the surface but is actually incomplete. Creator–Solver tests for these loopholes through independent solving, enabling the task requirements and scoring criteria to be repeatedly calibrated.
PLE also introduces a technical detail here: constraints in co-created tasks must identify their sources and distinguish mandatory safety and feasibility requirements from softer requirements such as document formatting. The Solver must also list the specification clauses and specific numerical values it used, and the validation program checks them against the relevant documents.
This amounts to converting requirements originally scattered across engineering specifications into conditions that can be checked programmatically. Leiphone(official account: Leiphone)
An engineering constraint must include its applicable target, parameter range, calculation conditions, and source basis. Only by making this information explicit can the verifier determine whether a deliverable meets the requirements.For example, the same cable specification may correspond to different capacity limits under different installation conditions. Directly extracting a single value from the specification is not sufficient to perform a reliable check.
Going a step further, this approach also needs to address the problem of matching objects across files. The geometric paths in CAD, component properties in BIM, and equipment parameters in spreadsheets may describe different parts of the same system. If the validation program cannot establish object associations, it can only inspect the files separately, making it difficult to determine whether the design results are consistent with one another.
Therefore, a production-grade validator needs to identify specific engineering objects and link physical calculations, specification requirements, and deliverable files. This way, when a check fails, the system can identify the object that violates the requirement, the corresponding conditions, and its current parameters, rather than merely returning a failure result.
This will further change how the Agent solves problems. If the validator can provide feedback on specific constraint violations, the Agent can regenerate a localized design around the relevant parameters and run the checks again.
For continuous parameters, the magnitude of the exceedance can assist with adjustment; for discrete variables such as equipment models, candidate configurations that meet the requirements can be reselected. As validation results are continuously fed back, the design search will gradually converge on the feasible range.
It should be noted that PLE's current Solver cannot access the hidden validator during execution and does not receive real-time scores. It has already developed the ability to independently inspect complex deliverables, but directly integrating the validator into the Agent's design loop requires additional system architecture.
This includes encapsulating the checking procedures as callable services, supporting revalidation based on the scope of changes, and binding the check results to specific design versions. This way, after the Agent modifies a localized design, it can determine which conditions need to be recalculated and which existing results remain valid.
The validator itself also needs to use different inspection methods. Physical calculations, numerical bounds, and geometric collisions are best handled by deterministic programs or specialized solvers, while relatively open-ended design descriptions can be reviewed with the assistance of LLM-Judge. PLE rescored 1,159 runs involving language-model adjudication. The item-by-item agreement rate for the same model reached 98.7%, although differences remained after changing the adjudication model. Leiphone
For practical engineering applications, validation results should also retain the calculation inputs, specification sources, and program versions so that engineers can review each key conclusion.
As these capabilities are established, the validator can gradually move from a post-delivery inspection tool into the Agent's actual problem-solving workflow. The model is responsible for proposing and adjusting designs, specialized programs are responsible for calculations and determinations, and the validation results determine which modifications can enter the next valid design version.
04
Physical AI Needs a Sign-Off-Ready Design Loop
This architecture has direct implications for the development of robots and industrial equipment. After the linkage structure of a robotic arm is modified, its stiffness, vibration characteristics, and actuator loads may be affected, requiring the existing simulation results to be rechecked. After the actuator or battery configuration changes, the thermal, assembly, and electrical designs must also be updated in sync.
If engineering agents can integrate with CAD, CAE, and product lifecycle management systems; automatically identify affected calculations when designs are modified; invoke specialized tools for validation; and preserve a complete record of the basis for every revision, then AI-generated designs would be eligible to enter the existing engineering review process.
PLE is currently an evaluation conducted in a controlled environment and cannot replace manufacturing tests or formal sign-off. However, it has moved the assessment of engineering agents’ capabilities closer to real-world delivery.
In the future, competition in Physical AI will increasingly hinge on whether models can consistently generate designs that satisfy constraints while providing engineers with calculations and validation evidence they can review for every critical modification.
Reference link:https://www.mercor.com/blog/introducing-physical-engineering-s-last-exam-ple-with-google-cloud-ai-research/
Original article by Leifeng Network. Unauthorized reproduction is prohibited. See the Reposting Guidelines for details.Reposting Guidelines.
