AI coding agents (like Claude, Copilot, etc) are a popular and powerful way to build software. While they work extremely well for building prototypes and smaller applications, programmers are struggling to use them for large software systems. My last blog post highlighted parts of systems design and implementation that cannot be replaced by AI. This post focuses on parts of systems building that could leverage coding agents and why they work poorly for systems building.

Unlike prototypes and applications, systems software is critical infrastructure and thus must be reliable and secure. For systems software that must behave in precise and correct ways, it is not enough for systems programmers to specify what they want to coding agents to do using natural language prompts. They usually must also review large amounts of generated code, find bugs in code they did not write, and manage failures caused by errors in—or attacks targeting—code generated by coding agents. As a result, building systems with coding agents often requires more time and energy while also producing a sub-par system.

The reason for this gap is that we are training coding models to use tools fundamentally designed for humans. Instead, we must meet the models halfway by modifying our tools to be shared between agents and humans. I argue that researchers must rethink our software engineering tools (i.e., programming languages, compilers, debuggers, etc.) for coding agents.

Background

Fundamentally, any type of computing that we do is a pipeline between the programmer’s intent to a concrete specification of what the computer should execute. We can consider this an abstract model that, for the last 40 years or so, has flowed from high level programming language to some type of compiler (with an intermediate representation) to assembly (ignoring the various types of macros and microcode in the middle) to running on a CPU and peripherals.

Coding agents introduce a new stage of the pipeline: one that translates from a natural language prompt to code usually in a high level programming language. High-level programming languages and their associated tools were not designed to be generated from a natural language prompt.

Let us evaluate natural language prompts like those used by Claude as a method of specifying software behavior, similar to programming languages (i.e., Rust), and verification (i.e., Verus). We evaluate across four axes and use Rust and Verus as relatively optimal examples of the properties that we care about.

1. Precision - the amount of undefined behavior allowed by the method of specification.
2. Determinism - the likelihood that the specification will generate the same system each time.
3. Correctness - the likelihood that the generated system will have bugs that deviate from the intended behavior.
4. Ease of Use - the amount of time and effort it takes the programmer to specify the system’s behavior.

	Precision	Determinism	Correctness	Ease of Use
Rust	Average	High	Average	Average
Verus	High	High	High	Low
Claude	Low	Low	Low	High

Rust and its associated compiler and runtime are average for every metric except for determinism. The Rust compiler and runtime are likely to generate the same instructions and execution every time, but Rust does have undefined behavior which makes it imprecise. Rust code is mostly correct but unlikely to be entirely bug-free, and the time and effort to write a program in Rust is about average for a programming language. Other programming languages might deviate slightly on each metric but generally, we’ve settled on programming languages of various forms as the best trade off of ease of use and other metrics for specifying the behavior of software.

More recently, verification has become a popular technique for increasing the correctness of programs. Verification requires the programmer to write a separate specification that formalizes the behavior of the software, then proves that the code conforms to that specification. Verification significantly reduces the likelihood of bugs (although a bug in the specification and code can occur); however, it requires much more time and effort from the programmers.

In comparision, LLMs perform poorly on every axes except for ease of use. This is well understood given most people’s experience with Claude, Copilot, etc. While large amounts of code are easy to generate, it is hard to generate code that: 1) is precisely what the programmer would have written, 2) is reliably the same every time, and 3) totally bug-free and correct.

For now, we assume that the programmer does not carefully read every line of generated code (much like I don’t read the assembly generated by my Rust code). Regardless, reviewing generated code decreases the ease of use without significantly improving any of the other metrics. For example, we can assume that the programmer is not likely to find all the bugs, as previous work has shown that it is very difficult even for experts to find bugs in others’ code. Likewise, the programmer might achieve slightly more precision through successive iterations of prompt engineering but there is a limit to how precise natural language can be when describing a software and LLMs cannot be relied on for determinism.

Research: Rethinking Software Engineering for AI Coding

Thus our goal is to increase the use of LLMs for specifying system behavior on one or more of these axes without reducing the ease of use. I argue that we cannot achieve this by only training more powerful models. Instead, we must change our programming tools designed for people to work better with AI.

There is already much work on improving correctness, however it often trades off ease of use for correctness. For example, adding Verus specifications means that either the programmer has to write the spec or understand both the LLM-generated code and spec. Thus we should be considering how to design verification tools that work better with natural language models and coding agents.

• RQ: Can we increase correctness without requiring the programmer to understand variables in the code (i.e., still using natural language)?

While we cannot increase the determinism of LLMs, we could decrease our reliance on determinism. Acceptability-oriented programming and approximate computing shows some potential in this direction. For example failure-oblivious Rust might be more resilient to non-determinism in cases where it doesn’t violate the acceptability envelope.

• RQ: Can we do other things with the language runtime or verification system to reduce the dependence on determinism for correctness?

Techniques developed for approximate computing might also work; however, approximate computing was largely about starting with precise programs and reducing that precision, rather than starting with imprecise programs, so we would need different techniques to refine rather than relax the program’s precision.

• RQ: Can we increase the precision with a more formal natural language specification? Can we train that into the model or add it to the context?