Back to Bigger Picture

BIGGER PICTURE Learning & work

A working web server in 71 minutes instead of 161

The experiment gave developers a specific job: build an HTTP server in JavaScript. A randomly selected group received an AI coding assistant; automated tests decided whether the result actually worked.

AI-generated illustration: A developer reviews abstract code on a laptop while assembling a small stack of glass blocks.
Glass building blocks symbolise suggestions that a human developer selects and checks. The scene and interface are imagined, not an actual coding-trial participant or product screenshot.AI-generated illustration of the idea, not a photograph of the actual people, equipment or event.
Task
One JavaScript HTTP server
Mean completion time
71.17 vs 160.89 minutes
Measured difference
55.8% less time among completers

How the idea works

  1. Specify a working server

    Requirements and automated tests establish completion.

  2. Draft code in context

    The assistant suggests code while the developer works.

  3. Run, inspect, compare

    Human review and tests determine success and elapsed time.

The finish line was executable

In the controlled experiment, developers had to produce code that passed a defined test suite. This gave the researchers a clearer outcome than asking people whether AI felt helpful. The task was small enough to finish in one sitting but included the practical work of translating instructions into a functioning server.

Sources: [1]

Suggestions stayed inside the developer’s workflow

GitHub Copilot proposed code inside the editor. The person still had to choose what to accept, fit the pieces together and run the result. Microsoft’s research account frames it as an AI pair programmer. The potential benefit is reducing the effort spent producing familiar code patterns while a developer remains responsible for the implementation.

Sources: [3]

Time fell substantially in this task

Among participants who completed the task, the assisted group averaged 71.17 minutes versus 160.89 minutes without Copilot: a 55.8% reduction. The study did not establish a statistically significant increase in the completion rate. GitHub’s separate developer surveys reported perceived benefits, but those opinions are not the experimental result.

Sources: [1][2]

Make the promise as testable as the code

This does not mean every software project becomes twice as fast. Maintaining an unfamiliar system, checking security or deciding what to build can dominate real development. The transferable idea is to start with bounded tasks, measurable acceptance tests and human review. That makes it possible to discover where assistance genuinely saves effort.

Sources: [1][3]

The original sources

  1. Peng and colleagues, arXivThe Impact of AI on Developer Productivity: Evidence from GitHub Copilot
  2. GitHubResearch: quantifying GitHub Copilot’s impact on developer productivity and happiness
  3. Microsoft ResearchThe Impact of AI on Developer Productivity

What to explore next

Twenty-five minutes back from the science planning pileA warning before blood pressure falls, with a plan for what to do
Explore all 99 applications