BIGGER PICTURE Learning & work
A working web server in 71 minutes instead of 161
The experiment gave developers a specific job: build an HTTP server in JavaScript. A randomly selected group received an AI coding assistant; automated tests decided whether the result actually worked.

- Task
- One JavaScript HTTP server
- Mean completion time
- 71.17 vs 160.89 minutes
- Measured difference
- 55.8% less time among completers
How the idea works
Specify a working server
Requirements and automated tests establish completion.
Draft code in context
The assistant suggests code while the developer works.
Run, inspect, compare
Human review and tests determine success and elapsed time.
The finish line was executable
In the controlled experiment, developers had to produce code that passed a defined test suite. This gave the researchers a clearer outcome than asking people whether AI felt helpful. The task was small enough to finish in one sitting but included the practical work of translating instructions into a functioning server.
Sources: [1]
Suggestions stayed inside the developer’s workflow
GitHub Copilot proposed code inside the editor. The person still had to choose what to accept, fit the pieces together and run the result. Microsoft’s research account frames it as an AI pair programmer. The potential benefit is reducing the effort spent producing familiar code patterns while a developer remains responsible for the implementation.
Sources: [3]
Time fell substantially in this task
Among participants who completed the task, the assisted group averaged 71.17 minutes versus 160.89 minutes without Copilot: a 55.8% reduction. The study did not establish a statistically significant increase in the completion rate. GitHub’s separate developer surveys reported perceived benefits, but those opinions are not the experimental result.
Make the promise as testable as the code
This does not mean every software project becomes twice as fast. Maintaining an unfamiliar system, checking security or deciding what to build can dominate real development. The transferable idea is to start with bounded tasks, measurable acceptance tests and human review. That makes it possible to discover where assistance genuinely saves effort.
The original sources
- Peng and colleagues, arXivThe Impact of AI on Developer Productivity: Evidence from GitHub Copilot
- GitHubResearch: quantifying GitHub Copilot’s impact on developer productivity and happiness
- Microsoft ResearchThe Impact of AI on Developer Productivity