A recent study found that an AI chatbot-powered software company could develop software in under seven minutes for less than $1 in costs, on average.
ILYA SEDYKH/Getty
Researchers tasked an AI-powered tech company to develop 70 different programs in a new study.They found AI could develop software in under 7 minutes for less than $1 in costs, on average.AI bots were assigned roles and were able to talk, make logical decisions, and troubleshoot bugs.
AI chatbots like OpenAI’s ChatGPT can operate a software company in a quick, cost-effective manner with minimal human intervention, a new study has found.
The findings come after researchers published another study in which AI agents powered by large language models were able to run a virtual town on their own.
In the recent paper, a team of researchers from Brown University and multiple Chinese universities conducted an experiment to see if AI bots powered by a version of ChatGPT’s 3.5 model could complete the software development process without prior training.
To test this, researchers created a hypothetical software-development company called ChatDev. Based on the waterfall model — a sequential approach to creating software — the company was broken down into four different stages, in chronological order: designing, coding, testing, and documenting.
From there, researchers assigned AI bots specific roles by prompting each one with “vital details” that described the “designated task and roles, communication protocols, termination criteria, and constraints.”
Once the researchers gave the AI bots their roles, each bot was allocated to their respective stages. The “CEO” and “CTO” of ChatDev, for instance, worked in the “designing” stage, and the “programmer” and “art designer” performed under the “coding” stage.
During each stage, the AI workers chatted with one another with minimal human input to complete specific parts of the software-development process — from deciding which programming language to use to identifying bugs in the code — until the software was complete.
The researchers ran the experiment across different software scenarios, and applied a series of analyses to them to see how long it took ChatDev to complete each type of software, and how much each one would cost.
Researchers, for example, tasked ChatDev to “design a basic Gomoku game,” an abstract strategy board game also known as “Five in a Row.”
At the designing stage, the CEO asked the CTO to “propose a concrete programming language” that would “satisfy the new user’s demand,” to which the CTO responded with Python. In turn, the CEO said, “Great!” and explained that the programming language’s “simplicity and readability make it a popular choice for beginners and experienced developers alike.”
After the CTO replied with, “Lets get started,” ChatDev moved on to the coding stage, where the CTO asked the programmer to write a file, followed by the programmer asking the designer to give the software a “beautiful graphical user interface.” The chat chain repeated at each stage until the software is developed.
After assigning ChatDev 70 different tasks, the study found that the AI-powered company was able to complete the full software development process “in under seven minutes at a cost of less than one dollar,” on average — all while identifying and troubleshooting “potential vulnerabilities” through its “memory” and “self-reflection” capabilities.
The study said 86.66% of the generated software systems were “executed flawlessly.”
“Our experimental results demonstrate the efficiency and cost-effectiveness of the automated software development process driven by CHATDEV,” the researchers wrote in the paper.
The researchers didn’t immediately respond to Insider’s request for comment before publication.
The study’s findings highlight one of the many ways powerful generative AI technologies like ChatGPT can perform specific job functions. Since the AI chatbot came out last November, workers across industries have used it on the job to save time and boost productivity.
Coders, in particular, may find generative AI tools beneficial to their personal and professional lives. Daniel Dippold, a Berlin-based coder, used ChatGPT to develop a program that helped him find an apartment, and Amazon employees were found to use ChatGPT for software development.
Nevertheless, the study isn’t perfect: Researchers identified limitations, such as errors and biases in the language models, that could cause issues in the creation of software. Still, the researchers said the findings “may potentially help junior programmers or engineers in the real world” down the line.