Increasing developer productivity with generative Artificial Intelligence

Technology leaders looking to accelerate software development can expect significant time savings from generative artificial intelligence. But they will need more than just tools to harness the full potential of this revolutionary technology.

Our recent empirical study has shown that generative AI-based tools provide impressive speed gains for many common developer tasks (see the "About the Study" sidebar). Documenting code functionality in terms of maintainability (which takes into account how easily code can be improved) can be done twice as fast, writing new code almost twice as fast, and optimizing existing code (aka code refactoring) almost two-thirds as fast (Figure 1). With the right skill development and enterprise support tools, these speed gains can be translated into productivity gains that surpass previous engineering productivity gains due to both new tools and processes.

However, despite the possible dramatic increase in productivity, the time savings, as our study showed, can vary significantly depending on the complexity of the task and the developer's experience. Time savings amounted to less than 10% on tasks that developers considered complex, for example, due to unfamiliarity with the required programming framework. A similar result was observed among developers with less than a year of experience: in some cases, tasks with the use of tools took 7-10% more time for novice developers than without them.

When developer and tool worked together, quality was not compromised in favor of speed. Code quality in terms of bugs, maintainability, and readability (important for reuse) was marginally higher in code written with AI. However, participant feedback indicated that developers actively iterated with the tools to achieve this quality, suggesting that this technology is best suited to supplement developers rather than replace them. Ultimately, to maintain code quality, developers need to understand what attributes make up quality code and prompt the tool for the desired results.

Taken together, these findings suggest that to maximize productivity gains and minimize risk when implementing generative AI-based tools, engineering managers need to take a structured approach that includes generative AI education and training, use case selection, employee development, and risk control. This article will highlight where generative AI has shown promise in our research, what tasks require developer expertise, and what engineering managers can do to ensure the most effective use of this emerging technology.

What generative AI excels at

In our study, we challenged developers with several common tasks that software development teams tackle on a regular basis: refactoring code into microservices to improve maintainability and reuse, creating new application functionality to improve customer experience, and documenting code features to facilitate subsequent changes.

According to our study, tools based on generative AI can achieve significant productivity gains in four key areas:

• Accelerating manual and repetitive work. Generative AI can handle routine tasks such as automatically populating standard functions used in coding, completing code statements as the developer types, and documenting code functionality in a given standard format at the developer's prompt. Thus, these tools can free developers to solve more complex business problems and accelerate the creation of new software features.
• Launching the first draft of new code. When faced with a blank screen, developers using generative AI-based tools can request suggestions by typing a prompt in a separate window or in the integrated development environment (IDE) they use to create software. Developers who did this noted that the generative AI-based tools provided useful code suggestions. This allowed them to avoid "writer's block" and get to work more quickly. According to one participant, these tools allow developers to get into the "flow" more quickly.
• Accelerating updates to existing code. Participants also reported that when using these tools with effective hints, they can make changes to existing code faster. For example, to spend less time adapting code from an online coding library and improving code already written, developers copy and paste it into a tooltip and submit iterative requests that require the tool to make adjustments based on criteria they specify.
• Increased ability of developers to solve new problems. Although developers' time savings when using AI-based generative tools were more modest for complex tasks, our study still found benefits: the technology can help developers quickly learn an unfamiliar code base, language, or framework needed to get the job done. Moreover, when developers are faced with a new task, they can turn to these tools for help they might otherwise get from an experienced colleague: e.g., explaining new concepts, synthesizing information (e.g., comparing and contrasting code from different repositories), providing step-by-step guidance on how to use a framework so they can get the job done. For example, developers who used generative AI tools to complete complex tasks were 25-30% more likely than those who did not use such tools to complete these tasks within the allotted timeframe (Figure 2).

The benefits are not just limited to increased productivity. The study found that equipping developers to be their most productive also significantly improves the developer experience, which in turn can help companies retain and attract the best talent. Developers using generative AI-based tools are more than twice as likely to report overall happiness, satisfaction, and flow state (Figure 3). They attribute this to the tools' ability to automate chores that were distracting them from more enjoyable tasks and provide information faster than searching for solutions on various online platforms.

Which tasks require the competence of a developer

Generative AI technology is capable of many things, but our research shows that the quality of the tools depends solely on the skill of the engineers who use them. Feedback from participants revealed three areas where human control and involvement are critical:

• Examining the code for bugs and errors. Participants in the study noted that sometimes the generative AI-based tools made incorrect coding recommendations and even introduced errors into the code. In one task, a developer noted that he had to enter multiple prompts to correct the tool's erroneous assumption and get an answer to a question. In another case, the developer said that he had to "spoon-feed" the tool to debug the code correctly.
• Bringing in organizational context. While off-the-shelf generative AI-based tools know a lot about coding, they cannot know the specific needs of a particular project and organization. Such knowledge is critical when coding so that the final software product can easily integrate with other applications, meet the company's performance and security requirements, and ultimately meet the needs of end users. As study participants noted in their feedback, it is up to the software developers to provide context to these tools with cues, including how and by whom the code will be used, the types of interfaces and other systems with which the program will interact, the data used, and more.
• Navigating complex code requirements. According to participants' feedback, generative AI tools are better suited for simple tasks, such as optimizing a piece of code, than complex ones, such as combining multiple frameworks with disparate code logic. One participant explained that in order to obtain a usable solution that met multifaceted requirements, he first had to either manually combine components or break the code into smaller segments. As another participant explained, "[generative AI] is least useful when the problem becomes more complex and the big picture needs to be considered."

What do these findings mean for technology leaders?

Given these findings, what can technology leaders do to translate time savings and quality improvements into real productivity gains while minimizing the risks of using generative AI in software development? The experience of our study participants suggests that it should start with four priorities: upskilling, finding promising use cases, planning for skill shifts, and managing risk.

Providing developers with training and coaching on generative AI

In order for developers to effectively use this technology in their day-to-day work, they will likely need a combination of training and coaching. Initial training should include best practices and hands-on exercises on how to input natural-language cues into tools, often referred to as cue development. In addition, workshops should provide developers with an understanding of the risks of generative AI, including any industry concerns related to data privacy and intellectual property, as well as best practices for reviewing AI-generated code for design, functionality, complexity, coding standards, and quality, including how to distinguish good recommendations from bad ones from tools.

For developers with less than a year of experience, the study also points to the need for additional courses in programming fundamentals - such as coding syntax, data structures, algorithms, design patterns, and debugging skills - to achieve the productivity gains seen in those with more experience.

Once developers start using the tools in their day-to-day activities, their development should continue with ongoing coaching from senior team members and community building, such as dedicated online channels and team meetings to share case studies. Such work promotes continuous learning, spreading best practices throughout the organization, and early identification of problems. In our study, participants noted that as they generated more prompts and shared their experiences with each other, the quality of their prompts improved.

Finding advanced use cases beyond code generation

While there is tremendous hype in the industry around generative AI's ability to generate new code, our research shows that the technology can impact many common developer tasks, including refactoring existing code, which can allow managers to contribute to traditionally resource-intensive modernization activities that are often sidelined due to time constraints. For example, if generative AI-based tools can help teams quickly refactor a legacy application, they can refocus their time on eliminating the backlog of enhancements that have been languishing on the company's to-do list or on improving the architectural performance of the entire software platform.

Deploying new use cases requires careful toolkit evaluation, as a flurry of new generative AI tools are emerging on the market, and different tools excel in different areas. Our study shows that using multiple tools can be more beneficial than a single tool. In our study, participants had access to two tools: one that used a baseline model trained to respond to user cues, and one that used a fine-tuned baseline model trained specifically for code. Participants noted that the first tool, with its conversational capabilities, did an excellent job of answering questions when refactoring code. The second tool, they said, was better at writing new code because of its ability to connect to the integrated development environment and suggest code based on the descriptive commentary they noted in the document. However, when developers used both generative AI tools for a particular task, rather than just one of them, they gained an additional 1.5 to 2.5 times improvement in time.

Planning for a change of qualification

As developer productivity increases, managers must be prepared to shift employees to more important tasks. Establishing a productivity baseline and then continuously measuring its improvement can reveal new capabilities as they emerge in the organization. Managers should consider how to capitalize on this additional capacity and what needs to be done to address emerging skill gaps. For example, they might use their talent to expand new business or update existing products more frequently. Such assignments will require new skills in software design and architecture.

Ensure risk control

Given the speed at which developers can write and update code using these tools, it's easy to imagine the snowball of problems that can arise from a bug in the code or a data problem, for example. When updating a management system, managers should consider the following potential risks:

• Data privacy and third-party security, such as the ability of developers to disclose sensitive information when requesting tools
• legislative and regulatory changes, including changes to the European Union's General Data Protection Regulation (GDPR) and other regulations restricting the use of technology
• Behavioral AI vulnerabilities, including the consequences that could arise if attackers put malicious or faulty code out in the open to affect the training of large language models or infiltrate organizations
• ethical and reputational issues that may arise from using a piece of code copyrighted by another organization, or from disputes over the ownership of code created by tools
• Security vulnerabilities that could appear in code generated by artificial intelligence and put systems (and the organization) at risk

Generative AI has the potential to transform software development in ways that no other tool or process improvement has been able to do. Using the current class of tools based on generative AI, developers can complete tasks twice as fast, and that's just the beginning. As the technology matures and is integrated into tools throughout the software development lifecycle, it is expected to further improve the speed and even the quality of the development process. However, as our research shows, tools alone are not enough to unlock the technology's full potential. A structured approach that includes generative AI education and training, use case selection, staff development, and risk control can provide a solid foundation for organizations to realize the promise of generative AI to deliver high performance and unprecedented software innovation.