AI is accelerating software development to breakneck speeds, but it's hard to measure it

Software development and deployment cycles continue to accelerate, thanks in large part to artificial intelligence (AI) that can generate code and make suggestions. But even with this hyperproductivity, IT managers and business leaders remain puzzled about how to measure the impact of AI.

These are the results of a new survey of 5,315 executives and IT professionals by GitLab on software development productivity and DevSecOps. Developing with artificial intelligence has become the norm - 78% of respondents said they are currently using artificial intelligence in software development or plan to do so in the next two years, up from 64% in 2023, according to the survey. In addition, 67% of respondents say the software development lifecycle is now mostly or completely automated.

The adoption of artificial intelligence can accelerate software development to dazzling speeds. Shockingly, the majority of executives (69%) say they are creating software twice as fast as they did last year. In addition, IT professionals are taking more time to get things done. More than half (52%) say it takes more than three months to bring on new developers - up from 42% a year ago.

C-level executives are much more wary of AI than their employees. Most executives (56%) believe that introducing AI into the software development lifecycle is risky in terms of data privacy and security. In contrast, only 40% of professionals have similar concerns.

Executives are also more concerned about AI skills, with 35% citing a lack of appropriate skills to apply AI or interpret AI results as a barrier to using AI. Only 26% of IT professionals agree.

Respondents who use AI for software development (43%) are much more likely than those who don't use AI (20%) to say that developer training typically takes less than a month. The survey found the same effect for DevSecOps platform usage, with 44% of respondents currently using the platform saying developer training takes less than a month, compared to 20% of respondents not using the platform.

The study also found that the most popular use of AI in IT companies is to generate code, as well as provide explanations of how it works. In terms of future work, the largest number of people would like AI to help them achieve predictive and productivity metrics.

How AI is being used in development

• Code generation and code suggestion/addition: 47%
• Explanations of how a particular piece of code works: 40%
• Summary of code changes: 38%
• Chatbots that allow users to ask documentation questions using natural language: 35%
• Summary of code reviews: 35%

What IT professionals and executives want to see in artificial intelligence

• Predicting performance metrics and identifying anomalies throughout the software development lifecycle: 38%
• Explanation of how the vulnerability can be exploited and how to fix it: 37%
• Chatbots that allow users to ask documentation questions using natural language: 36%
• Suggestions for who can review code changes: 34%
• Correction of failed pipeline works: 31%

Software supply chain security is a potential weakness: 67% of professionals report that a quarter or more of the code they work on comes from open source libraries. At the same time, only 21% of organizations currently use a software bill of materials (SBOM) to document the composition of their software.

Executives say developer productivity is a critical operational metric, but many don't know how to measure it. Just over half of executives (51%) say their current methods for measuring developer productivity are flawed, or want to measure it but don't know how to do so. At least 45% admit that they don't even measure developer productivity against business outcomes.

Most executives (55%) agree that developer productivity is important, and 57% agree that measuring that productivity is key to business growth. Only 42% currently measure developer productivity in their organization and are satisfied with their approach. More than a third (36%) believe their methods for measuring developer productivity are flawed, and 15% want to measure developer productivity but don't know how to do so.