Yes, you can measure software developer productivity (2023)

Compared to other key businessesFor functions such as sales or customer service, software development has always been undervalued. Many in the technology industry have long believed that it's impossible to get everything right and that only trained engineers have enough knowledge to judge the performance of their colleagues anyway. However, this status quo is no longer sustainable. Most companies nowTo become (to some extent) a software companyRegardless of the industry, leaders need to know that they are using the most valuable talent for the greatest possible success.

There's no denying that measuring developer productivity is difficult. Other functions can be measured quite well, some even with a single metric; while in software development the connection between input and output is quite ambiguous. Software development is also highly collaborative, complex, and creative work, requiring different metrics for different levels (such as system, team, and individual). Moreover, even with a true commitment to proper productivity monitoring, traditional metrics may require configuring systems and software for more detailed and comprehensive measurements. For some standard metrics, the entire technology and development process will need to be reconfigured to enable monitoring, and implementing the instrumentation and tools needed to generate meaningful insights may require significant long-term investment. Additionally, the software development environment is changing rapidly as generative AI tools like Copilot X and ChatGPT have potentialIt enables developers to complete tasks twice as fast.

To help overcome these challenges and make this critical task more feasible, we developed a method for measuring software developer productivity that is easier to implement through existing surveys or data, such as backlog management tools. In the process, we build on existing productivity metrics developed by industry leaders over the years to uncover opportunities to improve performance.

The new approach has been implemented in nearly 20 technology, financial and pharmaceutical companies, with encouraging initial results. They include the following improvements:

• 20-30% reduction in the number of reported product defects
• 20% improvement in employee experience results
• 60 percent increase in user satisfaction.

Leverage productivity insights

With access to more comprehensive productivity data and information, leaders can begin to answer the burning questions about the software engineering talent they want to attract and retain, such as:

• What are the barriers to engineers doing their best work?
• To what extent does the culture and organization affect your ability to do your best work?
• How do we know if we are devoting their time to activities that really create value?
• How will we know if we have all the software engineering talent we need?

understand the basics

To use a granular enough system to measure developer productivity, you need to understand the three types of metrics to track: system level, team level, and individual level. Unlike functions like sales, where individual and team performance can be measured using system-level metrics of dollars earned or deals closed, software development is collaborative in a unique way that requires a different perspective. For example, while deployment frequency is a perfect metric for evaluating a system or team, it depends on all team members performing their tasks and is therefore not a useful way to monitor individual performance.

Another key dimension to recognize is what different metrics can and cannot tell you. For example, measuring the frequency of deployments or time to change may give you a clear picture of some results, but not whether the engineering organization is optimized. While metrics such as completed stories or interruptions can help identify optimizations, they require further research to identify potentially useful improvements.

In building our set of metrics, we wanted to expand on two sets of metrics already developed by the software industry. The first is the DORA metric, named after Google's DevOps research and evaluation team. These are the closest standards in the tech industry and are very good at measuring results. When the DORA indicator gives poor results, it is a sign that the problem needs to be investigated, which can often involve lengthy investigations. For example, if a metric like deployment frequency is increasing or decreasing, it could be for a number of reasons. Determining what they are and how to fix them is often not easy.

Another set of industry-developed metrics are the SPACE (Satisfaction and Well-Being, Performance, Activity, Communication and Collaboration, and Efficiency and Process) metrics, developed by GitHub and Microsoft Research to augment the DORA metrics. Taking a separate perspective, especially around developer well-being, the SPACE metric is well-suited to shed light on whether an engineering organization is optimizing. For example, an increase in developer downtime indicates a need for optimization.

Beyond this already powerful metric, our approach is to identify what can be done to improve the way products are delivered and the value of those improvements without heavy instrumentation. Supplementing DORA and SPACE metrics with opportunity-focused metrics creates an end-to-end view of software developer productivity (Exhibit 1).

1

These opportunity-focused productivity metrics use several different lenses to generate a nuanced view of the complex range of activities associated with software product development.

Time spent in the inner/outer loop.In order to identify specific areas for improvement, it is useful to think about the activities involved in software development organized into two cycles (Appendix 2). The inner loop includes activities directly related to product creation: coding, building, and unit testing. The outer loop contains the other tasks that developers need to complete to bring the code to production: integration, integration testing, release, and deployment. From the point of view of productivity and personal experience, it is desirable to maximize the time developers spend in the inner loop: product creation directly creates value.gThis is something most developers are happy to do. Most programmers find the outer loop activity a necessary but often unsatisfying task. Investing time in better tools and automating the outer loop allows developers to spend more time on inner loop activities.

2

The goal of large technology companies is that developers spend 70% of their time on back-loop activities. For example, a company that had previously successfully completed an agile transformation found that its developers were not spending as much time on coding, but on low-value-added tasks such as setting up infrastructure, running manual unit tests, and managing test data. Armed with that information, he launched a number of new tools and automation projects to help with these tasks throughout the software development lifecycle.

The growing influence of business software

explore the collection

Developer Speed ​​Index Benchmark.The Developer Velocity Index (DVI) is a survey that measures companies' technology, hiring practices and organizational support and compares them to their peers. This benchmarking helps uncover opportunities in specific areas, whether it's backlog management, testing, or security and compliance.1 For more information on the McKinsey DVI research, see Shivam Srivastava, Kartik Trehan, Dilip Wagle, and Jane Wang, “Developer Speed: How Software Excellence Drives Business Performance,” McKinsey, April 20, 2020; and Chandra Gnanasambandam, Neha Jindal, Shivam Srivastava, and Dilip Wagle, "Developer Velocity: Key Lessons from Industry Leaders," McKinsey, February 22, 2021.For example, a company known for its technical prowess and top-notch developers sought to more carefully define standard work practices for cross-team collaboration after discovering high levels of reported developer dissatisfaction, overwork, and inefficiency.

Contribution analysis.Evaluating individual contributions to a team's backlog—starting with data from backlog management tools like Jira and using proprietary algorithms to normalize data and account for nuance—can help uncover trends that are holding back optimization. This information can enable team leaders to manage clear performance expectations and therefore improve performance. In addition, it can help to identify opportunities for individual training or upskilling and to reconsider the assignment of roles within the team (for example, whether quality assessors have enough work). For example, one company found that its most talented developers were spending too much time on non-coding activities, such as design meetings or managing interdependencies between teams. In response, the company changed its operating model and clarified roles and responsibilities to allow higher-value developers to do what they do best: code. Another company reexamined its onboarding and one-on-one mentoring programs after discovering relatively low contributions from new developers to the organization.

Assessment of talent ability.Based on an industry-standard competency map, the output is a summary of an individual's knowledge, skills and abilities for a specific organization. Ideally, organizations should look for a "diamond" skill distribution, where most developers fall into the middle skill range.2Klemens Hjartar, Peter Jacobs, Eric Lamarre, and Lars Vinter, "It's Time to Reset the IT Talent Model," MIT Sloan Management Review, March 5, 2020.This could lead to opportunities for mentoring and skills development and, in extreme cases, to rethinking talent acquisition strategies. For example, one company found that its developers were more than ideally focused on "newborn" capabilities. They implemented customized learning journeys based on specific gaps and were able to promote 30% of developers to a higher professional level within six months.

Avoid metric errors

While developer productivity data is valuable, it can be detrimental to an organization if misused, so it's important to avoid certain pitfalls. In our work, we see that there are two main types of mistakes: the misuse of metrics and the inability to break free from old ways of thinking.

Abuse is most common when companies try to apply overly simplistic measures, such as generated lines of code or code commits (when developers commit code to a version control system). These simple metrics not only fail to generate truly useful insights, but can also have unintended consequences, such as leaders making inappropriate trade-offs. For example, optimizing lead times or deployment frequency can affect quality. Focusing on a single metric or an overly simplistic set of metrics can also easily lead to poor practice; for example, when commitments are measured, developers may make smaller changes more often while trying to cheat the system.

To truly benefit from productivity measurement, both managers and developers must shake off the outdated notion that leaders "can't" understand the complexities of software engineering or that engineering is too complex to measure. The importance of engineering talent for company success iCompetition for development talent has been fierce in recent years., emphasizing the need to recognize that software development, like many other things, requires better measurement. In addition, attracting and retaining top software development talent is highly dependent on providing a workplace and tools that allow engineers to perform at their best and encourage their creativity. Measuring productivity at the system level allows employers to see hidden friction points that are holding back work and creativity.

leaving

The mechanics of creating a developer productivity program can seem daunting, but now is the perfect time to start laying the groundwork. The factors driving the need to elevate the conversation about software developer productivity to C-level executives are overcoming the barriers to doing so.

The rise of remote work and its popularity among developers is one of the biggest factors. Developers have long worked in agile teams, collaborating in the same physical space, and some technology leaders believe that face-to-face teamwork is essential to work. However, the digital tools that are central to their work have facilitated the transition to remote work during the pandemic, and like most industries, that shift will be difficult to reverse. As remote and hybrid work increasingly becomes the norm, organizations will need to rely on broad, objective metrics to maintain confidence in these new work arrangements and ensure they are constantly improving the features that can easily determine their success or future failure. In fact, the market is now placing more emphasis on efficient growth and return on investment, making it more important than ever to understand how to optimize the performance of your high-value engineering talent.

Another key driver of this need for greater visibility is the rapid development of artificial intelligence tools, especially large language models like generative artificial intelligence. They are already rapidly changing the way work is done, which means that measuring developer productivity is only the first step in understanding how these valuable resources are being used.

But as developer productivity becomes more important, companies shouldn't feel like they have to launch massive and dramatic reforms almost overnight. Instead, they can start the process with a few key steps:

Learn the basics.All senior management leaders who are not engineers or have been in management positions for a long time need to understand the software development process and how it is developed.

Rate your system.Because developer productivity is often not measured at the levels necessary to identify opportunities for improvement, most companies' technology stacks will likely require extensive reconfiguration. For example, to measure test coverage (how well an area of ​​code is tested), the development team needs to equip its codebase with a tool that can track the code executed during a test run.

make plans.As with most analytics initiatives, getting lost in the flood of data is risky. It's important to start with an area that you know will lead to a clear path to improvement, such as identifying pain points and bottlenecks. Be clear about the scope of such programs, as even the best approach, however comprehensive, is not a cure-all.

Remember, measuring productivity is done on a case-by-case basis.The point is to look at the whole system and see how it can work better by improving the development environment at the system, team or individual level.

Regardless of the specific methodology used, measuring productivity ideally creates transparency and insight into key areas for improvement. Only then can organizations develop concrete plans to impact developer productivity and experience, an impact that will benefit developers and the business as a whole.