June 20, 2023
Every tech leader is faced with the challenge of maximizing technology delivery performance and improving operational efficiency. It’s what drives success for an org and, increasingly, for a career. But how can you truly ensure that your technology delivery is as good as it can be? And how can you do this well when you rely on technology vendors? This is exactly what technology vendor performance benchmarking and analysis tries to do and what we cover in this post.
Technology Vendor Performance Benchmarking uses standard engineering metrics and data analysis to give tech leaders an objective and independent way of measuring the delivery performance of vendors. It gives visibility into the quality and efficiency of vendor work. It can also highlight potential areas of underperformance. And, perhaps most importantly, it gives leaders knowledge and insight to actively and strategically manage vendor relationships and budgets.
In large companies the technology delivery ecosystem typically consists of a blend of in-house teams and outsourced technology vendors. Teams can and will have different roles and goals. For example, in-house teams may handle strategic, business-critical, or sensitive aspects of technology delivery. Whereas, outsourced vendors may be brought in to provide specialized knowledge or expertise that the in-house teams might lack. This could include expertise in emerging technologies, specialized software development, or IT infrastructure management. Alternatively, vendors may be engaged for scalablity. Or, if a company operates internationally, having vendors in different time zones can ensure round-the-clock service and business continuity.
What all of this means is that each vendor team may have different goals and, as a consequence, the performance of each team is going to be slight different. That is, their focus and performance will be unique. So, if you are going to measure the performance of your vendors, you are going to need a framework that can embrace different performance models and goals but still deliver actionable insight.
For any analysis you are going to need to do some kind of benchmarking of the performance of your vendors. But choosing appropriate benchmarks for vendor comparison is often the first challenge. Do you look at how they compare against best practice? Against peers? What about your technology maturity? What about your regulatory environment? How do you choose? It can be complicated.
Benchmarking is really effective when you look at it as an exercise in optimization. It’s purpose is less about how good you are compared to somebody else and more about how good you are in your context. Therefore, what’s required is to establish Key Performance Indicators (KPIs) that allow you to gauge the performance of technology delivery in the context of those different performance models. These indicators must be meaningful, measurable, and align with your business objectives. Most importantly, these should not be vanity metrics or self-selected by the vendor.
Thankfully, in the last number of years a consensus has grown that DORA engineering metrics offer a valuable way to baseline technology performance. DORA benchmarks give engineering leaders concrete engineering objectives around delivery velocity and quality, but that break down into the metrics that can be used for deeper analysis.
For example, by looking at Change Failure Rate and Mean Time to Recover, leaders can understand if their vendor teams are building robust quality services. Similarly, monitoring Deployment Frequency and Mean Lead Time for Changes allows leaders to see if a team is working quickly. Together, the metrics provide insight into the team’s balance of speed and quality but framed by their particular performance context. These data and metrics give you the insight you need.
Gathering the necessary data from vendors can often be the most challenging part. Vendors may be reluctant to share it. Or they may seek to provide metrics and data that paint them only in a positive light. Many vendors only report outputs. In other words, you get what you get.
And then teams try to build tools to get the data. They find problems with integration, data normalization etc. The cost and resource required can be significant. Thankfully, getting access to the data is a lot easier than it once was. There are now many engineering metrics solutions that allow you to get and use this data. When we are conducting benchmarking for our clients, we use our Implement.io platform. It works by plugging into commonly-used engineering, release, project management and other tools and extracting the necessary data.
Now, we know that every team is different. And that means the quality of the data of every team is likely to be different too. This can add additional overheard to getting to useable data if you are doing this yourself or with in-house teams. Indeed, many tech leaders lose enthusiasm for data-driven engineering for vendor benchmarking because of this data issue. This is one of the reasons we created Implement Partners: to take away this pain. In our work with clients, we look after the data issues for them and get to clean data and metrics at minimum cost. For clients it requires 30 minutes to connect the systems, and we begin to see performance metrics within 3 to 4 hours. This then allows us to begin building each vendor performance model see how well they are working against agreed KPIs, benchmarks, or best practice. That’s the goal after all.
Technology delivery is a complex matter. It’s common for teams to perform differently over time. New teams and projects are often slow and complex to start, and then get into flow later. Codebases get more fragile with time and this impacts performance. And teams are made up of people with lives and those lives have a way of impacting work. The upshot is, a single snapshot of engineering behaviour or metrics in time isn’t enough to judge technology delivery performance of a vendor. It’s how vendor teams perform week on week, month on month, year on year – that’s what’s important. Again, in our work with clients we tend to look at six months of data as a minimum to establish performance trends, but we can go back years if required. Only by looking at the performance data over time will you see how well you vendor is performing and continuing to perform.
Whatever the benchmark and trend analysis brings back, it’s important to acknowledge that not all aspects of vendor performance are in the hands of your technology vendor. For example, we sometimes see from the metrics that the performance issues are the result of poor requirements or strategy from the in-house product team. In addition, we often see how inherited codebases and tooling have performance impact. This kind of insight is hugely valuable. Taking action on these external, indirect impacts will actually improve the performance of your vendor teams. That’s the goal, after all.
As you can see, benchmarking and measuring the performance of technology delivery from outsourced partners and vendors can be a complex task. It could almost appear overwhelming. However, the move to data-driven engineering in recent years has opened up the possibility of objective and accurate benchmarking of vendors.
For the the first time, you have visibility on vendor performance that is equal to the visibility you have of your internal teams. This new level of data, metrics, insight and understand is going to allow you to transform your relationship with your vendors. You can strategically manage your vendors to better meet your outcomes rather than outputs. You can challenge pricing. And you can ensure that the common danger of vendor complacency is avoided. The key to success, of course, is getting the data and metrics that provide the insight you need. Any benchmark is only as valuable as what you can do with the insight.
We specialize in helping technology leaders use data and metrics to improve technology delivery performance and achieve better outcomes.
Want to learn more? Drop us a message and we’ll get back to you.
A 30-minute call is usually enough to know whether a Delivery 360 would be useful — and what it would look at in your situation.
