Monitoring Software Delivery Process Efficiency

DEFINITION:

Treno can be used to review the delivery efficiency following each delivery cycle (sprint).  This is primarily done through reviewing key performance indicators - that show how well the delivery cycle was completed, but also gives insights into process effectiveness.

ROLES:

• CTO

• VP, Engineering

• Director, Engineering

   

RECOMMENDED USAGE:

• Monthly, Quarterly, On Demand

   

ANALYSIS ACTIVITIES:

1. Navigate to the Analyze/Trends page

2. Select the desired Workspace (rollup of all projects) or an individual project/team

3. View the Trend Analysis Monitors for the key aspects of delivery

   • Track the index score for each monitor over time

1. Navigate to the Analyze/Delivery/Spotlight Metrics page

2. Select the desired Workspace (rollup of all projects) or an individual project/team

3. Pay special attention to these metrics which will give good insights into how efficient the process is performing

   • Deployment

   • Bug/Error Ratio

   • Issue Duration/Cycle Time

   • Coding Efficiency

1. Review from Analyze/Delivery/Spotlight Metrics page

   • the cycle time for a project/roll-up.  Spotting issues in the review process or where branches spend most of their time will help to understand

   • the average time an issue stays in a workflow stage - issues that stay longer are “bottlenecked”

2. Review from the Analyze/Management page the type of work and the level of effort that has been done during the review period.

    

Trends

• When reviewing delivery trends it is important to view the baseline for each Index Monitor.  These baselines are updated every three months and represent how “normal” looks.

  • It should be expected that an index should either “hug” the baseline or show improvement.

  • If an index monitor is decreasing in value, you should click into the Constituent Metrics 

  • The Constituent Metrics break down the Index Monitor score into individual metrics that will allow you to dig into the details of what is going well or what requires improvement.

Key Metrics

• Deployments: the goal of the software delivery organization is to deploy code  Unless you are focusing solely on mobile applications, an elite organization deploys multiple times a day.  

  • low sprint completion % may result in lower than optimal deployments

  • If deployments are less than multiple times a day, a review of QA and devops process is merited - assuming that code is available to deploy.

• Bug / Error Resolution Ratio: creating more bugs or errors than are resolved will eventually result in more production errors or a lower percentage of resolved innovation issues

  • creating bugs / errors can be indicative of either an engineer in trouble, overloaded or a less than effective review or QA process

• Issue Duration / Cycle Time: steady increases in issue duration and cycle time are indicative of process issues.  Review the breakdown of both of these metrics to understand where process issues exist.

• Coding Efficiency: coding efficiency is not available at the project level if multiple projects are housed in the same repository.  If this is the case, review code efficiency at the rollup (workspace) level.

  • A low coding efficiency indicates either engineers that are overwhelmed or unclear stories.  Either way deeper analysis of why multiple changes per code review is warranted.

Monitor Flow / Bottleneck & Delivery Mix:

• While every organization is different, it is a good benchmark to expect the deal to focus at least 70% of their time on innovation

   

1. Review from Analyze/Management page the average time an issue stays in a workflow stage - issues that stay longer are “bottlenecked”

2. Add time in stage to the weekly 1:1s you have with the delivery leads.  Common bottleneck causes are:

   • Poorly written stories

   • Overloaded teams or engineers

   • Overly complicated QA process

   • Mobile focused deployment processes

   • Disconnected DevOps teams and processes

3. If bottlenecks are found, leverage the spotlight metrics to identify potential areas for improvement in your software development lifecycle