Management requires measuring effectiveness and efficiency. To measure efficiency, we need some metrics. Metrics are used to demonstrate if an initiative is succeeding or failing. Metrics are also used to measure how one person is performing against another. Metrics make comparisons data driven and objective.
The first challenge is finding the right set of metrics. It is a well-known fact that measuring lines of code or number of PRs is not a correct metric for developer productivity. But many organizations still use those to measure developer productivity. I used to work at an organization where engineers used to be rewarded for the number of bugs fixed. That again is a questionable metric. Finding the right metric may be very difficult. Some bugs are tricky to fix and some are easy. But measuring trickiness is tough. Some code changes can add a lot of value with a few lines and that value addition is hard to measure. When you don’t have the right metric, you use a proxy metric which gives an easy to measure signal.
The second challenge is ensuring the metrics are not gamed. Goodhart’s Law dictates if a metric becomes a target, then it loses its value as an indicator. People try to out compete each other by gaming the system and manipulating the metrics. Developers come up with shallow PRs to increase their count. I have seen engineers writing redundant tests or unnecessary lengthy code to ensure high line count. The general practice is to put some checks and balances in place to ensure fairness. For example, the code changes are often passed through reviews, which ensures fairness.
These days, many companies are blowing their GenAI tooling budgets as they are targeting the wrong metric. These companies are using token usage as a proxy for GenAI tool adoption. The problem remains the same. A naive user can create prompts which will use a lot of tokens. But an expert user can create efficient prompts to get the same work done with less number of tokens. Ideally management should reward the expert user. But when the incentive is to use more and more tokens, then token-maxxing becomes the goal. Every one tries to use as many tokens as possible to be viewed as a top performer. Things get even worse when a leaderboard is maintained to show how people are using the tokens. Nobody wants to be seen as a laggard. And the end result is a bloated tooling budget due to token-maxxing.
Recently Amazon scrapped their AI leaderboard due to the above reason. Other companies are also realizing the mistake. But many would have realized what will happen when people will measure token usage as a metric. The question remains why some reputed companies full of smart people are making obvious mistakes. One reason can be difficulty in finding a suitable metric. The other can be deliberately measuring it for some time to ensure people are coming out of their comfort zones and trying out the GenAI tools. The third reason can be cultural issues with organizations where anybody calling out emperors having no cloth is penalized.