The conventional product management wisdom suggests that one of a product leader’s responsibilities is to track and optimize metrics – quantitative measures that reflect how people benefit from a specific solution. Anyone who has read product management books, attended workshops or even gone through an interview knows that what cannot be measured cannot be managed.
However, the practice of product management is much more nuanced. Context is very important, and the realities of different organizations, regions, cultures and market segments greatly influence what can be measured and what actions can be taken based on these observations. In this article, I look at cybersecurity product governance and how product leaders are tempted to track and report on metrics that may not be what they seem.
Detection accuracy
While not all cybersecurity products are designed to generate some form of detection, many do. Detection accuracy is a metric that applies to the security tools that trigger alerts to inform users that a specific behavior has been detected.
Two types of metrics are useful to track in the context of detection accuracy:
- False positives (a false alarm, when the tool triggers a detection under normal behavior).
- False negatives (a missed attack, when the tool falsely identifies an attack as normal behavior and does not trigger detection).
Security vendors face a serious, and dare I say, impossible-to-win challenge: how to reduce the number of false positives and false negatives as close to zero as possible.
The reason this is impossible is that each customer’s environment is unique and applying generic detection logic across all organizations will inevitably lead to gaps in security coverage.
Product leaders should keep in mind that false positives make it more likely that a true, critical detection will be missed, while false negatives mean that the product is not doing what the tool was purchased for.
Conversion rate
Conversion rate is one of the most important metrics companies, and subsequently product teams, obsess over. This metric tracks the percentage of all users or visitors who take a desired action.
Who owns conversions in the organization depends on who can influence the outcome. For example:
- If the product is completely led by the sale and whether the deal is closed is in the hands of the sale, then the conversion belongs to the sale.
- If the product is completely product driven and whether a free user becomes a paying customer is in the hands of product, then the conversion is owned by marketing and product teams (marketing owns the website signup, product owns in-app conversion ).
