Five Causal Factors of Metric Drift

In business, we track metrics because we want to see them change over time (or stay the same). For the most part, metrics do what we want, thanks in part to the adage, “What’s measured gets managed.”  But sometimes, metrics don’t behave the way we want them to; they drift. This is when the analyst steps in.

When analysts seek to determine why a metric has drifted from one period to another, most will follow a few intuitional clues. Still, few have a reliable method for determining the cause of the drift. This article will introduce the vocabulary to describe and understand different causal factors and demonstrate how they all interrelate within the metric tree framework.

Before we get into causal factors, it's important to note that drift may be caused by multiple factors. Businesses are dynamic systems, and it's unlikely that any change is due to a single cause (let alone one kind of causal factor). It’s helpful to think of various causal factors as a collection—each with a degree of upward or downward pressure on the metric. Addressing the collection of causal factors will help ensure that remediations will be effective and durable.

We’ll discuss pressure in the next article, but it’s helpful to remember that context as we address the five types of drift.

Five causes of metric drift

In the previous article about Root Cause Analysis, the sales funnel analysis example listed five potential causes. Let’s establish formal definitions for each of them.

1. Component drift- Change due to change in component inputs in a metrics’s definition
2. Temporal variance - Change due to natural, often cyclic, behavior over time
3. Influence drift - Change due to change in the value of  or statistical relationship to a metric’s influence inputs
4. Dimension drift - Change in the dimensional composition or change in the values within “slices” of a metric
5. Event Shocks - Abrupt and significant changes due to specific events that alter outcomes beyond predictable trends

Now, let's put these into the context of business operations.

Component drift

Components are the foundation for understanding metric drift. The reason is simple: components have a deterministic effect on their downstream metrics. Change in compound metrics (those defined by an equation consisting of input components like traffic * conversion rate = conversions) is entirely explained by changes in their input metrics.

Examples:

• The simplest example is a metric defined by a value and a rate. Let’s use the Sales Qualified Leads example again.  If “SQLs” are up or down from one period to another, it’s because the Sales Accepted Lead, Sales Qualification Rate, or both changed simultaneously.
• There are more complicated examples: New MRR is a linear function New MRR + Expansion MRR - Contraction MRR - Churn MRR. If New MRR changes, it's because one of the components has changed.

Temporal variance

Most businesses experience normal and observable variance within and across periods—these parallel natural cycles like seasons and days and human-made cycles such as weeks, months, and holidays.

Examples: 

• Ecommerce: Shifting seasonal demand, including holidays, especially BFCM
• B2B: Quarterly and yearly cycles set the pace for budgets, which impacts buying behavior
• Marketplace: Seasonality impacts travel and leisure. Weekly cycles impact logistics.

Influence drift 

Influences, like components, define their output metrics. In this case, though, the relationship is not deterministic. Influence metrics define their output metrics with a statistical relationship with one or more sets of two expressions  (input * coefficient), where the coefficient represents the strength of the relationships.

There are two ways a metric defined by influence metrics can change:

1. The statistical relationship has changed
2. The input metric  itself has changed

Examples: 

• Let’s say that the Sales Opportunity Conversion Rate is entirely dependent upon one metric: Speed to Lead. In that case, the conversion rate would go up or down because one of two things changed: Speed to lead changed, or the relationship between Speed to lead and conversion rate is now even more strongly negative than before.
•  There are also cases where multiple inputs influence a metric. For example, the Trial Conversion Rate is influenced by the average time to activation among new users and the number of new users who reach a “product qualified lead” status. This relationship would look like this: Trial Conversion Rate =PQLs * {PQL coefficient}} +Time to Activation * {PQL coefficient} + {error}.

Dimension shift

Every metric can be segmented across a dimension. For example, website traffic can be divided quite discreetly into device types. Dimension shift (also known as mix shift)  is when a metric drifts due to three types of changes:

1. The metric value within a slice of a dimension changes
2. The composition or proportion of the slices within a dimension change
3. New slices are added or removed

Examples: 

• Let’s refer back to the web traffic device types. You could find that the web conversion rate has decreased. What you find is that mobile traffic used to be 50%. Now it's 70%.  Mobile traffic converts less than desktop traffic, so the conversion rate is diluted and decreases.
• The RCA process example from the previous article mentions a shift in Sales Qualified Leads due to a change in the marketing mix that led to lower-quality leads from new channels. Perhaps the marketing team gave up on events and rolled the dice on TikTok, only ending up with leads from teen influencers.

Event Shocks

Event shocks are significant and abrupt changes in metrics that result from external or internal events, disrupting the expected patterns. These events usually fall into four buckets: GTM, operations, market, and product (defined below). Unlike trends or cyclical behaviors, event shocks are characterized by their suddenness and the often temporary but profound impact they have on metrics.

1. GTM Event - An activity aimed at enhancing market presence and driving growth (or an accident with the opposite effect).
2. Operations Event - Activities within a company's operations that significantly affect its efficiency, productivity, or cost structures.
3. Market Event - External events in the broader market, such as economic shifts, regulatory changes, or competitive actions.
4. Product Event - The introduction, update, or discontinuation of a product, significantly influencing customer engagement, market share, and revenue streams.

Examples:

• A viral marketing campaign can lead to an unprecedented spike in website traffic and sales, significantly deviating from normal trends.
• A regulatory change in an industry can abruptly alter buying behaviors, leading to a sudden drop or spike in demand for certain products or services.
• A natural disaster affecting a popular travel destination can instantly decrease bookings, disrupting the usual seasonality patterns in the travel and leisure sectors.

With a solid understanding of the five causes of metric drift, you're now equipped to go beyond identification and start quantifying the impact of these factors. This transition is crucial because it shifts our focus from recognizing what causes metric changes to understanding how significantly each cause affects the metric. The next step in our journey is to evaluate these causes by assessing their pressure on the metrics. 

This approach allows us to prioritize interventions based on the magnitude of impact. We'll explore how to differentiate between significant factors and background noise, ensuring that our actions are informed by a deep understanding of underlying causes.