External Metrics

For some AB tests, the relevant metrics may not be tracked in Causal. For example, return rate for an e-commerce site, or downstream conversions for a media site. In those instances, Causal provides a mechanism for you to provide us with the relevant pre-calculated metric for the specific experiment + variant on the relevant time period.

By providing Causal with the following data points, we can easily calculate experiment statistics for you. For each experiment-variant, you’ll provide:

• Number of samples
• Sum of those samples
• Sum of those samples squared

We can use that data to calculate the experiment statistics:

• The number of samples
• The average sample value
• The standard deviation of those samples

Then we hook them into the significance calculation that is displayed in the experiment analysis page.

API Endpoints

Production: https://tools.causallabs.io/api/v1/external-metric

API Parameters

All of the fields are required

metric_id: The id assigned by causal to this metric. You can look this up on the Metrics detail page. To create an external metric, set the type to “external” during metric creation.

environment_id: The ID for the environment (dev / stage / prod) you want to publish the external metric data to.

variant_id: The ID for the experiment variant you want to provide metric data for

key: We use the key value to separate out different time buckets. You can use any key format and time bucket you prefer. The most important consideration is making sure we have the same number of buckets for each experiment-variant id.

Note: If you send the same key value twice, we will use an upsert and discard the old data.

count: the number of observations used when calculating the metric

sum: the sum of the observation metric

sumsq: The sum of the squared observation metric.

Example API call

curl -X POST 'https://tools.causallabs.io/api/v1/external-metric' \
  -H "Content-Type: application/json" \
  -H "X-API-KEY: db63xxxxxxxxxxxxxxxxxxx4a24"  \
  -d '{
  "metric_id": "3385cd54-db85-40ed-bb32-e0b096c51987",
  "environment_id": "b25b58f3-58e3-4f7a-bc67-7b02a9d176a5",
  "variant_id": "22222222-2222-2222-2222-222222222222",
  "key": "2023-01-01",
  "count": 120,
  "sum": 54.2,
  "sumsq": 48.432
}'

Example Use

Take an example where we are experimenting with the posting process with the goal of increasing the number of downstream comments on that post. The metric would be comments per post. Let’s say that we have a post feature that we can use to alter some instructive text:

feature PostDialog
{
    output {
      postPrompt = “Write stuff here.”
   }
  event Post
  {
     postId: ID!
  }
}

Where the Post event fires when the post is created.

Now we make an experiment that changes the postPrompt to something more helpful and want to measure the difference in average comments per post. This is an impression based metric, but the events we are counting are not on the feature, and indeed not even on the same session or user. Presumably, your data warehouse already has the necessary data:

post_id	num_comments
1	5
2	3

We can then join to the relevant Causal impression table to produce the necessary output:

select
    ds,
    hh,
    variant_id,
    count(post_id) as count,
    sum(num_comments) as sum,
    sum(num_comments*num_comments) as sumsq
from
    session
    join session.variants on T
    join post_dialog using (session_id)
    join post_dialog.post on T
where ds = $DATE and hh=$HOUR
group by ds, hh, variant_id;

That will produce one row per variant for the key. Sending this data to the Causal External Metrics API endpoint will allow us to update the Experiment metrics.