I am writing alerts to monitor the sign up conversion rate for an app. Sign up conversion rate here means the percent of users that open up the app, who end up making an account. Usually, this is around ~35 – 45% (meaning 35 – 45% of users that open the up make an account).
I want an alert to fire if it detects a significant drop in this conversion rate, such as due to buggy release where new users can’t sign up. I have the following:
- appOpenedCount: Number of users who opened the app
- signupCount: Number of users who created an account
conversionRate: signupCount / appOpenedCount
period: How far in the past to look. Or what time periods to use for the appOpenedCount or signupCount data. Usually, we want
this to be in the past 1 hour to be urgent.
So based on the above, how do I find the best condition to trigger an alert with minimal false positives? I have about months of past data for analysis. The system will check for the alert every 5 minutes.
My current condition: If appOpenedCount > 100 and conversionRate < 0.32 (2th percentile) in the past hour, fire an alert. However, I’m noticing a lot of false positives, so I’m thinking we could do better? Should I use something like 0.2th percentile instead? The conditions can be very flexible. For example, I can use week over week analysis, where I compare against past week’s data etc.
Here is a graph I made that might be useful. Each data point indicates the conversion rate and app opened in the last hour (over the course of a month). As you can see, with more data, it’s more accurate. App open count is lowest at night, and highest during noon.