📊 A/B Testing: Push Notification Timing and Player Engagement

This repository contains an analysis of an A/B test designed to determine the optimal timing for prompting players to allow push notifications in a mobile game. The goal is to evaluate how the timing of the pop-up affects hit rate (opt-in rate) and player retention.

---

• A detailed presentation of the findings can be found on this link
• The code that was used to obtain the results is contained in the analysis.ipynb notebook. To access and run the notebook either follow the instructinos at the bottom of this README, or access the notebook in Google Colab on this link

📁 Dataset

Two anonymized datasets were used in this analysis:

• notification_allowed.csv: Records whether a user allowed push notifications and when the prompt was shown.
• user_history.csv: Daily activity logs for players, including login status, gameplay stats, and registration metadata.

Players were randomly split into two groups based on their login_id:

• Group 1 (even login_ids) – Received the pop-up immediately after installation.
• Group 2 (odd login_ids) – Received the pop-up after completing the tutorial (5 battles).

---

🎯 Objectives

• Determine which group has a higher hit rate (percentage of users who allow notifications).
• Analyze whether the timing of the notification affects player retention over time.
• Make a data-driven recommendation for optimal pop-up timing.

---

🧪 Methodology

The following statistical methods were applied:

• ✅ Two-Proportion Z-Test: Used to compare opt-in rates and daily retention rates between the two groups.
• ✅ Chi-Squared Test for Independence: Used to validate whether group assignment is associated with differences in retention or opt-in behavior.
• 📉 Visualization: Bar charts and line graphs were used to visually assess trends in opt-in and retention behavior.
• 🎯 Imputation Strategy: A boolean mask was constructed to safely impute missing values with 0 when users were inactive but still within the observable period (21 days post-registration). Payers were excluded from imputation due to their extended visibility in the dataset.

---

📈 Results

• Hit Rate: There is a statistically significant difference in hit rate between the two groups. Group 2 (pop-up after tutorial) achieved a higher opt-in rate.
• Retention: There is no statistically significant difference in retention rates between the two groups across all observed days.

---

✅ Conclusion

Deferring the push notification prompt until after the tutorial leads to higher opt-in rates without negatively affecting player retention.
➡️ Recommendation: Adopt the post-tutorial timing as the default strategy for prompting push notifications.

---

🚀 Repository Setup

0. Clone the repo

Clone the repository using the following command in any terminal where git is available:

git clone https://github.com/olimpijada72/A_B_testing.git

1. Virtual environment setup

We will use virtual environments as it is more reliable for testing. Creating a virtual environment requires a certain version of Python, we'll work with 3.10.

1. To create and activate a virtual environment run the following, based on your operating system:

• Windows (cmd):

python3 -m venv venv
venv\Scripts\activate

• Windows (PowerShell):

python3 -m venv venv
venv\Scripts\Activate.ps1

• Windows (Git Bash):

python -m venv venv
source venv/Scripts/activate

• Linux/macOS:

python3.10 -m venv venv
source venv/bin/activate

• Note: To deactivate a virtual environment, simply run deactivate in the terminal.

2. Installing dependencies

To install the necessary requirements, run the following command:

pip install -r requirements.txt

3. Opening Jupyter Lab

To open the notebook in order to run and edit it, run the following command:

jupyter lab

📂 Project Structure

A_B_testing/
├── data/                  # Folder containing raw input data files
│ ├── notification_allowed.csv
│ └── user_history.csv       
├── analysis.ipynb         # Main analysis notebook with statistical testing and visualizations
├── assignment.md          # Project assignment or problem description
├── requirements.txt       # List of Python dependencies
├── README.md              # Project overview and setup instructions
├── LICENSE                # License file
└── .gitignore             # Files and folders to ignore in version control