1. Analyze_AB_Test_Results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze A/B Test Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Part I - Probability\n",
    "- Part II - A/B Test\n",
    "- Part III - Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part I - Probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "random.seed(42)\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(294478, 5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         290584\n",
       "timestamp       294478\n",
       "group                2\n",
       "landing_page         2\n",
       "converted            2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the number of rows where group and landing_page have mismatches\n",
    "len(df[(df['group'] == 'control') & (df['landing_page'] == 'new_page')])+len(df[(df['group'] == 'treatment') & (df['landing_page'] == 'old_page')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(290585, 5)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop rows with mismatches and reset index\n",
    "df_1 = df[~((df['group']=='control') & (df['landing_page']=='new_page'))]\n",
    "df_filtered = df_1[~((df['group']=='treatment') & (df['landing_page']=='old_page'))]\n",
    "df_filtered.reset_index()\n",
    "df_filtered.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         290584\n",
       "timestamp       290585\n",
       "group                2\n",
       "landing_page         2\n",
       "converted            2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_filtered.nunique() # There is still one duplicated row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find out duplicated value\n",
    "df_filtered[df_filtered.duplicated('user_id')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         290584\n",
       "timestamp       290584\n",
       "group                2\n",
       "landing_page         2\n",
       "converted            2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop duplicated value\n",
    "df_filtered = df_filtered[~df_filtered.duplicated('user_id')]\n",
    "df_filtered.reset_index()\n",
    "df_filtered.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conversion rate across both groups\n",
    "df_filtered['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conversion rate of control group\n",
    "df_filtered[df_filtered['group'] == 'control']['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conversion rate of treatment group\n",
    "df_filtered[df_filtered['group'] == 'treatment']['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the probability of receving the new group\n",
    "len(df_filtered[df_filtered['group'] == 'treatment'])/len(df_filtered)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no sufficient evidence to conclude that the new page leads to higher conversion rate as the conversion rate for both groups is at very similar level. In addition, the conversion rate of the control group is even slightly higher than the treatment group"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part II - A/B Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Null Hypothesis: p_new - p_old <= 0 \n",
    "- Alternative Hypothesis: p_new - p_old > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the control and treatment dataframe\n",
    "df_old = df_filtered[df_filtered['group'] == 'control']\n",
    "df_new = df_filtered[df_filtered['group'] == 'treatment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "145274 145310\n"
     ]
    }
   ],
   "source": [
    "# sample size of both group\n",
    "n_old = len(df_old)\n",
    "n_new = len(df_new)\n",
    "print(n_old,n_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Simulation method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# p_old and p_new under null hypothesis\n",
    "p_old = df_filtered['converted'].mean()\n",
    "p_new = df_filtered['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#simulate 10000 samples of the differences in conversion rates\n",
    "p_diffs = []\n",
    "for _ in range(10000):\n",
    "    new_page_converted = np.random.choice([0, 1], size=n_new, p=[(1 - p_new), p_new])\n",
    "    old_page_converted = np.random.choice([0, 1], size=n_old, p=[(1 - p_old), p_old])\n",
    "    p_diffs.append(new_page_converted.mean() - old_page_converted.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEZCAYAAABB4IgrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df7xVVZ3/8ddbRPwtIEjyQ/AHNakzkkNqUzPDZPnbtB8WjhmaRTX6LWdsCmtmJIvSGtNK0/EHSZmRZRoqpWQ5jpoCOmoiMiKiICQooqJJop/vH2ud2hzOvffcH+veC7yfj8d5nH3WXnvvtfbZZ3/2WnufvRURmJmZlbRFTxfAzMw2fQ42ZmZWnIONmZkV52BjZmbFOdiYmVlxDjZmZlZclwQbSSdIuqUr5tVg3ldK+kqJeTdY1kmS7ujkPOZJGpeHJel7kp6TNDunfUrS05LWSNq5C4rda0j6gqTLe7ocPWVzr39XkHSbpI/1dDlqqvsfSeMkLe3pMjWjVFmr+7f2ajrYSHqHpLskPS9plaQ7Jb0VICJ+GBGHdKQAXankhipplKTIQWJNDhg3Snp3NV9E7BMRt+WP7wDeDQyPiAMk9QW+CRwSEdtHxLMlytpTIuKrEdHh9S/pAEkzJa3O29hsSSd3ZRlL6mz9W5IPgl7L290Lkh6QdFQ7pi9ywNbSwZmkxZLe1dXLa6/Kb/amuvSrJE0usLyQtFdd2mRJV3X1sppZdgl1+7d2aSrYSNoRuBH4DjAQGAZ8CVjbkYVu5PpHxPbAfsAs4DpJJ7WQdySwOCJeyp+HAFsD8zqyYEl9OjLdxkDS24BfA/8N7AXsDHwKOLwny1UlacseXPxv83bXH/guMF1S/x4sT6/SxndzkKS3d1thrKFmWzZvBIiIH0XEaxHxh4i4JSIehA2PcHKU/SdJj0p6UdKXJe0p6bf5yOwaSVs1mrYy/QZRWtKA3JpYmbumbpQ0PI+bAvwtcGE+Arwwp/+FpFn5SHmBpA9W5rezpBm5TLOBPZtdcRHx+4j4FjAZOFfSFnmeiyW9S9IpwOXA23J5fgQsyJOvlvTrJsp3paSL89H+S8A/SOon6T8lPZlbV5dI2ibnHydpqaQzJK2QtLzaMpC0jaTzJD2RW6h3VKY9KLdcV+cj53GV6U6StCh/l49LOqHROqkexVWOKifksj4j6YutrNJvANMi4tyIeCaSeyOiuj4+LmlhXlczJA2tjAtJn8zb3HOSLlLSL9dp30rewZL+IGmX/PkoSffnfHdJ+qtK3sWSPi/pQeAlSVvmz0/l9bFA0sH19c+f36PU7bBaqdX95rr5flbSg/m7+LGkrVtZPwBExOvAD4DtgNGV+f1E0u/zvG6XtE9OnwicAHwub4c35PShkq7Nv6XHJX26Mq8DJM3Nv4unJX2zrXK1JG/DF0m6Ka+veyTtWRn/bkmP5HJfCKhu+o9Kmp+/05sljayMC0mnSnoUeLSVYnwdaNiyUzv2P52V18N5dWk3SDo9Dy+WdKakh3N9v1fdJlra/iXdnrM8kL/jD1WmaWlf0Np+ZJDSvrXWw/A/qtu/5eH2bScR0eYL2BF4FphGOtIcUDf+JOCOyucAZuTp9iG1gG4F9gB2Ah4GJjSatjL9Xnn4SuAreXhn4P3AtsAOwE+A6yvT3QZ8rPJ5O2AJcDKwJbA/8AywTx4/Hbgm59sXeKq+LJV5jcrl2rIufY+c/ub8eTHwrhbWy3rzaKJ8VwLPA28nHRhsDVyQ1+3AvA5uAL6W848D1gFnA32BI4CXa98XcFFeR8OAPsDfAP3y52dz/i1IXX/PAoNzGV8A3pTnsWutfA3W0WTgqrq6XgZsQ2oJrq2tp7rptgVeA/6hlW3wnXnd7J/L/B3g9rpt5kbSkf9uwErgsDxuKjClkvdU4Jd5eH9gBXBgXicT8nfYr/J93g+MyPV4U/7OhlbquWeD+r8ReCmvy77A54CFwFaV+c4Ghubvcj7wyRbqfhJ5O8plPBX4I7BLJc9H8/bQL28j91fGXUn+DeXPWwD3Av8BbEXahhcBh+bxvwVOzMPbAwe1Va669MX8+TdwJbAKOIC0jf8QmJ7HDSJtWx/I6+ifSdvvx/L4Y/M6e3Oe9t+Au+q+81l5/W3Tym92e9Jvu1amq4DJHdj/jAOWtrKN/mm6Fn4TBwDLgC0q9X8ZGFJZbw+RtrWBwJ2VZTez/e9V+TyO1vcFre1HvgZckqfrSzqIV4PvtqntpPZqqmUTES+Qzj/Udh4rc2Qd0spk50bECxExL6/AWyJiUUQ8D/wCeEszy64rx7MRcW1EvBwRLwJTgL9vZZKjSN1Y34uIdRFxH3At8AGlLqn3A/8RES9FxEOkYNpey/L7wA5M22L5Knl+HhF3RjqiXQt8HPjniFiV18FXgfGV/K8CZ0fEqxExE1gDvCkfmXwU+ExEPBWphXpXRKwFPgzMjIiZEfF6RMwC5pI2UIDXgX0lbRMRy/N32qwvRWoJPwA8QAo69QaQdoDLW5nPCcDUiLgvl/lMUqtxVCXPORGxOiKeBH4DjMnpVwPHV/L9Y06DtD7/KyLuyetkGmk9H1TJ/+2IWBIRfyAFxX7A3pL6RsTiiHisQXk/BNwUEbMi4lXgP0nB6m/q5rssIlaRfuxjGsyn5iBJq4FX8rw+HBEraiMjYmpEvJjXzWRgP0k7tTCvtwKDI+LsiPhjRCwi/a5r29GrwF6SBkXEmoi4u5VyNeNnETE7ItaRgk2tnkcAD0fET/M6ugD4fWW6T5B2gPPztF8FxlRbN3n8qvzdtOQV0r6iOy40ui+3CFbn72tSbUREzCYdPB6ck8YDt0XE05XpL8zb2qpc5tp228z2X6+lfYFofT/yKumgcmSe9n8iR5QG8296O2n6AoH8hZ8UEcNJrYChpI2jJdUV+IcGn7dvdtk1kraV9F9K3UAvALcD/dXyuYyRwIF1X/4JwBtIR+1bko5Sa55ob5lIrQJIR2/t1Vr5aqrlG0xqBdxbyf/LnF7zbP5h1rxMWteDSC2jRjvGkcBxdeV4B7BrpPNNHwI+CSzP3SF/0Y46VncetbLUe44U0HZtZT5DqXw/EbGG1PoaVsnT0rJ+DWwj6cC8oxoDXJfHjQTOqKv7iLy8mj99BxGxEDidtENfIWm6Kt15rZT39TyfZsrbyN0R0Z8UmGeQjjaBdC5P0jmSHsu/i8V51KAW5jUSGFpX5y+QzikCnEJqmT0iaY5avhhhHenIt15f0o6opqV6DmX9dRusv72PBL5VKeMqUjdbdR1W87fmMmCIpKObzN9R+0dE/9oLOKdu/DTSwR35/Qd14+v3R7Vtq5ntv15L+4K29iPfILUob1HqPp9EY81uJ0AHL32OiEdIzct928jajJdIFQdA0htayXsGqRvjwIjYEfi72mS1otXlXwL8d/XLj3QV2KdI3SzrSDuWmt06UP73krphFrSVsYHWyldTrdMzpEC9TyX/TpFOHLflGdIRXqPzUkuAH9SVY7uIOAcgIm6OiHeTgsEjpB9ul4mIl0lN8ve3km0ZaecDgKTtSN2qTzUx/9dJ3aXHk1o1N+ajOUh1n1JX920j4kfVWdTN7+qIeEcuTwDnNlFekba1NsvbRl3WAP8EnCip1jvwj8AxwLtI3dSjaottVH5SnR+vq/MOEXFEXsajEXE8sEuu20/z+q73JLBbrlutntvm6Zo5cFtO5fdXWUfVcn6irpzbRMRd1VXSxHLILacvAV9m/fNC7dn/dIWrgGMk7UfqHry+bnz9/qjWc9Lh7b+BVvcjuYV8RkTsARwN/IvyecmqdmwnQPNXo/1FPtFUOxk/gvTD7WzzGlLXyj6SxuSTYZNbybsDaSWtljQQOKtu/NOk/ueaG4E3SjpRUt/8equkN0fEa8DPgMm5xbQ3qb++KZKGSDotl+HMvENrrxbL1yhzXsZlwPn688ntYZIObWtBedqpwDeVTg73kfQ2Sf1IP4CjJR2a07dWuthgeK7ne/JGtJbUFH+tA3Vty+eAkyT9q/L/jyTtJ2l6Hn81cHLeTvqRmv33RMTiJud/NamFdgJ/7kKDtD4/mVs9krSdpCMl7dBoJpLeJOmduQyvkLbHRuvjGuBISQcrXfJ+Bmn93dUgb7tEumT+ctI5F0i/i7WkI91tSeumqv53MRt4QelCh23yd76v8l8ZJH1Y0uC8zazO0zSq4z2kdTApbzPbkY7k59JcsLmJ9Nt/n9LVZJ9m/Vb9JcCZ+vPFDjtJOq6J+bbkB6Qu0MMqae3Z/3RaRCwF5uSyXNug++/U/LsbSGpt/jint7X913/HrZWh1f2I0gUze+Xg/wLpu9/g+2/HdgI037J5kXQC9R6lq6LuJp2HOaPJ6VsUEf9HOon1K9IVJa39qfICUr/3M7kMv6wb/y3S+ZjnJH07H70eQuqLXEZqzp9L2uAATiM1K39Paql9r4kir87r4HekPufjImJqE9NtoInyNfJ5UhP37txl8itSa68Zn83lnkPqkjiXdLJyCenI+AukFt8S4F9J28cWpO95WZ7m70lH1l0qH62+M78WSVoFXArMzONvBf6ddE5rOamFNr7x3BrO/x7SUexQ0jnDWvpcUv/1haTuvIWkk8Yt6UfaoT5D+r52Ia23+uUtIHWTfCfnPRo4OiL+2GyZ23ABcITSlXPfJ+3cnyJdfFN/EHgF6RzTaknX5wOto0ndiY/n8l1OahVB2hnPk7SG9JsaHxGvNKjjWuBI8olz0kUGQ4EPttDHXz/9M8BxpPX5LOnqujsr468jbaPT87b+EJ24FD7X+ywq51fbuf/pKtOAv2TDLjRIQeUW0rpcRD7P1MT2PxmYlr/jD9K21vYjo/PnNaQeh+9G4//WNLWd1KiJbcLMzLqIpL8j9SaMqvaISFpMuhLvVz1VtpJ8bzQzs26Su1Q/A1zewa73jZaDjZlZN8jnYleTLrRp7UreTZK70czMrDi3bMzMrLievLFgmwYNGhSjRo3q6WJYsxbkvxq9qdmL48yshHvvvfeZiBjcds7u06uDzahRo5g7d25PF8OaNW5cer/ttp4shdlmT1JH7oZSlLvRzMysOAcbMzMrzsHGzMyKc7AxM7PiHGzMzKw4BxszMyvOwcbMzIpzsDEzs+IcbMzMrLhefQcBs7aMmnRTjy178TlH9tiyzTY2btmYmVlxDjZmZlacg42ZmRXXZrCRtLWk2ZIekDRP0pdy+u6S7pH0qKQfS9oqp/fLnxfm8aMq8zozpy+QdGipSpmZWe/STMtmLfDOiNgPGAMcJukg4Fzg/IgYDTwHnJLznwI8FxF7AefnfEjaGxgP7AMcBnxXUp+urIyZmfVObQabSNbkj33zK4B3Aj/N6dOAY/PwMfkzefzBkpTTp0fE2oh4HFgIHNAltTAzs16tqXM2kvpIuh9YAcwCHgNWR8S6nGUpMCwPDwOWAOTxzwM7V9MbTFNd1kRJcyXNXblyZftrZGZmvU5TwSYiXouIMcBwUmvkzY2y5Xe1MK6l9PplXRoRYyNi7ODBveqppmZm1kHtuhotIlYDtwEHAf0l1f4UOhxYloeXAiMA8vidgFXV9AbTmJnZJqyZq9EGS+qfh7cB3gXMB34DfCBnmwD8PA/PyJ/J438dEZHTx+er1XYHRgOzu6oiZmbWezVzu5pdgWn5yrEtgGsi4kZJDwPTJX0F+F/gipz/CuAHkhaSWjTjASJinqRrgIeBdcCpEfFa11bHzMx6ozaDTUQ8CLylQfoiGlxNFhGvAMe1MK8pwJT2F9PMzDZmvoOAmZkV52BjZmbFOdiYmVlxDjZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkV52BjZmbFOdiYmVlxDjZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWXJvBRtIISb+RNF/SPEmfyemTJT0l6f78OqIyzZmSFkpaIOnQSvphOW2hpEllqmRmZr3Nlk3kWQecERH3SdoBuFfSrDzu/Ij4z2pmSXsD44F9gKHAryS9MY++CHg3sBSYI2lGRDzcFRUxM7Peq81gExHLgeV5+EVJ84FhrUxyDDA9ItYCj0taCByQxy2MiEUAkqbnvA42ZmabuHads5E0CngLcE9OOk3Sg5KmShqQ04YBSyqTLc1pLaWbmdkmrulgI2l74Frg9Ih4AbgY2BMYQ2r5nFfL2mDyaCW9fjkTJc2VNHflypXNFs/MzHqxZs7ZIKkvKdD8MCJ+BhART1fGXwbcmD8uBUZUJh8OLMvDLaX/SURcClwKMHbs2A2CkVlvMWrSTT2y3MXnHNkjyzXrjGauRhNwBTA/Ir5ZSd+1ku29wEN5eAYwXlI/SbsDo4HZwBxgtKTdJW1FuohgRtdUw8zMerNmWjZvB04Efifp/pz2BeB4SWNIXWGLgU8ARMQ8SdeQTvyvA06NiNcAJJ0G3Az0AaZGxLwurIuZmfVSzVyNdgeNz7fMbGWaKcCUBukzW5vOzMw2Tb6DgJmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkV52BjZmbFOdiYmVlxDjZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkV52BjZmbFOdiYmVlxDjZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFddmsJE0QtJvJM2XNE/SZ3L6QEmzJD2a3wfkdEn6tqSFkh6UtH9lXhNy/kclTShXLTMz6022bCLPOuCMiLhP0g7AvZJmAScBt0bEOZImAZOAzwOHA6Pz60DgYuBASQOBs4CxQOT5zIiI57q6Utb9Rk26iemLngVg/KSberg0ZtbbtNmyiYjlEXFfHn4RmA8MA44BpuVs04Bj8/AxwPcjuRvoL2lX4FBgVkSsygFmFnBYl9bGzMx6pXads5E0CngLcA8wJCKWQwpIwC452zBgSWWypTmtpfT6ZUyUNFfS3JUrV7aneGZm1ks1HWwkbQ9cC5weES+0lrVBWrSSvn5CxKURMTYixg4ePLjZ4pmZWS/WVLCR1JcUaH4YET/LyU/n7jHy+4qcvhQYUZl8OLCslXQzM9vENXM1moArgPkR8c3KqBlA7YqyCcDPK+kfyVelHQQ8n7vZbgYOkTQgX7l2SE4zM7NNXDNXo70dOBH4naT7c9oXgHOAaySdAjwJHJfHzQSOABYCLwMnA0TEKklfBubkfGdHxKouqYWZmfVqbQabiLiDxudbAA5ukD+AU1uY11RgansKaGZmGz/fQcDMzIpzsDEzs+IcbMzMrDgHGzMzK87BxszMinOwMTOz4hxszMysOAcbMzMrzsHGzMyKc7AxM7PiHGzMzKw4BxszMyvOwcbMzIpzsDEzs+IcbMzMrDgHGzMzK87BxszMinOwMTOz4hxszMysOAcbMzMrzsHGzMyKc7AxM7PiHGzMzKw4BxszMyuuzWAjaaqkFZIeqqRNlvSUpPvz64jKuDMlLZS0QNKhlfTDctpCSZO6vipmZtZbNdOyuRI4rEH6+RExJr9mAkjaGxgP7JOn+a6kPpL6ABcBhwN7A8fnvGZmthnYsq0MEXG7pFFNzu8YYHpErAUel7QQOCCPWxgRiwAkTc95H253ic3MbKPTmXM2p0l6MHezDchpw4AllTxLc1pL6RuQNFHSXElzV65c2YnimZlZb9HRYHMxsCcwBlgOnJfT1SBvtJK+YWLEpRExNiLGDh48uIPFMzOz3qTNbrRGIuLp2rCky4Ab88elwIhK1uHAsjzcUrqZmW3iOtSykbRr5eN7gdqVajOA8ZL6SdodGA3MBuYAoyXtLmkr0kUEMzpebDMz25i02bKR9CNgHDBI0lLgLGCcpDGkrrDFwCcAImKepGtIJ/7XAadGxGt5PqcBNwN9gKkRMa/La2NmZr1SM1ejHd8g+YpW8k8BpjRInwnMbFfpzMxsk+A7CJiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkV52BjZmbFOdiYmVlxDjZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkVt2VPF8DM2mfUpJt6bNmLzzmyx5ZtGze3bMzMrDgHGzMzK87BxszMinOwMTOz4toMNpKmSloh6aFK2kBJsyQ9mt8H5HRJ+rakhZIelLR/ZZoJOf+jkiaUqY6ZmfVGzbRsrgQOq0ubBNwaEaOBW/NngMOB0fk1EbgYUnACzgIOBA4AzqoFKDMz2/S1GWwi4nZgVV3yMcC0PDwNOLaS/v1I7gb6S9oVOBSYFRGrIuI5YBYbBjAzM9tEdfSczZCIWA6Q33fJ6cOAJZV8S3NaS+kbkDRR0lxJc1euXNnB4pmZWW/S1RcIqEFatJK+YWLEpRExNiLGDh48uEsLZ2ZmPaOjwebp3D1Gfl+R05cCIyr5hgPLWkk3M7PNQEeDzQygdkXZBODnlfSP5KvSDgKez91sNwOHSBqQLww4JKeZmdlmoM17o0n6ETAOGCRpKemqsnOAaySdAjwJHJezzwSOABYCLwMnA0TEKklfBubkfGdHRP1FB2ZmtolqM9hExPEtjDq4Qd4ATm1hPlOBqe0qnZmZbRJ8BwEzMyvOwcbMzIpzsDEzs+IcbMzMrDgHGzMzK87BxszMinOwMTOz4hxszMysOAcbMzMrrs07CNjGZdSkm3q6CGZmG3DLxszMinOwMTOz4hxszMysOAcbMzMrzsHGzMyKc7AxM7PiHGzMzKw4BxszMyvOwcbMzIpzsDEzs+IcbMzMrDgHGzMzK87BxszMinOwMTOz4joVbCQtlvQ7SfdLmpvTBkqaJenR/D4gp0vStyUtlPSgpP27ogJmZtb7dUXL5h8iYkxEjM2fJwG3RsRo4Nb8GeBwYHR+TQQu7oJlm5nZRqBEN9oxwLQ8PA04tpL+/UjuBvpL2rXA8s3MrJfpbLAJ4BZJ90qamNOGRMRygPy+S04fBiypTLs0p5mZ2Saus4+FfntELJO0CzBL0iOt5FWDtNggUwpaEwF22223ThbPzMx6g061bCJiWX5fAVwHHAA8Xesey+8rcvalwIjK5MOBZQ3meWlEjI2IsYMHD+5M8czMrJfocLCRtJ2kHWrDwCHAQ8AMYELONgH4eR6eAXwkX5V2EPB8rbvNzMw2bZ3pRhsCXCepNp+rI+KXkuYA10g6BXgSOC7nnwkcASwEXgZO7sSyzcxsI9LhYBMRi4D9GqQ/CxzcID2AUzu6PDMz23j5DgJmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnIONmZkV19lHDJjZZmTUpJt6ZLmLzzmyR5ZrXcctGzMzK87BxszMinOwMTOz4hxszMysOAcbMzMrzsHGzMyK86XPBfTU5aFmZr2VWzZmZlacg42ZmRXnYGNmZsU52JiZWXEONmZmVpyDjZmZFedgY2ZmxXV7sJF0mKQFkhZKmtTdyzczs+7XrX/qlNQHuAh4N7AUmCNpRkQ83J3lMLONS0/+UdrP0uka3X0HgQOAhRGxCEDSdOAYoEiw8T/5zcx6h+4ONsOAJZXPS4EDqxkkTQQm5o9rJC3oprK1ZBDwTA+XoSe0u95vqw2ce1SXF6abba7fOWy+dW+x3jq3m0vSNUb2dAHqdXewUYO0WO9DxKXApd1TnLZJmhsRY3u6HN1tc603uO6bY90313p3p+6+QGApMKLyeTiwrJvLYGZm3ay7g80cYLSk3SVtBYwHZnRzGczMrJt1azdaRKyTdBpwM9AHmBoR87qzDB3Qa7r0utnmWm9w3TdHm2u9u40iou1cZmZmneA7CJiZWXEONmZmVtxmG2wkDZQ0S9Kj+X1AC/km5DyPSppQSf9rSb/Lt935tiTVTfdZSSFpUOm6tEepekv6hqRHJD0o6TpJ/burTm1p6xZJkvpJ+nEef4+kUZVxZ+b0BZIObXaevUFX11vSCEm/kTRf0jxJn+m+2rRPie88j+sj6X8l3Vi+FpuYiNgsX8DXgUl5eBJwboM8A4FF+X1AHh6Qx80m/Y9RwC+AwyvTjSBdBPEEMKin69od9QYOAbbMw+c2mm8P1bcP8BiwB7AV8ACwd12efwIuycPjgR/n4b1z/n7A7nk+fZqZZ0+/CtV7V2D/nGcH4P96W71L1b0y3b8AVwM39nQ9N7bXZtuyId0mZ1oengYc2yDPocCsiFgVEc8Bs4DDJO0K7BgRv420BX6/bvrzgc9R94fVXqJIvSPilohYl6e/m/Qfqt7gT7dIiog/ArVbJFVV18lPgYNzi+0YYHpErI2Ix4GFeX7NzLOndXm9I2J5RNwHEBEvAvNJdwXpbUp850gaDhwJXN4NddjkbM7BZkhELAfI77s0yNPo9jrD8mtpg3QkvQd4KiIeKFHoLlCk3nU+Smr19AYt1aVhnhwwnwd2bmXaZubZ00rU+09yt9NbgHu6sMxdpVTdLyAdRL7e9UXe9HX37Wq6laRfAW9oMOqLzc6iQVq0lC5p2zzvQ5qcfxHdXe+6ZX8RWAf8sMllldZmmVvJ01J6o4O03taKLVHvNJG0PXAtcHpEvNDhEpbT5XWXdBSwIiLulTSuk+XbLG3SwSYi3tXSOElPS9o1Ipbn7qEVDbItBcZVPg8Hbsvpw+vSlwF7kvp5H8jnzYcD90k6ICJ+34mqtEsP1Ls27wnAUcDBuZutN2jmFkm1PEslbQnsBKxqY9reftulIvWW1JcUaH4YET8rU/ROK1H39wDvkXQEsDWwo6SrIuLDZaqwCerpk0Y99QK+wfonyr/eIM9A4HHSSfIBeXhgHjcHOIg/nyg/osH0i+l9FwgUqTdwGOlREYN7uo51ddmSdIHD7vz5ZPE+dXlOZf2Txdfk4X1Y/2TxItLJ5zbn2dOvQvUW6TzdBT1dv+6ue9204/AFAu3/Xnq6AD1W8dQ/eyvwaH6v7UzHApdX8n2UdJJwIXByJX0s8BDpapULyXdjqFtGbww2Reqd8y0B7s+vS3q6rpUyH0G6cuox4Is57VITpEgAAAYVSURBVGzgPXl4a+AnuQ6zgT0q034xT7eA9a843GCeve3V1fUG3kHqanqw8j1vcJDVG14lvvPKeAebDrx8uxozMytuc74azczMuomDjZmZFedgY2ZmxTnYmJlZcQ42ZmZWnION9QqSJku6qoPTniTpjlbG/6J25+r6vJLWSNqjI8vtCEmXSPr3yudP5T/arpG0s6S35zttr5HU6L51ZhulTfoOAlaWpMXAEOA14CVgJvD/ImJNT5arXkQc3sq47WvDkq4ElkbEv3VkOZX1sY60Th4m/Qny0oh4PS/vk5X8fYFvAgdFvpeepLOBCyPiWx0pg1lv5ZaNddbReYe9P/BWYIMdtZLNZVs7OiJ2AEYC5wCfB65oIe8Q0p8L51XSRtZ9blq+7YpZr7S57ACssIh4inT7mn0BJN0maYqkO4GXgT0kDZU0Q9Kq/HCqj9fNZuv8QKsXJd0nab/aCEmTJD2Wxz0s6b1100rSdyQ9r/QQt4MrI26T9LFG5VZ6wN1ekiYCJwCfy11YN0j6V0nX1uX/jqQLmlgfz0fEDOBDwARJtfVypaSvSHoj6R/qAKsl/VpS7RksN+Qy9JO0k6QrJC2X9FSetk+e10mS7pR0vqRVwOSc/lGlB5w9J+lmSSPr6vvJ3FX3nKSL8q31a+M/nqetref9c/pQSddKWinpcUmfbmsdmFU52FiXkDSCdIuQ/60knwhMJD1o6wngR6QbHQ4FPgB8tRoUSM8S+Qnp3mxXA9fnriZItw/5W9INE78EXJVvJFpzIOk+VoOAs4CfSRrYbPkj4lLSnaq/HhHbR8TRwFWk5/j0z3XckhQ8ftCO+c7Odf7buvT/I92HC6B/RLwzIvYEniS3FiNiLemZK+uAvUi39D8EqAbOWr13Aabk8zxfAN4HDAb+h7Teq44itUL3Az5Ien4Rko4jBayPADuSbj75bG6V3kC6Z9gw4GDgdNU9xdKsNQ421lnXS1oN3AH8N/DVyrgrI2JepOeFvIF0b63PR8QrEXE/6SFUJ1by3xsRP42IV0nnMrYm3fSTiPhJRCyLiNcj4seke7sdUJl2BekGka/m8QtID7rqsEjP+7kdOC4nHQY8ExH3tnNWy0gBtF0kDQEOJ93K/6WIWEF6MN/46rwj4jsRsS4i/gB8AvhaRMzP6/2rwJhq6wY4JyJWR8STwG+AMTn9Y6RgOyeShRHxBCkwDY6IsyPijxGxCLisrhxmrXIfr3XWsRHxqxbGVR9CNRRYFekJjzVPkG7suUH+iHhdUq0VhKSPkB7JOypn2Z7Uiql5Kta/0d8TtWk7aRrwKdLO9cO0o1VTMYx0+/r2Ggn0BZZXerq2YP31uqTBNN+SdF4lTbkMT+TP1cddvExal5Burf9YC+UYmg8qavqQWk1mTXGwsZKqO/9lwEBJO1QCzm7AU5U8f3qOSO66GQ4sy0fll5G6b34bEa9Jup/1H3Q1TJIqAWc3YEYnyltzPXBxPudyFOlJjU2T9FbSjr7FS7NbsQRYS7pz+LoW8tSXeQkwJSI68vC6JaRnMjVKfzwiRndgnmaAu9Gsm0TEEuAu4GuStpb0V8AprP9Ez7+W9L58buR00o72bmA70k51JYCkk8kXIlTsAnxaUt987uHNpEux2+Np0gn6arlfIT2j/mpgdu56apOkHZWe7jgduCoiftfOstS68W4Bzsvz20LSnpL+vpXJLgHOlLRPLsdOeX0043Lgs5L+Ol9BuFcO9LOBFyR9XtI2kvpI2jcHUrOmONhYdzqe1A22DLgOOCsiZlXG/5x0Av450rmc9+VzMA8D5wG/JQWEvwTurJv3PcBo4BlgCvCBiHi2neW7Athb0mpJ11fSp+VlNtOFdoOkF0mtgS+Szj2d3M5yVH2E9ACwh0nr5afAri1ljojrgHOB6ZJeID17qMX/GdVN+xPSursaeJHUqhsYEa8BR5PO7TxOWseXky7WMGuKn2dj1gZJuwGPAG+IiBd6ujxmGyO3bMxakc8d/Qsw3YHGrON8gYBZCyRtR+q2e4J02bOZdZC70czMrDh3o5mZWXEONmZmVpyDjZmZFedgY2ZmxTnYmJlZcf8f8tkVLN7/3iMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the distribution of sample mean difference\n",
    "obs_diff = df_filtered[df_filtered['group'] == 'treatment']['converted'].mean()-df_filtered[df_filtered['group'] == 'control']['converted'].mean()\n",
    "plt.hist(p_diffs)\n",
    "plt.title(\"Simulated Differences in Conversion Rates Under Null Hypothesis\", fontsize=12)\n",
    "plt.xlabel(\"Probability Difference\", fontsize=12)\n",
    "plt.axvline(obs_diff, color='r');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate observed difference\n",
    "obs_diff = df_filtered[df_filtered['group'] == 'treatment']['converted'].mean()-df_filtered[df_filtered['group'] == 'control']['converted'].mean()\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8978"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate p_value\n",
    "(p_diffs>=obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the p-value is 0.9, which is significantly higher than αlevel of 0.05, therefore we fail to reject the null hypothesis.This indicates there there is no significant difference in conversion rate between the new and old webpages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Use built-in stats package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17489, 17264, 145274, 145310)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "converted_old = df_old[df_old['converted']==1]['converted'].count()\n",
    "converted_new = df_new[df_new['converted']==1]['converted'].count()\n",
    "converted_old, converted_new, n_old, n_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.3109241984234394, 0.9050583127590245)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use stats proportions z-test package \n",
    "z_score, p_value = sm.stats.proportions_ztest([converted_new, converted_old], [n_new, n_old], alternative='larger')\n",
    "z_score, p_value\n",
    "#In the two sample test, smaller means that the alternative hypothesis is p1 < p2 and larger means p1 > p2 where p1 is the proportion of the first sample and p2 of the second one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part III - Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal is to use logistic regression a. to see if there is a significant difference in conversion based on which page a customer receives.\n",
    "\n",
    "- Step 1: create a column for the intercept\n",
    "- Step 2: create a dummy variable column for which page each user received - 1 for treatment and 0 for control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a column for the intercept\n",
    "df_filtered['intercept'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get dummy variable for landing_page column\n",
    "df_filtered[['new_page','old_page']] = pd.get_dummies(df['landing_page'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  new_page  old_page  \n",
       "0          1         0         1  \n",
       "1          1         0         1  \n",
       "2          1         1         0  \n",
       "3          1         1         0  \n",
       "4          1         0         1  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_filtered.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "# apply logistic model\n",
    "logit_mod = sm.Logit(df_filtered['converted'], df_filtered[['intercept', 'new_page']])\n",
    "results = logit_mod.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Thu, 04 Jun 2020</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>11:24:55</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_page</th>  <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Thu, 04 Jun 2020   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        11:24:55   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "new_page      -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# summary of the model\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value associated with new_page is 0.19. It is different from the p-value found in Part II because logistic regression is based on a two-tailed test. As 0.19 is higher than the alpha value of 0.05, we fail to reject the null hypothesis. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conversion rate can be potentially related to other user properties such as country, gender, age etc. Adding additional information to the model will reveal the effect of these variables on conversion rate.Here I will add the country feature to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_country = pd.read_csv('data/countries.csv')\n",
    "df_country.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  new_page  old_page country  \n",
       "0          1         0         1      US  \n",
       "1          1         0         1      US  \n",
       "2          1         1         0      US  \n",
       "3          1         1         0      US  \n",
       "4          1         0         1      US  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a new dataframe by adding country feature\n",
    "df_updated = pd.merge(df_filtered, df_country, left_on = 'user_id', right_on = 'user_id', how = 'left')\n",
    "df_updated.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290584 entries, 0 to 290583\n",
      "Data columns (total 9 columns):\n",
      " #   Column        Non-Null Count   Dtype \n",
      "---  ------        --------------   ----- \n",
      " 0   user_id       290584 non-null  int64 \n",
      " 1   timestamp     290584 non-null  object\n",
      " 2   group         290584 non-null  object\n",
      " 3   landing_page  290584 non-null  object\n",
      " 4   converted     290584 non-null  int64 \n",
      " 5   intercept     290584 non-null  int64 \n",
      " 6   new_page      290584 non-null  uint8 \n",
      " 7   old_page      290584 non-null  uint8 \n",
      " 8   country       290584 non-null  object\n",
      "dtypes: int64(3), object(4), uint8(2)\n",
      "memory usage: 18.3+ MB\n"
     ]
    }
   ],
   "source": [
    "df_updated.info() # all users are mapped to a country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "US    203619\n",
       "UK     72466\n",
       "CA     14499\n",
       "Name: country, dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_updated['country'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_updated = pd.get_dummies(df_updated, columns=['country'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "      <th>country_CA</th>\n",
       "      <th>country_UK</th>\n",
       "      <th>country_US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  new_page  old_page  country_CA  country_UK  country_US  \n",
       "0          1         0         1           0           0           1  \n",
       "1          1         0         1           0           0           1  \n",
       "2          1         1         0           0           0           1  \n",
       "3          1         1         0           0           0           1  \n",
       "4          1         0         1           0           0           1  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_updated.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Thu, 04 Jun 2020</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>11:24:58</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th>  <td>   -2.0300</td> <td>    0.027</td> <td>  -76.249</td> <td> 0.000</td> <td>   -2.082</td> <td>   -1.978</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>country_UK</th> <td>    0.0506</td> <td>    0.028</td> <td>    1.784</td> <td> 0.074</td> <td>   -0.005</td> <td>    0.106</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>country_US</th> <td>    0.0408</td> <td>    0.027</td> <td>    1.516</td> <td> 0.130</td> <td>   -0.012</td> <td>    0.093</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Thu, 04 Jun 2020   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        11:24:58   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -2.0300      0.027    -76.249      0.000      -2.082      -1.978\n",
       "new_page      -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "country_UK     0.0506      0.028      1.784      0.074      -0.005       0.106\n",
       "country_US     0.0408      0.027      1.516      0.130      -0.012       0.093\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply logistic model\n",
    "log_mod_2 = sm.Logit(df_updated['converted'], df_updated[['intercept','new_page', 'country_UK','country_US']])\n",
    "result = log_mod_2.fit()\n",
    "result.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-values for the countries are above 0.05, so again we cannot reject the null hypothesis. Countries do not lead to significant difference in conversion rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "      <th>country_CA</th>\n",
       "      <th>country_UK</th>\n",
       "      <th>country_US</th>\n",
       "      <th>new_page_uk</th>\n",
       "      <th>new_page_us</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  new_page  old_page  country_CA  country_UK  country_US  \\\n",
       "0          1         0         1           0           0           1   \n",
       "1          1         0         1           0           0           1   \n",
       "2          1         1         0           0           0           1   \n",
       "3          1         1         0           0           0           1   \n",
       "4          1         0         1           0           0           1   \n",
       "\n",
       "   new_page_uk  new_page_us  \n",
       "0            0            0  \n",
       "1            0            0  \n",
       "2            0            1  \n",
       "3            0            1  \n",
       "4            0            0  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_updated['new_page_uk'] = df_updated['new_page'] * df_updated['country_UK']\n",
    "df_updated['new_page_us'] = df_updated['new_page'] * df_updated['country_US']\n",
    "df_updated.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290578</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     5</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Thu, 04 Jun 2020</td> <th>  Pseudo R-squ.:     </th>  <td>3.482e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>11:25:00</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1920</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>          <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th>   <td>   -2.0040</td> <td>    0.036</td> <td>  -55.008</td> <td> 0.000</td> <td>   -2.075</td> <td>   -1.933</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_page</th>    <td>   -0.0674</td> <td>    0.052</td> <td>   -1.297</td> <td> 0.195</td> <td>   -0.169</td> <td>    0.034</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>country_UK</th>  <td>    0.0118</td> <td>    0.040</td> <td>    0.296</td> <td> 0.767</td> <td>   -0.066</td> <td>    0.090</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>country_US</th>  <td>    0.0175</td> <td>    0.038</td> <td>    0.465</td> <td> 0.642</td> <td>   -0.056</td> <td>    0.091</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_page_uk</th> <td>    0.0783</td> <td>    0.057</td> <td>    1.378</td> <td> 0.168</td> <td>   -0.033</td> <td>    0.190</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_page_us</th> <td>    0.0469</td> <td>    0.054</td> <td>    0.872</td> <td> 0.383</td> <td>   -0.059</td> <td>    0.152</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290578\n",
       "Method:                           MLE   Df Model:                            5\n",
       "Date:                Thu, 04 Jun 2020   Pseudo R-squ.:               3.482e-05\n",
       "Time:                        11:25:00   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1920\n",
       "===============================================================================\n",
       "                  coef    std err          z      P>|z|      [0.025      0.975]\n",
       "-------------------------------------------------------------------------------\n",
       "intercept      -2.0040      0.036    -55.008      0.000      -2.075      -1.933\n",
       "new_page       -0.0674      0.052     -1.297      0.195      -0.169       0.034\n",
       "country_UK      0.0118      0.040      0.296      0.767      -0.066       0.090\n",
       "country_US      0.0175      0.038      0.465      0.642      -0.056       0.091\n",
       "new_page_uk     0.0783      0.057      1.378      0.168      -0.033       0.190\n",
       "new_page_us     0.0469      0.054      0.872      0.383      -0.059       0.152\n",
       "===============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply logistic model\n",
    "log_mod_3 = sm.Logit(df_updated['converted'], df_updated[['intercept','new_page', 'country_UK','country_US','new_page_uk','new_page_us']])\n",
    "result = log_mod_3.fit()\n",
    "result.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at p-values for all variables (all higher than 0.05), we can conclude that interactions between page and country have no significant effects on conversion rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analysis shows that we do not have sufficient evidence to conclude that he new webpage leads to higher conversion rate than the old one. Even though there can be other user properties potentially resulting in rejecting the null hypothesis, the analysis done so-far suggests that the new website doesn't bring additional value and we should keep the old webpage instead. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}