ABTest/AB_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze A/B Test Results\n",
    "\n",
    "This project will assure you have mastered the subjects covered in the statistics lessons.  The hope is to have this project be as comprehensive of these topics as possible.  Good luck!\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists.  It is important that you get some practice working with the difficulties of these \n",
    "\n",
    "For this project, you will be working to understand the results of an A/B test run by an e-commerce website.  Your goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.\n",
    "\n",
    "**As you work through this notebook, follow along in the classroom and answer the corresponding quiz questions associated with each question.** The labels for each classroom concept are provided for each question.  This will assure you are on the right track as you work through the project, and you can feel more confident in your final submission meeting the criteria.  As a final check, assure you meet all the criteria on the [RUBRIC](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric).\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1.` Now, read in the `ab_data.csv` data. Store it in `df`.  **Use your dataframe to answer the questions in Quiz 1 of the classroom.**\n",
    "\n",
    "a. Read in the dataset and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1\n",
       "5   936923  2017-01-10 15:20:49.083499    control     old_page          0\n",
       "6   679687  2017-01-19 03:26:46.940749  treatment     new_page          1\n",
       "7   719014  2017-01-17 01:48:29.539573    control     old_page          0\n",
       "8   817355  2017-01-04 17:58:08.979471  treatment     new_page          1\n",
       "9   839785  2017-01-15 18:11:06.610965  treatment     new_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Groups: ['control' 'treatment']\n",
      "Landing pages: ['old_page' 'new_page']\n",
      "Converted: [0 1]\n"
     ]
    }
   ],
   "source": [
    "# Investigating unique values in columns of interest\n",
    "print('Groups: {}'.format(df['group'].unique()))\n",
    "print('Landing pages: {}'.format(df['landing_page'].unique()))\n",
    "print('Converted: {}'.format(df['converted'].unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Use the below cell to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         294478 non-null int64\n",
      "timestamp       294478 non-null object\n",
      "group           294478 non-null object\n",
      "landing_page    294478 non-null object\n",
      "converted       294478 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique users: 290584\n"
     ]
    }
   ],
   "source": [
    "# obtaining unique number of values ofr user_id column\n",
    "unique_users_num = df.user_id.nunique()\n",
    "print('Unique users: {}'.format(unique_users_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "d. The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of users converted : 12.104 %\n"
     ]
    }
   ],
   "source": [
    "# Compute proportion of users converted without any further data investigation\n",
    "users_conv = df.query('converted == 1').user_id.nunique()\n",
    "print('Proportion of users converted : {:.3f} %'.format(100.0*users_conv/unique_users_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "e. The number of times the `new_page` and `treatment` don't line up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New page not allignes with treatment 1928 times\n",
      "Old page allignes with treatment (not control) 1965 times\n",
      "Treatment not alligned with new_page 3893 times\n"
     ]
    }
   ],
   "source": [
    "# Checking when the new_page and old_page do not line up with treatment and control respectively\n",
    "\n",
    "# First checking for treatment and new_page missmatch\n",
    "df_new_page = df.query('landing_page == \"new_page\"')\n",
    "df_new_page_not_treatment = df_new_page.query('group != \"treatment\"')\n",
    "\n",
    "# Then checking for control and old_page missmatch\n",
    "df_old_page = df.query('landing_page == \"old_page\"')\n",
    "df_old_page_not_control = df_old_page.query('group != \"control\"')\n",
    "\n",
    "new_page_not_treatment = df_new_page_not_treatment.shape[0]\n",
    "old_page_not_control = df_old_page_not_control.shape[0]\n",
    "\n",
    "print('New page not allignes with treatment {} times'.format(new_page_not_treatment))\n",
    "\n",
    "# means treatment not alligned with new_page, but with old_page\n",
    "print('Old page allignes with treatment (not control) {} times'.format(old_page_not_control)) \n",
    "\n",
    "# Total of missmatchs on both sides\n",
    "print('Treatment not alligned with new_page {} times'.format(new_page_not_treatment+old_page_not_control))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No missing values\n"
     ]
    }
   ],
   "source": [
    "print('No missing values')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` For the rows where **treatment** is not aligned with **new_page** or **control** is not aligned with **old_page**, we cannot be sure if this row truly received the new or old page.  Use **Quiz 2** in the classroom to provide how we should handle these rows.  \n",
    "\n",
    "a. Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df2 should have: 290585 rows\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290585 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         290585 non-null int64\n",
      "timestamp       290585 non-null object\n",
      "group           290585 non-null object\n",
      "landing_page    290585 non-null object\n",
      "converted       290585 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 13.3+ MB\n"
     ]
    }
   ],
   "source": [
    "#drop the rows that do not allign new_page w/ treatment and old_page w/ control (these might have innacurate data)\n",
    "df2 = df.drop(df_new_page_not_treatment.index).copy()\n",
    "df2.drop(df_old_page_not_control.index, inplace=True)\n",
    "\n",
    "print('df2 should have: {} rows\\n'.format(df.shape[0]-(new_page_not_treatment+old_page_not_control)) )\n",
    "\n",
    "df2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the correct rows were removed - this should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` Use **df2** and the cells below to answer questions for **Quiz3** in the classroom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique user id in df2: 290584\n"
     ]
    }
   ],
   "source": [
    "print('Unique user id in df2: {}'.format(df2.user_id.nunique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "b. There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicated user_id: 773192\n"
     ]
    }
   ],
   "source": [
    "# Finding a duplicated user_id\n",
    "dup_id = df2.user_id.value_counts().argmax()\n",
    "print('Duplicated user_id: {}'.format(dup_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is the row information for the repeat **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('user_id == {}'.format(dup_id)).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290584 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         290584 non-null int64\n",
      "timestamp       290584 non-null object\n",
      "group           290584 non-null object\n",
      "landing_page    290584 non-null object\n",
      "converted       290584 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 13.3+ MB\n"
     ]
    }
   ],
   "source": [
    "# Removing duplicated user_id\n",
    "df2.drop_duplicates('user_id',inplace=True)\n",
    "df2.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` Use **df2** in the below cells to answer the quiz questions related to **Quiz 4** in the classroom.\n",
    "\n",
    "a. What is the probability of an individual converting regardless of the page they receive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of converted individual regardless of the page: 0.1196\n"
     ]
    }
   ],
   "source": [
    "# Calculating probability of an individual convertion rate regardless of the page\n",
    "conv_ind = df2.query('converted == 1')\n",
    "num_conv = conv_ind.shape[0]\n",
    "print('Probability of converted individual regardless of the page: {:.4f}'.format(num_conv/df2.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of CONTROL page converted individual: 0.1204\n"
     ]
    }
   ],
   "source": [
    "# Control group individual conversion rate \n",
    "control_conv = conv_ind.query('group == \"control\"').shape[0]\n",
    "control_group = df2.query('group==\"control\"').shape[0]\n",
    "print('Probability of CONTROL page converted individual: {:.4f}'.format(control_conv/control_group))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of TREATMENT page converted individual: 0.1188\n"
     ]
    }
   ],
   "source": [
    "# Treatment group individual conversion rate \n",
    "treatment_conv = conv_ind.query('group == \"treatment\"').shape[0]\n",
    "treatment_group = df2.query('group == \"treatment\"').shape[0]\n",
    "print('Probability of TREATMENT page converted individual: {:.4f}'.format(treatment_conv/treatment_group))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability and individual got the NEW PAGE: 0.5001\n"
     ]
    }
   ],
   "source": [
    "# probability an individual got the new page\n",
    "new_page_ind = df2.query('landing_page == \"new_page\"').shape[0]\n",
    "print('Probability and individual got the NEW PAGE: {:.4f}'.format(new_page_ind/df2.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Consider your results from a. through d. above, and explain below whether you think there is sufficient evidence to say that the new treatment page leads to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information above tells us that:\n",
    "1. The probability of an individual to convert regardless of the page is 0.1196\n",
    "2. The probability of an individual in the control group/ old page to convert is 0.1204\n",
    "3. The probability of an individual in the treatment group/ new page to convert is 0.1188\n",
    "4. The probability of an individual to get the new page (be in treatment group) is 0.5001, which is basically 0.5\n",
    "\n",
    "Based on those probabilities one can see that the difference between 1 and 2 is 0.0015 which indicates they are very close to each other. The statistical significance might not exist, but there is not enough evidence to conclude anything until we do an A/B test with a proper hypothesis in which we want to know if the new page on average has a higher successful convert rate than the old page. After the A/B test we will understand statiscal significance, but it will be very important to consider the practical significance of our result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### Part II - A/B Test\n",
    "\n",
    "Notice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.  \n",
    "\n",
    "However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "`1.` For now, consider you need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should your null and alternative hypotheses be?  You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the converted rates for the old and new pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H_0: p_{new} - p_{old} \\leq 0 $$\n",
    "$$ H_1: p_{new} - p_{old} > 0 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have \"true\" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. <br><br>\n",
    "\n",
    "Use a sample size for each page equal to the ones in **ab_data.csv**.  <br><br>\n",
    "\n",
    "Perform the sampling distribution for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.  <br><br>\n",
    "\n",
    "Use the cells below to provide the necessary parts of this simulation.  If this doesn't make complete sense right now, don't worry - you are going to work through the problems below to complete this problem.  You can use **Quiz 5** in the classroom to make sure you are on the right track.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. What is the **convert rate** for $p_{new}$ under the null? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p_new under sample  : 11.8808 %\n",
      "p_new under the assumed H0 conditions: 11.9597%\n"
     ]
    }
   ],
   "source": [
    "def calc_p_new(df):\n",
    "    \"\"\"\n",
    "    The function calculates the proportion of new pages that have a successful converted rate\n",
    "    Arguments:\n",
    "        @df: Pandas.DataFrame that contains columns landing_page and converted.\n",
    "    Return the proportion of new pages with successful convertion\n",
    "    \"\"\"\n",
    "    p_new_succ = df.query('landing_page == \"new_page\" and converted == 1').shape[0]\n",
    "    p_new = df.query('landing_page == \"new_page\"').shape[0]\n",
    "    prop_p_new = p_new_succ/p_new\n",
    "    return prop_p_new\n",
    "\n",
    "#compute proportion of new pages with successful convertion\n",
    "p_new_prop = calc_p_new(df2)\n",
    "print('p_new under sample  : {:.4f} %'.format(100.0*p_new_prop))\n",
    "\n",
    "# H0: assumed that the nul p_new and p_old converted success rates are equal.\n",
    "# Meaning p_new and p_old converted success rates for null are calculated using the converted success rate regardless\n",
    "# of the type of page\n",
    "p_new_null = df2.converted.mean()\n",
    "print('p_new under the assumed H0 conditions: {:.4f}%'.format(100.0*p_new_null))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. What is the **convert rate** for $p_{old}$ under the null? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p_old under sample : 12.0386 %\n",
      "p_old under the assumed H0 conditions: 11.9597%\n"
     ]
    }
   ],
   "source": [
    "def calc_p_old(df):\n",
    "    \"\"\"\n",
    "    The function calculates the proportion of old pages that have a successful converted rate\n",
    "    Arguments:\n",
    "        @df: Pandas.DataFrame that contains columns landing_page and converted.\n",
    "    Return the proportion of old pages with successful convertion\n",
    "    \"\"\"\n",
    "    p_old_succ = df.query('landing_page == \"old_page\" and converted == 1').shape[0]\n",
    "    p_old = df.query('landing_page == \"old_page\"').shape[0]\n",
    "    prop_p_old = p_old_succ/p_old\n",
    "    return prop_p_old\n",
    "\n",
    "#compute proportion of old pages with successful convertion\n",
    "p_old_prop = calc_p_old(df2)\n",
    "print('p_old under sample : {:.4f} %'.format(100.0*p_old_prop))\n",
    "\n",
    "# H0: assumed that the nul p_new and p_old converted success rates are equal.\n",
    "p_old_null = p_new_null\n",
    "print('p_old under the assumed H0 conditions: {:.4f}%'.format(100.0*p_old_null))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is $n_{new}$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample size new_page: 145310\n"
     ]
    }
   ],
   "source": [
    "# Size of new page samples\n",
    "sample_size_new_page = df2.query('landing_page == \"new_page\"').shape[0]\n",
    "print('Sample size new_page: {}'.format(sample_size_new_page))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is $n_{old}$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample size old_page: 145274\n"
     ]
    }
   ],
   "source": [
    "# Size of old page samples\n",
    "sample_size_old_page = df2.query('landing_page == \"old_page\"').shape[0]\n",
    "print('Sample size old_page: {}'.format(sample_size_old_page)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Simulate $n_{new}$ transactions with a convert rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Simulate with H0 assumption new_page data\n",
    "new_page_converted = np.random.choice([1,0], size=sample_size_new_page, p=[p_new_null,1-p_new_null])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Simulate $n_{old}$ transactions with a convert rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Simulate with H0 assumption old_page data\n",
    "old_page_converted = np.random.choice([1,0], size=sample_size_old_page, p=[p_old_null, 1-p_old_null])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_p(array):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "        @array: numpy array that we will take its mean\n",
    "    Return mean of the @array argument\n",
    "    \"\"\"\n",
    "    p = array.mean()\n",
    "    return p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No simulated p_diff (under the sample): -0.1578 %\n",
      "Simulated samples p_diff (under H0 assumed conditions): 0.0879%\n"
     ]
    }
   ],
   "source": [
    "# Calculating the mean for H0 and H1\n",
    "p_new_null_mean = calc_p(new_page_converted)\n",
    "p_old_null_mean = calc_p(old_page_converted)\n",
    "\n",
    "p_diff = p_new_prop - p_old_prop\n",
    "\n",
    "p_diff_sim = p_new_null_mean - p_old_null_mean\n",
    "\n",
    "print('No simulated p_diff (under the sample): {:.4f} %'.format(100.0*p_diff))\n",
    "print('Simulated samples p_diff (under H0 assumed conditions): {:.4f}%'.format(100.0*p_diff_sim))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Simulate 10,000 $p_{new}$ - $p_{old}$ values using this same process similarly to the one you calculated in parts **a. through g.** above.  Store all 10,000 values in a numpy array called **p_diffs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Simulating the null values with 10000 samples \n",
    "p_diffs = []\n",
    "\n",
    "for _ in range(10000):\n",
    "    new = np.random.choice([1,0], size=sample_size_new_page, p=[p_new_null,1-p_new_null])\n",
    "    old = np.random.choice([1,0], size=sample_size_old_page, p=[p_old_null,1-p_old_null])\n",
    "    p_new_mean = calc_p(new)\n",
    "    p_old_mean = calc_p(old)\n",
    "    p_diffs.append(p_new_mean - p_old_mean)\n",
    "    \n",
    "p_diffs = np.array(p_diffs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean: 0.0000%\n",
      "Std: 0.0012\n"
     ]
    }
   ],
   "source": [
    "# Simulation descriptive statistics \n",
    "p_diffs_sim10000 = p_diffs.mean()\n",
    "print('Mean: {:.4f}%\\nStd: {:.4f}'.format(p_diffs_sim10000,p_diffs.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAFOCAYAAABaLaGCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4FtW58P/vLUSUYhEELAqKFvRVBDkqnkMRxNYiVi34\n+gpUrW3V+tbdXcVqK7q1G3/2p7tsbd20olhRqHiialU8UA9FKbAjragN9VBAtgcQPKFyWO8fzyRN\nQhICZBISv5/req7nmTVrZu55MuKdlXvWREoJSZIkSfnZobEDkCRJkpo7k25JkiQpZybdkiRJUs5M\nuiVJkqScmXRLkiRJOTPpliRJknJm0i2p0UXEixFR3NhxNKaIOCkilkbEhxHRN6dj3BoRV2Wfj4qI\nVyqs2z8i/jsiPoiICyJi54j4fUSsiYi7coonRUT3qrFt5b72yr67FvUU200R8ZPsc3FELKuP/Wb7\nq/TdS/p8MOmWlKuIeD0ijq3SNi4inilbTin1TCnN2cx+umVJWsucQm1sPwfOTym1SSn9d9WV2bl/\nlCWWyyPium1JMFNKT6eU9q/QdBEwJ6W0S0ppEnAKsDuwW0rp1GrimZDFdGqFtpZZW7etjas62fWy\nITv3DyPitYi4JSL2q3A+/8i+uw112NcztfXJ9vfdlNK/1VP85b9cZPuu+t1L+hww6ZYkCgljI4ew\nN/DiZvocnFJqAwwB/jfw7RyPvzfwt5TS+lq2WQVcWV+jy5sxNzv3tsCxwFpgQUQcVN8HaqDzkfQ5\nY9ItqdFVHA2PiEMiYn5EvB8Rb0XEdVm3p7L31dlo52ERsUNEXBYRb0TE2xFxW0S0rbDfMdm6lRHx\nkyrHmRARMyPi9oh4HxiXHXtuRKyOiBURcUNE7Fhhfykizo2I0qwM498i4svZNu9HxO8q9q9yjtXG\nGhGtIuJDoAXwQkT8fXPfV0rpZeBpoNaEMyL6RsTCLNYZwE4V1pWXTETEE8Bg4Ibsu70T+CkwKls+\nq4ZDPAx8BvyfGo4/JyLOrrBcp1Hm2qSUNqSU/p5SOhf4IzAh23elv4Rkx3o1O/fXIuL0iDgAuAk4\nLDuv1VnfWyPiVxHxUER8BAyOaspdIuLHEfFudh2dXpfzjIiy6/aF7Jijokq5SkQckO1jdRRKrUZU\nWHdrRNwYEQ9m5/J8RHx5W75DSY3DpFvS9uYXwC9SSl8Evgz8Lms/OnvfNSsjmAuMy16DgX2BNsAN\nABFxIPBL4HSgM4UR0j2rHOtEYCawKzAN2ABcCHQADqMwonxulW2GA/2BQRRKMiZnx+hKIQk+rYbz\nqjbWlNKn2QguFEayN5tQZed2FLBJGUqFPjsC9wG/BdoDdwEnV9c3pfQVCkl8WXnLacDPgBnZ8s01\nHCYBPwEuj4iizcWdg3sofA+VRMQXgEnA8SmlXYDDgZKU0kvAd8lGzVNKu1bY7H8DVwO7ANX9YvAl\nCtfFnsBYYHJEbLZEJKVUdt0enB1zRpVYi4DfA48CnYDvA9Oq7Ps04AqgHbAki1NSE2PSLakh3JeN\n4q3ORhd/WUvfdUD3iOiQUvowpfRcLX1PB65LKb2aUvoQuAQYnY12ngL8PqX0TErpMwojt6nK9nNT\nSvellDamlNamlBaklJ5LKa1PKb0O/BdwTJVtrkkpvZ9SehH4K/Bodvw1wB+Amm6CrC3WuloYEe9R\nSNJ+A9xSS99BQBHwHymldSmlmcCft+BYdZJSmgW8A5y9ub45eJPCLxTV2QgcFBE7p5RWZD+v2tyf\nUno2uxY+qaHPT7Jfkv4IPAh8cyvjrmgQhV/AJqaUPkspPQE8QOVf3u5JKc3LSn2mAX3q4biSGphJ\nt6SGMDKltGvZi01Hjys6C9gPeDki/hwRJ9TSdw/gjQrLbwAtKdwAuAewtGxFSuljYGWV7ZdWXIiI\n/SLigYj4n6zk5GcURjcreqvC57XVLLeherXFWlf9UkrtUkpfTildllLaWEvfPYDlKaWKv2i8UVPn\nbXQZcCkVylcayJ4U6sorSSl9BIyiMKq9IivN+F+b2dfSzax/L9tvmTcofMfbag9gaZWf5RtU/qvM\n/1T4/DE1X2OStmMm3ZK2Kyml0qy8oRNwDTAzKxeoOkoNhZHOvSss7wWsp5AIrwC6lK2IiJ2B3aoe\nrsryr4CXgR5ZecuPgdj6s6lzrHlYAewZERXj3yuPA6WUZlMoe6j6y9RHQOsKy1+q50OfRKEsprqY\nHkkpDaVQWvQy8OuyVTXsq6b2Mu2y67DMXhR+prBt5/km0DUiKv7/eC9g+RbsQ1ITYNItabsSEf8n\nIjpmI3+rs+YNFEoYNlKohy5zJ3BhROwTEW34Zx3yegq12l+PiMOz+uYr2HwCvQvwPvBhNjL6vXo7\nsdpjzcNcCkn9BVGYyu8bwCE5HQsKI90XVWkrAb4REa2jMGVeTTdk1llEtMi+w/8Eiin8XKv22T0i\nRmRJ8qfAhxSuISj8ktOlphteN+OKiNgxIo4CTqBQJw+bP8+3qHzdVvQ8haT9oogoisJ89V8Hpm9F\nfJK2YybdkrY3w4EXsxk9fgGMTil9kpWHXA08m9WGDwKmULhR8CngNeATCjeikdXwfp9C8rIC+AB4\nm0ISVpN/pXBD3QcURkZn1NJ3S9UYax6yOvZvULh58z0K5Rb35Hi8Z4F5VZqvpzC7yVvAVAr1yFvr\nsOyaeB+YA3wRGJhS+ks1fXcAfkhhFHkVhbr8slH4JyhMjfg/EfHuFhz/fyh8j29SOI/vZrPIwObP\ncwIwNbtuK9WBZz+nEcDxwLsU7ncYU2HfkpqJqFzuJ0nNUza6vJpC6chrjR2PJOnzxZFuSc1WRHw9\n+5P/Fyg88fEvwOuNG5Uk6fPIpFtSc3YihXKAN4EeFEpVms2f9yJir/jno9GrvnK5aVKStHUsL5Ek\nSZJylttId0TsFBHzIuKF7LG2V2Tt+2SPsS2NiBlld5BH4VHIMyJiSba+W4V9XZK1vxIRx+UVsyRJ\nkpSHPMtLPgW+klI6mMLTs4Znsw1cA1yfUupB4U7wsqmVzqLw8IHuFO4EvwbKH3c8GuhJYVaDX0ZE\nixzjliRJkurVljx+eItkdZMfZotF2SsBX6EwJRcUplaaQOGBFCdmn6Ewv+4N2UMdTgSmp5Q+BV6L\niCUU5pqdW9OxO3TokLp161aPZyOpRgsWVF7u379x4lC1PljwQaXlXfrv0kiR5GfBB5XPsf8uze8c\nJW2/FixY8G5KqePm+uWWdEPhIQbAAqA7cCPwd2B1hYdBLOOfj7rdk+wxvCml9RGxhsLT4/YEnquw\n24rbVKtbt27Mnz+/vk5DUm2iyvNm/G9vuzIn5lRaLp5f3Chx5CnmzKm0PL+4uFHikPT5FBFv1KVf\nrrOXpJQ2pJT6UHgU8yHAAdV1y96re1JcqqW9kog4JyLmR8T8d955Z2tDliRJkupdg0wZmFJaTeEJ\nYoOAXSOibIS9C4WpvKAwgt0VIFvflsKTxMrbq9mm4jEmp5QGpJQGdOy42RF+SZIkqcHkOXtJx4jY\nNfu8M3As8BLwJHBK1m0scH/2eVa2TLb+iawufBYwOpvdZB8Kc+1WfdSwJEmStN3Ks6a7MzA1q+ve\nAfhdSumBiFgMTI+Iq4D/Bm7O+t8M/Da7UXIVhRlLSCm9GBG/AxYD64HzUkobcoxbkiRtp9atW8ey\nZcv45JNPGjsUfc7stNNOdOnShaKioq3aPs/ZSxYBfatpf5VCfXfV9k+AU2vY19XA1fUdoyRJalqW\nLVvGLrvsQrdu3YiqN3JLOUkpsXLlSpYtW8Y+++yzVfvwMfCSJKnJ+OSTT9htt91MuNWgIoLddttt\nm/7CYtItSZKaFBNuNYZtve5MuiVJkrZAixYt6NOnDz179uTggw/muuuuY+PGjbVu8/rrr3PHHXfU\neyz/8R//wccff1ztuuLiYvbff38OPvhgjjjiCF555ZV6P35tSkpKeOihh8qXZ82axcSJE7dpn8OH\nD2f58uUUFxez1157UZhzo2DkyJG0adNmm/afJ5NuSZKkLbDzzjtTUlLCiy++yOzZs3nooYe44oor\nat2mMZJugGnTpvHCCy8wduxYfvSjH22yfsOGfOamWL9+/SZJ94gRIxg/fvxW73Pt2rWsWrWKPfcs\nPCNx11135dlnnwVg9erVrFixYtuCzplJtyRJ0lbq1KkTkydP5oYbbiClxOuvv85RRx1Fv3796Nev\nH3/6058AGD9+PE8//TR9+vTh+uuvr7HfihUrOProo+nTpw8HHXQQTz/9NACPPvoohx12GP369ePU\nU0/lww8/ZNKkSbz55psMHjyYwYMH1xrn0UcfzZIlS4DCk7uvvPJKjjzySO666y5KSkoYNGgQvXv3\n5qSTTuK9994DCiPlP/jBDzj88MM56KCDmDevMGPzqlWrGDlyJL1792bQoEEsWrQIgAkTJnDOOecw\nbNgwxowZw09/+lNmzJhBnz59mDFjBrfeeivnn38+AG+88QZDhgyhd+/eDBkyhH/84x8AjBs3jgsu\nuIDDDz+cfffdl5kzZ5afw5w5cyiu8MTZ0aNHM336dADuuecevvGNb1Q652uvvZaBAwfSu3dvLr/8\n8vL2kSNH0r9/f3r27MnkyZPL29u0acOll17KwQcfzKBBg3jrrbc2+/PfEibdkiSp6YnI97UF9t13\nXzZu3Mjbb79Np06dmD17NgsXLmTGjBlccMEFAEycOJGjjjqKkpISLrzwwhr73XHHHRx33HGUlJTw\nwgsv0KdPH959912uuuoqHnvsMRYuXMiAAQO47rrruOCCC9hjjz148sknefLJJ2uN8fe//z29evUq\nX95pp5145plnGD16NGPGjOGaa65h0aJF9OrVq9Ko/UcffcSf/vQnfvnLX3LmmWcCcPnll9O3b18W\nLVrEz372M8aMGVPef8GCBdx///3ccccdXHnllYwaNYqSkhJGjRpVKZ7zzz+fMWPGsGjRIk4//fTy\n84fCLx7PPPMMDzzwQKWR8T/84Q8MHz68fHnIkCE89dRTbNiwgenTp1c6xqOPPkppaSnz5s2jpKSE\nBQsW8NRTTwEwZcoUFixYwPz585k0aRIrV64sP9dBgwbxwgsvcPTRR/PrX/+61u90S+U5T7ckNVnd\nxj/Y2CHUi1v5QqXl2s7r9Ylfyzscqdkqqy1et24d559/PiUlJbRo0YK//e1v1favqd/AgQM588wz\nWbduHSNHjqRPnz788Y9/ZPHixRxxxBEAfPbZZxx22GF1iuv0009n5513plu3bvznf/5neXtZgrpm\nzRpWr17NMcccA8DYsWM59dR/zuB82mmnAYWR8vfff5/Vq1fzzDPPcPfddwPwla98hZUrV7JmzRqg\nUEKy8847bzauuXPncs899wBwxhlncNFFF5WvGzlyJDvssAMHHnhgpdHmZ599lp///Oflyy1atODI\nI49kxowZrF27lm7dupWve/TRR3n00Ufp27cwe/WHH35IaWkpRx99NJMmTeLee+8FYOnSpZSWlrLb\nbrux4447csIJJwDQv39/Zs+evdnz2BIm3ZIkSdvg1VdfpUWLFnTq1IkrrriC3XffnRdeeIGNGzey\n0047VbvN9ddfX22/o48+mqeeeooHH3yQM844gx/96Ee0a9eOoUOHcuedd25xbNOmTWPAgAGbtH/h\nC1+opvemqs7YERGVbl6s2q+u+63tOK1atSr/XHasV199la5du7LjjjtW2m706NGcdNJJTJgwoVJ7\nSolLLrmE73znO5Xa58yZw2OPPcbcuXNp3bo1xcXF5dMAFhUVlcfRokUL1q9fv1XnUhPLSyRJkrbS\nO++8w3e/+13OP/98IoI1a9bQuXNndthhB37729+W36i4yy678MEHH5RvV1O/N954g06dOvHtb3+b\ns846i4ULFzJo0CCeffbZ8prsjz/+uHxkvOp+t1Tbtm1p165dee34b3/72/JRb4AZM2YA8Mwzz9C2\nbVvatm3L0UcfzbRp04BCEtuhQwe++MUvbrLv2mI7/PDDy+uxp02bxpFHHllrnFVLS8ocddRRXHLJ\nJeUj8mWOO+44pkyZwocffgjA8uXLefvtt1mzZg3t2rWjdevWvPzyyzz33HO1Hrc+OdItSZKanmpG\nWxvK2rVr6dOnD+vWraNly5acccYZ/Mu//AsA5557LieffDJ33XUXgwcPLh/57d27Ny1btuTggw9m\n3LhxNfabM2cO1157LUVFRbRp04bbbruNjh07cuutt3Laaafx6aefAnDVVVex3377cc4553D88cfT\nuXPnzdZ112Tq1Kl897vf5eOPP2bffffllltuKV/Xrl07Dj/8cN5//32mTJkCFG6Y/Na3vkXv3r1p\n3bo1U6dOrXa/gwcPZuLEifTp04dLLrmk0rpJkyZx5plncu2119KxY8dKx6zOww8/XKk8pkxE8K//\n+q+btA8bNoyXXnqpvAynTZs23H777QwfPpybbrqJ3r17s//++zNo0KDav5x6FNX9iaCpGzBgQJo/\nf35jhyF9PlS94aiZ/JvSbGq6r6n8p95xF39UY9+mWtMdc+ZUWk4VZjdQ8/PSSy9xwAEHNHYYnwvF\nxcX8/Oc/r7Y8pSF9+umnHHHEEWwPuV11119ELEgpbfZLsrxEkiRJ261WrVptFwn3trK8RJIkSZuY\nU+WvSNo2jnRLkiRJOTPpliRJknJm0i1JkiTlzKRbkiRJyplJtyRJ0ha4+uqr6dmzJ71796ZPnz48\n//zzAJx99tksXry4Xo7RrVs33n333Vr7/OxnP9vi/d56662cf/75WxtWuTPPPJNOnTpx0EEHVWpf\ntWoVQ4cOpUePHgwdOpT33nsPKDwh8oILLqB79+707t2bhQsXlm8zdepUevToQY8ePSrN+b1gwQJ6\n9epF9+7dueCCC6p9EmZTYtItSZJUR3PnzuWBBx5g4cKFLFq0iMcee4yuXbsC8Jvf/IYDDzywwWLZ\nmqR7S61atara9nHjxvHwww9v0j5x4kSGDBlCaWkpQ4YMYeLEiUDhiZKlpaWUlpYyefJkvve975Xv\n/4orruD5559n3rx5XHHFFeWJ+ve+9z0mT55cvl11x2tKTLolSZLqaMWKFXTo0IFWrVoB0KFDB/bY\nYw+g8DCZsvmk27Rpw8UXX0z//v059thjmTdvHsXFxey7777MmjUL2HTU+YQTTqh2mr6RI0fSv39/\nevbsyeTJkwEYP358+ZMxTz/9dABuv/12DjnkEPr06cN3vvOd8kfL33LLLey3334cc8wxPPvss5s9\nx/Xr1zNr1ixGjBjBSSedVG2fo48+mvbt22/Sfv/99zN27FgAxo4dy3333VfePmbMGCKCQYMGsXr1\nalasWMEjjzzC0KFDad++Pe3atWPo0KE8/PDDrFixgvfff5/DDjuMiGDMmDHl+2qqnKdbkiQ1OVWf\nRFrfanqy6bBhw7jyyivZb7/9OPbYYxk1ahTHHHPMJv0++ugjiouLueaaazjppJO47LLLmD17NosX\nL2bs2LGMGDGizrFMmTKF9u3bs3btWgYOHMjJJ5/MxIkTueGGGygpKQEKT0qcMWMGzz77LEVFRZx7\n7rlMmzaNoUOHcvnll7NgwQLatm3L4MGD6du3b7XHWbJkCTfffDMzZ87k8MMP54c//GG151abt956\ni86dOwPQuXNn3n77bQCWL19e/hcBgC5durB8+fJa27t06bJJe1Nm0i1JklRHbdq0YcGCBTz99NM8\n+eSTjBo1iokTJzJu3LhK/XbccUeGDx8OQK9evWjVqhVFRUX06tWL119/fYuOOWnSJO69914Ali5d\nSmlpKbvttlulPo8//jgLFixg4MCBAKxdu5ZOnTrx/PPPU1xcTMeOHQEYNWoUf/vb3zY5xt13382o\nUaO49NJLWbhwIbvssssWxbg51dVjR8QWtzdllpdIkiRtgRYtWlBcXMwVV1zBDTfcwN13371Jn6Ki\novIkcYcddigvR9lhhx1Yv349AC1btmTjxo3l23zyySeb7GfOnDk89thjzJ07lxdeeIG+fftW2y+l\nxNixYykpKaGkpIRXXnmFCRMmAHVLVocOHcovfvELHnzwQU4++WTuvPPOao+zObvvvjsrVqwACqU4\nnTp1Agoj1UuXLi3vt2zZMvbYY49a25ctW7ZJe1Nm0i1JklRHr7zyCqWlpeXLJSUl7L333lu1r27d\nulFSUsLGjRtZunQp8+bN26TPmjVraNeuHa1bt+bll1/mueeeK19XVFTEunXrABgyZAgzZ84sL+dY\ntWoVb7zxBoceeihz5sxh5cqVrFu3jrvuuqvaWL74xS9y3nnnMX/+fK655hqeeeYZDjjgAC666KIt\nOqcRI0aUz0AydepUTjzxxPL22267jZQSzz33HG3btqVz584cd9xxPProo7z33nu89957PProoxx3\n3HF07tyZXXbZheeee46UErfddlv5vpoqy0skSVKTU1PNdd4+/PBDvv/977N69WpatmxJ9+7dy29u\n3FJHHHEE++yzD7169eKggw6iX79+m/QZPnw4N910E71792b//fdn0KBB5evOOeccevfuTb9+/Zg2\nbRpXXXUVw4YNY+PGjRQVFXHjjTcyaNAgJkyYwGGHHUbnzp3p169f+Q2WNenbty833ngjn3zyCU88\n8US1fU477TTmzJnDu+++S5cuXbjiiis466yzGD9+PN/85je5+eab2WuvvcqT/K9+9as89NBDdO/e\nndatW3PLLbcA0L59e37yk5+Ul8X89Kc/Lb9B81e/+hXjxo1j7dq1HH/88Rx//PFb/iVvR6Kpz3lY\nnQEDBqSyu4cl5azqny2byb8p3cY/2Ngh1Itbr/lCpeVxF39UY9/XJ34t73ByUfWGusZKxtQwXnrp\nJQ444IDGDkOfU9VdfxGxIKU0YHPbWl4iSZIk5cykW5IkScqZNd2SJKAJl9QMr1xCsyXn0VRLaiQ1\nPY50S5IkSTkz6ZYkSZJyZtItSZIk5cykW5IkaQtcffXV9OzZk969e9OnTx+ef/55AM4++2wWL15c\nL8fo1q0b7777bq19fvazn23xfm+99VbOP//8rQ2r3KWXXkrXrl1p06ZNpfZPP/2UUaNG0b17dw49\n9NBKj7z/93//d7p3787+++/PI488Ut7+8MMPs//++9O9e3cmTpxY3v7aa69x6KGH0qNHD0aNGsVn\nn322zXE3JpNuSZKkOpo7dy4PPPAACxcuZNGiRTz22GN07doVgN/85jcceOCBDRbL1iTd9eXrX/96\ntU/QvPnmm2nXrh1Llizhwgsv5OKLLwZg8eLFTJ8+nRdffJGHH36Yc889lw0bNrBhwwbOO+88/vCH\nP7B48WLuvPPO8l9cLr74Yi688EJKS0tp164dN998c4OeY30z6ZYkSaqjFStW0KFDB1q1agVAhw4d\n2GOPPQAoLi6m7OF8bdq04eKLL6Z///4ce+yxzJs3j+LiYvbdd19mzZoFbDrqfMIJJzCnysOeAEaO\nHEn//v3p2bNn+dMvx48fz9q1a+nTpw+nn346ALfffjuHHHIIffr04Tvf+U75kydvueUW9ttvP445\n5hieffbZevkeBg0aROfOnTdpv//++xk7diwAp5xyCo8//jgpJe6//35Gjx5Nq1at2GeffejevTvz\n5s1j3rx5dO/enX333Zcdd9yR0aNHc//995NS4oknnuCUU04BYOzYsdx33331EntjccpASZLU5MyJ\nObnuvzgVV9s+bNgwrrzySvbbbz+OPfZYRo0axTHHHLNJv48++oji4mKuueYaTjrpJC677DJmz57N\n4sWLGTt2LCNGjKhzLFOmTKF9+/asXbuWgQMHcvLJJzNx4kRuuOEGSkpKgMKTEmfMmMGzzz5LUVER\n5557LtOmTWPo0KFcfvnlLFiwgLZt2zJ48GD69u27yTGefPJJLrzwwk3aW7duzZ/+9Kc6x7p8+fLy\nkf+WLVvStm1bVq5cyfLlyys9wr5Lly4sX74coLx/Wfvzzz/PypUr2XXXXWnZsuUm/Zuq3JLuiOgK\n3AZ8CdgITE4p/SIiJgDfBt7Juv44pfRQts0lwFnABuCClNIjWftw4BdAC+A3KaWJSJIkNbA2bdqw\nYMECnn76aZ588klGjRrFxIkTGTduXKV+O+64I8OHDwegV69etGrViqKiInr16lWpzrkuJk2axL33\n3gvA0qVLKS0tZbfddqvU5/HHH2fBggUMHDgQgLVr19KpUyeef/55iouL6dixIwCjRo3ib3/72ybH\nGDx4cHkCvy1SSpu0RUSN7Rs3btyi/k1ZniPd64EfppQWRsQuwIKImJ2tuz6l9POKnSPiQGA00BPY\nA3gsIvbLVt8IDAWWAX+OiFkppfq5U0GSJGkLtGjRguLiYoqLi+nVqxdTp07dJOkuKioqTxJ32GGH\n8nKUHXbYgfXr1wOFkeCKSecnn3yyybHmzJnDY489xty5c2ndujXFxcXV9kspMXbsWP793/+9Uvt9\n991Xp2S1vka6u3TpwtKlS+nSpQvr169nzZo1tG/fvry9zLJly8rLcqpr79ChA6tXr2b9+vW0bNmy\nUv+mKrea7pTSipTSwuzzB8BLwJ61bHIiMD2l9GlK6TVgCXBI9lqSUno1pfQZMD3rK0mS1KBeeeUV\nSktLy5dLSkrYe++9t2pf3bp1o6SkhI0bN7J06dJqb0xcs2YN7dq1o3Xr1rz88ss899xz5euKiopY\nt24dAEOGDGHmzJm8/fbbAKxatYo33niDQw89lDlz5rBy5UrWrVvHXXfdVW0sZSPdVV9bknADjBgx\ngqlTpwIwc+ZMvvKVrxARjBgxgunTp/Ppp5/y2muvUVpayiGHHMLAgQMpLS3ltdde47PPPmP69OmM\nGDGCiGDw4MHMnDkTgKlTp3LiiU07/WuQmu6I6Ab0BZ4HjgDOj4gxwHwKo+HvUUjIn6uw2TL+maQv\nrdJ+aM4hS5Kk7VhNNdd5+/DDD/n+97/P6tWradmyJd27dy+/uXFLHXHEEeyzzz706tWLgw46iH79\n+m3SZ/jw4dx000307t2b/fffv1Jd9DnnnEPv3r3p168f06ZN46qrrmLYsGFs3LiRoqIibrzxRgYN\nGsSECROtlq78AAAaCUlEQVQ47LDD6Ny5M/369Su/wXJbXHTRRdxxxx18/PHHdOnShbPPPpsJEyZw\n1llnccYZZ9C9e3fat2/P9OnTAejZsyff/OY3OfDAA2nZsiU33ngjLVq0AOCGG27guOOOY8OGDZx5\n5pn07NkTgGuuuYbRo0dz2WWX0bdvX84666xtjrsxRXU1M/V6gIg2wB+Bq1NK90TE7sC7QAL+Deic\nUjozIm4E5qaUbs+2uxl4iMJo/HEppbOz9jOAQ1JK369ynHOAcwD22muv/m+88Uau5yUpU/XPljn/\nm9JQuo1/sLFDqBe3XvOFSsvjLv6okSLJzxvDK5/j3g/X/Rxfn/i1+g5HOXvppZc44IADGjsMfU5V\nd/1FxIKU0oDNbZvrlIERUQTcDUxLKd0DkFJ6K6W0IaW0Efg1hfIRKIxgd62weRfgzVraK0kpTU4p\nDUgpDSi7WUCSJEnaHuSWdEehav9m4KWU0nUV2itO6ngS8Nfs8yxgdES0ioh9gB7APODPQI+I2Cci\ndqRws+WsvOKWJEmS6lueNd1HAGcAf4mIsjlofgycFhF9KJSXvA58ByCl9GJE/A5YTGHmk/NSShsA\nIuJ84BEKUwZOSSm9mGPckiRJUr3KLelOKT0DVDdHzUO1bHM1cHU17Q/Vtp0kSfr8SCk1+Tmb1fRs\n632QPgZekiQ1GTvttBMrV67c5gRI2hIpJVauXMlOO+201fvwMfCSJKnJ6NKlC8uWLeOdd97ZfGep\nHu2000506dJlq7c36ZYkSU1GUVER++yzT2OHIW0xy0skSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk\n5cykW5IkScqZSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk\n5cykW5IkScqZSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk\n5cykW5IkScqZSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk\n5cykW5IkScqZSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknOWWdEdE14h4MiJeiogXI+L/Zu3t\nI2J2RJRm7+2y9oiISRGxJCIWRUS/Cvsam/UvjYixecUsSZIk5SHPke71wA9TSgcAg4DzIuJAYDzw\neEqpB/B4tgxwPNAje50D/AoKSTpwOXAocAhweVmiLkmSJDUFuSXdKaUVKaWF2ecPgJeAPYETgalZ\nt6nAyOzzicBtqeA5YNeI6AwcB8xOKa1KKb0HzAaG5xW3JEmSVN8apKY7IroBfYHngd1TSiugkJgD\nnbJuewJLK2y2LGurqV2SJElqEnJPuiOiDXA38IOU0vu1da2mLdXSXvU450TE/IiY/84772xdsJIk\nSVIOck26I6KIQsI9LaV0T9b8VlY2Qvb+dta+DOhaYfMuwJu1tFeSUpqcUhqQUhrQsWPH+j0RSZIk\naRvkOXtJADcDL6WUrquwahZQNgPJWOD+Cu1jsllMBgFrsvKTR4BhEdEuu4FyWNYmSZIkNQktc9z3\nEcAZwF8ioiRr+zEwEfhdRJwF/AM4NVv3EPBVYAnwMfAtgJTSqoj4N+DPWb8rU0qrcoxbkiRJqle5\nJd0ppWeovh4bYEg1/RNwXg37mgJMqb/oJEmSpIbjEyklSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk\n5cykW5IkScqZSbckSZKUM5NuSZIkKWcm3ZIkSVLO8nwipaRmpNv4B6ttf72O/SRJ+jxzpFuSJEnK\nmUm3JEmSlDOTbkmSJClnJt2SJElSzky6JUmSpJzVKemOiIPyDkSSJElqruo60n1TRMyLiHMjYtdc\nI5IkSZKamTol3SmlI4HTga7A/Ii4IyKG5hqZJEmS1EzUuaY7pVQKXAZcDBwDTIqIlyPiG3kFJ0mS\nJDUHda3p7h0R1wMvAV8Bvp5SOiD7fH2O8UmSJElNXl0fA38D8GvgxymltWWNKaU3I+KyXCKTJEmS\nmom6Jt1fBdamlDYARMQOwE4ppY9TSr/NLTpJkiSpGahrTfdjwM4VlltnbZIkSZI2o65J904ppQ/L\nFrLPrfMJSZIkSWpe6pp0fxQR/coWIqI/sLaW/pIkSZIyda3p/gFwV0S8mS13BkblE5IkSZLUvNQp\n6U4p/Tki/hewPxDAyymldblGJkmSJDUTdR3pBhgIdMu26RsRpJRuyyUqSZIkqRmpU9IdEb8FvgyU\nABuy5gSYdEuSJEmbUdeR7gHAgSmllGcwkiRJUnNU19lL/gp8Kc9AJEmSpOaqriPdHYDFETEP+LSs\nMaU0IpeoJEmSpGakrkn3hDyDkCRJkpqzuk4Z+MeI2BvokVJ6LCJaAy3yDU2SJElqHupU0x0R3wZm\nAv+VNe0J3JdXUJIkSVJzUtcbKc8DjgDeB0gplQKd8gpKkiRJak7qmnR/mlL6rGwhIlpSmKdbkiRJ\n0mbUNen+Y0T8GNg5IoYCdwG/r22DiJgSEW9HxF8rtE2IiOURUZK9vlph3SURsSQiXomI4yq0D8/a\nlkTE+C07PUmSJKnx1TXpHg+8A/wF+A7wEHDZZra5FRheTfv1KaU+2eshgIg4EBgN9My2+WVEtIiI\nFsCNwPHAgcBpWV9JkiSpyajr7CUbgV9nrzpJKT0VEd3q2P1EYHpK6VPgtYhYAhySrVuSUnoVICKm\nZ30X1zUOSZIkqbHVKemOiNeopoY7pbTvVhzz/IgYA8wHfphSeo/CbCjPVeizLGsDWFql/dCtOKYk\nSZLUaOpaXjIAGJi9jgImAbdvxfF+BXwZ6AOsAP7/rD2q6Ztqad9ERJwTEfMjYv4777yzFaFJkiRJ\n+ahT0p1SWlnhtTyl9B/AV7b0YCmlt1JKGyqUq5SVkCwDulbo2gV4s5b26vY9OaU0IKU0oGPHjlsa\nmiRJkpSbupaX9KuwuAOFke9dtvRgEdE5pbQiWzwJKJvZZBZwR0RcB+wB9ADmURjp7hER+wDLKdxs\n+b+39LiSJElSY6pT0s0/y0AA1gOvA9+sbYOIuBMoBjpExDLgcqA4IvpQKBF5ncJMKKSUXoyI31G4\nQXI9cF5KaUO2n/OBRyg8dn5KSunFOsYsSZIkbRfqOnvJ4C3dcUrptGqab66l/9XA1dW0P0RhikJJ\nkiSpSaprecm/1LY+pXRd/YQjSZIkNT91LS8pm71kVrb8deApKk/nJ0mSJKkadU26OwD9UkofQOFx\n7sBdKaWz8wpMkiRJai7qmnTvBXxWYfkzoFu9RyNJUgPqNv7Bxg6hwb0+8WuNHYL0uVTXpPu3wLyI\nuJfCzCMnAbflFpUkSZLUjNR19pKrI+IPFJ5GCfCtlNJ/5xeWJEmS1HzU9THwAK2B91NKvwCWZQ+s\nkSRJkrQZdUq6I+Jy4GLgkqypCLg9r6AkSZKk5qSuI90nASOAjwBSSm+yFY+BlyRJkj6P6pp0f5ZS\nShRuoiQivpBfSJIkSVLzUtek+3cR8V/ArhHxbeAx4Nf5hSVJkiQ1H3WdveTnETEUeB/YH/hpSml2\nrpFJkiRJzcRmk+6IaAE8klI6FjDRliRJkrbQZstLUkobgI8jom0DxCNJkiQ1O3V9IuUnwF8iYjbZ\nDCYAKaULcolKkiRJakbqmnQ/mL0kSZIkbaFak+6I2Cul9I+U0tSGCkiSJElqbjZX031f2YeIuDvn\nWCRJkqRmaXNJd1T4vG+egUiSJEnN1eaS7lTDZ0mSJEl1tLkbKQ+OiPcpjHjvnH0mW04ppS/mGp0k\nSZLUDNSadKeUWjRUIJIkSVJztdmH40iSJEnaNibdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm\n0i1JkiTlzKRbkiRJyplJtyRJkpQzk25JkiQpZybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm\n0i1JkiTlLLekOyKmRMTbEfHXCm3tI2J2RJRm7+2y9oiISRGxJCIWRUS/CtuMzfqXRsTYvOKVJEmS\n8pLnSPetwPAqbeOBx1NKPYDHs2WA44Ee2esc4FdQSNKBy4FDgUOAy8sSdUmSJKmpyC3pTik9Bayq\n0nwiMDX7PBUYWaH9tlTwHLBrRHQGjgNmp5RWpZTeA2azaSIvSZIkbdcauqZ795TSCoDsvVPWview\ntEK/ZVlbTe2SJElSk7G93EgZ1bSlWto33UHEORExPyLmv/POO/UanCRJkrQtGjrpfisrGyF7fztr\nXwZ0rdCvC/BmLe2bSClNTikNSCkN6NixY70HLkmSJG2thk66ZwFlM5CMBe6v0D4mm8VkELAmKz95\nBBgWEe2yGyiHZW2SJElSk9Eyrx1HxJ1AMdAhIpZRmIVkIvC7iDgL+Adwatb9IeCrwBLgY+BbACml\nVRHxb8Cfs35XppSq3pwpSZIkbddyS7pTSqfVsGpINX0TcF4N+5kCTKnH0CRJkqQGtb3cSClJkiQ1\nWybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm0i1JkiTlzKRbkiRJyplJtyRJkpQzk25JkiQp\nZybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm0i1JkiTlzKRbkiRJyplJtyRJkpQzk25JkiQp\nZybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm0i1JkiTlzKRbkiRJyplJtyRJkpQzk25JkiQp\nZybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm0i1JkiTlrGVjByA1Rd3GP9jYIUiSpCbEkW5J\nkiQpZybdkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknLWKEl3RLweEX+JiJKImJ+1tY+I2RFRmr23\ny9ojIiZFxJKIWBQR/RojZkmSJGlrNeZI9+CUUp+U0oBseTzweEqpB/B4tgxwPNAje50D/KrBI5Uk\nSZK2wfZUXnIiMDX7PBUYWaH9tlTwHLBrRHRujAAlSZKkrdFYSXcCHo2IBRFxTta2e0ppBUD23ilr\n3xNYWmHbZVmbJEmS1CQ01hMpj0gpvRkRnYDZEfFyLX2jmra0SadC8n4OwF577VU/UUqSJEn1oFFG\nulNKb2bvbwP3AocAb5WVjWTvb2fdlwFdK2zeBXizmn1OTikNSCkN6NixY57hS5IkSVukwZPuiPhC\nROxS9hkYBvwVmAWMzbqNBe7PPs8CxmSzmAwC1pSVoUiSJElNQWOUl+wO3BsRZce/I6X0cET8Gfhd\nRJwF/AM4Nev/EPBVYAnwMfCthg9ZkiRJ2noNnnSnlF4FDq6mfSUwpJr2BJzXAKFJkiRJudiepgyU\nJEmSmiWTbkmSJClnJt2SJElSzhprnm5JktQIuo1/sLFDaHCvT/xaY4cgOdItSZIk5c2kW5IkScqZ\nSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk5cykW5IkScqZ\nSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk5cykW5IkScqZ\nSbckSZKUM5NuSZIkKWcm3ZIkSVLOTLolSZKknJl0S5IkSTkz6ZYkSZJyZtItSZIk5cykW5IkScpZ\ny8YOQE1ft/EPNnYIkiRJ2zVHuiVJkqScmXRLkiRJObO8RJIkNWufxzLI1yd+rbFDUBWOdEuSJEk5\nM+mWJEmSctZkku6IGB4Rr0TEkogY39jxSJIkSXXVJJLuiGgB3AgcDxwInBYRBzZuVJIkSVLdNJUb\nKQ8BlqSUXgWIiOnAicDiRo2qGp/HmzUkSZJUu6aSdO8JLK2wvAw4tJFikSRJ2q59HgcBt/cZWyKl\n1NgxbFZEnAocl1I6O1s+AzgkpfT9Cn3OAc7JFvcHXmnwQLcfHYB3GzsIbRe8FgReB/onrwWV8Vqo\nP3unlDpurlNTGeleBnStsNwFeLNih5TSZGByQwa1vYqI+SmlAY0dhxqf14LA60D/5LWgMl4LDa9J\n3EgJ/BnoERH7RMSOwGhgViPHJEmSJNVJkxjpTimtj4jzgUeAFsCUlNKLjRyWJEmSVCdNIukGSCk9\nBDzU2HE0EZbZqIzXgsDrQP/ktaAyXgsNrEncSClJkiQ1ZU2lpluSJElqsky6m5CIaB8RsyOiNHtv\nV0O/sVmf0ogYW6G9f0T8JSKWRMSkiIgq2/1rRKSI6JD3uWjr5XUdRMS1EfFyRCyKiHsjYteGOidt\nmYgYHhGvZD/D8dWsbxURM7L1z0dEtwrrLsnaX4mI4+q6T22f6vtaiIiuEfFkRLwUES9GxP9tuLPR\ntsjj34VsXYuI+O+IeCD/s2jmUkq+msgL+P+A8dnn8cA11fRpD7yavbfLPrfL1s0DDgMC+ANwfIXt\nulK4UfUNoENjn6uvhr8OgGFAy+zzNdXt11fjvyjcTP53YF9gR+AF4MAqfc4Fbso+jwZmZJ8PzPq3\nAvbJ9tOiLvv0tf29croWOgP9sj67AH/zWtj+X3lcCxW2+xfgDuCBxj7Ppv5ypLtpORGYmn2eCoys\nps9xwOyU0qqU0nvAbGB4RHQGvphSmpsK/xXdVmX764GLAIv8t3+5XAcppUdTSuuz7Z+jMB++tj+H\nAEtSSq+mlD4DplO4JiqqeI3MBIZkf9E4EZieUvo0pfQasCTbX132qe1PvV8LKaUVKaWFACmlD4CX\nKDwVWtu3PP5dICK6AF8DftMA59DsmXQ3LbunlFYAZO+dqumzJ7C0wvKyrG3P7HPVdiJiBLA8pfRC\nHkGr3uVyHVRxJoVRcG1/avrZVtsn+0VqDbBbLdvWZZ/a/uRxLZTLyg/6As/XY8zKR17Xwn9QGJDb\nWP8hf/40mSkDPy8i4jHgS9WsurSuu6imLdXUHhGts30Pq+P+1QAa+jqocuxLgfXAtDoeSw1rsz/D\nWvrU1F7dAIx/9dr+5XEtFDaKaAPcDfwgpfT+VkeohlLv10JEnAC8nVJaEBHF2xifMOne7qSUjq1p\nXUS8FRGdU0orsjKBt6vptgworrDcBZiTtXep0v4m8GUKNVwvZPfTdQEWRsQhKaX/2YZT0TZohOug\nbN9jgROAIVn5ibY/yyjcg1Gm0s+wSp9lEdESaAus2sy2m9untj+5XAsRUUQh4Z6WUronn9BVz/K4\nFkYAIyLiq8BOwBcj4vaU0v/J5xSaP8tLmpZZQNksFGOB+6vp8wgwLCLaZbNaDAMeycoQPoiIQVkN\n1xjg/pTSX1JKnVJK3VJK3Sj8x9fPhHu7Vu/XARTufAcuBkaklD7O+yS01f4M9IiIfSJiRwo3RM2q\n0qfiNXIK8ET2S9QsYHQ2i8E+QA8KN9bWZZ/a/tT7tZD9u3Az8FJK6boGOQvVh3q/FlJKl6SUumS5\nweisvwn3tmjsOzl91f1FofbqcaA0e2+ftQ8AflOh35kUboRYAnyrQvsA4K8U7ky+gezhSFWO8TrO\nXrJdv/K6DrJ+S4GS7HVTY5+rrxqvga9SmFXi78ClWduVFH5hgsKo1F3Zz3QesG+FbS/NtnuFyjMY\nbbJPX9v/q76vBeBICiUHiyr8W/DVxj5PXw1/LVTZdzHOXrLNL59IKUmSJOXM8hJJkiQpZybdkiRJ\nUs5MuiVJkqScmXRLkiRJOTPpliRJknJm0i1JkiTlzKRbkhpIRHwpIqZHxN8jYnFEPBQR+zVSLOMi\nYo8t3KZbRPy1hva1EVGSnddNEVGn/79ExK0RcUr2+TcRcWD2+dSIeCkinsyW74yIRRFx4ZbELEnb\nCx8DL0kNIHvS373A1JTS6KytD7A7hQdaNGQsLYBxFB6SVF+Pe/97SqlP9njpJ4CRwBY9QjyldHaF\nxbOAc1NKT0bEl4DDU0p711OsktTgHOmWpIYxGFiXUrqprCGlVJJSejoKro2Iv0bEXyJiFEBEFEfE\nnIiYGREvR8S0rO/xEfG7sv1k/X6ffR4WEXMjYmFE3BURbbL21yPipxHxDHAahSeTTstGp3eOiP4R\n8ceIWBARj0RE52y7/hHxQkTMBc7b3EmmlNYDfwK6V7c+i/+GbET8QaBThXVzImJARPyUwpMRb4qI\na4FHgU5ZrEdtyZcuSdsLk25JahgHAQtqWPcNoA9wMHAscG1Z0gv0BX4AHAjsCxwBzAYGRcQXsj6j\ngBkR0QG4DDg2pdQPmA/8S4XjfJJSOjKldHu27vSUUh9gPfCfwCkppf7AFODqbJtbgAtSSofV5SQj\nojUwBPhLDV1OAvYHegHfBg6v2iGldGWF+H4EjCAbSU8pPV2XOCRpe2N5iSQ1viOBO1NKG4C3IuKP\nwEDgfWBeSmkZQESUAN1SSs9ExMPA1yNiJvA14CLgGArJ+bOFahZ2BOZWOM6MGo6/P4VfCmZn27UA\nVkREW2DXlNIfs36/BY6vYR9fzuJLwP0ppT/U0O/oCuf6ZkQ8UeO3IknNiEm3JDWMF4FTalgXtWz3\naYXPG/jnv9szKJR7rAL+nFL6IKsbn51SOq2GfX1Uy/FfrDqaHRG7Ukii6+Lv2ah5XdR1n5LUbFhe\nIkkN4wmgVUR8u6whIgZGxDHAU8CoiGgRER0pjAbP28z+5gD9KJRolI1gPwccERHds/23rmV2lA+A\nXbLPrwAdI+KwbLuiiOiZUloNrImII7N+p9f9dGv0FDA6O9fOFGrdJanZM+mWpAaQUkoU6pmHZlMG\nvghMoDB7yL3AIuAFCsn5RSml/9nM/jYAD1Ao93gga3uHwqwkd0bEIgpJ+P+qYRe3UrhRsYRCOckp\nwDUR8QJQwj9rrb8F3JjdSLl2i098U/cCpRRqvn8F/LH27pLUPETh/wOSJEmS8uJItyRJkpQzb6SU\nJNW7iOhFYbaTij5NKR3aGPFIUmOzvESSJEnKmeUlkiRJUs5MuiVJkqScmXRLkiRJOTPpliRJknJm\n0i1JkiTl7P8Bk5KY+8ABPVwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114905320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting histogram of means computed and the simulated H0\n",
    "plt.figure(figsize=(12,5))\n",
    "plt.hist(p_diffs)\n",
    "plt.axvline(p_diff,c='r',lw=4,label='Dataset Proportion/Mean')\n",
    "plt.axvline(p_diff_sim, c='c', lw=4, label='Simulated < 10000')\n",
    "plt.axvline(p_diffs_sim10000, c='m', lw=4, label='Simulated = 10000')\n",
    "plt.xlabel('Converted P_diff')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Histogram of P_diff Null Distribution')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No simulated p_diff (under the sample): -0.1578 %\n",
      "Simulated 10000 samples p_diff (under H0 assumed conditions): 0.0015%\n",
      "Proportion greater 0.1593%\n",
      "Pval: 0.9065\n"
     ]
    }
   ],
   "source": [
    "# Analyzing the means and the histogram values to understand if our null hypothesis is rejected\n",
    "print('No simulated p_diff (under the sample): {:.4f} %'.format(100.0*p_diff))\n",
    "print('Simulated 10000 samples p_diff (under H0 assumed conditions): {:.4f}%'.format(100.0*p_diffs_sim10000))\n",
    "\n",
    "prop_greater_pdiffs = p_diffs_sim10000 - p_diff\n",
    "print('Proportion greater {:.4f}%'.format(100.0*prop_greater_pdiffs))\n",
    "\n",
    "p_val = (p_diffs > p_diff).mean()\n",
    "print('Pval: {:.4f}'.format(p_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k. In words, explain what you just computed in part **j.**  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean of the simulated 10,000 samples is the difference of the treatment group mean of converted values and the control group mean of converted values. This is close to 0 which indicates the distribution is a normal distribution. Our hypothesis statement stated that for the null hypothesis the difference of means between treatement and control successful convert rates was 0. \n",
    "\n",
    ">**Note:**We were able to simulate such null hypothesis by assuming that the success rate for treatment and control were the same\n",
    "\n",
    "On the other hand, the alternative hypothesis has a mean different than 0. This leads to further analysis and to take the p-value to understand if we have something statistically significant. In this case our p-value was 0.9054 and if the typical Error I threshold is 0.05 then we do not have enough information to reject our $H_0$ (null hypothesis)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l. We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/arturops/anaconda/envs/sdcar/lib/python3.5/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    }
   ],
   "source": [
    "# Import stats models \n",
    "import statsmodels.api as sm\n",
    "\n",
    "# Getting back the sample sizes for the data as well as the success rate mean\n",
    "convert_old = df2.query('landing_page == \"old_page\" and converted == 1').shape[0]\n",
    "convert_new = df2.query('landing_page == \"new_page\" and converted == 1').shape[0]\n",
    "n_old = sample_size_old_page\n",
    "n_new = sample_size_new_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "m. Now use `stats.proportions_ztest` to compute your test statistic and p-value.  [Here](http://knowledgetack.com/python/statsmodels/proportions_ztest/) is a helpful link on using the built in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17489 17264 145274 145310\n",
      "Zscore: -1.311\n",
      "p-val: 0.9051\n"
     ]
    }
   ],
   "source": [
    "#Displaying the data that we will feed to compute the z_score and p-value using statsmodels\n",
    "print(convert_old, convert_new, n_old, n_new)\n",
    "\n",
    "# Computing z_score and p-value\n",
    "z_score, pval = sm.stats.proportions_ztest([convert_new, convert_old],[n_new, n_old], alternative='larger')\n",
    "\n",
    "print('Zscore: {:.3f}\\np-val: {:.4f}'.format(z_score, pval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $Z_{score}$ in this case determines if our alternative hypothesis mean is away from the mean and in which direction. The $Z_{score}$ we computed is -1.311 which indicates the alternative hypothesis mean is 1.311 standard deviations awaya from the mean and the fact that has negative sign means it is to the left. This matches what we saw in our histogram (red line).\n",
    "\n",
    "The p-value agrees with the value calculated above. It basically is trying to help us determine that the old page has more succesful conversion rate. Based on this information we do not have enough data to reject $H_0$.\n",
    "However, we might want to check the time trend between the old page samples and new page as the long time users might be experiencing change aversion and the moderate new users could see benefit on the change. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "`1.` In this final part, you will see that the result you acheived in the previous A/B test can also be acheived by performing regression.<br><br>\n",
    "\n",
    "a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Logistic regression**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. The goal is to use **statsmodels** to fit the regression model you specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives.  However, you first need to create a column for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page  old_page  \n",
       "0          1        0         1  \n",
       "1          1        0         1  \n",
       "2          1        1         0  \n",
       "3          1        1         0  \n",
       "4          1        0         1  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a new copy of our data frame to do all the regression fitting and column manipulation if needed\n",
    "df3 = df2.copy()\n",
    "\n",
    "# Adding intercept and dummy variables for the page type\n",
    "df3['intercept'] = 1\n",
    "df3[['ab_page','old_page']] = pd.get_dummies(df3['landing_page'])\n",
    "df3.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use **statsmodels** to import your regression model.  Instantiate the model, and fit the model using the two columns you created in part **b.** to predict whether or not an individual converts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "# Logistic fitting\n",
    "logit = sm.Logit(df3['converted'],df3[['intercept','ab_page']]) \n",
    "r = logit.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sun, 29 Apr 2018</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>14:29:37</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Sun, 29 Apr 2018   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        14:29:37   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Displaying results of our model\n",
    "r.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br>  **Hint**: What are the null and alternative hypotheses associated with your regression model, and how do they compare to the null and alternative hypotheses in the **Part II**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The null hypothesis ($H_0$) and alternative hypothesis ($H_1$) are basically related to the **ab_page** column. If we look at the $Z_{score}$ we can check that it is the same as what we computed above. The coefficient in this case has to be converted to $1/e^{-0.015}$ as we are using logistic regression and the sign is negative. Then we get a value of 0.985 which tell us that the relation of success covertion rate in new pages decreases 0.985 compared to the old pages. This is implying the old pages have a higher success convertion rate which is exactly what we saw before.\n",
    "\n",
    "The p-value in the section above (Part II) was 0.9065 because it was a one tail hypothesis testing in which the alternative hypothesis is $H_1: p_new - p_old > 0$. However, in this section part III the p-value was 0.190 since the alternative hypothesis for a logistic regression is $H_1: p_new - p_old \\neq 0$, which is a two-tailed  hypothesis test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the analysis above we can tell that the hypothesis is related to the column **ab_page** and it will be a good idea to add another variable to understand better if the model has generalized good enough or if there are more pieces to it that can make it more accurate. It can be that depending on the country the new page had more acceptance. Thus, it is always good to add and check if a model improved with a new variable, that way one can also check for multicolinearity and remove the variables that are correlated to fine tune the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives. You will need to read in the **countries.csv** dataset and merge together your datasets on the approporiate rows.  [Here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) are the docs for joining tables. \n",
    "\n",
    "Does it appear that country had an impact on conversion?  Don't forget to create dummy variables for these country columns - **Hint: You will need two columns for the three dummy variables.** Provide the statistical output as well as a written response to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>834778</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>928468</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>822059</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>711597</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>710616</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909908</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-06 20:44:26.334764</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>811617</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-02 18:42:11.851370</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938122</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-10 09:32:08.222716</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887018</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-06 11:09:40.487196</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>820683</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-14 11:52:06.521342</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        country                   timestamp      group landing_page  \\\n",
       "user_id                                                               \n",
       "834778       UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "928468       US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "822059       UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "711597       UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "710616       UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "909908       UK  2017-01-06 20:44:26.334764  treatment     new_page   \n",
       "811617       US  2017-01-02 18:42:11.851370  treatment     new_page   \n",
       "938122       US  2017-01-10 09:32:08.222716  treatment     new_page   \n",
       "887018       US  2017-01-06 11:09:40.487196  treatment     new_page   \n",
       "820683       US  2017-01-14 11:52:06.521342  treatment     new_page   \n",
       "\n",
       "         converted  intercept  ab_page  old_page  \n",
       "user_id                                           \n",
       "834778           0          1        0         1  \n",
       "928468           0          1        1         0  \n",
       "822059           1          1        1         0  \n",
       "711597           0          1        0         1  \n",
       "710616           0          1        1         0  \n",
       "909908           0          1        1         0  \n",
       "811617           1          1        1         0  \n",
       "938122           1          1        1         0  \n",
       "887018           0          1        1         0  \n",
       "820683           0          1        1         0  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Importing a joining a new variable (country) to add to the fit\n",
    "countries_df = pd.read_csv('./countries.csv')\n",
    "df_new = countries_df.set_index('user_id').join(df3.set_index('user_id'), how='inner')\n",
    "df_new.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Investigating number of unique countries in the dataset\n",
    "df_new.country.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['UK', 'US', 'CA'], dtype=object)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Investigating the unique countries in the dataset\n",
    "df_new.country.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>old_page</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>834778</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>928468</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>822059</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>711597</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>710616</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        country                   timestamp      group landing_page  \\\n",
       "user_id                                                               \n",
       "834778       UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "928468       US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "822059       UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "711597       UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "710616       UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "\n",
       "         converted  intercept  ab_page  old_page  CA  UK  US  \n",
       "user_id                                                       \n",
       "834778           0          1        0         1   0   1   0  \n",
       "928468           0          1        1         0   0   0   1  \n",
       "822059           1          1        1         0   0   1   0  \n",
       "711597           0          1        0         1   0   1   0  \n",
       "710616           0          1        1         0   0   1   0  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Create the necessary dummy variables\n",
    "country_dummies = pd.get_dummies(df_new['country'])\n",
    "df_new = df_new.join(country_dummies)\n",
    "df_new.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.  \n",
    "\n",
    "Provide the summary results, and your conclusions based on the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366116\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290581</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     2</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sun, 29 Apr 2018</td> <th>  Pseudo R-squ.:     </th>  <td>1.521e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>14:29:38</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1984</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9868</td> <td>    0.011</td> <td> -174.174</td> <td> 0.000</td> <td>   -2.009</td> <td>   -1.964</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0507</td> <td>    0.028</td> <td>   -1.786</td> <td> 0.074</td> <td>   -0.106</td> <td>    0.005</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>   -0.0099</td> <td>    0.013</td> <td>   -0.746</td> <td> 0.456</td> <td>   -0.036</td> <td>    0.016</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290581\n",
       "Method:                           MLE   Df Model:                            2\n",
       "Date:                Sun, 29 Apr 2018   Pseudo R-squ.:               1.521e-05\n",
       "Time:                        14:29:38   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1984\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9868      0.011   -174.174      0.000      -2.009      -1.964\n",
       "CA            -0.0507      0.028     -1.786      0.074      -0.106       0.005\n",
       "US            -0.0099      0.013     -0.746      0.456      -0.036       0.016\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Fit Logistic Model using intercept and country(UK as baseline) And Obtain the Results\n",
    "df_new['intercept'] = 1\n",
    "logit2 = sm.Logit(df_new['converted'],df_new[['intercept','CA','US']])\n",
    "r2 = logit2.fit()\n",
    "r2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sun, 29 Apr 2018</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>14:29:39</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9794</td> <td>    0.013</td> <td> -155.415</td> <td> 0.000</td> <td>   -2.004</td> <td>   -1.954</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0506</td> <td>    0.028</td> <td>   -1.784</td> <td> 0.074</td> <td>   -0.106</td> <td>    0.005</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>   -0.0099</td> <td>    0.013</td> <td>   -0.743</td> <td> 0.457</td> <td>   -0.036</td> <td>    0.016</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Sun, 29 Apr 2018   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        14:29:39   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9794      0.013   -155.415      0.000      -2.004      -1.954\n",
       "ab_page       -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "CA            -0.0506      0.028     -1.784      0.074      -0.106       0.005\n",
       "US            -0.0099      0.013     -0.743      0.457      -0.036       0.016\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Fitting using intercept, country(UK as baseline), and ab_page (old_page as baseline)\n",
    "logit3 = sm.Logit(df_new['converted'], df_new[['intercept','ab_page','CA','US']])\n",
    "r3 = logit3.fit()\n",
    "r3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After adding the new variable and comparing a model only with it we noticed that they added variables have a $pvalue < 0.5$ which means they have some significance added to our model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "## Conclusions\n",
    "\n",
    "After all the A/B Test we performed, we learned that we do **not have enough information to reject $H_0$** and the **model to predict the convert individual is a logistic regression** as the converted value is binary (meaning only True =1 or False = 0).\n",
    "\n",
    "During the whole analysis we obtained the mean corresponding to the $H_1$ and $H_0$, we noticed that the $H_1$ mean was on the left side of the distribution while the simulated $H_0$ mean was very close to the center in 0. The difference in means gave us a $Z_{score} = -1.311$ for the $H_1$ mean, meaning that it was on the left side of the distribution and not centered at all. We plotted a histogram with the means to verify that all of it was correct and it confirmed our assumption. Although it looked like the mean was further away in the histogram we know that we need to know the p-value to determine if that was true. \n",
    "When we computed p-value, this value turned out to be 0.9051 which is greater than the Error Type 1 threshold of 0.05. This indicates that there is some statistical significance the old page actually had a higher sucessful conversion of individuals. Nevertheless, we cannot confirm this as an absolute true, but only agree that there is no information to reject $H_0$. In fact, the data also showed the old page to have a higher proportion of conversion, but in reality boundary conditions the change is so small that even if the old page has more success on conversion it actually is for an approximately amount of 0.16% of the people, which to be honest shows no practical significance. As a result, we shouldn't stop using the old page, but probably that's what we want as implementing the new page will cost and take time while it might not apport anything. However, we should think that people might be experiencing an aversion to change and we probably wanted to run the test longer if needed. Another important question is if this change will be better for the future of the company then it might be wise to change to the new page as it doesn't have much difference with the old and it can bing greater benefit even though it seems riskier at first. In any case the external factors and not just the math should also be consider, but if we were to decide based on math an strongly believeing our study covers almost all we needed then we would stick to the old page.\n",
    "\n",
    "The logistic model we used also included the country. I believe it might be smart to decide per country if the new page performs better, but that will be future work.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}