DIEGO_GIMENEZ_A2_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apprentice chef \n",
    " <b>Assignment 1</b> <br>\n",
    "<i> Diego Gimenez </i>\n",
    "\n",
    "<br><br>\n",
    "\n",
    "<b>Introduction</b>\n",
    "\n",
    "The Following analysis is focused on analyzing which factors are most important to successfully cross sell Half Way there to customers. This analysis includes a model to classify if a customer will successfully cross sell. \n",
    "<br> <br>\n",
    "\n",
    "<b> Classification Model Notes: </b>\n",
    "    <br><br>\n",
    "False negatives (Recall) is the main classification metric for this model's business value. The opportunity cost of not offering Halfway There to someone that is willing to pay is higher than offering it to someone that would not purchase it. In addition, AUC was used for Grid Search model quality metric.\n",
    "<br> <br>\n",
    "\n",
    "<b> Target Variable Notes: </b> <br><br>\n",
    "\n",
    "Halfway There, a cross-selling promotion where subscribers receive a half bottle of wine from a local California\n",
    "vineyard every Wednesday (halfway through the work week).\n",
    "\n",
    "<br> \n",
    "<b> Technical Notes </b>\n",
    "- Seed across the notebook: 222 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries and Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.191551Z",
     "start_time": "2020-03-16T01:50:50.961911Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd # Data Munging library\n",
    "import matplotlib.pyplot as plt # Basic graphs\n",
    "import seaborn as sns # Enhanced Graphs\n",
    "import re # Regex\n",
    "import numpy as np # Linear Algebra \n",
    "\n",
    "# Machine Learning Modules\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import recall_score, confusion_matrix,accuracy_score, auc, roc_auc_score\n",
    "\n",
    "# Read CSVs\n",
    "df = pd.read_excel('Apprentice_Chef_Dataset.xlsx')\n",
    "dataDic = pd.read_excel('Apprentice_Chef_Data_Dictionary.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions\n",
    "\n",
    "<br>\n",
    "\n",
    "The following helper functions have been developed and open-sourced due to the iterative nature of the Data Science Workflow. These helper functions are based on Chase Kusterer's Machine Learning course taught at Hult International Business School  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Missing Value Flagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.196201Z",
     "start_time": "2020-03-16T01:50:52.192938Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def missingValueFlagger(dataframe):\n",
    "    \"\"\" \n",
    "    Creates binary values of columns with missing values. \n",
    "    Such values are included in new column(s) with the format m_<column_name>.\n",
    "    These columns will form part of the dataframe.\n",
    "    -------------\n",
    "    Params\n",
    "        dataframe: dataframe to be used for the creation of flags. \n",
    "    \"\"\"\n",
    "    \n",
    "    for col in dataframe:\n",
    "\n",
    "        # creating columns with 1s if missing and 0 if not\n",
    "        if dataframe[col].isnull().astype(int).sum() > 0:\n",
    "            dataframe['m_'+col] = dataframe[col].isnull().astype(int)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### outlierVisualEDA "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.260424Z",
     "start_time": "2020-03-16T01:50:52.197818Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def outlierVisualEDA(dataframe,subfolderName):    \n",
    "    '''\n",
    "    This function creates distplots for all numerical variables. \n",
    "    The purpose of this helper function is to aid in visual EDA for outlier detection. \n",
    "    The analyst must choose by his/her intuition a cutoff point where the distribution looks normal. \n",
    "    ______________________________________________\n",
    "    Params\n",
    "        dataframe: Dataframe to plot numerical values\n",
    "        subfolderName: Name of a folder in the current working directory to save the histograms. \n",
    "            THE SUBFOLDER MUST EXIST BEFORE RUNNING THIS HELPER FUNCTION\n",
    "    \n",
    "    \n",
    "    '''\n",
    "    \n",
    "    plt.rcParams['figure.figsize']=(9,7)\n",
    "    sns.set()\n",
    "    \n",
    "    lst_ = dataframe.select_dtypes(exclude=['object']).columns\n",
    "\n",
    "    \n",
    "    for i in range(len(lst_)):\n",
    "        \n",
    "        sns.distplot(dataframe[lst_[i]],\n",
    "                 bins  = 'fd',\n",
    "                 color = 'g')\n",
    "        plt.xlabel(lst_[i])\n",
    "        plt.savefig(f'{subfolderName}/{lst_[i]}_Histogram.png')\n",
    "        plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### outlierFlagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.268114Z",
     "start_time": "2020-03-16T01:50:52.261977Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "\n",
    "def outlierFlagger(dataframe, outlierDictionary):\n",
    "    '''\n",
    "This function creates outlier flags based on Visual Outler EDA. Please use the function outlierVisualEDA to do identify outliers.  \n",
    "A flag consists of a binary vector column where 1 identifies an outlier and 0 does not identify an outlier. \n",
    "This function creates the outlier columns in the original dataframe. The column name format is the following: \"Out_<ORIGINALCOLUMNNAME>\" \n",
    "USER NEEDS TO CREATE THEIR OWN OUTLIER DICTIONARY.\n",
    "\n",
    "This function is based on Chase Kusterer's Machine Learning course at Hult International Business School. \n",
    "---------------------------\n",
    "Params: \n",
    "    dataframe: Dataframe to create flags on. \n",
    "    outlierDictionary: Dictionary containing column names as keys and numerical flag (outlier threshold) as values. The outlier dictionary keys\n",
    "    most have the following format <ORIGINAL_COLUMN_NAME>_HI or <ORIGINAL_COLUMN_NAME>_LO. \n",
    "____________________________\n",
    "Examples: \n",
    "# Dataframe \n",
    "exampleDf = pd.DataFrame({\n",
    "                    'column1':[1,2,1,2,15],\n",
    "                    'column2':[1,20,50,70,60] \n",
    "                    })\n",
    "# outlierDictionary\n",
    "outlierDic = {\n",
    "                'column1_HI': 10,\n",
    "                'column2_LO':2\n",
    "            }\n",
    "\n",
    "# Columns before outlier flagger\n",
    "print(exampleDf.columns.to_list())\n",
    "\n",
    "# Function call\n",
    "outlierFlagger(exampleDf,outlierDic)\n",
    "\n",
    "# Columns after outlier Flagger\n",
    "print(exampleDf.columns.to_list())\n",
    "\n",
    "# DataFrame with new columns\n",
    "exampleDf\n",
    "\n",
    "'''\n",
    "\n",
    "    for key, value in outlierDictionary.items():\n",
    "\n",
    "        # If it's a high outlier run the following\n",
    "        if '_HI' in key:\n",
    "            m = re.search(\n",
    "                '\\w+(?=_)', key\n",
    "            )  # Extracts the original value of the column to be changed. If 'column1_HI', m will be 'column1'\n",
    "            dataframe['Out_' + m.group()] = 0  # Creates flag column\n",
    "            condition_hi = dataframe.loc[0:, 'Out_' + m.group()][\n",
    "                dataframe[m.group()] >\n",
    "                value]  # Subsets the rows that are greater than the\n",
    "            # value provided in the trend change dictionary\n",
    "\n",
    "            dataframe['Out_' + m.group()].replace(\n",
    "                to_replace=condition_hi, value=1, inplace=True\n",
    "            )  # Replaces the indices of condition_hi with 1 and puts it in the dataframe\n",
    "\n",
    "        # If it's a low outlier run the following\n",
    "        elif '_LO' in key:\n",
    "            m = re.search(\n",
    "                '\\w+(?=_)', key\n",
    "            )  # Extracts the original value of the column to be changed. If 'column1_LO', m will be 'column1'\n",
    "            dataframe['Out_' + m.group()] = 0  # Creates flag column\n",
    "            condition_lo = dataframe.loc[0:, 'Out_' + m.group()][dataframe[\n",
    "                m.group()] < value]  # Subsets the rows that are lower than the\n",
    "            # value provided in the trend change dictionary\n",
    "\n",
    "            dataframe['Out_' + m.group()].replace(\n",
    "                to_replace=condition_lo, value=1, inplace=True\n",
    "            )  # Replaces the indices of condition_hi with 1 and puts it in the dataframe\n",
    "        else:\n",
    "            print(\n",
    "                f'The following key from your outlierDictionary has not been properly named:{key} \\n Remember that the format is <COLUMNNAME>_HI or <COLUMNNAME>_LO'\n",
    "            )\n",
    "            print(\"Call Gandalf, there's an error\")\n",
    "\n",
    "# # NOT RUN\n",
    "# # Testing if the flags worked\n",
    "# dfTest = df.copy()\n",
    "# outlierFlagger(dfTest, outDic)\n",
    "# dfTest[dfTest.loc[:,'TOTAL_PHOTOS_VIEWED_LO'] > 0][['TOTAL_PHOTOS_VIEWED','TOTAL_PHOTOS_VIEWED_LO']]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### trendBasedVisualEDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.273559Z",
     "start_time": "2020-03-16T01:50:52.269407Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def trendBasedVisualEDA(dataframe,targetVariableStr,subfolderName):\n",
    "    '''\n",
    "    This function creates scatter for all numerical variables with the target variable. \n",
    "    The purpose of this helper function is to aid in visual EDA for bivariate outlier detection. \n",
    "    The analyst must choose by his/her intuition a cutoff point where the distribution starts to deviate. \n",
    "    ______________________________________________\n",
    "    Params\n",
    "        dataframe: Dataframe to plot numerical values\n",
    "        subfolderName: Name of a folder in the current working directory to save the histograms. \n",
    "            THE SUBFOLDER MUST EXIST BEFORE RUNNING THIS HELPER FUNCTION\n",
    "    \n",
    "    \n",
    "    '''\n",
    "    \n",
    "    plt.rcParams['figure.figsize']=(9,7)\n",
    "    sns.set()\n",
    "\n",
    "    lst_ = dataframe.select_dtypes(exclude=['object']).columns\n",
    "\n",
    "    \n",
    "    for i in range(len(lst_)):\n",
    "        sns.scatterplot(x = dataframe[lst_[i]],\n",
    "                        y = dataframe[targetVariableStr],\n",
    "                        color = 'b')\n",
    "        plt.xlabel(lst_[i])\n",
    "        plt.savefig(f'{subfolderName}/{lst_[i]}_Histogram.png')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### trendBaseFlagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.280861Z",
     "start_time": "2020-03-16T01:50:52.274783Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def trendBaseFlagger(dataframe,trendChangeDictionary):\n",
    "    import re\n",
    "    \n",
    "    \"\"\"Summary line.\n",
    "    \n",
    "    This function creates trend-based flags based on Visual Trend-Based EDA. \n",
    "    Please use the function trendBasedVisualEDA to identify Trend-Based outliers that distorts the distribution of the variable.  \n",
    "    A flag consists of a binary vector column where 1 identifies an outlier and 0 does not identify an outlier. \n",
    "    This function creates the trend change columns in the original dataframe. The column name format is the following: \"Change_<ORIGINALCOLUMNNAME>\" \n",
    "    USER NEEDS TO CREATE THEIR OWN TRENDCHANGE DICTIONARY.\n",
    "\n",
    "    This function is based on Chase Kusterer's Machine Learning course at Hult International Business School. \n",
    "    ---------------------------\n",
    "    Params: \n",
    "        dataframe: Dataframe to create flags on. \n",
    "        outlierDictionary: Dictionary containing column names as keys and numerical flag (outlier threshold) as values. The outlier dictionary keys\n",
    "        most have the following format <ORIGINAL_COLUMN_NAME>_HI or <ORIGINAL_COLUMN_NAME>_LO. \n",
    "    ____________________________\n",
    "    Example: \n",
    "    # Dataframe \n",
    "    exampleDf = pd.DataFrame({\n",
    "                        'column1':[1,2,1,2,15],\n",
    "                        'column2':[1,20,50,70,60] \n",
    "                        })\n",
    "    # outlierDictionary\n",
    "    outlierDic = {\n",
    "                    'column1_HI': 10,\n",
    "                    'column2_LO':2\n",
    "                }\n",
    "\n",
    "    # Columns before outlier flagger\n",
    "    print(exampleDf.columns.to_list())\n",
    "\n",
    "    # Function call\n",
    "    outlierFlagger(exampleDf,outlierDic)\n",
    "\n",
    "    # Columns after outlier Flagger\n",
    "    print(exampleDf.columns.to_list())\n",
    "\n",
    "    # DataFrame with new columns\n",
    "    exampleDf\n",
    "    .\"\"\"\n",
    "    \n",
    "    for key,value in trendChangeDictionary.items():\n",
    "        \n",
    "        # If it's a high outlier run the following\n",
    "        if '_HI' in key:\n",
    "            m = re.search('\\w+(?=_)',key) # Extracts the original value of the column to be changed. If 'column1_HI', m will be 'column1'\n",
    "            dataframe['Change_'+ m.group()] = 0 # Creates flag column \n",
    "            condition_hi = dataframe.loc[0:,'Change_'+ m.group()][dataframe[m.group()] > value] # Subsets the rows that are greater than the \n",
    "                                                                                                    # value provided in the trend change dictionary\n",
    "\n",
    "            dataframe['Change_'+m.group()].replace(to_replace = condition_hi, # Replaces the indices of condition_hi with 1 and puts it in the dataframe\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "        \n",
    "        \n",
    "        # If it's a low outlier run the following\n",
    "        elif '_LO' in key:   \n",
    "            \n",
    "            m = re.search('\\w+(?=_)',key)  # Extracts the original value of the column to be changed. If 'column1_LO', m will be 'column1'      \n",
    "            \n",
    "            dataframe['Change_'+ m.group()] = 0 #  Creates flag column\n",
    "            \n",
    "            condition_lo = dataframe.loc[0:,'Change_'+ m.group()][dataframe[m.group()] < value] # Subsets the rows that are lower than the \n",
    "                                                                                                    # value provided in the trend change dictionary\n",
    "                \n",
    "            dataframe['Change_'+ m.group()].replace(to_replace = condition_lo, # Replaces the indices of condition_hi with 1 and puts it in the dataframe\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "        else: \n",
    "             print(f'The following key from your trendChangeDictionary has not been properly named:{key} \\n Remember that the format is <COLUMNNAME>_HI or <COLUMNNAME>_LO')\n",
    "             print(\"Call GanOutdalf, there's an error\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration \n",
    "\n",
    "\n",
    "<br> \n",
    "<b>General Notes:</b>\n",
    "<br>\n",
    "\n",
    "This sections is focused on exploring the data at a high level. This includes: \n",
    "- Target's variable class imbalance. \n",
    "- Missing Value handling: Columns with NA's were flagged by creating a new column (m_Family_name). Original's column missing values were inputed by filling NAs as unknown. This is due that the only column with missing values is composed of nominal strings. \n",
    "- Data Measurement Classification: Features were divided in their measurement level (Continuous or Interval, Binary, Count, Categorical, Discrete or Nominal). I did this in order to segment the feature selection in section 4\n",
    "\n",
    "<br>\n",
    "\n",
    "_______________________\n",
    "\n",
    "<br> \n",
    "\n",
    "<b>Hypothesis Section</b>\n",
    "<br>\n",
    "\n",
    "- Recall is better as a metric because we want avoid false negatives. \n",
    "- Accuracy is not the proper metric due to class imbalance. \n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "______________\n",
    "\n",
    "\n",
    "Technical Notes:\n",
    "\n",
    "- The dataset only contains null values in family name (2% of total data) \n",
    "- 3 floats, 22 integers, 4 objects\n",
    "- FOLLOWED_RECOMMENDATIONS_PCT is a floats according to the data dictionary. In the dataset these columns are set as integers and all numbers are rounded to the nearest whole number. I assume that the data engineering team intended this. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Class Imbalance\n",
    "Counts: \n",
    "- 1321 class 1\n",
    "- 625 class 0\n",
    "\n",
    "Almost the double of class 1 than class 0. This unbalance is not huge, but the training and test set will be stratified by the target variable in order to be misrepresent the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.283737Z",
     "start_time": "2020-03-16T01:50:52.282009Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# 1321 class 1\n",
    "# 625 class 2\n",
    "\n",
    "# df.CROSS_SELL_SUCCESS.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.287366Z",
     "start_time": "2020-03-16T01:50:52.285560Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# The following code has been commented out to not have information overload\n",
    "\n",
    "\n",
    "# df.info() # To inspect datatypes\n",
    "# FOLLOWED_RECOMMENDATIONS_PCT should be float \n",
    "# df.isnull().mean().round(2).sort_values(ascending = False) # Percentage of null values per column \n",
    "# df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Creation of missing values flags \n",
    "<br> \n",
    "\n",
    "The only missing values in the dataset are in the FAMILY_NAME column. Thus, it's passed to the missing value flagger helper function. \n",
    "\n",
    "<br> \n",
    "\n",
    "<b> Note:</b> The missingValueFlagger helper function creates populates the column m_FAMILY_NAME. This is a binary column where if 1, there's a missing value in family name, if not 0 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.300626Z",
     "start_time": "2020-03-16T01:50:52.289261Z"
    },
    "code_folding": [],
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Missing Value Flagger uses in place in the function            \n",
    "missingValueFlagger(df)\n",
    "\n",
    "# Filling NA's\n",
    "df.loc[:,'FAMILY_NAME'].fillna('Unknown',inplace = True)\n",
    "\n",
    "\n",
    "# df.columns # Checking which columns where created.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Measurement Classification\n",
    "\n",
    "<br>\n",
    "\n",
    "I have divided categorized the dataset by data types in order to do feature specialized feature engineering by data type. \n",
    "\n",
    "<br> \n",
    "<br> \n",
    "\n",
    "<b>Heuristics of classification:</b> \n",
    "- Continuous or interval: Numerical values\n",
    "- Binary: Contains 1 and 0\n",
    "- Count: How many of something was measured.\n",
    "- Categorical: I assume multinomial independent categories.\n",
    "- Discrete / Nominal: Variables that are unique. \n",
    "\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "#### Hypotheses \n",
    "<br>\n",
    "\n",
    "- User that input mobile number might be more engaged? \n",
    "    - <b>THIS HYPOTHESIS WASN'T TRUE </b>\n",
    "\n",
    "- Users that input taste and preferences are more engaged?\n",
    "    - <b>THIS HYPOTHESIS WASN'T TRUE </b>\n",
    "    \n",
    "<br><br>\n",
    "________\n",
    "<br>\n",
    "<b>Data Measurement classification:</b>\n",
    "<br><br>\n",
    "\n",
    "- Continuous or Interval\n",
    "    - REVENUE\n",
    "    - AVG_TIME_PER_SITE_VISIT \n",
    "    - FOLLOWED_RECOMMENDATIONS_PCT\n",
    "    - AVG_PREP_VID_TIME\n",
    "    - AVG_CLICKS_PER_VISIT\n",
    "\n",
    "\n",
    "- Binary\n",
    "    - MOBILE_NUMBER\n",
    "    - TASTES_AND_PREFERENCES\n",
    "    - PACKAGE_LOCKER\n",
    "    - REFRIGERATED_LOCKER\n",
    "\n",
    "\n",
    "- Count\n",
    "    - TOTAL_MEALS_ORDERED\n",
    "    - UNIQUE_MEALS_PURCH\n",
    "    - CONTACTS_W_CUSTOMER_SERVICE\n",
    "    - PRODUCT_CATEGORIES_VIEWED\n",
    "    - CANCELLATIONS_BEFORE_NOON\n",
    "    - CANCELLATIONS_AFTER_NOON\n",
    "    - PC_LOGINS\n",
    "    - MOBILE_LOGINS\n",
    "    - WEEKLY_PLAN\n",
    "    - EARLY_DELIVERIES\n",
    "    - LATE_DELIVERIES\n",
    "    - LARGEST_ORDER_SIZE\n",
    "    - MASTER_CLASSES_ATTENDED\n",
    "    - TOTAL_PHOTOS_VIEWED\n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "- Categorical \n",
    "    - MEDIAN_MEAL_RATING\n",
    "\n",
    "\n",
    "\n",
    "- Discrete or Nominal\n",
    "    - Name \n",
    "    - Email\n",
    "    - First Name\n",
    "    - Last Name\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "\n",
    "<br> \n",
    "<b>General Notes:</b>\n",
    "<br>\n",
    "\n",
    "This sections is focused on performing feature engineering. You will find the following subsections: \n",
    "- Domain Knowledge Feature engineering\n",
    "    - **Meal Kit Industry Secondary Research**: Domain knowledge acquired by researching on internet and buying a meal kit in order to be in customer's shoes. \n",
    "    - **Cross Sell Sales Technique Secondary Research**: Domain Knowledge acquired by researching on internet. \n",
    "- **Outlier Feature Engineering**: Univariate and Bivariate Outlier analysis in order to identify anomalies or trends. Such anomalies were flagged by creating a binary column for each flagged feature. \n",
    "- **Discrete / Nominal Variable Feature Engineering**: Email types were featured by one hot encoding. \n",
    "- **Count Variables Feature Engineering**: All count variables were used to generate new features. \n",
    "\n",
    "<br>\n",
    "\n",
    "_______________________\n",
    "\n",
    "<br> \n",
    "\n",
    "<b>Hypothesis Section</b>\n",
    "<br>\n",
    "\n",
    "- I thought that count-variable and Univariate Outlier Feature Engineering was going to be successful because I captured anomalies in the data. This proved wrong.\n",
    "- The most successful features were from Discrete/Nominal and Domain Knowledge sections. I believe that this is because cross sales depend a lot in sub-groups in the data. Capturing anomalies wouldn't do the trick because these were two sparse.\n",
    "- One Bivariate Outlier feature proved to be useful in the final model. This might indicate that behavior of users that cross sell cannot be modeled by creating univariate features. For further analysis would be necessary to create more complex features (bivariate or multivariate) in order to test this hypothesis. \n",
    "- Many users have less total_meal_orders than Weekly plans. This is weird because for each weekly plan, the user receives at least 3 meals per week. I preferred to not create assumptions and to schedule a meeting with the analytics team to drill down more on this subject.  \n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "______________\n",
    "\n",
    "\n",
    "Technical Notes:\n",
    "\n",
    "- Domain Knowledge Features were created by the following steps:\n",
    "    1.  Simulate behavior stated by stakeholder (business case) using the available features. \n",
    "    2.  Calculate the mean of the target variable to serve as a benchmark (this was interpreted as the proportion of ones over zeros)\n",
    "    3. Change the logic operators of these behaviors. \n",
    "    4. Calculate mean of CROSS_SELL_SUCCESS\n",
    "    5. If mean is greater than benchmark mean (Step 2), flag is created and marked as a high (COLUMN_NAME_hi).\n",
    "    6. If mean is less than benchmark mean (Step 2), flag is created and marked as a low (COLUMN_NAME_hi).\n",
    "    \n",
    "If mean was higher than benchmark mean I assumed that these users had a higher tendency to cross sell. The same logic is applied vice-versa. \n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Domain Knowledge Feature Engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Meal Kit Industry Secondary Research\n",
    "<br>\n",
    "\n",
    "Many of the count variables feature engineering has been based on heuristics extracted from secondary research. \n",
    "<br>\n",
    "\n",
    "\n",
    "- <b>What Does the Customer Really Value in a Meal Kit?</b>\n",
    "    - Nearly 6 out of 10 consumers reported they quit their meal kit subscription service because the value wasn’t worth the money.\n",
    "    - Moreover, consumers are more likely to stick with a meal kit if it aligns with their dietary goals.\n",
    "- <b>Meal Kit Companies scramble to dodge disaster</b>\n",
    "    - Gist of the article\n",
    "        - Meal kit companies face a logistics nightmare. Because it's perishable and varies a lot. \n",
    "        - “Fresh food by its nature is perishable inventory,” said Rett Wallace of Triton Research, a provider of data on technology companies. “It’s easy to see           how marketing and delivery costs would zero out any margin they make on the food itself.”\n",
    "- <b>Packaged Facts article </b> <br>\n",
    "    <i>Verbatim</i>: The same poll reported 46% of respondents saying time management was a deciding factor in purchasing meal kit delivery. 44% of respondents said they would reconsider purchasing meal kits purely because of convenience.\n",
    "    - Time delivery is very important! \n",
    "        - Hypothesis: Create features based on how late meal kit is arriving - LATE_DELIVERIES. This might capture variations in revenue. \n",
    "- <b> Hello Fresh Grabs bigger slice of shrinking meal <b> <br>\n",
    "    - <i> Verbatim: </i>  “There is less prep time with it. All you do is chop, and everything is prepackaged and well-labeled. Directions are super simple,” Mr. Yarbro said. He has discontinued Blue Apron and plans to stick to HelloFresh.\n",
    "        - Preparation time and how easy is to do it is very important. \n",
    "        - Hypothesis:\n",
    "            - Create feature using AVG_PREP_VID_TIME. Avg prep time can capture how complicated is to do a meal\n",
    "- <b> Blue Shield Sheds customers as meal kit competition mounts <b>\n",
    "    - <i> Verbatim: </i> Mr. Dickerson said customer retention has improved as Blue Apron has introduced meals that are easier to make, added more variety and beef to its offerings, and struck partnerships to offer recipes that match dining trends, such as the Whole30 elimination diet.\n",
    "    - This corroborates previous point. Easiness to cook is important. How we can measure this by the count variables that we have? \n",
    "        \n",
    "        \n",
    "    \n",
    "_________\n",
    "<br> \n",
    "- Sources\n",
    "    - https://www.ehy.com/food-agency/insights/what-does-the-customer-really-value-in-a-meal-kit/ \n",
    "    - https://www.wsj.com/articles/meal-kit-companies-scramble-to-dodge-disaster-1532700025\n",
    "    - https://www.packagedfacts.com/Content/Featured-Markets/Meal-Kit-Delivery-Services\n",
    "    - https://www.wsj.com/articles/hellofresh-grabs-bigger-slice-of-shrinking-meal-kit-market-11577279059\n",
    "    - https://www.wsj.com/articles/blue-apron-sheds-customers-as-meal-kit-competition-mounts-1518528989"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T04:06:26.049270Z",
     "start_time": "2020-03-16T04:06:26.046343Z"
    }
   },
   "outputs": [],
   "source": [
    "# df.WEEKLY_PLAN.describe() # Count of weeks that a customer subscribed \n",
    "\n",
    "# It seems that users are not using the weekly meal plans! \n",
    "# df[df.WEEKLY_PLAN == 52]\n",
    "# df.TOTAL_MEALS_ORDERED.describe()\n",
    "\n",
    "# It looks that there's no effective cross sell success for people that order a lot of meals\n",
    "# df.loc[df.TOTAL_MEALS_ORDERED > 95, ['WEEKLY_PLAN',\n",
    "#                                      'NAME',\n",
    "#                                      'TOTAL_MEALS_ORDERED',\n",
    "#                                     'CROSS_SELL_SUCCESS']].sort_values('WEEKLY_PLAN',ascending = False).groupby('CROSS_SELL_SUCCESS').mean()\n",
    "\n",
    "# It looks that the Total Meals people that has ordered less than 60 meals are slightly more prone to do succesfull cross sell  \n",
    "# df.loc[df.TOTAL_MEALS_ORDERED < 60, ['WEEKLY_PLAN',\n",
    "#                                      'NAME',\n",
    "#                                      'TOTAL_MEALS_ORDERED',\n",
    "#                                     'CROSS_SELL_SUCCESS']].sort_values('WEEKLY_PLAN',ascending = False).groupby('CROSS_SELL_SUCCESS').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross Sell Sales Technique Domain knowledge research\n",
    "\n",
    "<b> Hubspot: </b>\n",
    "- Definition of Cross Selling\n",
    "    - Cross-selling is encouraging the purchase of anything in conjunction with the primary product. For example, if a customer has already purchased a subscription to your marketing tool, cross-selling would encourage that customer to purchase a subscription to your CRM.\n",
    "        - Hypothesis: Engagement can be leveraged to improve cross sell? What encourages customers to cross sell?\n",
    "- How to effectively cross sell according to hubspot\n",
    "    -  <u>Get to know your audience</u> : create personas for your customers and understand their goals and their challenges in order to identify the products you could cross-sell and upsell that are most useful to them.\n",
    "    - <u> Think about problems and offer solutions that map to the products </u>: Review your product offerings and try to align them with your customer journey. That way, you'll have a clear idea of common challenges your customers face, and exactly which of your products you can try to cross-sell or upsell as a possible solution.\n",
    "        - Hypotheses from section: Understanding customers' pain points and understanding their goal for the product usage are key to identify cross selling. \n",
    "        - Pain point Variables: \n",
    "            - Low Median Meal Ratings: Categorize this variable! \n",
    "            - High Late Deliveries: More late deliveries, more pain. Also, this is the biggest hurdle for meal kit companies\n",
    "            - Refrigerated locker: Users without this type of locker might have a bad experience. \n",
    "        - Cross Sell and customer Journey: \n",
    "            - Recreate customer journey\n",
    "                - Hypothesis: People that register with a landline number are older. Probably these people tend to buy less wine because they don't want to drink half a bottle of wine on a Wednesday . \n",
    "                \n",
    "<br> \n",
    "\n",
    "<b> EConsultancy <b> \n",
    "- Focus cross selling in complementary products: There is no point offering people something they are simply not going to be interested in. This is a waste of everyone’s time and a missed opportunity for a sale. You should always offer the customer something that compliments the product they are already interested in. \n",
    "- Ways to successfully cross-sell in e-commerce: \n",
    "    - Keep it relevant: You should always offer the customer something that compliments the product they are already interested in. The easiest and most obvious way to do this is to offer a product that directly relates to the one the customer is already looking at or has added to their basket. \n",
    "        - Hypothesis: Probably users in different parts of their journey have a higher tendency on cross selling. It would be interesting creating a longevity feature \n",
    "    - Offer a deal: You can go one step further than simply presenting additional products and offer a discount if people buy them as a ‘bundle’.\n",
    "\n",
    "_____________\n",
    "\n",
    "<br>\n",
    "\n",
    " References:\n",
    " - https://blog.hubspot.com/service/cross-selling\n",
    " -  https://econsultancy.com/cross-selling-online-why-it-s-important-how-to-do-it/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Customer Journey\n",
    "\n",
    "I'm going to recreate the customer journey with the available data in order to identify challenges that users might have. This is based on the hubspot article.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "###### New Users\n",
    "<br>\n",
    "New-user customer journey of Apprentice Chef:\n",
    "\n",
    "1. Go to the website or download the mobile app.\n",
    "2. Register using a phone number. Confirm the phone number with a one-time code sent\n",
    "via SMS message or voice call.\n",
    "3. (optional) Select tastes and preferences in terms of food and specify dietary\n",
    "restrictions/food allergies.\n",
    "4. (optional) May submit a picture of a government-issued ID card if also wanting to\n",
    "purchase alcohol.\n",
    "5. (optional) Register for a weekly meal plan and receive a discount on all purchases.\n",
    "6. Select meals and set meal delivery schedule.\n",
    "7. (optional) Fill in feedback survey on order.\n",
    "\n",
    "<b> Hypotheses </b>\n",
    "\n",
    " - Users that didn't say their preferences and Didn't ave a weekly plan\n",
    "     - These users convert 62% of times in avg \n",
    " \n",
    " - Users that say their preferences and haven't registered to weekly plans \n",
    "     - Tend to convert 74% of times in avg\n",
    " \n",
    "- Users that say their preferences and registered to 10 or more weekly plans \n",
    "    - Tend to convert 70% of times in avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T04:38:34.373369Z",
     "start_time": "2020-03-16T04:38:34.358067Z"
    },
    "code_folding": [],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "######## New users customer journey featuring ############\n",
    "\n",
    "############ Go to the website or download mobile app #################\n",
    "##df.MOBILE_LOGINS + df.PC_LOGINS\n",
    "df['uses_mobile_app'] = 0\n",
    "df.loc[df.MOBILE_LOGINS > 0, 'uses_mobile_app'] = 1\n",
    "\n",
    "######## SELECT TASTS AND PREFERENCES ################\n",
    "##df.TASTES_AND_PREFERENCES\n",
    "\n",
    "########## REGISTER FOR WEEKLY PLAN #####################\n",
    "\n",
    "## Users that didn't say their preferences and Didn't ave a weekly plan\n",
    "## These users convert 62% of times in avg \n",
    "# df[(df.MOBILE_NUMBER == 1) & df[(df.MOBILE_LOGINS == 1) | (df.TASTES_AND_PREFERENCES == 0) | (df.WEEKLY_PLAN == 0) ].describe().round(2)\n",
    "\n",
    "## Users that say their preferences and haven't registered to weekly plans \n",
    "## Tend to convert 74% of times in avg\n",
    "# df[((df.MOBILE_NUMBER == 1) & df.TASTES_AND_PREFERENCES == 1) & (df.WEEKLY_PLAN == 0) ].describe().round(2)\n",
    "\n",
    "## Users that say their preferences and registered to 10 or more weekly plans \n",
    "## Tend to convert 70% of times in avg\n",
    "# df[ (df.TASTES_AND_PREFERENCES == 1) & (df.WEEKLY_PLAN > 10) ].describe().round(2)\n",
    "\n",
    "########### More in depth Analysis of Weekly Plan and possible relationship with total meals ordered ###########3\n",
    "    ## I can't see anything valuable by combining weekly plan and Total Meals ordered!! \n",
    "# df.loc[(df.TOTAL_MEALS_ORDERED / df.WEEKLY_PLAN) > 3,  [ 'WEEKLY_PLAN','TOTAL_MEALS_ORDERED']].sort_values('WEEKLY_PLAN')\n",
    "# df.groupby('WEEKLY_PLAN')\n",
    "\n",
    "######### Valuable Feature #############333\n",
    "##  High New Engaged Customer\n",
    "df['high_new_customer_engagement'] = 0\n",
    "\n",
    "df.loc[(df.MOBILE_NUMBER == 1) & (df.TASTES_AND_PREFERENCES == 1) & (df.WEEKLY_PLAN == 0),'high_new_customer_engagement' ] = 1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Existing Users\n",
    "<br>\n",
    "Existing-user customer journey of Apprentice Chef:\n",
    "\n",
    "\n",
    "1. Log into the website or mobile app.\n",
    "2. (optional) Update profile and/or orders.\n",
    "3. Select meals and confirm meal delivery schedule.\n",
    "4. (optional) Fill in feedback survey on order.\n",
    "5. Food is delivered, cooked, and rated (created by me)\n",
    "\n",
    "\n",
    "Funnel was analyzed per section because the number of datapoints was highly reduced if the analysis was performed in a combined manner. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:52.388236Z",
     "start_time": "2020-03-16T01:50:52.338316Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "######## 1. Log into the website or mobile app ########### \n",
    "## df.MOBILE_LOGINS.describe()\n",
    "## df.PC_LOGINS.describe()\n",
    "df[(df.MOBILE_LOGINS > 1)  & (df.PC_LOGINS > 1 )] # least engaged users from first part of funnel\n",
    "# First part of funnel 65% of users convert \n",
    "df['existing_users_1_funnel_lo'] = 0\n",
    "df.loc[(df.MOBILE_LOGINS > 1)  & (df.PC_LOGINS > 1 ),'existing_users_1_funnel_lo'] = 1\n",
    "# df['existing_users_1_funnel_lo'].fillna(0, inplace = True)\n",
    "\n",
    "\n",
    "######### 2. (optional) Update profile and/or orders. #############\n",
    "df.AVG_CLICKS_PER_VISIT\n",
    "df.PRODUCT_CATEGORIES_VIEWED\n",
    "df[(df.AVG_CLICKS_PER_VISIT < 13) & (df.AVG_TIME_PER_SITE_VISIT > 94)]\n",
    "df['existing_users_2_funnel_hi'] = 0\n",
    "df.loc[(df.AVG_CLICKS_PER_VISIT < 13) & (df.AVG_TIME_PER_SITE_VISIT > 94),'existing_users_2_funnel_hi'] =1\n",
    "# df['existing_users_2_funnel_hi'].fillna(0, inplace = True)\n",
    "\n",
    "########3 3. Select meals and confirm meal delivery schedule. ########33\n",
    "# df.TOTAL_MEALS_ORDERED\n",
    "# df.AVG_TIME_PER_SITE_VISIT\n",
    "# df[(df.TOTAL_MEALS_ORDERED > 40) & (df.AVG_TIME_PER_SITE_VISIT > 94)]\n",
    "df['existing_users_3_funnel_hi'] = 0\n",
    "df.loc[(df.TOTAL_MEALS_ORDERED > 40) & (df.AVG_TIME_PER_SITE_VISIT > 94),'existing_users_3_funnel_hi'] = 1\n",
    "# df['existing_users_3_funnel_hi'].fillna(0, inplace = True)\n",
    "\n",
    "\n",
    "\n",
    "# Final part of Funnel \n",
    "## df.LATE_DELIVERIES\n",
    "## df.EARLY_DELIVERIES\n",
    "## df.FOLLOWED_RECOMMENDATIONS_PCT\n",
    "## df.MEDIAN_MEAL_RATING\n",
    "\n",
    "# Mean of cross sell 0.46, can be useful to extract low variance\n",
    "df['existing_users_final_funnel_lo'] = 0\n",
    "df.loc[(df.LATE_DELIVERIES > 40) |\\\n",
    "       (df.EARLY_DELIVERIES > 94) |\\\n",
    "       (df.FOLLOWED_RECOMMENDATIONS_PCT < 40)\\\n",
    "       , 'existing_users_final_funnel_lo'] = 1\n",
    "\n",
    "df['existing_users_final_funnel_hi'] = 0\n",
    "df.loc[(df.LATE_DELIVERIES > 40) |\\\n",
    "       (df.EARLY_DELIVERIES > 94) |\\\n",
    "       (df.FOLLOWED_RECOMMENDATIONS_PCT > 40),\\\n",
    "       'existing_users_final_funnel_hi'] = 1\n",
    "\n",
    "df['existing_users_final_funnel_hi_ALTERNATIVE'] = 0\n",
    "df.loc[(df.LATE_DELIVERIES > 40) |\\\n",
    "       (df.EARLY_DELIVERIES > 94) |\\\n",
    "       (df.FOLLOWED_RECOMMENDATIONS_PCT < 30), \\\n",
    "       'existing_users_final_funnel_hi_ALTERNATIVE'] = 1\n",
    "\n",
    "## df.loc[(df.TOTAL_MEALS_ORDERED < 90) & (df.AVG_TIME_PER_SITE_VISIT < 40),'CROSS_SELL_SUCCESS'].describe()\n",
    "\n",
    "df['tot_meal_per_week'] = (df.TOTAL_MEALS_ORDERED / df.loc[df.WEEKLY_PLAN > 0,'WEEKLY_PLAN'])\n",
    "df['tot_meal_per_week'].fillna(0, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Outlier Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "##### Univariate Outlier Feature Engineering \n",
    "This features didn't work. Hypothesis is explained section 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:53.117023Z",
     "start_time": "2020-03-16T01:50:52.389325Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# sns.catplot(x = 'CROSS_SELL_SUCCESS',\n",
    "#                             y = 'REFRIGERATED_LOCKER',\n",
    "#                             data = df,\n",
    "#            kind = 'violin')\n",
    "outlierFlagger(df,{\n",
    "                  'MEDIAN_MEAL_RATING_LO': 3, # It looks that it doesn't descriminate well the variance\n",
    "                  'LATE_DELIVERIES_HI': 1 # It looks that there is not much difference between users. \n",
    "                  \n",
    "                  })\n",
    "\n",
    "\n",
    "# sns.boxplot(x = df['CROSS_SELL_SUCCESS'],\n",
    "#                             y = df['REFRIGERATED_LOCKER'],\n",
    "#                             color = 'b')\n",
    "\n",
    "# sns.catplot(x = 'CROSS_SELL_SUCCESS',\n",
    "#                             y = 'REFRIGERATED_LOCKER',\n",
    "#                             data = df,\n",
    "#            kind = 'violin')\n",
    "\n",
    "# Double of the users that don't have a fridge decide to buy more wine!!\n",
    "# I don't know how to feature this out, but this can be a useful insight afterwards.\n",
    "# I need to dig in more into this\n",
    "# pd.pivot_table(df, values = 'CROSS_SELL_SUCCESS', index =  'REFRIGERATED_LOCKER', aggfunc= len ) # There are more values in the 0 \n",
    "\n",
    "\n",
    "# # Here we can see that from the ones that don't have refrigerated locker, converted more.\n",
    "# This also can be due to the fact that not that many people have refrigerated lockers\n",
    "# df.loc[df.REFRIGERATED_LOCKER == 0 ,'CROSS_SELL_SUCCESS'].hist()\n",
    "# df.groupby('REFRIGERATED_LOCKER').count()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "###### Outlier Analysis\n",
    "\n",
    "- TLDR; AVG_TIME_PER_SITE_VISIT, CANCELLATIONS_BEFORE_NOON, TOTAL_PHOTOS_VIEWED have a very high value on 100% quantile\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "- AVG_TIME_PER_SITE_VISIT: The 100% percentile is 15 times greater than the 0.99. This is weird\n",
    "-  CANCELLATIONS_BEFORE_NOON: The 100% percentile is 2 times greater than the 0.99 percentile. This is not very weird but worth inspecting. \n",
    "-  TOTAL_PHOTOS_VIEWED: The 100% percentile is ~ 2 times greater than the 0.99 percentile. This is not very weird but worth inspecting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:53.135575Z",
     "start_time": "2020-03-16T01:50:53.133320Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# We use the loc to look more in depth in a variable\n",
    "# df.iloc[:, 14:].quantile([0.20,\n",
    "#                          0.40,\n",
    "#                          0.60,\n",
    "#                          0.80,\n",
    "#                          1.00])\n",
    "\n",
    "# Extreme Variables Analysis based on quantiles\n",
    "\n",
    "# lst_ = ['AVG_TIME_PER_SITE_VISIT','CANCELLATIONS_BEFORE_NOON','TOTAL_PHOTOS_VIEWED']\n",
    "\n",
    "# for col in lst_:\n",
    "#     print(df.loc[:, col].quantile([ 0.90,\n",
    "#                                       0.95,\n",
    "#                                       0.99,\n",
    "#                                       1.00]),'\\n___________________')\n",
    "    \n",
    "# # Output \n",
    "# 0.90     146.3950\n",
    "# 0.95     179.6825\n",
    "# 0.99     248.5150\n",
    "# 1.00    1645.6000\n",
    "# Name: AVG_TIME_PER_SITE_VISIT, dtype: float64 \n",
    "# ___________________\n",
    "# 0.90     3.0\n",
    "# 0.95     4.0\n",
    "# 0.99     7.0\n",
    "# 1.00    13.0\n",
    "# Name: CANCELLATIONS_BEFORE_NOON, dtype: float64 \n",
    "# ___________________\n",
    "# 0.90     336.00\n",
    "# 0.95     471.50\n",
    "# 0.99     769.65\n",
    "# 1.00    1600.00\n",
    "# Name: TOTAL_PHOTOS_VIEWED, dtype: float64 \n",
    "# ___________________\n",
    "\n",
    "\n",
    "\n",
    "# I want to check the variables that cought my attention on the quantile analysis. \n",
    "# AVG_TIME_PER_SITE_VISIT: The 100% percentile is 15 times greater than the 0.99. This is weird\n",
    "# CANCELLATIONS_BEFORE_NOON: The 100% percentile is 2 times greater than the 0.99 percentile. This is not very weird but worth inspecting. \n",
    "# TOTAL_PHOTOS_VIEWED: The 100% percentile is ~ 2 times greater than the 0.99 percentile. This is not very weird but worth inspecting. \n",
    "\n",
    "# df[df['AVG_TIME_PER_SITE_VISIT'] > 1600].iloc[:,0:15]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "###### Outlier Analysis - Visual EDA\n",
    "\n",
    "- KDE plots where done for each column and outliers flags were created in new columns with the format Out_<COLUMN_NAME>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:53.139932Z",
     "start_time": "2020-03-16T01:50:53.137752Z"
    },
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# outlierVisualEDA(df,'hists') # Commented out to avoid content overload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "###### Outlier Flags\n",
    "\n",
    "   The following outlier flags were chosen based on the following heuristics: \n",
    " - Zero-inflated: If a variable have a spike on number of zeros, it was flagged. \n",
    " - Normal Distribution approximation: Outliers that deform the distribution of the feature. Outliers were flagged to make the feature have a normal distribution.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:55.029400Z",
     "start_time": "2020-03-16T01:50:53.141235Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Flags created after doing Visual Outlier Analysis EDA \n",
    "outDic = {\n",
    "    'REVENUE_HI':800,\n",
    "    'REVENUE_HI':2200,\n",
    "    'AVG_PREP_VID_TIME_HI': 230,\n",
    "    'AVG_PREP_VID_TIME_LO': 80,\n",
    "    'AVG_TIME_PER_SITE_VISIT_HI': 300,\n",
    "    'CANCELLATIONS_AFTER_NOON_HI': 1,\n",
    "    'CANCELLATIONS_BEFORE_NOON_HI': 5,\n",
    "    'CONTACTS_W_CUSTOMER_SERVICE_HI': 8,\n",
    "    'EARLY_DELIVERIES_LO': 0,\n",
    "    'EARLY_DELIVERIES_HI': 5,\n",
    "    'LARGEST_ORDER_SIZE_HI': 7,\n",
    "    'LATE_DELIVERIES_HI': 5,\n",
    "    'MASTER_CLASSES_ATTENDED_HI': 1,\n",
    "    'MEDIAN_MEAL_RATING_LO': 2,\n",
    "    'MEDIAN_MEAL_RATING_HI': 4,\n",
    "    'MOBILE_LOGINS_LO': 1,\n",
    "    'MOBILE_LOGINS_HI': 2.5,\n",
    "    'PC_LOGINS_LO': 5,\n",
    "    'PC_LOGINS_HI': 6,\n",
    "    'TOTAL_MEALS_ORDERED_HI': 200,\n",
    "    'TOTAL_MEALS_ORDERED_LO': 25,\n",
    "    'TOTAL_PHOTOS_VIEWED_HI': 500,\n",
    "    'TOTAL_PHOTOS_VIEWED_LO': 1,\n",
    "    'UNIQUE_MEALS_PURCH_HI': 9,\n",
    "    'WEEKLY_PLAN_HI': 12,\n",
    "    'WEEKLY_PLAN_LO': 1,\n",
    "    'AVG_CLICKS_PER_VISIT_LO': 8\n",
    "}\n",
    "\n",
    "\n",
    "outlierFlagger(df, outDic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Bivariate Feature Engineering\n",
    "\n",
    "The following analysis has been done in order to explore bivariate relationships between the features and the target variable.\n",
    "The purpose of this visual EDA is to detect outliers. I chose the cutoff points where the distribution starts to deviate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-04T19:35:00.934957Z",
     "start_time": "2020-03-04T19:35:00.919391Z"
    },
    "heading_collapsed": true
   },
   "source": [
    "###### Trend-Based Outlier Visual EDA\n",
    "\n",
    "This Section Contains the Boxplots of all features grouped by each class of the target variable. The thresholds were chosen from this analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:55.032291Z",
     "start_time": "2020-03-16T01:50:55.030546Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Commented out to not have content overload\n",
    "\n",
    "# trendBasedVisualEDA(df,'REVENUE','trendBased')\n",
    "\n",
    "# trendBasedVisualEDA(df, 'CROSS_SELL_SUCCESS','trendBased',binaryTarget=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "###### Trend-Based Flag Creation \n",
    "<br>\n",
    "Flags from visual trend-based visual EDA are  defined in the following dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:55.568110Z",
     "start_time": "2020-03-16T01:50:55.033741Z"
    },
    "code_folding": [],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "\n",
    "trendBaseDic = {\n",
    "\t'CANCELLATIONS_BEFORE_NOON_HI': 1,\n",
    "    'FOLLOWED_RECOMMENDATIONS_PCT_LO': 20\n",
    "    \n",
    "    \n",
    "}\n",
    "            \n",
    "trendBaseFlagger(df,trendBaseDic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Discrete / Nominal Variable Feature engineering \n",
    "\n",
    "In the following section I featured out the Family Name, First Name, and emails. \n",
    "\n",
    "These features were relevant in the final model. Explained in section 4.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.505758Z",
     "start_time": "2020-03-16T01:50:55.569319Z"
    },
    "code_folding": [],
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Users that have their family name equal to their first name have any impact on the target?\n",
    "# dfSameNameFamilyName = df[df['FIRST_NAME'] == df['FAMILY_NAME']] They don't\n",
    "\n",
    "# Divide the emails by domain knowledge provided by Customer Service Team\n",
    "professional = [\n",
    "    \"mmm.com\", \"amex.com\", \"apple.com\", \"boeing.com\", \"caterpillar.com\",\n",
    "    \"chevron.com\", \"cisco.com\", \"cocacola.com\", \"disney.com\", \"dupont.com\",\n",
    "    \"exxon.com\", \"ge.org\", \"goldmansacs.com\", \"homedepot.com\", \"ibm.com\",\n",
    "    \"intel.com\", \"jnj.com\", \"jpmorgan.com\", \"mcdonalds.com\", \"merck.com\",\n",
    "    \"microsoft.com\", \"nike.com\", \"pfizer.com\", \"pg.com\", \"travelers.com\",\n",
    "    \"unitedtech.com\", \"unitedhealth.com\", \"verizon.com\", \"visa.com\",\n",
    "    \"walmart.com\"\n",
    "]\n",
    "\n",
    "personal = [\"gmail.com\", \"yahoo.com\", \"protonmail.com\"]\n",
    "\n",
    "junk = [\n",
    "    \"me.com\", \"aol.com\", \"hotmail.com\", \"live.com\", \"msn.com\", \"passport.com\"\n",
    "]\n",
    "\n",
    "# Split by @ and select domains\n",
    "df['EMAIL_DOMAIN'] = df.EMAIL.str.split('@', expand=True)[1]\n",
    "\n",
    "# Create email_tue column\n",
    "df['EMAIL_TYPE'] = '0'\n",
    "\n",
    "# Populate the Email Type column by if else statements\n",
    "for index, row in df.iterrows():\n",
    "\n",
    "    if df.loc[index, 'EMAIL_DOMAIN'] in professional:\n",
    "        df.loc[index, 'EMAIL_TYPE'] = 'professional'\n",
    "    elif df.loc[index, 'EMAIL_DOMAIN'] in personal:\n",
    "        df.loc[index, 'EMAIL_TYPE'] = 'personal'\n",
    "    elif df.loc[index, 'EMAIL_DOMAIN'] in junk:\n",
    "        df.loc[index, 'EMAIL_TYPE'] = 'junk'\n",
    "    elif df.loc[index, 'EMAIL_DOMAIN'] not in professional or df.loc[\n",
    "            row, 'EMAIL'] not in personal or df.loc[row, 'EMAIL'] not in junk:\n",
    "        df.loc[index, 'EMAIL_TYPE'] = 'unknown'\n",
    "    else:\n",
    "        print('Call Gandalf, there has been a problem')\n",
    "\n",
    "email_dummies = pd.get_dummies(df.EMAIL_TYPE, drop_first=True)\n",
    "\n",
    "df = pd.concat([df, email_dummies], axis=1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Count Variable Feature Engineering\n",
    "\n",
    "The following section contains Count-Variable feature engineering. Many of these variables are important in the final model. \n",
    "\n",
    "These features were created based on domain knowledge from the section 4.1 and by trial and error. \n",
    "\n",
    "<br><br>\n",
    "<b> Count Variables </b> \n",
    "_____________\n",
    "\n",
    "- CANCELLATIONS_AFTER_NOON,\n",
    "- CANCELLATIONS_BEFORE_NOON,\n",
    "- CONTACTS_W_CUSTOMER_SERVICE,\n",
    "- EARLY_DELIVERIES,\n",
    "- LARGEST_ORDER_SIZE,\n",
    "- LATE_DELIVERIES,\n",
    "- MASTER_CLASSES_ATTENDED,\n",
    "- MOBILE_LOGINS,\n",
    "- PC_LOGINS,\n",
    "- PRODUCT_CATEGORIES_VIEWED,\n",
    "- TOTAL_MEALS_ORDERED,\n",
    "- TOTAL_PHOTOS_VIEWED,\n",
    "- UNIQUE_MEALS_PURCH,\n",
    "- WEEKLY_PLAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.524498Z",
     "start_time": "2020-03-16T01:50:56.506831Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df['late_over_total'] = df.LATE_DELIVERIES / df.TOTAL_MEALS_ORDERED # Late Deliveries per how many meals the user ordered in the year\n",
    "df['early_over_total'] = df.EARLY_DELIVERIES / df.TOTAL_MEALS_ORDERED #  Late Deliveries per how many meals the user ordered in the year\n",
    "df['product_cat_viewed_over_total_logins'] = df.PRODUCT_CATEGORIES_VIEWED / (df.PC_LOGINS + df.MOBILE_LOGINS) # Categories viewed per total number of logins \n",
    "df['late_deliveries_over_product_catViewed'] = df.LATE_DELIVERIES / df.PRODUCT_CATEGORIES_VIEWED # Late Deliveries per categories viewed\n",
    "df['total_logins_over_total_meals_ordered'] = (df.MOBILE_LOGINS + df.PC_LOGINS) / df.TOTAL_MEALS_ORDERED # Total Logins per total meals ordered\n",
    "\n",
    "\n",
    "df['total_unique_orders'] =  df.UNIQUE_MEALS_PURCH / df.TOTAL_MEALS_ORDERED # Total Unique orders\n",
    "df['total_orders_contact_custServ'] = df.CONTACTS_W_CUSTOMER_SERVICE /  df.TOTAL_MEALS_ORDERED # Total orders that contacted the customer service\n",
    "df['total_orders_without_cancellations'] = df.TOTAL_MEALS_ORDERED - df.CANCELLATIONS_BEFORE_NOON #Total orders that haven't had any cancellation\n",
    "df['late_deli_over_total_meals'] = df.LATE_DELIVERIES / df.TOTAL_MEALS_ORDERED # Late Deliveries over total meals ordered\n",
    "df['total_logins'] = df.PC_LOGINS + df.MOBILE_LOGINS # Total Logins\n",
    "\n",
    "# This Feature  is categorical \n",
    "# Median Meal Dummy Variable\n",
    "\n",
    "\n",
    "medianMealDummy = pd.get_dummies(df['MEDIAN_MEAL_RATING'],\n",
    "                                  drop_first=True)\n",
    "\n",
    "medianMealDummy.columns = ['median_meal_2','median_meal_3','median_meal_4','median_meal_5']\n",
    "\n",
    "df = df.drop('MEDIAN_MEAL_RATING', axis = 1)\n",
    "\n",
    "df = df.join(medianMealDummy)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelling \n",
    "\n",
    "<br> \n",
    "\n",
    "The following section contains: \n",
    "- Base Model: Logistic Regression\n",
    "- Model Tournament: Lasso, Ridge, LinearRegression, ElasticNet, ARDRegression, AdaBoostRegressor, BaggingRegressor, GradientBoostingRegressor,RandomForestRegressor\n",
    "- Winner: Logistic Regression with Hyper Parameter Tuning\n",
    "\n",
    "<br>\n",
    "\n",
    "_______\n",
    "\n",
    "<br>\n",
    "\n",
    "<b>Results:<b>\n",
    "\n",
    "~~~\n",
    "Train Metrics:\n",
    "            Recall:     0.851 \n",
    "            Accuracy:   0.777\n",
    "            AUC:        0.735 \n",
    "            \n",
    "____________________________________________________________________\n",
    "            \n",
    "    Test Metrics:\n",
    "            Recall:     0.819 \n",
    "            Accuracy:   0.749\n",
    "            AUC:        0.711 \n",
    "~~~\n",
    "\n",
    "<br>\n",
    "\n",
    "_______\n",
    "\n",
    "<br>\n",
    "\n",
    "<b>Notes:</b>\n",
    "- I used different preprocessing techniques (StandardScaler, MixManScaler) and Dimensionality Reduction (PCA). None of these proved to improve the final model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature sets\n",
    "<br> \n",
    "\n",
    "The Following feature sets were developed by different feature selection approaches:\n",
    "- Ensemble First Prune: Pruned full dataset by selecting features that had a relative importance higher than 0 (Gradient Boosting)\n",
    "- Logistic First Prube: Pruned full dataset by selecting features that had a coefficient lower than Zero (Logistic Regression)\n",
    "- First_log_reg_select_from_model_sklearn: Pruned full dataset by selecting features obtained by SelectFromModel method from SkLearn\n",
    "- Second_log_reg_select_from_model_sklearn: Pruned full First_log_reg_select_from_model_sklearn Features by selecting features obtained by SelectFromModel method from SkLearn\n",
    "- first_dec_tree_prune: Pruned full dataset by selecting features that had a relative importance higher than 0 (Decision Tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.580547Z",
     "start_time": "2020-03-16T01:50:56.527156Z"
    },
    "code_folding": [
     3,
     42
    ]
   },
   "outputs": [],
   "source": [
    "features = {\n",
    "\n",
    "    # Worse than full model on Logistic Regression\n",
    "'ensemble_first_prune': ['REVENUE',\n",
    " 'TOTAL_MEALS_ORDERED',\n",
    " 'UNIQUE_MEALS_PURCH',\n",
    " 'CONTACTS_W_CUSTOMER_SERVICE',\n",
    " 'AVG_TIME_PER_SITE_VISIT',\n",
    " 'MOBILE_NUMBER',\n",
    " 'CANCELLATIONS_BEFORE_NOON',\n",
    " 'CANCELLATIONS_AFTER_NOON',\n",
    " 'TASTES_AND_PREFERENCES',\n",
    " 'PC_LOGINS',\n",
    " 'MOBILE_LOGINS',\n",
    " 'WEEKLY_PLAN',\n",
    " 'LATE_DELIVERIES',\n",
    " 'PACKAGE_LOCKER',\n",
    " 'REFRIGERATED_LOCKER',\n",
    " 'FOLLOWED_RECOMMENDATIONS_PCT',\n",
    " 'AVG_PREP_VID_TIME',\n",
    " 'LARGEST_ORDER_SIZE',\n",
    " 'MASTER_CLASSES_ATTENDED',\n",
    " 'AVG_CLICKS_PER_VISIT',\n",
    " 'TOTAL_PHOTOS_VIEWED',\n",
    " 'high_new_customer_engagement',\n",
    " 'personal',\n",
    " 'professional',\n",
    " 'Out_CONTACTS_W_CUSTOMER_SERVICE',\n",
    " 'Out_WEEKLY_PLAN',\n",
    " 'late_over_total',\n",
    " 'early_over_total',\n",
    " 'product_cat_viewed_over_total_logins',\n",
    " 'late_deliveries_over_product_catViewed',\n",
    " 'total_logins_over_total_meals_ordered',\n",
    " 'total_unique_orders',\n",
    " 'total_orders_contact_custServ',\n",
    " 'total_orders_without_cancellations',\n",
    " 'late_deli_over_total_meals',\n",
    " 'median_meal_3',\n",
    " 'median_meal_4'],\n",
    "    \n",
    "    # Worse than full model on logistic regression\n",
    "'logistic_first_prune': ['REVENUE',\n",
    " 'TOTAL_MEALS_ORDERED',\n",
    " 'UNIQUE_MEALS_PURCH',\n",
    " 'CONTACTS_W_CUSTOMER_SERVICE',\n",
    " 'PRODUCT_CATEGORIES_VIEWED',\n",
    " 'AVG_TIME_PER_SITE_VISIT',\n",
    " 'CANCELLATIONS_AFTER_NOON',\n",
    " 'PC_LOGINS',\n",
    " 'WEEKLY_PLAN',\n",
    " 'EARLY_DELIVERIES',\n",
    " 'LATE_DELIVERIES',\n",
    " 'PACKAGE_LOCKER',\n",
    " 'REFRIGERATED_LOCKER',\n",
    " 'FOLLOWED_RECOMMENDATIONS_PCT',\n",
    " 'AVG_PREP_VID_TIME',\n",
    " 'LARGEST_ORDER_SIZE',\n",
    " 'MASTER_CLASSES_ATTENDED',\n",
    " 'AVG_CLICKS_PER_VISIT',\n",
    " 'TOTAL_PHOTOS_VIEWED',\n",
    " 'Out_AVG_PREP_VID_TIME',\n",
    " 'Out_AVG_TIME_PER_SITE_VISIT',\n",
    " 'Out_CANCELLATIONS_AFTER_NOON',\n",
    " 'Out_CANCELLATIONS_BEFORE_NOON',\n",
    " 'Out_CONTACTS_W_CUSTOMER_SERVICE',\n",
    " 'Out_EARLY_DELIVERIES',\n",
    " 'Out_LARGEST_ORDER_SIZE',\n",
    " 'Out_LATE_DELIVERIES',\n",
    " 'Out_MASTER_CLASSES_ATTENDED',\n",
    " 'Out_MEDIAN_MEAL_RATING',\n",
    " 'Out_MOBILE_LOGINS',\n",
    " 'Out_PC_LOGINS',\n",
    " 'Out_TOTAL_MEALS_ORDERED',\n",
    " 'Out_TOTAL_PHOTOS_VIEWED',\n",
    " 'Out_UNIQUE_MEALS_PURCH',\n",
    " 'Out_WEEKLY_PLAN',\n",
    " 'Out_AVG_CLICKS_PER_VISIT']\n",
    "    \n",
    "    # AUC:  Train  0.774 , Test 0.735 \n",
    ",'First_log_reg_select_from_model_sklearn':['MOBILE_NUMBER',\n",
    " 'CANCELLATIONS_AFTER_NOON',\n",
    " 'TASTES_AND_PREFERENCES',\n",
    " 'REFRIGERATED_LOCKER',\n",
    " 'MASTER_CLASSES_ATTENDED',\n",
    " 'high_new_customer_engagement',\n",
    " 'existing_users_3_funnel_hi',\n",
    " 'existing_users_final_funnel_lo',\n",
    " 'existing_users_final_funnel_hi',\n",
    " 'existing_users_final_funnel_hi_ALTERNATIVE',\n",
    " 'premium_weekly_plan',\n",
    " 'Out_TOTAL_PHOTOS_VIEWED',\n",
    " 'Change_CANCELLATIONS_BEFORE_NOON',\n",
    " 'Change_FOLLOWED_RECOMMENDATIONS_PCT',\n",
    " 'personal',\n",
    " 'professional',\n",
    " 'median_meal_3']\n",
    "    \n",
    "    #\n",
    "   ,'Second_log_reg_select_from_model_sklearn': ['MOBILE_NUMBER',\n",
    " 'existing_users_final_funnel_lo',\n",
    "#  'existing_users_final_funnel_hi',\n",
    " 'Change_CANCELLATIONS_BEFORE_NOON',\n",
    " 'personal',\n",
    " 'professional']\n",
    "    \n",
    "    \n",
    "    ,'first_dec_tree_prune':['MOBILE_NUMBER',\n",
    " 'CANCELLATIONS_BEFORE_NOON',\n",
    " 'FOLLOWED_RECOMMENDATIONS_PCT',\n",
    " 'AVG_PREP_VID_TIME',\n",
    " 'existing_users_final_funnel_lo',\n",
    " 'existing_users_final_funnel_hi_ALTERNATIVE',\n",
    " 'personal',\n",
    " 'professional',\n",
    " 'early_over_total',\n",
    " 'late_deliveries_over_product_catViewed',\n",
    " 'total_logins_over_total_meals_ordered',\n",
    " 'total_unique_orders',\n",
    " 'total_orders_without_cancellations',\n",
    " 'late_deli_over_total_meals']\n",
    "    \n",
    "}\n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prep For Models\n",
    "\n",
    "Train and test split is done in this section. \n",
    "- Test size: 0.25\n",
    "- Random Seed: 222\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.595229Z",
     "start_time": "2020-03-16T01:50:56.583100Z"
    },
    "code_folding": [],
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MOBILE_NUMBER',\n",
       " 'existing_users_final_funnel_lo',\n",
       " 'Change_CANCELLATIONS_BEFORE_NOON',\n",
       " 'personal',\n",
       " 'professional']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrFeat = df.loc[:,features['Second_log_reg_select_from_model_sklearn']] ### First_log_reg_select_from_model_sklearn\n",
    "\n",
    "\n",
    "## For full-variable model:\n",
    "# lrFeat = df.drop(['NAME',\n",
    "#  'EMAIL',\n",
    "#  'EMAIL_TYPE',\n",
    "#  'EMAIL_DOMAIN',\n",
    "#  'FIRST_NAME',\n",
    "#  'FAMILY_NAME',\n",
    "#  'CROSS_SELL_SUCCESS'\n",
    " \n",
    "# ], axis = 1)\n",
    "\n",
    "\n",
    "lrTarget = df.loc[:,'CROSS_SELL_SUCCESS']\n",
    "\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "st_scaler = StandardScaler()\n",
    "mn_max_scaler = MinMaxScaler()\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "\n",
    "# lrFeat_st_scaled = st_scaler.fit_transform(lrFeat)\n",
    "# lrFeat_mn_max_scaled = mn_max_scaler.fit_transform(lrFeat)\n",
    "# lrFeat_pca_scaled = pca.fit_transform(lrFeat)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(lrFeat,\n",
    "                                                    lrTarget,\n",
    "                                                    test_size = 0.25,\n",
    "                                                    random_state = 222,\n",
    "                                                   stratify = lrTarget)\n",
    "\n",
    "\n",
    "# # Features\n",
    "X_train.columns.to_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base Model - Logistic Regression\n",
    "~~~\n",
    "Train Metrics:\n",
    "            Recall:     0.851 \n",
    "            Accuracy:   0.777\n",
    "            AUC:        0.735 \n",
    "            \n",
    "____________________________________________________________________\n",
    "            \n",
    "    Test Metrics:\n",
    "            Recall:     0.819 \n",
    "            Accuracy:   0.749\n",
    "            AUC:        0.711 \n",
    "~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.620823Z",
     "start_time": "2020-03-16T01:50:56.596396Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[343 126]\n",
      " [165 825]]\n",
      " Train Metrics:\n",
      "            Recall:     0.833 \n",
      "            Accuracy:   0.801\n",
      "            AUC:        0.782 \n",
      "            \n",
      "____________________________________________________________________\n",
      "            \n",
      "    Test Metrics:\n",
      "            Recall:     0.801 \n",
      "            Accuracy:   0.782\n",
      "            AUC:        0.772 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/diego/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "/home/diego/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coeffs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [coeffs]\n",
       "Index: []"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model = LogisticRegression(C = 1,random_state=222).fit(X_train,y_train)\n",
    "lr_fit = lr_model.fit(X_train, y_train)\n",
    "\n",
    "y_pred_train = lr_fit.predict(X_train)\n",
    "y_pred_test = lr_fit.predict(X_test)\n",
    "\n",
    "\n",
    "print(confusion_matrix(y_train,y_pred_train))\n",
    "\n",
    "print(f\"\"\" Train Metrics:\n",
    "            Recall:     {round(recall_score(y_train,y_pred_train),3)} \n",
    "            Accuracy:   {round(accuracy_score(y_train, y_pred_train),3)}\n",
    "            AUC:        {round(roc_auc_score(y_train, y_pred_train),3)} \n",
    "            \n",
    "____________________________________________________________________\n",
    "            \n",
    "    Test Metrics:\n",
    "            Recall:     {round(recall_score(y_test,y_pred_test),3)} \n",
    "            Accuracy:   {round(accuracy_score(y_test, y_pred_test),3)}\n",
    "            AUC:        {round(roc_auc_score(y_test,y_pred_test),3)} \"\"\")\n",
    "\n",
    "lr_coeffs = pd.DataFrame(\n",
    "    \n",
    "    lr_fit.coef_.round(2)[0],\n",
    "    index = X_train.columns.to_list(),\n",
    "    columns = ['coeffs']\n",
    "\n",
    ")\n",
    "\n",
    "\n",
    "\n",
    "# Most \n",
    "# lr_coeffs[np.abs(lr_coeffs.coeffs) > 0 ].sort_values('coeffs') \n",
    "# Without pruning\n",
    "lr_coeffs.loc[np.abs(lr_coeffs.coeffs) < 0.05 ].sort_index()\n",
    "\n",
    "# lr_coeffs.loc[np.abs(lr_coeffs.coeffs) > 0.05 ].sort_index().index.to_list()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sklearn Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:50:56.633492Z",
     "start_time": "2020-03-16T01:50:56.622898Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/diego/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['existing_users_final_funnel_lo']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "lr_model = LogisticRegression(C = 1,random_state=222).fit(X_train,y_train)\n",
    "model = SelectFromModel(lr_model, prefit=True)\n",
    "X_new = model.transform(X_train)\n",
    "# LinearSVC(penalty=\"l1\")\n",
    "X_train.columns[model.get_support()].to_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Selection \n",
    "Models Tested: Ridge, LogisticRegression, SGDClassifier, KNeighborsClassifier, AdaBoostClassifier, BaggingClassifier, GradientBoostingClassifier, RandomForestClassifier\n",
    "\n",
    "Winner: Logistic Regression due to its interpretability and boosting/bagging methods' scores didn't differ that much. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T05:05:32.714952Z",
     "start_time": "2020-03-16T05:05:28.779126Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_name</th>\n",
       "      <th>train_results</th>\n",
       "      <th>test_results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RidgeClassifier</td>\n",
       "      <td>0.782258</td>\n",
       "      <td>0.718031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>0.782258</td>\n",
       "      <td>0.728805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SGDClassifier</td>\n",
       "      <td>0.726549</td>\n",
       "      <td>0.702355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>KNeighborsClassifier</td>\n",
       "      <td>0.746026</td>\n",
       "      <td>0.708179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AdaBoostClassifier</td>\n",
       "      <td>0.782258</td>\n",
       "      <td>0.725881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BaggingClassifier</td>\n",
       "      <td>0.779553</td>\n",
       "      <td>0.749978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>0.785749</td>\n",
       "      <td>0.710418</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.779627</td>\n",
       "      <td>0.719921</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   model_name  train_results  test_results\n",
       "0             RidgeClassifier       0.782258      0.718031\n",
       "1          LogisticRegression       0.782258      0.728805\n",
       "2               SGDClassifier       0.726549      0.702355\n",
       "3        KNeighborsClassifier       0.746026      0.708179\n",
       "4          AdaBoostClassifier       0.782258      0.725881\n",
       "5           BaggingClassifier       0.779553      0.749978\n",
       "6  GradientBoostingClassifier       0.785749      0.710418\n",
       "7      RandomForestClassifier       0.779627      0.719921"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import RidgeClassifier, LogisticRegression, SGDClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier, GradientBoostingClassifier,RandomForestClassifier\n",
    "from sklearn.model_selection import KFold, cross_val_score,GridSearchCV\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import recall_score, confusion_matrix,accuracy_score, auc, roc_auc_score\n",
    "\n",
    "\n",
    "seed = 222\n",
    "num_folds = 3\n",
    "scoring = make_scorer(roc_auc_score, needs_threshold = False)\n",
    "models = []\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "models.append(('RidgeClassifier', RidgeClassifier()))\n",
    "models.append(('LogisticRegression', LogisticRegression(solver = 'lbfgs', max_iter = 5000)))\n",
    "models.append(('SGDClassifier', SGDClassifier()))\n",
    "models.append(('KNeighborsClassifier',KNeighborsClassifier(n_neighbors = 5)))\n",
    "# models.append(('ARDRegression', ARDRegression()))\n",
    "# models.append(('AdaBoostRegressor', AdaBoostRegressor()))\n",
    "models.append(('AdaBoostClassifier', AdaBoostClassifier(random_state = 222)))\n",
    "models.append(('BaggingClassifier', BaggingClassifier(random_state = 222)))\n",
    "models.append(('GradientBoostingClassifier', GradientBoostingClassifier(random_state = 222)))\n",
    "models.append(('RandomForestClassifier', RandomForestClassifier(n_estimators = 100, \n",
    "                                                                random_state = 222)))\n",
    "results_train = []\n",
    "results_test = []\n",
    "names = []\n",
    "for name, model in models:\n",
    "    kfold = KFold(n_splits=num_folds, random_state=seed)\n",
    "    cv_results_train = cross_val_score(model, X_train, y_train, cv=kfold, n_jobs=-1, scoring=scoring)\n",
    "    cv_results_test = cross_val_score(model, X_test, y_test, cv=kfold, n_jobs=-1,  scoring=scoring)\n",
    "    results_train.append(cv_results_train.mean())\n",
    "    results_test.append(cv_results_test.mean())\n",
    "    names.append(name)\n",
    "#     msg = \"%s %f %f \" % (name, cv_results.mean(), cv_results.std())\n",
    "#     print(f'Training {name}')\n",
    "\n",
    "\n",
    "models_log = pd.DataFrame({'model_name':names,\n",
    "                             'train_results':results_train,\n",
    "                             'test_results':results_test})\n",
    "\n",
    "models_log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Winner Model: Logistic Regression\n",
    "\n",
    "The Gradient Boosting Parameters were found using GridsearchCV. The hyper-parameters are defined below. \n",
    "<br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GridSearch Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T01:51:03.558908Z",
     "start_time": "2020-03-16T01:51:00.697176Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      " Train Metrics:\n",
      "\n",
      "            CV AUC Score: 0.782\n",
      "            model Parameters: {'C': 0.21000000000000002, 'penalty': 'l1', 'solver': 'saga', 'warm_start': True},\n",
      "            best Model: LogisticRegression(C=0.21000000000000002, class_weight=None, dual=False,\n",
      "                   fit_intercept=True, intercept_scaling=1, l1_ratio=None,\n",
      "                   max_iter=100, multi_class='warn', n_jobs=None, penalty='l1',\n",
      "                   random_state=222, solver='saga', tol=0.0001, verbose=0,\n",
      "                   warm_start=True)\n",
      "            \n",
      "            \n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve, confusion_matrix\n",
    "from sklearn.metrics import make_scorer              # customizable scorer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "## Params Tested:\n",
    "params = {\n",
    "    \"penalty\": ['l1','l2'],\n",
    "    'solver':['saga'],\n",
    "    'C':np.arange(0.01,11,0.1),\n",
    "    'warm_start':[True,False]\n",
    "    }\n",
    "\n",
    "\n",
    "lr_tuned = LogisticRegression(random_state=222)\n",
    "\n",
    "\n",
    "lr_model = GridSearchCV(lr_tuned,\n",
    "             param_grid=params,\n",
    "             n_jobs = -1, \n",
    "             scoring= make_scorer(roc_auc_score,needs_threshold=False),\n",
    "             cv = 3)\n",
    "\n",
    "\n",
    "lr_model.fit(X_train,y_train)\n",
    "\n",
    "\n",
    "print(f\"\"\" \\n Train Metrics:\n",
    "\n",
    "            CV AUC Score: {round(lr_model.best_score_,3)}\n",
    "            model Parameters: {lr_model.best_params_},\n",
    "            best Model: {lr_model.best_estimator_}\n",
    "            \n",
    "            \"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regressions with final parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T05:56:28.149184Z",
     "start_time": "2020-03-16T05:56:28.117554Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[343 126]\n",
      " [165 825]]\n",
      " Train Metrics:\n",
      "            Recall:     0.833 \n",
      "            Accuracy:   0.801\n",
      "            AUC:        0.782 \n",
      "            \n",
      "____________________________________________________________________\n",
      "            \n",
      "    Test Metrics:\n",
      "            Recall:     0.801 \n",
      "            Accuracy:   0.782\n",
      "            AUC:        0.772 \n",
      "coeffs:\n",
      "                                   coeffs\n",
      "MOBILE_NUMBER                       0.54\n",
      "existing_users_final_funnel_lo     -4.87\n",
      "Change_CANCELLATIONS_BEFORE_NOON    0.68\n",
      "personal                            0.85\n",
      "professional                        1.46\n",
      "probs:\n",
      "                                     coeffs\n",
      "MOBILE_NUMBER                     0.631812\n",
      "existing_users_final_funnel_lo    0.007615\n",
      "Change_CANCELLATIONS_BEFORE_NOON  0.663739\n",
      "personal                          0.700567\n",
      "professional                      0.811533\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import recall_score, confusion_matrix,accuracy_score, auc, roc_auc_score\n",
    "\n",
    "\n",
    "\n",
    "lr_model = LogisticRegression(C=0.21, class_weight=None, dual=False,\n",
    "                   fit_intercept=True, intercept_scaling=1, l1_ratio=None,\n",
    "                   max_iter=100, multi_class='warn', n_jobs=None, penalty='l1',\n",
    "                   random_state=222, solver='saga', tol=0.0001, verbose=0,\n",
    "                   warm_start=True)\n",
    "\n",
    "lr_fit = lr_model.fit(X_train, y_train)\n",
    "\n",
    "y_pred_train = lr_fit.predict(X_train)\n",
    "y_pred_test = lr_fit.predict(X_test)\n",
    "\n",
    "\n",
    "print(confusion_matrix(y_train,y_pred_train))\n",
    "\n",
    "print(f\"\"\" Train Metrics:\n",
    "            Recall:     {round(recall_score(y_train,y_pred_train),3)} \n",
    "            Accuracy:   {round(accuracy_score(y_train, y_pred_train),3)}\n",
    "            AUC:        {round(roc_auc_score(y_train, y_pred_train),3)} \n",
    "            \n",
    "____________________________________________________________________\n",
    "            \n",
    "    Test Metrics:\n",
    "            Recall:     {round(recall_score(y_test,y_pred_test),3)} \n",
    "            Accuracy:   {round(accuracy_score(y_test, y_pred_test),3)}\n",
    "            AUC:        {round(roc_auc_score(y_test,y_pred_test),3)} \"\"\")\n",
    "\n",
    "lr_coeffs = pd.DataFrame(\n",
    "    \n",
    "    lr_fit.coef_.round(2)[0],\n",
    "    index = X_train.columns.to_list(),\n",
    "    columns = ['coeffs']\n",
    "\n",
    ")\n",
    "\n",
    "\n",
    "\n",
    "# Most \n",
    "# lr_coeffs[np.abs(lr_coeffs.coeffs) > 0 ].sort_values('coeffs') \n",
    "# Without pruning\n",
    "# lr_coeffs.loc[np.abs(lr_coeffs.coeffs) >= 0 ].sort_index()\n",
    "print('coeffs:\\n',lr_coeffs)\n",
    "\n",
    "odds = np.exp(lr_coeffs)\n",
    "# Probabilities of Coeffs (need to reverse existing users final because is negative)\n",
    "print('probs:\\n',(odds / (1 + odds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-16T06:19:41.924671Z",
     "start_time": "2020-03-16T06:19:41.716877Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRODUCT_CATEGORIES_VIEWED</th>\n",
       "      <th>AVG_TIME_PER_SITE_VISIT</th>\n",
       "      <th>MOBILE_NUMBER</th>\n",
       "      <th>CANCELLATIONS_BEFORE_NOON</th>\n",
       "      <th>CANCELLATIONS_AFTER_NOON</th>\n",
       "      <th>TASTES_AND_PREFERENCES</th>\n",
       "      <th>PC_LOGINS</th>\n",
       "      <th>MOBILE_LOGINS</th>\n",
       "      <th>WEEKLY_PLAN</th>\n",
       "      <th>EARLY_DELIVERIES</th>\n",
       "      <th>LATE_DELIVERIES</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "      <td>1946.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.38</td>\n",
       "      <td>99.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>1.40</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.71</td>\n",
       "      <td>5.52</td>\n",
       "      <td>1.48</td>\n",
       "      <td>11.33</td>\n",
       "      <td>1.49</td>\n",
       "      <td>2.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.04</td>\n",
       "      <td>62.34</td>\n",
       "      <td>0.33</td>\n",
       "      <td>1.55</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.58</td>\n",
       "      <td>0.53</td>\n",
       "      <td>13.57</td>\n",
       "      <td>2.32</td>\n",
       "      <td>2.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00</td>\n",
       "      <td>10.33</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3.00</td>\n",
       "      <td>72.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.00</td>\n",
       "      <td>94.16</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>6.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>7.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>8.00</td>\n",
       "      <td>117.29</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>6.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>13.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>4.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>10.00</td>\n",
       "      <td>1645.60</td>\n",
       "      <td>1.00</td>\n",
       "      <td>13.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>7.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>52.00</td>\n",
       "      <td>9.00</td>\n",
       "      <td>19.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRODUCT_CATEGORIES_VIEWED  AVG_TIME_PER_SITE_VISIT  MOBILE_NUMBER  \\\n",
       "count                    1946.00                  1946.00        1946.00   \n",
       "mean                        5.38                    99.60           0.88   \n",
       "std                         3.04                    62.34           0.33   \n",
       "min                         1.00                    10.33           0.00   \n",
       "25%                         3.00                    72.00           1.00   \n",
       "50%                         5.00                    94.16           1.00   \n",
       "75%                         8.00                   117.29           1.00   \n",
       "max                        10.00                  1645.60           1.00   \n",
       "\n",
       "       CANCELLATIONS_BEFORE_NOON  CANCELLATIONS_AFTER_NOON  \\\n",
       "count                    1946.00                   1946.00   \n",
       "mean                        1.40                      0.17   \n",
       "std                         1.55                      0.43   \n",
       "min                         0.00                      0.00   \n",
       "25%                         0.00                      0.00   \n",
       "50%                         1.00                      0.00   \n",
       "75%                         2.00                      0.00   \n",
       "max                        13.00                      3.00   \n",
       "\n",
       "       TASTES_AND_PREFERENCES  PC_LOGINS  MOBILE_LOGINS  WEEKLY_PLAN  \\\n",
       "count                 1946.00    1946.00        1946.00      1946.00   \n",
       "mean                     0.71       5.52           1.48        11.33   \n",
       "std                      0.45       0.58           0.53        13.57   \n",
       "min                      0.00       4.00           0.00         0.00   \n",
       "25%                      0.00       5.00           1.00         1.00   \n",
       "50%                      1.00       6.00           1.00         7.00   \n",
       "75%                      1.00       6.00           2.00        13.00   \n",
       "max                      1.00       7.00           3.00        52.00   \n",
       "\n",
       "       EARLY_DELIVERIES  LATE_DELIVERIES  \n",
       "count           1946.00          1946.00  \n",
       "mean               1.49             2.97  \n",
       "std                2.32             2.74  \n",
       "min                0.00             0.00  \n",
       "25%                0.00             1.00  \n",
       "50%                0.00             2.00  \n",
       "75%                3.00             4.00  \n",
       "max                9.00            19.00  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().iloc[:,5:16].round(2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "157px",
    "left": "1306px",
    "top": "77px",
    "width": "489px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "525px",
    "left": "1550px",
    "right": "20px",
    "top": "120px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}