2 - Imputation in machine learning pipeline and multiple imputation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TopzKHmOtVtL"
   },
   "source": [
    "### $\\color{purple}{\\text{Missing Data in the Age of Machine Learning and Artifical Neural Network (Part 3)}}$\n",
    "\n",
    "## $\\color{purple}{\\text{Imputation in Machine Learning and Multiple Imputation}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ILSdXRpPtiU8"
   },
   "source": [
    "## $\\color{purple}{\\text{How Imputation Fits Into Your Machine Learning Models}}$\n",
    "\n",
    "### Typical Machine Learning Workflow: \n",
    "  * Encoding\n",
    "  * Preprocessing\n",
    "  * Train\n",
    "  * Test\n",
    "  * Use\n",
    "  \n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/typical.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ILSdXRpPtiU8"
   },
   "source": [
    "### Machine Learning Workflow with Imputation: \n",
    "\n",
    "  * Encoding\n",
    "  * Preprocessing\n",
    "  * *Imputater*\n",
    "  * Train\n",
    "  * Test\n",
    "  * Use\n",
    "\n",
    "\n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/imputer.svg)\n",
    "\n",
    "\n",
    "* We demonstrate this using `sklearn`'s pipeline. But this is meant to describe abstractly what you should do\n",
    "\n",
    "Same data set is taken from the [Wine Quality Dataset at UCI](https://archive.ics.uci.edu/ml/datasets/Wine+Quality)\n",
    "\n",
    "This demonstrates a typical pipeline. The final column `quality` is the predicted value (ranging from 1 - 9). The `features` variable contains all the other column names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Libraries for this lesson}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from autoimpute.imputations import MiceImputer\n",
    "from autoimpute.imputations import SingleImputer\n",
    "from matplotlib.patches import Rectangle\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0AcjuKGBZBDN"
   },
   "outputs": [],
   "source": [
    "training = pd.read_csv('data/original_wine_training.csv')\n",
    "test = pd.read_csv('data/original_wine_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": [
    "training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list(training.columns[0:-1])\n",
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Typical ML Workflow\n",
    "\n",
    "We build the pipeline by one-hot encoding the `type` column which is categorical, then scale it, then apply random forest regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    ColumnTransformer([(\"type\", OneHotEncoder(), [\"type\"])],remainder='passthrough'), \n",
    "    StandardScaler(),\n",
    "    RandomForestRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.fit(training[features], training['quality'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.score(test[features], test['quality'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OZZt4mf7mpd6"
   },
   "source": [
    "### ML Workflow with Imputer\n",
    "\n",
    "Note: This is meant to demonstrate workflow and `autoimpute` is used as an example.\n",
    "\n",
    "One drawback is that `autoimpute` imputers require a `pandas` `DataFrame` as an input so custom transformers need to be used.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_hack = FunctionTransformer(\n",
    "    lambda x: pd.DataFrame(x, columns=['type_r', 'type_w'] + features[0:-1]))\n",
    "pandas_hack_full = FunctionTransformer(\n",
    "    lambda x: pd.DataFrame(x, columns=['type_r', 'type_w'] + features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can insert the imputer into the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    ColumnTransformer([(\"type\", OneHotEncoder(), ['type'])],remainder='passthrough'), \n",
    "    pandas_hack,\n",
    "    SingleImputer(strategy='least squares'), \n",
    "    StandardScaler(),\n",
    "    RandomForestRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_training = pd.read_csv('data/wine_training.csv')\n",
    "pipeline.fit(wine_training[features], wine_training['quality'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_test = pd.read_csv('data/wine_test.csv')\n",
    "pipeline.score(wine_test[features], wine_test['quality'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine_future = pd.read_csv('data/wine_future.csv')\n",
    "pipeline.predict(wine_future[features].iloc[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\color{purple}{\\text{How does Multiple Imputation fit in?}}$\n",
    "### $\\color{purple}{\\text{Approach 1: Augment Data with Multiple Copies}}$\n",
    "\n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/stack.svg)\n",
    "\n",
    "Augmentation teachs the model that the imputed values are \"fuzzy\" by providing different values.\n",
    "\n",
    "We create the same pipeline except we have a MiceImputer at the end.\n",
    "The resultant `dfs` are 5 copies of our dataframe with 5 separate imputations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "executionInfo": {
     "elapsed": 2640,
     "status": "ok",
     "timestamp": 1654712480068,
     "user": {
      "displayName": "Haw-minn Lu",
      "userId": "16109571175851064283"
     },
     "user_tz": 420
    },
    "id": "zwFrIcTcm87_",
    "outputId": "bf99ee06-2531-437c-988b-c1bd2d87652e"
   },
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    ColumnTransformer([(\"type\", OneHotEncoder(), ['type'])],remainder='passthrough'), \n",
    "    StandardScaler(), \n",
    "    pandas_hack,\n",
    "    MiceImputer(k=5, strategy='stochastic'))\n",
    "\n",
    "dfs = [each[1] for each in pipeline.fit_transform(wine_training[features])]\n",
    "print('Number of dataframe: ' + str(len(dfs)))\n",
    "\n",
    "dfs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HdHaAdHHm38b"
   },
   "source": [
    "We augment the training set by concatenating the 5 different data frame. Equivalently, we could rotate each epoch with different imputations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "augmented_training = pd.concat(dfs)\n",
    "\n",
    "quality = pd.concat([\n",
    "    wine_training.quality, wine_training.quality, wine_training.quality,\n",
    "    wine_training.quality, wine_training.quality\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "augmented_training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since test set is used to evaluate when the model may hit overtraining, it is not necessary to multiply impute the tests. But you may."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dfs = [each[1] for each in pipeline.transform(wine_test[features])]\n",
    "\n",
    "test1 = test_dfs[0]  # Variation one, just take one imputation\n",
    "quality1 = wine_test.quality\n",
    "\n",
    "test2 = pd.concat(test_dfs)  # Variation two augment in the same way\n",
    "quality2 = pd.concat([quality1, quality1, quality1, quality1, quality1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    RandomForestRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline.fit(augmented_training, quality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.score(test1, quality1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.score(test2, quality2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jsfJcQLfm9rh"
   },
   "source": [
    "Here we show an additional example of using neural networks to predict the imputed dataset. The network model was built as a classification problem. Bear in mind this is just for demonstration purposes, our model is not a particularly good estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Neural Network Approach\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e8ZUY3sEjJax"
   },
   "outputs": [],
   "source": [
    "model.fit(augmented_training, quality, epochs=10, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": [
    "model.evaluate(test1, quality1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": [
    "model.evaluate(test2, quality2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What about future values?\n",
    "Same options:\n",
    " * take one imputation\n",
    " * run all imputations through the model and use an ensemble technique to combine (e.g., majority voting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    ColumnTransformer([(\"type\", OneHotEncoder(), ['type'])],remainder='passthrough'), \n",
    "    StandardScaler(), \n",
    "    pandas_hack,\n",
    "    MiceImputer(k=5, strategy='stochastic'))\n",
    "\n",
    "future_dfs = [\n",
    "    each[1].iloc[[0]] for each in pipeline.fit_transform(wine_future[features])\n",
    "]\n",
    "\n",
    "future1 = future_dfs[0]  # Variation one, just take one imputation\n",
    "future2 = pd.concat(future_dfs)  # Variation two augment in the same way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.predict(future1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variation 1: Pick First Imputed\n",
    "# Argmax was used because the model predicted one-hot encoded label (i.e. 000000100 for 7)\n",
    "\n",
    "np.argmax(model.predict(future1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worthy to mention that here we simply take the sum to aggregate 5 imputation results. In practice, you can use other more robusted methods such as majority vote, averaging, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variation 2: Aggregate all Imputed\n",
    "np.argmax(model.predict(future2).sum(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Approach 2: Combine Multiple Models}}$\n",
    "\n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/ensemble.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [\n",
    "    tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Dropout(0.2),\n",
    "        tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "        tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "        tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "    ]) for _ in range(0, 5)\n",
    "]\n",
    "\n",
    "for model in models:\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each model is trained on a different imputation model. Each model should be tested under each imputation model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reminder: dfs stores a list of 5 imputed dataframes that was done indepdently\n",
    "\n",
    "for model, training in zip(models, dfs):\n",
    "    model.fit(training, wine_training.quality, epochs=10, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for model, test in zip(models, test_dfs):\n",
    "    print(model.evaluate(test, wine_test.quality, verbose=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np vstack turns the list of arrays into an array of arrays\n",
    "predictions = np.vstack(\n",
    "    [model.predict(future) for model, future in zip(models, future_dfs)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregate predictions\n",
    "np.argmax(predictions.sum(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Worth Mentioning: Use Single Imputation with Bagging}}$\n",
    "Short for bootstrap and aggregation\n",
    "\n",
    "Rather than multiple imputation, single imputations are performed from resampled datasets (bootstrapping)\n",
    "\n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/single.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = make_pipeline(\n",
    "    ColumnTransformer([(\"type\", OneHotEncoder(), ['type'])],remainder='passthrough'), \n",
    "    StandardScaler(),\n",
    "    pandas_hack_full, \n",
    "    MiceImputer(k=1, strategy='stochastic')\n",
    ")\n",
    "\n",
    "bagged_dfs = [\n",
    "    next(pipeline.fit_transform(wine_training.sample(frac=1, replace=True)))[1]\n",
    "    for _ in range(0, 5)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [\n",
    "    tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Dropout(0.2),\n",
    "        tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "        tf.keras.layers.Dense(20, activation=tf.nn.tanh),\n",
    "        tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "    ]) for _ in range(0, 5)\n",
    "]\n",
    "\n",
    "for model in models:\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for model, training in zip(models, dfs):\n",
    "    model.fit(training, wine_training.quality, epochs=10, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = np.vstack(\n",
    "    [model.predict(future) for model, future in zip(models, future_dfs)])\n",
    "# Aggregate predictions\n",
    "np.argmax(predictions.sum(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\color{purple}{\\text{Conclusion}}$\n",
    "\n",
    "* In a typical machine learning pipeline, imputation should be done before feeding into the machine learning model \n",
    "\n",
    "* Several strategies for multiple imputations:\n",
    "  * Augmenting dataset multiple times and predicted by the same imputer \n",
    "  * Perform imputation multiple times with independent imputer\n",
    "  * Bagging can be applied to single imputation performed multiply\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "colab": {
   "authorship_tag": "ABX9TyPpLv6R3L0XkmwitYmu11KS",
   "collapsed_sections": [],
   "name": "Untitled5.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}