1 - Imputation Crash Course and Autoencoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XGJmuVrOrepP"
   },
   "source": [
    "# $\\color{purple}{\\text{Missing Data in the Age of Machine Learning (Part 1)}}$\n",
    "\n",
    "## $\\color{purple}{\\text{Regression in Imputation}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Libraries for this lesson}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "from autoimpute.imputations import SingleImputer\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 450
    },
    "executionInfo": {
     "elapsed": 20431,
     "status": "ok",
     "timestamp": 1654721415225,
     "user": {
      "displayName": "Haw-minn Lu",
      "userId": "16109571175851064283"
     },
     "user_tz": 420
    },
    "id": "e7N2XN4NFG-d",
    "outputId": "6103427b-15a2-4d9f-9d57-579328cc0633"
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/full_set.csv')\n",
    "mar_df = pd.read_csv('data/mar_set.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\color{purple}{\\text{Quick Look at the Data Set}}$\n",
    "\n",
    " * Full Set - Synthetic Normally Distributed Data Set\n",
    " * MAR Set - Data in the `feature a` column clobbered using an MAR mechanism\n",
    " * Double MAR Set - Data in the `feature a` and `feature b` column clobbered using an MAR mechanism"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Assess the missingness}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mar_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Compare Statistics}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NGTqexiQMgmM"
   },
   "source": [
    "### $\\color{purple}{\\text{Conventional Imputation: Stochastic Linear Regression}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KSE9PBfSoF3c"
   },
   "outputs": [],
   "source": [
    "linear_regressor = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Perform the linear regresssion\n",
    "\n",
    "We base the prediction of `feature a` on the remaining features in `rest`. We only run the regression on data with full rows, `full_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rest = ['feature b', 'feature c', 'feature d', 'uncorrelated']\n",
    "full_data = mar_df.dropna()\n",
    "linear_regressor.fit(full_data[rest], full_data['feature a'])\n",
    "predicted = linear_regressor.predict(mar_df[rest])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A note about a code pattern\n",
    "\n",
    "I will be repeating the following code pattern or variation thereof. \n",
    "\n",
    "```.assign(**{'feature a': df['feature a'].where(~df['feature a'].isnull(), predicted)``` \n",
    "\n",
    "Depending on the use case, I'll either be filling in a value when the value is missing.\n",
    "\n",
    "This basically substitutes the predicted value only when values are missing.\n",
    "\n",
    "This is basically the same pattern as\n",
    "\n",
    "```df['feature a'] = df['feature a'].where(~df['feature a'].isnull(), predicted)```\n",
    "\n",
    "but allows for passing the dataframe or method chaining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputed = mar_df.assign(\n",
    "    **{\n",
    "        'feature a':\n",
    "        mar_df['feature a'].where(~mar_df['feature a'].isnull(), predicted)\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze the Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### $\\color{purple}{\\text{Adding Stochastic Element to Linear Regression}}$\n",
    "* Extends Linear Regression by adding noise modelling the residuals\n",
    "* Better simulates variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We rely on the linear regression prediction above. And calculate the statistics behind the residuals of the linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "residual = mar_df['feature a'] - predicted\n",
    "residual.mean()\n",
    "residual.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the prediction we model the residual noise as a normal distribution and adjust predictions accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "residual_noise = np.random.normal(residual.mean(), residual.std(), 20000)\n",
    "predicted += residual_noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputed = mar_df.assign(\n",
    "    **{\n",
    "        'feature a':\n",
    "        mar_df['feature a'].where(~mar_df['feature a'].isnull(), predicted)\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze the Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Built into}}$ `autoimpute`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer = SingleImputer('least squares')\n",
    "ls_imputations = imputer.fit_transform(mar_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autoimpute.imputations import SingleImputer\n",
    "\n",
    "imputer = SingleImputer('stochastic')\n",
    "st_imputations = imputer.fit_transform(mar_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Random Forest Regression}}$\n",
    "Let's use a Random Forest Regression instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_regressor = RandomForestRegressor()\n",
    "rest = ['feature b', 'feature c', 'feature d', 'uncorrelated']\n",
    "full_data = mar_df.dropna()\n",
    "rf_regressor.fit(full_data[rest], full_data['feature a'])\n",
    "predicted = rf_regressor.predict(mar_df[rest])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputed = mar_df.assign(\n",
    "    **{\n",
    "        'feature a':\n",
    "        mar_df['feature a'].where(~mar_df['feature a'].isnull(), predicted)\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D13nfWuLsk-u"
   },
   "source": [
    "\n",
    "## $\\color{purple}{\\text{Out on the Fringe: Let's try an Artificial Neural Network}}$\n",
    "\n",
    "Imputation of categorical variables employs classification in place of regression. Most common is multinomial logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Dense(4, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(15, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mse')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rest = ['feature b', 'feature c', 'feature d', 'uncorrelated']\n",
    "full_data = mar_df.dropna()\n",
    "history=model.fit(full_data[rest], full_data['feature a'], epochs=50, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": [
    "plt.plot(history.history['loss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = model.predict(mar_df[rest])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputed = mar_df.assign(\n",
    "    **{\n",
    "        'feature a':\n",
    "        mar_df['feature a'].where(~mar_df['feature a'].isnull(), predicted[0])\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WvQYUhOqtN0d"
   },
   "source": [
    "# $\\color{purple}{\\text{Missing Data in the Age of Machine Learning (Part 2)}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TopzKHmOtVtL"
   },
   "source": [
    "## $\\color{purple}{\\text{Neural Network Imputers}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Libraries for this lesson}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from autoimpute.imputations import SingleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Denoising Autoencoders}}$\n",
    "\n",
    "* The missing data (or deviation from an imputed value) is treated as noise.\n",
    "* Denoising autoencoders are neural networks trained on the same input and output.\n",
    "* Theory is that the output is trained so that the output is the input with noise removed.\n",
    "* To work properly, data should be normalized during the imputation.\n",
    "\n",
    "`scaler` uses `sklearn`'s `StandardScaler`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/full_set.csv')\n",
    "dmar_df = pd.read_csv('data/double_mar_set.csv')\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(dmar_df)\n",
    "sdmar_df = pd.DataFrame(scaler.transform(dmar_df), columns=dmar_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmar_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def restore_df(scaler, x):\n",
    "    \"\"\"\n",
    "    Inverse the scaler and created a dataframe\n",
    "    \"\"\"\n",
    "    return pd.DataFrame(scaler.inverse_transform(x), columns=dmar_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic autoencoder proposed by [Gondara and Wang](https://arxiv.org/abs/1705.02737)\n",
    "![](https://raw.githubusercontent.com/WestHealth/pydataglobal-2022/main/images/dae.svg)\n",
    "* Deep neural network with 5 hidden layers with a dropout layer\n",
    "* $\\Theta$ is a hyperparameter governing the expansion and contraction of the layer\n",
    "* $\\Theta=7$ is suggested by best practice.\n",
    "* In the first 3 hidden layers, each layer expands by $\\Theta$ and contracts by $\\Theta$ in the last 2 hidden layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1 Impute the data set using univariate imputation\n",
    "The recommendation is that mean or median imputation of numeric data and mode imputation of categorical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "univariate_imputed = SingleImputer('median').fit_transform(sdmar_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2 Split data into training and test sets\n",
    "This is only necessary if you are building a model that accepts future data (open configuration). If the data set is closed (i.e. you don't expect any new data) then you can set the test_size to 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = 7\n",
    "# Divide into training and test sets\n",
    "training, test = train_test_split(univariate_imputed, test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3 Build, Compile and Train a Deep Neural Network Model\n",
    "* theta and activation function are hyperparameters\n",
    "\n",
    "See `tensorflow` and `keras` documentation for further detail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(5 + theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 2 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 3 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 2 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mse')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(training, training, epochs=50, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can visualize the progress of the loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(history.history['loss'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4 Make Prediction based on initial imputation.\n",
    "We replace the missing values with the predicted value. We also convert back to `pandas` `DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = pd.DataFrame(model.predict(univariate_imputed),\n",
    "                         columns=dmar_df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Don't forget to rescale the data after filling in missing data\n",
    "imputed = restore_df(scaler, sdmar_df.combine_first(predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "source": [
    "### $\\color{purple}{\\text{Analyze the Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### $\\color{purple}{\\text{Improved Feedback Denoising Autoencoders}}$\n",
    "\n",
    "My own enhancement to the denoising autoencoder see [here](https://arxiv.org/abs/2002.08338)\n",
    "\n",
    "The algorithm was designed for closed data sets. This example shows one enhancement to the denoising autoencoder (DAE), the iterative refinement of the imputed values. It starts similarly by univariate imputation as **step 1**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "univariate_imputation = SingleImputer('median').fit_transform(sdmar_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2 Build and Compile Deep Neural Network Model\n",
    "We use the same architecture as the DAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(5 + theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 2 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 3 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + 2 * theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5 + theta, activation=tf.nn.tanh),\n",
    "    tf.keras.layers.Dense(5)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(optimizer='adam', loss='mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3 Initial Fit\n",
    "Fewer epochs than standard DAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = [\n",
    "    model.fit(univariate_imputation,\n",
    "              univariate_imputation,\n",
    "              epochs=10,\n",
    "              verbose=False)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4 Iteration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = pd.DataFrame(model.predict(univariate_imputation),\n",
    "                         columns=dmar_df.columns)\n",
    "iterated_imputation = sdmar_df.combine_first(predicted)\n",
    "history.append(\n",
    "    model.fit(iterated_imputation,\n",
    "              iterated_imputation,\n",
    "              epochs=2,\n",
    "              verbose=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Repeat the Iteration a Prescribed Number of Times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(0, 19):\n",
    "    predicted = pd.DataFrame(model.predict(iterated_imputation),\n",
    "                             columns=dmar_df.columns)\n",
    "    iterated_imputation = sdmar_df.combine_first(predicted)\n",
    "    history.append(\n",
    "        model.fit(iterated_imputation,\n",
    "                  iterated_imputation,\n",
    "                  epochs=2,\n",
    "                  verbose=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we collected history in several batches, concatenate them so we can see a plot\n",
    "losses = sum([each.history['loss'] for each in history], [])\n",
    "plt.plot(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plug the final prediction into the missing values and rescale the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = pd.DataFrame(model.predict(iterated_imputation),\n",
    "                         columns=dmar_df.columns)\n",
    "imputed = restore_df(scaler, sdmar_df.combine_first(predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\color{purple}{\\text{Analyze the Results}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "master"
    ]
   },
   "source": [
    "#### $\\color{purple}{\\text{Another Improvement}}$\n",
    "\n",
    "* The loss function can be adjusted to eliminate the influence of missing/imputed values.\n",
    "* Most NN packages are not equipped to handle this, requires a complicated modification to the package. Beyond the scope of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyOzI6saudMRGj5Q9rKp9Mrq",
   "collapsed_sections": [],
   "name": "4 - Imputation Techniques.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}