02_identification_estimation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f51698b9-84bb-44bf-bfc5-b8d97b68080a",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "This solution accelerator notebook is also available at https://github.com/databricks-industry-solutions/causal-incentive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "89bb7ad9-8c8e-4101-baca-e38dc78bd1ad",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Cluster Configuration\n",
    "We recommend using a cluster with the following or similar specifications to run this solution accelerator:\n",
    "- Unity Catalog enabled cluster\n",
    "- Databricks Runtime 15.4 LTS ML or above\n",
    "- Single-node cluster: e.g. `m5d.2xlarge` on AWS or `Standard_D8ds_v5` on Azure Databricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "b1bd99ae-10cc-43ed-9f42-fd47f693767e",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Install Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "821a838b-1da6-4288-8ad0-76035e75b1b9",
     "showTitle": true,
     "tableResultSettingsMap": {},
     "title": "Install requirements"
    }
   },
   "outputs": [],
   "source": [
    "%pip install -r ./requirements.txt --quiet\n",
    "dbutils.library.restartPython()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "19b7bbc9-c08a-4a2b-87ac-5069175bfcb6",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%run ./util/notebook-config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "270d5fe1-fa4d-4774-b9d9-e003abae1ab1",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Influence Identification and Estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f0607752-b4d8-4ec8-93d8-ce0bd7a2ea59",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "We will use the causal graph obtained from the previous step to guide the identification step.  In this step the best method to isolate the effect of a given promotional offer on ```Revenue``` is identified.  The package DoWhy automates this step by relying on the well established [Do-Calculus](https://ftp.cs.ucla.edu/pub/stat_ser/r402.pdf) theoretical framework.\n",
    "\n",
    "First, let's load the graph from [MLflow](https://www.databricks.com/product/managed-mlflow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a5ba81e1-2a9d-4515-a41c-d0e147267a6d",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "graph = load_graph_from_latest_mlflow_run(experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4f64fdf2-ea9e-4f3e-8310-7653dee1d10a",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "We will now identify the total effect of ```Tech Support``` in ```Revenue``` using [DoWhy](https://github.com/py-why/dowhy) to obtain the <b>[Average Treatement Effect (ATE)](https://en.wikipedia.org/wiki/Average_treatment_effect)</b> estimand. In other words,  we will isolate the average influence ``Tech Support`` had over the ``Revenue`` of accounts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "06921b02-d3e2-455f-af01-a919dbaf18d6",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Instantiate a model object to estimate the tech support effect\n",
    "tech_support_effect_model = dowhy.CausalModel(\n",
    "    data=input_df, graph=graph, treatment=\"Tech Support\", outcome=\"Revenue\"\n",
    ")\n",
    "\n",
    "# Identify methods we can use to estimate the tech support effect (estimands)\n",
    "tech_support_total_effect_identified_estimand = (\n",
    "    tech_support_effect_model.identify_effect(\n",
    "        estimand_type=\"nonparametric-ate\",\n",
    "        method_name=\"maximal-adjustment\",\n",
    "    )\n",
    ")\n",
    "\n",
    "# Print out all identified estimands\n",
    "print(tech_support_total_effect_identified_estimand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "cdd8d10c-7102-42e0-9c7b-f6594eb52020",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "[DoWhy](https://github.com/py-why/dowhy) selects the [backdoor](http://causality.cs.ucla.edu/blog/index.php/category/back-door-criterion/) method as the best one to identify the effect.  It also determines which features should be used for the estimation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2afe3068-7875-4b22-a724-0e0bfc5a4eda",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Estimating \"Tech Support\" total effect on \"Revenue\"\n",
    "\n",
    "In order to obtain an unbias estimation we will use an approach call [Double Machine Learning (DML)](https://academic.oup.com/ectj/article/21/1/C1/5056401)  which is implemented in the [PyWhy](https://github.com/py-why) package [EconML](https://github.com/py-why/EconML). We use a [logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) model for estimating if the promotional offer was given and a [lasso regression](https://en.wikipedia.org/wiki/Lasso_(statistics) model for estimating the ``Revenue`` of the account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "26e4e3f5-8c38-45b7-9a7b-cee6970447d9",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Disable the mlflow autolog feature\n",
    "mlflow.autolog(disable=True)\n",
    "\n",
    "# Set up the treatment (t) and outcome (y) models for DML. See notebook-config for detail.\n",
    "model_t, model_y = setup_treatment_and_out_models()\n",
    "\n",
    "# Specify the effect modifiers, which are variables that can change the magnitude of the effect based on the groups.\n",
    "effect_modifiers = [\"Size\", \"Global Flag\"]\n",
    "\n",
    "# Specify the estimand recommended in the previous cell\n",
    "method_name = \"backdoor.econml.dml.LinearDML\"\n",
    "\n",
    "init_params = {\n",
    "    \"model_t\": model_t,\n",
    "    \"model_y\": model_y,\n",
    "    \"linear_first_stages\": True,\n",
    "    \"discrete_treatment\": True,\n",
    "    \"cv\": 3,\n",
    "    \"mc_iters\": 10,\n",
    "}\n",
    "\n",
    "# Estimate the effect of tech support\n",
    "tech_support_total_effect_estimate = tech_support_effect_model.estimate_effect(\n",
    "    tech_support_total_effect_identified_estimand,\n",
    "    effect_modifiers=effect_modifiers,\n",
    "    method_name=method_name,\n",
    "    method_params={\"init_params\": init_params},\n",
    ")\n",
    "\n",
    "# Extract the interpretation of the estimate\n",
    "tech_support_total_effect_estimate.interpret()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "86a7b8e8-b634-44d2-b17f-a58973ae3595",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Registering in MLflow the resulting model\n",
    "Please notice the ```autolog``` functionality of [MLflow](https://www.databricks.com/product/managed-mlflow) is disable in the above block of code.  This was done to have more control of what is logged into MLflow.  EconML uses sklearn models trained and evaluated usgin cross-validation.  If \"autolog\" is enable, all the trained models are logged in [MLflow](https://www.databricks.com/product/managed-mlflow) (including the once not ultimately selected by EconML),  this results in a lot of noise and slow performance.  Instead, we will control what is logged in [MLflow](https://www.databricks.com/product/managed-mlflow) by using the helper function ```register_dowhy_model```.  This function will register the EconML model together with the artefacts created by DoWhy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "20ca3b6d-731d-4840-8c29-9e5126e62db3",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "model_details = register_dowhy_model(\n",
    "    model_name=f\"{catalog}.{db}.tech_support_total_effect_dowhy_model\",\n",
    "    model=tech_support_effect_model,\n",
    "    estimand=tech_support_total_effect_identified_estimand,\n",
    "    estimate=tech_support_total_effect_estimate,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "9a9b6c67-7a78-45e3-999d-c56bf4de97b8",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Estimating \"Tech Support\" direct effect on \"Revenue\"\n",
    "\n",
    "In the graph obtained from the previous notebook we can appricate ```Tech Support``` has a direct effect on ```Revenue``` and a mediated effect through ```New Product Adoption```.  In other words,  ```Tech Support``` besides directly influencing ```Revenue```, also impacts ```New Product Adoption``` which itself has an effect on ```Revenue```.  The estimation done in the commands above covered the total influence on this promotional offer (direct and indirect).  We will now identify the direct influence only by using the [Control Direct Effect (CDE)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4193506/) estimand type of DoWhy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "440e4943-d277-420b-a0a7-93417879be0c",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "tech_support_direct_effect_identified_estimand = (\n",
    "    tech_support_effect_model.identify_effect(\n",
    "        estimand_type=\"nonparametric-cde\",\n",
    "        method_name=\"maximal-adjustment\",\n",
    "    )\n",
    ")\n",
    "print(tech_support_direct_effect_identified_estimand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "b464cd92-a05f-4e6e-a6e4-84944eba7b03",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "We will use again the [DML algorithm](https://academic.oup.com/ectj/article/21/1/C1/5056401) implemented in [EconML]() for this estimation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "923607d7-b98c-469e-836b-505502108f74",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "mlflow.autolog(disable=True)\n",
    "\n",
    "model_t, model_y = setup_treatment_and_out_models()\n",
    "\n",
    "effect_modifiers = [\"Size\", \"Global Flag\"]\n",
    "method_name = \"backdoor.econml.dml.LinearDML\"\n",
    "init_params = {\n",
    "    \"model_t\": model_t,\n",
    "    \"model_y\": model_y,\n",
    "    \"linear_first_stages\": True,\n",
    "    \"discrete_treatment\": True,\n",
    "    \"cv\": 3,\n",
    "    \"mc_iters\": 1,\n",
    "}\n",
    "\n",
    "tech_support_direct_effect_estimate = tech_support_effect_model.estimate_effect(\n",
    "    tech_support_direct_effect_identified_estimand,\n",
    "    effect_modifiers=effect_modifiers,\n",
    "    method_name=method_name,\n",
    "    method_params={\"init_params\": init_params},\n",
    ")\n",
    "\n",
    "tech_support_direct_effect_estimate.interpret()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "95152568-5e08-4a38-b878-0c29746fc06b",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "Let's now register the resulting model with all the associated DoWhy artifacts in [MLflow](https://www.databricks.com/product/managed-mlflow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e681358f-ce1b-4a57-a00d-51826f184195",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "model_details = register_dowhy_model(\n",
    "    model_name=f\"{catalog}.{db}.tech_support_direct_effect_dowhy_model\",\n",
    "    model=tech_support_effect_model,\n",
    "    estimand=tech_support_direct_effect_identified_estimand,\n",
    "    estimate=tech_support_direct_effect_estimate,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "edb0944a-5523-4308-803f-bcc8146c19ca",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Estimating effect of \"Discount\" in \"Revenue\"\n",
    "\n",
    "We will follow a similar approach as the one use to identify and estimate the total effect of ```Tech Support``` to now identify and estimate the effect of ```Discount``` on ```Revenue```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fe3a2163-841b-4166-a119-2ce485d91538",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "discount_effect_model = dowhy.CausalModel(\n",
    "    data=input_df, graph=graph, treatment=\"Discount\", outcome=\"Revenue\"\n",
    ")\n",
    "\n",
    "discount_effect_identified_estimand = discount_effect_model.identify_effect(\n",
    "    estimand_type=\"nonparametric-ate\",\n",
    "    method_name=\"maximal-adjustment\",\n",
    ")\n",
    "\n",
    "print(discount_effect_identified_estimand)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2c909cb2-2577-481e-badb-0683f987ca5d",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "mlflow.autolog(disable=True)\n",
    "\n",
    "model_t, model_y = setup_treatment_and_out_models()\n",
    "\n",
    "effect_modifiers = [\"Size\", \"Global Flag\"]\n",
    "method_name = \"backdoor.econml.dml.LinearDML\"\n",
    "init_params = {\n",
    "    \"model_t\": model_t,\n",
    "    \"model_y\": model_y,\n",
    "    \"linear_first_stages\": True,\n",
    "    \"discrete_treatment\": True,\n",
    "    \"cv\": 3,\n",
    "    \"mc_iters\": 10,\n",
    "}\n",
    "\n",
    "discount_effect_estimate = discount_effect_model.estimate_effect(\n",
    "    discount_effect_identified_estimand,\n",
    "    confidence_intervals=True,\n",
    "    effect_modifiers=effect_modifiers,\n",
    "    method_name=method_name,\n",
    "    method_params={\"init_params\": init_params},\n",
    ")\n",
    "\n",
    "discount_effect_estimate.interpret()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0f17cddd-3978-49fb-a13e-beaa397793d8",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "model_details = register_dowhy_model(\n",
    "    model_name=f\"{catalog}.{db}.discount_dowhy_model\",\n",
    "    model=discount_effect_model,\n",
    "    estimand=discount_effect_identified_estimand,\n",
    "    estimate=discount_effect_estimate,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5b52b2d6-7047-4544-bec4-d12ddccaa5d8",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Estimating the effect of \"New Engagment Strategy\" in \"Revenue\"\n",
    "\n",
    "Finally we will estimate the effect of the ```New Engagement Strategy``` incentive.  The graph obtained from the previous notebook displayed no effect on ```Revenue```.  We should see the same when identifying this effect and estimating it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8daac7f0-1286-45ce-aef5-3ea506e9ac42",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "new_strategy_effect_model = dowhy.CausalModel(\n",
    "    data=input_df, graph=graph, treatment=\"New Engagement Strategy\", outcome=\"Revenue\"\n",
    ")\n",
    "\n",
    "new_strategy_effect_identified_estimand = new_strategy_effect_model.identify_effect(\n",
    "    proceed_when_unidentifiable=True\n",
    ")\n",
    "\n",
    "print(new_strategy_effect_identified_estimand)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8226a19b-8575-49cc-ae9a-bad7e92cfed4",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "warnings.simplefilter(\"ignore\")\n",
    "\n",
    "new_strategy_effect_estimate = new_strategy_effect_model.estimate_effect(\n",
    "    new_strategy_effect_identified_estimand,\n",
    "    method_name=\"backdoor.propensity_score_matching\",\n",
    "    target_units=\"att\",\n",
    ")\n",
    "\n",
    "new_strategy_effect_estimate.value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "9373cd46-22ca-4b83-8a46-c3df70768340",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "[DoWhy](https://github.com/py-why/dowhy) also finds no effect.\n",
    "\n",
    "Please notice [DoWhy](https://github.com/py-why/dowhy) decides not to use ```Plan Summit``` as a feature for the estimation.  If included, a spurious effect would be percived, leading us to wrong conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "495cfae7-1e55-499a-8013-0aa2378f40b2",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "###Comparing estimated effects with ground thruth\n",
    "\n",
    "As mentioned before the data for this accelerator was generated using probabilistic methods.  The ground truth is provided in the original dataset.  When compared with the estimated effect we see the estimations are very close."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "dbe3b251-ae2a-4253-bca3-9d31e9870593",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "estimates_df = pd.DataFrame(\n",
    "    {\n",
    "        \"Estimated Direct Treatment Effect: Tech Support\": [\n",
    "            tech_support_direct_effect_estimate.value\n",
    "        ],\n",
    "        \"Estimated Total Treatment Effect: Tech Support\": [\n",
    "            tech_support_total_effect_estimate.value\n",
    "        ],\n",
    "        \"Estimated Total Treatment Effect: Discount\": [\n",
    "            discount_effect_estimate.value\n",
    "        ],\n",
    "        \"Estimated Total Treatment Effect: New Engagement Strategy\": [\n",
    "            new_strategy_effect_estimate.value\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "\n",
    "compare_estimations_vs_ground_truth(ground_truth_df, estimates_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bf5ea988-f494-4a20-8572-b0e31fc3443e",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "As we can see, the estimations of the models are not only directioanlly correct but very closed the the actual values used in the simulations:\n",
    "- ``Tech Support`` direct influence is only off by around $12 \n",
    "- ``Tech Support`` accounting for the direct and indirect influences is only off by around $110\n",
    "- ``Discount`` direct influence is only off by $4\n",
    "- ``New Strategy`` has no influence in ``Revenue``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a5a469e8-a40e-4428-8884-dfd9c637c677",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "© 2023 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License. All included or referenced third party libraries are subject to the licenses set forth below.\n",
    "\n",
    "| library                                | description             | license    | source                                              |\n",
    "|----------------------------------------|-------------------------|------------|-----------------------------------------------------|\n",
    "| dowhy   | Python library for causal inference that supports explicit modeling and testing of causal assumptions | MIT   | https://pypi.org/project/dowhy/          \n",
    "| econml   |  contains several methods for calculating Conditional Average Treatment Effects | MIT    | https://pypi.org/project/econml/  \n",
    "| causal-learn   | python package for causal discovery  | MIT    | https://pypi.org/project/causal-learn/"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "computePreferences": null,
   "dashboards": [],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "mostRecentlyExecutedCommandWithImplicitDF": {
     "commandId": 1375557022084086,
     "dataframes": [
      "_sqldf"
     ]
    },
    "pythonIndentUnit": 2
   },
   "notebookName": "02_identification_estimation",
   "widgets": {}
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}