01_causal_discovery.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c0be0376-f008-4fb8-8ca5-cd67241d7ec5",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "This solution accelerator notebook is available at https://github.com/databricks-industry-solutions/causal-incentive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "946b617e-bb55-4933-a769-c73cea2f4916",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Cluster Configuration\n",
    "We recommend using a cluster with the following or similar specifications to run this solution accelerator:\n",
    "- Unity Catalog enabled cluster\n",
    "- Databricks Runtime 15.4 LTS ML or above\n",
    "- Single-node cluster: e.g. `m5d.2xlarge` on AWS or `Standard_D8ds_v5` on Azure Databricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "aa9221e4-ab26-447e-9c21-20b3cb69db44",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "## Install Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "60f2f5e1-6ae9-4eea-b39d-8275fdc66a76",
     "showTitle": true,
     "tableResultSettingsMap": {},
     "title": "Install graphviz from nicer visualization"
    },
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%sh \n",
    "sudo apt-get -qq update\n",
    "sudo apt-get -y -qq install graphviz libgraphviz-dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "61a0f224-8195-4c8c-82c5-c273e0d7ee9b",
     "showTitle": true,
     "tableResultSettingsMap": {},
     "title": "Install requirements"
    }
   },
   "outputs": [],
   "source": [
    "%pip install -r ./requirements.txt pygraphviz==1.10 --quiet\n",
    "dbutils.library.restartPython()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d2f6fd8a-0cb6-44ef-9312-8d6d27403d5a",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    },
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%run ./util/notebook-config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8be398d8-971b-4f43-a35d-782ad8eb618e",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "#Discovering the network of influences among the Features\n",
    "\n",
    "In order to isolate the influence we plan to estimate,  we need first to understand the relations among the available features.  We will use [PC algorithm](https://www.youtube.com/watch?v=o2A61bJ0UCw) implemented in the [PyWhy](https://www.pywhy.org/) package called [causal-learn](https://github.com/py-why/causal-learn), to discover the basic skeleton of the network. Before generating the skeleton, it is possible to set some requirements on the graph. For example, we discussed in the previous notebook that ```Size``` affects ```IT Spend``` and ```Employee Count``` influences ```PC Count```. We will add these causal connections when we generate the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "539c43b9-846f-498b-807e-dd0864091d6b",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import causallearn\n",
    "from causallearn.search.ConstraintBased.PC import pc\n",
    "\n",
    "required_edges = [\n",
    "    {\"from\": \"Size\", \"to\": \"IT Spend\"},\n",
    "    {\"from\": \"Employee Count\", \"to\": \"PC Count\"},\n",
    "]\n",
    "\n",
    "# Get the background_knowledge parameter from the required edges defined above. \n",
    "background_knowledge = add_background_knowledge(\n",
    "  edges = required_edges\n",
    ")\n",
    "\n",
    "# Parameters\n",
    "parameters = {\"node_names\": input_df.columns, \n",
    "              \"background_knowledge\": background_knowledge, \n",
    "              \"alpha\": 0.01, \n",
    "              \"indep_test\": \"fisherz\"}\n",
    "\n",
    "cg = pc(data=np.vstack(input_df.to_numpy()), **parameters)\n",
    "\n",
    "# Visualization using pydot\n",
    "cg.draw_pydot_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5fb82a8b-4352-43fe-966a-60d7a7160458",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "The bare bones skeleton discovered offers some interesting insights:\n",
    "\n",
    "- ```Discount``` seems to have a direct impact in ```Revenue```.\n",
    "- ```Tech Support``` appears to have a direct impact in ```Revenue``` and a mediated one through ```New Product Adoption```.\n",
    "- The ```New Engagement Strategy``` seems not to influence ```Revenue```.\n",
    "- Both ```Revenue``` and ```New Engagement Strategy``` influence ```Planning Summit```. This representas a [collider pattern](https://en.wikipedia.org/wiki/Collider_(statistics)) which could result in creating a fictitious relation between ```New Engagement Strategy``` and ```Revenue``` if ```Planning Summit``` is included as a feature during the influence estimation!! (this pattern is also known as [selection bias](https://catalogofbias.org/biases/collider-bias/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "04e61f57-7fdf-4467-b6d8-fad10dcc3024",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "#Adding Domain Knowledge Assumptions to the Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3d04da71-222a-4869-91e6-13a713ac1d99",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "The skeleton lacks directions in some of the relations.  Some of the missing directions are obvious:\n",
    "\n",
    "- ```IT Spend``` -> ```Tech Support```\n",
    "- ```Tech Support``` -> ```New Product Adoption```\n",
    "- ```Major Flag``` -> ```New Engagment Strategy```\n",
    "\n",
    "We will add these directions to the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bcfe02e4-3fae-426d-a738-73d5835920d0",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Adding missing directions\n",
    "added_directions = [\n",
    "    {\"from\": \"IT Spend\", \"to\": \"Tech Support\"},\n",
    "    {\"from\": \"Tech Support\", \"to\": \"New Product Adoption\"},\n",
    "    {\"from\": \"Major Flag\", \"to\": \"New Engagement Strategy\"},\n",
    "]\n",
    "\n",
    "add_directions(\n",
    "    causal_graph=cg,\n",
    "    directions=added_directions,\n",
    ")\n",
    "\n",
    "cg.draw_pydot_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5991b566-77f8-440d-b5cd-23ec2fcde25c",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "Some of the discovered directions are clearly wrong:\n",
    "\n",
    "- ```Revenue``` should not convert a company from the commercial sector to the public sector\n",
    "- Providing the ```New Engagement Strategy``` incentive to a company will not converted it from the commerical sector to the public sector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "910811fe-3f44-4c5f-b779-d81e013a983a",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Correcting directions\n",
    "inverted_directions = [\n",
    "    {\"from\": \"Revenue\", \"to\": \"Commercial Flag\"},\n",
    "    {\"from\": \"New Engagement Strategy\", \"to\": \"Commercial Flag\"},\n",
    "]\n",
    "invert_directions(\n",
    "    causal_graph=cg,\n",
    "    directions=inverted_directions,\n",
    ")\n",
    "\n",
    "cg.draw_pydot_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "b05c4958-c7d6-45e8-bc6c-7868cea8fd3f",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "Even though some relations were not automatically discovered, The business experts of the software company are pretty convinced ```Global Flag``` and ```Major Flag``` have an influence in ```Revenue```.  We will add these assumptions to the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "662f9d42-276e-4c09-b27d-6504c0bb1ba3",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Adding missing relations based on domain knowledge\n",
    "added_missing_directed_relations = [\n",
    "    {\"from\": \"Global Flag\", \"to\": \"Revenue\"},\n",
    "    {\"from\": \"Major Flag\", \"to\": \"Revenue\"},\n",
    "]\n",
    "\n",
    "add_directions(\n",
    "    causal_graph=cg,\n",
    "    directions=added_missing_directed_relations,\n",
    ")\n",
    "\n",
    "cg.draw_pydot_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "dbc21d42-1b38-446b-ac49-888c69a40829",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "Finally, even thought there was no incentive assignation policy, we believe the basic characteristics of the company had an influence in the assignation.  We will add a directed relation from each of these characteristics to each of the promotional offers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2795b8e8-75c0-41d8-b7b4-84a366cd6df7",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Add effect from all basic characteristics to incentives\n",
    "account_basic_characteristics = [\n",
    "    \"Major Flag\",\n",
    "    \"SMC Flag\",\n",
    "    \"Commercial Flag\",\n",
    "    \"IT Spend\",\n",
    "    \"Employee Count\",\n",
    "    \"PC Count\",\n",
    "]\n",
    "\n",
    "add_relations_influencing_incentives(\n",
    "    causal_graph=cg,\n",
    "    incentives=[\"Discount\", \"Tech Support\", \"New Engagement Strategy\"],\n",
    "    account_basic_characteristics=account_basic_characteristics,\n",
    ")\n",
    "\n",
    "cg.draw_pydot_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7fc7ea1a-21ff-483f-8b1c-05a312df0022",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "The network defined above,  also known as the Causal Graph, will guide the <b>identification</b> and <b>estimation</b> phases.  We will proceed to store it as an artifact in an MLFlow experiment run to then use it in the next step (the ```02_identification_estimation``` notebook).  \n",
    "\n",
    "Please notice we are also storing in the [MLfLow](https://databricks.atlassian.net/wiki/spaces/UN/pages/2893873880/Brickstore) the algorithm used for the discovery, the parameters applied to the algorithm,  and all the alterations made to the discovered graph skeleton."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7ef03eaa-5715-4bc7-967b-b9a52baae38d",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "with mlflow.start_run(run_name=\"causal_discovery\") as run:\n",
    "    mlflow.log_params(\n",
    "        {\n",
    "            **{\n",
    "                \"algorithm\": \"PC\",\n",
    "                \"library\": f\"causal-learn=={get_version('causal-learn')}\",\n",
    "            },\n",
    "            **parameters,\n",
    "            **{\n",
    "                \"added_directions\": str(added_directions),\n",
    "                \"inverted_directions\": str(inverted_directions),\n",
    "                \"added_missing_directed_relations\": str(\n",
    "                    added_missing_directed_relations\n",
    "                ),\n",
    "                \"account_basic_characteristics\": str(account_basic_characteristics),\n",
    "            },\n",
    "        }\n",
    "    )\n",
    "\n",
    "    # Serialize the graph to a file and log it to MLflow for reference.\n",
    "    graph = to_pydot(cg.G, labels=input_df.columns)\n",
    "    with open(\"/databricks/driver/graph.txt\", \"w\") as f:\n",
    "        f.write(graph)\n",
    "\n",
    "    mlflow.log_artifact(\"/databricks/driver/graph.txt\", artifact_path=\"graph\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "eb6f6fd0-d18c-4aa6-9e01-00e6760e74e6",
     "showTitle": false,
     "tableResultSettingsMap": {},
     "title": ""
    }
   },
   "source": [
    "© 2023 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License. All included or referenced third party libraries are subject to the licenses set forth below.\n",
    "\n",
    "| library                                | description             | license    | source                                              |\n",
    "|----------------------------------------|-------------------------|------------|-----------------------------------------------------|\n",
    "| dowhy   | Python library for causal inference that supports explicit modeling and testing of causal assumptions | MIT   | https://pypi.org/project/dowhy/          \n",
    "| econml   |  contains several methods for calculating Conditional Average Treatment Effects | MIT    | https://pypi.org/project/econml/  \n",
    "| causal-learn   | python package for causal discovery  | MIT    | https://pypi.org/project/causal-learn/"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "computePreferences": null,
   "dashboards": [],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "mostRecentlyExecutedCommandWithImplicitDF": {
     "commandId": 1375557022083871,
     "dataframes": [
      "_sqldf"
     ]
    },
    "pythonIndentUnit": 2
   },
   "notebookName": "01_causal_discovery",
   "widgets": {}
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}