MSB1015-Practical.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ffb3d642",
   "metadata": {},
   "source": [
    "## Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aa21c97",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install camog"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f4eba6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa43dbad",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4df39e13",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f48613cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install opencv-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "472cfeaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5753e4db",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f718560f",
   "metadata": {},
   "source": [
    "## Load packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92bda307",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "import multiprocessing as mp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba9bded8",
   "metadata": {},
   "source": [
    "## Check the number of logical cores (threads) you have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4a1a806",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_OF_THREADS = mp.cpu_count()\n",
    "\n",
    "print(\"You have {} logical cores\".format(NUM_OF_THREADS))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca0df56",
   "metadata": {},
   "source": [
    "**Jupyter notebook is capable of measuring the time needed to execute code in a cell using \"cell magic\".**\n",
    "\n",
    "**However, if you want to have a similar functionality in your Python scripts aside from Jupyter, you can use the following code snippet.**\n",
    "\n",
    "**For this practical, we will use Jupyter notbook cell magic to time the code execution**\n",
    "\n",
    "Documentation: https://ipython.readthedocs.io/en/stable/interactive/magics.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "481e546d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# timing code without jupyter notebook cell magic\n",
    "\n",
    "total_runs = 10\n",
    "\n",
    "time_runs = np.empty((0,0))\n",
    "\n",
    "for i in range(total_runs):\n",
    "\n",
    "    start_time = time.time() \n",
    "\n",
    "    ### Start of code\n",
    "    \n",
    "    \n",
    "    ### End of code\n",
    "\n",
    "    end_time = time.time() \n",
    "\n",
    "    time_taken = round(end_time - start_time, 2)\n",
    "    \n",
    "    print(\"Time taken for run {}: {} seconds\".format(i, time_taken))\n",
    "    \n",
    "    time_runs = np.append(time_runs, time_taken)\n",
    "\n",
    "print('Average run time: {}'.format(round(np.mean(time_runs), 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1152fc7",
   "metadata": {},
   "source": [
    "## Exercise 1: Read files in parallel, using the \"camog\" package"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cc8b0d0",
   "metadata": {},
   "source": [
    "Download the augmented gene expression file from the following link, unzip the file and place the resulting CSV inside the data/ folder\n",
    "https://drive.google.com/file/d/1xpaueGzBUpK2lSECN-JpbReg3pKUqFcq/view?usp=sharing\n",
    "\n",
    "This is a made-up large file simulating gene expression data for 3000 sample and ~43000 human genes\n",
    "\n",
    "The file size is ~2.4GB and the large size is made on purpose to allow measuring reasonable difference in performance when reading it in a parallel way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b244a405",
   "metadata": {},
   "outputs": [],
   "source": [
    "import camog"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42597dee",
   "metadata": {},
   "source": [
    "The following 4 code snippets read the file \"augmented_gene_expression.csv\" using 1,2,4,8 threads respectively and print the execution time underneath each cell.\n",
    "Run those cells and compare the obtained times.\n",
    "\n",
    "What are your observations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfd9c9c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n10\n",
    "\n",
    "headers, columns = camog.load('data/augmented_gene_expression.csv', nthreads=1)\n",
    "pandas_df = pd.DataFrame({key: value for key, value in zip(headers, columns)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "605657ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n10\n",
    "\n",
    "headers, columns = camog.load('data/augmented_gene_expression.csv', nthreads=2)\n",
    "pandas_df = pd.DataFrame({key: value for key, value in zip(headers, columns)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41a90c51",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n10\n",
    "\n",
    "headers, columns = camog.load('data/augmented_gene_expression.csv', nthreads=4)\n",
    "pandas_df = pd.DataFrame({key: value for key, value in zip(headers, columns)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ca1195e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n10\n",
    "\n",
    "headers, columns = camog.load('data/augmented_gene_expression.csv', nthreads=8)\n",
    "pandas_df = pd.DataFrame({key: value for key, value in zip(headers, columns)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da59590c",
   "metadata": {},
   "source": [
    "Lets check now the run time when using Pandas directly to read the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93e7a6a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n5 \n",
    "\n",
    "pandas_df = pd.read_csv('data/augmented_gene_expression.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a158c54",
   "metadata": {},
   "source": [
    "The \"camog\" package is an open source project available on GitHub https://github.com/walshb/camog/tree/master\n",
    "\n",
    "Check the source code file reponsible for reading the csv file in parallel and **explain what type of parallel programming is used in this package, which programming language and what library is used.**\n",
    "\n",
    "**Hint:** start with this file https://github.com/walshb/camog/blob/master/camog/_csv.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "242373e8",
   "metadata": {},
   "source": [
    "## Exercise 2: Populate a Pandas dataframe with API calls (enrich UniProt IDs with protein information)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "557b755e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d02de5f2",
   "metadata": {},
   "source": [
    "The following code line reads a dataframe of 100 rows and five columns. The first column contains UniProt IDs for 100 proteins and the remaing columns are empty.\n",
    "\n",
    "The purpose of this exercise is to apply parallel computing on Pandas datafram and make external API calls to retrieve information about the protiens in the first column and fill the rest of empty column with relevant information about those proteins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfd409d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "uniprot_df = pd.read_csv('data/uniprot_ids_df.csv', encoding='utf-8', dtype=str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "658ddae9",
   "metadata": {},
   "outputs": [],
   "source": [
    "uniprot_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79fcecb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "uniprot_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21b77086",
   "metadata": {},
   "source": [
    "The code cell bellow define a function that will be applied on each dataframe chunk processed in parallel. It takes a dataframe as input (in the parallel way, the dataframe is split into multiple chunks, each to be handled by a different thread) and returns the same dataframe after filling the empty column."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6620692",
   "metadata": {},
   "source": [
    "Explain the function code below using comments showing the role of each piece of code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bc51887",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_df_row(df):\n",
    "    \n",
    "    uniprot_url = \"https://rest.uniprot.org/uniprotkb/{}.json\"\n",
    "    \n",
    "    for i, row in df.iterrows():\n",
    "        \n",
    "        uniprot_id = row['uniprot_id'].strip()\n",
    "        \n",
    "        try:\n",
    "            response = requests.get(uniprot_url.format(uniprot_id))\n",
    "            \n",
    "            response_json = json.loads(response.content)\n",
    "            \n",
    "            try:\n",
    "                proten_name = response_json['proteinDescription']['recommendedName']['fullName']['value']\n",
    "            except KeyError:\n",
    "                proten_name = \"\"\n",
    "            \n",
    "            try:\n",
    "                proten_length = response_json['sequence']['length']\n",
    "            except KeyError:\n",
    "                proten_length = \"\"\n",
    "            \n",
    "            try:\n",
    "                proten_organism = response_json['organism']['scientificName']\n",
    "            except KeyError:\n",
    "                proten_organism = \"\"\n",
    "                \n",
    "                \n",
    "            try:\n",
    "                proten_sequence = response_json['sequence']['value']\n",
    "            except KeyError:\n",
    "                proten_sequence = \"\"\n",
    "            \n",
    "            \n",
    "            df.at[i, 'protein_name'] = proten_name\n",
    "            df.at[i, 'protein_length'] = proten_length\n",
    "            df.at[i, 'protein_organism'] = proten_organism\n",
    "            df.at[i, 'protein_sequence'] = proten_sequence\n",
    "            \n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "        \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57fe06d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the defined function on the dataframe\n",
    "# this line is meant to show how the output look like and not meant to assess performance\n",
    "uniprot_enriched_df = process_df_row(uniprot_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c598ccb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "uniprot_enriched_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a808f5df",
   "metadata": {},
   "source": [
    "The following two cells compare the sequential and parallel processing of the dataframe.\n",
    "\n",
    "Run the code and compare the results.\n",
    "\n",
    "You can try the parallel part with different number of cores and see how does that effect the execution time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "181766db",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n5\n",
    "\n",
    "process_df_row(uniprot_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c935fb3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n5\n",
    "\n",
    "NUM_OF_THREADS = 4\n",
    "\n",
    "df_splits = np.array_split(uniprot_df, NUM_OF_THREADS)\n",
    "\n",
    "pool = mp.Pool(NUM_OF_THREADS)\n",
    "\n",
    "results = pool.map(process_df_row, df_splits)\n",
    "\n",
    "uniprot_enriched_df = pd.concat(results)\n",
    "\n",
    "pool.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c03936",
   "metadata": {},
   "source": [
    "## Exercise 3: Apply edge detection to blood cell image files in parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d27769",
   "metadata": {},
   "source": [
    "In this exercise, we will work with applying an image processing function on a large number of blood cell images (simulating what you would do in a similar research project) and we will compare this process with and without parallel computing.\n",
    "\n",
    "The dataset was originally obtained from Kaggle (https://www.kaggle.com/datasets/paultimothymooney/blood-cells/). However, for this practical, the image were copied and multiplied a couple of time to increase their number in order the observe reasonable differences between sequential and parallel approaches.\n",
    "\n",
    "Therefore, download the image dataset from the following URL: https://drive.google.com/file/d/1a5EPJPSrrpaKTu6tvIY37sdtqoPdTMNd/view?usp=sharing\n",
    "\n",
    "Unzip the folder into the data/ folder and make sure that you have the images directly under data/original_images/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "651e72bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import cv2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d64f8746",
   "metadata": {},
   "outputs": [],
   "source": [
    "files_path = 'data/original_images'\n",
    "\n",
    "# Get a list of all image file names in the specified path\n",
    "list_of_files = [f for f in listdir(files_path) if isfile(join(files_path, f))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f83b904",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(list_of_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e912c655",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_files[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "158e7113",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_image(image_file):\n",
    "    \n",
    "    folder = \"data/original_images\"\n",
    "    folder_processed = \"data/processed_images\"\n",
    "        \n",
    "    image = cv2.imread(folder+\"/\"+image_file)\n",
    "  \n",
    "    # convert to gray scale image\n",
    "    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "    # apply median filter for smoothing\n",
    "    blurM = cv2.medianBlur(gray, 5)\n",
    "    \n",
    "    \n",
    "    # apply Canny edge detector and save the output to processed_images folder\n",
    "    edgeM = cv2.Canny(blurM, 10, 50)\n",
    "    cv2.imwrite(folder_processed+\"/\"+image_file, edgeM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15b0cedc",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n5\n",
    "\n",
    "for image_file in list_of_files:\n",
    "    \n",
    "    process_image(image_file)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "933c9acf",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit -r1 -n5\n",
    "\n",
    "NUM_OF_THREADS = 4\n",
    "\n",
    "pool = mp.Pool(NUM_OF_THREADS)\n",
    "pool.map(process_image, list_of_files)\n",
    "\n",
    "pool.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e725528f",
   "metadata": {},
   "source": [
    "What is the difference in using pool.map() in this excercise compared to excercise 2?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4361d1fc",
   "metadata": {},
   "source": [
    "## Exercise 4: machine learning model training using parallel computing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "deb4a0a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "from sklearn.datasets import make_regression \n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import RepeatedStratifiedKFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95ca8dd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make up a regression dataset sample to use it for machine learning model training\n",
    "X, y = make_regression(n_samples=30000, n_features=25, n_informative=10,\n",
    "                        random_state=0, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6fffbe",
   "metadata": {},
   "source": [
    "Explain in comments the code below, run the code, what are your observations from the output plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c227e42b",
   "metadata": {},
   "outputs": [],
   "source": [
    "time_runs = np.empty((0,0))\n",
    "\n",
    "n_cores = [1,2,3,4,5,6,7,8]\n",
    "\n",
    "for n in n_cores:\n",
    "    \n",
    "    start_time = time.time() \n",
    "\n",
    "    model = RandomForestRegressor(n_jobs=n)\n",
    "    \n",
    "    model.fit(X, y)\n",
    "\n",
    "    end_time = time.time() \n",
    "\n",
    "    time_taken = round(end_time - start_time, 2)\n",
    "\n",
    "    print(\"Time taken for run {}: {} seconds\".format(n, time_taken))\n",
    "\n",
    "    time_runs = np.append(time_runs, time_taken)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74f6da89",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(n_cores, time_runs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22bf8500",
   "metadata": {},
   "source": [
    "To learn more about parallism in scikit-learn, check this link: https://scikit-learn.org/stable/computing/parallelism.html\n",
    "\n",
    "Which parallel computing paradigm is used by default in scikit-learn?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c764a9bd",
   "metadata": {},
   "source": [
    "## Exercise 5: Solve the following problem using a parallel programming approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99efe2d1",
   "metadata": {},
   "source": [
    "1. Load the CSV file in the \"data\" folder, named \"pdb_ids.csv\" using Pandas\n",
    "2. Create a function to process each row of the dataframe. The function should take one argument of type DataFrame and return a dataframe object.\n",
    "The function should iterate through the dataframe rows and perform the following steps:\n",
    "    * Get the PDB ID from the relevant column and make an HTTP call to download the protein image from PDB (use the following URL template: http://cdn.rcsb.org/images/structures/dl/{}/{}_assembly-1.jpeg).\n",
    "    * Save the content of the response (binary content) to an image file stored in the folder \"data/pdb_images\" named with the PDB id and the extension \"jpeg\"\n",
    "    * read the image file from the folder using OpenCV and extract the size of the image (i.e. width and height)\n",
    "    * store the width and the height of the image in the relevant columns in the dataframe\n",
    "3. Save the dataframe to a file\n",
    "4. Use the timing template provided at the beginning of this notebook to time your code and test it using 2, 4 and 8 cores"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}