SI650_Homework_2_Covid_Ranking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MtfWvIMlT1L4"
   },
   "source": [
    "# Homework 2\n",
    "Please see the write-up on Canvas for full details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3iHEKIUSK6aL"
   },
   "source": [
    "### - **Name**: **Yashaswini**\n",
    "### - **Kaggle Name**: **Yashaswini Joshi**\n",
    "### - **Unique Name**: **yjoshi**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0oLdz9VmT1L7"
   },
   "source": [
    "# install metapy, it may take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "id": "XjUUOguhUCgE",
    "outputId": "107b5efb-1f1c-4b8c-d40b-c5d8a20601ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting metapy\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/81/a4/92dae084446597d6bbf355e7eaff3e83dcb51e33d434f43ecdea4c0c4b0a/metapy-0.2.13-cp36-cp36m-manylinux1_x86_64.whl (14.3MB)\n",
      "\u001b[K     |████████████████████████████████| 14.3MB 297kB/s \n",
      "\u001b[?25hInstalling collected packages: metapy\n",
      "Successfully installed metapy-0.2.13\n"
     ]
    }
   ],
   "source": [
    "!pip install metapy\n",
    "import metapy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "43ocIrmPT1MC"
   },
   "source": [
    "# Download the dataset files for this assignment. \n",
    "\n",
    "These files are also on canvas. The `wget` and `tar` commands may not work on Windows, so if this command doesn't work, just download them on canvas. These commands should work if you're doing the assignment on Google Collab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "pWwB7o8zUHAV",
    "outputId": "8f8484b0-d24e-4ad0-dbf3-e136087c8716"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-10-14 03:23:10--  https://raw.githubusercontent.com/meta-toolkit/meta/master/data/lemur-stopwords.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2747 (2.7K) [text/plain]\n",
      "Saving to: ‘lemur-stopwords.txt’\n",
      "\n",
      "\r",
      "lemur-stopwords.txt   0%[                    ]       0  --.-KB/s               \r",
      "lemur-stopwords.txt 100%[===================>]   2.68K  --.-KB/s    in 0s      \n",
      "\n",
      "2020-10-14 03:23:11 (45.9 MB/s) - ‘lemur-stopwords.txt’ saved [2747/2747]\n",
      "\n",
      "--2020-10-14 03:23:11--  http://www-personal.umich.edu/~shiyansi/covid_ir.tar.gz\n",
      "Resolving www-personal.umich.edu (www-personal.umich.edu)... 141.211.243.103\n",
      "Connecting to www-personal.umich.edu (www-personal.umich.edu)|141.211.243.103|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 69098957 (66M) [application/x-gzip]\n",
      "Saving to: ‘covid_ir.tar.gz’\n",
      "\n",
      "covid_ir.tar.gz     100%[===================>]  65.90M  1.25MB/s    in 57s     \n",
      "\n",
      "2020-10-14 03:24:08 (1.15 MB/s) - ‘covid_ir.tar.gz’ saved [69098957/69098957]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -nc https://raw.githubusercontent.com/meta-toolkit/meta/master/data/lemur-stopwords.txt\n",
    "!wget -N http://www-personal.umich.edu/~shiyansi/covid_ir.tar.gz\n",
    "!tar xf covid_ir.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QxwJO9V4T1MG"
   },
   "source": [
    "# Generate the metapy header configuration for you\n",
    "Metapy is a powerful IR library. To lower the barrier for entry, we're generating the configuration that tells metapy how the task is setup and what is needed. You should keep this file the same when verifying your BM25 implementation in Part 1 of the assignment.  However, you can generate a different version of it if you want when trying to outperform BM25 in Part 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "R1mkpb7oUJ3S"
   },
   "outputs": [],
   "source": [
    "with open('covid_ir/tutorial.toml', 'w') as f:\n",
    "    f.write('type = \"line-corpus\"\\n')\n",
    "    f.write('store-full-text = true\\n')\n",
    "\n",
    "config = \"\"\"prefix = \".\" # tells MeTA where to search for datasets\n",
    "\n",
    "dataset = \"covid_ir\" # a subfolder under the prefix directory\n",
    "corpus = \"tutorial.toml\" # a configuration file for the corpus specifying its format & additional args\n",
    "\n",
    "index = \"covid_ir-idx\" # subfolder of the current working directory to place index files\n",
    "\n",
    "query-judgements = \"covid_ir/covid_ir-qrels.txt\" # file containing the relevance judgments for this dataset\n",
    "\n",
    "stop-words = \"lemur-stopwords.txt\"\n",
    "\n",
    "[[analyzers]]\n",
    "method = \"ngram-word\"\n",
    "ngram = 1\n",
    "filter = \"default-unigram-chain\"\n",
    "\"\"\"\n",
    "with open('covid_ir-config.toml', 'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C70KJbp_T1ML"
   },
   "source": [
    "### Build the inverted index with metapy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "ztJSZMExVkMW"
   },
   "outputs": [],
   "source": [
    "inv_idx = metapy.index.make_inverted_index('covid_ir-config.toml') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2V_NNwl2T1MP"
   },
   "source": [
    "## Problem 1: Re-implemented BM25 (25 points) and Pivoted Indexing (25 points)\n",
    "We've provided a skeleton of a ranking function below with examples of commonly-used parameters. For each, you should re-implement it using the formulas as defined in the lecture slides. You are welcome to use any hyperparameter choices you want (NOTE: as mentioned in the homework, changing these does not count as a new method for Problem 2). \n",
    "\n",
    "To test for correctness, you can compare your method against metapy's implementations. We've included one below to get you started. Your solution should return the exact same ranking as their implementation when using identical hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "r2-p9AGfV1Ia"
   },
   "outputs": [],
   "source": [
    "# You can define your own retrieval function \n",
    "import math \n",
    "class MyBM25Reimplementation(metapy.index.RankingFunction):                                                                                                                    \n",
    "    def __init__(self,  k1 = 1.2, b = 0.75, k3 = 500):                                             \n",
    "        self.k1 = k1\n",
    "        self.b = b\n",
    "        self.k3 = k3\n",
    "        # You *must* invoke the base class __init__() here!\n",
    "        super(MyBM25Reimplementation, self).__init__()                                        \n",
    "                                                                                 \n",
    "    def score_one(self, sd):\n",
    "        \"\"\"\n",
    "        You need to override this function to return a score for a single term.\n",
    "        \n",
    "        You may want to call some of the following variables when implementing your retrieval function:\n",
    "        \n",
    "        sd.avg_dl: average document length of the collection\n",
    "        sd.num_docs: total number of documents in the index\n",
    "        sd.total_terms: total number of terms in the index\n",
    "        sd.query_length: the total length of the current query (sum of all term weights)\n",
    "        sd.query_term_weight: query term count (or weight in case of feedback)\n",
    "        sd.doc_count: number of documents that a term t_id appears in\n",
    "        sd.corpus_term_count: number of times a term t_id appears in the collection\n",
    "        sd.doc_term_count: number of times the term appears in the current document\n",
    "        sd.doc_size: total number of terms in the current document\n",
    "        sd.doc_unique_terms: number of unique terms in the current document\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        k1 = self.k1\n",
    "        b = self.b\n",
    "        k3 = self.k3      \n",
    "        #Fill your answers here\n",
    "\n",
    "        V_IDF = math.log((sd.num_docs - sd.doc_count + 0.5) /(sd.doc_count + 0.5))\n",
    "        N_TF = (k1+1)* sd.doc_term_count / (k1*(1-b+b*sd.doc_size/sd.avg_dl) + sd.doc_term_count)\n",
    "        QTF = (k3+1)*sd.query_term_weight / (k3 + sd.query_term_weight)\n",
    "        return (V_IDF * N_TF * QTF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "GTxPdqbyVoWj"
   },
   "outputs": [],
   "source": [
    "#ranker = metapy.index.OkapiBM25(k1 = 1.2, b = 0.5, k3 = 500)\n",
    "ranker = MyBM25Reimplementation()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 248
    },
    "id": "DjzVYEQzVJAn",
    "outputId": "60e437e8-d48a-4f57-a42f-2090e04daa90"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query:  coronavirus origin\n",
      "Retrieved Results\n",
      "1 Bat-Origin Coronaviruses Expand Their Host Range to Pigs Infections with bat-origin coronaviruses have caused severe illness in humans by ‘host jump’. Recently, novel bat-origin coronaviruses were found in pigs. The large number of mutations on the receptor-binding domain allowed the viruses to infect the new host, posing a potential threat to both agriculture and public health.\n",
      "2 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "3 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "4 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "5 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "6 Origin and evolution of pathogenic coronaviruses Severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) are two highly transmissible and pathogenic viruses that emerged in humans at the beginning of the 21st century. Both viruses likely originated in bats, and genetically diverse coronaviruses that are related to SARS-CoV and MERS-CoV were discovered in bats worldwide. In this Review, we summarize the current knowledge on the origin and evolution of these two pathogenic coronaviruses and discuss their receptor usage; we also highlight the diversity and potential of spillover of bat-borne coronaviruses, as evidenced by the recent spillover of swine acute diarrhoea syndrome coronavirus (SADS-CoV) to pigs.\n",
      "7 Novel Avian Coronavirus and Fulminating Disease in Guinea Fowl, France For decades, French guinea fowl have been affected by fulminating enteritis of unclear origin. By using metagenomics, we identified a novel avian gammacoronavirus associated with this disease that is distantly related to turkey coronaviruses. Fatal respiratory diseases in humans have recently been caused by coronaviruses of animal origin.\n",
      "8 Possible Bat Origin of Severe Acute Respiratory Syndrome Coronavirus 2 We showed that severe acute respiratory syndrome coronavirus 2 is probably a novel recombinant virus. Its genome is closest to that of severe acute respiratory syndrome–related coronaviruses from horseshoe bats, and its receptor-binding domain is closest to that of pangolin viruses. Its origin and direct ancestral viruses have not been identified.\n",
      "9 Possible Bat Origin of Severe Acute Respiratory Syndrome Coronavirus 2 We showed that severe acute respiratory syndrome coronavirus 2 is probably a novel recombinant virus. Its genome is closest to that of severe acute respiratory syndrome-related coronaviruses from horseshoe bats, and its receptor-binding domain is closest to that of pangolin viruses. Its origin and direct ancestral viruses have not been identified.\n",
      "10 Diversity of Coronaviruses in Bats: Insights Into Origin of SARS Coronavirus \n"
     ]
    }
   ],
   "source": [
    "num_results = 10\n",
    "retrieval_results = []\n",
    "with open('covid_ir/covid_ir-queries.txt') as query_file:\n",
    "    for query_num, line in enumerate(query_file):\n",
    "        query = metapy.index.Document()\n",
    "        query.content(line.strip())\n",
    "        results = ranker.score(inv_idx, query, num_results)  \n",
    "        res_list = [(query_num + 1, x[0]) for x in results]\n",
    "        retrieval_results += res_list\n",
    "\n",
    "        \n",
    "        print(\"Query: \", query.content())\n",
    "        print(\"Retrieved Results\")\n",
    "        for num, (d_id, _) in enumerate(results):\n",
    "           content = inv_idx.metadata(d_id).get('content')\n",
    "           print(str(num + 1), content)\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "3X9Jo-8Ri59J"
   },
   "outputs": [],
   "source": [
    "class Pivoted(metapy.index.RankingFunction):                                                                                                                    \n",
    "    def __init__(self, s = 0.1):                                             \n",
    "        self.s = s\n",
    "        # You *must* invoke the base class __init__() here!\n",
    "        super(Pivoted, self).__init__()                                        \n",
    "                                                                                 \n",
    "    def score_one(self, sd):\n",
    "        \"\"\"\n",
    "        You need to override this function to return a score for a single term.\n",
    "        \n",
    "        You may want to call some of the following variables when implementing your retrieval function:\n",
    "        \n",
    "        \n",
    "        sd.avg_dl: average document length of the collection\n",
    "        sd.num_docs: total number of documents in the index\n",
    "        sd.total_terms: total number of terms in the index\n",
    "        sd.query_length: the total length of the current query (sum of all term weights)\n",
    "        sd.query_term_weight: query term count (or weight in case of feedback)\n",
    "        sd.doc_count: number of documents that a term t_id appears in\n",
    "        sd.corpus_term_count: number of times a term t_id appears in the collection\n",
    "        sd.doc_term_count: number of times the term appears in the current document\n",
    "        sd.doc_size: total number of terms in the current document\n",
    "        sd.doc_unique_terms: number of unique terms in the current document\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        s = self.s    \n",
    "        #Fill your answers here\n",
    "        IDF =  math.log((sd.num_docs+1) / sd.doc_count) \n",
    "        Nor_TF = (1 + math.log(1+math.log(sd.doc_term_count))) / (1-s +s *sd.doc_size/sd.avg_dl)\n",
    "        TF = sd.query_term_weight\n",
    "        \n",
    "        return (IDF*Nor_TF*TF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "ETQ9gPaKi6LF"
   },
   "outputs": [],
   "source": [
    "ranker = Pivoted()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 248
    },
    "id": "4L9nOa6BVrPF",
    "outputId": "1c39a506-4f93-40bb-ae14-dc3240544b73"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query:  coronavirus origin\n",
      "Retrieved Results\n",
      "1 Origin and evolution of pathogenic coronaviruses Severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) are two highly transmissible and pathogenic viruses that emerged in humans at the beginning of the 21st century. Both viruses likely originated in bats, and genetically diverse coronaviruses that are related to SARS-CoV and MERS-CoV were discovered in bats worldwide. In this Review, we summarize the current knowledge on the origin and evolution of these two pathogenic coronaviruses and discuss their receptor usage; we also highlight the diversity and potential of spillover of bat-borne coronaviruses, as evidenced by the recent spillover of swine acute diarrhoea syndrome coronavirus (SADS-CoV) to pigs.\n",
      "2 Bat origin of human coronaviruses Bats have been recognized as the natural reservoirs of a large variety of viruses. Special attention has been paid to bat coronaviruses as the two emerging coronaviruses which have caused unexpected human disease outbreaks in the 21st century, Severe Acute Respiratory Syndrome Coronavirus (SARS-CoV) and Middle East Respiratory Syndrome Coronavirus (MERS-CoV), are suggested to be originated from bats. Various species of horseshoe bats in China have been found to harbor genetically diverse SARS-like coronaviruses. Some strains are highly similar to SARS-CoV even in the spike protein and are able to use the same receptor as SARS-CoV for cell entry. On the other hand, diverse coronaviruses phylogenetically related to MERS-CoV have been discovered worldwide in a wide range of bat species, some of which can be classified to the same coronavirus species as MERS-CoV. Coronaviruses genetically related to human coronavirus 229E and NL63 have been detected in bats as well. Moreover, intermediate hosts are believed to play an important role in the transmission and emergence of these coronaviruses from bats to humans. Understanding the bat origin of human coronaviruses is helpful for the prediction and prevention of another pandemic emergence in the future.\n",
      "3 Mosaic evolution of the severe acute respiratory syndrome coronavirus. Severe acute respiratory syndrome (SARS) is a deadly form of pneumonia caused by a novel coronavirus, a viral family responsible for mild respiratory tract infections in a wide variety of animals including humans, pigs, cows, mice, cats, and birds. Analyses to date have been unable to identify the precise origin of the SARS coronavirus. We used Bayesian, neighbor-joining, and split decomposition phylogenetic techniques on the SARS virus replicase, surface spike, matrix, and nucleocapsid proteins to reveal the evolutionary origin of this recently emerging infectious agent. The analyses support a mammalian-like origin for the replicase protein, an avian-like origin for the matrix and nucleocapsid proteins, and a mammalian-avian mosaic origin for the host-determining spike protein. A bootscan recombination analysis of the spike gene revealed high nucleotide identity between the SARS virus and a feline infectious peritonitis virus throughout the gene, except for a 200- base-pair region of high identity to an avian sequence. These data support the phylogenetic analyses and suggest a possible past recombination event between mammalian-like and avian-like parent viruses. This event occurred near a region that has been implicated to be the human receptor binding site and may have been directly responsible for the switch of host of the SARS coronavirus from animals to humans.\n",
      "4 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "5 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "6 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "7 Zoonotic origins of human coronavirus 2019 (HCoV-19 / SARS-CoV-2): why is this work important? The ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by infection with human coronavirus 2019 (HCoV-19 / SARS-CoV-2 / 2019-nCoV), is a global threat to the human population. Here, we briefly summarize the available data for the zoonotic origins of HCoV-19, with reference to the other two epidemics of highly virulent coronaviruses, SARS-CoV and MERS-CoV, which cause severe pneumonia in humans. We propose to intensify future efforts for tracing the origins of HCoV-19, which is a very important scientific question for the control and prevention of the pandemic.\n",
      "8 Bat-Origin Coronaviruses Expand Their Host Range to Pigs Infections with bat-origin coronaviruses have caused severe illness in humans by ‘host jump’. Recently, novel bat-origin coronaviruses were found in pigs. The large number of mutations on the receptor-binding domain allowed the viruses to infect the new host, posing a potential threat to both agriculture and public health.\n",
      "9 The coronaviruses. Coronaviruses of human origin have emerged as probable important etiologic agents of acute upper respiratory tract illnesses in adults; their role in the etiology of respiratory illnesses of childhood is less certain. The first coronavirus of human origin, B814, was described in 1965 and since that time only 31 additional strains have been recovered. Ten of these were originally recovered in human embryonic tracheal organ culturesonly, and the remainder in monolayer cell cultures. All of the latter group were antigenically related to a prototype strain designated 229e. As a result of the fastidious growth requirements of these agents, most of the knowledge concerning the clinical syndromes associated with them has come from chajlenge studies in volunteers and seroepidemiologic investigations. Coronaviruses have been shown to be associated with a substantial number of adult respiratory illnesses especially during certain periods when rhinovirus infections are infrequent. Progress in propagating coronaviruses and in detecting coronavirus infections has been achieved recently: viruses similar to 229E were successfully recovered in human embryonic intestine cell cultures from patients with upper respiratory tract illnesses; 30C strains (B814, LP and EVS) were recovered in L132 cell cultures from infectious nasal washings; OC43(and OC38) viruses which were originally recovered in OC and later adapted to grow in suckling mice were not only shown to directly hemagglutinate various erythrocytes but were also adapted to grow in MK cell cultures; OC43 virus was also found to induce hemadsorption of rat and mouse erythrocytes in certain cell cultures; and a coronavirus strain was detected by immune electron microscopy. Coronaviruses are also associated with certain diseases in animals.\n",
      "10 Origin of Novel Coronavirus (COVID-19): A Computational Biology Study using Artificial Intelligence Origin of the COVID-19 virus has been intensely debated in the scientific community since the first infected cases were detected in December 2019. The disease has caused a global pandemic, leading to deaths of thousands of people across the world and thus finding origin of this novel coronavirus is important in responding and controlling the pandemic. Recent research results suggest that bats or pangolins might be the original hosts for the virus based on comparative studies using its genomic sequences. This paper investigates the COVID-19 virus origin by using artificial intelligence (AI) and raw genomic sequences of the virus. More than 300 genome sequences of COVID-19 infected cases collected from different countries are explored and analysed using unsupervised clustering methods. The results obtained from various AI-enabled experiments using clustering algorithms demonstrate that all examined COVID-19 virus genomes belong to a cluster that also contains bat and pangolin coronavirus genomes. This provides evidences strongly supporting scientific hypotheses that bats and pangolins are probable hosts for the COVID-19 virus. At the whole genome analysis level, our findings also indicate that bats are more likely the hosts for the COVID-19 virus than pangolins.\n"
     ]
    }
   ],
   "source": [
    "num_results = 10\n",
    "retrieval_results = []\n",
    "with open('covid_ir/covid_ir-queries.txt') as query_file:\n",
    "    for query_num, line in enumerate(query_file):\n",
    "        query = metapy.index.Document()\n",
    "        query.content(line.strip())\n",
    "        results = ranker.score(inv_idx, query, num_results)  \n",
    "        res_list = [(query_num + 1, x[0]) for x in results]\n",
    "        retrieval_results += res_list\n",
    "\n",
    "        \n",
    "        print(\"Query: \", query.content())\n",
    "        print(\"Retrieved Results\")\n",
    "        for num, (d_id, _) in enumerate(results):\n",
    "           content = inv_idx.metadata(d_id).get('content')\n",
    "           print(str(num + 1), content)\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "9mMaxMifWkWT"
   },
   "outputs": [],
   "source": [
    "# You can check your results by comparing the two rankers here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "k6auOVd-YEuG",
    "outputId": "8cd94185-5ca5-4155-c210-7f7469ff3572"
   },
   "source": [
    "## Part 2: Define your own ranking function! (50 points)\n",
    "Implement at least one retrieval function *different* from BM25, Dirichlet Prior, and Pivoted Normalization. You will be graded based on your best performing function. You’ll get full credit if your retrieval function can beat the provided baseline in the dataset. By \"beat,\" we mean that your implemented function and your choice of parameters should reach higher NDCG@10 than the baseline on Kaggle for our dataset, which you can check at any time. Report this information in your submission: the code to implement the retrieval function, the parameter you used that achieved the best performance, and the best performance. In addition,_explain_ what you have explored and why you decide to try those. You will lose points if you cannot explain why your function can reach a higher performance. You can include your explanations in the end of the submitted notebook.\n",
    "\n",
    "*Note:* Simply varying the value of parameters in Okapi/BM25, Dirichlet Prior or Pivoted Normalization does not count as a new retrieval function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "gMdpgK-WT1Mi"
   },
   "outputs": [],
   "source": [
    "# You can define your own retrieval function \n",
    "import math \n",
    "class MyCustomRanker(metapy.index.RankingFunction):                                                                                                                    \n",
    "    def __init__(self, k1 = 1.2, b = 0.75, k3 = 500):                                             \n",
    "        self.k1 = k1\n",
    "        self.b = b\n",
    "        self.k3 = k3\n",
    "        # You *must* invoke the base class __init__() here!\n",
    "        super(MyCustomRanker, self).__init__()                                       \n",
    "                                                                                 \n",
    "    def score_one(self, sd):\n",
    "        \"\"\"\n",
    "        You need to override this function to return a score for a single term.\n",
    "        \n",
    "        You may want to call some of the following variables when implementing your retrieval function:\n",
    "        \n",
    "        sd.avg_dl: average document length of the collection\n",
    "        sd.num_docs: total number of documents in the index\n",
    "        sd.total_terms: total number of terms in the index\n",
    "        sd.query_length: the total length of the current query (sum of all term weights)\n",
    "        sd.query_term_weight: query term count (or weight in case of feedback)\n",
    "        sd.doc_count: number of documents that a term t_id appears in\n",
    "        sd.corpus_term_count: number of times a term t_id appears in the collection\n",
    "        sd.doc_term_count: number of times the term appears in the current document\n",
    "        sd.doc_size: total number of terms in the current document\n",
    "        sd.doc_unique_terms: number of unique terms in the current document\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        k1 = self.k1\n",
    "        b = self.b\n",
    "        k3 = self.k3\n",
    "        \n",
    "        #Fill your answer here\n",
    "        # modified ES \n",
    "        TF = sd.doc_term_count/(sd.doc_term_count + b * math.sqrt(sd.doc_size/sd.avg_dl))\n",
    "        IDF = ((sd.corpus_term_count ** 3) * sd.num_docs / (sd.doc_count**4)) ** k1 \n",
    "        QTF = (k3 + 1)* sd.query_term_weight / (k3 + sd.query_term_weight)\n",
    "        return IDF * TF * QTF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "hz7bh59GT1Ml"
   },
   "outputs": [],
   "source": [
    "ranker = MyCustomRanker()\n",
    "\n",
    "num_results = 10\n",
    "custom_ranking_retrieval_results = []\n",
    "query_id_list = []\n",
    "with open('covid_ir/covid_ir-queries.txt') as query_file:\n",
    "    for query_num, line in enumerate(query_file):\n",
    "        # print(type(query_num))\n",
    "        # query_id_list.append(query_num)\n",
    "        query = metapy.index.Document()\n",
    "        query.content(line.strip())\n",
    "        results = ranker.score(inv_idx, query, num_results)  \n",
    "        res_list = [(query_num + 1, x[0]) for x in results]\n",
    "        custom_ranking_retrieval_results += res_list\n",
    "\n",
    "        \n",
    "        # print(\"Query: \", query.content())\n",
    "        # print(\"Retrieved Results\")\n",
    "        # for num, (d_id, _) in enumerate(results):\n",
    "        #   content = inv_idx.metadata(d_id).get('content')\n",
    "        #   print(str(num + 1), content)\n",
    "        # break     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z0HJLzwlT1Mo"
   },
   "source": [
    "### Write your ranking to a file and upload it to the [Kaggle competition](https://www.kaggle.com/t/a8345852fdab42da9e210f833b9f50b1) for the class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "iidS_eU-T1Mp"
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "with open(\"my_kaggle_submission.csv\",\"w\") as f:\n",
    "    csv_writer = csv.writer(f)\n",
    "    csv_writer.writerow([\"queryid\", \"docid\"])\n",
    "    for x in custom_ranking_retrieval_results:\n",
    "        csv_writer.writerow(list(x))\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l36tG2VZ61Fm"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "SI650 Homework 2 - Covid Ranking.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}