B. Getting started with NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b0ad4785",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 1. Goals of this Unit\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "The goal of this unit is to introduce the field of natural language processing and provide an overview of common applications, techniques, and challenges.\n",
    "\n",
    "<br/>After this unit, you will be able to:\n",
    "- Understand what natural language processing is.\n",
    "- Gain an introduction to common applications and challenges within natural language processing.\n",
    "- Identify several natural language processing techniques and how they relate to each other.\n",
    "- Try out a few natural language processing techniques using Python."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d84634",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 2. Intro to NLP\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "Look at the technologies around us:\n",
    "- Spellcheck and autocorrect\n",
    "- Auto-generated video captions\n",
    "- Virtual assistants like Amazon’s Alexa\n",
    "- Autocomplete\n",
    "- Your news site’s suggested articles\n",
    "\n",
    "<br/>What do they have in common?\n",
    "\n",
    "<br/>All of these handy technologies exist because of *natural language processing!* Also known as NLP, the field is at the intersection of linguistics, artificial intelligence, and computer science. The goal? Enabling computers to interpret, analyze, and approximate the generation of human languages (like English or Spanish).\n",
    "\n",
    "<br/>NLP got its start around 1950 with Alan Turing’s test for artificial intelligence evaluating whether a computer can use language to fool humans into believing it’s human.\n",
    "\n",
    "<br/>But approximating human speech is only one of a wide range of applications for NLP! Applications from detecting spam emails or bias in tweets to improving accessibility for people with disabilities all rely heavily on natural language processing techniques.\n",
    "\n",
    "<br/>NLP can be conducted in several programming languages. However, Python has some of the most extensive open-source NLP libraries, including the Natural Language Toolkit or *NLTK.* Because of this, you’ll be using Python to get your first taste of NLP.\n",
    "<img src=\"Images/Natural_Language_Processing_Overview.webp\" width=\"30%\" height=\"30%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1728bc94",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 3. Text Preprocessing\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "> \"You never know what you have... until you clean your data.\"\n",
    "\n",
    "Cleaning and preparation are crucial for many tasks, and NLP is no exception. *Text preprocessing* is usually the first step you’ll take when faced with an NLP task.\n",
    "\n",
    "<br/>Without preprocessing, your computer interprets `\"the\"`, `\"The\"`, and `\"<p>The\"` as entirely different words. There is a LOT you can do here, depending on the formatting you need. Lucky for you, Regex and NLTK will do most of it for you! Common tasks include:\n",
    "\n",
    "<br/>**Noise removal** — stripping text of formatting (e.g., HTML tags).\n",
    "\n",
    "<br/>**Tokenization** — breaking text into individual words.\n",
    "\n",
    "<br/>**Normalization** — cleaning text data in any other way:\n",
    "- *Stemming* is a blunt axe to chop off word prefixes and suffixes. “booing” and “booed” become “boo”, but “computer” may become “comput” and “are” would remain “are.”\n",
    "- *Lemmatization* is a scalpel to bring words down to their root forms. For example, NLTK’s savvy lemmatizer knows “am” and “are” are related to “be.”\n",
    "- Other common tasks include lowercasing, stopwords removal, spelling correction, etc.\n",
    "\n",
    "<br/>*Exercise:*\n",
    "1. We used NLTK’s PorterStemmer to normalize the text — see example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "498a9f10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stemmed text:\n",
      "['so', 'mani', 'squid', 'are', 'jump', 'out', 'of', 'suitcas', 'these', 'day', 'that', 'you', 'can', 'bare', 'go', 'anywher', 'without', 'see', 'one', 'burst', 'forth', 'from', 'a', 'tightli', 'pack', 'valis', 'i', 'went', 'to', 'the', 'dentist', 'the', 'other', 'day', 'and', 'sure', 'enough', 'i', 'saw', 'an', 'angri', 'one', 'jump', 'out', 'of', 'my', 'dentist', 's', 'bag', 'within', 'minut', 'of', 'arriv', 'she', 'hardli', 'even', 'notic']\n",
      "\n",
      "Lemmatized text:\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# regex for removing punctuation!\n",
    "import re\n",
    "# nltk preprocessing magic\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "# grabbing a part of speech function:\n",
    "#from part_of_speech import get_part_of_speech\n",
    "\n",
    "text = \"So many squids are jumping out of suitcases these days that you can barely go anywhere without seeing one burst forth from a tightly packed valise. I went to the dentist the other day, and sure enough I saw an angry one jump out of my dentist's bag within minutes of arriving. She hardly even noticed.\"\n",
    "\n",
    "cleaned = re.sub('\\W+', ' ', text)\n",
    "tokenized = word_tokenize(cleaned)\n",
    "\n",
    "stemmer = PorterStemmer()\n",
    "stemmed = [stemmer.stem(token) for token in tokenized]\n",
    "\n",
    "## -- CHANGE these -- ##\n",
    "lemmatizer = None\n",
    "lemmatized = []\n",
    "\n",
    "print(\"Stemmed text:\")\n",
    "print(stemmed)\n",
    "print(\"\\nLemmatized text:\")\n",
    "print(lemmatized)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59001670",
   "metadata": {},
   "source": [
    "2. In the output terminal you’ll see our program counts `\"go\"` and `\"went\"` as different words! Also, what’s up with `\"mani\"` and `\"hardli\"`? A lemmatizer will fix this. Let’s do it. Where `lemmatizer` is defined, replace `None` with `WordNetLemmatizer()`. Where we defined `lemmatized`, replace the empty list with a list comprehension that uses `lemmatizer` to `lemmatize()` each `token` in `tokenized`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5e4fcb2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stemmed text:\n",
      "['so', 'mani', 'squid', 'are', 'jump', 'out', 'of', 'suitcas', 'these', 'day', 'that', 'you', 'can', 'bare', 'go', 'anywher', 'without', 'see', 'one', 'burst', 'forth', 'from', 'a', 'tightli', 'pack', 'valis', 'i', 'went', 'to', 'the', 'dentist', 'the', 'other', 'day', 'and', 'sure', 'enough', 'i', 'saw', 'an', 'angri', 'one', 'jump', 'out', 'of', 'my', 'dentist', 's', 'bag', 'within', 'minut', 'of', 'arriv', 'she', 'hardli', 'even', 'notic']\n",
      "\n",
      "Lemmatized text:\n",
      "['So', 'many', 'squid', 'are', 'jumping', 'out', 'of', 'suitcase', 'these', 'day', 'that', 'you', 'can', 'barely', 'go', 'anywhere', 'without', 'seeing', 'one', 'burst', 'forth', 'from', 'a', 'tightly', 'packed', 'valise', 'I', 'went', 'to', 'the', 'dentist', 'the', 'other', 'day', 'and', 'sure', 'enough', 'I', 'saw', 'an', 'angry', 'one', 'jump', 'out', 'of', 'my', 'dentist', 's', 'bag', 'within', 'minute', 'of', 'arriving', 'She', 'hardly', 'even', 'noticed']\n"
     ]
    }
   ],
   "source": [
    "lemmatizer = WordNetLemmatizer()\n",
    "lemmatized = [lemmatizer.lemmatize(token) for token in tokenized]\n",
    "\n",
    "print(\"Stemmed text:\")\n",
    "print(stemmed)\n",
    "print(\"\\nLemmatized text:\")\n",
    "print(lemmatized)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88eadd15",
   "metadata": {},
   "source": [
    "3. Why are the lemmatized verbs like `\"went\"` still conjugated? By default `lemmatize()` treats every word as a noun. Give `lemmatize()` a second argument: `get_part_of_speech(token)`. This will tell our lemmatizer what part of speech the word is. Run your code again to see the result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4765eea0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stemmed text:\n",
      "['so', 'mani', 'squid', 'are', 'jump', 'out', 'of', 'suitcas', 'these', 'day', 'that', 'you', 'can', 'bare', 'go', 'anywher', 'without', 'see', 'one', 'burst', 'forth', 'from', 'a', 'tightli', 'pack', 'valis', 'i', 'went', 'to', 'the', 'dentist', 'the', 'other', 'day', 'and', 'sure', 'enough', 'i', 'saw', 'an', 'angri', 'one', 'jump', 'out', 'of', 'my', 'dentist', 's', 'bag', 'within', 'minut', 'of', 'arriv', 'she', 'hardli', 'even', 'notic']\n",
      "\n",
      "Lemmatized text:\n",
      "['So', 'many', 'squid', 'be', 'jumping', 'out', 'of', 'suitcase', 'these', 'day', 'that', 'you', 'can', 'barely', 'go', 'anywhere', 'without', 'see', 'one', 'burst', 'forth', 'from', 'a', 'tightly', 'packed', 'valise', 'I', 'go', 'to', 'the', 'dentist', 'the', 'other', 'day', 'and', 'sure', 'enough', 'I', 'saw', 'an', 'angry', 'one', 'jump', 'out', 'of', 'my', 'dentist', 's', 'bag', 'within', 'minute', 'of', 'arrive', 'She', 'hardly', 'even', 'notice']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tag import pos_tag\n",
    "from nltk.corpus import wordnet\n",
    "\n",
    "def get_part_of_speech(word):\n",
    "    \"\"\"Map POS tag to first character lemmatize() accepts\"\"\"\n",
    "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
    "    tag_dict = {\"J\": wordnet.ADJ, \"N\": wordnet.NOUN, \"V\": wordnet.VERB, \"R\": wordnet.ADV}\n",
    "    return tag_dict.get(tag, wordnet.NOUN)\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "lemmatized = [lemmatizer.lemmatize(token, get_part_of_speech(token)) for token in tokenized]\n",
    "\n",
    "print(\"Stemmed text:\")\n",
    "print(stemmed)\n",
    "print(\"\\nLemmatized text:\")\n",
    "print(lemmatized)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc527e3",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 4. Parsing Text\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "You now have a preprocessed, clean list of words. Now what? It may be helpful to know how the words relate to each other and the underlying syntax (grammar). Parsing is an NLP process concerned with segmenting text based on syntax.\n",
    "\n",
    "<br/>You probably do not want to be doing any parsing by hand and NLTK has a few tricks up its sleeve to help you out:\n",
    "\n",
    "<br/>*Part-of-speech tagging (POS tagging)* identifies parts of speech (verbs, nouns, adjectives, etc.). NLTK can do it faster (and maybe more accurately) than your grammar teacher.\n",
    "\n",
    "<br/>*Named entity recognition (NER)* helps identify the proper nouns (e.g., “Natalia” or “Berlin”) in a text. This can be a clue as to the topic of the text and NLTK captures many for you.\n",
    "\n",
    "<br/>*Dependency grammar trees* help you understand the relationship between the words in a sentence. It can be a tedious task for a human, so the Python library spaCy is at your service, even if it isn’t always perfect.\n",
    "\n",
    "<br/>In English we leave a lot of ambiguity, so syntax can be tough, even for a computer program. Take a look at the following sentence:\n",
    "<img src=\"Images/parsing_syntactic_ambiguity.gif\" width=\"30%\" height=\"30%\">\n",
    "Do I have the binoculars? Does the cow have binoculars? Does the tree have binoculars?\n",
    "\n",
    "<br/>*Regex parsing,* using Python’s `re` library, allows for a bit more nuance. When coupled with POS tagging, you can identify specific phrase chunks. On its own, it can find you addresses, emails, and many other common patterns within large chunks of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ac11296e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        saw                     \n",
      "  _______|___                    \n",
      " |          cow                 \n",
      " |    _______|____               \n",
      " |   |          under           \n",
      " |   |            |              \n",
      " |   |           tree           \n",
      " |   |        ____|_______       \n",
      " |   |       |           with   \n",
      " |   |       |            |      \n",
      " I   a       a        binoculars\n",
      "\n",
      "    jumped              \n",
      "  ____|_________         \n",
      " |    |        over     \n",
      " |    |         |        \n",
      " |   fox       dog      \n",
      " |    |      ___|____    \n",
      " !   The   the      lazy\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from nltk import Tree\n",
    "dependency_parser = spacy.load('en_core_web_sm')\n",
    "parsed_squids = dependency_parser(\"I saw a cow under a tree with binoculars\")\n",
    "\n",
    "# Assign a new value:\n",
    "my_parsed_sentence = dependency_parser(\"The fox jumped over the lazy dog!\")\n",
    "\n",
    "def to_nltk_tree(node):\n",
    "    if node.n_lefts + node.n_rights > 0:\n",
    "        parsed_child_nodes = [to_nltk_tree(child) for child in node.children]\n",
    "        return Tree(node.orth_, parsed_child_nodes)\n",
    "    else: return node.orth_\n",
    "\n",
    "for sent in parsed_squids.sents: to_nltk_tree(sent.root).pretty_print()\n",
    "for sent in my_parsed_sentence.sents: to_nltk_tree(sent.root).pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "678a0359",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 5. Language Models: Bag-of-Words\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "How can we help a machine make sense of a bunch of word tokens? We can help computers make predictions about language by training a language model on a corpus (a bunch of example text).\n",
    "\n",
    "<br/>*Language models* are probabilistic computer models of language. We build and use these models to figure out the likelihood that a given sound, letter, word, or phrase will be used. Once a model has been trained, it can be tested out on new texts.\n",
    "\n",
    "<br/>One of the most common language models is the unigram model, a statistical language model commonly known as *bag-of-words*. As its name suggests, bag-of-words does not have much order to its chaos! What it does have is a tally count of each instance for each word. Consider the following text example:\n",
    "<img src=\"Images/bag-of-words.gif\" width=\"30%\" height=\"30%\">\n",
    "Provided some initial preprocessing, bag-of-words would result in a mapping like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "938bf7ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 2, 'squid': 1, 'jump': 1, 'out': 1, 'of': 1, 'suitcase': 1}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{\"the\": 2, \"squid\": 1, \"jump\": 1, \"out\": 1, \"of\": 1, \"suitcase\": 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de2689df",
   "metadata": {},
   "source": [
    "Now look at this sentence and mapping: “Why are your suitcases full of jumping squids?”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8feaad97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'why': 1,\n",
       " 'be': 1,\n",
       " 'your': 1,\n",
       " 'suitcase': 1,\n",
       " 'full': 1,\n",
       " 'of': 1,\n",
       " 'jump': 1,\n",
       " 'squid': 1}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{\"why\": 1, \"be\": 1, \"your\": 1, \"suitcase\": 1, \"full\": 1, \"of\": 1, \"jump\": 1, \"squid\": 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d89ec4e9",
   "metadata": {},
   "source": [
    "You can see how even with different word order and sentence structures, “jump,” “squid,” and “suitcase” are shared topics between the two examples. Bag-of-words can be an excellent way of looking at language when you want to make predictions concerning topic or sentiment of a text. When grammar and word order are irrelevant, this is probably a good model to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "19042c86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'never': 1, 'reach': 1, 'destination': 1, 'stop': 1, 'throw': 1, 'stone': 1, 'every': 1, 'dog': 1, 'bark': 1})\n"
     ]
    }
   ],
   "source": [
    "# importing regex and nltk\n",
    "import re, nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "# importing Counter to get word counts for bag of words\n",
    "from collections import Counter\n",
    "\n",
    "# Change text to another string:\n",
    "text = \"You will never reach your destination if you stop and throw stones at every dog that barks.\"\n",
    "\n",
    "cleaned = re.sub('\\W+', ' ', text).lower()\n",
    "tokenized = word_tokenize(cleaned)\n",
    "\n",
    "stop_words = stopwords.words('english')\n",
    "filtered = [word for word in tokenized if word not in stop_words]\n",
    "\n",
    "normalizer = WordNetLemmatizer()\n",
    "normalized = [normalizer.lemmatize(token, get_part_of_speech(token)) for token in filtered]\n",
    "# Comment out the print statement below\n",
    "# print(normalized)\n",
    "\n",
    "# Define bag_of_looking_glass_words & print:\n",
    "bag_of_looking_glass_words = Counter(normalized)\n",
    "print(bag_of_looking_glass_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e62537e",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 6. Language Models: N-Gram and NLM\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "For parsing entire phrases or conducting language prediction, you will want to use a model that pays attention to each word’s neighbors. Unlike bag-of-words, the *n-gram* model considers a sequence of some number (*n*) units and calculates the probability of each unit in a body of language given the preceding sequence of length *n*. Because of this, n-gram probabilities with larger *n* values can be impressive at language prediction.\n",
    "\n",
    "<br/>Take a look at our revised squid example: “The squids jumped out of the suitcases. The squids were furious.”\n",
    "\n",
    "<br/>A bigram model (where *n* is 2) might give us the following count frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e6299110",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('', 'the'): 2,\n",
       " ('the', 'squids'): 2,\n",
       " ('squids', 'jumped'): 1,\n",
       " ('jumped', 'out'): 1,\n",
       " ('out', 'of'): 1,\n",
       " ('of', 'the'): 1,\n",
       " ('the', 'suitcases'): 1,\n",
       " ('suitcases', ''): 1,\n",
       " ('squids', 'were'): 1,\n",
       " ('were', 'furious'): 1,\n",
       " ('furious', ''): 1}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{('', 'the'): 2, ('the', 'squids'): 2, ('squids', 'jumped'): 1, ('jumped', 'out'): 1, ('out', 'of'): 1, ('of', 'the'): 1, ('the', 'suitcases'): 1, ('suitcases', ''): 1, ('squids', 'were'): 1, ('were', 'furious'): 1, ('furious', ''): 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "655c66c9",
   "metadata": {},
   "source": [
    "There are a couple problems with the n gram model:\n",
    "1. How can your language model make sense of the sentence “The cat fell asleep in the mailbox” if it’s never seen the word “mailbox” before? During training, your model will probably come across test words that it has never encountered before (this issue also pertains to bag of words). A tactic known as *language smoothing* can help adjust probabilities for unknown words, but it isn’t always ideal.\n",
    "2. For a model that more accurately predicts human language patterns, you want n (your sequence length) to be as large as possible. That way, you will have more natural sounding language, right? Well, as the sequence length grows, the number of examples of each sequence within your training corpus shrinks. With too few examples, you won’t have enough data to make many predictions.\n",
    "\n",
    "<br/>Enter *neural language models (NLMs)*! Much recent work within NLP has involved developing and training neural networks to approximate the approach our human brains take towards language. This deep learning approach allows computers a much more adaptive tack to processing human language. Common NLMs include LSTMs and transformer models.\n",
    "\n",
    "<br/>*Exercise:*\n",
    "1. If you run the code below, you’ll see the 10 most commonly used words in one of Churchill's quotes, parsed with NLTK’s `ngrams` module — if you’re thinking this looks like a bag of words, that’s because it is one!\n",
    "2. At what `n` are you just getting lines from poems repeated in the text? This is where there may be too few examples of each sequence within your training corpus to make any helpful predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fa799b24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking Glass Bigrams:\n",
      "[(('you', 'will'), 1), (('will', 'never'), 1), (('never', 'reach'), 1), (('reach', 'your'), 1), (('your', 'destination'), 1), (('destination', 'if'), 1), (('if', 'you'), 1), (('you', 'stop'), 1), (('stop', 'and'), 1), (('and', 'throw'), 1)]\n",
      "\n",
      "Looking Glass Trigrams:\n",
      "[(('you', 'will', 'never'), 1), (('will', 'never', 'reach'), 1), (('never', 'reach', 'your'), 1), (('reach', 'your', 'destination'), 1), (('your', 'destination', 'if'), 1), (('destination', 'if', 'you'), 1), (('if', 'you', 'stop'), 1), (('you', 'stop', 'and'), 1), (('stop', 'and', 'throw'), 1), (('and', 'throw', 'stones'), 1)]\n",
      "\n",
      "Looking Glass n-grams:\n",
      "[(('you', 'will', 'never', 'reach'), 1), (('will', 'never', 'reach', 'your'), 1), (('never', 'reach', 'your', 'destination'), 1), (('reach', 'your', 'destination', 'if'), 1), (('your', 'destination', 'if', 'you'), 1), (('destination', 'if', 'you', 'stop'), 1), (('if', 'you', 'stop', 'and'), 1), (('you', 'stop', 'and', 'throw'), 1), (('stop', 'and', 'throw', 'stones'), 1), (('and', 'throw', 'stones', 'at'), 1)]\n"
     ]
    }
   ],
   "source": [
    "import nltk, re\n",
    "from nltk.tokenize import word_tokenize\n",
    "# importing ngrams module from nltk\n",
    "from nltk.util import ngrams\n",
    "from collections import Counter\n",
    "\n",
    "cleaned = re.sub('\\W+', ' ', \"You will never reach your destination if you stop and throw stones at every dog that barks.\").lower()\n",
    "tokenized = word_tokenize(cleaned)\n",
    "\n",
    "# Change the n value to 2:\n",
    "looking_glass_bigrams = ngrams(tokenized, 2)\n",
    "looking_glass_bigrams_frequency = Counter(looking_glass_bigrams)\n",
    "\n",
    "# Change the n value to 3:\n",
    "looking_glass_trigrams = ngrams(tokenized, 3)\n",
    "looking_glass_trigrams_frequency = Counter(looking_glass_trigrams)\n",
    "\n",
    "# Change the n value to a number greater than 3:\n",
    "looking_glass_ngrams = ngrams(tokenized, 4)\n",
    "looking_glass_ngrams_frequency = Counter(looking_glass_ngrams)\n",
    "\n",
    "print(\"Looking Glass Bigrams:\")\n",
    "print(looking_glass_bigrams_frequency.most_common(10))\n",
    "\n",
    "print(\"\\nLooking Glass Trigrams:\")\n",
    "print(looking_glass_trigrams_frequency.most_common(10))\n",
    "\n",
    "print(\"\\nLooking Glass n-grams:\")\n",
    "print(looking_glass_ngrams_frequency.most_common(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66432fe6",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 7. Topic Models\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "We’ve touched on the idea of finding topics within a body of language. But what if the text is long and the topics aren’t obvious?\n",
    "\n",
    "<br/>*Topic modeling* is an area of NLP dedicated to uncovering latent, or hidden, topics within a body of language.\n",
    "\n",
    "<br/>A common technique is to deprioritize the most common words and prioritize less frequently used terms as topics in a process known as *term frequency-inverse document frequency (tf-idf)*. Say what?! This may sound counter-intuitive at first. Why would you want to give more priority to less-used words? Well, when you’re working with a lot of text, it makes a bit of sense if you don’t want your topics filled with words like “the” and “is.” The Python libraries `gensim` and `sklearn` have modules to handle tf-idf.\n",
    "\n",
    "<br/>Whether you use your plain bag of words (which will give you term frequency) or run it through tf-idf, the next step in your topic modeling journey is often *latent Dirichlet allocation (LDA)*. LDA is a statistical model that takes your documents and determines which words keep popping up together in the same contexts (i.e., documents). We’ll use sklearn to tackle this for us.\n",
    "\n",
    "<br/>If you have any interest in visualizing your newly minted topics, *word2vec* is a great technique to have up your sleeve. word2vec can map out your topic model results spatially as vectors so that similarly used words are closer together. In the case of a language sample consisting of “The squids jumped out of the suitcases. The squids were furious. Why are your suitcases full of jumping squids?”, we might see that “suitcase”, “jump”, and “squid” were words used within similar contexts. This word-to-vector mapping is known as a *word embedding*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbada46e",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 8. Text Similarity\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "Most of us have a good autocorrect story. Our phone’s messenger quietly swaps one letter for another as we type and suddenly the meaning of our message has changed (to our horror or pleasure). However, addressing *text similarity* — including spelling correction — is a major challenge within natural language processing.\n",
    "\n",
    "<br/>Addressing word similarity and misspelling for spellcheck or autocorrect often involves considering the *Levenshtein distance* or minimal edit distance between two words. The distance is calculated through the minimum number of insertions, deletions, and substitutions that would need to occur for one word to become another. For example, turning “bees” into “beans” would require one substitution (“a” for “e”) and one insertion (“n”), so the Levenshtein distance would be two.\n",
    "\n",
    "<br/>*Phonetic similarity* is also a major challenge within speech recognition. English-speaking humans can easily tell from context whether someone said “euthanasia” or “youth in Asia,” but it’s a far more challenging task for a machine! More advanced autocorrect and spelling correction technology additionally considers key distance on a keyboard and phonetic similarity (how much two words or phrases sound the same).\n",
    "\n",
    "<br/>It’s also helpful to find out if texts are the same to guard against plagiarism, which we can identify through *lexical similarity* (the degree to which texts use the same vocabulary and phrases). Meanwhile, *semantic similarity* (the degree to which documents contain similar meaning or topics) is useful when you want to find (or recommend) an article or book similar to one you recently finished.\n",
    "\n",
    "<br/>*Exercise:*\n",
    "<br/>Assign the variable `three_away_from_code` a word with a Levenshtein distance of 3 from “code”. Assign `two_away_from_chunk` a word with a Levenshtein distance of 2 from “chunk”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "41ea8230",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Levenshtein distance from 'fart' to 'target' is 3!\n",
      "The Levenshtein distance from 'code' to 'babe' is 3!\n",
      "The Levenshtein distance from 'chunk' to 'dunk' is 2!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "# NLTK has a built-in function\n",
    "# to check Levenshtein distance:\n",
    "from nltk.metrics import edit_distance\n",
    "\n",
    "def print_levenshtein(string1, string2):\n",
    "    print(f\"The Levenshtein distance from '{string1}' to '{string2}' is {edit_distance(string1, string2)}!\")\n",
    "\n",
    "# Check the distance between\n",
    "# any two words here!\n",
    "print_levenshtein(\"fart\", \"target\")\n",
    "\n",
    "# Assign passing strings here:\n",
    "three_away_from_code = \"babe\"\n",
    "\n",
    "two_away_from_chunk = \"dunk\"\n",
    "\n",
    "print_levenshtein(\"code\", three_away_from_code)\n",
    "print_levenshtein(\"chunk\", two_away_from_chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea4d5fd5",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 9. Language Prediction & Text Generation\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "How does your favorite search engine complete your search queries? How does your phone’s keyboard know what you want to type next? Language prediction is an application of NLP concerned with predicting text given preceding text. Autosuggest, autocomplete, and suggested replies are common forms of language prediction.\n",
    "\n",
    "<br/>Your first step to language prediction is picking a language model. Bag of words alone is generally not a great model for language prediction; no matter what the preceding word was, you will just get one of the most commonly used words from your training corpus.\n",
    "\n",
    "<br/>If you go the n-gram route, you will most likely rely on *Markov chains* to predict the statistical likelihood of each following word (or character) based on the training corpus. Markov chains are memory-less and make statistical predictions based entirely on the current n-gram on hand.\n",
    "\n",
    "<br/>For example, let’s take a sentence beginning, “I ate so many grilled cheese”. Using a trigram model (where *n* is 3), a Markov chain would predict the following word as “sandwiches” based on the number of times the sequence “grilled cheese sandwiches” has appeared in the training data out of all the times “grilled cheese” has appeared in the training data.\n",
    "\n",
    "<br/>A more advanced approach, using a neural language model, is the *Long Short Term Memory (LSTM) model*. LSTM uses deep learning with a network of artificial “cells” that manage memory, making them better suited for text prediction than traditional neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f327bc",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 10. Advanced NLP Topics\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "Believe it or not, you’ve just scratched the surface of natural language processing. There are a slew of advanced topics and applications of NLP, many of which rely on deep learning and neural networks.\n",
    "- *Naive Bayes classifiers* are supervised machine learning algorithms that leverage a probabilistic theorem to make predictions and classifications. They are widely used for sentiment analysis (determining whether a given block of language expresses negative or positive feelings) and spam filtering.\n",
    "- We’ve made enormous gains in *machine translation,* but even the most advanced translation software using neural networks and LSTM still has far to go in accurately translating between languages.\n",
    "- Some of the most life-altering applications of NLP are focused on improving *language accessibility* for people with disabilities. Text-to-speech functionality and speech recognition have improved rapidly thanks to neural language models, making digital spaces far more accessible places.\n",
    "- NLP can also be used to detect bias in writing and speech. Feel like a political candidate, book, or news source is biased but can’t put your finger on exactly how? Natural language processing can help you identify the language at issue."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75758468",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 11. Challenges and Considerations\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "As you’ve seen, there are a vast array of applications for NLP. However, as they say, “with great language processing comes great responsibility” (or something along those lines). When working with NLP, we have several important considerations to take into account:\n",
    "- Different NLP tasks may be more or less difficult in different languages. Because so many NLP tools are built by and for English speakers, these tools may lag behind in processing other languages. The tools may also be programmed with cultural and linguistic biases specific to English speakers.\n",
    "- What if your Amazon Alexa could only understand wealthy men from coastal areas of the United States? English itself is not a homogeneous body. English varies by person, by dialect, and by many sociolinguistic factors. When we build and train NLP tools, are we only building them for one type of English speaker?\n",
    "- You can have the best intentions and still inadvertently program a bigoted tool. While NLP can limit bias, it can also propagate bias. As an NLP developer, it’s important to consider biases, both within your code and within the training corpus. A machine will learn the same biases you teach it, whether intentionally or unintentionally.\n",
    "- As you become someone who builds tools with natural language processing, it’s vital to take into account your users’ privacy. There are many powerful NLP tools that come head-to-head with privacy concerns. Who is collecting your data? How much data is being collected and what do those companies plan to do with your data?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f371a949",
   "metadata": {},
   "source": [
    "<img src=\"Images/atom.png\" alt=\"Atom\" style=\"width:60px\" align=\"left\" vertical-align=\"middle\">\n",
    "\n",
    "## 12. NLP Review\n",
    "*Natural Language Processing*\n",
    "\n",
    "----\n",
    "Check out how much you’ve learned about natural language processing!\n",
    "- Natural language processing combines computer science, linguistics, and artificial intelligence to enable computers to process human languages.\n",
    "- NLTK is a Python library used for NLP.\n",
    "- Text preprocessing is a stage of NLP focused on cleaning and preparing text for other NLP tasks.\n",
    "- Parsing is an NLP technique concerned with breaking up text based on syntax.\n",
    "- Language models are probabilistic machine models of language use for NLP comprehension tasks. Common models include bag-of-words, n-gram models, and neural language modeling.\n",
    "- Topic modeling is the NLP process by which hidden topics are identified given a body of text.\n",
    "- Text similarity is a facet of NLP concerned with semblance between instances of language.\n",
    "- Language prediction is an application of NLP concerned with predicting language given preceding language.\n",
    "- There are many social and ethical considerations to take into account when designing NLP tools.\n",
    "\n",
    "<br/>You can build a lot of fun tools with NLP knowledge and a bit of Python. This is just the beginning.\n",
    "\n",
    "<br/>Feel free to test out the plagiarism classifier we built below (does it work?) or use the space to play around with other NLP code you’ve encountered in this lesson!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "99da99d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "# Levenshtein distance:\n",
    "from nltk.metrics import edit_distance\n",
    "\n",
    "# an arbitrary plagiarism classifier:\n",
    "def is_plagiarized(text1, text2):\n",
    "    n = 7\n",
    "    if edit_distance(text1.lower(), text2.lower()) > ((len(text1) + len(text2)) / n): return False\n",
    "    return True\n",
    "\n",
    "doc1 = \"is this plagiarized\"\n",
    "doc2 = \"maybe it's plagiarized\"\n",
    "\n",
    "print(is_plagiarized(doc1, doc2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}