2_data_representation_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Representation in Natural Language Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"NLP-image.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We all know that computer understand binary language in the form of 0's and 1's. \n",
    "* It is impossible to make them understand words naturally. \n",
    "* But encoding such words into numeric form can solve our problem.\n",
    "* The process of converting textual information into numbers is called Vectorization. \n",
    "* It is also termed as feature extraction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (1) Bag of Words "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* simplest approach to convert text into numbers\n",
    "* Why this name: This model is only concerned with with the occurrence of the word and not where it is placed (i.e. order) in bag.\n",
    "* The intuition behind such approach is that similar documents contain similar words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"bag1.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"bag2.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_vec(text):\n",
    "    vectorizer = CountVectorizer()\n",
    "    vocabulary=vectorizer.fit(text)\n",
    "    doc_term_matrix= vectorizer.transform(text)\n",
    "    final=doc_term_matrix.toarray()\n",
    "    return final\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1, 1, 1, 1, 1, 2]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vec([\"The quick brown fox jumped over the lazy dog.\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " If value error occurs: ValueError: Iterable over raw text documents expected, string object received.\n",
    "* This means The solution to this problem is because input is just a String, \n",
    "  but what is needed is a list (or an iterable) containing a single element (which is nothing but the String itself).\n",
    "\n",
    "Input is a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2.) Tf-idf Vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"tf1.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"idf.jpeg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tf_idf_vec(text):\n",
    "    vectorizer = TfidfVectorizer()\n",
    "    vocabulary=vectorizer.fit(text)\n",
    "    doc_term_matrix= vectorizer.transform(text)\n",
    "    final=doc_term_matrix.toarray()\n",
    "    return final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.42471719, 0.30218978, 0.        , 0.30218978, 0.30218978,\n",
       "        0.42471719, 0.60437955, 0.        ],\n",
       "       [0.        , 0.30218978, 0.42471719, 0.30218978, 0.30218978,\n",
       "        0.        , 0.60437955, 0.42471719]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_idf_vec([\"The car is driven on the road.\",\"the truck is driven on the highway.\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Size of the matrix: no. of documents* unique words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.freecodecamp.org/news/how-to-process-textual-data-using-tf-idf-in-python-cd2bbc0a94a3/\n",
    "# https://datascience.stackexchange.com/questions/22250/what-is-the-difference-between-a-hashing-vectorizer-and-a-tfidf-vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (3)Hashing Vectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hash_vec(text):\n",
    "    vectorizer = HashingVectorizer()\n",
    "    vocabulary=vectorizer.fit(text)\n",
    "    doc_term_matrix= vectorizer.transform(text)\n",
    "    final=doc_term_matrix.toarray()\n",
    "    return final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hash_vec([\"The car is driven on the road.\",\"The truck is driven on the highway.\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (1) Implementation of Word2Vec via Gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package abc to /home/shivangi/nltk_data...\n",
      "[nltk_data]   Package abc is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import abc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['PM', 'denies', 'knowledge', 'of', 'AWB', 'kickbacks', 'The', 'Prime', 'Minister', 'has', 'denied', 'he', 'knew', 'AWB', 'was', 'paying', 'kickbacks', 'to', 'Iraq', 'despite', 'writing', 'to', 'the', 'wheat', 'exporter', 'asking', 'to', 'be', 'kept', 'fully', 'informed', 'on', 'Iraq', 'wheat', 'sales', '.'], ['Letters', 'from', 'John', 'Howard', 'and', 'Deputy', 'Prime', 'Minister', 'Mark', 'Vaile', 'to', 'AWB', 'have', 'been', 'released', 'by', 'the', 'Cole', 'inquiry', 'into', 'the', 'oil', 'for', 'food', 'program', '.'], ...]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# splitting sentences into tokens, Word2Vec Model takes list of lists as input where each sublist contains\n",
    "# tokens for a sentence.\n",
    "abc.sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PM', 'denies', 'knowledge', 'of', 'AWB', 'kickbacks', ...]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Function to find different word in corpora 'abc'\n",
    "nltk.corpus.abc.words()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading model or training the model\n",
    "model= gensim.models.Word2Vec(abc.sents())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PM',\n",
       " 'denies',\n",
       " 'knowledge',\n",
       " 'of',\n",
       " 'AWB',\n",
       " 'kickbacks',\n",
       " 'The',\n",
       " 'Prime',\n",
       " 'Minister',\n",
       " 'has',\n",
       " 'denied',\n",
       " 'he',\n",
       " 'knew',\n",
       " 'was',\n",
       " 'paying',\n",
       " 'to',\n",
       " 'Iraq',\n",
       " 'despite',\n",
       " 'writing',\n",
       " 'the',\n",
       " 'wheat',\n",
       " 'exporter',\n",
       " 'asking',\n",
       " 'be',\n",
       " 'kept',\n",
       " 'fully',\n",
       " 'informed',\n",
       " 'on',\n",
       " 'sales',\n",
       " '.',\n",
       " 'Letters',\n",
       " 'from',\n",
       " 'John',\n",
       " 'Howard',\n",
       " 'and',\n",
       " 'Deputy',\n",
       " 'Mark',\n",
       " 'Vaile',\n",
       " 'have',\n",
       " 'been',\n",
       " 'released',\n",
       " 'by',\n",
       " 'Cole',\n",
       " 'inquiry',\n",
       " 'into',\n",
       " 'oil',\n",
       " 'for',\n",
       " 'food',\n",
       " 'program',\n",
       " 'In',\n",
       " 'one',\n",
       " 'letters',\n",
       " 'Mr',\n",
       " 'asks',\n",
       " 'managing',\n",
       " 'director',\n",
       " 'Andrew',\n",
       " 'Lindberg',\n",
       " 'remain',\n",
       " 'in',\n",
       " 'close',\n",
       " 'contact',\n",
       " 'with',\n",
       " 'Government',\n",
       " 'Opposition',\n",
       " \"'\",\n",
       " 's',\n",
       " 'Gavan',\n",
       " 'O',\n",
       " 'Connor',\n",
       " 'says',\n",
       " 'letter',\n",
       " 'sent',\n",
       " '2002',\n",
       " ',',\n",
       " 'same',\n",
       " 'time',\n",
       " 'though',\n",
       " 'a',\n",
       " 'trucking',\n",
       " 'company',\n",
       " 'He',\n",
       " 'can',\n",
       " 'longer',\n",
       " 'wipe',\n",
       " 'its',\n",
       " 'hands',\n",
       " 'illicit',\n",
       " 'payments',\n",
       " 'which',\n",
       " '$',\n",
       " '290',\n",
       " 'million',\n",
       " '\"',\n",
       " 'responsibility',\n",
       " 'this',\n",
       " 'must',\n",
       " 'lay',\n",
       " 'may',\n",
       " 'at',\n",
       " 'feet',\n",
       " 'Coalition',\n",
       " 'ministers',\n",
       " 'trade',\n",
       " 'agriculture',\n",
       " ',\"',\n",
       " 'said',\n",
       " 'But',\n",
       " 'show',\n",
       " 'about',\n",
       " 'future',\n",
       " 'do',\n",
       " 'not',\n",
       " 'prove',\n",
       " 'It',\n",
       " 'would',\n",
       " 'if',\n",
       " 'as',\n",
       " 'I',\n",
       " 'hadn',\n",
       " 't',\n",
       " 'done',\n",
       " 'anything',\n",
       " 'possibly',\n",
       " 'could',\n",
       " 'preserve',\n",
       " 'Australia',\n",
       " 'very',\n",
       " 'valuable',\n",
       " 'market',\n",
       " 'questions',\n",
       " 'Today',\n",
       " 'trading',\n",
       " 'manager',\n",
       " 'Peter',\n",
       " 'questioned',\n",
       " 'an',\n",
       " 'email',\n",
       " 'received',\n",
       " 'May',\n",
       " '2000',\n",
       " 'indicated',\n",
       " 'that',\n",
       " 'Iraqi',\n",
       " 'Grains',\n",
       " 'Board',\n",
       " 'had',\n",
       " 'approached',\n",
       " 'provide',\n",
       " 'after',\n",
       " '-',\n",
       " 'service',\n",
       " '\".',\n",
       " 'two',\n",
       " 'colleagues',\n",
       " 'did',\n",
       " 'remember',\n",
       " 'reading',\n",
       " 'it',\n",
       " 'although',\n",
       " 'still',\n",
       " 'plenty',\n",
       " 'support',\n",
       " 'among',\n",
       " 'grain',\n",
       " 'growers',\n",
       " 'central',\n",
       " 'western',\n",
       " 'New',\n",
       " 'South',\n",
       " 'Wales',\n",
       " 'Producers',\n",
       " 'say',\n",
       " 'they',\n",
       " 'broadly',\n",
       " 'attempts',\n",
       " 'get',\n",
       " 'best',\n",
       " 'prices',\n",
       " 'their',\n",
       " 'products',\n",
       " 'think',\n",
       " 'all',\n",
       " 'overseas',\n",
       " 'interests',\n",
       " 'try',\n",
       " 'single',\n",
       " 'desk',\n",
       " 'put',\n",
       " 'aside',\n",
       " 'stories',\n",
       " 'are',\n",
       " 'going',\n",
       " 'round',\n",
       " 'commission',\n",
       " 'everything',\n",
       " 'way',\n",
       " 'people',\n",
       " 'got',\n",
       " 'things',\n",
       " 'business',\n",
       " 'Middle',\n",
       " 'East',\n",
       " 'Asian',\n",
       " 'countries',\n",
       " 'producer',\n",
       " 'actually',\n",
       " 'pretty',\n",
       " 'reasonable',\n",
       " 'system',\n",
       " 'd',\n",
       " 'give',\n",
       " 'them',\n",
       " 'fair',\n",
       " 'moment',\n",
       " 'average',\n",
       " 've',\n",
       " 'performed',\n",
       " 'fairly',\n",
       " 'well',\n",
       " 'another',\n",
       " 'biggest',\n",
       " 'thing',\n",
       " 'someone',\n",
       " 'else',\n",
       " 'taking',\n",
       " 'over',\n",
       " 'is',\n",
       " 'whether',\n",
       " 'will',\n",
       " 'too',\n",
       " 'much',\n",
       " 'foothold',\n",
       " 'there',\n",
       " 'take',\n",
       " 'advantage',\n",
       " '.\"',\n",
       " 'Grain',\n",
       " 'analyst',\n",
       " 'predicts',\n",
       " 'drop',\n",
       " '20',\n",
       " 'tonne',\n",
       " 'back',\n",
       " 'Malcolm',\n",
       " 'Bartholomaeus',\n",
       " 'pool',\n",
       " 'returns',\n",
       " 'already',\n",
       " 'dropped',\n",
       " 'year',\n",
       " 'price',\n",
       " 'past',\n",
       " 'five',\n",
       " 'years',\n",
       " 'through',\n",
       " 'export',\n",
       " 'monopoly',\n",
       " 'severely',\n",
       " 'eroded',\n",
       " 'SA',\n",
       " 'farmers',\n",
       " 'help',\n",
       " 'fire',\n",
       " 'ravaged',\n",
       " 'neighbours',\n",
       " 'Farmers',\n",
       " 'south',\n",
       " 'east',\n",
       " 'hay',\n",
       " 'across',\n",
       " 'border',\n",
       " 'wake',\n",
       " 'Grampians',\n",
       " 'bushfires',\n",
       " 'just',\n",
       " 'few',\n",
       " 'days',\n",
       " 'donated',\n",
       " '250',\n",
       " 'tonnes',\n",
       " 'agistment',\n",
       " 'cattle',\n",
       " 'They',\n",
       " 'beginning',\n",
       " 'Fodder',\n",
       " 'drive',\n",
       " 'coordinator',\n",
       " 'response',\n",
       " 'All',\n",
       " 'week',\n",
       " 'gone',\n",
       " 'places',\n",
       " 'load',\n",
       " 'or',\n",
       " 'up',\n",
       " 'loads',\n",
       " 'We',\n",
       " 'man',\n",
       " 'full',\n",
       " 'rest',\n",
       " 'straight',\n",
       " 'we',\n",
       " 're',\n",
       " 'moving',\n",
       " 'A',\n",
       " 'major',\n",
       " 'between',\n",
       " 'Northern',\n",
       " 'Territory',\n",
       " 'Western',\n",
       " 'remains',\n",
       " 'blocked',\n",
       " 'floodwaters',\n",
       " 'today',\n",
       " 'Victoria',\n",
       " 'River',\n",
       " 'cut',\n",
       " 'Highway',\n",
       " 'also',\n",
       " 'flooded',\n",
       " 'remote',\n",
       " 'Hole',\n",
       " 'Aboriginal',\n",
       " 'community',\n",
       " 'Simon',\n",
       " 'describes',\n",
       " 'hundred',\n",
       " 'higher',\n",
       " 'ground',\n",
       " 'vehicles',\n",
       " 'moved',\n",
       " 'out',\n",
       " 'set',\n",
       " 'were',\n",
       " 'more',\n",
       " 'couple',\n",
       " 'boats',\n",
       " 'onto',\n",
       " 'only',\n",
       " '500',\n",
       " 'metres',\n",
       " 'river',\n",
       " 'might',\n",
       " 'little',\n",
       " 'but',\n",
       " '[',\n",
       " ']',\n",
       " 'start',\n",
       " 'down',\n",
       " 'sold',\n",
       " 'Tasmania',\n",
       " 'main',\n",
       " 'changed',\n",
       " 'second',\n",
       " 'three',\n",
       " 'former',\n",
       " 'state',\n",
       " 'owned',\n",
       " 'Tasmanian',\n",
       " 'local',\n",
       " 'agribusiness',\n",
       " 'Roberts',\n",
       " 'Limited',\n",
       " '9',\n",
       " 'deal',\n",
       " 'includes',\n",
       " 'silos',\n",
       " 'Launceston',\n",
       " 'northern',\n",
       " 'bid',\n",
       " 'when',\n",
       " 'board',\n",
       " 'first',\n",
       " 'disappointed',\n",
       " 'weren',\n",
       " 'successful',\n",
       " 'point',\n",
       " 'Wine',\n",
       " 'workers',\n",
       " 'stop',\n",
       " 'work',\n",
       " 'Workers',\n",
       " 'Wines',\n",
       " 'Stanley',\n",
       " 'winery',\n",
       " 'walked',\n",
       " 'off',\n",
       " 'job',\n",
       " 'staff',\n",
       " 'morning',\n",
       " 'dispute',\n",
       " 'new',\n",
       " 'bargaining',\n",
       " 'agreement',\n",
       " 'comes',\n",
       " 'region',\n",
       " 'wine',\n",
       " 'grape',\n",
       " 'crush',\n",
       " 'gets',\n",
       " 'under',\n",
       " 'took',\n",
       " 'matter',\n",
       " 'Industrial',\n",
       " 'Commission',\n",
       " '(',\n",
       " ')',\n",
       " 'Friday',\n",
       " 'Wool',\n",
       " 'body',\n",
       " 'eyes',\n",
       " 'industry',\n",
       " '50',\n",
       " 'billion',\n",
       " 'global',\n",
       " 'target',\n",
       " 'wool',\n",
       " 'promotion',\n",
       " 'Australian',\n",
       " 'Innovation',\n",
       " 'AWI',\n",
       " ').',\n",
       " 'showing',\n",
       " 'blend',\n",
       " 'casual',\n",
       " 'wear',\n",
       " 'manufacturers',\n",
       " 'largest',\n",
       " 'being',\n",
       " 'held',\n",
       " 'Germany',\n",
       " 'Len',\n",
       " 'Stephens',\n",
       " 'end',\n",
       " 'shoppers',\n",
       " 'willing',\n",
       " 'pay',\n",
       " 'sports',\n",
       " 'sector',\n",
       " 'certainly',\n",
       " 'fastest',\n",
       " 'growing',\n",
       " 'world',\n",
       " 'no',\n",
       " 'secret',\n",
       " 'hasn',\n",
       " 'big',\n",
       " 'share',\n",
       " 'level',\n",
       " 'particularly',\n",
       " 'merino',\n",
       " 'really',\n",
       " 'almost',\n",
       " 'below',\n",
       " 'radar',\n",
       " 'Organisation',\n",
       " 'step',\n",
       " 'banning',\n",
       " 'rodeos',\n",
       " 'campaign',\n",
       " 'rodeo',\n",
       " 'sustained',\n",
       " 'attack',\n",
       " 'animal',\n",
       " 'rights',\n",
       " 'activists',\n",
       " 'destroyed',\n",
       " 'weekend',\n",
       " 'event',\n",
       " 'horse',\n",
       " 'breaking',\n",
       " 'leg',\n",
       " 'Saturday',\n",
       " 'Two',\n",
       " 'weeks',\n",
       " 'ago',\n",
       " 'bull',\n",
       " 'apparently',\n",
       " 'during',\n",
       " 'riding',\n",
       " 'competition',\n",
       " 'owner',\n",
       " 'both',\n",
       " 'animals',\n",
       " 'Brian',\n",
       " 'Fish',\n",
       " 'unfortunate',\n",
       " 'accidents',\n",
       " 'welfare',\n",
       " 'issue',\n",
       " 'Emma',\n",
       " 'Haswell',\n",
       " 'Animal',\n",
       " 'her',\n",
       " 'organisation',\n",
       " 'efforts',\n",
       " 'banned',\n",
       " 'used',\n",
       " 'cancer',\n",
       " 'research',\n",
       " 'investigate',\n",
       " 'sea',\n",
       " 'snails',\n",
       " 'eventually',\n",
       " 'treat',\n",
       " 'Until',\n",
       " 'now',\n",
       " 'snail',\n",
       " 'population',\n",
       " 'virtually',\n",
       " 'Flinders',\n",
       " 'University',\n",
       " 'hopes',\n",
       " 'discover',\n",
       " 'beneficial',\n",
       " 'compounds',\n",
       " 'predatory',\n",
       " 'known',\n",
       " 'Marine',\n",
       " 'biologist',\n",
       " 'Dr',\n",
       " 'follows',\n",
       " 'where',\n",
       " 'clinical',\n",
       " 'trials',\n",
       " 'investigating',\n",
       " 'anti',\n",
       " 'properties',\n",
       " 'm',\n",
       " 'looking',\n",
       " 'currently',\n",
       " 'harvested',\n",
       " 'considered',\n",
       " 'useful',\n",
       " 'resource',\n",
       " 'so',\n",
       " 'lot',\n",
       " 'potential',\n",
       " 'economic',\n",
       " 'benefits',\n",
       " 'side',\n",
       " 'she',\n",
       " 'Processors',\n",
       " 'fail',\n",
       " 'meet',\n",
       " 'kangaroo',\n",
       " 'meat',\n",
       " 'demand',\n",
       " 'International',\n",
       " 'putting',\n",
       " 'pressure',\n",
       " 'processors',\n",
       " 'Despite',\n",
       " 'high',\n",
       " 'harvesting',\n",
       " 'quotas',\n",
       " 'met',\n",
       " 'leaving',\n",
       " 'gap',\n",
       " 'Phil',\n",
       " 'King',\n",
       " 'Perth',\n",
       " 'Europeans',\n",
       " 'attracted',\n",
       " 'low',\n",
       " 'fat',\n",
       " 'content',\n",
       " 'consuming',\n",
       " 'large',\n",
       " 'amounts',\n",
       " 'As',\n",
       " 'far',\n",
       " 'know',\n",
       " 'every',\n",
       " 'manufactured',\n",
       " 'produced',\n",
       " 'human',\n",
       " 'consumption',\n",
       " 'goes',\n",
       " 'Russian',\n",
       " 'People',\n",
       " 'warned',\n",
       " 'snake',\n",
       " 'alert',\n",
       " 'With',\n",
       " 'temperatures',\n",
       " 'soaring',\n",
       " 'summer',\n",
       " 'southern',\n",
       " 'snakes',\n",
       " 'move',\n",
       " 'keep',\n",
       " 'open',\n",
       " 'paddock',\n",
       " 'especially',\n",
       " 'areas',\n",
       " 'hit',\n",
       " 'forced',\n",
       " 'usual',\n",
       " 'habitat',\n",
       " 'Maria',\n",
       " 'these',\n",
       " 'words',\n",
       " 'advice',\n",
       " 'Any',\n",
       " 'who',\n",
       " 'regularly',\n",
       " 'bush',\n",
       " 'walking',\n",
       " 'land',\n",
       " 'working',\n",
       " 'always',\n",
       " 'carry',\n",
       " 'least',\n",
       " 'mobile',\n",
       " 'phone',\n",
       " 'Aerial',\n",
       " 'spraying',\n",
       " 'begins',\n",
       " 'control',\n",
       " 'locust',\n",
       " 'threat',\n",
       " 'country',\n",
       " 'begun',\n",
       " 'aerial',\n",
       " 'locusts',\n",
       " 'plagues',\n",
       " 'made',\n",
       " 'continued',\n",
       " 'rain',\n",
       " 'warm',\n",
       " 'weather',\n",
       " 'assisting',\n",
       " 'breeding',\n",
       " 'conditions',\n",
       " 'crops',\n",
       " 'wiped',\n",
       " 'pest',\n",
       " 'won',\n",
       " 'runner',\n",
       " 'Queensland',\n",
       " 'farmer',\n",
       " 'knee',\n",
       " 'injury',\n",
       " 'him',\n",
       " 'running',\n",
       " 'Queen',\n",
       " 'lead',\n",
       " 'Commonwealth',\n",
       " 'Games',\n",
       " 'Stanthorpe',\n",
       " 'Bill',\n",
       " 'travelled',\n",
       " 'Harbour',\n",
       " 'run',\n",
       " 'his',\n",
       " 'And',\n",
       " 'prevent',\n",
       " 'making',\n",
       " 'metre',\n",
       " 'About',\n",
       " 'jumping',\n",
       " 'my',\n",
       " 'haven',\n",
       " 'able',\n",
       " 'than',\n",
       " 'walk',\n",
       " 'since',\n",
       " 'promise',\n",
       " 'you',\n",
       " 'because',\n",
       " 'such',\n",
       " 'rush',\n",
       " 'NZ',\n",
       " 'apple',\n",
       " 'debate',\n",
       " 'continues',\n",
       " 'Zealand',\n",
       " 'newspaper',\n",
       " 'members',\n",
       " 'concern',\n",
       " 'regarding',\n",
       " 'science',\n",
       " 'draft',\n",
       " 'Risk',\n",
       " 'Analysis',\n",
       " 'last',\n",
       " 'month',\n",
       " 'Biosecurity',\n",
       " 'determines',\n",
       " 'risk',\n",
       " 'fireblight',\n",
       " 'other',\n",
       " 'pests',\n",
       " 'diseases',\n",
       " 'apples',\n",
       " 'imported',\n",
       " 'Under',\n",
       " 'World',\n",
       " 'Trade',\n",
       " 'rules',\n",
       " 'refuse',\n",
       " 'scientific',\n",
       " 'evidence',\n",
       " 'substantial',\n",
       " 'disease',\n",
       " 'according',\n",
       " 'professional',\n",
       " 'shows',\n",
       " 'mature',\n",
       " 'pose',\n",
       " 'However',\n",
       " 'spokesperson',\n",
       " 'based',\n",
       " 'Apple',\n",
       " 'Group',\n",
       " 'provided',\n",
       " 'mathematical',\n",
       " 'model',\n",
       " 'determine',\n",
       " ':',\n",
       " 'assessment',\n",
       " 'Draft',\n",
       " 'Report',\n",
       " 'published',\n",
       " 'December',\n",
       " 'anyone',\n",
       " 'read',\n",
       " 'reproduce',\n",
       " 'analysis',\n",
       " 'given',\n",
       " 'report',\n",
       " 'chairman',\n",
       " 'task',\n",
       " 'force',\n",
       " 'surprising',\n",
       " 'exercise',\n",
       " 'right',\n",
       " 'presented',\n",
       " 'trees',\n",
       " 'producing',\n",
       " 'pears',\n",
       " 'trial',\n",
       " 'plot',\n",
       " 'young',\n",
       " 'pear',\n",
       " 'fruit',\n",
       " 'planted',\n",
       " 'Usually',\n",
       " 'don',\n",
       " 'produce',\n",
       " 'until',\n",
       " 'six',\n",
       " 'old',\n",
       " 'part',\n",
       " 'experimental',\n",
       " 'project',\n",
       " 'funded',\n",
       " 'Association',\n",
       " 'Horticulture',\n",
       " 'family',\n",
       " 'orchard',\n",
       " 'located',\n",
       " 'consultant',\n",
       " 'Van',\n",
       " 'aim',\n",
       " 'increase',\n",
       " 'number',\n",
       " 'grown',\n",
       " 'encouraging',\n",
       " 'younger',\n",
       " 'earlier',\n",
       " 'Local',\n",
       " 'federal',\n",
       " 'National',\n",
       " 'MP',\n",
       " 'supports',\n",
       " 'Member',\n",
       " 'Parkes',\n",
       " 'brought',\n",
       " 'This',\n",
       " 'examines',\n",
       " 'role',\n",
       " 'On',\n",
       " 'top',\n",
       " 'party',\n",
       " 'itself',\n",
       " 'producers',\n",
       " 'become',\n",
       " 'pro',\n",
       " 'active',\n",
       " 'make',\n",
       " 'themselves',\n",
       " 'different',\n",
       " 'Liberals',\n",
       " 'represent',\n",
       " 'regional',\n",
       " 'better',\n",
       " 'then',\n",
       " 'Liberal',\n",
       " 'deaths',\n",
       " 'believed',\n",
       " 'due',\n",
       " 'lack',\n",
       " 'oxygen',\n",
       " 'Department',\n",
       " 'Primary',\n",
       " 'Industries',\n",
       " 'fish',\n",
       " 'kill',\n",
       " 'Lachlan',\n",
       " 'upstream',\n",
       " 'Lake',\n",
       " 'Between',\n",
       " '150',\n",
       " '200',\n",
       " 'native',\n",
       " 'died',\n",
       " 'incident',\n",
       " 'District',\n",
       " 'Fisheries',\n",
       " 'Phillip',\n",
       " 'cod',\n",
       " 'golden',\n",
       " 'silver',\n",
       " 'killed',\n",
       " 'introduced',\n",
       " 'species',\n",
       " 'inquiries',\n",
       " 'suggest',\n",
       " 'result',\n",
       " 'increased',\n",
       " 'water',\n",
       " 'flow',\n",
       " 'chemical',\n",
       " 'poisoning',\n",
       " 'Vegetable',\n",
       " 'labels',\n",
       " 'Agriculture',\n",
       " 'group',\n",
       " 'AUSVEG',\n",
       " 'labelling',\n",
       " 'laws',\n",
       " 'nobody',\n",
       " 'enforcement',\n",
       " 'non',\n",
       " 'consumers',\n",
       " 'thinking',\n",
       " 'buying',\n",
       " 'false',\n",
       " 'poor',\n",
       " 'Red',\n",
       " 'tape',\n",
       " 'slows',\n",
       " 'west',\n",
       " 'Kimberley',\n",
       " 'horticulture',\n",
       " 'WA',\n",
       " 'soil',\n",
       " 'available',\n",
       " 'facing',\n",
       " 'hurdles',\n",
       " 'expand',\n",
       " 'potato',\n",
       " 'grower',\n",
       " 'Paul',\n",
       " 'Higgins',\n",
       " 'relies',\n",
       " 'generator',\n",
       " 'Broome',\n",
       " 'ones',\n",
       " 'connected',\n",
       " 'grid',\n",
       " 'applying',\n",
       " 'nine',\n",
       " 'kilometre',\n",
       " 'extension',\n",
       " 'current',\n",
       " 'bring',\n",
       " 'line',\n",
       " 'possible',\n",
       " 'customers',\n",
       " 'connection',\n",
       " 'costs',\n",
       " 'regardless',\n",
       " 'distance',\n",
       " 'existing',\n",
       " 'Land',\n",
       " 'acquisition',\n",
       " 'causing',\n",
       " 'headaches',\n",
       " 'purchased',\n",
       " 'June',\n",
       " 'title',\n",
       " 'improve',\n",
       " 'cleared',\n",
       " 'issues',\n",
       " 'resolved',\n",
       " 'area',\n",
       " 'surveyed',\n",
       " 'bed',\n",
       " 'paid',\n",
       " 'clear',\n",
       " 'apply',\n",
       " 'Planning',\n",
       " 'Infrastructure',\n",
       " 'seven',\n",
       " 'delay',\n",
       " 'should',\n",
       " 'aware',\n",
       " 'hold',\n",
       " 'West',\n",
       " 'Office',\n",
       " 'correctly',\n",
       " 'Though',\n",
       " 'Ms',\n",
       " 'worked',\n",
       " 'favour',\n",
       " '2004',\n",
       " 'property',\n",
       " 'official',\n",
       " 'before',\n",
       " 'ploughing',\n",
       " 'potatoes',\n",
       " 'Richmond',\n",
       " 'stay',\n",
       " 'fishers',\n",
       " 'decided',\n",
       " 'against',\n",
       " 'closing',\n",
       " 'commercial',\n",
       " 'recreational',\n",
       " '300',\n",
       " 'dead',\n",
       " 'eels',\n",
       " 'discovered',\n",
       " 'near',\n",
       " 'Thursday',\n",
       " 'levels',\n",
       " 'acting',\n",
       " 'fisheries',\n",
       " 'management',\n",
       " 'Anthony',\n",
       " 'inspectors',\n",
       " 'further',\n",
       " 'samples',\n",
       " 'tomorrow',\n",
       " 'Rural',\n",
       " 'woman',\n",
       " 'next',\n",
       " '10',\n",
       " 'seem',\n",
       " 'like',\n",
       " 'Catherine',\n",
       " 'Ford',\n",
       " 'find',\n",
       " 'Year',\n",
       " 'macadamia',\n",
       " 'win',\n",
       " 'guide',\n",
       " 'promoting',\n",
       " 'sustainability',\n",
       " 'admits',\n",
       " 'amazed',\n",
       " 'even',\n",
       " 'carrying',\n",
       " 'proactive',\n",
       " 'burning',\n",
       " 'rural',\n",
       " 'yesterday',\n",
       " 'heard',\n",
       " 'officers',\n",
       " 'feel',\n",
       " 'intensity',\n",
       " 'fires',\n",
       " 'fuel',\n",
       " 'reduction',\n",
       " 'burns',\n",
       " 'carried',\n",
       " 'national',\n",
       " 'park',\n",
       " 'VFF',\n",
       " 'branch',\n",
       " 'secretary',\n",
       " 'felt',\n",
       " 'enough',\n",
       " 'contributed',\n",
       " 'destructive',\n",
       " 'Sustainability',\n",
       " 'district',\n",
       " 'Geoff',\n",
       " 'Evans',\n",
       " 'agency',\n",
       " 'policy',\n",
       " 'some',\n",
       " 'landowners',\n",
       " 'keen',\n",
       " 'does',\n",
       " 'allow',\n",
       " ...]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X= list(model.wv.vocab)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('law', 0.9468981027603149),\n",
       " ('agriculture', 0.9328317046165466),\n",
       " ('general', 0.9301407337188721),\n",
       " ('policy', 0.9299634099006653),\n",
       " ('media', 0.9234398603439331),\n",
       " ('practice', 0.918952226638794),\n",
       " ('Crean', 0.9177979230880737),\n",
       " ('discussion', 0.9150807857513428),\n",
       " ('tight', 0.9143961668014526),\n",
       " ('Hooke', 0.9128788709640503)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data=model.most_similar('science')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('Federal', 0.8229790925979614),\n",
       " ('inquiry', 0.8202506899833679),\n",
       " ('government', 0.8166568279266357),\n",
       " ('Court', 0.811873197555542),\n",
       " ('company', 0.8014136552810669),\n",
       " ('Government', 0.787051796913147),\n",
       " ('exporter', 0.7665734887123108),\n",
       " ('Labor', 0.7493975162506104),\n",
       " ('veto', 0.7421144247055054),\n",
       " ('party', 0.7347246408462524)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data=model.most_similar('AWB')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "you\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `doesnt_match` (Method will be removed in 4.0.0, use self.wv.doesnt_match() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n",
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/gensim/models/keyedvectors.py:876: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
      "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
     ]
    }
   ],
   "source": [
    "dissimlar_words = model.doesnt_match('See you later, thanks for visiting'.split())\n",
    "print(dissimlar_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please provide the similarity between these two words:\n",
      "0.54881704\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `similarity` (Method will be removed in 4.0.0, use self.wv.similarity() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "similarity_two_words = model.similarity('science','AWB')\n",
    "print(\"Please provide the similarity between these two words:\")\n",
    "print(similarity_two_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2) Pre-trained Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### variouspre-trained models are available like Google Word2Vec, Godin, FastText, GloVe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Showcasing pre-trained model working on Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "path= '/media/shivangi/DATA/GoogleNews-vectors-negative300.bin' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/shivangi/anaconda3/envs/WWC_part1/lib/python3.7/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "# Load vectors directly from the file\n",
    "model = KeyedVectors.load_word2vec_format(path, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access vectors for specific words with a keyed lookup:\n",
    "vector = model['easy']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.06640625e-01,  6.83593750e-02, -1.60156250e-01,  1.19628906e-01,\n",
       "       -6.56127930e-03,  4.39453125e-03,  1.44531250e-01,  6.20117188e-02,\n",
       "        7.17773438e-02,  2.67333984e-02,  9.91210938e-02, -2.30712891e-02,\n",
       "        5.66406250e-02, -1.74804688e-01, -5.32226562e-02,  8.98437500e-02,\n",
       "        2.94921875e-01, -6.59179688e-02,  1.35742188e-01, -1.73828125e-01,\n",
       "        7.32421875e-02,  2.08007812e-01,  7.27539062e-02,  2.19726562e-01,\n",
       "       -5.02929688e-02, -1.15234375e-01, -1.80664062e-01, -4.29153442e-06,\n",
       "       -1.69921875e-01, -7.61718750e-02, -4.30297852e-03,  1.71875000e-01,\n",
       "        2.57812500e-01, -1.33789062e-01,  3.95507812e-02,  4.24194336e-03,\n",
       "       -2.80761719e-02, -1.54296875e-01,  1.76757812e-01,  6.68945312e-02,\n",
       "        2.71484375e-01, -1.43554688e-01,  4.02343750e-01, -1.19140625e-01,\n",
       "       -2.58789062e-02, -5.63964844e-02,  3.78417969e-02,  4.29687500e-02,\n",
       "        2.92968750e-02, -2.11181641e-02, -4.15039062e-02,  6.29882812e-02,\n",
       "       -1.90429688e-02, -6.10351562e-02, -1.66992188e-01,  1.05468750e-01,\n",
       "        4.40597534e-04, -1.81640625e-01,  1.90429688e-01,  6.98242188e-02,\n",
       "       -8.64257812e-02, -6.83593750e-02, -3.08593750e-01, -1.25000000e-01,\n",
       "       -1.67236328e-02, -1.03515625e-01, -2.59765625e-01,  2.29492188e-02,\n",
       "       -1.01074219e-01, -2.53295898e-03,  2.92968750e-02,  1.67968750e-01,\n",
       "       -1.64794922e-02, -2.53906250e-01, -1.81640625e-01, -1.24511719e-01,\n",
       "        9.61914062e-02,  9.42382812e-02,  1.46484375e-01,  1.94335938e-01,\n",
       "       -2.61718750e-01,  1.19140625e-01, -1.04003906e-01,  1.85546875e-02,\n",
       "        3.02734375e-02, -1.68945312e-01, -1.00585938e-01, -1.43554688e-01,\n",
       "        1.24511719e-01,  1.55029297e-02, -1.60156250e-01, -4.63867188e-02,\n",
       "       -8.64257812e-02, -1.51367188e-01, -2.11914062e-01,  3.93066406e-02,\n",
       "       -3.05175781e-03,  5.82885742e-03,  8.88671875e-02, -8.98437500e-02,\n",
       "       -6.68945312e-02,  2.24609375e-01, -1.20605469e-01,  3.95507812e-02,\n",
       "       -2.07031250e-01,  1.77734375e-01,  4.51660156e-03, -1.22558594e-01,\n",
       "        2.01171875e-01, -7.51953125e-02, -1.63085938e-01, -1.79687500e-01,\n",
       "       -2.35351562e-01, -2.81250000e-01,  3.28125000e-01,  6.59179688e-02,\n",
       "        1.35498047e-02, -6.25610352e-03, -1.88446045e-03, -5.17578125e-02,\n",
       "       -1.80664062e-01,  7.37304688e-02,  1.48437500e-01, -1.46484375e-01,\n",
       "       -1.96289062e-01, -8.30078125e-03, -1.02539062e-01, -1.27929688e-01,\n",
       "        1.44531250e-01,  1.84570312e-01, -2.34375000e-01,  1.87988281e-02,\n",
       "       -3.80859375e-02,  4.72656250e-01, -1.02050781e-01, -4.93164062e-02,\n",
       "        4.51660156e-02,  1.20605469e-01,  3.24707031e-02,  1.27929688e-01,\n",
       "       -4.76074219e-02, -2.83203125e-02,  1.32812500e-01, -1.03515625e-01,\n",
       "       -1.19140625e-01, -5.60760498e-04, -1.11816406e-01, -1.31835938e-01,\n",
       "        1.38671875e-01, -4.19921875e-02,  4.00390625e-02,  9.91210938e-02,\n",
       "       -1.20117188e-01, -4.63867188e-02, -2.62451172e-02,  7.86132812e-02,\n",
       "       -1.64062500e-01,  1.49414062e-01, -2.55859375e-01, -1.18164062e-01,\n",
       "       -4.32128906e-02,  2.55859375e-01,  1.79687500e-01, -9.22851562e-02,\n",
       "        8.10546875e-02, -4.05273438e-02, -3.02734375e-01, -1.06933594e-01,\n",
       "        7.03125000e-02, -2.61718750e-01, -1.70898438e-01,  1.08398438e-01,\n",
       "       -2.79296875e-01,  1.65039062e-01, -3.46679688e-02,  3.63281250e-01,\n",
       "       -1.48437500e-01, -7.08007812e-02, -8.39843750e-02,  5.37109375e-02,\n",
       "       -1.95312500e-01, -2.71484375e-01,  2.46582031e-02, -2.19726562e-01,\n",
       "       -2.18505859e-02,  4.66308594e-02, -1.08398438e-01,  9.13085938e-02,\n",
       "       -8.74023438e-02, -7.12890625e-02, -1.13281250e-01, -9.71679688e-02,\n",
       "        1.30004883e-02,  4.19921875e-02, -2.81250000e-01, -8.39843750e-02,\n",
       "       -1.14257812e-01, -1.68945312e-01, -1.76757812e-01, -3.20312500e-01,\n",
       "        2.96875000e-01,  1.50390625e-01, -2.09960938e-02,  1.65039062e-01,\n",
       "        9.42382812e-02,  9.37500000e-02, -2.96630859e-02, -1.19628906e-02,\n",
       "        2.73437500e-02, -2.81982422e-02, -1.23046875e-01,  3.75000000e-01,\n",
       "       -7.95898438e-02,  1.97265625e-01,  1.73828125e-01, -1.02050781e-01,\n",
       "        1.90429688e-01,  2.55859375e-01, -2.61718750e-01,  3.06396484e-02,\n",
       "       -5.51757812e-02,  1.66992188e-01, -1.69921875e-01, -1.43554688e-01,\n",
       "       -3.58886719e-02, -1.18652344e-01,  2.00195312e-01, -2.79296875e-01,\n",
       "        5.78613281e-02,  4.61425781e-02,  6.54296875e-02,  2.21679688e-01,\n",
       "       -3.00292969e-02, -2.81982422e-02,  1.74804688e-01, -7.56835938e-02,\n",
       "       -9.86328125e-02, -6.00585938e-02,  1.01928711e-02,  1.94335938e-01,\n",
       "        4.10156250e-02, -8.00781250e-02, -4.88281250e-02, -2.65625000e-01,\n",
       "       -1.54296875e-01,  7.76367188e-02, -4.19921875e-02,  1.85546875e-01,\n",
       "       -5.27343750e-02, -1.32812500e-01, -8.66699219e-03, -5.02929688e-02,\n",
       "        2.75390625e-01,  1.19628906e-01,  2.50000000e-01,  1.05957031e-01,\n",
       "        2.02636719e-02, -8.98437500e-02, -1.25000000e-01, -1.87500000e-01,\n",
       "        5.34667969e-02, -1.44531250e-01, -1.81640625e-01, -7.22656250e-02,\n",
       "        3.54003906e-02,  1.28906250e-01, -2.85644531e-02, -3.80859375e-01,\n",
       "       -1.03515625e-01, -6.93359375e-02,  1.39648438e-01,  1.19140625e-01,\n",
       "        2.27050781e-02,  9.61914062e-02, -8.64257812e-02,  1.39648438e-01,\n",
       "        3.20312500e-01, -1.84570312e-01,  8.59375000e-02, -3.51562500e-02,\n",
       "       -6.59179688e-02, -1.38671875e-01, -2.10937500e-01, -7.32421875e-02,\n",
       "       -2.71484375e-01,  1.83593750e-01, -9.81445312e-02,  6.07910156e-02,\n",
       "       -7.91015625e-02,  1.75781250e-01, -1.17187500e-01,  1.53320312e-01,\n",
       "        8.25195312e-02, -4.54101562e-02, -2.14843750e-01,  2.64892578e-02,\n",
       "        1.31835938e-01, -1.59912109e-02, -2.16796875e-01,  1.01562500e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see the shape of the vector (300,)\n",
    "vector.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Processing sentences is not as simple as with Spacy:\n",
    "vectors = [model[x] for x in \"This is some text I am processing with Spacy\".split(' ')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "vectors=np.array(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix=np.vstack(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(embedding_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.2890625 ,  0.19921875,  0.16015625, ...,  0.12792969,\n",
       "         0.12109375, -0.22949219],\n",
       "       [ 0.00704956, -0.07324219,  0.171875  , ...,  0.01123047,\n",
       "         0.1640625 ,  0.10693359],\n",
       "       [ 0.17871094,  0.09130859, -0.00165558, ...,  0.125     ,\n",
       "         0.08056641,  0.01672363],\n",
       "       ...,\n",
       "       [-0.09033203,  0.04394531,  0.11621094, ..., -0.3359375 ,\n",
       "        -0.15234375,  0.00254822],\n",
       "       [-0.02490234,  0.02197266, -0.03540039, ...,  0.01080322,\n",
       "        -0.01879883, -0.06884766],\n",
       "       [ 0.06054688,  0.09326172, -0.07373047, ..., -0.07177734,\n",
       "        -0.02893066, -0.02185059]], dtype=float32)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 300)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Refer to source[8] of the Medium Blog: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5717044"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.similarity('straightforward', 'easy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.similar_by_word('kind')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2444769"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity= model.similarity('please','see')\n",
    "similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most-similar words\n",
    "* Find the top-N most similar words. Positive words contribute positively towards the similarity, \n",
    "negative words negatively.\n",
    "\n",
    "This method computes cosine similarity between a simple mean of the projection weight vectors of \n",
    "the given words and the vectors for each word in the model. \n",
    "The method corresponds to the word-analogy and distance scripts in the original word2vec implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# similar_words = model.most_similar('thanks')\n",
    "# print(similar_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Custom Word Embeddings"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (WWC_part1)",
   "language": "python",
   "name": "wwc_part1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}