07. Recurrent Neural Networks/04. Sequence to Sequence - MXNet.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sequence to Sequence - MXNet (From Scratch).ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "BEvn8oD_MIK_",
        "colab_type": "code",
        "outputId": "b04437b4-53de-4343-b753-cc581c21bf1b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 151
        }
      },
      "source": [
        "!pip install mxnet-cu100"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: mxnet-cu100 in /usr/local/lib/python3.6/dist-packages (1.4.1)\n",
            "Requirement already satisfied: requests>=2.20.0 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (2.21.0)\n",
            "Requirement already satisfied: numpy<1.15.0,>=1.8.2 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (1.14.6)\n",
            "Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (0.8.4)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20.0->mxnet-cu100) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20.0->mxnet-cu100) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20.0->mxnet-cu100) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20.0->mxnet-cu100) (2019.6.16)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DfwJArQUMZKI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import mxnet as mx\n",
        "from mxnet import nd, autograd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "ctx = mx.gpu()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DK1pMOcRlVgE",
        "colab_type": "text"
      },
      "source": [
        "##Reading the data from text file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5aBCNwVSlYpT",
        "colab_type": "code",
        "outputId": "3cd5578a-6809-4ab6-e8b6-54e5d6a4b99b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "with open('text.txt') as file:\n",
        "    text = file.read()\n",
        "\n",
        "print(text)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Artificial intelligence (AI) is the simulation of human intelligence processes by machines, especially computer systems. These processes include learning (the acquisition of information and rules for using the information), reasoning (using rules to reach approximate or definite conclusions) and self-correction.\n",
            "Particular applications of AI include expert systems, speech recognition and machine vision. Artificial intelligence (AI), the ability of a digital computer or computer-controlled robot to perform tasks commonly associated with intelligent beings. The term is frequently applied to the project of developing systems endowed with the intellectual processes characteristic of humans, such as the ability to reason, discover meaning, generalize, or learn from past experience. Since the development of the digital computer in the 1940s, it has been demonstrated that computers can be programmed to carry out very complex tasks—as, for example, discovering proofs for mathematical theorems or playing chess—with great proficiency.\n",
            "Still, despite continuing advances in computer processing speed and memory capacity, there are as yet no programs that can match human flexibility over wider domains or in tasks requiring much everyday knowledge. On the other hand, some programs have attained the performance levels of human experts and professionals in performing certain specific tasks, so that artificial intelligence in this limited sense is found in applications as diverse as medical diagnosis, computer search engines, and voice or handwriting recognition.\n",
            "All but the simplest human behaviour is ascribed to intelligence, while even the most complicated insect behaviour is never taken as an indication of intelligence. What is the difference? Consider the behaviour of the digger wasp, Sphex ichneumoneus. When the female wasp returns to her burrow with food, she first deposits it on the threshold, checks for intruders inside her burrow, and only then, if the coast is clear, carries her food inside. The real nature of the wasp’s instinctual behaviour is revealed if the food is moved a few inches away from the entrance to her burrow while she is inside: on emerging, she will repeat the whole procedure as often as the food is displaced. Intelligence—conspicuously absent in the case of Sphex—must include the ability to adapt to new circumstances.\n",
            "Psychologists generally do not characterize human intelligence by just one trait but by the combination of many diverse abilities. Research in AI has focused chiefly on the following components of intelligence: learning, reasoning, problem solving, perception, and using language.\n",
            "AI can be categorized as either weak or strong. Weak AI, also known as narrow AI, is an AI system that is designed and trained for a particular task. Virtual personal assistants, such as Apple's Siri, are a form of weak AI. Strong AI, also known as artificial general intelligence, is an AI system with generalized human cognitive abilities. When presented with an unfamiliar task, a strong AI system is able to find a solution without human intervention.\n",
            "Because hardware, software and staffing costs for AI can be expensive, many vendors are including AI components in their standard offerings, as well as access to Artificial Intelligence as a Service (AIaaS) platforms. AI as a Service allows individuals and companies to experiment with AI for various business purposes and sample multiple platforms before making a commitment. Popular AI cloud offerings include Amazon AI services, IBM Watson Assistant, Microsoft Cognitive Services and Google AI services. While AI tools present a range of new functionality for businesses ,the use of artificial intelligence raises ethical questions. This is because deep learning algorithms, which underpin many of the most advanced AI tools, are only as smart as the data they are given in training. Because a human selects what data should be used for training an AI program, the potential for human bias is inherent and must be monitored closely.\n",
            "Some industry experts believe that the term artificial intelligence is too closely linked to popular culture, causing the general public to have unrealistic fears about artificial intelligence and improbable expectations about how it will change the workplace and life in general.\n",
            "Researchers and marketers hope the label augmented intelligence, which has a more neutral connotation, will help people understand that AI will simply improve products and services, not replace the humans that use them. Knowledge engineering is a core part of AI research. Machines can often act and react like humans only if they have abundant information relating to the world. Artificial intelligence must have access to objects, categories, properties and relations between all of them to implement knowledge engineering. Initiating common sense, reasoning and problem-solving power in machines is a difficult and tedious task.Machine learning is also a core part of AI.\n",
            "There are a number of different forms of learning as applied to artificial intelligence. The simplest is learning by trial and error. For example, a simple computer program for solving mate-in-one chess problems might try moves at random until mate is found. The program might then store the solution with the position so that the next time the computer encountered the same position it would recall the solution. This simple memorizing of individual items and procedures—known as rote learning—is relatively easy to implement on a computer. More challenging is the problem of implementing what is called generalization. Generalization involves applying past experience to analogous new situations. For example, a program that learns the past tense of regular English verbs by rote will not be able to produce the past tense of a word such as jump unless it previously had been presented with jumped, whereas a program that is able to generalize can learn the “add ed” rule and so form the past tense of jump based on experience with similar verbs.\n",
            "Learning without any kind of supervision requires an ability to identify patterns in streams of inputs, whereas learning with adequate supervision involves classification and numerical regressions. Classification determines the category an object belongs to and regression deals with obtaining a set of numerical input or output examples, thereby discovering functions enabling the generation of suitable outputs from respective inputs. Mathematical analysis of machine learning algorithms and their performance is a well-defined branch of theoretical computer science often referred to as computational learning theory. Machine perception deals with the capability to use sensory inputs to deduce the different aspects of the world, while computer vision is the power to analyze visual inputs with a few sub-problems such as facial, object and gesture recognition. Robotics is also a major field related to AI. Robots require intelligence to handle tasks such as object manipulation and navigation, along with sub-problems of localization, motion planning and mapping.\n",
            "To reason is to draw inferences appropriate to the situation. Inferences are classified as either deductive or inductive. An example of the former is, “Fred must be in either the museum or the café. He is not in the café; therefore he is in the museum,” and of the latter, “Previous accidents of this sort were caused by instrument failure; therefore this accident was caused by instrument failure.” The most significant difference between these forms of reasoning is that in the deductive case the truth of the premises guarantees the truth of the conclusion, whereas in the inductive case the truth of the premise lends support to the conclusion without giving absolute assurance. Inductive reasoning is common in science, where data are collected and tentative models are developed to describe and predict future behaviour—until the appearance of anomalous data forces the model to be revised. Deductive reasoning is common in mathematics and logic, where elaborate structures of irrefutable theorems are built up from a small set of basic axioms and rules.\n",
            "There has been considerable success in programming computers to draw inferences, especially deductive inferences. However, true reasoning involves more than just drawing inferences; it involves drawing inferences relevant to the solution of the particular task or situation. This is one of the hardest problems confronting AI.\n",
            "Problem solving, particularly in artificial intelligence, may be characterized as a systematic search through a range of possible actions in order to reach some predefined goal or solution. Problem-solving methods divide into special purpose and general purpose. A special-purpose method is tailor-made for a particular problem and often exploits very specific features of the situation in which the problem is embedded. In contrast, a general-purpose method is applicable to a wide variety of problems. One general-purpose technique used in AI is means-end analysis—a step-by-step, or incremental, reduction of the difference between the current state and the final goal. The program selects actions from a list of means—in the case of a simple robot this might consist of PICKUP, PUTDOWN, MOVEFORWARD, MOVEBACK, MOVELEFT, and MOVERIGHT—until the goal is reached.\n",
            "In perception the environment is scanned by means of various sensory organs, real or artificial, and the scene is decomposed into separate objects in various spatial relationships. Analysis is complicated by the fact that an object may appear different depending on the angle from which it is viewed, the direction and intensity of illumination in the scene, and how much the object contrasts with the surrounding field.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vJcQon60lYmX",
        "colab_type": "code",
        "outputId": "f61edd30-41cd-4cbe-b53c-6c087dfb61bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "characters = list(set(text))\n",
        "vocab_length = len(characters)\n",
        "print(characters)\n",
        "print(f'Number of unique characters: {vocab_length}')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['K', 'é', 'N', 'a', 'f', 's', ':', 'V', 'O', '”', 'c', '“', 'h', '?', 'b', 'P', 'm', 'G', 'R', 'g', 'd', 'I', 'E', '—', ';', 'A', 'o', '1', 't', 'B', 'M', 'L', 'i', 'D', 'n', 'y', 'T', 'q', 'l', 'S', 'H', 'u', '(', '0', 'j', \"'\", 'e', 'r', 'p', '.', 'F', '9', '4', 'w', 'k', 'W', '’', 'v', 'C', 'U', ')', 'x', 'z', ' ', ',', '-', '\\n']\n",
            "Number of unique characters: 67\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E2LeNuEDlYhJ",
        "colab_type": "code",
        "outputId": "5ce63bfa-c668-4355-fe3b-7fa958e58116",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "char_to_int = dict([(char, i) for i, char in enumerate(characters)])\n",
        "char_to_int"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'\\n': 66,\n",
              " ' ': 63,\n",
              " \"'\": 45,\n",
              " '(': 42,\n",
              " ')': 60,\n",
              " ',': 64,\n",
              " '-': 65,\n",
              " '.': 49,\n",
              " '0': 43,\n",
              " '1': 27,\n",
              " '4': 52,\n",
              " '9': 51,\n",
              " ':': 6,\n",
              " ';': 24,\n",
              " '?': 13,\n",
              " 'A': 25,\n",
              " 'B': 29,\n",
              " 'C': 58,\n",
              " 'D': 33,\n",
              " 'E': 22,\n",
              " 'F': 50,\n",
              " 'G': 17,\n",
              " 'H': 40,\n",
              " 'I': 21,\n",
              " 'K': 0,\n",
              " 'L': 31,\n",
              " 'M': 30,\n",
              " 'N': 2,\n",
              " 'O': 8,\n",
              " 'P': 15,\n",
              " 'R': 18,\n",
              " 'S': 39,\n",
              " 'T': 36,\n",
              " 'U': 59,\n",
              " 'V': 7,\n",
              " 'W': 55,\n",
              " 'a': 3,\n",
              " 'b': 14,\n",
              " 'c': 10,\n",
              " 'd': 20,\n",
              " 'e': 46,\n",
              " 'f': 4,\n",
              " 'g': 19,\n",
              " 'h': 12,\n",
              " 'i': 32,\n",
              " 'j': 44,\n",
              " 'k': 54,\n",
              " 'l': 38,\n",
              " 'm': 16,\n",
              " 'n': 34,\n",
              " 'o': 26,\n",
              " 'p': 48,\n",
              " 'q': 37,\n",
              " 'r': 47,\n",
              " 's': 5,\n",
              " 't': 28,\n",
              " 'u': 41,\n",
              " 'v': 57,\n",
              " 'w': 53,\n",
              " 'x': 61,\n",
              " 'y': 35,\n",
              " 'z': 62,\n",
              " 'é': 1,\n",
              " '—': 23,\n",
              " '’': 56,\n",
              " '“': 11,\n",
              " '”': 9}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3A--N_D3lYe6",
        "colab_type": "code",
        "outputId": "a2ba14e3-975e-49b5-dc9d-1008b618a180",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "int_to_char = dict([(i, char) for i, char in enumerate(characters)])\n",
        "int_to_char"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 'K',\n",
              " 1: 'é',\n",
              " 2: 'N',\n",
              " 3: 'a',\n",
              " 4: 'f',\n",
              " 5: 's',\n",
              " 6: ':',\n",
              " 7: 'V',\n",
              " 8: 'O',\n",
              " 9: '”',\n",
              " 10: 'c',\n",
              " 11: '“',\n",
              " 12: 'h',\n",
              " 13: '?',\n",
              " 14: 'b',\n",
              " 15: 'P',\n",
              " 16: 'm',\n",
              " 17: 'G',\n",
              " 18: 'R',\n",
              " 19: 'g',\n",
              " 20: 'd',\n",
              " 21: 'I',\n",
              " 22: 'E',\n",
              " 23: '—',\n",
              " 24: ';',\n",
              " 25: 'A',\n",
              " 26: 'o',\n",
              " 27: '1',\n",
              " 28: 't',\n",
              " 29: 'B',\n",
              " 30: 'M',\n",
              " 31: 'L',\n",
              " 32: 'i',\n",
              " 33: 'D',\n",
              " 34: 'n',\n",
              " 35: 'y',\n",
              " 36: 'T',\n",
              " 37: 'q',\n",
              " 38: 'l',\n",
              " 39: 'S',\n",
              " 40: 'H',\n",
              " 41: 'u',\n",
              " 42: '(',\n",
              " 43: '0',\n",
              " 44: 'j',\n",
              " 45: \"'\",\n",
              " 46: 'e',\n",
              " 47: 'r',\n",
              " 48: 'p',\n",
              " 49: '.',\n",
              " 50: 'F',\n",
              " 51: '9',\n",
              " 52: '4',\n",
              " 53: 'w',\n",
              " 54: 'k',\n",
              " 55: 'W',\n",
              " 56: '’',\n",
              " 57: 'v',\n",
              " 58: 'C',\n",
              " 59: 'U',\n",
              " 60: ')',\n",
              " 61: 'x',\n",
              " 62: 'z',\n",
              " 63: ' ',\n",
              " 64: ',',\n",
              " 65: '-',\n",
              " 66: '\\n'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r0yP2BxkTTXz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def toint(sentence):\n",
        "    return [char_to_int[char] for char in sentence]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7YR1HaIlTdeB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def tosentence(ints):\n",
        "    return ''.join([int_to_char[_int] for _int in ints])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "58oqFgyHnkt2",
        "colab_type": "code",
        "outputId": "e0ba0a56-37db-4d07-f6de-05788feece24",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "int_text = toint(text)\n",
        "int_text"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[25,\n",
              " 47,\n",
              " 28,\n",
              " 32,\n",
              " 4,\n",
              " 32,\n",
              " 10,\n",
              " 32,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 28,\n",
              " 46,\n",
              " 38,\n",
              " 38,\n",
              " 32,\n",
              " 19,\n",
              " 46,\n",
              " 34,\n",
              " 10,\n",
              " 46,\n",
              " 63,\n",
              " 42,\n",
              " 25,\n",
              " 21,\n",
              " 60,\n",
              " 63,\n",
              " 32,\n",
              " 5,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 5,\n",
              " 32,\n",
              " 16,\n",
              " 41,\n",
              " 38,\n",
              " 3,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 12,\n",
              " 41,\n",
              " 16,\n",
              " 3,\n",
              " 34,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 28,\n",
              " 46,\n",
              " 38,\n",
              " 38,\n",
              " 32,\n",
              " 19,\n",
              " 46,\n",
              " 34,\n",
              " 10,\n",
              " 46,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 10,\n",
              " 46,\n",
              " 5,\n",
              " 5,\n",
              " 46,\n",
              " 5,\n",
              " 63,\n",
              " 14,\n",
              " 35,\n",
              " 63,\n",
              " 16,\n",
              " 3,\n",
              " 10,\n",
              " 12,\n",
              " 32,\n",
              " 34,\n",
              " 46,\n",
              " 5,\n",
              " 64,\n",
              " 63,\n",
              " 46,\n",
              " 5,\n",
              " 48,\n",
              " 46,\n",
              " 10,\n",
              " 32,\n",
              " 3,\n",
              " 38,\n",
              " 38,\n",
              " 35,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 41,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 63,\n",
              " 5,\n",
              " 35,\n",
              " 5,\n",
              " 28,\n",
              " 46,\n",
              " 16,\n",
              " 5,\n",
              " 49,\n",
              " 63,\n",
              " 36,\n",
              " 12,\n",
              " 46,\n",
              " 5,\n",
              " 46,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 10,\n",
              " 46,\n",
              " 5,\n",
              " 5,\n",
              " 46,\n",
              " 5,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 10,\n",
              " 38,\n",
              " 41,\n",
              " 20,\n",
              " 46,\n",
              " 63,\n",
              " 38,\n",
              " 46,\n",
              " 3,\n",
              " 47,\n",
              " 34,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 42,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 3,\n",
              " 10,\n",
              " 37,\n",
              " 41,\n",
              " 32,\n",
              " 5,\n",
              " 32,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 16,\n",
              " 3,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 63,\n",
              " 3,\n",
              " 34,\n",
              " 20,\n",
              " 63,\n",
              " 47,\n",
              " 41,\n",
              " 38,\n",
              " 46,\n",
              " 5,\n",
              " 63,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 41,\n",
              " 5,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 16,\n",
              " 3,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 60,\n",
              " 64,\n",
              " 63,\n",
              " 47,\n",
              " 46,\n",
              " 3,\n",
              " 5,\n",
              " 26,\n",
              " 34,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 42,\n",
              " 41,\n",
              " 5,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 47,\n",
              " 41,\n",
              " 38,\n",
              " 46,\n",
              " 5,\n",
              " 63,\n",
              " 28,\n",
              " 26,\n",
              " 63,\n",
              " 47,\n",
              " 46,\n",
              " 3,\n",
              " 10,\n",
              " 12,\n",
              " 63,\n",
              " 3,\n",
              " 48,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 61,\n",
              " 32,\n",
              " 16,\n",
              " 3,\n",
              " 28,\n",
              " 46,\n",
              " 63,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 20,\n",
              " 46,\n",
              " 4,\n",
              " 32,\n",
              " 34,\n",
              " 32,\n",
              " 28,\n",
              " 46,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 34,\n",
              " 10,\n",
              " 38,\n",
              " 41,\n",
              " 5,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 5,\n",
              " 60,\n",
              " 63,\n",
              " 3,\n",
              " 34,\n",
              " 20,\n",
              " 63,\n",
              " 5,\n",
              " 46,\n",
              " 38,\n",
              " 4,\n",
              " 65,\n",
              " 10,\n",
              " 26,\n",
              " 47,\n",
              " 47,\n",
              " 46,\n",
              " 10,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 49,\n",
              " 66,\n",
              " 15,\n",
              " 3,\n",
              " 47,\n",
              " 28,\n",
              " 32,\n",
              " 10,\n",
              " 41,\n",
              " 38,\n",
              " 3,\n",
              " 47,\n",
              " 63,\n",
              " 3,\n",
              " 48,\n",
              " 48,\n",
              " 38,\n",
              " 32,\n",
              " 10,\n",
              " 3,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 5,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 25,\n",
              " 21,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 10,\n",
              " 38,\n",
              " 41,\n",
              " 20,\n",
              " 46,\n",
              " 63,\n",
              " 46,\n",
              " 61,\n",
              " 48,\n",
              " 46,\n",
              " 47,\n",
              " 28,\n",
              " 63,\n",
              " 5,\n",
              " 35,\n",
              " 5,\n",
              " 28,\n",
              " 46,\n",
              " 16,\n",
              " 5,\n",
              " 64,\n",
              " 63,\n",
              " 5,\n",
              " 48,\n",
              " 46,\n",
              " 46,\n",
              " 10,\n",
              " 12,\n",
              " 63,\n",
              " 47,\n",
              " 46,\n",
              " 10,\n",
              " 26,\n",
              " 19,\n",
              " 34,\n",
              " 32,\n",
              " 28,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 63,\n",
              " 3,\n",
              " 34,\n",
              " 20,\n",
              " 63,\n",
              " 16,\n",
              " 3,\n",
              " 10,\n",
              " 12,\n",
              " 32,\n",
              " 34,\n",
              " 46,\n",
              " 63,\n",
              " 57,\n",
              " 32,\n",
              " 5,\n",
              " 32,\n",
              " 26,\n",
              " 34,\n",
              " 49,\n",
              " 63,\n",
              " 25,\n",
              " 47,\n",
              " 28,\n",
              " 32,\n",
              " 4,\n",
              " 32,\n",
              " 10,\n",
              " 32,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 28,\n",
              " 46,\n",
              " 38,\n",
              " 38,\n",
              " 32,\n",
              " 19,\n",
              " 46,\n",
              " 34,\n",
              " 10,\n",
              " 46,\n",
              " 63,\n",
              " 42,\n",
              " 25,\n",
              " 21,\n",
              " 60,\n",
              " 64,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 3,\n",
              " 14,\n",
              " 32,\n",
              " 38,\n",
              " 32,\n",
              " 28,\n",
              " 35,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 3,\n",
              " 63,\n",
              " 20,\n",
              " 32,\n",
              " 19,\n",
              " 32,\n",
              " 28,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 41,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 63,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 41,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 65,\n",
              " 10,\n",
              " 26,\n",
              " 34,\n",
              " 28,\n",
              " 47,\n",
              " 26,\n",
              " 38,\n",
              " 38,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 47,\n",
              " 26,\n",
              " 14,\n",
              " 26,\n",
              " 28,\n",
              " 63,\n",
              " 28,\n",
              " 26,\n",
              " 63,\n",
              " 48,\n",
              " 46,\n",
              " 47,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 16,\n",
              " 63,\n",
              " 28,\n",
              " 3,\n",
              " 5,\n",
              " 54,\n",
              " 5,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 16,\n",
              " 26,\n",
              " 34,\n",
              " 38,\n",
              " 35,\n",
              " 63,\n",
              " 3,\n",
              " 5,\n",
              " 5,\n",
              " 26,\n",
              " 10,\n",
              " 32,\n",
              " 3,\n",
              " 28,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 53,\n",
              " 32,\n",
              " 28,\n",
              " 12,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 28,\n",
              " 46,\n",
              " 38,\n",
              " 38,\n",
              " 32,\n",
              " 19,\n",
              " 46,\n",
              " 34,\n",
              " 28,\n",
              " 63,\n",
              " 14,\n",
              " 46,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 5,\n",
              " 49,\n",
              " 63,\n",
              " 36,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 16,\n",
              " 63,\n",
              " 32,\n",
              " 5,\n",
              " 63,\n",
              " 4,\n",
              " 47,\n",
              " 46,\n",
              " 37,\n",
              " 41,\n",
              " 46,\n",
              " 34,\n",
              " 28,\n",
              " 38,\n",
              " 35,\n",
              " 63,\n",
              " 3,\n",
              " 48,\n",
              " 48,\n",
              " 38,\n",
              " 32,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 28,\n",
              " 26,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 44,\n",
              " 46,\n",
              " 10,\n",
              " 28,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 20,\n",
              " 46,\n",
              " 57,\n",
              " 46,\n",
              " 38,\n",
              " 26,\n",
              " 48,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 5,\n",
              " 35,\n",
              " 5,\n",
              " 28,\n",
              " 46,\n",
              " 16,\n",
              " 5,\n",
              " 63,\n",
              " 46,\n",
              " 34,\n",
              " 20,\n",
              " 26,\n",
              " 53,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 53,\n",
              " 32,\n",
              " 28,\n",
              " 12,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 28,\n",
              " 46,\n",
              " 38,\n",
              " 38,\n",
              " 46,\n",
              " 10,\n",
              " 28,\n",
              " 41,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 10,\n",
              " 46,\n",
              " 5,\n",
              " 5,\n",
              " 46,\n",
              " 5,\n",
              " 63,\n",
              " 10,\n",
              " 12,\n",
              " 3,\n",
              " 47,\n",
              " 3,\n",
              " 10,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 32,\n",
              " 5,\n",
              " 28,\n",
              " 32,\n",
              " 10,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 12,\n",
              " 41,\n",
              " 16,\n",
              " 3,\n",
              " 34,\n",
              " 5,\n",
              " 64,\n",
              " 63,\n",
              " 5,\n",
              " 41,\n",
              " 10,\n",
              " 12,\n",
              " 63,\n",
              " 3,\n",
              " 5,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 3,\n",
              " 14,\n",
              " 32,\n",
              " 38,\n",
              " 32,\n",
              " 28,\n",
              " 35,\n",
              " 63,\n",
              " 28,\n",
              " 26,\n",
              " 63,\n",
              " 47,\n",
              " 46,\n",
              " 3,\n",
              " 5,\n",
              " 26,\n",
              " 34,\n",
              " 64,\n",
              " 63,\n",
              " 20,\n",
              " 32,\n",
              " 5,\n",
              " 10,\n",
              " 26,\n",
              " 57,\n",
              " 46,\n",
              " 47,\n",
              " 63,\n",
              " 16,\n",
              " 46,\n",
              " 3,\n",
              " 34,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 64,\n",
              " 63,\n",
              " 19,\n",
              " 46,\n",
              " 34,\n",
              " 46,\n",
              " 47,\n",
              " 3,\n",
              " 38,\n",
              " 32,\n",
              " 62,\n",
              " 46,\n",
              " 64,\n",
              " 63,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 38,\n",
              " 46,\n",
              " 3,\n",
              " 47,\n",
              " 34,\n",
              " 63,\n",
              " 4,\n",
              " 47,\n",
              " 26,\n",
              " 16,\n",
              " 63,\n",
              " 48,\n",
              " 3,\n",
              " 5,\n",
              " 28,\n",
              " 63,\n",
              " 46,\n",
              " 61,\n",
              " 48,\n",
              " 46,\n",
              " 47,\n",
              " 32,\n",
              " 46,\n",
              " 34,\n",
              " 10,\n",
              " 46,\n",
              " 49,\n",
              " 63,\n",
              " 39,\n",
              " 32,\n",
              " 34,\n",
              " 10,\n",
              " 46,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 20,\n",
              " 46,\n",
              " 57,\n",
              " 46,\n",
              " 38,\n",
              " 26,\n",
              " 48,\n",
              " 16,\n",
              " 46,\n",
              " 34,\n",
              " 28,\n",
              " 63,\n",
              " 26,\n",
              " 4,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 20,\n",
              " 32,\n",
              " 19,\n",
              " 32,\n",
              " 28,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 41,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 63,\n",
              " 32,\n",
              " 34,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 63,\n",
              " 27,\n",
              " 51,\n",
              " 52,\n",
              " 43,\n",
              " 5,\n",
              " 64,\n",
              " 63,\n",
              " 32,\n",
              " 28,\n",
              " 63,\n",
              " 12,\n",
              " 3,\n",
              " 5,\n",
              " 63,\n",
              " 14,\n",
              " 46,\n",
              " 46,\n",
              " 34,\n",
              " 63,\n",
              " 20,\n",
              " 46,\n",
              " 16,\n",
              " 26,\n",
              " 34,\n",
              " 5,\n",
              " 28,\n",
              " 47,\n",
              " 3,\n",
              " 28,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 3,\n",
              " 28,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 41,\n",
              " 28,\n",
              " 46,\n",
              " 47,\n",
              " 5,\n",
              " 63,\n",
              " 10,\n",
              " 3,\n",
              " 34,\n",
              " 63,\n",
              " 14,\n",
              " 46,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 19,\n",
              " 47,\n",
              " 3,\n",
              " 16,\n",
              " 16,\n",
              " 46,\n",
              " 20,\n",
              " 63,\n",
              " 28,\n",
              " 26,\n",
              " 63,\n",
              " 10,\n",
              " 3,\n",
              " 47,\n",
              " 47,\n",
              " 35,\n",
              " 63,\n",
              " 26,\n",
              " 41,\n",
              " 28,\n",
              " 63,\n",
              " 57,\n",
              " 46,\n",
              " 47,\n",
              " 35,\n",
              " 63,\n",
              " 10,\n",
              " 26,\n",
              " 16,\n",
              " 48,\n",
              " 38,\n",
              " 46,\n",
              " 61,\n",
              " 63,\n",
              " 28,\n",
              " 3,\n",
              " 5,\n",
              " 54,\n",
              " 5,\n",
              " 23,\n",
              " 3,\n",
              " 5,\n",
              " 64,\n",
              " 63,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 46,\n",
              " 61,\n",
              " 3,\n",
              " 16,\n",
              " 48,\n",
              " 38,\n",
              " 46,\n",
              " 64,\n",
              " 63,\n",
              " 20,\n",
              " 32,\n",
              " 5,\n",
              " 10,\n",
              " 26,\n",
              " 57,\n",
              " 46,\n",
              " 47,\n",
              " 32,\n",
              " 34,\n",
              " 19,\n",
              " 63,\n",
              " 48,\n",
              " 47,\n",
              " 26,\n",
              " 26,\n",
              " 4,\n",
              " 5,\n",
              " 63,\n",
              " 4,\n",
              " 26,\n",
              " 47,\n",
              " 63,\n",
              " 16,\n",
              " 3,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 16,\n",
              " 3,\n",
              " 28,\n",
              " 32,\n",
              " 10,\n",
              " 3,\n",
              " 38,\n",
              " 63,\n",
              " 28,\n",
              " 12,\n",
              " 46,\n",
              " 26,\n",
              " 47,\n",
              " 46,\n",
              " 16,\n",
              " 5,\n",
              " 63,\n",
              " ...]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F2UFBUgfn-wJ",
        "colab_type": "code",
        "outputId": "67dc5138-c776-478a-b12d-42fa42ee87bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "print(f'Total sequence length of the text: {len(int_text)}')\n",
        "print(f'First 20 integers: {int_text[:20]}')\n",
        "print(f'First 20 characters: {tosentence(int_text[:20])}')"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total sequence length of the text: 9792\n",
            "First 20 integers: [25, 47, 28, 32, 4, 32, 10, 32, 3, 38, 63, 32, 34, 28, 46, 38, 38, 32, 19, 46]\n",
            "First 20 characters: Artificial intellige\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3PF8ysC8n-so",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def one_hot(ints, vocab_length = vocab_length):\n",
        "    outputs = nd.zeros((len(ints), vocab_length))\n",
        "    for i, idx in enumerate(ints):\n",
        "        outputs[i, idx] = 1.0\n",
        "    return outputs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jSNFuFern-rE",
        "colab_type": "code",
        "outputId": "9a7bfa19-b6a4-40b4-e31e-bfbdade322a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 218
        }
      },
      "source": [
        "print('Before one hot:')\n",
        "print(int_text[:20])\n",
        "print('After one hot:')\n",
        "print(one_hot(int_text[:20]))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Before one hot:\n",
            "[25, 47, 28, 32, 4, 32, 10, 32, 3, 38, 63, 32, 34, 28, 46, 38, 38, 32, 19, 46]\n",
            "After one hot:\n",
            "\n",
            "[[0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]]\n",
            "<NDArray 20x67 @cpu(0)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VeOFs9OdqDAR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def onehot_to_text(embeddings):\n",
        "    outputs = \"\"\n",
        "    ints = nd.argmax(embeddings, axis = 1).asnumpy()\n",
        "    for i in ints:\n",
        "        outputs += int_to_char[i]\n",
        "    return outputs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uX0Ut6_dqaRD",
        "colab_type": "code",
        "outputId": "4ad53d37-867f-46c0-e242-88213e3c0f26",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "source": [
        "onehot = one_hot(int_text[:20])\n",
        "print('First 20 characters in Onehot:')\n",
        "print(onehot)\n",
        "print('\\nAfter converting First 20 one hot embeddings to text:')\n",
        "print(f'\"{onehot_to_text(onehot)}\"')"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First 20 characters in Onehot:\n",
            "\n",
            "[[0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]]\n",
            "<NDArray 20x67 @cpu(0)>\n",
            "\n",
            "After converting First 20 one hot embeddings to text:\n",
            "\"Artificial intellige\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1cIWC8aq0s5",
        "colab_type": "text"
      },
      "source": [
        "##Building the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihhJIxcsq4f1",
        "colab_type": "code",
        "outputId": "b331f614-9b83-4b51-8da7-e64cc4d7c0d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "seq_length = 32\n",
        "num_samples = (len(int_text))//seq_length\n",
        "features = one_hot(int_text[: seq_length * num_samples]).reshape((num_samples, seq_length, vocab_length))\n",
        "features.shape"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(306, 32, 67)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z087k0tkq4bU",
        "colab_type": "code",
        "outputId": "12d26935-6c69-404a-80d6-1e8e85066c6a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "output = onehot_to_text(features[0])\n",
        "print(output)\n",
        "print(f'Sequence length: {len(output)}')"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Artificial intelligence (AI) is \n",
            "Sequence length: 32\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MLDVN4GgSD55",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def reverse(sentence):\n",
        "    words = sentence.split(' ')\n",
        "    words.reverse()\n",
        "    return ' '.join(words)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RBmFWs5WSRiX",
        "colab_type": "code",
        "outputId": "7c2db41c-e499-4ba4-93d4-c5a41e342750",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "reverse('This is seq to seq scratch implementation')"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'implementation scratch seq to seq is This'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xotlv5I1R3df",
        "colab_type": "code",
        "outputId": "fcf06af7-803d-4f9f-eeb6-8e44ad57e152",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "labels = nd.array([one_hot(toint(reverse(onehot_to_text(sentence)))).asnumpy() for sentence in features])\n",
        "labels.shape"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(306, 32, 67)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xHbIu4m8UJR1",
        "colab_type": "code",
        "outputId": "7200d9ce-11dd-4be5-c7ea-064dbc91bce8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "print('Original sentence:', onehot_to_text(features[0]))\n",
        "print('Reversed sentence:', onehot_to_text(labels[0]))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original sentence: Artificial intelligence (AI) is \n",
            "Reversed sentence:  is (AI) intelligence Artificial\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QrWeMXJ3q4aF",
        "colab_type": "code",
        "outputId": "50e10026-b4ad-43a8-fc8a-845c6bc0c15c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "batch_size = 8\n",
        "batches = features.shape[0] // batch_size\n",
        "print(f'Number of samples in features: {features.shape[0]}')\n",
        "print(f'Number of batches: {batches}')"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of samples in features: 306\n",
            "Number of batches: 38\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U4TLvbZ5MZY2",
        "colab_type": "code",
        "outputId": "fd1ec1e8-1066-429c-9f3b-6bc6f94667b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "train_features = features[:batches * batch_size]\n",
        "print(train_features.shape)\n",
        "train_features = train_features.reshape((batch_size, batches, seq_length, vocab_length))\n",
        "print(train_features.shape)\n",
        "train_features = train_features.swapaxes(0, 1)\n",
        "print(train_features.shape)\n",
        "train_features = train_features.swapaxes(1, 2)\n",
        "print(train_features.shape)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(304, 32, 67)\n",
            "(8, 38, 32, 67)\n",
            "(38, 8, 32, 67)\n",
            "(38, 32, 8, 67)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sEkfKumCMZW0",
        "colab_type": "code",
        "outputId": "4ca62744-9098-4760-e930-6b1e8652a8f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "train_labels = labels[:batches * batch_size]\n",
        "print(train_labels.shape)\n",
        "train_labels = train_labels.reshape((batch_size, batches, seq_length, vocab_length))\n",
        "print(train_labels.shape)\n",
        "train_labels = train_labels.swapaxes(0, 1)\n",
        "print(train_labels.shape)\n",
        "train_labels = train_labels.swapaxes(1, 2)\n",
        "print(train_labels.shape)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(304, 32, 67)\n",
            "(8, 38, 32, 67)\n",
            "(38, 8, 32, 67)\n",
            "(38, 32, 8, 67)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QSAYcQzRMZVA",
        "colab_type": "code",
        "outputId": "129f663e-e2b3-4530-e281-21394a9005d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "print(onehot_to_text(train_features[0, :, 0])) # [0(for first batch), :(for all letters in first example), 0(for first example)]\n",
        "print(onehot_to_text(train_labels[0, :, 0])) # [0(for first batch), :(for all letters in first example), 0(for first example)]"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Artificial intelligence (AI) is \n",
            " is (AI) intelligence Artificial\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8TkdY970mKg",
        "colab_type": "text"
      },
      "source": [
        "##Defining Encoder Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_p8RFnKw0qI4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_inputs = vocab_length\n",
        "num_hidden = 512\n",
        "num_outputs = vocab_length\n",
        "\n",
        "eWxz = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "eWxr = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "eWxh = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "\n",
        "eWhz = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "eWhr = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "eWhh = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "\n",
        "ebz = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "ebr = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "ebh = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "\n",
        "eWhy = nd.random.normal(shape = (num_hidden,num_outputs), ctx = ctx) * .001\n",
        "eby = nd.random.normal(shape = num_outputs, ctx = ctx) * .001\n",
        "\n",
        "encoder_params = [eWxz, eWxr, eWxh, eWhz, eWhr, eWhh, ebz, ebr, ebh, eWhy, eby]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Lc4i4w_hHtB",
        "colab_type": "text"
      },
      "source": [
        "##Defining Encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTGOYb7b0qDO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def softmax(linear):\n",
        "    exp = nd.exp((linear - nd.max(linear, axis = 1).reshape((-1, 1))))\n",
        "    partition = nd.sum(exp, axis = 1).reshape((-1, 1))\n",
        "    return exp / partition"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bCRIpoHdhHDo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def encode(inputs, h):\n",
        "    outputs = []\n",
        "    for X in inputs:\n",
        "        z = nd.sigmoid(nd.dot(X, eWxz) + nd.dot(h, eWhz) + ebz)\n",
        "        r = nd.sigmoid(nd.dot(X, eWxr) + nd.dot(h, eWhr) + ebr)\n",
        "        \n",
        "        g = nd.tanh(nd.dot(X, eWxh) + nd.dot(r * h, eWhh) + ebh)\n",
        "        h = z * h + (1 - z) * g\n",
        "\n",
        "        linear = nd.dot(h, eWhy) + eby\n",
        "        output = softmax(linear)\n",
        "        outputs.append(output)\n",
        "    return outputs, h"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yvQbxt9TRrnT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        },
        "outputId": "14559b3b-c534-4522-8265-0136227518a4"
      },
      "source": [
        "print(train_features[0].shape)\n",
        "h = nd.zeros((1, num_hidden)).as_in_context(ctx)\n",
        "encoded, h = encode(train_features[0].as_in_context(ctx), h)\n",
        "print(len(encoded), encoded[0].shape)\n",
        "print(h.shape)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(32, 8, 67)\n",
            "32 (8, 67)\n",
            "(8, 512)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0nIKEoNhA0R",
        "colab_type": "text"
      },
      "source": [
        "##Defining Decoder Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONJqBHBohARI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_inputs = vocab_length\n",
        "num_hidden = 512\n",
        "num_outputs = vocab_length\n",
        "\n",
        "dWxz = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "dWxr = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "dWxh = nd.random.normal(shape = (num_inputs, num_hidden), ctx = ctx) * .001\n",
        "\n",
        "dWhz = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "dWhr = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "dWhh = nd.random.normal(shape = (num_hidden, num_hidden), ctx = ctx) * .001\n",
        "\n",
        "dbz = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "dbr = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "dbh = nd.random.normal(shape = num_hidden, ctx = ctx) * .001\n",
        "\n",
        "dWhy = nd.random.normal(shape = (num_hidden,num_outputs), ctx = ctx) * .001\n",
        "dby = nd.random.normal(shape = num_outputs, ctx = ctx) * .001\n",
        "\n",
        "decoder_params = [dWxz, dWxr, dWxh, dWhz, dWhr, dWhh, dbz, dbr, dbh, dWhy, dby]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Clyi0ZHvh90G",
        "colab_type": "text"
      },
      "source": [
        "## Defining Decoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Xgf_bGU1OJg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def decode(inputs, h):\n",
        "    outputs = []\n",
        "    X = inputs[-1]\n",
        "    for i in range(len(inputs)):\n",
        "        z = nd.sigmoid(nd.dot(X, dWxz) + nd.dot(h, dWhz) + dbz)\n",
        "        r = nd.sigmoid(nd.dot(X, dWxr) + nd.dot(h, dWhr) + dbr)\n",
        "        \n",
        "        g = nd.tanh(nd.dot(X, dWxh) + nd.dot(r * h, dWhh) + dbh)\n",
        "        h = z * h + (1 - z) * g\n",
        "\n",
        "        linear = nd.dot(h, dWhy) + dby\n",
        "        X = softmax(linear)\n",
        "        outputs.append(X)\n",
        "        X = X.argmax(1).one_hot(vocab_length)\n",
        "    return outputs, h"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Efqyl5CuTflY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        },
        "outputId": "40f7653c-0242-4720-cc74-ff2a2eb6d756"
      },
      "source": [
        "decoded, h = decode(encoded, h)\n",
        "print(len(decoded), decoded[0].shape)\n",
        "print(h.shape)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "32 (8, 67)\n",
            "(8, 512)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I7WtIi4gVfKc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def seq_to_seq(inputs, h):\n",
        "    encoded, h = encode(inputs, h)\n",
        "    decoded, _ = decode(encoded, h)\n",
        "    return decoded"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QYz98J1oiYQU",
        "colab_type": "text"
      },
      "source": [
        "##Defining Generator"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ojb_FXh152M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def generate(prefix):\n",
        "    string = ''\n",
        "\n",
        "    int_prefix = [char_to_int[char] for char in prefix]\n",
        "    input_sequence = one_hot(int_prefix).as_in_context(ctx)\n",
        "\n",
        "    h = nd.zeros(shape = (1, num_hidden), ctx = ctx)\n",
        "    \n",
        "    decoded = seq_to_seq(input_sequence, h)\n",
        "    \n",
        "    for d in decoded:\n",
        "        choice = int(d[0].argmax(0).asscalar())\n",
        "        string += int_to_char[choice]\n",
        "    return string"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L9bctEbC6sY6",
        "colab_type": "code",
        "outputId": "c757aa52-4081-492f-b798-370c27a19f9c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(generate(\"Artificial intelligence\"))"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "’’’’’’’’’’’’’’’’’’’’’’’\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJbClx0_1OGd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cross_entropy(pred, label):\n",
        "    return -nd.sum(label * nd.log(pred))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WHjdxaUO_2Vj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def objective(outputs, labels):\n",
        "    total_loss = 0.\n",
        "    for (output, label) in zip(outputs,labels):\n",
        "        total_loss = total_loss + cross_entropy(output, label)\n",
        "    return total_loss / len(outputs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fs_zhF-BXMaa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "params = encoder_params + decoder_params\n",
        "vs = []\n",
        "sqrs = []\n",
        "\n",
        "for param in params:\n",
        "    param.attach_grad()\n",
        "    vs.append(param.zeros_like())\n",
        "    sqrs.append(param.zeros_like())\n",
        "    \n",
        "def adam(t, params, vs = vs, sqrs = sqrs, lr = 0.001):\n",
        "    beta1 = 0.9\n",
        "    beta2 = 0.999\n",
        "    eps = 1e-8\n",
        "\n",
        "    for param, v, sqr in zip(params, vs, sqrs):\n",
        "        grad = param.grad / batch_size\n",
        "\n",
        "        v[:] = beta1 * v + (1. - beta1) * grad\n",
        "        sqr[:] = beta2 * sqr + (1. - beta2) * nd.square(grad + 1e-8)\n",
        "\n",
        "        v_bias_corr = v / (1. - beta1 ** t)\n",
        "        sqr_bias_corr = sqr / (1. - beta2 ** t)\n",
        "\n",
        "        update = lr * v_bias_corr / (nd.sqrt(sqr_bias_corr + 1e-8) + eps)\n",
        "        param[:] = param - update"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YRJTLBco1OEg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sgd(params, lr):\n",
        "    for param in params:\n",
        "        param[:] = param - lr * param.grad"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WlmjfU8B1N_x",
        "colab_type": "code",
        "outputId": "7ffd8751-a184-4463-8e2c-233409dab5a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "epochs = 200\n",
        "learning_rate = .001\n",
        "losses = []\n",
        "\n",
        "t = 0\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    if (epoch + 1) % 50 == 0:\n",
        "        learning_rate = learning_rate / 1.5\n",
        "    state = nd.zeros(shape = (batch_size, num_hidden), ctx = ctx)\n",
        "    for i in range(batches):\n",
        "        t += 1\n",
        "        features = train_features[i].as_in_context(ctx)\n",
        "        labels = train_labels[i].as_in_context(ctx)\n",
        "        with autograd.record():\n",
        "            outputs = seq_to_seq(features, state)\n",
        "            loss = objective(outputs, labels)\n",
        "            loss.backward()\n",
        "        adam(t, params, lr = learning_rate)\n",
        "    losses.append(loss.asscalar())\n",
        "    print(f'\\nEpoch: {epoch} | Loss: {loss.asscalar():.5f}')\n",
        "    print(generate(\"Artificial intelligence\"))\n",
        "    print('-'*50)\n",
        "    print(generate(\"Research in AI has focused chiefly on\"))"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch: 0 | Loss: 24.84329\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 1 | Loss: 24.67238\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 2 | Loss: 24.67748\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 3 | Loss: 24.67402\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 4 | Loss: 24.66671\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 5 | Loss: 24.65792\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 6 | Loss: 24.64633\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 7 | Loss: 24.63495\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 8 | Loss: 24.63571\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 9 | Loss: 24.63094\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 10 | Loss: 24.62269\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 11 | Loss: 24.59300\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 12 | Loss: 24.53072\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 13 | Loss: 24.35965\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 14 | Loss: 24.64026\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 15 | Loss: 24.58506\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 16 | Loss: 24.43198\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 17 | Loss: 24.31618\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 18 | Loss: 24.53993\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 19 | Loss: 24.56241\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 20 | Loss: 24.56851\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 21 | Loss: 24.48960\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 22 | Loss: 24.49474\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 23 | Loss: 24.54252\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 24 | Loss: 24.53773\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 25 | Loss: 24.51336\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 26 | Loss: 24.49743\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 27 | Loss: 24.48391\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 28 | Loss: 24.47135\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 29 | Loss: 24.44433\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 30 | Loss: 24.43177\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 31 | Loss: 24.42158\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 32 | Loss: 24.41117\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 33 | Loss: 24.52290\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 34 | Loss: 24.39367\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 35 | Loss: 24.41075\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 36 | Loss: 24.31936\n",
            "                       \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 37 | Loss: 24.34030\n",
            "  eeee                 \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 38 | Loss: 24.47693\n",
            "peeee                  \n",
            "--------------------------------------------------\n",
            "  e                                  \n",
            "\n",
            "Epoch: 39 | Loss: 24.49640\n",
            " eeeeee                \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 40 | Loss: 24.28419\n",
            "eeeeee                 \n",
            "--------------------------------------------------\n",
            " eeee                                \n",
            "\n",
            "Epoch: 41 | Loss: 24.40454\n",
            "peeeeeee               \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 42 | Loss: 24.35356\n",
            "peeeeeeeee             \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 43 | Loss: 24.25239\n",
            "peeeeeeee              \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 44 | Loss: 24.20752\n",
            "peeeeeeeeee            \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 45 | Loss: 24.17213\n",
            "peeeeeeeee             \n",
            "--------------------------------------------------\n",
            "                                     \n",
            "\n",
            "Epoch: 46 | Loss: 23.96729\n",
            "peeeee                 \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 47 | Loss: 24.00169\n",
            "peeeeee                \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 48 | Loss: 23.95916\n",
            "peeeeee                \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 49 | Loss: 23.84545\n",
            "peeeee                 \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 50 | Loss: 23.71374\n",
            "reeeee                 \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 51 | Loss: 23.60847\n",
            "ieeee                  \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 52 | Loss: 23.49284\n",
            "ieeeee                 \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 53 | Loss: 23.37577\n",
            "ieeeee                 \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 54 | Loss: 23.28144\n",
            "innnnnn                \n",
            "--------------------------------------------------\n",
            "a                                    \n",
            "\n",
            "Epoch: 55 | Loss: 23.34014\n",
            "iniennn                \n",
            "--------------------------------------------------\n",
            "aa a                                 \n",
            "\n",
            "Epoch: 56 | Loss: 22.88268\n",
            "ininnnnnn              \n",
            "--------------------------------------------------\n",
            "aa a                                 \n",
            "\n",
            "Epoch: 57 | Loss: 22.71820\n",
            "ininnnnnnin            \n",
            "--------------------------------------------------\n",
            "a  sa                                \n",
            "\n",
            "Epoch: 58 | Loss: 22.60781\n",
            "inlnnnnnninin          \n",
            "--------------------------------------------------\n",
            "oh sa  ss                            \n",
            "\n",
            "Epoch: 59 | Loss: 22.39802\n",
            "ininnnnnnininin i      \n",
            "--------------------------------------------------\n",
            "aa sae  ss                           \n",
            "\n",
            "Epoch: 60 | Loss: 22.53920\n",
            "ininnnnnnnininin i     \n",
            "--------------------------------------------------\n",
            "in sae   ss                          \n",
            "\n",
            "Epoch: 61 | Loss: 22.57201\n",
            "inleennnnnininin i     \n",
            "--------------------------------------------------\n",
            "aa caas  ses                         \n",
            "\n",
            "Epoch: 62 | Loss: 22.29249\n",
            "inlnlnnnninin i        \n",
            "--------------------------------------------------\n",
            "in aaaae se s                        \n",
            "\n",
            "Epoch: 63 | Loss: 22.14672\n",
            "inlnlnnnniiiii         \n",
            "--------------------------------------------------\n",
            "an caase ss                          \n",
            "\n",
            "Epoch: 64 | Loss: 22.06280\n",
            "inlnlnnnniiiii         \n",
            "--------------------------------------------------\n",
            "in ceesee                            \n",
            "\n",
            "Epoch: 65 | Loss: 22.02580\n",
            "inlnllnnn iiiii        \n",
            "--------------------------------------------------\n",
            "in ceccss                            \n",
            "\n",
            "Epoch: 66 | Loss: 22.22825\n",
            "intnnnniniiii i i  i  i\n",
            "--------------------------------------------------\n",
            "in aaase s                           \n",
            "\n",
            "Epoch: 67 | Loss: 21.70392\n",
            "inlnnnnnniniiii i  i   \n",
            "--------------------------------------------------\n",
            "in catse                             \n",
            "\n",
            "Epoch: 68 | Loss: 21.94471\n",
            "intnnnninii i ii i i  i\n",
            "--------------------------------------------------\n",
            "ie chese                             \n",
            "\n",
            "Epoch: 69 | Loss: 21.78730\n",
            "indnnnnnnnnnininininini\n",
            "--------------------------------------------------\n",
            "in catse                             \n",
            "\n",
            "Epoch: 70 | Loss: 21.41565\n",
            "intnnnnnnnniniiii  i  i\n",
            "--------------------------------------------------\n",
            "in catses                            \n",
            "\n",
            "Epoch: 71 | Loss: 21.34496\n",
            "intnlinnn inininin i i \n",
            "--------------------------------------------------\n",
            "in catuet                            \n",
            "\n",
            "Epoch: 72 | Loss: 21.12307\n",
            "intnnnnnnininininininin\n",
            "--------------------------------------------------\n",
            "in aaasess                           \n",
            "\n",
            "Epoch: 73 | Loss: 21.35871\n",
            "intnliinn inininin inin\n",
            "--------------------------------------------------\n",
            "in chase                             \n",
            "\n",
            "Epoch: 74 | Loss: 21.03351\n",
            "inteliiin  iiiiii ii ii\n",
            "--------------------------------------------------\n",
            "in cetuee  s                         \n",
            "\n",
            "Epoch: 75 | Loss: 29.60205\n",
            "intelinnn iniiii  i i i\n",
            "--------------------------------------------------\n",
            "in thasee  a                         \n",
            "\n",
            "Epoch: 76 | Loss: 21.68072\n",
            "indeleing  ieiiin   in \n",
            "--------------------------------------------------\n",
            "in temuee  he                        \n",
            "\n",
            "Epoch: 77 | Loss: 21.12252\n",
            "intellingniiiiiiniiniii\n",
            "--------------------------------------------------\n",
            "in ceauuse th                        \n",
            "\n",
            "Epoch: 78 | Loss: 21.08176\n",
            "intellingeiiiiii  iiiii\n",
            "--------------------------------------------------\n",
            "in feauis  sh                        \n",
            "\n",
            "Epoch: 79 | Loss: 20.55551\n",
            "intelinggeiiinininiiii \n",
            "--------------------------------------------------\n",
            "in  omuuse ss                        \n",
            "\n",
            "Epoch: 80 | Loss: 20.75091\n",
            "inteliningiiiiiiiiiiiii\n",
            "--------------------------------------------------\n",
            "in feauee  ss                        \n",
            "\n",
            "Epoch: 81 | Loss: 20.51998\n",
            "intelenngee  ininiiniin\n",
            "--------------------------------------------------\n",
            "in aemuse the                        \n",
            "\n",
            "Epoch: 82 | Loss: 20.32029\n",
            "intelenlege einaiii  in\n",
            "--------------------------------------------------\n",
            "in temuet  ss                        \n",
            "\n",
            "Epoch: 83 | Loss: 20.12387\n",
            "inteleling  eiiin ii  i\n",
            "--------------------------------------------------\n",
            "in aemuee th                         \n",
            "\n",
            "Epoch: 84 | Loss: 19.92932\n",
            "inteleline  ie iiiiiiii\n",
            "--------------------------------------------------\n",
            "in teruee th                         \n",
            "\n",
            "Epoch: 85 | Loss: 19.40551\n",
            "intellinneeeiiiii iiiii\n",
            "--------------------------------------------------\n",
            "in cumuee  h                         \n",
            "\n",
            "Epoch: 86 | Loss: 19.69361\n",
            "intelliine ieiiiin iiii\n",
            "--------------------------------------------------\n",
            "in cumuie  s                         \n",
            "\n",
            "Epoch: 87 | Loss: 19.31022\n",
            "Intelelinge  i iiiiiiii\n",
            "--------------------------------------------------\n",
            "in seruee  ss                        \n",
            "\n",
            "Epoch: 88 | Loss: 19.03918\n",
            "intellinge  i iiiiiiiii\n",
            "--------------------------------------------------\n",
            "in fumuut  s                         \n",
            "\n",
            "Epoch: 89 | Loss: 19.18756\n",
            "intelliigee     iiiiiii\n",
            "--------------------------------------------------\n",
            "in cumuute  s                        \n",
            "\n",
            "Epoch: 90 | Loss: 19.57896\n",
            "Intellnlggee    iiiiiii\n",
            "--------------------------------------------------\n",
            "in cerpeee  s                        \n",
            "\n",
            "Epoch: 91 | Loss: 19.07737\n",
            "Intellelegee   iaiiiiii\n",
            "--------------------------------------------------\n",
            "in ceruuee  s                ee      \n",
            "\n",
            "Epoch: 92 | Loss: 18.62447\n",
            "Intelliggeeeeiiiiiiiiii\n",
            "--------------------------------------------------\n",
            "in ceruue  ssss               eeeee  \n",
            "\n",
            "Epoch: 93 | Loss: 18.22796\n",
            "Intellllegee  iaiiiiiii\n",
            "--------------------------------------------------\n",
            "an reruie  AI  s                     \n",
            "\n",
            "Epoch: 94 | Loss: 17.75922\n",
            "Intellegeeee    iiiiiii\n",
            "--------------------------------------------------\n",
            "sn ceruue  sn                        \n",
            "\n",
            "Epoch: 95 | Loss: 17.59847\n",
            "Intellilegee   iaiiiiii\n",
            "--------------------------------------------------\n",
            "sn eeruue  sss                       \n",
            "\n",
            "Epoch: 96 | Loss: 18.06679\n",
            "Intelligeee  i iiiiiiii\n",
            "--------------------------------------------------\n",
            "in eeruee  has                       \n",
            "\n",
            "Epoch: 97 | Loss: 18.78783\n",
            "Indeeleleeee  i iiiiiii\n",
            "--------------------------------------------------\n",
            "se cersue  sss   s                   \n",
            "\n",
            "Epoch: 98 | Loss: 17.83536\n",
            "Inteeleleee   i iiiiiii\n",
            "--------------------------------------------------\n",
            "an curuuee sas                       \n",
            "\n",
            "Epoch: 99 | Loss: 17.54344\n",
            "Intelleleeeee iiaiiiiii\n",
            "--------------------------------------------------\n",
            "in euuuee  hs                        \n",
            "\n",
            "Epoch: 100 | Loss: 16.90441\n",
            "Intellegeeee  iaiiiiiii\n",
            "--------------------------------------------------\n",
            "in funuue  has                       \n",
            "\n",
            "Epoch: 101 | Loss: 16.52453\n",
            "Intellegeeee  iiiiiiiii\n",
            "--------------------------------------------------\n",
            "sn suauue  fon  s                    \n",
            "\n",
            "Epoch: 102 | Loss: 16.50079\n",
            "Intellegeeee  inaiiiiii\n",
            "--------------------------------------------------\n",
            "gn aumpud  fon  ss                   \n",
            "\n",
            "Epoch: 103 | Loss: 16.31366\n",
            "Intelleleeeece i aiiiii\n",
            "--------------------------------------------------\n",
            "in auauure  wo  s   s                \n",
            "\n",
            "Epoch: 104 | Loss: 17.04798\n",
            "Intelligeeee in iiiiiii\n",
            "--------------------------------------------------\n",
            "an sunuud  in  he                    \n",
            "\n",
            "Epoch: 105 | Loss: 16.29687\n",
            "Intelligeee  in iiiiii \n",
            "--------------------------------------------------\n",
            "an cursud  fo  as                    \n",
            "\n",
            "Epoch: 106 | Loss: 16.13713\n",
            "Intelligeeee eraiiiii  \n",
            "--------------------------------------------------\n",
            "in auuped  fo  ass              eee  \n",
            "\n",
            "Epoch: 107 | Loss: 16.08144\n",
            "Intelligence  iaiiiiii \n",
            "--------------------------------------------------\n",
            "an puapute won  ass                  \n",
            "\n",
            "Epoch: 108 | Loss: 16.43732\n",
            "Intelligencec Ai  itiii\n",
            "--------------------------------------------------\n",
            "gn euauute win  uss                  \n",
            "\n",
            "Epoch: 109 | Loss: 16.58910\n",
            "Intelligence  i iiiiiia\n",
            "--------------------------------------------------\n",
            "gn erauun  Ao  fo   te               \n",
            "\n",
            "Epoch: 110 | Loss: 15.86194\n",
            "intelligenee er iiiiil \n",
            "--------------------------------------------------\n",
            "gn eunpud  in  bus                   \n",
            "\n",
            "Epoch: 111 | Loss: 14.93285\n",
            "Intelligenc  ir iiiii  \n",
            "--------------------------------------------------\n",
            "gn puasuss insuues                   \n",
            "\n",
            "Epoch: 112 | Loss: 14.80074\n",
            "Intelligence  iatiiiii \n",
            "--------------------------------------------------\n",
            "gn puapute wint  as                  \n",
            "\n",
            "Epoch: 113 | Loss: 14.86630\n",
            "Intelligence  in iitiil\n",
            "--------------------------------------------------\n",
            "gn auasute wnf aume                  \n",
            "\n",
            "Epoch: 114 | Loss: 14.73453\n",
            "intelligence er  iatial\n",
            "--------------------------------------------------\n",
            "gn cursun  winllle                   \n",
            "\n",
            "Epoch: 115 | Loss: 14.43135\n",
            "Intelligence Ara iiiia \n",
            "--------------------------------------------------\n",
            "gn cursent win   ee              eeee\n",
            "\n",
            "Epoch: 116 | Loss: 14.13458\n",
            "IntelligencMArrt aatial\n",
            "--------------------------------------------------\n",
            "gn obapues  ant                      \n",
            "\n",
            "Epoch: 117 | Loss: 13.72648\n",
            "intelligenceceratiiiill\n",
            "--------------------------------------------------\n",
            "gn auasune  and         s ss s    a  \n",
            "\n",
            "Epoch: 118 | Loss: 13.89568\n",
            "intellilence  ra iitial\n",
            "--------------------------------------------------\n",
            "gn fursuns win  as   s s      a  aeer\n",
            "\n",
            "Epoch: 119 | Loss: 13.81786\n",
            "intelligence Ara fitial\n",
            "--------------------------------------------------\n",
            "gn corsune win  uus               rer\n",
            "\n",
            "Epoch: 120 | Loss: 13.29452\n",
            "intelligence  ra fitial\n",
            "--------------------------------------------------\n",
            "gn oaamunt win  o   s                \n",
            "\n",
            "Epoch: 121 | Loss: 13.19635\n",
            "Intelligence ratificial\n",
            "--------------------------------------------------\n",
            "gn caasute wan  os                   \n",
            "\n",
            "Epoch: 122 | Loss: 13.77694\n",
            "Intelligence  ra ficial\n",
            "--------------------------------------------------\n",
            "go aaasud, wan  o  s               r \n",
            "\n",
            "Epoch: 123 | Loss: 13.27242\n",
            "intelligence  riffitill\n",
            "--------------------------------------------------\n",
            "go cormutt wan                       \n",
            "\n",
            "Epoch: 124 | Loss: 13.95362\n",
            "Intelligence Ardfficial\n",
            "--------------------------------------------------\n",
            "gn cormute wan  o   ase              \n",
            "\n",
            "Epoch: 125 | Loss: 13.44120\n",
            "intelligence Arffficial\n",
            "--------------------------------------------------\n",
            "gn aaahed, win aaes    tes           \n",
            "\n",
            "Epoch: 126 | Loss: 13.13175\n",
            "intelligence Artificial\n",
            "--------------------------------------------------\n",
            "go auathed wwn  ores                r\n",
            "\n",
            "Epoch: 127 | Loss: 13.27525\n",
            "intelligence  ratfitial\n",
            "--------------------------------------------------\n",
            "go saakede wwa  ar                   \n",
            "\n",
            "Epoch: 128 | Loss: 12.31177\n",
            "intelligence  an ffiill\n",
            "--------------------------------------------------\n",
            "gn orasedd wwn  o  o               r \n",
            "\n",
            "Epoch: 129 | Loss: 12.16019\n",
            "intelligence Ard fitill\n",
            "--------------------------------------------------\n",
            "gn orasedd wan  s  s   ee            \n",
            "\n",
            "Epoch: 130 | Loss: 12.31031\n",
            "intelligence  iitiicill\n",
            "--------------------------------------------------\n",
            "go aaaseds wan                       \n",
            "\n",
            "Epoch: 131 | Loss: 12.49773\n",
            "intelligence  rifficicl\n",
            "--------------------------------------------------\n",
            "gn arashed waas AI                   \n",
            "\n",
            "Epoch: 132 | Loss: 11.56542\n",
            "intelligence Ard ficial\n",
            "--------------------------------------------------\n",
            "gn orasees wans AI  hhee            r\n",
            "\n",
            "Epoch: 133 | Loss: 11.49457\n",
            "intelligence ird ficial\n",
            "--------------------------------------------------\n",
            "gn prasens wans  s  h              r \n",
            "\n",
            "Epoch: 134 | Loss: 12.06283\n",
            "intelligence  rd ficial\n",
            "--------------------------------------------------\n",
            "gn prasens wans                     a\n",
            "\n",
            "Epoch: 135 | Loss: 10.60509\n",
            "intelligence  adtiiicia\n",
            "--------------------------------------------------\n",
            "gn prasens wans  s                rr \n",
            "\n",
            "Epoch: 136 | Loss: 10.75187\n",
            "Intelligence Ard fitial\n",
            "--------------------------------------------------\n",
            "go srasens waas  a                  r\n",
            "\n",
            "Epoch: 137 | Loss: 11.14866\n",
            "Intelligence Ardificial\n",
            "--------------------------------------------------\n",
            "go prasen  wins  A plle      fff rrrr\n",
            "\n",
            "Epoch: 138 | Loss: 10.49068\n",
            "intelligence Ardificial\n",
            "--------------------------------------------------\n",
            "go arapen  aan  A  h le           re \n",
            "\n",
            "Epoch: 139 | Loss: 11.51447\n",
            "intelligence Ardificial\n",
            "--------------------------------------------------\n",
            "go prapen  aaal     ll         ararrr\n",
            "\n",
            "Epoch: 140 | Loss: 11.28568\n",
            "Intelligence  rdificial\n",
            "--------------------------------------------------\n",
            "gn paasens wan   ll\n",
            "\n",
            "II           ar \n",
            "\n",
            "Epoch: 141 | Loss: 10.85071\n",
            "Intelligence  rd ficial\n",
            "--------------------------------------------------\n",
            "gn puasest win iAI. heere     fff ree\n",
            "\n",
            "Epoch: 142 | Loss: 11.12847\n",
            "intelligence Artificial\n",
            "--------------------------------------------------\n",
            "gn prasest waa   AI  tll             \n",
            "\n",
            "Epoch: 143 | Loss: 11.07998\n",
            "intelligence iasificial\n",
            "--------------------------------------------------\n",
            "gn prasens waas  a  al   r          r\n",
            "\n",
            "Epoch: 144 | Loss: 10.45396\n",
            "intelligence  adificial\n",
            "--------------------------------------------------\n",
            "gn paasute waas AI   arre            \n",
            "\n",
            "Epoch: 145 | Loss: 9.48877\n",
            "Intelligence Aid ficial\n",
            "--------------------------------------------------\n",
            "gn paasute waas AI  t rere           \n",
            "\n",
            "Epoch: 146 | Loss: 8.87414\n",
            "intelligence iad ficicl\n",
            "--------------------------------------------------\n",
            "gn prasede wans AI  trrrar           \n",
            "\n",
            "Epoch: 147 | Loss: 8.57479\n",
            "intelligence  ad ficial\n",
            "--------------------------------------------------\n",
            "go paaseth waas AI   aarerest        \n",
            "\n",
            "Epoch: 148 | Loss: 8.52383\n",
            "intelligence Ardificica\n",
            "--------------------------------------------------\n",
            "go praseds waas AI  trararte t       \n",
            "\n",
            "Epoch: 149 | Loss: 8.11486\n",
            "intelligence Ard ficica\n",
            "--------------------------------------------------\n",
            "go prasets wans AI hh rmasst       rr\n",
            "\n",
            "Epoch: 150 | Loss: 7.57973\n",
            "intelligence ind ficial\n",
            "--------------------------------------------------\n",
            "gn prasest wans AI hhrer  s       rrr\n",
            "\n",
            "Epoch: 151 | Loss: 7.61604\n",
            "intelligence  adificial\n",
            "--------------------------------------------------\n",
            "go praseds aas  AI th rm s   ss  rrrr\n",
            "\n",
            "Epoch: 152 | Loss: 7.12822\n",
            "intelligence  ndificicl\n",
            "--------------------------------------------------\n",
            "go prasest wall AI  o mmt    s   arrr\n",
            "\n",
            "Epoch: 153 | Loss: 6.91965\n",
            "intelligence  artficial\n",
            "--------------------------------------------------\n",
            "go praset  aals AI th mmmss   s   rre\n",
            "\n",
            "Epoch: 154 | Loss: 7.44745\n",
            "intelligence  artficial\n",
            "--------------------------------------------------\n",
            "go projet  aal  s  h mem s       rrra\n",
            "\n",
            "Epoch: 155 | Loss: 7.83300\n",
            "intelligence  ad ficica\n",
            "--------------------------------------------------\n",
            "go prasent wal  AI t mampts   s  fn r\n",
            "\n",
            "Epoch: 156 | Loss: 6.87155\n",
            "intelligence  artficial\n",
            "--------------------------------------------------\n",
            "go prasent win AI  t memstt  s   f  r\n",
            "\n",
            "Epoch: 157 | Loss: 6.70751\n",
            "intelligence iad ficica\n",
            "--------------------------------------------------\n",
            "go prasent wis  AI t ramptt  rs  s  f\n",
            "\n",
            "Epoch: 158 | Loss: 6.85327\n",
            "intelligence indificicl\n",
            "--------------------------------------------------\n",
            "go prasens was  AI frrrt    s      rr\n",
            "\n",
            "Epoch: 159 | Loss: 7.16044\n",
            "intelligence  artficiaa\n",
            "--------------------------------------------------\n",
            "go prasent wanl AI forttt    s  karrr\n",
            "\n",
            "Epoch: 160 | Loss: 6.60990\n",
            "intelligence Ardificial\n",
            "--------------------------------------------------\n",
            "go prasens wans AI torrtt r    ss    \n",
            "\n",
            "Epoch: 161 | Loss: 6.39027\n",
            "intelligence Aartficial\n",
            "--------------------------------------------------\n",
            "gn prasens aans    ro   I    aaarrrrr\n",
            "\n",
            "Epoch: 162 | Loss: 6.18523\n",
            "intelligence Aarificial\n",
            "--------------------------------------------------\n",
            "go prasent wins A  fot      kk aarrrr\n",
            "\n",
            "Epoch: 163 | Loss: 5.72408\n",
            "intelligence Aartficial\n",
            "--------------------------------------------------\n",
            "go prasent wans is tirtt   as  karrrr\n",
            "\n",
            "Epoch: 164 | Loss: 5.93747\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "go prasent wan  s  fottt    as kk rrr\n",
            "\n",
            "Epoch: 165 | Loss: 5.65268\n",
            "intelligence Aarificial\n",
            "--------------------------------------------------\n",
            "go prasent win is  tirtthurt ss s   r\n",
            "\n",
            "Epoch: 166 | Loss: 5.22311\n",
            "intelligence  artiiciia\n",
            "--------------------------------------------------\n",
            "go prasent wans  s  htat r  ra  nn  r\n",
            "\n",
            "Epoch: 167 | Loss: 5.07329\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "oo pursett wins AI hhrmatt  ass  s  r\n",
            "\n",
            "Epoch: 168 | Loss: 5.75737\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "go prasett wans AI hrrmat   as  s  fr\n",
            "\n",
            "Epoch: 169 | Loss: 5.09734\n",
            "intelligence  astificia\n",
            "--------------------------------------------------\n",
            "oo suasett wass AI  trmpl   as  ant r\n",
            "\n",
            "Epoch: 170 | Loss: 4.99808\n",
            "intelligence intificial\n",
            "--------------------------------------------------\n",
            "oo prasen  aass I  frmpl   as   n  rr\n",
            "\n",
            "Epoch: 171 | Loss: 5.04350\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "oo prasent hals iI h   muu   ss  k  r\n",
            "\n",
            "Epoch: 172 | Loss: 4.93260\n",
            "intelligence  irtiacial\n",
            "--------------------------------------------------\n",
            "oo prasen  als oI  h    u    nn  r rr\n",
            "\n",
            "Epoch: 173 | Loss: 4.33201\n",
            "intelligence iartiicial\n",
            "--------------------------------------------------\n",
            "oo prasent wass AI ot mpt  sss int  r\n",
            "\n",
            "Epoch: 174 | Loss: 5.06039\n",
            "intelligence  artiicicl\n",
            "--------------------------------------------------\n",
            "go prasent winh io tith   kk  rrrriii\n",
            "\n",
            "Epoch: 175 | Loss: 4.68399\n",
            "intelligence  artficicl\n",
            "--------------------------------------------------\n",
            "gn prasent wis iA Rtthaltt rssiiku rr\n",
            "\n",
            "Epoch: 176 | Loss: 4.56605\n",
            "intelligence  artficice\n",
            "--------------------------------------------------\n",
            "go prasent win iI Rormalt trts ns  rr\n",
            "\n",
            "Epoch: 177 | Loss: 4.54633\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "oo prasent was  i  ittt  t  aas rrrir\n",
            "\n",
            "Epoch: 178 | Loss: 3.76993\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "go prasent wass AI otrttt   as kknrrr\n",
            "\n",
            "Epoch: 179 | Loss: 3.67521\n",
            "intelligence Aartficial\n",
            "--------------------------------------------------\n",
            "go prasent has  II tormatt aass fff r\n",
            "\n",
            "Epoch: 180 | Loss: 3.38096\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "go prasent aass II form   aass kn rrr\n",
            "\n",
            "Epoch: 181 | Loss: 4.06698\n",
            "intelligence  artiiciaa\n",
            "--------------------------------------------------\n",
            "go prasen  als  I  for   aas skn rrfr\n",
            "\n",
            "Epoch: 182 | Loss: 3.84253\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ansi o  th m—— as arr rrri\n",
            "\n",
            "Epoch: 183 | Loss: 3.18584\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais is Rtrthuus ass kn rrr\n",
            "\n",
            "Epoch: 184 | Loss: 3.40673\n",
            "intelligence iartficial\n",
            "--------------------------------------------------\n",
            "go prasen  als is ttttt   asaskrrrrri\n",
            "\n",
            "Epoch: 185 | Loss: 3.07047\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "go prasent aas  AI tt mptt  as rint r\n",
            "\n",
            "Epoch: 186 | Loss: 2.79581\n",
            "intelligence iartiicial\n",
            "--------------------------------------------------\n",
            "go prasen  aas  I  ootttt aassasr rrr\n",
            "\n",
            "Epoch: 187 | Loss: 2.46658\n",
            "intelligence iartficial\n",
            "--------------------------------------------------\n",
            "on prasent aas  I  fo  mt  ass rar rr\n",
            "\n",
            "Epoch: 188 | Loss: 2.55933\n",
            "intelligence iirtiacial\n",
            "--------------------------------------------------\n",
            "on prasent aas  Io ft  m ssss  rn rrf\n",
            "\n",
            "Epoch: 189 | Loss: 3.23013\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "on prasent aas  Io tt  mss ass rrr ri\n",
            "\n",
            "Epoch: 190 | Loss: 2.76055\n",
            "intelligence iartiicial\n",
            "--------------------------------------------------\n",
            "go prasent aash oo ttt ss ,a,,arrriwi\n",
            "\n",
            "Epoch: 191 | Loss: 3.02750\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  oo tt mt —sassknn  ff\n",
            "\n",
            "Epoch: 192 | Loss: 2.51313\n",
            "intelligence iirtiicial\n",
            "--------------------------------------------------\n",
            "Ao prasent wish io tttttt   akkndrow,\n",
            "\n",
            "Epoch: 193 | Loss: 2.51810\n",
            "intelligence iartficial\n",
            "--------------------------------------------------\n",
            "gn prasent was  I  totttttsss ararrrf\n",
            "\n",
            "Epoch: 194 | Loss: 2.30951\n",
            "intelligence iartiicial\n",
            "--------------------------------------------------\n",
            "on prasent aas  s  ottt  aas ard rifo\n",
            "\n",
            "Epoch: 195 | Loss: 1.85263\n",
            "intelligence iartiicial\n",
            "--------------------------------------------------\n",
            "go prasen  aas  I  ot m  aaass rn  rr\n",
            "\n",
            "Epoch: 196 | Loss: 1.89299\n",
            "intelligence dartificia\n",
            "--------------------------------------------------\n",
            "on prasen  alsi I  tr mt aaass kn rrf\n",
            "\n",
            "Epoch: 197 | Loss: 2.38829\n",
            "intelligence  istiicial\n",
            "--------------------------------------------------\n",
            "oo prased halsis  R mith   as   rrrrr\n",
            "\n",
            "Epoch: 198 | Loss: 4.65027\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "on prasen has io  Rmm9sss  frr  rrtio\n",
            "\n",
            "Epoch: 199 | Loss: 2.50626\n",
            "intelligence dartificia\n",
            "--------------------------------------------------\n",
            "on parsett wish ss s hetas ss  rrr  a\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DH6XJsGvMZTG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8a0142bf-ae12-4088-8424-f9c5e8151060"
      },
      "source": [
        "generate('All but the simplest human behaviour is ascribed to intelligence')"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'intelligence displipet  th thate teeteeste the eccence  tte hee '"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KwxV2SFn88H9",
        "colab_type": "text"
      },
      "source": [
        "##Lets have some more training here"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hvm8Yr5UMZRA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "69742a00-5383-4e7e-8617-a33dc961069a"
      },
      "source": [
        "epochs = 100\n",
        "learning_rate = .001/1.5/1.5/1.5\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    if (epoch + 1) % 50 == 0:\n",
        "        learning_rate = learning_rate / 1.5\n",
        "    state = nd.zeros(shape = (batch_size, num_hidden), ctx = ctx)\n",
        "    for i in range(batches):\n",
        "        t += 1\n",
        "        features = train_features[i].as_in_context(ctx)\n",
        "        labels = train_labels[i].as_in_context(ctx)\n",
        "        with autograd.record():\n",
        "            outputs = seq_to_seq(features, state)\n",
        "            loss = objective(outputs, labels)\n",
        "            loss.backward()\n",
        "        adam(t, params, lr = learning_rate)\n",
        "    losses.append(loss.asscalar())\n",
        "    print(f'\\nEpoch: {epoch} | Loss: {loss.asscalar():.5f}')\n",
        "    print(generate(\"Artificial intelligence\"))\n",
        "    print('-'*50)\n",
        "    print(generate(\"Research in AI has focused chiefly on\"))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch: 0 | Loss: 2.48920\n",
            "intelligence  irtificia\n",
            "--------------------------------------------------\n",
            "gn prasent  ins  ss titmmssssss rfn r\n",
            "\n",
            "Epoch: 1 | Loss: 2.80859\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "gn prasett aas  in Rotmtt ass rrnd in\n",
            "\n",
            "Epoch: 2 | Loss: 3.63204\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "gn Srasent wiss  sstthhak,, arrrriinc\n",
            "\n",
            "Epoch: 3 | Loss: 2.29681\n",
            "Intelligence  irtricial\n",
            "--------------------------------------------------\n",
            "gn prasent wass  I  th  sssass, rrrri\n",
            "\n",
            "Epoch: 4 | Loss: 1.92425\n",
            "Intelligence iirtiicial\n",
            "--------------------------------------------------\n",
            "gn Srasens wass  I  th  ssssas,rrrr i\n",
            "\n",
            "Epoch: 5 | Loss: 1.67435\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "gn prasent aash io tth ssssss, rrriii\n",
            "\n",
            "Epoch: 6 | Loss: 1.78172\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  AI Rttrtsss as rar rf\n",
            "\n",
            "Epoch: 7 | Loss: 1.36993\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent wish io ttthas,,aasrrrrici\n",
            "\n",
            "Epoch: 8 | Loss: 1.33910\n",
            "intelligence dirtiacial\n",
            "--------------------------------------------------\n",
            "go prasent aasi oo thim—ssas rarrriic\n",
            "\n",
            "Epoch: 9 | Loss: 0.96399\n",
            "intelligence dirtricial\n",
            "--------------------------------------------------\n",
            "go prasent ansi oo ttim sssas, arrrii\n",
            "\n",
            "Epoch: 10 | Loss: 0.83496\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "gn prasent aash io ttttsssssa, rrriii\n",
            "\n",
            "Epoch: 11 | Loss: 0.75508\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "gn prasent aas  io ttt sssas,,arrriiw\n",
            "\n",
            "Epoch: 12 | Loss: 0.70695\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  io Rtt  ssssarerrriow\n",
            "\n",
            "Epoch: 13 | Loss: 0.67700\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  io tth mssss, arrriii\n",
            "\n",
            "Epoch: 14 | Loss: 0.61832\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  io tth msssss,arrrrii\n",
            "\n",
            "Epoch: 15 | Loss: 0.59432\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  oo tti mssss,,arrriii\n",
            "\n",
            "Epoch: 16 | Loss: 0.59249\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  io Rtt  sssas, arrrii\n",
            "\n",
            "Epoch: 17 | Loss: 0.59190\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ansi s  Rtt ——as, aar rifi\n",
            "\n",
            "Epoch: 18 | Loss: 0.56498\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "An prasent aash io tttt sssa,,arrrrii\n",
            "\n",
            "Epoch: 19 | Loss: 0.49265\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasen  ansi I  tor huaaas  as rrf\n",
            "\n",
            "Epoch: 20 | Loss: 0.54227\n",
            "intelligence districial\n",
            "--------------------------------------------------\n",
            "go prasent aasi o  Rtt —has  aas irfi\n",
            "\n",
            "Epoch: 21 | Loss: 0.52369\n",
            "intelligence dirtricial\n",
            "--------------------------------------------------\n",
            "go prasen  and is RRtrthu aa Ian  rrf\n",
            "\n",
            "Epoch: 22 | Loss: 0.56805\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "gn prasent aas  in Rttttssssa,,arr ii\n",
            "\n",
            "Epoch: 23 | Loss: 0.53702\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasen  als is Rrtrthans  ass rrfi\n",
            "\n",
            "Epoch: 24 | Loss: 0.54297\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  so fti  —sas, aar  ri\n",
            "\n",
            "Epoch: 25 | Loss: 0.52510\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "go prasent ans  s  Rt  massas  rr rrf\n",
            "\n",
            "Epoch: 26 | Loss: 0.63575\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "Ao prasent aash is Rttt——s al, ar iri\n",
            "\n",
            "Epoch: 27 | Loss: 0.57795\n",
            "intelligence distiicial\n",
            "--------------------------------------------------\n",
            "gn prasent wish oo Rtt ss,,aarrriiig,\n",
            "\n",
            "Epoch: 28 | Loss: 0.53776\n",
            "intelligence  irtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  o  Rtttt ——aalInns ir\n",
            "\n",
            "Epoch: 29 | Loss: 0.58231\n",
            "intelligence  artiicial\n",
            "--------------------------------------------------\n",
            "gn prasent aas  s  Roth ———aalIIn  ir\n",
            "\n",
            "Epoch: 30 | Loss: 0.54454\n",
            "intelligence iistiicial\n",
            "--------------------------------------------------\n",
            "An prasent ans  s  ro t —sas,aarr iif\n",
            "\n",
            "Epoch: 31 | Loss: 1.03260\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aas  s  yttt —ssa,,aars ri\n",
            "\n",
            "Epoch: 32 | Loss: 1.78505\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aasi io ttt —ssaasaras rii\n",
            "\n",
            "Epoch: 33 | Loss: 1.27784\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent andi o  Rtr th— tattns  kr\n",
            "\n",
            "Epoch: 34 | Loss: 1.16155\n",
            "intelligence dirtiiciia\n",
            "--------------------------------------------------\n",
            "go prasent ansh io tth —ksaat sstiiiz\n",
            "\n",
            "Epoch: 35 | Loss: 0.92903\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent wish on tth stsaarrrriiz  \n",
            "\n",
            "Epoch: 36 | Loss: 0.93161\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "An prasent aash i  Rttt ss asaass rrr\n",
            "\n",
            "Epoch: 37 | Loss: 0.87577\n",
            "intelligence  irtficial\n",
            "--------------------------------------------------\n",
            "go prasent and  is Rttt  ssaalrrarrii\n",
            "\n",
            "Epoch: 38 | Loss: 0.67036\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "gn prasent wash on tttttssssaarr riii\n",
            "\n",
            "Epoch: 39 | Loss: 0.65818\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "An prasent wish o  ttttttsssararr ifi\n",
            "\n",
            "Epoch: 40 | Loss: 0.60696\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "go prasent wish on Rttttsssalarrs rii\n",
            "\n",
            "Epoch: 41 | Loss: 0.46053\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "Ao prasent wish on Rttttsssaarrrrifwi\n",
            "\n",
            "Epoch: 42 | Loss: 0.34408\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "Ao prasent wash io ttttsss,aarrrriiic\n",
            "\n",
            "Epoch: 43 | Loss: 0.25860\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssaarr rrifi\n",
            "\n",
            "Epoch: 44 | Loss: 0.22956\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent wish on tttttsssa,rarrriii\n",
            "\n",
            "Epoch: 45 | Loss: 0.21419\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssaarr rriwi\n",
            "\n",
            "Epoch: 46 | Loss: 0.20073\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssaarr rrifi\n",
            "\n",
            "Epoch: 47 | Loss: 0.18960\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrrrriii\n",
            "\n",
            "Epoch: 48 | Loss: 0.18008\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrrrriiw\n",
            "\n",
            "Epoch: 49 | Loss: 0.17341\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrrrriii\n",
            "\n",
            "Epoch: 50 | Loss: 0.16804\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 51 | Loss: 0.16265\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 52 | Loss: 0.15784\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 53 | Loss: 0.15302\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 54 | Loss: 0.14866\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 55 | Loss: 0.14439\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 56 | Loss: 0.14026\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent aish io tttttsssalrrarrifi\n",
            "\n",
            "Epoch: 57 | Loss: 0.13629\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io tttttsssal,rarriii\n",
            "\n",
            "Epoch: 58 | Loss: 0.13264\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io tttttsssal,rarriii\n",
            "\n",
            "Epoch: 59 | Loss: 0.12890\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io tttttsssal,rarriii\n",
            "\n",
            "Epoch: 60 | Loss: 0.12566\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io ttt ts sa, arrriii\n",
            "\n",
            "Epoch: 61 | Loss: 0.12213\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io ttt ts sa, arrriii\n",
            "\n",
            "Epoch: 62 | Loss: 0.11909\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io ttt ts sa, arrrifi\n",
            "\n",
            "Epoch: 63 | Loss: 0.11591\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  so tttthstas,,arrriic\n",
            "\n",
            "Epoch: 64 | Loss: 0.11302\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth stas,ararrirw\n",
            "\n",
            "Epoch: 65 | Loss: 0.10999\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth stas,,arr ric\n",
            "\n",
            "Epoch: 66 | Loss: 0.10737\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  io ttt ts sa, arrrifi\n",
            "\n",
            "Epoch: 67 | Loss: 0.10455\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth stas,,arr ric\n",
            "\n",
            "Epoch: 68 | Loss: 0.10203\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth staa,,arr riw\n",
            "\n",
            "Epoch: 69 | Loss: 0.09937\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sas,aarr rii\n",
            "\n",
            "Epoch: 70 | Loss: 0.09725\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sas,aarr rii\n",
            "\n",
            "Epoch: 71 | Loss: 0.09456\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sa, aarrrrwi\n",
            "\n",
            "Epoch: 72 | Loss: 0.09277\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sas,aarr rii\n",
            "\n",
            "Epoch: 73 | Loss: 0.09010\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sa, aarrrrwi\n",
            "\n",
            "Epoch: 74 | Loss: 0.08823\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sa, aarrrrwi\n",
            "\n",
            "Epoch: 75 | Loss: 0.08593\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sa, aarrrrwi\n",
            "\n",
            "Epoch: 76 | Loss: 0.08403\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  s  rtth —sa, aarrrrwi\n",
            "\n",
            "Epoch: 77 | Loss: 0.08194\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  io tttmth ss, aarrifi\n",
            "\n",
            "Epoch: 78 | Loss: 0.08018\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  io tttmth ss, aarrifi\n",
            "\n",
            "Epoch: 79 | Loss: 0.07820\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  io tttmth ss, aarrifi\n",
            "\n",
            "Epoch: 80 | Loss: 0.07662\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  is Rtt thass, arrrifi\n",
            "\n",
            "Epoch: 81 | Loss: 0.07495\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "go prasent ans  io tttmth ss, aarrifi\n",
            "\n",
            "Epoch: 82 | Loss: 0.07358\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  is Rtt thass, arr ifi\n",
            "\n",
            "Epoch: 83 | Loss: 0.07378\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "go prasent ans  is Rttmthass, arr rdi\n",
            "\n",
            "Epoch: 84 | Loss: 0.07634\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ans  io tttmth ss, aarrifi\n",
            "\n",
            "Epoch: 85 | Loss: 0.07271\n",
            "intelligence dirtficial\n",
            "--------------------------------------------------\n",
            "Ao prasent aas  io ttt thas,,aarr rii\n",
            "\n",
            "Epoch: 86 | Loss: 0.08420\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "Ao prasent aas  i  Rtt t staalass rri\n",
            "\n",
            "Epoch: 87 | Loss: 0.31840\n",
            "intelligence dirtiicial\n",
            "--------------------------------------------------\n",
            "go prasent ais  i  ttr hustal, ar rin\n",
            "\n",
            "Epoch: 88 | Loss: 0.39855\n",
            "intelligence districila\n",
            "--------------------------------------------------\n",
            "go prasent ans is trom ssaaa,asstriii\n",
            "\n",
            "Epoch: 89 | Loss: 1.40530\n",
            "Intelligence direlicial\n",
            "--------------------------------------------------\n",
            "AI arasent was  o  tot —ssa,,aart iri\n",
            "\n",
            "Epoch: 90 | Loss: 3.89216\n",
            "intelligence iatificial\n",
            "--------------------------------------------------\n",
            "An prasent was  on tht———s,, arrircic\n",
            "\n",
            "Epoch: 91 | Loss: 4.70942\n",
            "intelligence  istriciua\n",
            "--------------------------------------------------\n",
            "on prasent was  iI ft rmsssararrrrrri\n",
            "\n",
            "Epoch: 92 | Loss: 3.71111\n",
            "intelligence  istficial\n",
            "--------------------------------------------------\n",
            "gn prasent wish  I nithtrsss aarrsrif\n",
            "\n",
            "Epoch: 93 | Loss: 1.67066\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "gn prasent wily oI Rtrmttst atas  rrf\n",
            "\n",
            "Epoch: 94 | Loss: 0.61738\n",
            "intelligence  artificia\n",
            "--------------------------------------------------\n",
            "gn prasent wis  I  tth rrtsss sff rar\n",
            "\n",
            "Epoch: 95 | Loss: 0.35921\n",
            "intelligence  astiiiial\n",
            "--------------------------------------------------\n",
            "gn prasent wis  o  tth rrsssss ran  i\n",
            "\n",
            "Epoch: 96 | Loss: 0.31197\n",
            "intelligence  artiiiial\n",
            "--------------------------------------------------\n",
            "gn prasent wis  AI ttr trts ss fri  r\n",
            "\n",
            "Epoch: 97 | Loss: 0.27797\n",
            "intelligence  artiiiiia\n",
            "--------------------------------------------------\n",
            "gn prasent wis  AI ttr trsts s trn, r\n",
            "\n",
            "Epoch: 98 | Loss: 0.25412\n",
            "intelligence  artiiicia\n",
            "--------------------------------------------------\n",
            "gn prasent wis  AI ttrrttsts starn  r\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-45-242e52eeefcd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     17\u001b[0m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobjective\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m             \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 19\u001b[0;31m         \u001b[0madam\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlearning_rate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     20\u001b[0m     \u001b[0mlosses\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masscalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'\\nEpoch: {epoch} | Loss: {loss.asscalar():.5f}'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-37-45bbb6cc32fc>\u001b[0m in \u001b[0;36madam\u001b[0;34m(t, params, vs, sqrs, lr)\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0mgrad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m         \u001b[0mv\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbeta1\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mv\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1.\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta1\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     19\u001b[0m         \u001b[0msqr\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbeta2\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0msqr\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1.\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta2\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msquare\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1e-8\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/ndarray.py\u001b[0m in \u001b[0;36m__setitem__\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m    442\u001b[0m         \u001b[0mindexing_dispatch_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_indexing_dispatch_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    443\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mindexing_dispatch_code\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_NDARRAY_BASIC_INDEXING\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 444\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_nd_basic_indexing\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    445\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mindexing_dispatch_code\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_NDARRAY_ADVANCED_INDEXING\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    446\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_set_nd_advanced_indexing\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/ndarray.py\u001b[0m in \u001b[0;36m_set_nd_basic_indexing\u001b[0;34m(self, key, value)\u001b[0m\n\u001b[1;32m    701\u001b[0m                         \u001b[0;32mif\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    702\u001b[0m                             \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbroadcast_to\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 703\u001b[0;31m                         \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopyto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    704\u001b[0m                 \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumeric_types\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    705\u001b[0m                     _internal._full(shape=shape, ctx=self.context,\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/ndarray.py\u001b[0m in \u001b[0;36mcopyto\u001b[0;34m(self, other)\u001b[0m\n\u001b[1;32m   2072\u001b[0m                 \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'You are attempting to copy an array to itself'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mRuntimeWarning\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2073\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2074\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0m_internal\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_copyto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2075\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mContext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2076\u001b[0m             \u001b[0mhret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNDArray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_new_alloc_handle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/register.py\u001b[0m in \u001b[0;36m_copyto\u001b[0;34m(data, out, name, **kwargs)\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/_ctypes/ndarray.py\u001b[0m in \u001b[0;36m_imperative_invoke\u001b[0;34m(handle, ndargs, keys, vals, out)\u001b[0m\n\u001b[1;32m     88\u001b[0m         \u001b[0mctypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbyref\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     89\u001b[0m         \u001b[0mctypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc_int\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 90\u001b[0;31m         \u001b[0mc_str_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     91\u001b[0m         \u001b[0mc_str_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvals\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m         ctypes.byref(out_stypes)))\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/mxnet/base.py\u001b[0m in \u001b[0;36mc_str_array\u001b[0;34m(strings)\u001b[0m\n\u001b[1;32m    328\u001b[0m         \"\"\"\n\u001b[1;32m    329\u001b[0m         \u001b[0marr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mctypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc_char_p\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstrings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 330\u001b[0;31m         \u001b[0marr\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mstrings\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0marr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    332\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UPRSG6a1MZPV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "48ba30fd-ea80-48a0-a000-06806ffb4c73"
      },
      "source": [
        "generate('All but the simplest human behaviour is ascribed to intelligence')"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'intelligenced isplupedt to thed bo elee ieecectthe enesseee, the'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E9rvD_PJMZNx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d3d6b6e1-3de8-45ca-9b13-5273e5e60fc4"
      },
      "source": [
        "generate('I love AI')"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'An Sroje '"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FBMAusT7MZMG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8f223073-2e9c-4c81-c28b-818fd573e88a"
      },
      "source": [
        "generate('Artificial Intelligence')"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Intelligence Aartificia'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQtR5HICMZI4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h689kGsyMZHl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "INM0cg7vMZGS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nOAJUgosMZE8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}