Machine Learning Algos - Neural Network - Hands-on Introduction to Python And Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on Introduction to Python And Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instructor: Tak-Kei Lam\n",
    "\n",
    "(Readers are assumed to have a little bit programming background.)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Neural networks\n",
    "\n",
    "In our brain, we have around $10^{11}$ basic units called neurons. These neurons\n",
    "are massively interconnected as neural networks. A neuron, when stimulated,\n",
    "communicate with its neighbours electro-chemically at ultra high speed. The\n",
    "sender start the communication by releasing chemical substances to excite the\n",
    "receivers. Upon receiving the signals, the inner electric potential of the\n",
    "receiver will change. If the inner electric potential has reached a certain\n",
    "threshold, the receiver will then in turn release chemical substances and\n",
    "trigger the communication with its own neighbours. This happens repeatedly and\n",
    "simultaneously among all neurons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Artificial neural network\n",
    "![Artificial neural network](ann.png)\n",
    "\n",
    "In order to understand how our brain process information, and to create a computer that can learn, scientists have been developing artificial neural networks using computers. Since artificial neural network has the ability to learn from examples, they can be used to solve problems even if we do not understand the nature of problems well (and thus cannot give explicit instructions to the computer). They have been found to be very flexible and can be potentially useful for solving many problems.\n",
    "\n",
    "An example of artificial neural network is shown in the above figure. The circles represent neurons. The arrows represents connections and indicate the direction of information flow. The network has one layer of four input neurons, one hidden layer of five neurous, and one output layer containing only one neuron. Every artificial neural network must have one layer of input and one output layer of output neurons. Each layer of neurons can have any number of neurons.\n",
    "\n",
    "An artificial neural network is usually fully connected, where every neuron in a layer is connected with all neurons in the neighbour layers; it can also be sparse, which may be more suitable in some situations. It is also possible to have feedback loops and use the output neurons as the inputs of some neurons in the hidden layers.\n",
    "\n",
    "Each input connection of the receiver has a weighting value. The weights of the input connections sum to $< 1$ for each receiver. In the above figure, the weights of the inputs connections of the output neuron are shown. The output of a neuron is calculated based on a function and the weighted inputs, and is usually in the range 0 to 1.\n",
    "\n",
    "The number of layers, the number of neurons in each layer, how the neurons should be connected, the functions that neurons implement, and other parameters have to be tailor-made for the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How do artificial networks learn?\n",
    "Let's learn by the following example; the simplest example ever!\n",
    "\n",
    "![A simple example of neural network](nn-example.png)\n",
    "\n",
    "Artificial neural networks actually learn by adjusting the weights of each\n",
    "neuron. Consider the super simple artificial neural network in\n",
    "the above figure. The output neuron implement the function $f$:\n",
    "\\begin{align*}\n",
    "    f &= \\frac{1}{1+e^{-x}} \\\\\n",
    "    \\text{where } x &= w_a \\times a + w_b \\times b + w_c \\times c \\\\\n",
    "    c & = -0.5\n",
    "\\end{align*}\n",
    "\n",
    "We want to use it to learn whether $a > b$:\n",
    "\n",
    "| condition |  target value of $f$, $t$    |\n",
    "|:---------:|:----------------------------:|\n",
    "|$a > b$    | 1                            |\n",
    "|$a \\le b$   | 0                            |\n",
    "\n",
    "Since $f$ is continuous in the range 0 to 1, we assume that:\n",
    "\n",
    "|result  &nbsp;&nbsp;&nbsp;       | meaning |\n",
    "|:-------------:|:-------:|\n",
    "|$f > 0.5 $     | $a > b$   |\n",
    "|$f \\le 0.5$    | $a \\le b$  |\n",
    "\n",
    "\n",
    "Base on the initial configuration as shown in the figure,\n",
    "\\begin{align*}\n",
    "    \\text{if } a &=0.5, b =0.3 \\\\\n",
    "    \\text{then } f & = \\frac{1}{1+e^{-(0.0 \\times 0.5 + 0.0 \\times 0.3 + 1 \\times -0.5)}} \\\\\n",
    "                   & = 0.377541\n",
    "\\end{align*}\n",
    "\n",
    "The target value of $f$ should be $1$ instead; now it is not even $>0.5$!\n",
    "Obviously, our artificial neural network fails to determine whether $a>b$ at\n",
    "this moment. We can train the artificial neural network to improve it. But how? We need to use a little bit maths.\n",
    "\n",
    "First of all, we have to define how far we are from the target result. Let $E$\n",
    "be the squared error:\n",
    "\\begin{align*}\n",
    "    E = \\frac{1}{2} (t - f)^2\n",
    "\\end{align*}\n",
    "where $t$ is the target result. Referring to the previous example, if $a=0.5,\n",
    "b=0.3$, $f=0.377541$ and $t=1$.\n",
    "\n",
    "Then, we can determine how $E$ responds to the change of each of the weights.\n",
    "That is, the rates of change of $E$ with respect to\n",
    "$w_a$($\\frac{\\partial{E}}{\\partial{w_a}}$),\n",
    "$w_b$($\\frac{\\partial{E}}{\\partial{w_b}}$) and\n",
    "$w_c$($\\frac{\\partial{E}}{\\partial{w_c}}$). If we know these values, we can know\n",
    "whether a weight should be increased or decreased. For example, if\n",
    "$\\frac{\\partial{E}}{\\partial{w_a}} > 0$, we know that increasing $w_a$ will\n",
    "increase the error, and hence we should decrease it. We can also decide how\n",
    "large the adjustment should be.\n",
    "\n",
    "\n",
    "By the **chain rule**, we know that:\n",
    "\\begin{align*}\n",
    "    \\frac{\\partial{E}}{\\partial{w_i}} &= \n",
    "        \\frac{\\partial{E}}{\\partial{f}} \\times\n",
    "        \\frac{\\partial{f}}{\\partial{x}} \\times \n",
    "        \\frac{\\partial{x}}{\\partial{w_i}}\n",
    "        & \\text{ where } i \\text{ can be } a,\n",
    "        b, \\text{ or } c \\\\\n",
    "\\end{align*}\n",
    "\n",
    "We also know all the derivatives in the formula:\n",
    "\\begin{align*}\n",
    "    \\frac{\\partial{E}}{\\partial{f}} &= -(t-f) \\\\\n",
    "    \\frac{\\partial{f}}{\\partial{x}} &= (\\frac{1}{1+e^{-x}})(1-\\frac{1}{1+e^{-x}}) \\\\\n",
    "    \\frac{\\partial{x}}{\\partial{w_a}} &= a \\\\\n",
    "    \\frac{\\partial{x}}{\\partial{w_b}} &= b  \\\\\n",
    "    \\frac{\\partial{x}}{\\partial{w_c}} &= c = -0.5 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "\n",
    "Therefore, each of the weights can be updated as follows:\n",
    "\\begin{align*}\n",
    "    w_i &= w_i - \\eta \\times \\frac{\\partial{E}}{\\partial{w_i}} & \\text{ where } i\n",
    "    \\text{ can be } a, b, \\text{ or } c, \\text{and } 0<\\eta\\le 1 \\text{ is known as the\n",
    "    learning rate } \\\\\n",
    "\\end{align*}\n",
    "\n",
    "In our example, if $a=0.5, b=0.3$, the corrected weights are:\n",
    "- $w_a=0.073140$, \n",
    "- $w_b=0.043884$,\n",
    "- $w_c=0.926860$.\n",
    "\n",
    "The value of $f$ based on these new weights will increase to $0.398027$.\n",
    "\n",
    "Suppose we train our artificial neural network using the following *training data* using\n",
    "$\\eta=1$ for 100 iterations:\n",
    "\n",
    "|        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$a, b$ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | $t$ |\n",
    "|:-------------:|:---:|\n",
    "|        $a=0.5,\\\\ b=0.3$ | 1 |\n",
    "|        $a=0.3,\\\\ b=0.5$ | 0 |\n",
    "|        $a=1.0,\\\\ b=0.2$ | 1 |\n",
    "|        $a=0.2,\\\\ b=1.0$ | 0 |\n",
    "|        $a=0.9,\\\\ b=0.8$ | 1 |\n",
    "|        $a=0.5,\\\\ b=0.5$ | 0 |\n",
    "\n",
    "\n",
    "After training, the final weights will be:\n",
    "- $w_a=4.887006$,\n",
    "- $w_b=-2.908520$,\n",
    "- $w_c=2.280892$.\n",
    "\n",
    "The result for some *test* data is listed in the following table. Only the entry $a=0.51, b=0.5$ is wrong. Our artificial neural network performs way better than before! It may work even better if we train it with more examples. Generally speaking, the more\n",
    "examples the artificial neural network has learnt, the less error it will produce.\n",
    "\n",
    "|       &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $a, b$&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | $t$ | $f$| correct?|\n",
    "|:--:|:--:|:--:|:--:|\n",
    "|        $a=0.5,\\\\ b=0.3$ | 1 | 0.605993| ✔|\n",
    "|        $a=0.3,\\\\ b=0.5$ | 0 | 0.244419| ✔|\n",
    "|        $a=1.0,\\\\ b=0.2$ | 1 | 0.959490| ✔|\n",
    "|        $a=0.2,\\\\ b=1.0$ | 0 | 0.044296| ✔|\n",
    "|        $a=0.9,\\\\ b=0.8$ | 1 | 0.717287| ✔|\n",
    "|        $a=0.5,\\\\ b=0.5$ | 0 | 0.462271| ✔|\n",
    "|        $a=0.51,\\\\ b=0.5$ | 1 | 0.474439| |\n",
    "|        $a=0.61,\\\\ b=0.6$ | 1 | 0.523861| ✔|\n",
    "\n",
    "\n",
    "We have demonstrated a artificial neural network with only one neuron can be used to solve problems without explicit if-then-else code. Using an artificial neural network to solve $a > b?$ is of course an overkill. How about using artificial neural network to \"predict\" stock trend? That may be possible using more complex artificial neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you ever wondered how a neural network program looks like? Let's see the code for the above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "\n",
    "A = np.array([0.5, 0.3, 1, 0.2, 0.9, 0.5])\n",
    "B = np.array([0.3, 0.5, 0.2, 1, 0.8, 0.5])\n",
    "Expected_f = np.array([1, 0, 1, 0, 1, 0])\n",
    "\n",
    "# initial weights\n",
    "w_a = 0\n",
    "w_b = 0\n",
    "w_c = 1\n",
    "\n",
    "epoch = 1\n",
    "alpha = 1 # the learning rate \"eta\" in our formula\n",
    "\n",
    "# training\n",
    "while epoch <= 100:\n",
    "    print('---------------- epoch: {}'.format(i))\n",
    "    for i in range(0, len(A)):\n",
    "        is_update = 0\n",
    "\n",
    "        a = A[i]\n",
    "        b = B[i]\n",
    "        c = -0.5\n",
    "        expected_f = Expected_f[i]\n",
    "\n",
    "        x = w_a * a + w_b * b + w_c *c\n",
    "        f = 1/(1+math.exp(-x))\n",
    "        \n",
    "        E = 0.5 * ((expected_f - f)**2)\n",
    "\n",
    "        df_dx = f * (1-f)\n",
    "        df_dw_a = df_dx * a\n",
    "        df_dw_b = df_dx * b\n",
    "        df_dw_c = df_dx * c\n",
    "\n",
    "\n",
    "        dE_df = (-expected_f+f)\n",
    "        dE_dw_a = dE_df * df_dw_a\n",
    "        dE_dw_b = dE_df * df_dw_b\n",
    "        dE_dw_c = dE_df * df_dw_c\n",
    "\n",
    "        w_a = w_a - alpha*dE_dw_a\n",
    "        w_b = w_b - alpha*dE_dw_b\n",
    "        w_c = w_c - alpha*dE_dw_c\n",
    "\n",
    "        print('x: {}'.format( x))\n",
    "        print('a: {}, b: {}, f: {}, expected f: {}'.format( a, b, f, expected_f))\n",
    "        print('df_dw_a: {}'.format( df_dw_a))\n",
    "        print('df_dw_b: {}'.format( df_dw_b))\n",
    "        print('df_dw_c: {}'.format( df_dw_c))\n",
    "        print('dE_df: {}'.format( dE_df))\n",
    "        print('dE_dw_a: {}'.format( dE_dw_a))\n",
    "        print('dE_dw_b: {}'.format( dE_dw_b))\n",
    "        print('dE_dw_c: {}'.format( dE_dw_c))\n",
    "        print('w_a: {}'.format( w_a))\n",
    "        print('w_b: {}'.format( w_b))\n",
    "        print('w_c: {}'.format( w_c))\n",
    "\n",
    "        x = w_a * a + w_b * b + w_c *c\n",
    "        f = 1/(1+math.exp(-x))\n",
    "\n",
    "        print('corrected a: {}, b: {}, f: {}, expected f: {}'.format(a, b, f, expected_f))\n",
    "        print('')\n",
    "        \n",
    "        i = i+1\n",
    "\n",
    "    epoch = epoch +1 \n",
    "\n",
    "\n",
    "print('Finish training!')\n",
    "\n",
    "\n",
    "\n",
    "# testing\n",
    "A = np.array([0.5, 0.3, 1, 0.2, 0.9, 0.5, 0.51, 0.61, 0.2, 0.4])\n",
    "B = np.array([0.3, 0.5, 0.2, 1, 0.8, 0.5, 0.5, 0.60, 0.1, 0.5])\n",
    "Expected_f = np.array([1, 0, 1, 0, 1, 0, 1, 1, 1, 0])\n",
    "for i in range(0, len(A)):\n",
    "    a = A[i]\n",
    "    b = B[i]\n",
    "    expected_f = Expected_f[i]\n",
    "\n",
    "    x = w_a * a + w_b * b + w_c *c\n",
    "    f = 1/(1+math.exp(-x))\n",
    "\n",
    "    correct = (expected_f == 1 and f > 0.5) or (expected_f == 0 and f<=0.5)\n",
    "    print('a: {}, b: {}, f: {}, expected f: {}, correct? {}'.format(a, b, f, expected_f, correct))\n",
    "    i = i + 1\n",
    "\n",
    "print('Finish testing!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that the code is just for our particular single-neuron example. It is highly specialised and is not a general neural network implementation. Now you have realised how difficult it is to write a general neural network library, haven't you?\n",
    "\n",
    "Efficiency and user-friendliness are the major factors that neural network library developers should pay attention to. The number of parameters to be calculated can easily go up to millions or billions. That is why specialised hardwares such as *GPU* and *FPGA* are prefered to general CPU for training and running neural networks.\n",
    "\n",
    "For general neural network maching learning frameworks, please refer to:\n",
    "1. [scikit-learn (neural network)](http://scikit-learn.org/stable/modules/neural_networks_supervised.html)\n",
    "2. [Tensorflow](https://www.tensorflow.org/) and [Keras](https://keras.io/)\n",
    "3. [pytorch](https://pytorch.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A demo of using scikit-learn to classify Pokemons:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "import pandas as pd    \n",
    "\n",
    "# Load the dataset\n",
    "pokemons = pd.read_csv(\"pokemon.csv\")\n",
    "print(pokemons['Type 1'].unique())\n",
    "\n",
    "pokemons = pokemons.sample(frac=1) # .sample(frac=1) randomize the data, and select 100% from the randomized\n",
    "\n",
    "label_column = ['Type 1']\n",
    "features_columns = ['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed']\n",
    "\n",
    "pokemons_features = pokemons[features_columns]\n",
    "pokemons_label = pokemons[label_column]\n",
    "\n",
    "# normalise every columns in pokemons_features\n",
    "pokemons_features = pokemons_features.apply(lambda x: (x - x.min())/(x.max() - x.min()))\n",
    "\n",
    "# .values convert the datastructure from pandas's dataframe into numpy's array\n",
    "\n",
    "# Split the data into training/test sets\n",
    "last_index = -int(0.20*len(pokemons_features))\n",
    "pokemons_features_train = pokemons_features[:last_index].values\n",
    "pokemons_features_test = pokemons_features[last_index:].values \n",
    "\n",
    "last_index = -int(0.20*len(pokemons_label))\n",
    "pokemons_label_train = pokemons_label[:last_index].values.flatten() # the expected labels\n",
    "pokemons_label_test = pokemons_label[last_index:].values.flatten()  # the expected labels\n",
    "\n",
    "\n",
    "clf = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(50, 18), random_state=1)\n",
    "clf.fit(pokemons_features_train, pokemons_label_train)\n",
    "\n",
    "# Make predictions using the testing set\n",
    "pokemons_label_pred = clf.predict(pokemons_features_test) # the actual labels\n",
    "\n",
    "correct = 0.0\n",
    "for i in range(0, len(pokemons_label_test)):\n",
    "    print('expected {} VS. actual {}'.format(pokemons_label_test[i], pokemons_label_pred[i]))\n",
    "    if pokemons_label_pred[i] == pokemons_label_test[i]:\n",
    "        correct = correct+1\n",
    "print('Accuracy: {}%'.format(correct/len(pokemons_label_test) * 100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have just learnt the most fundamental concepts  of artificial neural network. Here are a few \"jargons\" regarding neural network basics that we have talked about:\n",
    "- Feed forward\n",
    "- Back propagation\n",
    "- Bias\n",
    "- Fully connected layers\n",
    "- Activation function\n",
    "- (Stochastic) Gradient descent\n",
    "\n",
    ">> Don't remember the keywords; remember the concepts. --- Tak-Kei Lam (the Great)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People found that there is a huge potential of using artificial neural networks to solve problems because of its simplicity$^*$. And therefore now we have a huge family of artificial neural networks in which each member is unique and best at doing something...\n",
    "\n",
    "##### The family of artificial neural networks:\n",
    "- Basic feed forward neural network (NN)\n",
    "- Convolutional neural network (CNN)\n",
    "- Recurrent neural network (RNN)\n",
    "- Long short term memory (LSTM)\n",
    "- Convolutional recurrent neural network (CRNN)\n",
    "- Region-based convolutional neural network (RCNN)\n",
    "- (an extremely lengthy and boring list of entries are not presented here...) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### An example of CNN using Tensorflow and Keras:\n",
    "In this example, we are building a neural network to report whether a given Pokemon image is Pichu or Pikachu.\n",
    "\n",
    "|          |         |\n",
    "|:----:|:----:|\n",
    "|![Pichu?](pichu-1.png) | ![Pikachu?](pikachu-1.png)|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.pooling import MaxPooling2D\n",
    "from keras.optimizers import SGD\n",
    "from  skimage import io, color, exposure, transform\n",
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "NUM_CLASSES = 2\n",
    "IMG_SIZE = 48\n",
    "BATCH_SIZE = 1\n",
    "EPOCHS  = 20\n",
    "\n",
    "root_dir = './pokemon-images/'\n",
    "img_paths = []\n",
    "imgs = []\n",
    "labels = []\n",
    "label_names = []\n",
    "\n",
    "# ---------------------------------------------------------------------------- model definition\n",
    "\n",
    "class AccuracyHistory(keras.callbacks.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.acc = []\n",
    "\n",
    "    def on_epoch_end(self, batch, logs={}):\n",
    "        self.acc.append(logs.get('acc'))\n",
    "        \n",
    "history = AccuracyHistory()\n",
    "\n",
    "# preprocess the input image: kind of normalise it, and standardise the size\n",
    "def preprocess_img(img):\n",
    "    # Histogram normalization in v channel\n",
    "    hsv = color.rgb2hsv(color.rgba2rgb(img))\n",
    "    hsv[:, :, 2] = exposure.equalize_hist(hsv[:, :, 2])\n",
    "    img = color.hsv2rgb(hsv)\n",
    "\n",
    "    # central square crop\n",
    "    min_side = min(img.shape[:-1])\n",
    "    centre = img.shape[0] // 2, img.shape[1] // 2\n",
    "    img = img[centre[0] - min_side // 2:centre[0] + min_side // 2,\n",
    "              centre[1] - min_side // 2:centre[1] + min_side // 2,\n",
    "              :]\n",
    "\n",
    "    # rescale to standard size\n",
    "    img = transform.resize(img, (IMG_SIZE, IMG_SIZE))\n",
    "    return img\n",
    "\n",
    "# construct the neural network model\n",
    "# hey we're using more than one neurons connected in some ways! \n",
    "def constructModel(batchSize):\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),\n",
    "                     activation='relu',\n",
    "                     input_shape=(IMG_SIZE, IMG_SIZE, 3)))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
    "    model.add(Conv2D(64, (5, 5), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(1000, activation='relu'))\n",
    "    model.add(Dense(NUM_CLASSES, activation='softmax'))\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(lr=0.01),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "\n",
    "\n",
    "# ---------------------------------------------------------------------------- model training\n",
    "# read images\n",
    "all_img_paths = glob.glob(os.path.join(root_dir, '*.png'))\n",
    "np.random.shuffle(all_img_paths)\n",
    "for img_path in all_img_paths:\n",
    "    img = preprocess_img(io.imread(img_path))\n",
    "    if 'pikachu' in img_path:\n",
    "        label = 1\n",
    "        label_name = 'Pikachu'\n",
    "    elif 'pichu' in img_path:\n",
    "            label = 0\n",
    "            label_name = 'Pichu'\n",
    "    print('Image: {} is of class: {}'.format(img_path, label))\n",
    "    imgs.append(img)\n",
    "    labels.append(label)\n",
    "    label_names.append(label_name)\n",
    "    img_paths.append(img_path)\n",
    "\n",
    "X = np.array(imgs)\n",
    "#print(X.shape)\n",
    "# Make one hot targets\n",
    "Y = np.eye(NUM_CLASSES)[labels]\n",
    "\n",
    "\n",
    "num_training_imgs = int(0.8*len(X))\n",
    "num_test_imgs = len(X) - num_training_imgs\n",
    "\n",
    "img_paths_train = img_paths[0:num_training_imgs]\n",
    "x_train = X[0:num_training_imgs]\n",
    "y_train = Y[0:num_training_imgs]\n",
    "labels_train = label_names[0:num_training_imgs]\n",
    "\n",
    "for i,v in enumerate(x_train):\n",
    "    print('training image: {} is {}'.format(img_paths_train[i], labels_train[i]))\n",
    "\n",
    "model = constructModel(len(x_train))\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=BATCH_SIZE,\n",
    "          epochs=EPOCHS,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test),\n",
    "          callbacks=[history])\n",
    "\n",
    "\n",
    "# plot a graph of the change of model accuracy over time\n",
    "plt.plot(range(1,EPOCHS+1), history.acc)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.show()\n",
    "\n",
    "\n",
    "# ---------------------------------------------------------------------------- model testing\n",
    "img_paths_test = img_paths[-num_test_imgs:]\n",
    "x_test = X[-num_test_imgs:]\n",
    "y_test = Y[-num_test_imgs:]\n",
    "labels_test = label_names[-num_test_imgs:]\n",
    "\n",
    "for i,v in enumerate(x_test):\n",
    "    print('test image: {} is {}'.format(img_paths_test[i], labels_test[i]))\n",
    "    \n",
    "# predict and print the probability that an image belongs to a class\n",
    "print(model.predict(x_test))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise **:\n",
    "- Try to use scikit-learn's neural network to model the following classification problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [0, 1]\n",
    "classes  = [0, 1] # classes[i] is the class of the data point data[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise **:\n",
    "- Try to use scikit-learn's neural network to model the following classification problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [[0, 0], [0,1], [1,0], [1,1]]\n",
    "classes  = [0, 0, 1, 1] # classes[i] is the class of the data point data[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise **:\n",
    "- Try to use scikit-learn's neural network to model the following classification problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [[0, 0], [0,1], [1,0], [1,1]]\n",
    "classes  = [0, 1, 2, 3] # classes[i] is the class of the data point data[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise **:\n",
    "- Try to use scikit-learn's neural network to model the following classification problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ['a', 'e', 'i', 'o', 'u', 'b', 'c', 'd', 'f', 'g', 'h', 'j', 'k', 'l', 'm', 'n', 'p', 'q', 'r', 's', 't', 'v', 'w', 'x', 'y', 'z']\n",
    "classes  = [1, 1, 1,1, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0,0, 0, 0, 0, 0,0, 0, 0, 0, 0,0] # classes[i] is the class of the data point data[i]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}