4-machine-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 機械学習をやってみる\n",
    "\n",
    "## この章で扱うもの\n",
    "\n",
    "- 前章で作った各種データを再利用\n",
    "- 機械学習を行ってみる\n",
    "- 学習結果の評価を行う\n",
    "- 機械学習の改善方法\n",
    "\n",
    "\n",
    "## この章で取り扱う手順\n",
    "\n",
    "- 学習データとテストデータの分割\n",
    "- sklearnを用いて学習\n",
    "- 評価\n",
    "- 交差検証\n",
    "- グリットサーチ\n",
    "- 各種モデルを試す\n",
    "- 評価結果の再確認\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle(\"df.db\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学習データとテストデータの分割\n",
    "\n",
    "http://qiita.com/terapyon/items/8f8d3518ee8eeb4f96b2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>報告数</th>\n",
       "      <th>流行</th>\n",
       "      <th>増加</th>\n",
       "      <th>平均気温(℃)</th>\n",
       "      <th>最高気温(℃)</th>\n",
       "      <th>最低気温(℃)</th>\n",
       "      <th>平均湿度(％)</th>\n",
       "      <th>最小相対湿度(％)</th>\n",
       "      <th>平均現地気圧(hPa)</th>\n",
       "      <th>降水量の合計(mm)</th>\n",
       "      <th>日照時間(時間)</th>\n",
       "      <th>平均風速(m/s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-01-01</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.8</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.9</td>\n",
       "      <td>54.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1005.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.2</td>\n",
       "      <td>5.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-02</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.9</td>\n",
       "      <td>4.4</td>\n",
       "      <td>41.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1011.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.1</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-03</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>9.9</td>\n",
       "      <td>2.7</td>\n",
       "      <td>43.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1014.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.1</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-04</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>11.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>47.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1009.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-05</th>\n",
       "      <td>0.178571</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.4</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2.3</td>\n",
       "      <td>40.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1016.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 報告数  流行  増加  平均気温(℃)  最高気温(℃)  最低気温(℃)  平均湿度(％)  最小相対湿度(％)  \\\n",
       "2014-01-01  0.178571   0   0      9.8     13.7      3.9     54.0       37.0   \n",
       "2014-01-02  0.178571   0   0      8.0     12.9      4.4     41.0       26.0   \n",
       "2014-01-03  0.178571   0   0      5.9      9.9      2.7     43.0       32.0   \n",
       "2014-01-04  0.178571   0   0      6.7     11.5      2.1     47.0       29.0   \n",
       "2014-01-05  0.178571   0   0      4.4      6.9      2.3     40.0       28.0   \n",
       "\n",
       "            平均現地気圧(hPa)  降水量の合計(mm)  日照時間(時間)  平均風速(m/s)  \n",
       "2014-01-01       1005.3         0.0       9.2        5.3  \n",
       "2014-01-02       1011.3         0.0       9.1        3.0  \n",
       "2014-01-03       1014.9         0.0       4.1        1.6  \n",
       "2014-01-04       1009.5         0.0       5.9        2.4  \n",
       "2014-01-05       1016.6         0.0       1.1        2.5  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df.iloc[:, 3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>平均気温(℃)</th>\n",
       "      <th>最高気温(℃)</th>\n",
       "      <th>最低気温(℃)</th>\n",
       "      <th>平均湿度(％)</th>\n",
       "      <th>最小相対湿度(％)</th>\n",
       "      <th>平均現地気圧(hPa)</th>\n",
       "      <th>降水量の合計(mm)</th>\n",
       "      <th>日照時間(時間)</th>\n",
       "      <th>平均風速(m/s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-01-01</th>\n",
       "      <td>9.8</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3.9</td>\n",
       "      <td>54.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>1005.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.2</td>\n",
       "      <td>5.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-02</th>\n",
       "      <td>8.0</td>\n",
       "      <td>12.9</td>\n",
       "      <td>4.4</td>\n",
       "      <td>41.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1011.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.1</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-03</th>\n",
       "      <td>5.9</td>\n",
       "      <td>9.9</td>\n",
       "      <td>2.7</td>\n",
       "      <td>43.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1014.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.1</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-04</th>\n",
       "      <td>6.7</td>\n",
       "      <td>11.5</td>\n",
       "      <td>2.1</td>\n",
       "      <td>47.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1009.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.9</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-05</th>\n",
       "      <td>4.4</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2.3</td>\n",
       "      <td>40.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1016.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            平均気温(℃)  最高気温(℃)  最低気温(℃)  平均湿度(％)  最小相対湿度(％)  平均現地気圧(hPa)  \\\n",
       "2014-01-01      9.8     13.7      3.9     54.0       37.0       1005.3   \n",
       "2014-01-02      8.0     12.9      4.4     41.0       26.0       1011.3   \n",
       "2014-01-03      5.9      9.9      2.7     43.0       32.0       1014.9   \n",
       "2014-01-04      6.7     11.5      2.1     47.0       29.0       1009.5   \n",
       "2014-01-05      4.4      6.9      2.3     40.0       28.0       1016.6   \n",
       "\n",
       "            降水量の合計(mm)  日照時間(時間)  平均風速(m/s)  \n",
       "2014-01-01         0.0       9.2        5.3  \n",
       "2014-01-02         0.0       9.1        3.0  \n",
       "2014-01-03         0.0       4.1        1.6  \n",
       "2014-01-04         0.0       5.9        2.4  \n",
       "2014-01-05         0.0       1.1        2.5  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = df['流行']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2014-01-01    0\n",
       "2014-01-02    0\n",
       "2014-01-03    0\n",
       "2014-01-04    0\n",
       "2014-01-05    0\n",
       "Name: 流行, dtype: int32"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ロジスティック回帰"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_pred = clf.predict(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 正答率を確認"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87671232876712324"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_val_pred = clf.predict(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87727272727272732"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 混同行列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_val, y_val_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[168,   9],\n",
       "       [ 18,  25]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[682,  48],\n",
       "       [ 60,  86]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm_t = confusion_matrix(y_train, y_train_pred)\n",
    "cm_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 混同行列の評価\n",
    "\n",
    "```\n",
    "[[TP, FN],\n",
    " [FP, TN]]\n",
    " ```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 適合率(precision)・再現率(recall)・F値(f1-score)\n",
    "\n",
    "## 適合率\n",
    " \n",
    "- P(今回の場合は、流行していない) に判定された率 (178 / (178+14) = 0.93)\n",
    "- N(今回の場合は、流行している) に判定された率 (17 / (17+11) = 0.61)\n",
    " \n",
    "## 再現率\n",
    " \n",
    "- Tと正しく予測できた割合 (178 / (178+11) = 0.94)\n",
    "- Fと正しく予測できた割合 (17 / (17+14) = 0.55)\n",
    " \n",
    " \n",
    "## F値\n",
    " \n",
    "`2 / (1/適合率+1/再現率) = 2 * 適合率 * 再現率 / (適合率+再現率）`\n",
    " \n",
    "- 0のF値 `(2 * 0.93 * 0.94 / (0.93 + 0.94) = 0.93`\n",
    "- 1のF値 `(2 * 0.61 * 0.55 / (0.61 + 0.55) = 0.58`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.90      0.95      0.93       177\n",
      "          1       0.74      0.58      0.65        43\n",
      "\n",
      "avg / total       0.87      0.88      0.87       220\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_val, y_val_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## レポート関係を関数化し再利用可能にする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def report(y, pred):\n",
    "    print(accuracy_score(y, pred))\n",
    "    cm = confusion_matrix(y, pred)\n",
    "    print(cm)\n",
    "    cr = classification_report(y, pred)\n",
    "    print(cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.876712328767\n",
      "[[682  48]\n",
      " [ 60  86]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.93      0.93       730\n",
      "          1       0.64      0.59      0.61       146\n",
      "\n",
      "avg / total       0.87      0.88      0.87       876\n",
      "\n"
     ]
    }
   ],
   "source": [
    "report(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学習から評価までを関数化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fit_to_pred(clf, X_train, X_val, y_train, y_val):\n",
    "    # 学習\n",
    "    clf.fit(X_train, y_train)\n",
    "    \n",
    "    # 学習データで評価\n",
    "    y_train_pred = clf.predict(X_train)\n",
    "    print(\"y_train_pred: \")\n",
    "    report(y_train, y_train_pred)\n",
    "    \n",
    "    # テストデータで評価\n",
    "    y_val_pred = clf.predict(X_val)\n",
    "    print(\"y_val_pred: \")\n",
    "    report(y_val, y_val_pred)\n",
    "    \n",
    "    # 学習済みデータを返す\n",
    "    return clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.876712328767\n",
      "[[682  48]\n",
      " [ 60  86]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.93      0.93       730\n",
      "          1       0.64      0.59      0.61       146\n",
      "\n",
      "avg / total       0.87      0.88      0.87       876\n",
      "\n",
      "y_val_pred: \n",
      "0.877272727273\n",
      "[[168   9]\n",
      " [ 18  25]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.90      0.95      0.93       177\n",
      "          1       0.74      0.58      0.65        43\n",
      "\n",
      "avg / total       0.87      0.88      0.87       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "fit_to_pred(clf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# その他の機械学習アルゴリズム\n",
    "\n",
    "- サポートベクターマシン SVC\n",
    "- カーネルSVM\n",
    "- 決定木 DecisionTreeClassifier\n",
    "- ランダムフォレスト RandomForestClassifier\n",
    "- k近傍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.881278538813\n",
      "[[681  49]\n",
      " [ 55  91]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.93      0.93       730\n",
      "          1       0.65      0.62      0.64       146\n",
      "\n",
      "avg / total       0.88      0.88      0.88       876\n",
      "\n",
      "y_val_pred: \n",
      "0.859090909091\n",
      "[[167  10]\n",
      " [ 21  22]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.89      0.94      0.92       177\n",
      "          1       0.69      0.51      0.59        43\n",
      "\n",
      "avg / total       0.85      0.86      0.85       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc = SVC(kernel=\"linear\")\n",
    "fit_to_pred(svc, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.996575342466\n",
      "[[730   0]\n",
      " [  3 143]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       730\n",
      "          1       1.00      0.98      0.99       146\n",
      "\n",
      "avg / total       1.00      1.00      1.00       876\n",
      "\n",
      "y_val_pred: \n",
      "0.804545454545\n",
      "[[177   0]\n",
      " [ 43   0]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      1.00      0.89       177\n",
      "          1       0.00      0.00      0.00        43\n",
      "\n",
      "avg / total       0.65      0.80      0.72       220\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/terapyon/dev/scsk/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "fit_to_pred(k_svc, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.867579908676\n",
      "[[709  21]\n",
      " [ 95  51]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.97      0.92       730\n",
      "          1       0.71      0.35      0.47       146\n",
      "\n",
      "avg / total       0.85      0.87      0.85       876\n",
      "\n",
      "y_val_pred: \n",
      "0.840909090909\n",
      "[[171   6]\n",
      " [ 29  14]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.85      0.97      0.91       177\n",
      "          1       0.70      0.33      0.44        43\n",
      "\n",
      "avg / total       0.82      0.84      0.82       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=2,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=None, splitter='best')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = DecisionTreeClassifier(max_depth=2)\n",
    "fit_to_pred(tree, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.996575342466\n",
      "[[729   1]\n",
      " [  2 144]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       730\n",
      "          1       0.99      0.99      0.99       146\n",
      "\n",
      "avg / total       1.00      1.00      1.00       876\n",
      "\n",
      "y_val_pred: \n",
      "0.845454545455\n",
      "[[170   7]\n",
      " [ 27  16]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.86      0.96      0.91       177\n",
      "          1       0.70      0.37      0.48        43\n",
      "\n",
      "avg / total       0.83      0.85      0.83       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "fit_to_pred(rf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.897260273973\n",
      "[[693  37]\n",
      " [ 53  93]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.95      0.94       730\n",
      "          1       0.72      0.64      0.67       146\n",
      "\n",
      "avg / total       0.89      0.90      0.89       876\n",
      "\n",
      "y_val_pred: \n",
      "0.827272727273\n",
      "[[166  11]\n",
      " [ 27  16]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.86      0.94      0.90       177\n",
      "          1       0.59      0.37      0.46        43\n",
      "\n",
      "avg / total       0.81      0.83      0.81       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = KNeighborsClassifier()\n",
    "fit_to_pred(knn, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 交差検証(クロスバリデーション)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cv = KFold(5, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.83636364,  0.87214612,  0.88584475,  0.89954338,  0.85388128])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "cross_val_score(clf, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.79545455,  0.85844749,  0.80821918,  0.85388128,  0.83561644])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "cross_val_score(k_svc, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.84090909,  0.87671233,  0.84474886,  0.85388128,  0.81278539])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "cross_val_score(rf, X, y, cv=cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## F1-score で評価"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.58823529,  0.55384615,  0.5       ,  0.66666667,  0.63736264])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = LogisticRegression()\n",
    "cross_val_score(clf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/terapyon/dev/scsk/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/Users/terapyon/dev/scsk/env/lib/python3.6/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.        ,  0.        ,  0.08      ,  0.12121212,  0.05714286])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_svc = SVC(kernel=\"rbf\")\n",
    "cross_val_score(k_svc, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.23255814,  0.45070423,  0.49230769,  0.48717949,  0.50666667])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "cross_val_score(rf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# グリッドサーチ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param_grid = {'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(max_depth=2, n_estimators=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(rf, param_grid, cv=cv, n_jobs=-1, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 64 candidates, totalling 320 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 320 out of 320 | elapsed:    4.6s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=2, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=2, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=1)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87317518248175185"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 2, 'n_estimators': 40}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 64 candidates, totalling 320 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 320 out of 320 | elapsed:    4.9s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=2, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=2, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'max_depth': [2, 3, 4, 5, 10, 15, 20, 30], 'n_estimators': [2, 3, 4, 5, 10, 20, 30, 40]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='f1', verbose=1)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=2, n_estimators=2)\n",
    "grid_search = GridSearchCV(rf, param_grid, cv=cv, n_jobs=-1, verbose=1, scoring='f1')\n",
    "grid_search.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.628228780268\n",
      "{'max_depth': 2, 'n_estimators': 3}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.47368421,  0.63636364,  0.63636364,  0.66666667,  0.68493151])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=3, n_estimators=40)\n",
    "cross_val_score(rf, X, y, cv=cv, scoring=\"f1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 最終確認"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train_pred: \n",
      "0.883561643836\n",
      "[[681  49]\n",
      " [ 53  93]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.93      0.93      0.93       730\n",
      "          1       0.65      0.64      0.65       146\n",
      "\n",
      "avg / total       0.88      0.88      0.88       876\n",
      "\n",
      "y_val_pred: \n",
      "0.877272727273\n",
      "[[170   7]\n",
      " [ 20  23]]\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.89      0.96      0.93       177\n",
      "          1       0.77      0.53      0.63        43\n",
      "\n",
      "avg / total       0.87      0.88      0.87       220\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=3, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            n_estimators=40, n_jobs=1, oob_score=False, random_state=None,\n",
       "            verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(max_depth=3, n_estimators=40)\n",
    "fit_to_pred(rf, X_train, X_val, y_train, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0,\n",
       "       0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1,\n",
       "       0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0], dtype=int32)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.predict(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['clf_rf.db']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(rf, \"clf_rf.db\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}