Phishing_Website_Detection_featureExtraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phishing Website Detection - Feature Extraction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step -1 : Data preprocessing \n",
    "\n",
    "This dataset contains few website links (Some of them are legitimate websites and a few are fake websites)\n",
    "\n",
    "Pre-Processing the data before building a model and also Extracting the features from the data based on certain conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#importing numpy and pandas which are required for data pre-processing\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Loading the data\n",
    "raw_data = pd.read_csv(\"C:100-legitimate-art.txt\") #loading only 100 samples (art websites data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>websites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http://www.emuck.com:3000/archive/egan.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http://danoday.com/summit.shtml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http://groups.yahoo.com/group/voice_actor_appr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http://voice-international.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http://www.livinglegendsltd.com/</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            websites\n",
       "0        http://www.emuck.com:3000/archive/egan.html\n",
       "1                    http://danoday.com/summit.shtml\n",
       "2  http://groups.yahoo.com/group/voice_actor_appr...\n",
       "3                    http://voice-international.com/\n",
       "4                   http://www.livinglegendsltd.com/"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to split the data according to parts of the URL\n",
    "\n",
    "A typical URL could have the form http://www.example.com/index.html, which indicates a protocol (http), a hostname (www.example.com), and a file name (index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Detailed explanation about URL:  1.https://doepud.co.uk/blog/anatomy-of-a-url                                                    \n",
    "                                 2.https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL                  \n",
    "                                 3.https://techwelkin.com/understanding-the-components-and-structure-of-a-url\n",
    "                                        \n",
    "                                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         [http, www.emuck.com:3000/archive/egan.html]\n",
       "1                     [http, danoday.com/summit.shtml]\n",
       "2    [http, groups.yahoo.com/group/voice_actor_appr...\n",
       "3                     [http, voice-international.com/]\n",
       "4                    [http, www.livinglegendsltd.com/]\n",
       "Name: websites, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_data['websites'].str.split(\"://\").head() #Here we divided the protocol from the entire URL. but need it to be divided it \n",
    "                                                 #seperate column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "refer to this link for splitting of data --> https://apassionatechie.wordpress.com/2018/02/24/how-do-i-split-a-string-into-several-columns-in-a-dataframe-with-pandas-python/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seperation_of_protocol = raw_data['websites'].str.split(\"://\",expand = True) #expand argument in the split method will give you a new column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000/archive/egan.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com/summit.shtml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com/group/voice_actor_appreciatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com/</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      0                                                  1\n",
       "0  http               www.emuck.com:3000/archive/egan.html\n",
       "1  http                           danoday.com/summit.shtml\n",
       "2  http  groups.yahoo.com/group/voice_actor_appreciatio...\n",
       "3  http                           voice-international.com/\n",
       "4  http                          www.livinglegendsltd.com/"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seperation_of_protocol.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(seperation_of_protocol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seperation_domain_name = seperation_of_protocol[1].str.split(\"/\",1,expand = True) #split(seperator,no of splits according to seperator(delimiter),expand)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(seperation_domain_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seperation_domain_name.columns=[\"domain_name\",\"address\"] #renaming columns of data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                domain_name                                            address\n",
       "0        www.emuck.com:3000                                  archive/egan.html\n",
       "1               danoday.com                                       summit.shtml\n",
       "2          groups.yahoo.com  group/voice_actor_appreciation/links/events_an...\n",
       "3   voice-international.com                                                   \n",
       "4  www.livinglegendsltd.com                                                   "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seperation_domain_name.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Concatenation of data frames\n",
    "splitted_data = pd.concat([seperation_of_protocol[0],seperation_domain_name],axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data.columns = ['protocol','domain_name','address']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  \n",
       "0                                  archive/egan.html  \n",
       "1                                       summit.shtml  \n",
       "2  group/voice_actor_appreciation/links/events_an...  \n",
       "3                                                     \n",
       "4                                                     "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Domain name column can be further sub divided into domain_names as well as sub_domain_names \n",
    "\n",
    "Similarly, address column can also be further sub divided into path,query_string,file.................."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(splitted_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Feature-1\n",
    "\n",
    "1.Long URL to Hide the Suspicious Part\n",
    "\n",
    "If the length of the URL is greater than or equal 54 characters then the URL classified as phishing\n",
    "\n",
    "\n",
    "0 --- indicates legitimate\n",
    "\n",
    "1 --- indicates Phishing\n",
    "\n",
    "2 --- indicates Suspicious"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def long_url(l):\n",
    "    \"\"\"This function is defined in order to differntiate website based on the length of the URL\"\"\"\n",
    "    if len(l) < 54:\n",
    "        return 0\n",
    "    elif len(l) >= 54 and len(l) <= 75:\n",
    "        return 2\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Applying the above defined function in order to divide the websites into 3 categories\n",
    "splitted_data['long_url'] = raw_data['websites'].apply(long_url) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>http</td>\n",
       "      <td>voicechasers.com</td>\n",
       "      <td>forum/viewforum.php?f=8</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name                  address  long_url\n",
       "0     http        www.emuck.com:3000        archive/egan.html         0\n",
       "1     http               danoday.com             summit.shtml         0\n",
       "3     http   voice-international.com                                  0\n",
       "4     http  www.livinglegendsltd.com                                  0\n",
       "5     http          voicechasers.com  forum/viewforum.php?f=8         0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Will show the results only the websites which are legitimate according to above condition as 0 is legitimate website\n",
    "splitted_data[splitted_data.long_url == 0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature-2\n",
    "\n",
    "2.URL’s having “@” Symbol\n",
    "\n",
    "Using “@” symbol in the URL leads the browser to ignore everything preceding the “@” symbol and the real address often follows the “@” symbol.\n",
    "\n",
    "IF {Url Having @ Symbol→ Phishing\n",
    "    Otherwise→ Legitimate }\n",
    "\n",
    "\n",
    "0 --- indicates legitimate\n",
    "\n",
    "1 --- indicates Phishing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def have_at_symbol(l):\n",
    "    \"\"\"This function is used to check whether the URL contains @ symbol or not\"\"\"\n",
    "    if \"@\" in l:\n",
    "        return 1\n",
    "    return 0\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['having_@_symbol'] = raw_data['websites'].apply(have_at_symbol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  \n",
       "0                0  \n",
       "1                0  \n",
       "2                0  \n",
       "3                0  \n",
       "4                0  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature-3\n",
    "\n",
    "3.Redirecting using “//”\n",
    "\n",
    "The existence of “//” within the URL path means that the user will be redirected to another website.\n",
    "An example of such URL’s is: “http://www.legitimate.com//http://www.phishing.com”. \n",
    "We examine the location where the “//” appears. \n",
    "We find that if the URL starts with “HTTP”, that means the “//” should appear in the sixth position. \n",
    "However, if the URL employs “HTTPS” then the “//” should appear in seventh position.\n",
    "\n",
    "IF {ThePosition of the Last Occurrence of \"//\" in the URL > 7→ Phishing\n",
    "    \n",
    "    Otherwise→ Legitimate\n",
    "\n",
    "0 --- indicates legitimate\n",
    "\n",
    "1 --- indicates Phishing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def redirection(l):\n",
    "    \"\"\"If the url has symbol(//) after protocol then such URL is to be classified as phishing \"\"\"\n",
    "    if \"//\" in l:\n",
    "        return 1\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['redirection_//_symbol'] = seperation_of_protocol[1].apply(redirection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  \n",
       "0                0                      0  \n",
       "1                0                      0  \n",
       "2                0                      0  \n",
       "3                0                      0  \n",
       "4                0                      0  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature-4\n",
    "\n",
    "4.Adding Prefix or Suffix Separated by (-) to the Domain\n",
    "\n",
    "The dash symbol is rarely used in legitimate URLs. Phishers tend to add prefixes or suffixes separated by (-) to the domain name\n",
    "so that users feel that they are dealing with a legitimate webpage. \n",
    "\n",
    "For example http://www.Confirme-paypal.com/.\n",
    "    \n",
    "IF {Domain Name Part Includes (−) Symbol → Phishing\n",
    "    \n",
    "    Otherwise → Legitimate\n",
    "    \n",
    "1 --> indicates phishing\n",
    "\n",
    "0 --> indicates legitimate\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prefix_suffix_seperation(l):\n",
    "    if '-' in l:\n",
    "        return 1\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['prefix_suffix_seperation'] = seperation_domain_name['domain_name'].apply(prefix_suffix_seperation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \n",
       "0                0                      0                         0  \n",
       "1                0                      0                         0  \n",
       "2                0                      0                         0  \n",
       "3                0                      0                         1  \n",
       "4                0                      0                         0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 5\n",
    "\n",
    "5. Sub-Domain and Multi Sub-Domains\n",
    "\n",
    "The legitimate URL link has two dots in the URL since we can ignore typing “www.”. \n",
    "If the number of dots is equal to three then the URL is classified as “Suspicious” since it has one sub-domain.\n",
    "However, if the dots are greater than three it is classified as “Phishy” since it will have multiple sub-domains\n",
    "\n",
    "0 --- indicates legitimate\n",
    "\n",
    "1 --- indicates Phishing\n",
    "\n",
    "2 --- indicates Suspicious\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sub_domains(l):\n",
    "    if l.count('.') < 3:\n",
    "        return 0\n",
    "    elif l.count('.') == 3:\n",
    "        return 2\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['sub_domains'] = splitted_data['domain_name'].apply(sub_domains)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  \n",
       "0            0  \n",
       "1            0  \n",
       "2            0  \n",
       "3            0  \n",
       "4            0  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Feature-6\n",
    "\n",
    "6.Using the IP Address\n",
    "\n",
    "If an IP address is used as an alternative of the domain name in the URL, such as “http://125.98.3.123/fake.html”,\n",
    "users can be sure that someone is trying to steal their personal information. Sometimes, \n",
    "the IP address is even transformed into hexadecimal code as shown in the following link “http://0x58.0xCC.0xCA.0x62/2/paypal.ca/index.html”.\n",
    "\n",
    "        Rule: IF{If The Domain Part has an IP Address → Phishing\n",
    "                 Otherwise→ Legitimate\n",
    "\n",
    "                 1 --> indicates phishing\n",
    "\n",
    "                 0 --> indicates legitimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "def having_ip_address(url):\n",
    "    match=re.search('(([01]?\\\\d\\\\d?|2[0-4]\\\\d|25[0-5])\\\\.([01]?\\\\d\\\\d?|2[0-4]\\\\d|25[0-5])\\\\.([01]?\\\\d\\\\d?|2[0-4]\\\\d|25[0-5])\\\\.([01]?\\\\d\\\\d?|2[0-4]\\\\d|25[0-5])\\\\/)|'  #IPv4\n",
    "                    '((0x[0-9a-fA-F]{1,2})\\\\.(0x[0-9a-fA-F]{1,2})\\\\.(0x[0-9a-fA-F]{1,2})\\\\.(0x[0-9a-fA-F]{1,2})\\\\/)'  #IPv4 in hexadecimal\n",
    "                    '(?:[a-fA-F0-9]{1,4}:){7}[a-fA-F0-9]{1,4}',url)     #Ipv6\n",
    "    if match:\n",
    "        #print match.group()\n",
    "        return 1\n",
    "    else:\n",
    "        #print 'No matching pattern found'\n",
    "        return 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['having_ip_address'] = raw_data['websites'].apply(having_ip_address)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "      <th>having_ip_address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  having_ip_address  \n",
       "0            0                  0  \n",
       "1            0                  0  \n",
       "2            0                  0  \n",
       "3            0                  0  \n",
       "4            0                  0  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature-7\n",
    "\n",
    "7.Using URL Shortening Services “TinyURL”\n",
    "\n",
    "URL shortening is a method on the “World Wide Web” in which a URL may be made considerably smaller in length and still lead to the required webpage. \n",
    "This is accomplished by means of an “HTTP Redirect” on a domain name that is short, which links to the webpage that has a long URL. \n",
    "For example, the URL “http://portal.hud.ac.uk/” can be shortened to “bit.ly/19DXSk4”.\n",
    "    \n",
    "Rule: IF{TinyURL → Phishing\n",
    "         \n",
    "         Otherwise→ Legitimate\n",
    "         \n",
    "                 1 --> indicates phishing\n",
    "\n",
    "                 0 --> indicates legitimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#we have imported re module in the above feature. So need not to import again\n",
    "def shortening_service(url):\n",
    "    match=re.search('bit\\.ly|goo\\.gl|shorte\\.st|go2l\\.ink|x\\.co|ow\\.ly|t\\.co|tinyurl|tr\\.im|is\\.gd|cli\\.gs|'\n",
    "                    'yfrog\\.com|migre\\.me|ff\\.im|tiny\\.cc|url4\\.eu|twit\\.ac|su\\.pr|twurl\\.nl|snipurl\\.com|'\n",
    "                    'short\\.to|BudURL\\.com|ping\\.fm|post\\.ly|Just\\.as|bkite\\.com|snipr\\.com|fic\\.kr|loopt\\.us|'\n",
    "                    'doiop\\.com|short\\.ie|kl\\.am|wp\\.me|rubyurl\\.com|om\\.ly|to\\.ly|bit\\.do|t\\.co|lnkd\\.in|'\n",
    "                    'db\\.tt|qr\\.ae|adf\\.ly|goo\\.gl|bitly\\.com|cur\\.lv|tinyurl\\.com|ow\\.ly|bit\\.ly|ity\\.im|'\n",
    "                    'q\\.gs|is\\.gd|po\\.st|bc\\.vc|twitthis\\.com|u\\.to|j\\.mp|buzurl\\.com|cutt\\.us|u\\.bb|yourls\\.org|'\n",
    "                    'x\\.co|prettylinkpro\\.com|scrnch\\.me|filoops\\.info|vzturl\\.com|qr\\.net|1url\\.com|tweez\\.me|v\\.gd|tr\\.im|link\\.zip\\.net',url)\n",
    "    if match:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['shortening_service'] = raw_data['websites'].apply(shortening_service)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "      <th>having_ip_address</th>\n",
       "      <th>shortening_service</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  having_ip_address  shortening_service  \n",
       "0            0                  0                   0  \n",
       "1            0                  0                   0  \n",
       "2            0                  0                   0  \n",
       "3            0                  0                   0  \n",
       "4            0                  0                   0  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 8 \n",
    "\n",
    "8.The Existence of “HTTPS” Token in the Domain Part of the URL\n",
    "\n",
    "The phishers may add the “HTTPS” token to the domain part of a URL in order to trick users.\n",
    "For example, http://https-www-paypal-it-webapps-mpp-home.soft-hair.com/.\n",
    "\n",
    "    Rule: IF{Using HTTP Token in Domain Part of The URL→ Phishing\n",
    "             \n",
    "             Otherwise→ Legitimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def https_token(url):\n",
    "    match=re.search('https://|http://',url)\n",
    "    if match.start(0)==0:\n",
    "        url=url[match.end(0):]\n",
    "    match=re.search('http|https',url)\n",
    "    if match:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['https_token'] = raw_data['websites'].apply(https_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "      <th>having_ip_address</th>\n",
       "      <th>shortening_service</th>\n",
       "      <th>https_token</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  having_ip_address  shortening_service  https_token  \n",
       "0            0                  0                   0            0  \n",
       "1            0                  0                   0            0  \n",
       "2            0                  0                   0            0  \n",
       "3            0                  0                   0            0  \n",
       "4            0                  0                   0            0  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Feature - 9\n",
    "\n",
    "9.Abnormal_URL\n",
    "\n",
    "This feature can be extracted from WHOIS database. For a legitimate website, identity is typically part of its URL.\n",
    "\n",
    "Rule: IF {The Host Name Is Not Included In URL → Phishing \n",
    "          \n",
    "          Otherwise→ Legitimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def abnormal_url_sub(domain,url):\n",
    "    hostname=domain.name\n",
    "    match=re.search(hostname,url)\n",
    "    if match:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def abnormal_url_main(domain,url):\n",
    "    dns = 0\n",
    "    try:\n",
    "        domain_name = whois.whois(domain)\n",
    "    except:\n",
    "        dns = 1\n",
    "        \n",
    "    if dns == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return abnormal_url_sub(domain_name,url)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ?\n",
    " #x - - cross check \n",
    "splitted_data['abnormal_url'] = splitted_data['domain_name'].apply(abnormal_url_main)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 10:\n",
    "    \n",
    "10.Google Index\n",
    "\n",
    "This feature examines whether a website is in Google’s index or not. When a site is indexed by Google,\n",
    "it is displayed on search results (Webmaster resources, 2014). \n",
    "Usually, phishing webpages are merely accessible for a short period and as a result, \n",
    "many phishing webpages may not be found on the Google index.\n",
    "\n",
    "Rule: IF{Webpage Indexed by Google → Legitimate\n",
    "         \n",
    "         Otherwise → Phishing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import google\n",
    "def google_index(url):\n",
    "    site=google.search(url,5)\n",
    "    if site:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['google_index'] = raw_data['websites'].apply(google_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 11\n",
    "\n",
    "11.Website Traffic\n",
    "\n",
    "This feature measures the popularity of the website by determining the number of visitors and the number of pages they visit.\n",
    "However, since phishing websites live for a short period of time, they may not be recognized by the Alexa database \n",
    "(Alexa the Web Information Company., 1996). By reviewing our dataset, we find that in worst scenarios,\n",
    "legitimate websites ranked among the top 100,000. Furthermore,\n",
    "if the domain has no traffic or is not recognized by the Alexa database, it is classified as “Phishing”.\n",
    "Otherwise, it is classified as “Suspicious”.\n",
    "\n",
    "Rule: IF{Website Rank<100,000 → LegitimateWebsite Rank>100,\n",
    "         \n",
    "         000 →SuspiciousOtherwise → Phish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import urllib.request\n",
    "def web_traffic(url):\n",
    "    try:\n",
    "        rank = BeautifulSoup(urllib.request.urlopen(\"http://data.alexa.com/data?cli=10&dat=s&url=\" + url).read(), \"xml\").find(\"REACH\")['RANK']\n",
    "    except TypeError:\n",
    "        return 1\n",
    "    rank= int(rank)\n",
    "    if (rank<100000):\n",
    "        return 0\n",
    "    else:\n",
    "        return 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitted_data['web_traffic'] = raw_data['websites'].apply(web_traffic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "      <th>having_ip_address</th>\n",
       "      <th>shortening_service</th>\n",
       "      <th>https_token</th>\n",
       "      <th>web_traffic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  having_ip_address  shortening_service  https_token  \\\n",
       "0            0                  0                   0            0   \n",
       "1            0                  0                   0            0   \n",
       "2            0                  0                   0            0   \n",
       "3            0                  0                   0            0   \n",
       "4            0                  0                   0            0   \n",
       "\n",
       "   web_traffic  \n",
       "0            1  \n",
       "1            2  \n",
       "2            0  \n",
       "3            1  \n",
       "4            1  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 11\n",
    "\n",
    "Domain Registration Length\n",
    "\n",
    "Based on the fact that a phishing website lives for a short period of time, \n",
    "we believe that trustworthy domains are regularly paid for several years in advance. \n",
    "In our dataset, we find that the longest fraudulent domains have been used for one year only.\n",
    "\n",
    "Rule: IF{Domains Expires on≤ 1 years → Phishing\n",
    "         \n",
    "         Otherwise→ Legitimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import whois\n",
    "from datetime import datetime\n",
    "import time\n",
    "def domain_registration_length_sub(domain):\n",
    "    expiration_date = domain.expiration_date\n",
    "    today = time.strftime('%Y-%m-%d')\n",
    "    today = datetime.strptime(today, '%Y-%m-%d')\n",
    "    if expiration_date is None:\n",
    "        return 1\n",
    "    elif type(expiration_date) is list or type(today) is list :\n",
    "        return 2             #If it is a type of list then we can't select a single value from list. So,it is regarded as suspected website  \n",
    "    else:\n",
    "        registration_length = abs((expiration_date - today).days)\n",
    "        if registration_length / 365 <= 1:\n",
    "            return 1\n",
    "        else:\n",
    "            return 0\n",
    "\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def domain_registration_length_main(domain):\n",
    "    dns = 0\n",
    "    try:\n",
    "        domain_name = whois.whois(domain)\n",
    "    except:\n",
    "        dns = 1\n",
    "        \n",
    "    if dns == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return domain_registration_length_sub(domain_name)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitted_data['domain_registration_length'] = splitted_data['domain_name'].apply(domain_registration_length_main)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>domain_name</th>\n",
       "      <th>address</th>\n",
       "      <th>long_url</th>\n",
       "      <th>having_@_symbol</th>\n",
       "      <th>redirection_//_symbol</th>\n",
       "      <th>prefix_suffix_seperation</th>\n",
       "      <th>sub_domains</th>\n",
       "      <th>having_ip_address</th>\n",
       "      <th>shortening_service</th>\n",
       "      <th>https_token</th>\n",
       "      <th>web_traffic</th>\n",
       "      <th>domain_registration_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http</td>\n",
       "      <td>www.emuck.com:3000</td>\n",
       "      <td>archive/egan.html</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http</td>\n",
       "      <td>danoday.com</td>\n",
       "      <td>summit.shtml</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>groups.yahoo.com</td>\n",
       "      <td>group/voice_actor_appreciation/links/events_an...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http</td>\n",
       "      <td>voice-international.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http</td>\n",
       "      <td>www.livinglegendsltd.com</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol               domain_name  \\\n",
       "0     http        www.emuck.com:3000   \n",
       "1     http               danoday.com   \n",
       "2     http          groups.yahoo.com   \n",
       "3     http   voice-international.com   \n",
       "4     http  www.livinglegendsltd.com   \n",
       "\n",
       "                                             address  long_url  \\\n",
       "0                                  archive/egan.html         0   \n",
       "1                                       summit.shtml         0   \n",
       "2  group/voice_actor_appreciation/links/events_an...         1   \n",
       "3                                                            0   \n",
       "4                                                            0   \n",
       "\n",
       "   having_@_symbol  redirection_//_symbol  prefix_suffix_seperation  \\\n",
       "0                0                      0                         0   \n",
       "1                0                      0                         0   \n",
       "2                0                      0                         0   \n",
       "3                0                      0                         1   \n",
       "4                0                      0                         0   \n",
       "\n",
       "   sub_domains  having_ip_address  shortening_service  https_token  \\\n",
       "0            0                  0                   0            0   \n",
       "1            0                  0                   0            0   \n",
       "2            0                  0                   0            0   \n",
       "3            0                  0                   0            0   \n",
       "4            0                  0                   0            0   \n",
       "\n",
       "   web_traffic  domain_registration_length  \n",
       "0            1                           1  \n",
       "1            2                           1  \n",
       "2            0                           2  \n",
       "3            1                           1  \n",
       "4            1                           1  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitted_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Testing the above function\n",
    "domain_registration_length_main('www.google.com')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "domain_registration_length_main('www.w3schools.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 12\n",
    "\n",
    "This feature can be extracted from WHOIS database (Whois 2005). \n",
    "Most phishing websites live for a short period of time. \n",
    "By reviewing our dataset, we find that the minimum age of the legitimate domain is 6 months.\n",
    "\n",
    "Rule: IF {Age Of Domain≥6 months → Legitimate\n",
    "          \n",
    "          Otherwise → Phishing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def age_of_domain_sub(domain):\n",
    "    creation_date = domain.creation_date\n",
    "    expiration_date = domain.expiration_date\n",
    "    if ((expiration_date is None) or (creation_date is None)):\n",
    "        return 1\n",
    "    elif ((type(expiration_date) is list) or (type(creation_date) is list)):\n",
    "        return 2\n",
    "    else:\n",
    "        ageofdomain = abs((expiration_date - creation_date).days)\n",
    "        if ((ageofdomain/30) < 6):\n",
    "            return 1\n",
    "        else:\n",
    "            return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def age_of_domain_main(domain):\n",
    "    dns = 0\n",
    "    try:\n",
    "        domain_name = whois.whois(domain)\n",
    "    except:\n",
    "        dns = 1\n",
    "        \n",
    "    if dns == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return age_of_domain_sub(domain_name)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['age_of_domain'] = splitted_data['domain_name'].apply(age_of_domain_main)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitted_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 13\n",
    "\n",
    "DNS Record\n",
    "\n",
    "For phishing websites, either the claimed identity is not recognized by the WHOIS database (Whois 2005) or \n",
    "no records founded for the hostname (Pan and Ding 2006). \n",
    "If the DNS record is empty or not found then the website is classified as “Phishing”, otherwise it is classified as “Legitimate”.\n",
    "\n",
    "Rule: IF{no DNS Record For The Domain → Phishing\n",
    "         \n",
    "         Otherwise→ Legitimate\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dns_record(domain):\n",
    "    dns = 0\n",
    "    try:\n",
    "        domain_name = whois.whois(domain)\n",
    "        print(domain_name)\n",
    "    except:\n",
    "        dns = 1\n",
    "        \n",
    "    if dns == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return dns\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['dns_record'] = splitted_data['domain_name'].apply(dns_record)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitted_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature - 14\n",
    "\n",
    "Statistical-Reports Based Feature\n",
    "\n",
    "Several parties such as PhishTank (PhishTank Stats, 2010-2012),\n",
    "and StopBadware (StopBadware, 2010-2012) formulate numerous statistical reports on phishing websites at every given period of time\n",
    "some are monthly and others are quarterly. \n",
    "\n",
    "Rule: IF{Host Belongs to Top Phishing IPs or Top Phishing Domains → Phishing\n",
    "         \n",
    "         Otherwise → Legitimate\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import socket\n",
    "def statistical_report(url):\n",
    "    hostname = url\n",
    "    h = [(x.start(0), x.end(0)) for x in re.finditer('https://|http://|www.|https://www.|http://www.', hostname)]\n",
    "    z = int(len(h))\n",
    "    if z != 0:\n",
    "        y = h[0][1]\n",
    "        hostname = hostname[y:]\n",
    "        h = [(x.start(0), x.end(0)) for x in re.finditer('/', hostname)]\n",
    "        z = int(len(h))\n",
    "        if z != 0:\n",
    "            hostname = hostname[:h[0][0]]\n",
    "    url_match=re.search('at\\.ua|usa\\.cc|baltazarpresentes\\.com\\.br|pe\\.hu|esy\\.es|hol\\.es|sweddy\\.com|myjino\\.ru|96\\.lt|ow\\.ly',url)\n",
    "    try:\n",
    "        ip_address = socket.gethostbyname(hostname)\n",
    "        ip_match=re.search('146\\.112\\.61\\.108|213\\.174\\.157\\.151|121\\.50\\.168\\.88|192\\.185\\.217\\.116|78\\.46\\.211\\.158|181\\.174\\.165\\.13|46\\.242\\.145\\.103|121\\.50\\.168\\.40|83\\.125\\.22\\.219|46\\.242\\.145\\.98|107\\.151\\.148\\.44|107\\.151\\.148\\.107|64\\.70\\.19\\.203|199\\.184\\.144\\.27|107\\.151\\.148\\.108|107\\.151\\.148\\.109|119\\.28\\.52\\.61|54\\.83\\.43\\.69|52\\.69\\.166\\.231|216\\.58\\.192\\.225|118\\.184\\.25\\.86|67\\.208\\.74\\.71|23\\.253\\.126\\.58|104\\.239\\.157\\.210|175\\.126\\.123\\.219|141\\.8\\.224\\.221|10\\.10\\.10\\.10|43\\.229\\.108\\.32|103\\.232\\.215\\.140|69\\.172\\.201\\.153|216\\.218\\.185\\.162|54\\.225\\.104\\.146|103\\.243\\.24\\.98|199\\.59\\.243\\.120|31\\.170\\.160\\.61|213\\.19\\.128\\.77|62\\.113\\.226\\.131|208\\.100\\.26\\.234|195\\.16\\.127\\.102|195\\.16\\.127\\.157|34\\.196\\.13\\.28|103\\.224\\.212\\.222|172\\.217\\.4\\.225|54\\.72\\.9\\.51|192\\.64\\.147\\.141|198\\.200\\.56\\.183|23\\.253\\.164\\.103|52\\.48\\.191\\.26|52\\.214\\.197\\.72|87\\.98\\.255\\.18|209\\.99\\.17\\.27|216\\.38\\.62\\.18|104\\.130\\.124\\.96|47\\.89\\.58\\.141|78\\.46\\.211\\.158|54\\.86\\.225\\.156|54\\.82\\.156\\.19|37\\.157\\.192\\.102|204\\.11\\.56\\.48|110\\.34\\.231\\.42',ip_address)  \n",
    "    except:\n",
    "        return 1\n",
    "\n",
    "    if url_match:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitted_data['statistical_report'] = raw_data['websites'].apply(statistical_report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "splitted_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whois."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}