Data Wrangling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Data Wrangling Lab**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we will perform the following:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   Identify duplicate values in the dataset.\n",
    "\n",
    "-   Remove duplicate values from the dataset.\n",
    "\n",
    "-   Identify missing values in the dataset.\n",
    "\n",
    "-   Impute the missing values in the dataset.\n",
    "\n",
    "-   Normalize data in the dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing pandas module.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the dataset into a dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/LargeData/m1_survey_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding duplicates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "154"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicated_rows = df[df.duplicated()]\n",
    "len(duplicated_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing duplicates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify if duplicates were actually dropped.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11398, 85)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Missing values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "      <th>NormalizedAnnualCompensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11547</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11548</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11549</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11550</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11551</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11398 rows × 86 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Respondent  MainBranch  Hobbyist  OpenSourcer  OpenSource  Employment  \\\n",
       "0           False       False     False        False       False       False   \n",
       "1           False       False     False        False       False       False   \n",
       "2           False       False     False        False       False       False   \n",
       "3           False       False     False        False       False       False   \n",
       "4           False       False     False        False       False       False   \n",
       "...           ...         ...       ...          ...         ...         ...   \n",
       "11547       False       False     False        False       False       False   \n",
       "11548       False       False     False        False       False       False   \n",
       "11549       False       False     False        False       False       False   \n",
       "11550       False       False     False        False       False       False   \n",
       "11551       False       False     False        False       False       False   \n",
       "\n",
       "       Country  Student  EdLevel  UndergradMajor  ...  SONewContent    Age  \\\n",
       "0        False    False    False           False  ...         False  False   \n",
       "1        False    False    False           False  ...          True  False   \n",
       "2        False    False    False           False  ...         False  False   \n",
       "3        False    False    False            True  ...         False  False   \n",
       "4        False    False    False           False  ...         False  False   \n",
       "...        ...      ...      ...             ...  ...           ...    ...   \n",
       "11547    False    False    False           False  ...         False  False   \n",
       "11548    False    False    False           False  ...         False  False   \n",
       "11549    False    False    False           False  ...         False  False   \n",
       "11550    False    False    False            True  ...          True  False   \n",
       "11551    False    False    False           False  ...         False  False   \n",
       "\n",
       "       Gender  Trans  Sexuality  Ethnicity  Dependents  SurveyLength  \\\n",
       "0       False  False      False      False       False         False   \n",
       "1       False  False      False      False       False         False   \n",
       "2       False  False      False      False       False         False   \n",
       "3       False  False      False      False       False         False   \n",
       "4       False  False      False      False       False         False   \n",
       "...       ...    ...        ...        ...         ...           ...   \n",
       "11547   False  False      False      False       False         False   \n",
       "11548   False  False      False      False       False         False   \n",
       "11549   False  False      False      False       False         False   \n",
       "11550   False  False      False      False       False         False   \n",
       "11551   False  False      False      False       False         False   \n",
       "\n",
       "       SurveyEase  NormalizedAnnualCompensation  \n",
       "0           False                         False  \n",
       "1           False                         False  \n",
       "2           False                         False  \n",
       "3           False                         False  \n",
       "4           False                         False  \n",
       "...           ...                           ...  \n",
       "11547       False                         False  \n",
       "11548       False                         False  \n",
       "11549       False                         False  \n",
       "11550       False                         False  \n",
       "11551       False                          True  \n",
       "\n",
       "[11398 rows x 86 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "missing_data = df.isnull()\n",
    "missing_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "112"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data = df[\"EdLevel\"].isnull().sum()\n",
    "missing_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_data = df[\"Country\"].isnull().sum()\n",
    "missing_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find out how many rows are missing in the column 'WorkLoc'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "False    11366\n",
      "True        32\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('WorkLoc')\n",
    "print (missing_data['WorkLoc'].value_counts()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imputing missing values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "Office                                            6806\n",
      "Home                                              3589\n",
      "Other place, such as a coworking space or cafe     971\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('WorkLoc')\n",
    "print (df['WorkLoc'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Employment\n",
      "Employed full-time    10968\n",
      "Employed part-time      430\n",
      "Name: Employment, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Employment')\n",
    "print (df['Employment'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UndergradMajor\n",
      "Computer science, computer engineering, or software engineering          6953\n",
      "Information systems, information technology, or system administration     794\n",
      "Another engineering discipline (ex. civil, electrical, mechanical)        759\n",
      "Web development or web design                                             410\n",
      "A natural science (ex. biology, chemistry, physics)                       403\n",
      "Mathematics or statistics                                                 372\n",
      "A business discipline (ex. accounting, finance, marketing)                244\n",
      "A social science (ex. anthropology, psychology, political science)        210\n",
      "A humanities discipline (ex. literature, history, philosophy)             207\n",
      "Fine arts or performing arts (ex. graphic design, music, studio art)      161\n",
      "I never declared a major                                                  124\n",
      "A health science (ex. nursing, pharmacy, radiology)                        24\n",
      "Name: UndergradMajor, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('UndergradMajor')\n",
    "print (df['UndergradMajor'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify the value that is most frequent (majority) in the WorkLoc column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The most frequent value in WorkLoc column is Office"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Impute (replace) all the empty rows in the column WorkLoc with the value that you have identified as majority.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['WorkLoc'].replace(np.nan, 'Office', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After imputation there should ideally not be any empty rows in the WorkLoc column.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify if imputing was successful.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WorkLoc\n",
      "False    11398\n",
      "Name: WorkLoc, dtype: int64\n",
      "\n",
      "WorkLoc\n",
      "Office                                            6838\n",
      "Home                                              3589\n",
      "Other place, such as a coworking space or cafe     971\n",
      "Name: WorkLoc, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "missing_data = df.isnull()\n",
    "print('WorkLoc')\n",
    "print (missing_data['WorkLoc'].value_counts())  \n",
    "print('')\n",
    "print('WorkLoc')\n",
    "print (df['WorkLoc'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Listing out the various categories in the column 'CompFreq'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompFreq\n",
      "Yearly     6073\n",
      "Monthly    4788\n",
      "Weekly      331\n",
      "Name: CompFreq, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('CompFreq')\n",
    "print (df['CompFreq'].value_counts()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a new column named 'NormalizedAnnualCompensation'. Use the hint given below if needed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "      <th>NormalizedAnnualCompensation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>61000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>New Zealand</td>\n",
       "      <td>No</td>\n",
       "      <td>Some college/university study without earning ...</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Bisexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "      <td>138000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Cour...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "      <td>348000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Australia</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>29.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;Multiracial</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 86 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Respondent                      MainBranch Hobbyist  \\\n",
       "0           4  I am a developer by profession       No   \n",
       "1           9  I am a developer by profession      Yes   \n",
       "2          13  I am a developer by profession      Yes   \n",
       "3          16  I am a developer by profession      Yes   \n",
       "4          17  I am a developer by profession      Yes   \n",
       "\n",
       "                                         OpenSourcer  \\\n",
       "0                                              Never   \n",
       "1                         Once a month or more often   \n",
       "2  Less than once a month but more than once per ...   \n",
       "3                                              Never   \n",
       "4  Less than once a month but more than once per ...   \n",
       "\n",
       "                                          OpenSource          Employment  \\\n",
       "0  The quality of OSS and closed source software ...  Employed full-time   \n",
       "1  The quality of OSS and closed source software ...  Employed full-time   \n",
       "2  OSS is, on average, of HIGHER quality than pro...  Employed full-time   \n",
       "3  The quality of OSS and closed source software ...  Employed full-time   \n",
       "4  The quality of OSS and closed source software ...  Employed full-time   \n",
       "\n",
       "          Country Student                                            EdLevel  \\\n",
       "0   United States      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "1     New Zealand      No  Some college/university study without earning ...   \n",
       "2   United States      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "3  United Kingdom      No        Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "4       Australia      No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                                      UndergradMajor  ...  \\\n",
       "0  Computer science, computer engineering, or sof...  ...   \n",
       "1  Computer science, computer engineering, or sof...  ...   \n",
       "2  Computer science, computer engineering, or sof...  ...   \n",
       "3                                                NaN  ...   \n",
       "4  Computer science, computer engineering, or sof...  ...   \n",
       "\n",
       "                                        SONewContent   Age Gender Trans  \\\n",
       "0  Tech articles written by other developers;Indu...  22.0    Man    No   \n",
       "1                                                NaN  23.0    Man    No   \n",
       "2  Tech articles written by other developers;Cour...  28.0    Man    No   \n",
       "3  Tech articles written by other developers;Indu...  26.0    Man    No   \n",
       "4  Tech articles written by other developers;Indu...  29.0    Man    No   \n",
       "\n",
       "                 Sexuality                              Ethnicity Dependents  \\\n",
       "0  Straight / Heterosexual           White or of European descent         No   \n",
       "1                 Bisexual           White or of European descent         No   \n",
       "2  Straight / Heterosexual           White or of European descent        Yes   \n",
       "3  Straight / Heterosexual           White or of European descent         No   \n",
       "4  Straight / Heterosexual  Hispanic or Latino/Latina;Multiracial         No   \n",
       "\n",
       "            SurveyLength                  SurveyEase  \\\n",
       "0  Appropriate in length                        Easy   \n",
       "1  Appropriate in length  Neither easy nor difficult   \n",
       "2  Appropriate in length                        Easy   \n",
       "3  Appropriate in length  Neither easy nor difficult   \n",
       "4  Appropriate in length                        Easy   \n",
       "\n",
       "  NormalizedAnnualCompensation  \n",
       "0                      61000.0  \n",
       "1                     138000.0  \n",
       "2                      90000.0  \n",
       "3                     348000.0  \n",
       "4                      90000.0  \n",
       "\n",
       "[5 rows x 86 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_annual_comp = []\n",
    "\n",
    "for i in range(len(df)):\n",
    "    \n",
    "    if df['CompFreq'].iloc[i] == 'Weekly':\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*52)\n",
    "    elif df['CompFreq'].iloc[i] == 'Monthly':\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*12)\n",
    "    else:\n",
    "        norm_annual_comp.append(df['CompTotal'].iloc[i]*1)\n",
    "\n",
    "df['NormalizedAnnualCompensation']= norm_annual_comp\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['NormalizedAnnualCompensation'].median()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}