01 - Python and Pandas-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01 - Python and Pandas Introduction\n",
    "<p class=\"lead\">\n",
    "Michelle Brown Notes v 1.5<br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outline\n",
    "\n",
    "<!-- MarkdownTOC autolink=true autoanchor=true bracket=round -->\n",
    "\n",
    "- [Why Python?](#why)\n",
    "- [Introduction to DataFrames](#df)\n",
    "- [Quickly viewing the data](#view)\n",
    "- [Filtering the dataframe](#filter)\n",
    "- [Renaming, adding, and dropping columns](#rename)\n",
    "- [Pivot Tables](#pivot)\n",
    "<!-- /MarkdownTOC -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"why\"></a>\n",
    "# Python and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>What is Jupyter Notebook?</b>\n",
    "Jupyter Notebook, previously called \"iPython Notebook\" prior to version 4.0, is a way of interacting with Python code using a web browser. It is a very useful instructional tool that we will be using to introduce you to analysis with Python. Notebooks have the file extensions \"ipynb\" which are abbreviations of \"iPython Notebook\". Some websites such as nbviewer.ipython.org or www.github.com can view .ipynb files directly as rendered HTML. However, these are not interactive versions of the notebook, meaning, they are not running the python kernel which evaluates/interacts with the code. So the notebook is just a static version of the code contained inside of it.\n",
    "\n",
    "In order to interact with notebook and start coding, you will need to launch Terminal (for Mac and Linux users). For Windows users, you will need to launch the Command Prompt Tool using the `cmd.exe` program.<br>\n",
    "\n",
    "<b>Why Python?</b><br>\n",
    "Python is an easy-to-get-started language which handles data wrangling tasks in a simple and straightforward way. Python code is often said to be almost like pseudocode, since it allows you to express very powerful ideas in few lines of code while being very readable. It does not cost anything to install and use. Python also has a very open community that is quite supportive of new users. It also has many helpful packages that you can import and use that are well developed and documented. The Python language has the advantage of these Jupyter notebooks where you can easily use to add in comments and instructions. And it is easy to share the notebooks. The Python language is also very versatile and can be used to build applications. Python is a high-level, <a href=\"/files/images/matrix.gif\">dynamically typed multiparadigm</a> programming language. \n",
    "\n",
    "<b>Pandas Data Analysis Library</b><br>\n",
    "Python and <a href=\"http://pandas.pydata.org/\">Pandas</a> can easily read in and handle very large datasets (like voters lists with millions of records) which are not possible in excel. The pandas package for python was built to use data structures that are tabular (which is a commonn structure for election datasets). Here is a <a href=\"https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf\"> Pandas Cheat Sheet </a> for common tasks. \n",
    "\n",
    "A Note on the 2.7 version of Python:<br>\n",
    "There are two major Python versions (or interpreters). Python 2.X and Python 3.X. The most recent stable version of Python 2.X is 2.7, which is the Python version used in this notebook. The newest recent version of Python 3.X is Python 3.4, which is also the newest Python version available but is not considered stable (as of Jan 2017) and not all of the packages will work in 3.4. For now, assume code you write for 2.7 will not work in 3.4. (The term used to describe this is to say that 3.4 breaks <i>backwards compatibility</i>.)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"df\"></a>\n",
    "# Dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we are going to import the analysis module called Pandas as a variable called 'pd' so we can use it's associated methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# As an aside if you want to see what version of pandas you have, uncomment (remove the #) in the last line below. \n",
    "# I had version 0.18\n",
    "# pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a small and simple dataset, the Polling Station List 2008 file. We'll use pandas to read the file as a dataframe and we'll call the dataframe ps2008:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008 = pd.read_csv('data/Polling_Station_List_2008_1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a <b>DataFrame</b>. A dataframe is one of the basic Pandas data structures.\n",
    "A dataframe is similar to an Excel workbook – you have column names referring to columns and you have rows, which can be accessed with use of row numbers. Each row is an observation or measurment and each column is a (vector of) variable.\n",
    "<br>Those who are familiar with R know the data frame as a way to store data in rectangular (or 'tabular') grids that can easily be overviewed. Each row of these grids corresponds to measurements or values of an instance, while each column is (a vector containing) data for a specific variable.\n",
    "The essential difference in a dataframe is that the column names and row numbers are known as column _index_ and row _index_.\n",
    "<br> And a tricky thing to keep in mind is that indices start their numbering with 0 not 1. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Previewing the Data</b><br>The shape method gives you the dimensions of your DataFrame. So in this case it will tell us how many rows (20,928) and how many columns (6) are in our dataframe: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the .head method to look at the first few rows of all the columns: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that pandas automatically created a row index on the lefthand side (and again note that it started at 0):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember if you want to learn more about the methods available for your dataframe, then type the name followed by the \".\" and hit tab and it will show you a list like below: <br> \n",
    "<img src='images/tabhelp_screenshot.png' width=300 style=\"float: left;\")>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about a specific method, select it, type a \"?\" after it, hit return and the help should show up. Try it by running the below code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.astype?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"view\"></a>\n",
    "# Viewing the dataframe "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a list of all the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View a specific column of the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008['PS_Name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Let's look at the first 10 rows of all the columns:'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view  2 specific columns: The PS_Name and Registered_Voters "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008[['PS_Name', 'Registered_Voters']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also just get the first 10 rows of a specific column. Also note that it also tells us the type of data for that variable. In this case, we see that the data type is an integer (i.e., int64):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008['Registered_Voters'][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"filter\"></a>\n",
    "# Filtering the dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's view only those rows where the number of registered voters is greater than 750:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008[ps2008['Registered_Voters'] > 750 ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Viewing based on a row's specific value in one variable:</b>\n",
    "Now let's look in the column labeled PS_Code column and find the location of the value of W00770 and then show the it's values for the other columns: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.loc[ps2008['PS_Code'] == 'W00770'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter for those rows where the Region_Name is western or eastern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008[ps2008['Region_Name'].isin(['WESTERN', 'EASTERN'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a name=\"rename\"></a>\n",
    "# Renaming, adding, and dropping columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Rename a column</b><br>Let's say we want to take the column currently named 'Registered_Voters' and rename it to RV'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.rename(columns={'Registered_Voters': 'rv'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Define the new names of several columns at once \n",
    "newcols = {\n",
    "    'Region_Name': 'region', \n",
    "    'District_Name': 'district', \n",
    "}\n",
    "\n",
    "ps2008.rename(columns=newcols, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make add a new column where all values are 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ps2008['new_column'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ps2008['new_column']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new dataframe called \"smaller_ps2008\" but without a specific column: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "smaller_ps2008 = ps2008.drop('new_column', axis = 1)\n",
    "smaller_ps2008.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"pivot\"></a>\n",
    "# Pivot Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total (sum) number of registered voters for each Constituency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_const = ps2008.pivot_table('rv', columns='Constituency_Name', aggfunc='sum')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_const.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Average number of registered voters for each region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_rvregion = ps2008.pivot_table('rv', columns='region', aggfunc='mean')\n",
    "print mean_rvregion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total number of registered voters for each region:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum_rvregion = ps2008.pivot_table('rv', columns='region', aggfunc='sum')\n",
    "print sum_rvregion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's do the same calculations as above but set up the table so it looks nicer with some labels: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tablesumrv_region = pd.pivot_table(ps2008, index=[\"region\"],\n",
    "               values=[\"rv\"],\n",
    "               aggfunc=[np.sum],fill_value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tablesumrv_region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is the end of this introductory notebook. Next let's start summarizing a bigger dataset in the next notebook. "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:py27]",
   "language": "python",
   "name": "conda-env-py27-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}