Day_10_Pandas_part_3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day_10_Pandas_part_3.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9U8B1dFPZzox"
      },
      "source": [
        "In machine learning you will most likely use databases from many sources to train your learning algorithms. Pandas allows us to load databases of different formats into DataFrames. One of the most popular data formats used to store databases is csv. CSV stands for Comma Separated Values and offers a simple format to store data. We can load CSV files into Pandas DataFrames using the ```pd.read_csv()``` function. Let's load Google stock data into a Pandas DataFrame. The GOOG.csv file contains Google stock data from 8/19/2004 till 10/13/2017 taken from Yahoo Finance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XStain6bVH4x",
        "outputId": "21ba7179-f0c0-4633-fd85-b6da8be9bd09",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# We load Google stock data in a DataFrame\n",
        "Google_stock = pd.read_csv('/content/sample_data/stocks.csv')\n",
        "\n",
        "# We print some information about Google_stock\n",
        "print('Google_stock is of type:', type(Google_stock))\n",
        "print('Google_stock has shape:', Google_stock.shape)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Google_stock is of type: <class 'pandas.core.frame.DataFrame'>\n",
            "Google_stock has shape: (3313, 7)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7wzCKjscdd1i"
      },
      "source": [
        "We see that we have loaded the stocks.csv file into a Pandas DataFrame and it consists of 3,313 rows and 7 columns. Now let's look at the stock data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zh8u2vdMce7D",
        "outputId": "647be954-8468-43fb-e3dd-d0defb7e16b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 614
        }
      },
      "source": [
        "Google_stock"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Open</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Close</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2004-08-19</td>\n",
              "      <td>49.676899</td>\n",
              "      <td>51.693783</td>\n",
              "      <td>47.669952</td>\n",
              "      <td>49.845802</td>\n",
              "      <td>49.845802</td>\n",
              "      <td>44994500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2004-08-20</td>\n",
              "      <td>50.178635</td>\n",
              "      <td>54.187561</td>\n",
              "      <td>49.925285</td>\n",
              "      <td>53.805050</td>\n",
              "      <td>53.805050</td>\n",
              "      <td>23005800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2004-08-23</td>\n",
              "      <td>55.017166</td>\n",
              "      <td>56.373344</td>\n",
              "      <td>54.172661</td>\n",
              "      <td>54.346527</td>\n",
              "      <td>54.346527</td>\n",
              "      <td>18393200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2004-08-24</td>\n",
              "      <td>55.260582</td>\n",
              "      <td>55.439419</td>\n",
              "      <td>51.450363</td>\n",
              "      <td>52.096165</td>\n",
              "      <td>52.096165</td>\n",
              "      <td>15361800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2004-08-25</td>\n",
              "      <td>52.140873</td>\n",
              "      <td>53.651051</td>\n",
              "      <td>51.604362</td>\n",
              "      <td>52.657513</td>\n",
              "      <td>52.657513</td>\n",
              "      <td>9257400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3308</th>\n",
              "      <td>2017-10-09</td>\n",
              "      <td>980.000000</td>\n",
              "      <td>985.424988</td>\n",
              "      <td>976.109985</td>\n",
              "      <td>977.000000</td>\n",
              "      <td>977.000000</td>\n",
              "      <td>891400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3309</th>\n",
              "      <td>2017-10-10</td>\n",
              "      <td>980.000000</td>\n",
              "      <td>981.570007</td>\n",
              "      <td>966.080017</td>\n",
              "      <td>972.599976</td>\n",
              "      <td>972.599976</td>\n",
              "      <td>968400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3310</th>\n",
              "      <td>2017-10-11</td>\n",
              "      <td>973.719971</td>\n",
              "      <td>990.710022</td>\n",
              "      <td>972.250000</td>\n",
              "      <td>989.250000</td>\n",
              "      <td>989.250000</td>\n",
              "      <td>1693300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3311</th>\n",
              "      <td>2017-10-12</td>\n",
              "      <td>987.450012</td>\n",
              "      <td>994.119995</td>\n",
              "      <td>985.000000</td>\n",
              "      <td>987.830017</td>\n",
              "      <td>987.830017</td>\n",
              "      <td>1262400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3312</th>\n",
              "      <td>2017-10-13</td>\n",
              "      <td>992.000000</td>\n",
              "      <td>997.210022</td>\n",
              "      <td>989.000000</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>1157700</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3313 rows × 7 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            Date        Open        High  ...       Close   Adj Close    Volume\n",
              "0     2004-08-19   49.676899   51.693783  ...   49.845802   49.845802  44994500\n",
              "1     2004-08-20   50.178635   54.187561  ...   53.805050   53.805050  23005800\n",
              "2     2004-08-23   55.017166   56.373344  ...   54.346527   54.346527  18393200\n",
              "3     2004-08-24   55.260582   55.439419  ...   52.096165   52.096165  15361800\n",
              "4     2004-08-25   52.140873   53.651051  ...   52.657513   52.657513   9257400\n",
              "...          ...         ...         ...  ...         ...         ...       ...\n",
              "3308  2017-10-09  980.000000  985.424988  ...  977.000000  977.000000    891400\n",
              "3309  2017-10-10  980.000000  981.570007  ...  972.599976  972.599976    968400\n",
              "3310  2017-10-11  973.719971  990.710022  ...  989.250000  989.250000   1693300\n",
              "3311  2017-10-12  987.450012  994.119995  ...  987.830017  987.830017   1262400\n",
              "3312  2017-10-13  992.000000  997.210022  ...  989.679993  989.679993   1157700\n",
              "\n",
              "[3313 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EB2qpQEzdt3W"
      },
      "source": [
        "We see that it is quite a large dataset and that Pandas has automatically assigned numerical row indices to the DataFrame. Pandas also used the labels that appear in the data in the CSV file to assign the column labels.\n",
        "\n",
        "When dealing with large datasets like this one, it is often useful just to take a look at the first few rows of data instead of the whole dataset. We can take a look at the first 5 rows of data using the ```.head()``` method, as shown below"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8imtUbLudi_U",
        "outputId": "6dc8e6df-2ccf-40d3-99ca-9c65ce6e9647",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "Google_stock.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Open</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Close</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2004-08-19</td>\n",
              "      <td>49.676899</td>\n",
              "      <td>51.693783</td>\n",
              "      <td>47.669952</td>\n",
              "      <td>49.845802</td>\n",
              "      <td>49.845802</td>\n",
              "      <td>44994500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2004-08-20</td>\n",
              "      <td>50.178635</td>\n",
              "      <td>54.187561</td>\n",
              "      <td>49.925285</td>\n",
              "      <td>53.805050</td>\n",
              "      <td>53.805050</td>\n",
              "      <td>23005800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2004-08-23</td>\n",
              "      <td>55.017166</td>\n",
              "      <td>56.373344</td>\n",
              "      <td>54.172661</td>\n",
              "      <td>54.346527</td>\n",
              "      <td>54.346527</td>\n",
              "      <td>18393200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2004-08-24</td>\n",
              "      <td>55.260582</td>\n",
              "      <td>55.439419</td>\n",
              "      <td>51.450363</td>\n",
              "      <td>52.096165</td>\n",
              "      <td>52.096165</td>\n",
              "      <td>15361800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2004-08-25</td>\n",
              "      <td>52.140873</td>\n",
              "      <td>53.651051</td>\n",
              "      <td>51.604362</td>\n",
              "      <td>52.657513</td>\n",
              "      <td>52.657513</td>\n",
              "      <td>9257400</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         Date       Open       High        Low      Close  Adj Close    Volume\n",
              "0  2004-08-19  49.676899  51.693783  47.669952  49.845802  49.845802  44994500\n",
              "1  2004-08-20  50.178635  54.187561  49.925285  53.805050  53.805050  23005800\n",
              "2  2004-08-23  55.017166  56.373344  54.172661  54.346527  54.346527  18393200\n",
              "3  2004-08-24  55.260582  55.439419  51.450363  52.096165  52.096165  15361800\n",
              "4  2004-08-25  52.140873  53.651051  51.604362  52.657513  52.657513   9257400"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sk6a7_Nrh_RV"
      },
      "source": [
        "We can also take a look at the last 5 rows of data by using the ```.tail()``` method:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "59nQSBchhHyR",
        "outputId": "07b884e6-6598-408a-92d4-8a2db61cc1ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 320
        }
      },
      "source": [
        "Google_stock.tail()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Open</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Close</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3308</th>\n",
              "      <td>2017-10-09</td>\n",
              "      <td>980.000000</td>\n",
              "      <td>985.424988</td>\n",
              "      <td>976.109985</td>\n",
              "      <td>977.000000</td>\n",
              "      <td>977.000000</td>\n",
              "      <td>891400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3309</th>\n",
              "      <td>2017-10-10</td>\n",
              "      <td>980.000000</td>\n",
              "      <td>981.570007</td>\n",
              "      <td>966.080017</td>\n",
              "      <td>972.599976</td>\n",
              "      <td>972.599976</td>\n",
              "      <td>968400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3310</th>\n",
              "      <td>2017-10-11</td>\n",
              "      <td>973.719971</td>\n",
              "      <td>990.710022</td>\n",
              "      <td>972.250000</td>\n",
              "      <td>989.250000</td>\n",
              "      <td>989.250000</td>\n",
              "      <td>1693300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3311</th>\n",
              "      <td>2017-10-12</td>\n",
              "      <td>987.450012</td>\n",
              "      <td>994.119995</td>\n",
              "      <td>985.000000</td>\n",
              "      <td>987.830017</td>\n",
              "      <td>987.830017</td>\n",
              "      <td>1262400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3312</th>\n",
              "      <td>2017-10-13</td>\n",
              "      <td>992.000000</td>\n",
              "      <td>997.210022</td>\n",
              "      <td>989.000000</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>1157700</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Date        Open        High  ...       Close   Adj Close   Volume\n",
              "3308  2017-10-09  980.000000  985.424988  ...  977.000000  977.000000   891400\n",
              "3309  2017-10-10  980.000000  981.570007  ...  972.599976  972.599976   968400\n",
              "3310  2017-10-11  973.719971  990.710022  ...  989.250000  989.250000  1693300\n",
              "3311  2017-10-12  987.450012  994.119995  ...  987.830017  987.830017  1262400\n",
              "3312  2017-10-13  992.000000  997.210022  ...  989.679993  989.679993  1157700\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZmCx_A_3iUVX"
      },
      "source": [
        "We can also optionally use ```.head(N)``` or ```.tail(N)``` to display the first and last N rows of data, respectively.\n",
        "\n",
        "Let's do a quick check to see whether we have any ```NaN``` values in our dataset. To do this, we will use the ```.isnull()``` method followed by the ```.any()``` method to check whether any of the columns contain ```NaN``` values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KTvuj2BIiMcA",
        "outputId": "c1f9453d-6434-429f-c0b3-7947d786bef8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "Google_stock.isnull().any()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Date         False\n",
              "Open         False\n",
              "High         False\n",
              "Low          False\n",
              "Close        False\n",
              "Adj Close    False\n",
              "Volume       False\n",
              "dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zH8nfXlLio3j"
      },
      "source": [
        "We see that we have no ```NaN``` values.\n",
        "\n",
        "When dealing with large datasets, it is often useful to get statistical information from them. Pandas provides the ```.describe()``` method to get descriptive statistics on each column of the DataFrame. Let's see how this works:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXJrX81Kimgc",
        "outputId": "d43ca237-d5f1-4ce0-eac8-d2e27b0d9337",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "source": [
        "# We get descriptive statistics on our stock data\n",
        "Google_stock.describe()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Open</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Close</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>3313.000000</td>\n",
              "      <td>3313.000000</td>\n",
              "      <td>3313.000000</td>\n",
              "      <td>3313.000000</td>\n",
              "      <td>3313.000000</td>\n",
              "      <td>3.313000e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>380.186092</td>\n",
              "      <td>383.493740</td>\n",
              "      <td>376.519309</td>\n",
              "      <td>380.072458</td>\n",
              "      <td>380.072458</td>\n",
              "      <td>8.038476e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>223.818650</td>\n",
              "      <td>224.974534</td>\n",
              "      <td>222.473232</td>\n",
              "      <td>223.853780</td>\n",
              "      <td>223.853780</td>\n",
              "      <td>8.399521e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>49.274517</td>\n",
              "      <td>50.541279</td>\n",
              "      <td>47.669952</td>\n",
              "      <td>49.681866</td>\n",
              "      <td>49.681866</td>\n",
              "      <td>7.900000e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>226.556473</td>\n",
              "      <td>228.394516</td>\n",
              "      <td>224.003082</td>\n",
              "      <td>226.407440</td>\n",
              "      <td>226.407440</td>\n",
              "      <td>2.584900e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>293.312286</td>\n",
              "      <td>295.433502</td>\n",
              "      <td>289.929291</td>\n",
              "      <td>293.029114</td>\n",
              "      <td>293.029114</td>\n",
              "      <td>5.281300e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>536.650024</td>\n",
              "      <td>540.000000</td>\n",
              "      <td>532.409973</td>\n",
              "      <td>536.690002</td>\n",
              "      <td>536.690002</td>\n",
              "      <td>1.065370e+07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>992.000000</td>\n",
              "      <td>997.210022</td>\n",
              "      <td>989.000000</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>989.679993</td>\n",
              "      <td>8.276810e+07</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Open         High  ...    Adj Close        Volume\n",
              "count  3313.000000  3313.000000  ...  3313.000000  3.313000e+03\n",
              "mean    380.186092   383.493740  ...   380.072458  8.038476e+06\n",
              "std     223.818650   224.974534  ...   223.853780  8.399521e+06\n",
              "min      49.274517    50.541279  ...    49.681866  7.900000e+03\n",
              "25%     226.556473   228.394516  ...   226.407440  2.584900e+06\n",
              "50%     293.312286   295.433502  ...   293.029114  5.281300e+06\n",
              "75%     536.650024   540.000000  ...   536.690002  1.065370e+07\n",
              "max     992.000000   997.210022  ...   989.679993  8.276810e+07\n",
              "\n",
              "[8 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ddXb_1Eoiwyj"
      },
      "source": [
        "If desired, we can apply the ```.describe()``` method on a single column as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XVc9LUDeiulN",
        "outputId": "28f26246-9e3f-4aac-a1cd-4f8f027cd547",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 186
        }
      },
      "source": [
        "# We get descriptive statistics on a single column of our DataFrame\n",
        "Google_stock['Adj Close'].describe()"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    3313.000000\n",
              "mean      380.072458\n",
              "std       223.853780\n",
              "min        49.681866\n",
              "25%       226.407440\n",
              "50%       293.029114\n",
              "75%       536.690002\n",
              "max       989.679993\n",
              "Name: Adj Close, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bq8yeKE6i44l"
      },
      "source": [
        "Similarly, you can also look at one statistic by using one of the many statistical functions Pandas provides. Let's look at some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XG5MbGkui1-r",
        "outputId": "9365a3bb-0d82-4d5b-aeeb-33854fa8729c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 385
        }
      },
      "source": [
        "# We print information about our DataFrame  \n",
        "print()\n",
        "print('Maximum values of each column:\\n', Google_stock.max())\n",
        "print()\n",
        "print('Minimum Close value:', Google_stock['Close'].min())\n",
        "print()\n",
        "print('Average value of each column:\\n', Google_stock.mean())"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Maximum values of each column:\n",
            " Date         2017-10-13\n",
            "Open                992\n",
            "High             997.21\n",
            "Low                 989\n",
            "Close            989.68\n",
            "Adj Close        989.68\n",
            "Volume         82768100\n",
            "dtype: object\n",
            "\n",
            "Minimum Close value: 49.681866\n",
            "\n",
            "Average value of each column:\n",
            " Open         3.801861e+02\n",
            "High         3.834937e+02\n",
            "Low          3.765193e+02\n",
            "Close        3.800725e+02\n",
            "Adj Close    3.800725e+02\n",
            "Volume       8.038476e+06\n",
            "dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMYy30hfjebQ"
      },
      "source": [
        "Another important statistical measure is data correlation. Data correlation can tell us, for example, if the data in different columns are correlated. We can use the .corr() method to get the correlation between different columns, as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZrVhnTUujbd-",
        "outputId": "479839ae-f718-450b-9c33-5f508d5ee40c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 252
        }
      },
      "source": [
        "# We display the correlation between columns\n",
        "Google_stock.corr()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Open</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Close</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Open</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.999904</td>\n",
              "      <td>0.999845</td>\n",
              "      <td>0.999745</td>\n",
              "      <td>0.999745</td>\n",
              "      <td>-0.564258</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>High</th>\n",
              "      <td>0.999904</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.999834</td>\n",
              "      <td>0.999868</td>\n",
              "      <td>0.999868</td>\n",
              "      <td>-0.562749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Low</th>\n",
              "      <td>0.999845</td>\n",
              "      <td>0.999834</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.999899</td>\n",
              "      <td>0.999899</td>\n",
              "      <td>-0.567007</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Close</th>\n",
              "      <td>0.999745</td>\n",
              "      <td>0.999868</td>\n",
              "      <td>0.999899</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.564967</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adj Close</th>\n",
              "      <td>0.999745</td>\n",
              "      <td>0.999868</td>\n",
              "      <td>0.999899</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.564967</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Volume</th>\n",
              "      <td>-0.564258</td>\n",
              "      <td>-0.562749</td>\n",
              "      <td>-0.567007</td>\n",
              "      <td>-0.564967</td>\n",
              "      <td>-0.564967</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Open      High       Low     Close  Adj Close    Volume\n",
              "Open       1.000000  0.999904  0.999845  0.999745   0.999745 -0.564258\n",
              "High       0.999904  1.000000  0.999834  0.999868   0.999868 -0.562749\n",
              "Low        0.999845  0.999834  1.000000  0.999899   0.999899 -0.567007\n",
              "Close      0.999745  0.999868  0.999899  1.000000   1.000000 -0.564967\n",
              "Adj Close  0.999745  0.999868  0.999899  1.000000   1.000000 -0.564967\n",
              "Volume    -0.564258 -0.562749 -0.567007 -0.564967  -0.564967  1.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xBzK1crrkfLt"
      },
      "source": [
        "A correlation value of 1 tells us there is a high correlation and a correlation of 0 tells us that the data is not correlated at all.\n",
        "\n",
        "We will end this Introduction to Pandas by taking a look at the .groupby() method. The .groupby() method allows us to group data in different ways. Let's see how we can group data to get different types of information. For the next examples we are going to load fake data about a fictitious company."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bbRBKoJFkc2n",
        "outputId": "d6b37673-2d87-4a0f-aa52-adf1d45f6981",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "# We load fake Company data in a DataFrame\n",
        "data = pd.read_csv('/fake_company.csv')\n",
        "\n",
        "data"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Year</th>\n",
              "      <th>Name</th>\n",
              "      <th>Department</th>\n",
              "      <th>Age</th>\n",
              "      <th>Salary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1990</td>\n",
              "      <td>Alice</td>\n",
              "      <td>HR</td>\n",
              "      <td>25</td>\n",
              "      <td>50000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1990</td>\n",
              "      <td>Bob</td>\n",
              "      <td>RD</td>\n",
              "      <td>30</td>\n",
              "      <td>48000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1990</td>\n",
              "      <td>Charlie</td>\n",
              "      <td>Admin</td>\n",
              "      <td>45</td>\n",
              "      <td>55000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1991</td>\n",
              "      <td>Alice</td>\n",
              "      <td>HR</td>\n",
              "      <td>26</td>\n",
              "      <td>52000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1991</td>\n",
              "      <td>Bob</td>\n",
              "      <td>RD</td>\n",
              "      <td>31</td>\n",
              "      <td>50000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1991</td>\n",
              "      <td>Charlie</td>\n",
              "      <td>Admin</td>\n",
              "      <td>46</td>\n",
              "      <td>60000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1992</td>\n",
              "      <td>Alice</td>\n",
              "      <td>Admin</td>\n",
              "      <td>27</td>\n",
              "      <td>60000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1992</td>\n",
              "      <td>Bob</td>\n",
              "      <td>RD</td>\n",
              "      <td>32</td>\n",
              "      <td>52000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1992</td>\n",
              "      <td>Charlie</td>\n",
              "      <td>Admin</td>\n",
              "      <td>28</td>\n",
              "      <td>62000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Year     Name Department  Age  Salary\n",
              "0  1990    Alice         HR   25   50000\n",
              "1  1990      Bob         RD   30   48000\n",
              "2  1990  Charlie      Admin   45   55000\n",
              "3  1991    Alice         HR   26   52000\n",
              "4  1991      Bob         RD   31   50000\n",
              "5  1991  Charlie      Admin   46   60000\n",
              "6  1992    Alice      Admin   27   60000\n",
              "7  1992      Bob         RD   32   52000\n",
              "8  1992  Charlie      Admin   28   62000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_0sbu0Qo00-"
      },
      "source": [
        "We see that the data contains information for the year 1990 through 1992. For each year we see name of the employees, the department they work for, their age, and their annual salary. Now, let's use the ```.groupby()``` method to get information.\n",
        "\n",
        "Let's calculate how much money the company spent in salaries each year. To do this, we will group the data by Year using the ```.groupby()``` method and then we will add up the salaries of all the employees by using the ```.sum()``` method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "504rMBEiki5h",
        "outputId": "57c31bdf-e611-423e-f6ce-3fc413b0e87d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# We display the total amount of money spent in salaries each year\n",
        "data.groupby(['Year'])['Salary'].sum()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Year\n",
              "1990    153000\n",
              "1991    162000\n",
              "1992    174000\n",
              "Name: Salary, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NuqQeRX2o9lJ"
      },
      "source": [
        "We see that the company spent a total of 153,000 dollars in 1990, 162,000 in 1991, and 174,000 in 1992.\n",
        "\n",
        "Now, let's suppose I want to know what was the average salary for each year. In this case, we will group the data by Year using the ```.groupby()``` method, just as we did before, and then we use the ```.mean()``` method to get the average salary. Let's see how this works"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "51eARHI4pEg8",
        "outputId": "04c287e7-7f40-482a-96e7-e8759ccf87e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        }
      },
      "source": [
        "# We display the average salary per year\n",
        "data.groupby(['Year'])['Salary'].mean()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Year\n",
              "1990    51000\n",
              "1991    54000\n",
              "1992    58000\n",
              "Name: Salary, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-NsUOKyPpH5X"
      },
      "source": [
        "We see that the average salary in 1990 was 51,000 dollars, 54,000 in 1991, and 58,000 in 1992.\n",
        "\n",
        "Now let's see how much did each employee get paid in those three years. In this case, we will group the data by Name using the ```.groupby()``` method and then we will add up the salaries for each year. Let's see the result"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Rh4BaZ2pMGr",
        "outputId": "e207694a-ed45-4d3e-bb4d-e6f69598e535",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        }
      },
      "source": [
        "# We display the salary distribution per department per year.\n",
        "data.groupby(['Year', 'Department'])['Salary'].sum()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Year  Department\n",
              "1990  Admin          55000\n",
              "      HR             50000\n",
              "      RD             48000\n",
              "1991  Admin          60000\n",
              "      HR             52000\n",
              "      RD             50000\n",
              "1992  Admin         122000\n",
              "      RD             52000\n",
              "Name: Salary, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pc6lM-xypU22"
      },
      "source": [
        "We see that in 1990 the Admin department paid 55,000 dollars in salaries,the HR department paid 50,000, and the RD department 48,0000. While in 1992 the Admin department paid 122,000 dollars in salaries and the RD department paid 52,000."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0XftCH5JsnU3"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}