Q_Learning_with_FrozenLakev2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Q* Learning with FrozenLakev2.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/simoninithomas/Deep_reinforcement_learning_Course/blob/master/Q_Learning_with_FrozenLakev2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "5aQKQMJTJBPH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Q* Learning with FrozenLake 4x4 \n",
        "\n",
        "In this Notebook, we'll implement an agent <b>that plays FrozenLake.</b>\n",
        "\n",
        "![alt text](http://simoninithomas.com/drlc/Qlearning/frozenlake4x4.png)\n",
        "\n",
        "The goal of this game is <b>to go from the starting state (S) to the goal state (G)</b> by walking only on frozen tiles (F) and avoid holes (H). However, the ice is slippery, **so you won't always move in the direction you intend (stochastic environment)**\n",
        "\n",
        "Thanks to [lukewys](https://github.com/lukewys) for his help"
      ]
    },
    {
      "metadata": {
        "id": "Xr9nI6dcQM8I",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# This is a notebook from [Deep Reinforcement Learning Course, new version](https://simoninithomas.github.io/Deep_reinforcement_learning_Course/)\n",
        "<img src=\"https://raw.githubusercontent.com/simoninithomas/Deep_reinforcement_learning_Course/master/docs/assets/img/DRLC%20Environments.png\" alt=\"Deep Reinforcement Course\"/>\n",
        "<br>\n",
        "<p>  Deep Reinforcement Learning Course is a free series of articles and videos tutorials 🆕 about Deep Reinforcement Learning, where **we'll learn the main algorithms (Q-learning, Deep Q Nets, Dueling Deep Q Nets, Policy Gradients, A2C, Proximal Policy Gradients, Prediction Based rewards agents…), and how to implement them with Tensorflow and PyTorch.**\n",
        "\n",
        "  ![alt text](http://simoninithomas.com/drlc/libraries.png)\n",
        "<br><br>\n",
        "    \n",
        "📜The articles explain the architectures from the big picture to the mathematical details behind them.\n",
        "<br>\n",
        "📹 The videos explain how to build the agents with Tensorflow </b></p>\n",
        "<br>\n",
        "This course will give you a **solid foundation for understanding and implementing the future state of the art algorithms**. And, you'll build a strong professional portfolio by creating **agents that learn to play awesome environments**: Doom© 👹, Space invaders 👾, Outrun, Sonic the Hedgehog©, Michael Jackson’s Moonwalker, agents that will be able to navigate in 3D environments with DeepMindLab (Quake) and able to walk with Mujoco. \n",
        "<br><br>\n",
        "</p> \n",
        "\n",
        "## 📚 The complete [Syllabus HERE](https://simoninithomas.github.io/Deep_reinforcement_learning_Course/)\n",
        "\n",
        "\n",
        "## Any questions 👨‍💻\n",
        "<p> If you have any questions, feel free to ask me: </p>\n",
        "<p> 📧: <a href=\"mailto:hello@simoninithomas.com\">hello@simoninithomas.com</a>  </p>\n",
        "<p> Github: https://github.com/simoninithomas/Deep_reinforcement_learning_Course </p>\n",
        "<p> 🌐 : https://simoninithomas.github.io/Deep_reinforcement_learning_Course/ </p>\n",
        "<p> Twitter: <a href=\"https://twitter.com/ThomasSimonini\">@ThomasSimonini</a> </p>\n",
        "<p> Don't forget to <b> follow me on <a href=\"https://twitter.com/ThomasSimonini\">twitter</a>, <a href=\"https://github.com/simoninithomas/Deep_reinforcement_learning_Course\">github</a> and <a href=\"https://medium.com/@thomassimonini\">Medium</a> to be alerted of the new articles that I publish </b></p>\n",
        "    \n",
        "## How to help  🙌\n",
        "3 ways:\n",
        "- **Clap our articles and like our videos a lot**:Clapping in Medium means that you really like our articles. And the more claps we have, the more our article is shared Liking our videos help them to be much more visible to the deep learning community.\n",
        "- **Share and speak about our articles and videos**: By sharing our articles and videos you help us to spread the word. \n",
        "- **Improve our notebooks**: if you found a bug or **a better implementation** you can send a pull request.\n",
        "<br>\n",
        "\n",
        "## Important note 🤔\n",
        "<b> You can run it on your computer but it's better to run it on GPU based services</b>, personally I use Microsoft Azure and their Deep Learning Virtual Machine (they offer 170$)\n",
        "https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning\n",
        "<br>\n",
        "⚠️ I don't have any business relations with them. I just loved their excellent customer service.\n",
        "\n",
        "If you have some troubles to use Microsoft Azure follow the explainations of this excellent article here (without last the part fast.ai): https://medium.com/@manikantayadunanda/setting-up-deeplearning-machine-and-fast-ai-on-azure-a22eb6bd6429"
      ]
    },
    {
      "metadata": {
        "id": "QK8fD0zAQkkg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Prerequisites 🏗️\n",
        "Before diving on the notebook **you need to understand**:\n",
        "- The foundations of Reinforcement learning (MC, TD, Rewards hypothesis...) [Article](https://medium.freecodecamp.org/an-introduction-to-reinforcement-learning-4339519de419)\n",
        "- Q-learning [Article](https://medium.freecodecamp.org/diving-deeper-into-reinforcement-learning-with-q-learning-c18d0db58efe)\n",
        "- In the [video version](https://www.youtube.com/watch?v=q2ZOEFAaaI0)  we implemented a Q-learning agent that learns to play OpenAI Taxi-v2 🚕 with Numpy."
      ]
    },
    {
      "metadata": {
        "id": "54AIvDov_7aa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step -1: Install the dependencies on Google Colab"
      ]
    },
    {
      "metadata": {
        "id": "gxxpHDIs_lvg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "outputId": "432b66bf-e5b9-4018-f77c-16d9c5ace440"
      },
      "cell_type": "code",
      "source": [
        "!pip install numpy\n",
        "!pip install gym"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (1.14.6)\n",
            "Collecting gym\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/22/4ff09745ade385ffe707fb5f053548f0f6a6e7d5e98a2b9d6c07f5b931a7/gym-0.10.9.tar.gz (1.5MB)\n",
            "\u001b[K    100% |████████████████████████████████| 1.5MB 6.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym) (1.1.0)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym) (1.14.6)\n",
            "Requirement already satisfied: requests>=2.0 in /usr/local/lib/python3.6/dist-packages (from gym) (2.18.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym) (1.11.0)\n",
            "Collecting pyglet>=1.2.0 (from gym)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1c/fc/dad5eaaab68f0c21e2f906a94ddb98175662cc5a654eee404d59554ce0fa/pyglet-1.3.2-py2.py3-none-any.whl (1.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 1.0MB 10.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (1.22)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (3.0.4)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2018.11.29)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet>=1.2.0->gym) (0.16.0)\n",
            "Building wheels for collected packages: gym\n",
            "  Running setup.py bdist_wheel for gym ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \bdone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/6c/3a/0e/b86dee98876bb56cdb482cc1f72201035e46d1baf69d10d028\n",
            "Successfully built gym\n",
            "Installing collected packages: pyglet, gym\n",
            "Successfully installed gym-0.10.9 pyglet-1.3.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "G9qH33L_QoBk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 0: Import the dependencies 📚\n",
        "We use 3 libraries:\n",
        "- `Numpy` for our Qtable\n",
        "- `OpenAI Gym` for our FrozenLake Environment\n",
        "- `Random` to generate random numbers"
      ]
    },
    {
      "metadata": {
        "id": "oU8zRXv8QHlm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import gym\n",
        "import random"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0fz-X3HTQueX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 1: Create the environment 🎮\n",
        "- Here we'll create the FrozenLake 8x8 environment. \n",
        "- OpenAI Gym is a library <b> composed of many environments that we can use to train our agents.</b>\n",
        "- In our case we choose to use Frozen Lake."
      ]
    },
    {
      "metadata": {
        "id": "mh9jBR_cQ5_a",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "outputId": "95abe453-2638-459e-c699-27ee3d53a1f7"
      },
      "cell_type": "code",
      "source": [
        "env = gym.make(\"FrozenLake-v0\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.\n",
            "  result = entry_point.load(False)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "JEtXMldxQ7uw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 2: Create the Q-table and initialize it 🗄️\n",
        "- Now, we'll create our Q-table, to know how much rows (states) and columns (actions) we need, we need to calculate the action_size and the state_size\n",
        "- OpenAI Gym provides us a way to do that: `env.action_space.n` and `env.observation_space.n`"
      ]
    },
    {
      "metadata": {
        "id": "Uc0xDVd_Q-C8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "action_size = env.action_space.n\n",
        "state_size = env.observation_space.n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "17XBPCecAsZG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        },
        "outputId": "a9f938da-3f83-4790-ba25-235f60e3da46"
      },
      "cell_type": "code",
      "source": [
        "# Create our Q table with state_size rows and action_size columns (64x4)\n",
        "qtable = np.zeros((state_size, action_size))\n",
        "print(qtable)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]\n",
            " [0. 0. 0. 0.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WEGeWKKsAu7X",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 3: Create the hyperparameters ⚙️\n",
        "- Here, we'll specify the hyperparameters"
      ]
    },
    {
      "metadata": {
        "id": "FJhPxx7UAunE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "total_episodes = 20000       # Total episodes\n",
        "learning_rate = 0.7          # Learning rate\n",
        "max_steps = 99               # Max steps per episode\n",
        "gamma = 0.95                 # Discounting rate\n",
        "\n",
        "# Exploration parameters\n",
        "epsilon = 1.0                 # Exploration rate\n",
        "max_epsilon = 1.0             # Exploration probability at start\n",
        "min_epsilon = 0.01            # Minimum exploration probability \n",
        "decay_rate = 0.005            # Exponential decay rate for exploration prob"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rcdaN_DbA3ES",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 4: The Q learning algorithm 🧠\n",
        "- Now we implement the Q learning algorithm:\n",
        "  ![alt text](http://simoninithomas.com/drlc/Qlearning//qtable_algo.png)\n"
      ]
    },
    {
      "metadata": {
        "id": "5Dqqo_8LA5De",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        },
        "outputId": "0e3fac86-e74c-462c-869d-5970f8c548a2"
      },
      "cell_type": "code",
      "source": [
        "# List of rewards\n",
        "rewards = []\n",
        "\n",
        "# 2 For life or until learning is stopped\n",
        "for episode in range(total_episodes):\n",
        "    # Reset the environment\n",
        "    state = env.reset()\n",
        "    step = 0\n",
        "    done = False\n",
        "    total_rewards = 0\n",
        "    \n",
        "    for step in range(max_steps):\n",
        "        # 3. Choose an action a in the current world state (s)\n",
        "        ## First we randomize a number\n",
        "        exp_exp_tradeoff = random.uniform(0, 1)\n",
        "        \n",
        "        ## If this number > greater than epsilon --> exploitation (taking the biggest Q value for this state)\n",
        "        if exp_exp_tradeoff > epsilon:\n",
        "            action = np.argmax(qtable[state,:])\n",
        "            #print(exp_exp_tradeoff, \"action\", action)\n",
        "\n",
        "        # Else doing a random choice --> exploration\n",
        "        else:\n",
        "            action = env.action_space.sample()\n",
        "            #print(\"action random\", action)\n",
        "            \n",
        "        \n",
        "        # Take the action (a) and observe the outcome state(s') and reward (r)\n",
        "        new_state, reward, done, info = env.step(action)\n",
        "\n",
        "        # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
        "        # qtable[new_state,:] : all the actions we can take from new state\n",
        "        qtable[state, action] = qtable[state, action] + learning_rate * (reward + gamma * np.max(qtable[new_state, :]) - qtable[state, action])\n",
        "        \n",
        "        total_rewards += reward\n",
        "        \n",
        "        # Our new state is state\n",
        "        state = new_state\n",
        "        \n",
        "        # If done (if we're dead) : finish episode\n",
        "        if done == True: \n",
        "            break\n",
        "        \n",
        "    # Reduce epsilon (because we need less and less exploration)\n",
        "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode) \n",
        "    rewards.append(total_rewards)\n",
        "    \n",
        "\n",
        "print (\"Score over time: \" +  str(sum(rewards)/total_episodes))\n",
        "print(qtable)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Score over time: 0.50795\n",
            "[[3.31076341e-01 9.62218123e-02 7.25300527e-02 8.93317624e-02]\n",
            " [3.09107962e-02 2.11238465e-02 1.53940941e-03 8.15916450e-02]\n",
            " [2.30580059e-02 1.10205893e-02 3.05104836e-02 5.81989088e-02]\n",
            " [1.89392072e-03 1.42514492e-03 3.61505851e-03 3.33071549e-02]\n",
            " [4.10216080e-01 7.15773401e-02 1.85316436e-02 8.37170734e-02]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [3.27832769e-04 4.25377235e-04 1.36010420e-01 5.37383451e-04]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [5.36724237e-02 1.49053850e-01 1.32325076e-01 5.76347785e-01]\n",
            " [1.16731702e-02 7.47882851e-01 5.26449538e-02 5.56077547e-02]\n",
            " [6.90443526e-01 7.89701771e-02 1.79099164e-02 7.15556071e-03]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [1.35647619e-01 3.20232106e-01 8.67775997e-01 5.98079233e-04]\n",
            " [4.08023861e-01 9.53201064e-01 2.40692710e-01 3.94354541e-01]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "R5czk9qTBQIU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Step 5: Use our Q-table to play FrozenLake ! 👾\n",
        "- After 10 000 episodes, our Q-table can be used as a \"cheatsheet\" to play FrozenLake\"\n",
        "- By running this cell you can see our agent playing FrozenLake."
      ]
    },
    {
      "metadata": {
        "id": "Bt8UsREaBNkJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 728
        },
        "outputId": "559f5868-411e-487c-b2b9-6c2f4dc7a48d"
      },
      "cell_type": "code",
      "source": [
        "env.reset()\n",
        "\n",
        "for episode in range(5):\n",
        "    state = env.reset()\n",
        "    step = 0\n",
        "    done = False\n",
        "    print(\"****************************************************\")\n",
        "    print(\"EPISODE \", episode)\n",
        "\n",
        "    for step in range(max_steps):\n",
        "        \n",
        "        # Take the action (index) that have the maximum expected future reward given that state\n",
        "        action = np.argmax(qtable[state,:])\n",
        "        \n",
        "        new_state, reward, done, info = env.step(action)\n",
        "        \n",
        "        if done:\n",
        "            # Here, we decide to only print the last state (to see if our agent is on the goal or fall into an hole)\n",
        "            env.render()\n",
        "            if new_state == 15:\n",
        "                print(\"We reached our Goal 🏆\")\n",
        "            else:\n",
        "                print(\"We fell into a hole ☠️\")\n",
        "            \n",
        "            # We print the number of step it took.\n",
        "            print(\"Number of steps\", step)\n",
        "            \n",
        "            break\n",
        "        state = new_state\n",
        "env.close()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "****************************************************\n",
            "EPISODE  0\n",
            "  (Down)\n",
            "SFFF\n",
            "FHFH\n",
            "FFFH\n",
            "HFF\u001b[41mG\u001b[0m\n",
            "We reached our Goal 🏆\n",
            "Number of steps 10\n",
            "****************************************************\n",
            "EPISODE  1\n",
            "****************************************************\n",
            "EPISODE  2\n",
            "  (Right)\n",
            "SFFF\n",
            "FHF\u001b[41mH\u001b[0m\n",
            "FFFH\n",
            "HFFG\n",
            "We fell into a hole ☠️\n",
            "Number of steps 20\n",
            "****************************************************\n",
            "EPISODE  3\n",
            "  (Down)\n",
            "SFFF\n",
            "FHFH\n",
            "FFFH\n",
            "HFF\u001b[41mG\u001b[0m\n",
            "We reached our Goal 🏆\n",
            "Number of steps 75\n",
            "****************************************************\n",
            "EPISODE  4\n",
            "  (Right)\n",
            "SFFF\n",
            "FHF\u001b[41mH\u001b[0m\n",
            "FFFH\n",
            "HFFG\n",
            "We fell into a hole ☠️\n",
            "Number of steps 15\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}