Neural_Machine_Translation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/elmahygurl/Neural-Machine-Translation/blob/main/Neural_Machine_Translation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6czvz5VKO5M"
      },
      "source": [
        "# Notebook for Programming in Problem 4\n",
        "Welcome to the programming portion of the assignment! Each assignment throughout the semester will have a written portion and a programming portion. We will be using [Google Colab](https://colab.research.google.com/notebooks/intro.ipynb#recent=true), so if you have never used it before, take a quick look through this introduction: [Working with Google Colab](https://docs.google.com/document/d/1LlnXoOblXwW3YX-0yG_5seTXJsb3kRdMMRYqs8Qqum4/edit?usp=sharing)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5o8HI5JqTvU5"
      },
      "source": [
        "Learning Objectives\n",
        "In this problem, we will use [PyTorch](https://pytorch.org/) to implement a sequence-to-sequence (seq2seq) transformer model to build a nerual machine translation (NMT) system, which translates from French to English."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tnYMKJlKNXYe"
      },
      "source": [
        "## 1. Installing Packages and NMT data\n",
        "\n",
        "Install PyTorch using pip. See [https://pytorch.org/](https://pytorch.org/) if you want to install it on your computer.\n",
        "In addition, we will also be needing [huggingface](https://huggingface.co/)'s `transformers` and `datasets` libraries, and [nltk](https://www.nltk.org/) to compute the BLEU score."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-dRVuiP_JVdT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "09eb8495-e1f6-411c-9d23-b6fedda66f36",
        "collapsed": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting transformers==4.27.0\n",
            "  Downloading transformers-4.27.0-py3-none-any.whl (6.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (3.14.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (0.20.3)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (1.25.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (24.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (2023.12.25)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (2.31.0)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.27.0)\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.27.0) (4.66.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.27.0) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.27.0) (4.11.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.27.0) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.27.0) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.27.0) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.27.0) (2024.2.2)\n",
            "Installing collected packages: tokenizers, transformers\n",
            "  Attempting uninstall: tokenizers\n",
            "    Found existing installation: tokenizers 0.19.1\n",
            "    Uninstalling tokenizers-0.19.1:\n",
            "      Successfully uninstalled tokenizers-0.19.1\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.40.2\n",
            "    Uninstalling transformers-4.40.2:\n",
            "      Successfully uninstalled transformers-4.40.2\n",
            "Successfully installed tokenizers-0.13.3 transformers-4.27.0\n",
            "Collecting datasets==2.10.0\n",
            "  Downloading datasets-2.10.0-py3-none-any.whl (469 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m469.0/469.0 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (1.25.2)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (14.0.2)\n",
            "Collecting dill<0.3.7,>=0.3.0 (from datasets==2.10.0)\n",
            "  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m15.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (2.0.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (2.31.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (4.66.4)\n",
            "Collecting xxhash (from datasets==2.10.0)\n",
            "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets==2.10.0)\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m16.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (3.9.5)\n",
            "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (0.20.3)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (24.0)\n",
            "Collecting responses<0.19 (from datasets==2.10.0)\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets==2.10.0) (6.0.1)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets==2.10.0) (4.0.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets==2.10.0) (3.14.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets==2.10.0) (4.11.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets==2.10.0) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets==2.10.0) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets==2.10.0) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets==2.10.0) (2024.2.2)\n",
            "INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting multiprocess (from datasets==2.10.0)\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==2.10.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==2.10.0) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==2.10.0) (2024.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets==2.10.0) (1.16.0)\n",
            "Installing collected packages: xxhash, dill, responses, multiprocess, datasets\n",
            "Successfully installed datasets-2.10.0 dill-0.3.6 multiprocess-0.70.14 responses-0.18.0 xxhash-3.4.1\n"
          ]
        }
      ],
      "source": [
        "# Pytorch is typically already installed in Google Colab (uncomment to install):\n",
        "# !pip install torch==1.8.0\n",
        "# or for GPU support:\n",
        "# !pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "!pip install transformers==4.27.0\n",
        "!pip install datasets==2.10.0\n",
        "# NLTK is typically also already installed in Google Colab (uncomment to install):\n",
        "# !pip install nltk==3.8.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AL42TT6Tb0S7"
      },
      "source": [
        "We first download the data for NMT, which contains pairs of parallel sentences."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TmD7lPOXOlm2"
      },
      "outputs": [],
      "source": [
        "!wget --quiet https://princeton-nlp.github.io/cos484/assignments/a4/resources.zip\n",
        "!unzip -qo resources.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l5TS8PG0Os0f"
      },
      "source": [
        "## 2. Data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "In this section, we load and preprocess the dataset that will be used for training our neural machine translation (NMT) model. The dataset consists of parallel English and French sentences, which are partitioned into training, validation, and test splits.\n",
        "\n",
        "We use the `load_from_disk` function from the `datasets` library to load the pre-saved dataset from disk. The dataset is loaded into a `DatasetDict` object, which contains three splits:\n",
        "\n",
        "- **Train**: Contains 8,701 pairs of English and French sentences.\n",
        "- **Validation**: Contains 485 pairs of English and French sentences.\n",
        "- **Test**: Contains 486 pairs of English and French sentences.\n",
        "\n",
        "Each example in the dataset is a dictionary with two keys: `text_en` for the English sentence and `text_fr` for the French sentence.\n",
        "\n",
        "To get a better understanding of the data, we display a few examples from each split. This helps us verify the content and structure of the dataset before proceeding with further preprocessing steps.\n",
        "\n",
        "Below is the summary of the dataset splits and a few examples from each split.\n"
      ],
      "metadata": {
        "id": "psRStjCNYIRK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dxi3D2oX1iIK",
        "outputId": "534f8eb4-f4ae-472d-b2c0-f47b6cee4e1e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Summary of splits: DatasetDict({\n",
            "    train: Dataset({\n",
            "        features: ['text_en', 'text_fr'],\n",
            "        num_rows: 8701\n",
            "    })\n",
            "    validation: Dataset({\n",
            "        features: ['text_en', 'text_fr'],\n",
            "        num_rows: 485\n",
            "    })\n",
            "    test: Dataset({\n",
            "        features: ['text_en', 'text_fr'],\n",
            "        num_rows: 486\n",
            "    })\n",
            "})\n",
            "\n",
            "First 3 examples from the train split:\n",
            "Example 1: English: i m tough . - French: je suis dure .\n",
            "Example 2: English: i m not in a rush . - French: je ne suis pas aux pieces .\n",
            "Example 3: English: i m not sleeping well . - French: je ne dors pas bien .\n",
            "\n",
            "First 3 examples from the validation split:\n",
            "Example 1: English: she is a most gracious neighbor . - French: c est une voisine des plus courtoises .\n",
            "Example 2: English: i m not sitting here . - French: je ne vais pas m asseoir ici .\n",
            "Example 3: English: you re not wanted here . - French: on ne veut pas de toi ici .\n",
            "\n",
            "First 3 examples from the test split:\n",
            "Example 1: English: i m trustworthy . - French: on peut me faire confiance .\n",
            "Example 2: English: i m not miserable . - French: je ne suis pas malheureux .\n",
            "Example 3: English: i m going to take my car . - French: je vais prendre ma voiture .\n"
          ]
        }
      ],
      "source": [
        "from datasets import load_from_disk\n",
        "raw_text_datasets = load_from_disk(\"resources/parallel_en_fr_corpus\")\n",
        "\n",
        "# Print summary of splits\n",
        "print(\"Summary of splits:\", raw_text_datasets)\n",
        "\n",
        "# Display a few examples from each split\n",
        "for split in [\"train\", \"validation\", \"test\"]:\n",
        "    print(f\"\\nFirst 3 examples from the {split} split:\")\n",
        "    for i in range(3):\n",
        "        example = raw_text_datasets[split][i]\n",
        "        print(f\"Example {i+1}: English: {example['text_en']} - French: {example['text_fr']}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.1 Tokenisation"
      ],
      "metadata": {
        "id": "LSwdhUd2aQe7"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1NmS9j01iIK"
      },
      "source": [
        "You are also provided with two pre-trained tokenizers for the source and target languages respectively, which we can load with the hugginface transfomers library. [This tutorial](https://huggingface.co/docs/transformers/preprocessing#natural-language-processing) provides an introduction to using pre-trained tokenizers and the powerful `AutoTokenizer` class. The tokenizers are based on byte-pair encodings which break words into smaller units. This is aimed at reducing the sparsity of words, as subwords can be shared between different rare words. If you are interested in learning more, see the paper [Neural Machine Translation of Rare Words with Subword Units](https://www.aclweb.org/anthology/P16-1162.pdf)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### What is Tokenization?\n",
        "\n",
        "Tokenization is the process of converting raw text into smaller units called tokens. Tokens can be words, subwords, or even characters. This process is essential in natural language processing (NLP) as it transforms text data into a numerical format that machine learning models can process.\n",
        "\n",
        "#### Why is Tokenization Important?\n",
        "\n",
        "In the context of neural machine translation (NMT), tokenization is crucial for several reasons:\n",
        "\n",
        "1. **Handling Vocabulary Size**: Directly working with whole words would result in a very large vocabulary, making the model less efficient and harder to train. By breaking words into subwords, we can significantly reduce the vocabulary size.\n",
        "2. **Dealing with Rare Words**: Tokenizing into subwords allows the model to handle rare words more effectively. Even if a word is not seen during training, its subwords might be, enabling the model to understand and translate it.\n",
        "3. **Model Compatibility**: Machine learning models require numerical input. Tokenization converts text into numerical IDs that the model can process.\n",
        "\n",
        "#### Example of Tokenization\n",
        "\n",
        "Let's consider the English sentence: \"we have an example\".\n",
        "\n",
        "The tokenizer converts this sentence into the following tokens:\n",
        "- `<s>`: Start of sentence token.\n",
        "- `▁we`: The word \"we\" with a special character indicating the start of a new word.\n",
        "- `▁have`: The word \"have\" with the start of a new word indicator.\n",
        "- `▁an`: The word \"an\" with the start of a new word indicator.\n",
        "- `▁ex`: The beginning of the word \"example\".\n",
        "- `amp`: The middle part of the word \"example\".\n",
        "- `le`: The end part of the word \"example\".\n",
        "- `</s>`: End of sentence token.\n",
        "\n",
        "The output tokens are:\n",
        "```\n",
        "['<s>', '▁we', '▁have', '▁an', '▁ex', 'amp', 'le', '</s>']\n",
        "```\n",
        "\n",
        "These tokens are then converted into numerical IDs that the model can process:\n",
        "```\n",
        "{'input_ids': [1, 64, 324, 103, 266, 1490, 92, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}\n",
        "```\n",
        "\n",
        "Here:\n",
        "- `input_ids` are the numerical representations of the tokens.\n",
        "- `token_type_ids` are used to differentiate between different types of sequences (not relevant here as we have a single sentence).\n",
        "- `attention_mask` indicates which tokens should be attended to (1 for real tokens, 0 for padding).\n"
      ],
      "metadata": {
        "id": "e6I87JYKafgS"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Iqnyk7Xo1iIK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "34a60bef-50af-4f11-fcc1-c5e31aeda97c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vocab size of source language (French): 3200\n",
            "Vocab size of target language (English): 3200\n",
            "\n",
            "*** Example Sentence ***\n",
            "Example sentence: we have an example\n",
            "\n",
            "*** Tokenizer Output ***\n",
            "Token IDs: [1, 64, 324, 103, 266, 1490, 92, 2]\n",
            "Token Type IDs: [0, 0, 0, 0, 0, 0, 0, 0]\n",
            "Attention Mask: [1, 1, 1, 1, 1, 1, 1, 1]\n",
            "\n",
            "*** Tokens ***\n",
            "Tokens: ['<s>', '▁we', '▁have', '▁an', '▁ex', 'amp', 'le', '</s>']\n",
            "\n",
            "*** Reconstructed Sentence ***\n",
            "Reconstructed sentence: <s> we have an example</s>\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "# Load the pre-trained tokenizers for French and English\n",
        "source_tokenizer = AutoTokenizer.from_pretrained(\"resources/tokenizer_fr\")\n",
        "target_tokenizer = AutoTokenizer.from_pretrained(\"resources/tokenizer_en\")\n",
        "\n",
        "# Display the vocabulary sizes of both tokenizers\n",
        "print(\"Vocab size of source language (French):\", source_tokenizer.vocab_size)\n",
        "print(\"Vocab size of target language (English):\", target_tokenizer.vocab_size)\n",
        "\n",
        "# Define an example sentence in English\n",
        "example_sentence = \"we have an example\"\n",
        "\n",
        "# Tokenize the example sentence using the target (English) tokenizer\n",
        "tokenizer_output = target_tokenizer(example_sentence)\n",
        "\n",
        "# Display the original example sentence\n",
        "print(\"\\n*** Example Sentence ***\")\n",
        "print(\"Example sentence:\", example_sentence)\n",
        "\n",
        "# Display the tokenizer output in a more detailed manner\n",
        "print(\"\\n*** Tokenizer Output ***\")\n",
        "print(\"Token IDs:\", tokenizer_output[\"input_ids\"])\n",
        "print(\"Token Type IDs:\", tokenizer_output[\"token_type_ids\"])\n",
        "print(\"Attention Mask:\", tokenizer_output[\"attention_mask\"])\n",
        "\n",
        "# Convert the token IDs back to their corresponding tokens\n",
        "tokens = [target_tokenizer.decode(token) for token in tokenizer_output[\"input_ids\"]]\n",
        "print(\"\\n*** Tokens ***\")\n",
        "print(\"Tokens:\", tokens)\n",
        "\n",
        "# Reconstruct the sentence from the tokens, replacing the special character '▁' with a whitespace\n",
        "reconstructed_sentence = \"\".join(tokens).replace(\"▁\", \" \")\n",
        "print(\"\\n*** Reconstructed Sentence ***\")\n",
        "print(\"Reconstructed sentence:\", reconstructed_sentence)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.3 Mapping"
      ],
      "metadata": {
        "id": "1OfjscCaaXyJ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ui6m-oh1iIL"
      },
      "source": [
        "We now want to convert the entire dataset to token ids.\n",
        "Specifically, we want to use the tokenizers to create a dataset with\n",
        "features \"encoder_input_ids\" and \"decoder_input_ids\", which both have type `List[int]`\n",
        "and which will later be the inputs to our encoder-decoder model. We will implement this using the powerful `map` function. You can find its API reference [here](https://huggingface.co/docs/datasets/v2.10.0/en/package_reference/main_classes#datasets.Dataset.map)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XPb6HpAH1iIL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 399,
          "referenced_widgets": [
            "7dfe2f2ac30b4d9ab67986811c0ccbd0",
            "cb4c22c7368244e38b3b11c127eec3e0",
            "81179660118347ed89171284b2105581",
            "8bc0b8ec0ca74f4e81beb42b8faf98b3",
            "46aed8a90aca40ffb63395e7ce62974c",
            "ed1b19e6383d4d608c7b35ee12640886",
            "7ed640b661884f6990a7ede89b8d51d6",
            "d713374e9e734979af7f61e3a07f6e6c",
            "99490363e8684f24ba312ac1b9fb7d14",
            "60c18afc86c74e208cfca91c38a9816b",
            "a9451ba2b3b14cc9b30e9ff0ed74f447",
            "d455915de97449ad9b17e5b960ac1382",
            "98c6467e746d442a945ae2ee7be354bc",
            "00ac3e34416243968ac55bdc1ebdc344",
            "84e3a234d31944ee8e5b568ec4fcb2cf",
            "ec5937350f3748a1b2a77b6858a88863",
            "4e029bd4bf174fe88506998c053e9501",
            "1cf048da69c64c96b714e513b78c40f5",
            "15942d888f0f402f94f9806118b595e2",
            "810c669e4fa14d48874999466e81e6cd",
            "819aed5bdc384a43b7459a149e3421d6",
            "952dffd25f914cbeaf206f4c7c424d07",
            "2a2a11b3cd9c4db38fd8d8430cdfec68",
            "39290a9c0e0d419cb5c7afea75014281",
            "52d4211666ed48bd9c9939fb1750c178",
            "cf6cb81cdbf048559b83ac760e26dfc2",
            "56d3f69b0fc246f69ac6374e2d56b8ca",
            "06f8892c4a7c4b748eadcc258fedebf5",
            "c1f80019ca4d4108940adf6676688152",
            "2f3b31d8047848be8cbd383863aa9e60",
            "5aaa8ff26cc4480e95d49ca984c5566e",
            "f19415e01a3644c586240fe17253ca80",
            "fd3f172898e94e50976c2634a1c2ccc9"
          ]
        },
        "outputId": "a7b0b296-4a0e-4d6b-de85-c7eaba099070"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/8701 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7dfe2f2ac30b4d9ab67986811c0ccbd0"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Asking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.\n",
            "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
            "Asking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.\n",
            "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/485 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d455915de97449ad9b17e5b960ac1382"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/486 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2a2a11b3cd9c4db38fd8d8430cdfec68"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example 0:\n",
            "Encoder input IDs: [1, 47, 60, 1449, 36, 2]\n",
            "Decoder input IDs: [1, 37, 42, 1044, 35, 2]\n",
            "Source text (decoded): ▁je ▁suis ▁dure ▁.\n",
            "Target text (decoded): ▁i ▁m ▁tough ▁.\n",
            "\n",
            "Example 1:\n",
            "Encoder input IDs: [1, 47, 86, 60, 68, 531, 1037, 385, 36, 2]\n",
            "Decoder input IDs: [1, 37, 42, 60, 89, 39, 259, 2914, 35, 2]\n",
            "Source text (decoded): ▁je ▁ne ▁suis ▁pas ▁aux ▁pie ces ▁.\n",
            "Target text (decoded): ▁i ▁m ▁not ▁in ▁a ▁r ush ▁.\n",
            "\n",
            "Example 2:\n",
            "Encoder input IDs: [1, 47, 86, 48, 883, 68, 239, 36, 2]\n",
            "Decoder input IDs: [1, 37, 42, 60, 1678, 501, 35, 2]\n",
            "Source text (decoded): ▁je ▁ne ▁d ors ▁pas ▁bien ▁.\n",
            "Target text (decoded): ▁i ▁m ▁not ▁sleeping ▁well ▁.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from typing import Dict, List\n",
        "\n",
        "def map_example(example: Dict[str, str]) -> Dict[str, List[int]]:\n",
        "    # TODO: Tokenize the source and target text for an entry in the parallel dataset\n",
        "    # and return a dictionary with the keys \"encoder_input_ids\" and \"decoder_input_ids\".\n",
        "    # You can use `source_tokenizer` and `target_tokenizer`\n",
        "    # Print example to understand its structure\n",
        "    #print(example)\n",
        "\n",
        "    # Assuming the keys are 'text_en' for English and 'text_fr' for French\n",
        "    source_text = example[\"text_fr\"]\n",
        "    target_text = example[\"text_en\"]\n",
        "\n",
        "    # Tokenize the source text (French)\n",
        "    source_encoded = source_tokenizer(source_text, truncation=True, padding='max_length') #trial:, max_length=128\n",
        "    # Tokenize the target text (English)\n",
        "    target_encoded = target_tokenizer(target_text, truncation=True, padding='max_length')\n",
        "\n",
        "    return {\n",
        "        \"encoder_input_ids\": source_encoded['input_ids'],\n",
        "        \"decoder_input_ids\": target_encoded['input_ids']\n",
        "    }\n",
        "\n",
        "# When mapped is applied to the DatasetDict object, it will apply `map` separately to each split.\n",
        "tokenized_datasets = raw_text_datasets.map(map_example, batched=False)\n",
        "\n",
        "# The `remove_columns` removes the existing text features from the new dataset, as they are no longer needed.\n",
        "tokenized_datasets = tokenized_datasets.remove_columns(raw_text_datasets.column_names[\"train\"])\n",
        "\n",
        "# Sanity checks on the new dataset\n",
        "assert set(tokenized_datasets.column_names[\"train\"]) == {\"decoder_input_ids\", \"encoder_input_ids\"}\n",
        "assert len(tokenized_datasets[\"train\"]) == len(raw_text_datasets[\"train\"])\n",
        "# Print a few examples to visualize and do a sanity check\n",
        "for i in range(3):\n",
        "    example = tokenized_datasets[\"train\"][i]\n",
        "    print(f\"Example {i}:\")\n",
        "    print(f\"Encoder input IDs: {example['encoder_input_ids']}\")\n",
        "    print(f\"Decoder input IDs: {example['decoder_input_ids']}\")\n",
        "    print(\"Source text (decoded):\", source_tokenizer.decode(example['encoder_input_ids'], skip_special_tokens=True))\n",
        "    print(\"Target text (decoded):\", target_tokenizer.decode(example['decoder_input_ids'], skip_special_tokens=True))\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " #Example dataset as before\n",
        "example_dataset = [\n",
        "    {\"text_en\": \"we have an example\", \"text_fr\": \"nous avons un exemple\"}\n",
        "]\n",
        "mapped_example = map_example(example_dataset[0])\n",
        "\n",
        "print(\"Mapped example:\", mapped_example)"
      ],
      "metadata": {
        "id": "gNROclsDBq_c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32849899-f274-4772-d52a-dbd71954e382"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mapped example: {'encoder_input_ids': [1, 79, 349, 82, 298, 13, 130, 52, 2], 'decoder_input_ids': [1, 64, 324, 103, 266, 1490, 92, 2]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3qPsuS71iIL"
      },
      "source": [
        "## Transformer model for NMT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R90j2WL61iIM"
      },
      "source": [
        "We will now implement a encoder-decoder transformer model.\n",
        "We already provide code for the Feedforward Layers and Transformer Blocks, but you will have to implement the MultiHeadAttention and Embedding layer from scratch, as well as registering all the layers in the final EncoderDecoderModel. Pay attention to doc-strings and typing information to understand the context and purpose of each missing code block!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nVjxNtLz1iIM"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "import math\n",
        "from typing import Optional, Dict, List, Tuple"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Multihead Attention is basically a dot product and scaling\n"
      ],
      "metadata": {
        "id": "HEGRU4nTMJN1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Implementing a Multi-Head Attention Layer for NMT\n",
        "\n",
        "In this section, we will implement a multi-head attention (MHA) layer, which is a critical component of the transformer model used in neural machine translation (NMT). The MHA layer allows the model to focus on different parts of the input sequence for each position in the output sequence. Here's a detailed explanation of the implementation steps:\n",
        "\n",
        "#### Multi-Head Attention Class\n",
        "\n",
        "1. **Initialization:**\n",
        "   - The `MultiHeadAttention` class initializes the necessary components for the attention mechanism. It takes the `hidden_size`, `num_attention_heads`, and flags for causal and cross-attention as inputs.\n",
        "   - The hidden size is divided by the number of attention heads to get the dimension of each head.\n",
        "   - Query, key, and value projections are defined using linear layers. These projections transform the input vectors into query, key, and value vectors used for computing attention.\n",
        "   - An output projection is also defined to transform the concatenated attention outputs back to the original hidden size.\n",
        "\n",
        "2. **Causal Attention Mask:**\n",
        "   - The `causal_attention_mask` method creates a mask that prevents attending to future tokens in the sequence. This is important for autoregressive models where the prediction for each token should not depend on future tokens.\n",
        "   - The mask is a triangular matrix with large negative values above the diagonal, ensuring that positions can only attend to themselves and previous positions.\n",
        "\n",
        "3. **Forward Method:**\n",
        "   - **Projections:**\n",
        "     - If `is_cross_attention` is true, keys and values are derived from the encoder outputs, while queries come from the decoder inputs.\n",
        "     - For self-attention, all projections are derived from the hidden states of the current layer.\n",
        "   - **Reshaping:**\n",
        "     - The query, key, and value tensors are reshaped to separate the different attention heads.\n",
        "   - **Scaled Dot-Product Attention:**\n",
        "     - Attention scores are computed by taking the dot product of query and key vectors, scaled by the square root of the head dimension.\n",
        "   - **Masking:**\n",
        "     - The key padding mask is applied to ignore padding tokens in the input.\n",
        "     - If causal attention is enabled, the causal mask is added to prevent attending to future tokens.\n",
        "   - **Softmax:**\n",
        "     - The attention scores are normalized using the softmax function to get the attention weights.\n",
        "   - **Attention Output:**\n",
        "     - The attention weights are used to compute a weighted sum of the value vectors.\n",
        "   - **Combining Heads:**\n",
        "     - The outputs from all heads are concatenated and projected back to the original hidden size using the output projection layer.\n",
        "   - The method returns the final attention output and the attention weights.\n",
        "\n",
        "#### Example Output\n",
        "\n",
        "Consider an example with an input batch containing the sentence \"we have an example\" (English) and \"nous avons un exemple\" (French). After tokenization, the attention layer processes these inputs:\n",
        "\n",
        "- **Encoder Input IDs (`encoder_input_ids`):** [1, 79, 349, 82, 298, 13, 130, 52, 2]\n",
        "  - These IDs represent the French sentence after tokenization, including special start (`<s>`) and end (`</s>`) tokens.\n",
        "- **Decoder Input IDs (`decoder_input_ids`):** [1, 64, 324, 103, 266, 1490, 92, 2]\n",
        "  - These IDs represent the English sentence after tokenization, with the same special tokens.\n",
        "\n",
        "The attention mechanism will use these IDs to compute attention scores and aggregate the information from different parts of the input sequences. This allows the model to focus on relevant parts of the input when generating translations, making it more effective in capturing dependencies and context.\n",
        "\n"
      ],
      "metadata": {
        "id": "mciwCZ00fBBO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7pMQHadt1iIM"
      },
      "outputs": [],
      "source": [
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self,\n",
        "                 hidden_size: int,\n",
        "                 num_attention_heads: int,\n",
        "                 is_causal_attention: bool = False,\n",
        "                 is_cross_attention: bool = False):\n",
        "        \"\"\"Defines a flexible multi-head attention layer.\n",
        "\n",
        "        This layer should define parameters for the query, key and value projections, as well as the output projection,\n",
        "        and implement the following steps:\n",
        "        (1) Project the input vectors using query projection and key projection matrices.\n",
        "        (2) Compute the head-wise attention scores scaled by 1/sqrt(head_dim)\n",
        "        (3) Perform appropriate masking to the attention scores using key_padding_mask and optionally causal attention.\n",
        "        (4) Normalize the head-wise attention scores using softmax.\n",
        "        (5) Compute the value projections and then aggregate using the normalized attention scores.\n",
        "        (6) Use the output projection to obtain the final output vectors.\n",
        "        When is_cross_attention is True, the key and value projections are computed from the encoder outputs.\n",
        "        Note that we do not use attention weight dropout in this implementation.\n",
        "\n",
        "        Args:\n",
        "            hidden_size: The dimensionality of the input vectors.\n",
        "            num_attention_heads: The number of attention heads.\n",
        "            is_causal_attention: Whether to use causal masking,\n",
        "                    where tokens cannot attend to the future tokens on their right.\n",
        "            is_cross_attention: Whether to use cross attention,\n",
        "                    where we use different inputs for the key/value vs. query vectors.\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "        assert hidden_size % num_attention_heads == 0, \"The hidden size must be divisible by the number of attention heads.\"\n",
        "        self.head_dim = hidden_size // num_attention_heads  # embedding dimension of query and key vectors per head\n",
        "\n",
        "        # TODO Initialize the module and its parameters here.\n",
        "        # This module should be able to handle both full self-attention, causal masked self-attention and cross-attention.\n",
        "        # IMPORTANT: You are not allowed to use `nn.MultiheadAttention` or `nn.functional.scaled_dot_product_attention`!\n",
        "        self.head_dim = hidden_size // num_attention_heads  # embedding dimension of query and key vectors per head\n",
        "        self.num_attention_heads = num_attention_heads\n",
        "        self.is_causal_attention = is_causal_attention\n",
        "        self.is_cross_attention = is_cross_attention\n",
        "\n",
        "        # Query, Key, and Value projections\n",
        "        self.query_proj = nn.Linear(hidden_size, hidden_size)\n",
        "        self.key_proj = nn.Linear(hidden_size, hidden_size)\n",
        "        self.value_proj = nn.Linear(hidden_size, hidden_size)\n",
        "\n",
        "        # Output projection\n",
        "        self.out_proj = nn.Linear(hidden_size, hidden_size)\n",
        "\n",
        "    def causal_attention_mask(self,\n",
        "                              sequence_length: int,\n",
        "                              device: Optional[torch.device] = None) -> torch.FloatTensor:\n",
        "        \"\"\"Return a Float tensor that can be added to the (un-normalized) attention scores for causal masking.\n",
        "\n",
        "        Args:\n",
        "            sequence_length: width and height of the attention mask tensor.\n",
        "            device: which torch device the resulting tensor should be on (important if you use GPU).\n",
        "\n",
        "        Returns:\n",
        "            A Float tensor of shape (1, 1, sequence_length, sequence_length) on device `device`,\n",
        "            where the entries above the diagonal contain large negative values,\n",
        "            which means that a query at position i can't attend to a key at position j>i.\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement the forward function.\n",
        "        # IMPORTANT: For full credit, you should not use python loops.\n",
        "        #\n",
        "        # Hint 1: You can pick an arbitrary large value (e.g., -10^{6}), but note that\n",
        "        #         using `float(\"-inf\")` might lead to numerical issues and 'nan' values during training.\n",
        "        #\n",
        "        # Hint 2: Useful pytorch functions for this are `torch.arange` or `torch.triu`.\n",
        "        #\n",
        "        # Hint 3: You can move the tensor you create to a device by calling `tensor.to(device)`\n",
        "        #\n",
        "        # You should use this function in `forward` and use the returned tensor to implement causal masking\n",
        "        # by adding it to the un-normalized attention scores of shape (batch_size, num_heads, sequence_length, sequence_length),\n",
        "        # as torch will handle broadcasting and expand the first two dimensions to batch size and num_heads.\n",
        "        #\n",
        "        # You will the masking tensor to be on the same device as the attention scores's device,\n",
        "        # which you can via the attribute `tensor.device`.\n",
        "\n",
        "        mask = torch.triu(torch.ones((sequence_length, sequence_length), device=device) * float('-inf'), diagonal=1)\n",
        "\n",
        "        return mask.unsqueeze(0).unsqueeze(0)  # (1, 1, sequence_length, sequence_length)\n",
        "\n",
        "\n",
        "    def forward(self,\n",
        "                hidden_states: torch.FloatTensor,\n",
        "                key_padding_mask: torch.BoolTensor,\n",
        "                encoder_outputs: Optional[torch.FloatTensor] = None) -> Tuple[torch.FloatTensor, torch.FloatTensor]:\n",
        "        \"\"\"Computes scaled dot-product attention and returns the output of the attention layer.\n",
        "\n",
        "        Args:\n",
        "            hidden_states: Tensor of shape (batch_size, sequence_length, hidden_size) - the input vectors to the layer.\n",
        "            key_padding_mask: Tensor of shape (batch_size, sequence_length) indicating which tokens are padding tokens.\n",
        "                    A `True` entry means that this token should be ignored for the purpose of attention.\n",
        "                    In the case of cross-attention, the tensor has shape (batch_size, encoder_sequence_length).\n",
        "            encoder_outputs: Optional tensor of shape (batch_size, encoder_sequence_length, hidden_size).\n",
        "                    The output vectors of the encoder and only passed if the layer performs cross-attention.\n",
        "\n",
        "        Returns:\n",
        "            A (layer_output, attention_weights) where layer_output is a tensor of shape (batch_size, sequence_length, hidden_size)\n",
        "            and attention_weights are the normalized attention scores in the form of\n",
        "            a tensor of shape (batch_size, num_attention_heads, number_of_query_tokens, number_of_key_tokens).\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement the forward function.\n",
        "        # IMPORTANT: For full credit, you should not use python loops. Furthermore,\n",
        "        #            you are not allowed to use `nn.MultiheadAttention` or `nn.functional.scaled_dot_product_attention`!\n",
        "        #\n",
        "        # Hint 1: Use `torch.reshape` to add a new axis for the attention head,\n",
        "        #         which will allow you to process all attention heads in parallel.\n",
        "        #\n",
        "        # Hint 2: You can use `torch.transpose` to swap the order of two axes,\n",
        "        #         As the attention head dimension should be next to the batch size,\n",
        "        #         see the shape of the output attention weights.\n",
        "        #\n",
        "        # Hint 3: `torch.bmm(matrix1, matrix2)` is useful for computing batched matrix multiplications\n",
        "        #         If matrix1 has shape (B, M, N) and matrix2 has shape (B, N, P),\n",
        "        #         it performs `B` matrix multiplications and outputs a tensor of shape (B, M, P).\n",
        "        #         Alternatively, `torch.einsum` should be very useful.\n",
        "        #         (We really encourage you to check out the documentation of `torch.einsum`,\n",
        "        #         it can really make your life easier here.)\n",
        "        batch_size, sequence_length, hidden_size = hidden_states.size()\n",
        "\n",
        "        if self.is_cross_attention:\n",
        "            # Cross-attention: keys and values come from the encoder outputs\n",
        "            query = self.query_proj(hidden_states)\n",
        "            key = self.key_proj(encoder_outputs)\n",
        "            value = self.value_proj(encoder_outputs)\n",
        "            key_padding_mask = key_padding_mask[:, :encoder_outputs.size(1)]\n",
        "        else:\n",
        "            # Self-attention: keys and values come from the hidden states\n",
        "            query = self.query_proj(hidden_states)\n",
        "            key = self.key_proj(hidden_states)\n",
        "            value = self.value_proj(hidden_states)\n",
        "\n",
        "        # Reshape to separate heads\n",
        "        query = query.view(batch_size, sequence_length, self.num_attention_heads, self.head_dim).transpose(1, 2)\n",
        "        key = key.view(batch_size, -1, self.num_attention_heads, self.head_dim).transpose(1, 2)\n",
        "        value = value.view(batch_size, -1, self.num_attention_heads, self.head_dim).transpose(1, 2)\n",
        "\n",
        "        # Scaled dot-product attention\n",
        "        attn_scores = torch.einsum(\"bhqd, bhkd -> bhqk\", query, key) / (self.head_dim ** 0.5)\n",
        "\n",
        "        # Apply key padding mask\n",
        "        attn_scores = attn_scores.masked_fill(key_padding_mask.unsqueeze(1).unsqueeze(2), float('-inf'))\n",
        "\n",
        "        # Apply causal mask if needed\n",
        "        if self.is_causal_attention:\n",
        "            causal_mask = self.causal_attention_mask(sequence_length, device=hidden_states.device)\n",
        "            attn_scores = attn_scores + causal_mask\n",
        "\n",
        "        # Softmax to get attention weights\n",
        "        attn_weights = torch.softmax(attn_scores, dim=-1)\n",
        "\n",
        "        # Attention output\n",
        "        attn_output = torch.einsum(\"bhqk, bhkd -> bhqd\", attn_weights, value)\n",
        "\n",
        "        # Combine heads\n",
        "        attn_output = attn_output.transpose(1, 2).contiguous().view(batch_size, sequence_length, hidden_size)\n",
        "\n",
        "        # Final linear projection\n",
        "        output = self.out_proj(attn_output)\n",
        "\n",
        "        return output, attn_weights\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNhiLcSw1iIM"
      },
      "source": [
        "Before we move on to the other modules, you should implement a sanity check for your attention implementation:\n",
        "1. We check the dimensions of the output of the layer and\n",
        "2. We plot the attention weights to some toy embedding inputs.\n",
        "We assume that the last token in the encoder and the last two tokens in the decoder are pad tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "epu_wgGi1iIM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "597b7b6b-8209-4edb-bb95-71d2ee55d1da"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAogAAAIjCAYAAACAkuqXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXVklEQVR4nO3de1yUZf7/8feAMoAIngUUBQ+pqGiJuh5KSxLtpB3N3NWw7LceSpdt27WDeGjDToalaUetvpmWle1uLWas6Jp4zuygpoZ5BDUVBBOUuX5/mLPegRNjA4PM6/l43I91rrnmuj/3zPrw0+e6r+u2GWOMAAAAgJ/5eTsAAAAAVC0kiAAAALAgQQQAAIAFCSIAAAAsSBABAABgQYIIAAAACxJEAAAAWJAgAgAAwIIEEQAAABYkiAAuKdHR0br77ru9HYYkaffu3bLZbHrmmWe8HYpTZmambDabMjMzvR0KgEsYCSIApwMHDmjy5MnavHlzqfcWLFigtLS0Solj9erVmjx5so4fP14p5/NFBQUFSklJ0YABA1SvXj3ZbDbNnz/f22EBqCJIEAE4HThwQFOmTKkSCeKUKVPKTBC3b9+uV155pVLiqM6OHDmiqVOnauvWrerUqZO3wwFQxdTwdgBAdVVYWKhatWp5O4xqx263ezuEaiEiIkIHDx5UeHi4NmzYoK5du3o7JABVCBVEoBz279+ve+65R5GRkbLb7YqJidHo0aNVXFwsSZo/f75sNptWrFihMWPGqFGjRmratKnz8y+++KLat28vu92uyMhIjR07tlR1bMeOHbr11lsVHh6uwMBANW3aVHfeeafy8vKcfZYtW6bevXurTp06CgkJUZs2bfTwww//avxHjx7Vgw8+qI4dOyokJEShoaEaOHCgvvzyS2efzMxMZ5KQlJQkm83mnHbs27evPv74Y/3www/O9ujoaOdni4qKlJKSolatWslutysqKkoPPfSQioqKLHHYbDaNGzdOS5YsUYcOHWS329W+fXulp6c7+0yePFl/+ctfJEkxMTHO8+3evVtS2fcgfv/997r99ttVr149BQcH63e/+50+/vhjS59z9+a9++67+vvf/66mTZsqMDBQ/fr1086dO3/1O/w1L7/8slq2bCm73a6uXbtq/fr1pfps27ZNt912m+rVq6fAwEDFx8frH//4h6VPeX6rc/bt26fBgwerVq1aatSokf70pz+V+s4vxG63Kzw8/OIuFkC1RwUR+BUHDhxQt27ddPz4cd13331q27at9u/fr8WLF+vkyZMKCAhw9h0zZowaNmyoSZMmqbCwUNLZhGfKlClKSEjQ6NGjtX37ds2ZM0fr16/X559/rpo1a6q4uFiJiYkqKirS/fffr/DwcO3fv1//+te/dPz4cYWFhembb77RDTfcoLi4OE2dOlV2u107d+7U559//qvX8P3332vJkiW6/fbbFRMTo9zcXL300kvq06ePvv32W0VGRqpdu3aaOnWqJk2apPvuu09XXnmlJKlnz55q0qSJ8vLytG/fPj333HOSpJCQEEmSw+HQTTfdpFWrVum+++5Tu3bt9NVXX+m5557Td999pyVLllhiWbVqlT744AONGTNGtWvX1vPPP69bb71Ve/bsUf369XXLLbfou+++0zvvvKPnnntODRo0kCQ1bNiwzGvLzc1Vz549dfLkST3wwAOqX7++3njjDd10001avHixbr75Zkv/6dOny8/PTw8++KDy8vL01FNPadiwYVq7dm05/t9QtgULFujEiRP6f//v/8lms+mpp57SLbfcou+//141a9aUJH3zzTfq1auXmjRpor/97W+qVauW3n33XQ0ePFjvv/++M87y/FaS9NNPP6lfv37as2ePHnjgAUVGRuqtt97Sf/7zn4u+DgBwMgBcGj58uPHz8zPr168v9Z7D4TDGGDNv3jwjyfTu3ducOXPG+f6hQ4dMQECA6d+/vykpKXG2z5o1y0gyr7/+ujHGmC+++MJIMu+9994F43juueeMJHP48GG3r+HUqVOW8xtjTHZ2trHb7Wbq1KnOtvXr1xtJZt68eaXGuP76603z5s1Ltb/11lvGz8/P/Pe//7W0z50710gyn3/+ubNNkgkICDA7d+50tn355ZdGknnhhRecbU8//bSRZLKzs0udr3nz5mbEiBHO1xMmTDCSLOc/ceKEiYmJMdHR0c7rXr58uZFk2rVrZ4qKipx9Z86caSSZr776qtS5fk12draRZOrXr2+OHj3qbP/oo4+MJPPPf/7T2davXz/TsWNHc+rUKWebw+EwPXv2NK1bt3a2lfe3SktLM5LMu+++62wrLCw0rVq1MpLM8uXLy30drn53AL6JKWbABYfDoSVLlujGG29UfHx8qfdtNpvl9ahRo+Tv7+98/dlnn6m4uFgTJkyQn5+fpV9oaKhzGjQsLEyStHTpUp08ebLMWOrUqSNJ+uijj+RwONy6Drvd7jx/SUmJfvzxR+cU9aZNm9wa65fee+89tWvXTm3bttWRI0ecxzXXXCNJWr58uaV/QkKCWrZs6XwdFxen0NBQff/99xd1/k8++UTdunVT7969nW0hISG67777tHv3bn377beW/klJSZaq77lK6cWeX5KGDBmiunXrXnDMo0eP6j//+Y/uuOMOnThxwvkd/fjjj0pMTNSOHTu0f/9+SeX/rT755BNFRETotttuc7YFBwfrvvvuu+jrAIBzSBABFw4fPqz8/Hx16NChXP1jYmIsr3/44QdJUps2bSztAQEBatGihfP9mJgYJScn69VXX1WDBg2UmJio2bNnW+4/HDJkiHr16qV7771XjRs31p133ql3333Xkizm5ORYjp9++knS2UT3ueeeU+vWrWW329WgQQM1bNhQW7ZssZzjYuzYsUPffPONGjZsaDkuu+wySdKhQ4cs/Zs1a1ZqjLp16+rYsWMXdf4ffvih1PcrSe3atXO+7+r85xK7iz1/ecbcuXOnjDF67LHHSn1PKSkpkv73PZX3t/rhhx/UqlWrUv+RUtZ3AQDu4h5EwIOCgoIu+rPPPvus7r77bn300Uf69NNP9cADDyg1NVVr1qxR06ZNFRQUpJUrV2r58uX6+OOPlZ6erkWLFumaa67Rp59+Kn9/f0VERFjGnDdvnu6++2498cQTeuyxxzRy5EhNmzZN9erVk5+fnyZMmOB2NfKXHA6HOnbsqBkzZpT5flRUlOX1+RXW8xljflMc5VUR5/+1Mc99xw8++KASExPL7NuqVStJqtDfCgDKiwQRcKFhw4YKDQ3V119/fVGfb968uaSze/e1aNHC2V5cXKzs7GwlJCRY+nfs2FEdO3bUo48+qtWrV6tXr16aO3euHn/8cUmSn5+f+vXrp379+mnGjBl64okn9Mgjj2j58uVKSEjQsmXLLOO1b99ekrR48WJdffXVeu211yzvHz9+3LkIRCo9ZX6+C73XsmVLffnll+rXr5/Lz7vDnXGaN2+u7du3l2rftm2b831vO/fb16xZs9Rv/kvl/a2aN2+ur7/+WsYYy/dV1ncBAO5iihlwwc/PT4MHD9Y///lPbdiwodT7v1Z1SkhIUEBAgJ5//nlL39dee015eXm6/vrrJUn5+fk6c+aM5bMdO3aUn5+fc9uSo0ePlhq/c+fOkuTsk5CQYDnOVRT9/f1Lxfree+8573s759y+jWVtUF2rVq0yp6PvuOMO7d+/v8zNq3/66Sfnam53uIrjl6677jqtW7dOWVlZzrbCwkK9/PLLio6OVmxsrNvn97RGjRqpb9++eumll3Tw4MFS7x8+fNj55/L+Vtddd50OHDigxYsXO9tOnjypl19+2cPRA/BFVBCBX/HEE0/o008/VZ8+fZzbuBw8eFDvvfeeVq1a5Vw8UpaGDRtq4sSJmjJligYMGKCbbrpJ27dv14svvqiuXbvq97//vSTpP//5j8aNG6fbb79dl112mc6cOaO33npL/v7+uvXWWyVJU6dO1cqVK3X99derefPmOnTokF588UU1bdrUskCjLDfccIOmTp2qpKQk9ezZU1999ZXefvttS1VTOlsNrFOnjubOnavatWurVq1a6t69u2JiYtSlSxctWrRIycnJ6tq1q0JCQnTjjTfqD3/4g95991398Y9/1PLly9WrVy+VlJRo27Ztevfdd7V06dIyF/i40qVLF0nSI488ojvvvFM1a9bUjTfeWObG43/729/0zjvvaODAgXrggQdUr149vfHGG8rOztb7779vWRxUXpmZmbr66quVkpKiyZMnu/35ssyePVu9e/dWx44dNWrUKLVo0UK5ubnKysrSvn37nPsclve3GjVqlGbNmqXhw4dr48aNioiI0FtvvaXg4OByxzRr1iwdP35cBw4ckCT985//1L59+yRJ999/v3PxFAAf5LX108Al5IcffjDDhw83DRs2NHa73bRo0cKMHTvWuV3KuW1uytoKx5iz29q0bdvW1KxZ0zRu3NiMHj3aHDt2zPn+999/b0aOHGlatmxpAgMDTb169czVV19tPvvsM2efjIwMM2jQIBMZGWkCAgJMZGSkGTp0qPnuu+9+Nf5Tp06ZP//5zyYiIsIEBQWZXr16maysLNOnTx/Tp08fS9+PPvrIxMbGmho1ali2PikoKDB33XWXqVOnjpFk2fKmuLjYPPnkk6Z9+/bGbrebunXrmi5dupgpU6aYvLw8Zz9JZuzYsaXi++XWNcYYM23aNNOkSRPj5+dn2fKmrL67du0yt912m6lTp44JDAw03bp1M//6178sfc5tc/PLrYTObVVz/hYv//znP40kM3fu3At/qed99umnny71niSTkpJSKs7hw4eb8PBwU7NmTdOkSRNzww03mMWLFzv7uPNb/fDDD+amm24ywcHBpkGDBmb8+PEmPT293NvcNG/e3Egq8yhriyEAvsNmTCXdGQ4Al4iHHnpI77zzjnbu3Mmj/QD4JO5BBIBfWL58uR577DGSQwA+iwoiAAAALKggAgAAwIIEEQAAABYkiAAAALAgQQQAAIDFJb1RtsPh0IEDB1S7dm2PPeILAABULGOMTpw4ocjIyIvazP63OnXqlIqLiytk7ICAAAUGBlbI2JXpkk4QDxw4oKioKG+HAQAALsLevXvVtGnTSj3nqVOnFNM8RDmHSipk/PDwcGVnZ1/ySeIlnSDWrl1bkjT200TZa9X0cjT4YFtnb4eA8/wuOtvbIeBnh0+FeDsE/Kzk+hxvhwBJZ3Raq/SJ89/xylRcXKycQyX6YWO0Qmt7tnqZf8Kh5l12q7i4mATRm85NK9tr1ZQ9hATR2/yCL+2/DNVNQEiAt0PAz2r4s+F2VWGz8W9FlfDzDszevD0spLZNIbU9e36Hqs/tbpd0gggAAHAxSoxDJR5+VEiJcXh2QC9iFTMAAAAsqCACAACf45CRQ54tIXp6PG+igggAAAALKogAAMDnOOSQp+8Y9PyI3kMFEQAAABZUEAEAgM8pMUYlxrP3DHp6PG+igggAAAALKogAAMDnsIrZNRJEAADgcxwyKiFBvCCmmAEAAGBBBREAAPgcpphdo4IIAAAACyqIAADA57DNjWtUEAEAAGBBBREAAPgcx8+Hp8esLqggAgAAwIIKIgAA8DklFbAPoqfH8yYSRAAA4HNKzNnD02NWF0wxAwAAwIIKIgAA8DksUnGNCiIAAAAsqCACAACf45BNJbJ5fMzqggoiAAAALKggAgAAn+MwZw9Pj1ldUEEEAACARZVIEGfPnq3o6GgFBgaqe/fuWrdunbdDAgAA1VjJz/cgevqoLryeIC5atEjJyclKSUnRpk2b1KlTJyUmJurQoUPeDg0AAFRTJIiueT1BnDFjhkaNGqWkpCTFxsZq7ty5Cg4O1uuvv+7t0AAAAHySVxepFBcXa+PGjZo4caKzzc/PTwkJCcrKyirVv6ioSEVFRc7X+fn5lRInAACoXhzGJofx8DY3Hh7Pm7xaQTxy5IhKSkrUuHFjS3vjxo2Vk5NTqn9qaqrCwsKcR1RUVGWFCgAA4DO8PsXsjokTJyovL8957N2719shAQCASxD3ILrm1SnmBg0ayN/fX7m5uZb23NxchYeHl+pvt9tlt9srKzwAAACf5NUKYkBAgLp06aKMjAxnm8PhUEZGhnr06OHFyAAAQHVWIr8KOaoLrz9JJTk5WSNGjFB8fLy6deumtLQ0FRYWKikpyduhAQAA+CSvJ4hDhgzR4cOHNWnSJOXk5Khz585KT08vtXAFAADAU0wFrGI21WgVs9cTREkaN26cxo0b5+0wAACAj6iIRSXVaZFK9ZksBwAAgEdUiQoiAABAZSoxfioxnq2TlRiPDudVVBABAABgQQURAAD4HIdscni4TuZQ9SkhUkEEAACABRVEAADgc1jF7BoVRAAAAFhQQQQAAD6nYlYxV597EEkQAQCAzzm7SMWzU8KeHs+bmGIGAACABRVEAADgcxzyUwnb3FwQFUQAAABYUEEEAAA+h0UqrlFBBAAAgAUVRAAA4HMc8uNRey5QQQQAAIAFFUQAAOBzSoxNJcbDj9rz8HjeRIIIAAB8TkkFbHNTwhQzAAAAqisqiAAAwOc4jJ8cHt7mxsE2NwAAAKiuSBABAIDPOXcPoqePizF79mxFR0crMDBQ3bt317p16y7Y94MPPlB8fLzq1KmjWrVqqXPnznrrrbcsfYwxmjRpkiIiIhQUFKSEhATt2LHDrZhIEAEAALxk0aJFSk5OVkpKijZt2qROnTopMTFRhw4dKrN/vXr19MgjjygrK0tbtmxRUlKSkpKStHTpUmefp556Ss8//7zmzp2rtWvXqlatWkpMTNSpU6fKHRcJIgAA8DkO/W+rG08djouIY8aMGRo1apSSkpIUGxuruXPnKjg4WK+//nqZ/fv27aubb75Z7dq1U8uWLTV+/HjFxcVp1apVks5WD9PS0vToo49q0KBBiouL05tvvqkDBw5oyZIl5Y6LBBEAAMCD8vPzLUdRUVGZ/YqLi7Vx40YlJCQ42/z8/JSQkKCsrKxfPY8xRhkZGdq+fbuuuuoqSVJ2drZycnIsY4aFhal79+7lGvOcarGK+bGG3yq0Nrmut2XmtPZ2CDjPinWx3g4BP2vc+oi3Q8DPQr0dAKqMinnU3tnxoqKiLO0pKSmaPHlyqf5HjhxRSUmJGjdubGlv3Lixtm3bdsHz5OXlqUmTJioqKpK/v79efPFFXXvttZKknJwc5xi/HPPce+VRLRJEAAAAd5QYP5V4eJubc+Pt3btXoaH/+88Ru93u0fPUrl1bmzdvVkFBgTIyMpScnKwWLVqob9++HjsHCSIAAIAHhYaGWhLEC2nQoIH8/f2Vm5trac/NzVV4ePgFP+fn56dWrVpJkjp37qytW7cqNTVVffv2dX4uNzdXERERljE7d+5c7mtgXhYAAPgch2wVcrgjICBAXbp0UUZGxv/icjiUkZGhHj16lP9aHA7nfY4xMTEKDw+3jJmfn6+1a9e6NSYVRAAAAC9JTk7WiBEjFB8fr27duiktLU2FhYVKSkqSJA0fPlxNmjRRamqqJCk1NVXx8fFq2bKlioqK9Mknn+itt97SnDlzJEk2m00TJkzQ448/rtatWysmJkaPPfaYIiMjNXjw4HLHRYIIAAB8TkXeg+iOIUOG6PDhw5o0aZJycnLUuXNnpaenOxeZ7NmzR35+/xu3sLBQY8aM0b59+xQUFKS2bdvq//7v/zRkyBBnn4ceekiFhYW67777dPz4cfXu3Vvp6ekKDAwsd1w2Yy7dBwfm5+crLCxMx75rwSrmKqD3llu8HQLOc+C7ht4OAT9jFXPVETpwl7dDgKQz5rQy9ZHy8vLKda+eJ53LHZ7b0FNBIZ6tk/1UcEZ/il/tlevyNCqIAADA5/yWR+O5GrO6qD5XAgAAAI+ggggAAHyOw9jkMO6tOi7PmNUFFUQAAABYUEEEAAA+x1EB9yB6+tF93kSCCAAAfI7D+Mnh4W1uPD2eN1WfKwEAAIBHUEEEAAA+p0Q2lbj5aLzyjFldUEEEAACABRVEAADgc7gH0bXqcyUAAADwCCqIAADA55TI8/cMlnh0NO+igggAAAALKogAAMDncA+iaySIAADA55QYP5V4OKHz9HjeVH2uBAAAAB5BBREAAPgcI5scHl6kYtgoGwAAANUVFUQAAOBzuAfRtepzJQAAAPAIKogAAMDnOIxNDuPZewY9PZ43UUEEAACABRVEAADgc0rkpxIP18k8PZ43efVKVq5cqRtvvFGRkZGy2WxasmSJN8MBAAA+4twUs6eP6sKrCWJhYaE6deqk2bNnezMMAAAAnMerU8wDBw7UwIEDvRkCAADwQQ75yeHhOpmnx/OmS+oexKKiIhUVFTlf5+fnezEaAACA6umSSnVTU1MVFhbmPKKiorwdEgAAuASVGFuFHNXFJZUgTpw4UXl5ec5j79693g4JAACg2rmkppjtdrvsdru3wwAAAJc4Nsp27ZKqIAIAAKDiebWCWFBQoJ07dzpfZ2dna/PmzapXr56aNWvmxcgAAEB1ZoyfHMazdTLj4fG8yasJ4oYNG3T11Vc7XycnJ0uSRowYofnz53spKgAAUN2VyKYSeXZK2NPjeZNXE8S+ffvKGOPNEAAAAPALl9QiFQAAAE9wGM8vKnFUo5pX9ZksBwAAgEdQQQQAAD7HUQGLVDw9njdVnysBAACAR1BBBAAAPschmxweXnXs6fG8iQoiAAAALKggAgAAn1NibCrx8CpmT4/nTSSIAADA57BIxbXqcyUAAADwCCqIAADA5zhk8/xG2SxSAQAAQHVFBREAAPgcUwHb3BgqiAAAAKiuqCACAACf4zAVcA9iNdrmhgoiAAAALKggAgAAn8M+iK6RIAIAAJ/DFLNr1SfVBQAAgEdQQQQAAD7HUQHb3LBRNgAAAKotKogAAMDncA+ia1QQAQAAvGj27NmKjo5WYGCgunfvrnXr1l2w7yuvvKIrr7xSdevWVd26dZWQkFCq/9133y2bzWY5BgwY4FZMJIgAAMDnnKsgevpw16JFi5ScnKyUlBRt2rRJnTp1UmJiog4dOlRm/8zMTA0dOlTLly9XVlaWoqKi1L9/f+3fv9/Sb8CAATp48KDzeOedd9yKiwQRAADAS2bMmKFRo0YpKSlJsbGxmjt3roKDg/X666+X2f/tt9/WmDFj1LlzZ7Vt21avvvqqHA6HMjIyLP3sdrvCw8OdR926dd2KiwQRAAD4nIqsIObn51uOoqKiMmMoLi7Wxo0blZCQ4Gzz8/NTQkKCsrKyynUdJ0+e1OnTp1WvXj1Le2Zmpho1aqQ2bdpo9OjR+vHHH936fqrFIpWcMwUqPEOu623dG+72dgg4z79XhXs7BPwsOPa0t0MA8AsVuUglKirK0p6SkqLJkyeX6n/kyBGVlJSocePGlvbGjRtr27Zt5TrnX//6V0VGRlqSzAEDBuiWW25RTEyMdu3apYcfflgDBw5UVlaW/P39yzVutUgQAQAAqoq9e/cqNDTU+dput1fIeaZPn66FCxcqMzNTgYGBzvY777zT+eeOHTsqLi5OLVu2VGZmpvr161eusSm7AQAAn2P0v82yPXWYn8cODQ21HBdKEBs0aCB/f3/l5uZa2nNzcxUe7noW6JlnntH06dP16aefKi4uzmXfFi1aqEGDBtq5c2d5vx4SRAAAAG8ICAhQly5dLAtMzi046dGjxwU/99RTT2natGlKT09XfHz8r55n3759+vHHHxUREVHu2JhiBgAAPqeqbJSdnJysESNGKD4+Xt26dVNaWpoKCwuVlJQkSRo+fLiaNGmi1NRUSdKTTz6pSZMmacGCBYqOjlZOTo4kKSQkRCEhISooKNCUKVN06623Kjw8XLt27dJDDz2kVq1aKTExsdxxkSACAAB4yZAhQ3T48GFNmjRJOTk56ty5s9LT050LV/bs2SM/v/9N+M6ZM0fFxcW67bbbLOOcWwjj7++vLVu26I033tDx48cVGRmp/v37a9q0aW7dC0mCCAAAfE5VqSBK0rhx4zRu3Lgy38vMzLS83r17t8uxgoKCtHTp0ouK43zcgwgAAAALKogAAMDnVKUKYlVEgggAAHwOCaJrTDEDAADAggoiAADwOcbYZDxc8fP0eN5EBREAAAAWVBABAIDPOfd4PE+PWV1QQQQAAIAFFUQAAOBzWMXsGhVEAAAAWFBBBAAAPodVzK5RQQQAAIAFFUQAAOBzuAfRNRJEAADgc5hido0pZgAAAFhQQQQAAD7HVMAUMxVEAAAAVFtUEAEAgM8xkozx/JjVBRVEAAAAWFBBBAAAPschm2zy8DY3Hh7Pm6ggAgAAwIIKIgAA8Dnsg+gaCSIAAPA5DmOTjSepXNBFJYgZGRnKyMjQoUOH5HA4LO+9/vrr5R4nNTVVH3zwgbZt26agoCD17NlTTz75pNq0aXMxYQEAAMAD3L4HccqUKerfv78yMjJ05MgRHTt2zHK4Y8WKFRo7dqzWrFmjZcuW6fTp0+rfv78KCwvdDQsAAKDcjKmYo7pwu4I4d+5czZ8/X3/4wx9+88nT09Mtr+fPn69GjRpp48aNuuqqq37z+AAAAHCf2wlicXGxevbsWRGxKC8vT5JUr169Mt8vKipSUVGR83V+fn6FxAEAAKo3Fqm45vYU87333qsFCxZ4PBCHw6EJEyaoV69e6tChQ5l9UlNTFRYW5jyioqI8HgcAAICvc7uCeOrUKb388sv67LPPFBcXp5o1a1renzFjxkUFMnbsWH399ddatWrVBftMnDhRycnJztf5+fkkiQAAwG1UEF1zO0HcsmWLOnfuLEn6+uuvLe/ZbBf3xYwbN07/+te/tHLlSjVt2vSC/ex2u+x2+0WdAwAAAOXjdoK4fPlyj53cGKP7779fH374oTIzMxUTE+OxsQEAAC6EfRBd+00bZe/bt0+SXFb9XBk7dqwWLFigjz76SLVr11ZOTo4kKSwsTEFBQb8lNAAAgAuqiG1pqtM2N24vUnE4HJo6darCwsLUvHlzNW/eXHXq1NG0adNKbZr9a+bMmaO8vDz17dtXERERzmPRokXuhgUAAAAPcbuC+Mgjj+i1117T9OnT1atXL0nSqlWrNHnyZJ06dUp///vfyz2WqU6pNgAAuGScrSB6epGKR4fzKrcTxDfeeEOvvvqqbrrpJmdbXFycmjRpojFjxriVIAIAAKDqcTtBPHr0qNq2bVuqvW3btjp69KhHggIAAKhIbHPjmtv3IHbq1EmzZs0q1T5r1ix16tTJI0EBAADAe9yuID711FO6/vrr9dlnn6lHjx6SpKysLO3du1effPKJxwMEAADwNPPz4ekxqwu3K4h9+vTRd999p5tvvlnHjx/X8ePHdcstt2j79u268sorKyJGAAAAVKKL2gcxMjKSxSgAAOCSxT2IrpUrQdyyZYs6dOggPz8/bdmyxWXfuLg4jwQGAABQYZhjdqlcCWLnzp2Vk5OjRo0aqXPnzrLZbGXuYWiz2VRSUuLxIAEAAFB5ypUgZmdnq2HDhs4/AwAAXNIqYIpZvjbF3Lx5c+eff/jhB/Xs2VM1alg/eubMGa1evdrSFwAAAJcet1cxX3311WVuiJ2Xl6err77aI0EBAABUpLOP2vP8UV24nSAaY2SzlS6h/vjjj6pVq5ZHggIAAID3lHubm1tuuUXS2YUod999t+x2u/O9kpISbdmyRT179vR8hAAAAB7GNjeulTtBDAsLk3S2gli7dm0FBQU53wsICNDvfvc7jRo1yvMRAgAAoFKVO0GcN2+eJCk6OloPPvgg08kAAODSZWyeX3XsixXEc1JSUioiDgAAgEpTEYtKqtMilXIliFdccYUyMjJUt25dXX755WUuUjln06ZNHgsOAAAAla9cCeKgQYOci1IGDx5ckfEAAABUPB6151K5EsTzp5WZYgYAAKje3N4Hce/evdq3b5/z9bp16zRhwgS9/PLLHg0MAACgopzb5sbTR3XhdoJ41113afny5ZKknJwcJSQkaN26dXrkkUc0depUjwcIAACAyuV2gvj111+rW7dukqR3331XHTt21OrVq/X2229r/vz5no4PAACgYhgPHxdp9uzZio6OVmBgoLp3765169ZdsO8rr7yiK6+8UnXr1lXdunWdhTrLZRmjSZMmKSIiQkFBQUpISNCOHTvcisntBPH06dPOBSufffaZbrrpJklS27ZtdfDgQXeHAwAA8FmLFi1ScnKyUlJStGnTJnXq1EmJiYk6dOhQmf0zMzM1dOhQLV++XFlZWYqKilL//v21f/9+Z5+nnnpKzz//vObOnau1a9eqVq1aSkxM1KlTp8odl9sJYvv27TV37lz997//1bJlyzRgwABJ0oEDB1S/fn13hwMAAKh0VeUexBkzZmjUqFFKSkpSbGys5s6dq+DgYL3++utl9n/77bc1ZswYde7cWW3bttWrr74qh8OhjIyMn6/LKC0tTY8++qgGDRqkuLg4vfnmmzpw4ICWLFlS7rjcThCffPJJvfTSS+rbt6+GDh2qTp06SZL+8Y9/OKeeAQAAqjRPTy+fN82cn59vOYqKisoMobi4WBs3blRCQoKzzc/PTwkJCcrKyirXZZw8eVKnT59WvXr1JEnZ2dnONSLnhIWFqXv37uUeU7qIJ6n07dtXR44cUX5+vurWretsv++++xQcHOzucB7xQUGsAt2/FHjY3p/q/nonVJridj95OwT8rKZfibdDwM+q0TZ1qMKioqIsr1NSUjR58uRS/Y4cOaKSkhI1btzY0t64cWNt27atXOf661//qsjISGdCmJOT4xzjl2Oee688Liqr8vf315kzZ7Rq1SpJUps2bRQdHX0xQwEAAHiB7efD02Oe3RIwNDTU2Xpu7YanTZ8+XQsXLlRmZqYCAwM9OrbbU8yFhYUaOXKkIiIidNVVV+mqq65SZGSk7rnnHp08edKjwQEAAFxqQkNDLceFEsQGDRrI399fubm5lvbc3FyFh4e7PMczzzyj6dOn69NPP1VcXJyz/dznLmbM87mdICYnJ2vFihX65z//qePHj+v48eP66KOPtGLFCv35z392dzgAAIDKV4H3IJZXQECAunTp4lxgIsm54KRHjx4X/NxTTz2ladOmKT09XfHx8Zb3YmJiFB4ebhkzPz9fa9eudTnmL7k9xfz+++9r8eLF6tu3r7PtuuuuU1BQkO644w7NmTPH3SEBAAB8UnJyskaMGKH4+Hh169ZNaWlpKiwsVFJSkiRp+PDhatKkiVJTUyWdXSw8adIkLViwQNHR0c77CkNCQhQSEiKbzaYJEybo8ccfV+vWrRUTE6PHHntMkZGRGjx4cLnjcjtBPHnyZKkbHyWpUaNGTDEDAIBLw2/c3PqCY7ppyJAhOnz4sCZNmqScnBx17txZ6enpzlxrz5498vP734TvnDlzVFxcrNtuu80yzvkLYR566CEVFhbqvvvu0/Hjx9W7d2+lp6e7dZ+izRjj1uX069dP9evX15tvvuk80U8//aQRI0bo6NGj+uyzz9wZ7jfJz89XWFiY/r6urwJDWMXsbauPtfR2CDjPph+aeTsE/KxVRNkb3qLymWv2/3onVLgz5rQy9ZHy8vIsizkqw7ncIerFyfIL8uzCDsdPp7R3zGSvXJenuZ1VzZw5U4mJiWratKlzD8Qvv/xSgYGBWrp0qccDBAAA8DhjO3t4esxqwu0EsUOHDtqxY4fefvtt5x49Q4cO1bBhwxQUFOTxAAEAADzNmLOHp8esLi5qXjY4OFijRo3ydCwAAACoAi4qQdy+fbteeOEFbd26VZLUrl07jRs3Tm3btvVocAAAABWiiixSqarc3gfx/fffV4cOHbRx40Z16tRJnTp10qZNm9SxY0e9//77FREjAAAAKpHbFcSHHnpIEydO1NSpUy3tKSkpeuihh3Trrbd6LDgAAIAKwSIVl9yuIB48eFDDhw8v1f773/9eBw8e9EhQAAAA8B63E8S+ffvqv//9b6n2VatW6corr/RIUAAAABXJZirmqC7cnmK+6aab9Ne//lUbN27U7373O0nSmjVr9N5772nKlCn6xz/+YekLAACAS4vbCeKYMWMkSS+++KJefPHFMt+TJJvNppKSkt8YHgAAQAVgFbNLbieIDoejIuIAAACoPCxSccntexABAABQvV3URtkAAACXNKaYXaKCCAAAAAsqiAAAwPdQQXSJCiIAAAAsLqqC6HA4tHPnTh06dKjUquarrrrKI4EBAABUGCqILrmdIK5Zs0Z33XWXfvjhBxlj/SbY+xAAAODS53aC+Mc//lHx8fH6+OOPFRERIZut+uz5AwAAfAT7ILrkdoK4Y8cOLV68WK1ataqIeAAAAOBlbi9S6d69u3bu3FkRsQAAAFQKm6mYo7pwu4J4//33689//rNycnLUsWNH1axZ0/J+XFycx4IDAACoECxSccntBPHWW2+VJI0cOdLZZrPZZIxhkQoAAEA14HaCmJ2d7bGTz5kzR3PmzNHu3bslSe3bt9ekSZM0cOBAj50DAAAA7nE7QWzevLnHTt60aVNNnz5drVu3ljFGb7zxhgYNGqQvvvhC7du399h5AAAAUH4X9SSVt956S7169VJkZKR++OEHSVJaWpo++ugjt8a58cYbdd1116l169a67LLL9Pe//10hISFas2bNxYQFAABQLjZVwCIVb1+UB7mdIM6ZM0fJycm67rrrdPz4cec9h3Xq1FFaWtpFB1JSUqKFCxeqsLBQPXr0KLNPUVGR8vPzLQcAAAA8y+0E8YUXXtArr7yiRx55RP7+/s72+Ph4ffXVV24H8NVXXykkJER2u11//OMf9eGHHyo2NrbMvqmpqQoLC3MeUVFRbp8PAADAuVG2p49qwu0EMTs7W5dffnmpdrvdrsLCQrcDaNOmjTZv3qy1a9dq9OjRGjFihL799tsy+06cOFF5eXnOY+/evW6fDwAAAK65vUglJiZGmzdvLrVYJT09Xe3atXM7gICAAOdTWbp06aL169dr5syZeumll0r1tdvtstvtbp8DAADAgn0QXXI7QUxOTtbYsWN16tQpGWO0bt06vfPOO0pNTdWrr776mwNyOBwqKir6zeMAAABcEAmiS24niPfee6+CgoL06KOP6uTJk7rrrrsUGRmpmTNn6s4773RrrIkTJ2rgwIFq1qyZTpw4oQULFigzM1NLly51NywAAAB4iNsJYn5+voYNG6Zhw4bp5MmTKigoUKNGjSRJO3fudE4Xl8ehQ4c0fPhwHTx4UGFhYYqLi9PSpUt17bXXuhsWAABAuVXEs5N9+lnM119/vT777DPZ7XYFBwcrODhYkrR9+3b169dP+/btK/dYr732mrunBwAAQAVzexVzSEiIbr75Zp05c8bZtnXrVvXt29f5nGYAAIAqzVTQUU24nSB+8MEHysvL07Bhw2SM0ddff62+fftq6NChmjlzZkXECAAAgErkdoIYFBSkjz/+WNu3b9cdd9yhfv36afjw4ZoxY0ZFxAcAAOB5VBBdKtc9iL98pJ2fn58WLVqka6+9Vrfeeqsee+wxZ5/Q0FDPRwkAAIBKU64EsU6dOrLZSj8+xhijuXPn6qWXXpIxRjabzflsZgAAgKqKVcyulStBXL58eUXHAQAAUHkq4tnJ1ehZzOVKEPv06VPRcQAAAKCKcHsfREk6fvy4XnvtNW3dulWS1L59e40cOVJhYWEeDQ4AAKBC8Kg9l9xexbxhwwa1bNlSzz33nI4ePaqjR49qxowZatmypTZt2lQRMQIAAKASuV1B/NOf/qSbbrpJr7zyimrUOPvxM2fO6N5779WECRO0cuVKjwcJAADgSSxScc3tBHHDhg2W5FCSatSooYceekjx8fEeDQ4AAACVz+0p5tDQUO3Zs6dU+969e1W7dm2PBAUAAFCh2CjbJbcTxCFDhuiee+7RokWLtHfvXu3du1cLFy7Uvffeq6FDh1ZEjAAAAKhEbk8xP/PMM7LZbBo+fLjOnDkjSapZs6ZGjx6t6dOnezxAAAAAj6uAexCrUwXR7QQxICBAM2fOVGpqqnbt2iVJatmypYKDgz0eHAAAQIVgmxuX3J5iHjlypE6cOKHg4GB17NhRHTt2VHBwsAoLCzVy5MiKiBEAAACVyO0E8Y033tBPP/1Uqv2nn37Sm2++6ZGgAAAAKhSLVFwq9xRzfn6+jDEyxujEiRMKDAx0vldSUqJPPvlEjRo1qpAgAQAAUHnKnSDWqVNHNptNNptNl112Wan3bTabpkyZ4tHgAAAAKgIbZbtW7gRx+fLlMsbommuu0fvvv6969eo53wsICFDz5s0VGRlZIUECAACg8pQ7QezTp48kKTs7W82aNZPNZquwoAAAAOA9bm9z07x584qIAwAAAFWE26uYAQAALnlVaBXz7NmzFR0drcDAQHXv3l3r1q27YN9vvvlGt956q6Kjo2Wz2ZSWllaqz+TJk53rRs4dbdu2dSsmEkQAAOBzzi1S8fThrkWLFik5OVkpKSnatGmTOnXqpMTERB06dKjM/idPnlSLFi00ffp0hYeHX3Dc9u3b6+DBg85j1apVbsVFgggAAOBB+fn5lqOoqOiCfWfMmKFRo0YpKSlJsbGxmjt3roKDg/X666+X2b9r1656+umndeedd8put19w3Bo1aig8PNx5NGjQwK1rcPsexJSUFI0cObJK3Yv4/clGCvCr6e0wfF5+ceCvd0KliY/+wdsh4Gfbj7BHbFXBLwGLCtqWJioqyvI6JSVFkydPLtWvuLhYGzdu1MSJE51tfn5+SkhIUFZW1m+KYceOHYqMjFRgYKB69Oih1NRUNWvWrNyfd7uC+NFHH6lly5bq16+fFixY4DIrBgAA8DV79+5VXl6e8zg/ATzfkSNHVFJSosaNG1vaGzdurJycnIs+f/fu3TV//nylp6drzpw5ys7O1pVXXqkTJ06Uewy3E8TNmzdr/fr1at++vcaPH6/w8HCNHj1a69evd3coAAAA76jARSqhoaGWw9VUcEUYOHCgbr/9dsXFxSkxMVGffPKJjh8/rnfffbfcY1zUPYiXX365nn/+eR04cECvvfaa9u3bp169eikuLk4zZ85UXl7exQwLAADgMxo0aCB/f3/l5uZa2nNzc10uQHFXnTp1dNlll2nnzp3l/sxvWqRijNHp06dVXFwsY4zq1q2rWbNmKSoqSosWLfotQwMAAFSYqrCKOSAgQF26dFFGRoazzeFwKCMjQz169PDYtRYUFGjXrl2KiIgo92cuKkHcuHGjxo0bp4iICP3pT3/S5Zdfrq1bt2rFihXasWOH/v73v+uBBx64mKEBAAB8RnJysl555RW98cYb2rp1q0aPHq3CwkIlJSVJkoYPH265h7G4uFibN2/W5s2bVVxcrP3792vz5s2W6uCDDz6oFStWaPfu3Vq9erVuvvlm+fv7a+jQoeWOy+1VzB07dtS2bdvUv39/vfbaa7rxxhvl7+9v6TN06FCNHz/e3aEBAAAqx2/Y2NrlmG4aMmSIDh8+rEmTJiknJ0edO3dWenq6c+HKnj175Of3v3regQMHdPnllztfP/PMM3rmmWfUp08fZWZmSpL27dunoUOH6scff1TDhg3Vu3dvrVmzRg0bNix3XG4niHfccYdGjhypJk2aXLBPgwYN5HA43B0aAACgUlzsxta/NubFGDdunMaNG1fme+eSvnOio6NljOsTLVy48OICOY9bU8ynT5/W/PnzlZ+f/5tPDAAAgKrJrQpizZo1derUqYqKBQAAoHJUkSnmqsrtRSpjx47Vk08+qTNnzlREPAAAAPAyt+9BXL9+vTIyMvTpp5+qY8eOqlWrluX9Dz74wGPBAQAAVAgqiC65nSDWqVNHt956a0XEAgAAgCrA7QRx3rx5FREHAABApalKq5iroovaKPvMmTP67LPP9NJLLzkf/HzgwAEVFBR4NDgAAABUPrcriD/88IMGDBigPXv2qKioSNdee61q166tJ598UkVFRZo7d25FxAkAAOA53IPoktsVxPHjxys+Pl7Hjh1TUFCQs/3mm2+2PEsQAACgyjIVdFQTblcQ//vf/2r16tUKCAiwtEdHR2v//v0eCwwAAADe4XaC6HA4VFJSUqp93759ql27tkeCAgAAqEgsUnHN7Snm/v37Ky0tzfnaZrOpoKBAKSkpuu666zwZGwAAALzA7Qris88+q8TERMXGxurUqVO66667tGPHDjVo0EDvvPNORcQIAADgWSxSccntBLFp06b68ssvtXDhQm3ZskUFBQW65557NGzYMMuiFQAAAFya3E4QJalGjRr6/e9/7+lYAAAAKgX3ILrmdoL45ptvunx/+PDhFx0MAAAAvM/tBHH8+PGW16dPn9bJkycVEBCg4OBgEkQAAFD1cQ+iS26vYj527JjlKCgo0Pbt29W7d28WqQAAgEsDG2W7dFHPYv6l1q1ba/r06aWqiwAAALj0XNQilTIHqlFDBw4c8NRwAAAAFcb28+HpMasLtxPEf/zjH5bXxhgdPHhQs2bNUq9evTwWGAAAALzD7QRx8ODBltc2m00NGzbUNddco2effdZTcQEAAFQcFqm45PY9iA6Hw3KUlJQoJydHCxYsUERExEUHMn36dNlsNk2YMOGixwAAAMBvd9H3IB45ckQBAQEKDQ39zUGsX79eL730kuLi4n7zWAAAAL+GjbJdc6uCePz4cY0dO1YNGjRQ48aNVbduXYWHh2vixIk6efLkRQVQUFCgYcOG6ZVXXlHdunUvagwAAAB4TrkriEePHlWPHj20f/9+DRs2TO3atZMkffvtt3rhhRe0bNkyrVq1Slu2bNGaNWv0wAMPlGvcsWPH6vrrr1dCQoIef/xxl32LiopUVFTkfJ2fn1/e8AEAAP6HexBdKneCOHXqVAUEBGjXrl1q3Lhxqff69++vP/zhD/r000/1/PPPl2vMhQsXatOmTVq/fn25+qempmrKlCnlDRkAAODCqlFC52nlnmJesmSJnnnmmVLJoSSFh4frqaee0vvvv6/k5GSNGDHiV8fbu3evxo8fr7fffluBgYHlimHixInKy8tzHnv37i1v+AAAACinclcQDx48qPbt21/w/Q4dOsjPz08pKSnlGm/jxo06dOiQrrjiCmdbSUmJVq5cqVmzZqmoqEj+/v6Wz9jtdtnt9vKGDAAAUCYWqbhW7gSxQYMG2r17t5o2bVrm+9nZ2WrUqFG5T9yvXz999dVXlrakpCS1bdtWf/3rX0slhwAAAKgc5U4QExMT9cgjj2jZsmUKCAiwvFdUVKTHHntMAwYMKPeJa9eurQ4dOljaatWqpfr165dqBwAA8CgWqbjk1iKV+Ph4tW7dWmPHjlXbtm1ljNHWrVv14osvqqioSG+++WZFxgoAAIBKUO4EsWnTpsrKytKYMWM0ceJEGXM2TbbZbLr22ms1a9YsNWvW7DcFk5mZ+Zs+DwAAUB7cg+iaW09SiYmJ0b///W8dO3ZMO3bskCS1atVK9erVq5DgAAAAUPku6lF7devWVbdu3TwdCwAAQOXgHkSX3HrUHgAAAKq/i6ogAgAAXMq4B9E1EkQAAOB7mGJ2iSlmAAAAWFBBBAAAvocKoktUEAEAAGBBBREAAPgcFqm4RgURAAAAFlQQAQCA7+EeRJeoIAIAAMCCCiIAAPA5NmNkM54t+Xl6PG8iQQQAAL6HKWaXmGIGAACABRVEAADgc9jmxjUqiAAAALCggggAAHwP9yC6RAURAAAAFlQQAQCAz+EeRNeoIAIAAMCCCiIAAPA93IPoEhVEAADgc85NMXv6uBizZ89WdHS0AgMD1b17d61bt+6Cfb/55hvdeuutio6Ols1mU1pa2m8esywkiAAAAF6yaNEiJScnKyUlRZs2bVKnTp2UmJioQ4cOldn/5MmTatGihaZPn67w8HCPjFkWEkQAAOB7TAUdbpoxY4ZGjRqlpKQkxcbGau7cuQoODtbrr79eZv+uXbvq6aef1p133im73e6RMctCgggAAOBB+fn5lqOoqKjMfsXFxdq4caMSEhKcbX5+fkpISFBWVtZFndtTY1aLRSrLsuLkFxjo7TB8Xp3WR70dAs6T3vZjb4eAn11XfJ23Q8DPSrwdAKqUitqWJioqyvI6JSVFkydPLtXvyJEjKikpUePGjS3tjRs31rZt2y7q3J4as1okiAAAAFXF3r17FRoa6nx9oangqowEEQAA+B5jzh6eHlNSaGioJUG8kAYNGsjf31+5ubmW9tzc3AsuQKmsMbkHEQAAwAsCAgLUpUsXZWRkONscDocyMjLUo0cPr45JBREAAPicqvKoveTkZI0YMULx8fHq1q2b0tLSVFhYqKSkJEnS8OHD1aRJE6Wmpko6uwjl22+/df55//792rx5s0JCQtSqVatyjVkeJIgAAMD3VJEnqQwZMkSHDx/WpEmTlJOTo86dOys9Pd25yGTPnj3y8/vfhO+BAwd0+eWXO18/88wzeuaZZ9SnTx9lZmaWa8zyIEEEAADwonHjxmncuHFlvncu6TsnOjpaphz3TroaszxIEAEAgM+xOc4enh6zumCRCgAAACyoIAIAAN9TRe5BrKqoIAIAAMCCCiIAAPA5VWWbm6qKCiIAAAAsqCACAADfU4GP2qsOSBABAIDPYYrZNaaYAQAAYEEFEQAA+B62uXGJCiIAAAAsqCACAACfwz2IrlFBBAAAgAUVRAAA4HvY5sYlKogAAACwoIIIAAB8DvcgukaCCAAAfA/b3LjEFDMAAAAsqCACAACfwxSza1QQAQAAYEEFEQAA+B6HOXt4esxqggoiAAAALLyaIE6ePFk2m81ytG3b1pshAQAAX2Aq6KgmvD7F3L59e3322WfO1zVqeD0kAAAAn+b1bKxGjRoKDw/3dhgAAMCH2FQBq5g9O5xXef0exB07digyMlItWrTQsGHDtGfPngv2LSoqUn5+vuUAAABw27lnMXv6qCa8miB2795d8+fPV3p6uubMmaPs7GxdeeWVOnHiRJn9U1NTFRYW5jyioqIqOWIAAIDqz6sJ4sCBA3X77bcrLi5OiYmJ+uSTT3T8+HG9++67ZfafOHGi8vLynMfevXsrOWIAAFAdnNso29NHdeH1exDPV6dOHV122WXauXNnme/b7XbZ7fZKjgoAAMC3eP0exPMVFBRo165dioiI8HYoAACgOmObG5e8miA++OCDWrFihXbv3q3Vq1fr5ptvlr+/v4YOHerNsAAAAHyaV6eY9+3bp6FDh+rHH39Uw4YN1bt3b61Zs0YNGzb0ZlgAAKCasxkjm4dXHXt6PG/yaoK4cOFCb54eAAAAZahSi1QAAAAqhePnw9NjVhMkiAAAwOcwxexalVrFDAAAAO+jgggAAHxPRWxLU30KiFQQAQAAYEUFEQAA+B5jzh6eHrOaoIIIAAAACyqIAADA59jM2cPTY1YXVBABAABgQQURAAD4Hu5BdIkKIgAAACyoIAIAAJ9jc5w9PD1mdUGCCAAAfA9TzC4xxQwAAAALKogAAMD38Kg9l6ggAgAAwIIKIgAA8Dk2Y2Tz8D2Dnh7Pm6ggAgAAwIIKIgAA8D2sYnaJCiIAAAAsqCACAADfYyR5emPr6lNAJEEEAAC+h0UqrjHFDAAA4EWzZ89WdHS0AgMD1b17d61bt85l//fee09t27ZVYGCgOnbsqE8++cTy/t133y2bzWY5BgwY4FZMJIgAAMD3GP1voYrHDvfDWLRokZKTk5WSkqJNmzapU6dOSkxM1KFDh8rsv3r1ag0dOlT33HOPvvjiCw0ePFiDBw/W119/bek3YMAAHTx40Hm88847bsVFgggAAOAlM2bM0KhRo5SUlKTY2FjNnTtXwcHBev3118vsP3PmTA0YMEB/+ctf1K5dO02bNk1XXHGFZs2aZelnt9sVHh7uPOrWretWXCSIAADA93i8evi/bXPy8/MtR1FRUZkhFBcXa+PGjUpISHC2+fn5KSEhQVlZWWV+Jisry9JfkhITE0v1z8zMVKNGjdSmTRuNHj1aP/74o1tfT7VYpDKkz2rZQ2p6Owyft3hnZ2+HgPP8fndfb4eAn508HeDtEPAzu7cDgE+IioqyvE5JSdHkyZNL9Tty5IhKSkrUuHFjS3vjxo21bdu2MsfOyckps39OTo7z9YABA3TLLbcoJiZGu3bt0sMPP6yBAwcqKytL/v7+5bqGapEgAgAAuMUhyVYBY0rau3evQkNDnc12e+X+p8mdd97p/HPHjh0VFxenli1bKjMzU/369SvXGEwxAwAAeFBoaKjluFCC2KBBA/n7+ys3N9fSnpubq/Dw8DI/Ex4e7lZ/SWrRooUaNGignTt3lvsaSBABAIDPObcPoqcPdwQEBKhLly7KyMhwtjkcDmVkZKhHjx5lfqZHjx6W/pK0bNmyC/aXpH379unHH39UREREuWMjQQQAAL6nAhepuCM5OVmvvPKK3njjDW3dulWjR49WYWGhkpKSJEnDhw/XxIkTnf3Hjx+v9PR0Pfvss9q2bZsmT56sDRs2aNy4cZKkgoIC/eUvf9GaNWu0e/duZWRkaNCgQWrVqpUSExPLHRf3IAIAAHjJkCFDdPjwYU2aNEk5OTnq3Lmz0tPTnQtR9uzZIz+//9XzevbsqQULFujRRx/Vww8/rNatW2vJkiXq0KGDJMnf319btmzRG2+8oePHjysyMlL9+/fXtGnT3LoXkgQRAAD4nous+P3qmBdh3LhxzgrgL2VmZpZqu/3223X77beX2T8oKEhLly69qDjOxxQzAAAALKggAgAA31OFKohVERVEAAAAWFBBBAAAvqcCN8quDqggAgAAwIIKIgAA8DkXs7F1ecasLkgQAQCA72GRiktMMQMAAMCCCiIAAPA9DiPZPFzxc1BBBAAAQDVFBREAAPge7kF0iQoiAAAALKggAgAAH1QBFURRQQQAAEA1RQURAAD4Hu5BdIkEEQAA+B6HkcenhNnmBgAAANUVFUQAAOB7jOPs4ekxqwkqiAAAALCggggAAHwPi1RcooIIAAAACyqIAADA97CK2SWvVxD379+v3//+96pfv76CgoLUsWNHbdiwwdthAQAA+CyvVhCPHTumXr166eqrr9a///1vNWzYUDt27FDdunW9GRYAAKjuuAfRJa8miE8++aSioqI0b948Z1tMTIwXIwIAAD7BqAISRM8O501enWL+xz/+ofj4eN1+++1q1KiRLr/8cr3yyisX7F9UVKT8/HzLAQAAAM/yaoL4/fffa86cOWrdurWWLl2q0aNH64EHHtAbb7xRZv/U1FSFhYU5j6ioqEqOGAAAVAvnppg9fVQTXk0QHQ6HrrjiCj3xxBO6/PLLdd9992nUqFGaO3dumf0nTpyovLw857F3795KjhgAAKD68+o9iBEREYqNjbW0tWvXTu+//36Z/e12u+x2e2WEBgAAqjOHQ5KHH43n4FF7HtGrVy9t377d0vbdd9+pefPmXooIAAAAXq0g/ulPf1LPnj31xBNP6I477tC6dev08ssv6+WXX/ZmWAAAoLpjmxuXvFpB7Nq1qz788EO988476tChg6ZNm6a0tDQNGzbMm2EBAAD4NK8/au+GG27QDTfc4O0wAACAL6GC6JLXE0QAAIBKx7OYXfL6s5gBAABQtVBBBAAAPscYh4zx7LY0nh7Pm6ggAgAAwIIKIgAA8D3GeP6ewWq0SIUKIgAAACyoIAIAAN9jKmAVMxVEAAAAVFdUEAEAgO9xOCSbh1cdV6NVzCSIAADA9zDF7BJTzAAAALCggggAAHyOcThkPDzFzEbZAAAAqLaoIAIAAN/DPYguUUEEAACABRVEAADgexxGslFBvBAqiAAAALCggggAAHyPMZI8vVE2FUQAAABUU1QQAQCAzzEOI+PhexBNNaogkiACAADfYxzy/BQzG2UDAACgmiJBBAAAPsc4TIUcF2P27NmKjo5WYGCgunfvrnXr1rns/95776lt27YKDAxUx44d9cknn1ivzRhNmjRJERERCgoKUkJCgnbs2OFWTCSIAAAAXrJo0SIlJycrJSVFmzZtUqdOnZSYmKhDhw6V2X/16tUaOnSo7rnnHn3xxRcaPHiwBg8erK+//trZ56mnntLzzz+vuXPnau3atapVq5YSExN16tSpcsdFgggAAHyPcVTM4aYZM2Zo1KhRSkpKUmxsrObOnavg4GC9/vrrZfafOXOmBgwYoL/85S9q166dpk2bpiuuuEKzZs06e1nGKC0tTY8++qgGDRqkuLg4vfnmmzpw4ICWLFlS7rgu6UUq51YLFRWe9nIkkKSSk0XeDgHnOV1Y7O0Q8LMzhfzdqCr8Df9eVAVndPZ38Oaq3zM67fFHMZ+7rvz8fEu73W6X3W4v1b+4uFgbN27UxIkTnW1+fn5KSEhQVlZWmefIyspScnKypS0xMdGZ/GVnZysnJ0cJCQnO98PCwtS9e3dlZWXpzjvvLNe1XNIJ4okTJyRJz1+7zMuR4KxPfr0LKo17d5sAQOU7ceKEwsLCKvWcAQEBCg8P16qcivk3KyQkRFFRUZa2lJQUTZ48uVTfI0eOqKSkRI0bN7a0N27cWNu2bStz/JycnDL75+TkON8/13ahPuVxSSeIkZGR2rt3r2rXri2bzebtcC5afn6+oqKitHfvXoWGhno7HJ/Gb1F18FtUHfwWVUt1+D2MMTpx4oQiIyMr/dyBgYHKzs5WcXHFzLIYY0rlJGVVD6u6SzpB9PPzU9OmTb0dhseEhoZesn/Zqxt+i6qD36Lq4LeoWi7136OyK4fnCwwMVGBgoNfOf06DBg3k7++v3NxcS3tubq7Cw8PL/Ex4eLjL/uf+Nzc3VxEREZY+nTt3LndsLFIBAADwgoCAAHXp0kUZGRnONofDoYyMDPXo0aPMz/To0cPSX5KWLVvm7B8TE6Pw8HBLn/z8fK1du/aCY5blkq4gAgAAXMqSk5M1YsQIxcfHq1u3bkpLS1NhYaGSkpIkScOHD1eTJk2UmpoqSRo/frz69OmjZ599Vtdff70WLlyoDRs26OWXX5Yk2Ww2TZgwQY8//rhat26tmJgYPfbYY4qMjNTgwYPLHRcJYhVgt9uVkpJySd6jUN3wW1Qd/BZVB79F1cLvUb0MGTJEhw8f1qRJk5STk6POnTsrPT3duchkz5498vP734Rvz549tWDBAj366KN6+OGH1bp1ay1ZskQdOnRw9nnooYdUWFio++67T8ePH1fv3r2Vnp7u1rS6zVSnJ0sDAADgN+MeRAAAAFiQIAIAAMCCBBEAAAAWJIgAAACwIEH0stmzZys6OlqBgYHq3r271q1b5+2QfNLKlSt14403KjIyUjabza0HmsOzUlNT1bVrV9WuXVuNGjXS4MGDtX37dm+H5ZPmzJmjuLg454bMPXr00L///W9vhwVJ06dPd25nAlQEEkQvWrRokZKTk5WSkqJNmzapU6dOSkxM1KFDh7wdms8pLCxUp06dNHv2bG+H4vNWrFihsWPHas2aNVq2bJlOnz6t/v37q7Cw0Nuh+ZymTZtq+vTp2rhxozZs2KBrrrlGgwYN0jfffOPt0Hza+vXr9dJLLykuLs7boaAaY5sbL+revbu6du2qWbNmSTq7e3pUVJTuv/9+/e1vf/NydL7LZrPpww8/dGtDUVScw4cPq1GjRlqxYoWuuuoqb4fj8+rVq6enn35a99xzj7dD8UkFBQW64oor9OKLL+rxxx9X586dlZaW5u2wUA1RQfSS4uJibdy4UQkJCc42Pz8/JSQkKCsry4uRAVVLXl6epLOJCbynpKRECxcuVGFhoVuP64JnjR07Vtdff73l3w6gIvAkFS85cuSISkpKnDuln9O4cWNt27bNS1EBVYvD4dCECRPUq1cvy1MCUHm++uor9ejRQ6dOnVJISIg+/PBDxcbGejssn7Rw4UJt2rRJ69ev93Yo8AEkiACqrLFjx+rrr7/WqlWrvB2Kz2rTpo02b96svLw8LV68WCNGjNCKFStIEivZ3r17NX78eC1btsytx6UBF4sE0UsaNGggf39/5ebmWtpzc3MVHh7upaiAqmPcuHH617/+pZUrV6pp06beDsdnBQQEqFWrVpKkLl26aP369Zo5c6ZeeuklL0fmWzZu3KhDhw7piiuucLaVlJRo5cqVmjVrloqKiuTv7+/FCFHdcA+ilwQEBKhLly7KyMhwtjkcDmVkZHB/D3yaMUbjxo3Thx9+qP/85z+KiYnxdkg4j8PhUFFRkbfD8Dn9+vXTV199pc2bNzuP+Ph4DRs2TJs3byY5hMdRQfSi5ORkjRgxQvHx8erWrZvS0tJUWFiopKQkb4fmcwoKCrRz507n6+zsbG3evFn16tVTs2bNvBiZ7xk7dqwWLFigjz76SLVr11ZOTo4kKSwsTEFBQV6OzrdMnDhRAwcOVLNmzXTixAktWLBAmZmZWrp0qbdD8zm1a9cudR9urVq1VL9+fe7PRYUgQfSiIUOG6PDhw5o0aZJycnLUuXNnpaenl1q4goq3YcMGXX311c7XycnJkqQRI0Zo/vz5XorKN82ZM0eS1LdvX0v7vHnzdPfdd1d+QD7s0KFDGj58uA4ePKiwsDDFxcVp6dKluvbaa70dGoAKxj6IAAAAsOAeRAAAAFiQIAIAAMCCBBEAAAAWJIgAAACwIEEEAACABQkiAAAALEgQAQAAYEGCCAAAAAsSRAAeM3/+fNWpU8fbYXjc3XffrcGDB7vsk5mZKZvNpuPHj1dKTABQkUgQgSqurORk8eLFCgwM1LPPPlsh56yuid7FmjlzpuWRi3379tWECRMsfXr27Ol8JB0AXOp4FjNwiXn11Vc1duxYzZ07V0lJSd4OxyeUJ+kLCAhQeHh4JUQDABWPCiJwCXnqqad0//33a+HChZbk8KOPPtIVV1yhwMBAtWjRQlOmTNGZM2ckSSNHjtQNN9xgGef06dNq1KiRXnvttVLnyMzMVFJSkvLy8mSz2WSz2TR58mRJ0rFjxzR8+HDVrVtXwcHBGjhwoHbs2HHBeA8fPqz4+HjdfPPNKioqksPhUGpqqmJiYhQUFKROnTpp8eLFlnPbbDZlZGQoPj5ewcHB6tmzp7Zv337Bc+zevVs2m00LFy5Uz549FRgYqA4dOmjFihWWfitWrFC3bt1kt9sVERGhv/3tb87vSDpble3YsaOCgoJUv359JSQkqLCwUJK1inv33XdrxYoVmjlzpvP72b17d5lTzO+//77at28vu92u6OjoUhXf6OhoPfHEExo5cqRq166tZs2a6eWXX77gtQJApTEAqrQRI0aYQYMGmYceesiEhISYzz77zPL+ypUrTWhoqJk/f77ZtWuX+fTTT010dLSZPHmyMcaYzz//3Pj7+5sDBw44P/PBBx+YWrVqmRMnTpQ6X1FRkUlLSzOhoaHm4MGD5uDBg85+N910k2nXrp1ZuXKl2bx5s0lMTDStWrUyxcXFxhhj5s2bZ8LCwowxxuzZs8e0adPGjBgxwpw5c8YYY8zjjz9u2rZta9LT082uXbvMvHnzjN1uN5mZmcYYY5YvX24kme7du5vMzEzzzTffmCuvvNL07Nnzgt9Pdna2kWSaNm1qFi9ebL799ltz7733mtq1a5sjR44YY4zZt2+fCQ4ONmPGjDFbt241H374oWnQoIFJSUkxxhhz4MABU6NGDTNjxgyTnZ1ttmzZYmbPnu287nO/gTHGHD9+3PTo0cOMGjXK+f2cOXPGGfuxY8eMMcZs2LDB+Pn5malTp5rt27ebefPmmaCgIDNv3jxn7M2bNzf16tUzs2fPNjt27DCpqanGz8/PbNu2zeX/JwCgopEgAlXciBEjTEBAgJFkMjIySr3fr18/88QTT1ja3nrrLRMREeF8HRsba5588knn6xtvvNHcfffdFzzn+YneOd99952RZD7//HNn25EjR0xQUJB59913LZ/btm2biYqKMg888IBxOBzGGGNOnTplgoODzerVqy3j3nPPPWbo0KHGmP8liOcnwR9//LGRZH766acyYz2XIE6fPt3Zdvr0adO0aVPnNT/88MOmTZs2zliMMWb27NkmJCTElJSUmI0bNxpJZvfu3WWe4/wE0Rhj+vTpY8aPH2/p88sE8a677jLXXnutpc9f/vIXExsb63zdvHlz8/vf/9752uFwmEaNGpk5c+aUGQcAVBammIFLQFxcnKKjo5WSkqKCggLLe19++aWmTp2qkJAQ5zFq1CgdPHhQJ0+elCTde++9mjdvniQpNzdX//73vzVy5Ei3Yti6datq1Kih7t27O9vq16+vNm3aaOvWrc62n376SVdeeaVuueUW5zSsJO3cuVMnT57Utddea4n1zTff1K5du0pd7zkRERGSpEOHDrmMr0ePHs4/16hRQ/Hx8c64tm7dqh49ejhjkaRevXqpoKBA+/btU6dOndSvXz917NhRt99+u1555RUdO3bMre/nl7Zu3apevXpZ2nr16qUdO3aopKTE2Xb+tdpsNoWHh//qtQJARSNBBC4BTZo0UWZmpvbv368BAwboxIkTzvcKCgo0ZcoUbd682Xl89dVX2rFjhwIDAyVJw4cP1/fff6+srCz93//9n2JiYnTllVdWSKx2u10JCQn617/+pf3791vilKSPP/7YEuu3335ruQ9RkmrWrOn887mkzuFwVEi8kuTv769ly5bp3//+t2JjY/XCCy+oTZs2ys7OrrBznnP+tUpnr7cirxUAyoMEEbhENG/eXCtWrFBOTo4lSbziiiu0fft2tWrVqtTh53f2r3j9+vU1ePBgzZs3T/Pnz//V1c8BAQGWKpcktWvXTmfOnNHatWudbT/++KO2b9+u2NhYZ5ufn5/eeustdenSRVdffbUOHDggSYqNjZXdbteePXtKxRkVFfWbv581a9Y4/3zmzBlt3LhR7dq1c8aelZUlY4yzz+eff67atWuradOmks4mZr169dKUKVP0xRdfKCAgQB9++GGZ5yrr+/mldu3a6fPPP7e0ff7557rsssvk7+9/UdcIAJWFbW6AS0hUVJQyMzN19dVXKzExUenp6Zo0aZJuuOEGNWvWTLfddpv8/Pz05Zdf6uuvv9bjjz/u/Oy9996rG264QSUlJRoxYoTL80RHR6ugoEAZGRnq1KmTgoOD1bp1aw0aNEijRo3SSy+9pNq1a+tvf/ubmjRpokGDBlk+7+/vr7fffltDhw7VNddco8zMTIWHh+vBBx/Un/70JzkcDvXu3Vt5eXn6/PPPFRoa+qsx/ZrZs2erdevWateunZ577jkdO3bMOY0+ZswYpaWl6f7779e4ceO0fft2paSkKDk5WX5+flq7dq0yMjLUv39/NWrUSGvXrtXhw4edCWZZ38/atWu1e/duhYSEqF69eqX6/PnPf1bXrl01bdo0DRkyRFlZWZo1a5ZefPHF33SdAFApvH0TJADXfrlAwpizq3Jbt25tfve735m8vDyTnp5uevbsaYKCgkxoaKjp1q2befnlly2fcTgcpnnz5ua6664r13n/+Mc/mvr16xtJztW+R48eNX/4wx9MWFiYCQoKMomJiea7775zfuaXi1tOnz5tbrnlFtOuXTuTm5trHA6HSUtLM23atDE1a9Y0DRs2NImJiWbFihXGmNILPYwx5osvvjCSTHZ2dplxnluksmDBAtOtWzcTEBBgYmNjzX/+8x9Lv8zMTNO1a1cTEBBgwsPDzV//+ldz+vRpY4wx3377rUlMTDQNGzY0drvdXHbZZeaFF15wfvaXv8H27dvN7373OxMUFOSMrazYFy9ebGJjY03NmjVNs2bNzNNPP22JqXnz5ua5556ztHXq1Mn5fQOAt9iMOW/OBUC1VVBQoCZNmmjevHm65ZZbvB2Ox+zevVsxMTH64osv1LlzZ2+HAwDVAlPMQDXncDh05MgRPfvss6pTp45uuukmb4cEAKjiSBCBam7Pnj2KiYlR06ZNNX/+fNWowV97AIBrTDEDAADAgm1uAAAAYEGCCAAAAAsSRAAAAFiQIAIAAMCCBBEAAAAWJIgAAACwIEEEAACABQkiAAAALP4/NKDRYP1y2UYAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAogAAAIjCAYAAACAkuqXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABaqElEQVR4nO3de1yUZf7/8feAMoACnkEUBQ95BC1RQystSbSTdjRrF8Wy37e0dNm2zQ6i1oa2ZViadlLLzbS0rO2AGSu6Fh7SzA5qappHUFNBMEGZ6/eHOdsdiIwNjM68no/H/Vjmmmuu+zPQ7n76XIfbZowxAgAAAH7l5+kAAAAAcH4hQQQAAIAFCSIAAAAsSBABAABgQYIIAAAACxJEAAAAWJAgAgAAwIIEEQAAABYkiAAAALAgQQRwQYmOjtbQoUM9HYYkaceOHbLZbHrmmWc8HYpTdna2bDabsrOzPR0KgAsYCSIAp71792rcuHFav359mffmzp2rjIyMaonjiy++0Lhx43TkyJFquZ8vWrNmjUaOHKkOHTqoVq1aatasmW677Tb98MMPng4NwHmABBGA0969ezV+/PjzIkEcP358uQni5s2b9corr1RLHN5s0qRJWrhwofr06aMpU6bonnvu0fLly3XJJZfo22+/9XR4ADyshqcDALxVUVGRatWq5ekwvI7dbvd0CF4hNTVVc+fOVUBAgLNt0KBBio2N1cSJE/Wvf/3Lg9EB8DQqiEAl7NmzR3fddZciIyNlt9sVExOje++9VyUlJZKk2bNny2azadmyZbrvvvvUqFEjNW3a1Pn5F198UR06dJDdbldkZKRGjBhRpjq2ZcsW3XzzzYqIiFBgYKCaNm2q22+/Xfn5+c4+S5Ys0WWXXaY6deqodu3aatOmjR555JGzxn/o0CE9+OCDio2NVe3atRUaGqr+/fvr66+/dvbJzs5W165dJUkpKSmy2Wyy2WyaPXu2evfurY8++kg//fSTsz06Otr52eLiYqWlpalVq1ay2+2KiorSQw89pOLiYkscNptNI0eO1KJFi9SxY0fZ7XZ16NBBmZmZzj7jxo3T3/72N0lSTEyM8347duyQVP4axB9//FG33nqr6tWrp+DgYF166aX66KOPLH1Or817++239Y9//ENNmzZVYGCg+vTpo61bt571d3g2L7/8slq2bCm73a6uXbtqzZo1Zfps2rRJt9xyi+rVq6fAwEDFx8frgw8+sPSpzN/qtN27d2vgwIGqVauWGjVqpL/85S9lfudn0qNHD0tyKEmtW7dWhw4dtHHjRhe+OQBvRAUROIu9e/eqW7duOnLkiO655x61bdtWe/bs0YIFC3Ts2DHL/8ned999atiwocaOHauioiJJpxKe8ePHKzExUffee682b96s6dOna82aNfr8889Vs2ZNlZSUKCkpScXFxbr//vsVERGhPXv26MMPP9SRI0cUFham7777Ttddd53i4uI0YcIE2e12bd26VZ9//vlZv8OPP/6oRYsW6dZbb1VMTIzy8vL00ksvqVevXvr+++8VGRmpdu3aacKECRo7dqzuueceXX755ZJOJRJNmjRRfn6+du/ereeee06SVLt2bUmSw+HQDTfcoBUrVuiee+5Ru3bt9M033+i5557TDz/8oEWLFlliWbFihd59913dd999CgkJ0fPPP6+bb75ZO3fuVP369XXTTTfphx9+0FtvvaXnnntODRo0kCQ1bNiw3O+Wl5enHj166NixY3rggQdUv359vf7667rhhhu0YMEC3XjjjZb+EydOlJ+fnx588EHl5+fr6aef1p133qlVq1ZV4p+G8s2dO1dHjx7V//t//082m01PP/20brrpJv3444+qWbOmJOm7775Tz5491aRJEz388MOqVauW3n77bQ0cOFALFy50xlmZv5Uk/fLLL+rTp4927typBx54QJGRkZozZ47+85//nPP3MMYoLy9PHTp0OOcxAHgJA6BCycnJxs/Pz6xZs6bMew6HwxhjzKxZs4wkc9lll5mTJ08639+/f78JCAgwffv2NaWlpc72qVOnGklm5syZxhhjvvrqKyPJvPPOO2eM47nnnjOSzIEDB1z+DsePH7fc3xhjtm/fbux2u5kwYYKzbc2aNUaSmTVrVpkxrr32WtO8efMy7XPmzDF+fn7mv//9r6V9xowZRpL5/PPPnW2STEBAgNm6dauz7euvvzaSzAsvvOBs++c//2kkme3bt5e5X/Pmzc2QIUOcr0ePHm0kWe5/9OhRExMTY6Kjo53fe+nSpUaSadeunSkuLnb2nTJlipFkvvnmmzL3Opvt27cbSaZ+/frm0KFDzvb333/fSDL//ve/nW19+vQxsbGx5vjx4842h8NhevToYVq3bu1sq+zfKiMjw0gyb7/9trOtqKjItGrVykgyS5cudfn7zJkzx0gyr732msufBeBdmGIGKuBwOLRo0SJdf/31io+PL/O+zWazvB4+fLj8/f2drz/77DOVlJRo9OjR8vPzs/QLDQ11ToOGhYVJkhYvXqxjx46VG0udOnUkSe+//74cDodL38NutzvvX1paqp9//tk5Rb1u3TqXxvq9d955R+3atVPbtm118OBB53XVVVdJkpYuXWrpn5iYqJYtWzpfx8XFKTQ0VD/++OM53f/jjz9Wt27ddNlllznbateurXvuuUc7duzQ999/b+mfkpJiqfqerpSe6/2lU2v36tate8YxDx06pP/85z+67bbbdPToUefv6Oeff1ZSUpK2bNmiPXv2SKr83+rjjz9W48aNdcsttzjbgoODdc8995zTd9i0aZNGjBihhIQEDRky5JzGAOA9SBCBChw4cEAFBQXq2LFjpfrHxMRYXv/000+SpDZt2ljaAwIC1KJFC+f7MTExSk1N1auvvqoGDRooKSlJ06ZNs6w/HDRokHr27Km7775b4eHhuv322/X2229bksXc3FzL9csvv0g6leg+99xzat26tex2uxo0aKCGDRtqw4YNlnuciy1btui7775Tw4YNLddFF10kSdq/f7+lf7NmzcqMUbduXR0+fPic7v/TTz+V+f1KUrt27ZzvV3T/04ndud6/MmNu3bpVxhg9/vjjZX5PaWlpkv73e6rs3+qnn35Sq1atyvxLSnm/i7PJzc3Vtddeq7CwMC1YsMDyLzkAfBNrEAE3CgoKOufPPvvssxo6dKjef/99ffrpp3rggQeUnp6ulStXqmnTpgoKCtLy5cu1dOlSffTRR8rMzNT8+fN11VVX6dNPP5W/v78aN25sGXPWrFkaOnSonnrqKT3++OMaNmyYnnjiCdWrV09+fn4aPXq0y9XI33M4HIqNjdXkyZPLfT8qKsry+kzJhzHmD8VRWVVx/7ONefp3/OCDDyopKancvq1atZKkKv1blSc/P1/9+/fXkSNH9N///te5xhGAbyNBBCrQsGFDhYaGnvO5cM2bN5d06uy+Fi1aONtLSkq0fft2JSYmWvrHxsYqNjZWjz32mL744gv17NlTM2bM0JNPPilJ8vPzU58+fdSnTx9NnjxZTz31lB599FEtXbpUiYmJWrJkiWW805sNFixYoCuvvFKvvfaa5f0jR444N4FIZafMf+tM77Vs2VJff/21+vTpU+HnXeHKOM2bN9fmzZvLtG/atMn5vqed/tvXrFmzzN/89yr7t2revLm+/fZbGWMsv6/yfhdncvz4cV1//fX64Ycf9Nlnn6l9+/aV/iwA78YUM1ABPz8/DRw4UP/+97/15Zdflnn/bFWnxMREBQQE6Pnnn7f0fe2115Sfn69rr71WklRQUKCTJ09aPhsbGys/Pz/nsSWHDh0qM37nzp0lydknMTHRcp2uKPr7+5eJ9Z133nGuezvt9LmN5R1QXatWrXKno2+77Tbt2bOn3MOrf/nlF+dubldUFMfvXXPNNVq9erVycnKcbUVFRXr55ZcVHR19XiQ9jRo1Uu/evfXSSy9p3759Zd4/cOCA8+fK/q2uueYa7d27VwsWLHC2HTt2TC+//HKlYiotLdWgQYOUk5Ojd955RwkJCa58JQBejgoicBZPPfWUPv30U/Xq1ct5jMu+ffv0zjvvaMWKFc7NI+Vp2LChxowZo/Hjx6tfv3664YYbtHnzZr344ovq2rWr/vSnP0mS/vOf/2jkyJG69dZbddFFF+nkyZOaM2eO/P39dfPNN0uSJkyYoOXLl+vaa69V8+bNtX//fr344otq2rSpZYNGea677jpNmDBBKSkp6tGjh7755hu9+eablqqmdKoaWKdOHc2YMUMhISGqVauWunfvrpiYGHXp0kXz589Xamqqunbtqtq1a+v666/Xn//8Z7399tv6v//7Py1dulQ9e/ZUaWmpNm3apLfffluLFy8ud4NPRbp06SJJevTRR3X77berZs2auv7668s9ePzhhx/WW2+9pf79++uBBx5QvXr19Prrr2v79u1auHChZXNQZWVnZ+vKK69UWlqaxo0b5/LnyzNt2jRddtllio2N1fDhw9WiRQvl5eUpJydHu3fvdp5zWNm/1fDhwzV16lQlJydr7dq1aty4sebMmaPg4OBKxfPXv/5VH3zwga6//nodOnSozMHYp//ZBOCjPLZ/GriA/PTTTyY5Odk0bNjQ2O1206JFCzNixAjncSmnj7kp7ygcY04da9O2bVtTs2ZNEx4ebu69915z+PBh5/s//vijGTZsmGnZsqUJDAw09erVM1deeaX57LPPnH2ysrLMgAEDTGRkpAkICDCRkZFm8ODB5ocffjhr/MePHzd//etfTePGjU1QUJDp2bOnycnJMb169TK9evWy9H3//fdN+/btTY0aNSxH3hQWFpo77rjD1KlTx0iyHHlTUlJiJk2aZDp06GDsdrupW7eu6dKlixk/frzJz8939pNkRowYUSa+3x9dY4wxTzzxhGnSpInx8/OzHHlTXt9t27aZW265xdSpU8cEBgaabt26mQ8//NDS5/QxN78/Suj0UTW/Pdrn3//+t5FkZsyYceZf6m8++89//rPMe5JMWlpamTiTk5NNRESEqVmzpmnSpIm57rrrzIIFC5x9XPlb/fTTT+aGG24wwcHBpkGDBmbUqFEmMzOzUsfc9OrVy0g64wXAt9mMqaaV4QBwgXjooYf01ltvaevWrTzaD4BPYg0iAPzO0qVL9fjjj5McAvBZVBABAABgQQURAAAAFiSIAAAAsCBBBAAAgAUJIgAAACwu6IOyHQ6H9u7dq5CQELc94gsAAFQtY4yOHj2qyMjIczrM/o86fvy4SkpKqmTsgIAABQYGVsnY1emCThD37t2rqKgoT4cBAADOwa5du9S0adNqvefx48cV07y2cveXVsn4ERER2r59+wWfJF7QCWJISIgk6dpFd6hmrQAPR4PtR+p7OgT8xuEf63o6BPwqpHnZZ1jDMxre/oOnQ4CkkzqhFfrY+f/j1amkpES5+0v109pohYa4t3pZcNSh5l12qKSkhATRk05PK9esFUCCeB7wL+FQ4fOJ3wX+P07exD/4uKdDwK9q2Gp6OgRIpx7oKHl0eVjtEJtqh7j3/g55z3K3CzpBBAAAOBelxqFSNz8qpNQ43DugB7GLGQAAABZUEAEAgM9xyMgh95YQ3T2eJ1FBBAAAgAUVRAAA4HMccsjdKwbdP6LnUEEEAACABRVEAADgc0qNUalx75pBd4/nSVQQAQAAYEEFEQAA+Bx2MVeMBBEAAPgch4xKSRDPiClmAAAAWFBBBAAAPocp5opRQQQAAIAFFUQAAOBzOOamYlQQAQAAYEEFEQAA+BzHr5e7x/QWVBABAABgQQURAAD4nNIqOAfR3eN5EgkiAADwOaXm1OXuMb0FU8wAAACwoIIIAAB8DptUKkYFEQAAABZUEAEAgM9xyKZS2dw+preggggAAAALKogAAMDnOMypy91jegsqiAAAALA4LxLEadOmKTo6WoGBgerevbtWr17t6ZAAAIAXK/11DaK7L2/h8QRx/vz5Sk1NVVpamtatW6dOnTopKSlJ+/fv93RoAADAS5EgVszjCeLkyZM1fPhwpaSkqH379poxY4aCg4M1c+ZMT4cGAADgkzy6SaWkpERr167VmDFjnG1+fn5KTExUTk5Omf7FxcUqLi52vi4oKKiWOAEAgHdxGJscxs3H3Lh5PE/yaAXx4MGDKi0tVXh4uKU9PDxcubm5Zfqnp6crLCzMeUVFRVVXqAAAAD7D41PMrhgzZozy8/Od165duzwdEgAAuACxBrFiHp1ibtCggfz9/ZWXl2dpz8vLU0RERJn+drtddru9usIDAADwSR6tIAYEBKhLly7KyspytjkcDmVlZSkhIcGDkQEAAG9WKr8qubyFx5+kkpqaqiFDhig+Pl7dunVTRkaGioqKlJKS4unQAAAAfJLHE8RBgwbpwIEDGjt2rHJzc9W5c2dlZmaW2bgCAADgLqYKdjEbL9rF7PEEUZJGjhypkSNHejoMAADgI6piU4k3bVLxnslyAAAAuMV5UUEEAACoTqXGT6XGvXWyUuPW4TyKCiIAAAAsqCACAACf45BNDjfXyRzynhIiFUQAAABYUEEEAAA+h13MFaOCCAAAAAsSRAAA4HNO72J293Uupk2bpujoaAUGBqp79+5avXr1Gfu+++67io+PV506dVSrVi117txZc+bMsfQZOnSobDab5erXr59LMTHFDAAAfM6pTSrunRI+l/Hmz5+v1NRUzZgxQ927d1dGRoaSkpK0efNmNWrUqEz/evXq6dFHH1Xbtm0VEBCgDz/8UCkpKWrUqJGSkpKc/fr166dZs2Y5X9vtdpfiooIIAADgRgUFBZaruLj4jH0nT56s4cOHKyUlRe3bt9eMGTMUHBysmTNnltu/d+/euvHGG9WuXTu1bNlSo0aNUlxcnFasWGHpZ7fbFRER4bzq1q3r0ncgQQQAAD7HIT+Vuvk6fWxOVFSUwsLCnFd6enq5MZSUlGjt2rVKTEx0tvn5+SkxMVE5OTln/Q7GGGVlZWnz5s264oorLO9lZ2erUaNGatOmje699179/PPPLv1+mGIGAABwo127dik0NNT5+kzTuwcPHlRpaanCw8Mt7eHh4dq0adMZx8/Pz1eTJk1UXFwsf39/vfjii7r66qud7/fr10833XSTYmJitG3bNj3yyCPq37+/cnJy5O/vX6nvQIIIAAB8TtU8au/UQdmhoaGWBNHdQkJCtH79ehUWFiorK0upqalq0aKFevfuLUm6/fbbnX1jY2MVFxenli1bKjs7W3369KnUPUgQAQAAPKBBgwby9/dXXl6epT0vL08RERFn/Jyfn59atWolSercubM2btyo9PR0Z4L4ey1atFCDBg20devWSieIrEEEAAA+x/HrmkF3X64ICAhQly5dlJWV9b+4HA5lZWUpISGh8t/F4ahwI8zu3bv1888/q3HjxpUekwoiAACAh6SmpmrIkCGKj49Xt27dlJGRoaKiIqWkpEiSkpOT1aRJE+dGl/T0dMXHx6tly5YqLi7Wxx9/rDlz5mj69OmSpMLCQo0fP14333yzIiIitG3bNj300ENq1aqV5RicsyFBBAAAPqfU2FRq3PyovXMYb9CgQTpw4IDGjh2r3Nxcde7cWZmZmc6NKzt37pSf3/8qk0VFRbrvvvu0e/duBQUFqW3btvrXv/6lQYMGSZL8/f21YcMGvf766zpy5IgiIyPVt29fPfHEEy6dhWgz5tcVlReggoIChYWFaeCSoapZK8DT4fi8bYcbeDoE/MahrfU8HQJ+FRpzxNMh4FeNBpx5Zyiqz0lzQtl6X/n5+VW6maM8p3OH2V91UnBI5Xb0Vtaxo6UaevHXHvle7sYaRAAAAFgwxQwAAHyOw/jJ4eZjbhwX7qRsGVQQAQAAYEEFEQAA+JzTj8dz75hUEAEAAOClqCACAACf49C5HUtztjG9BRVEAAAAWHhFBfHSsO0KrO0VX+WC1iCgyNMh4DeyVnMu5fmiQHU8HQJ+1cjTAeC8cS6PxqvMmN6CrAoAAPicUuOnUjcfc+Pu8TzJe74JAAAA3IIKIgAA8DkO2eSQuzepuHc8T6KCCAAAAAsqiAAAwOewBrFi3vNNAAAA4BZUEAEAgM+pmkfteU/dzXu+CQAAANyCCiIAAPA5DmOTw92P2nPzeJ5EBREAAAAWVBABAIDPcVTBGkQetQcAAHABcxg/Odx8LI27x/Mk7/kmAAAAcAsqiAAAwOeUyqZSNz8az93jeRIVRAAAAFhQQQQAAD6HNYgV855vAgAAALeggggAAHxOqdy/ZrDUraN5FhVEAAAAWFBBBAAAPoc1iBUjQQQAAD6n1Pip1M0JnbvH8yTv+SYAAABwCyqIAADA5xjZ5HDzJhXDQdkAAADwVlQQAQCAz2ENYsW855sAAADALaggAgAAn+MwNjmMe9cMuns8T6KCCAAAAAsqiAAAwOeUyk+lbq6TuXs8T/LoN1m+fLmuv/56RUZGymazadGiRZ4MBwAA+IjTU8zuvryFRxPEoqIiderUSdOmTfNkGAAAAPgNj04x9+/fX/379/dkCAAAwAc55CeHm+tk7h7Pky6oNYjFxcUqLi52vi4oKPBgNAAAAN7pgkp109PTFRYW5ryioqI8HRIAALgAlRpblVze4oJKEMeMGaP8/HzntWvXLk+HBAAA4HUuqClmu90uu93u6TAAAMAFjoOyK3ZBVRABAABQ9TxaQSwsLNTWrVudr7dv367169erXr16atasmQcjAwAA3swYPzmMe+tkxs3jeZJHE8Qvv/xSV155pfN1amqqJGnIkCGaPXu2h6ICAADerlQ2lcq9U8LuHs+TPJrq9u7dW8aYMhfJIQAA8BXTpk1TdHS0AgMD1b17d61evfqMfd99913Fx8erTp06qlWrljp37qw5c+ZY+hhjNHbsWDVu3FhBQUFKTEzUli1bXIrJe2qhAAAAleQwVfG4PdfjmD9/vlJTU5WWlqZ169apU6dOSkpK0v79+8vtX69ePT366KPKycnRhg0blJKSopSUFC1evNjZ5+mnn9bzzz+vGTNmaNWqVapVq5aSkpJ0/PjxSsdFgggAAOAhkydP1vDhw5WSkqL27dtrxowZCg4O1syZM8vt37t3b914441q166dWrZsqVGjRikuLk4rVqyQdKp6mJGRoccee0wDBgxQXFyc3njjDe3du1eLFi2qdFwkiAAAwOc4ft2k4u5LOvWkt99ev30K3G+VlJRo7dq1SkxMdLb5+fkpMTFROTk5Z/0OxhhlZWVp8+bNuuKKKySd2vCbm5trGTMsLEzdu3ev1JjOOCrdEwAAAGcVFRVlefJbenp6uf0OHjyo0tJShYeHW9rDw8OVm5t7xvHz8/NVu3ZtBQQE6Nprr9ULL7ygq6++WpKcn3N1zN+7oA7KBgAAcAeHbHK4edfx6fF27dql0NBQZ7u7H/IREhKi9evXq7CwUFlZWUpNTVWLFi3Uu3dvt92DBBEAAMCNQkNDLQnimTRo0ED+/v7Ky8uztOfl5SkiIuKMn/Pz81OrVq0kSZ07d9bGjRuVnp6u3r17Oz+Xl5enxo0bW8bs3Llzpb8DU8wAAMDnlBpblVyuCAgIUJcuXZSVleVsczgcysrKUkJCQqXHcTgcznWOMTExioiIsIxZUFCgVatWuTQmFUQAAOBzHFXwJJVzGS81NVVDhgxRfHy8unXrpoyMDBUVFSklJUWSlJycrCZNmjjXMaanpys+Pl4tW7ZUcXGxPv74Y82ZM0fTp0+XJNlsNo0ePVpPPvmkWrdurZiYGD3++OOKjIzUwIEDKx0XCSIAAICHDBo0SAcOHNDYsWOVm5urzp07KzMz07nJZOfOnfLz+1/iWVRUpPvuu0+7d+9WUFCQ2rZtq3/9618aNGiQs89DDz2koqIi3XPPPTpy5Iguu+wyZWZmKjAwsNJx2Ywx53Cs4/mhoKBAYWFhmri6lwJrk+t62uZjZ14vgeqX9e8ung4Bvyqp5/B0CPhVq9ErPR0CJJ00J5St95Wfn1+ptXrudDp3uC3rzwqoFeDWsUuKSvR2nzke+V7uxhpEAAAAWFB2AwAAPsdUwTE3xs3jeRIVRAAAAFhQQQQAAD7HYWxyuHgsTWXG9BZUEAEAAGBBBREAAPic8+UcxPMVCSIAAPA5TDFXzHtSXQAAALgFFUQAAOBzHFVwzI27x/MkKogAAACwoIIIAAB8DmsQK0YFEQAAABZUEAEAgM+hglgxKogAAACwoIIIAAB8DhXEinlFgjhtXW/5BQV6Ogwc9Yp/nLxGo0v3ezoE/OryiG2eDgG/2uDpAHDeIEGsGFPMAAAAsKDkAwAAfI6R+w+2Nm4dzbOoIAIAAMCCCiIAAPA5rEGsGBVEAAAAWFBBBAAAPocKYsWoIAIAAMCCCiIAAPA5VBArRoIIAAB8DglixZhiBgAAgAUVRAAA4HOMscm4ueLn7vE8iQoiAAAALKggAgAAn+OQze2P2nP3eJ5EBREAAAAWVBABAIDPYRdzxaggAgAAwIIKIgAA8DnsYq4YFUQAAABYUEEEAAA+hzWIFSNBBAAAPocp5ooxxQwAAAALKogAAMDnmCqYYqaCCAAAAK9FBREAAPgcI8kY94/pLaggAgAAwIIKIgAA8DkO2WSTm4+5cfN4nkQFEQAAABZUEAEAgM/hHMSKkSACAACf4zA22XiSyhmdU4KYlZWlrKws7d+/Xw6Hw/LezJkzKz1Oenq63n33XW3atElBQUHq0aOHJk2apDZt2pxLWAAAAHADl9cgjh8/Xn379lVWVpYOHjyow4cPWy5XLFu2TCNGjNDKlSu1ZMkSnThxQn379lVRUZGrYQEAAFSaMVVzeQuXK4gzZszQ7Nmz9ec///kP3zwzM9Pyevbs2WrUqJHWrl2rK6644g+PDwAAANe5XEEsKSlRjx49qiIW5efnS5Lq1atX7vvFxcUqKCiwXAAAAK46vUnF3de5mDZtmqKjoxUYGKju3btr9erVZ+z7yiuv6PLLL1fdunVVt25dJSYmluk/dOhQ2Ww2y9WvXz+XYnI5Qbz77rs1d+5cVz92Vg6HQ6NHj1bPnj3VsWPHcvukp6crLCzMeUVFRbk9DgAAgOoyf/58paamKi0tTevWrVOnTp2UlJSk/fv3l9s/OztbgwcP1tKlS5WTk6OoqCj17dtXe/bssfTr16+f9u3b57zeeustl+JyeYr5+PHjevnll/XZZ58pLi5ONWvWtLw/efJkV4eUJI0YMULffvutVqxYccY+Y8aMUWpqqvN1QUEBSSIAAHDZ+XLMzeTJkzV8+HClpKRIOrWU76OPPtLMmTP18MMPl+n/5ptvWl6/+uqrWrhwobKyspScnOxst9vtioiIcDme01xOEDds2KDOnTtLkr799lvLezbbuf2iR44cqQ8//FDLly9X06ZNz9jPbrfLbref0z0AAACqw++XwJ0pfykpKdHatWs1ZswYZ5ufn58SExOVk5NTqXsdO3ZMJ06cKLM8Lzs7W40aNVLdunV11VVX6cknn1T9+vUr/R1cThCXLl3q6kfOyBij+++/X++9956ys7MVExPjtrEBAADOpCrPQfz97GZaWprGjRtXpv/BgwdVWlqq8PBwS3t4eLg2bdpUqXv+/e9/V2RkpBITE51t/fr100033aSYmBht27ZNjzzyiPr376+cnBz5+/tXatw/dFD27t27JanCql9FRowYoblz5+r9999XSEiIcnNzJUlhYWEKCgr6I6EBAACcUVUcS3N6vF27dik0NNTZXlWznxMnTtS8efOUnZ2twMBAZ/vtt9/u/Dk2NlZxcXFq2bKlsrOz1adPn0qN7fImFYfDoQkTJigsLEzNmzdX8+bNVadOHT3xxBNlDs0+m+nTpys/P1+9e/dW48aNndf8+fNdDQsAAOC8EBoaarnOlCA2aNBA/v7+ysvLs7Tn5eWddf3gM888o4kTJ+rTTz9VXFxchX1btGihBg0aaOvWrZX+Di5XEB999FG99tprmjhxonr27ClJWrFihcaNG6fjx4/rH//4R6XHMt50oiQAALhgnKogunuTimv9AwIC1KVLF2VlZWngwIGSThXisrKyNHLkyDN+7umnn9Y//vEPLV68WPHx8We9z+7du/Xzzz+rcePGlY7N5QTx9ddf16uvvqobbrjB2RYXF6cmTZrovvvucylBBAAA8GWpqakaMmSI4uPj1a1bN2VkZKioqMi5qzk5OVlNmjRRenq6JGnSpEkaO3as5s6dq+joaOfyvNq1a6t27doqLCzU+PHjdfPNNysiIkLbtm3TQw89pFatWikpKanScbmcIB46dEht27Yt0962bVsdOnTI1eEAAACq3flyzM2gQYN04MABjR07Vrm5uercubMyMzOdG1d27twpP7//rQicPn26SkpKdMstt1jGOb0Rxt/fXxs2bNDrr7+uI0eOKDIyUn379tUTTzzh0lpIlxPETp06aerUqXr++ect7VOnTlWnTp1cHQ4AAMCnjRw58oxTytnZ2ZbXO3bsqHCsoKAgLV68+A/H5HKC+PTTT+vaa6/VZ599poSEBElSTk6Odu3apY8//vgPBwQAAFDVzK+Xu8f0Fi7vYu7Vq5d++OEH3XjjjTpy5IiOHDmim266SZs3b9bll19eFTECAACgGp3TOYiRkZFsRgEAABes82UN4vmqUgnihg0b1LFjR/n5+WnDhg0V9j3bWTwAAAAexxxzhSqVIHbu3Fm5ublq1KiROnfuLJvNVu4ZhjabTaWlpW4PEgAAANWnUgni9u3b1bBhQ+fPAAAAF7QqmGKWr00xN2/e3PnzTz/9pB49eqhGDetHT548qS+++MLSFwAAABcel3cxX3nlleUeiJ2fn68rr7zSLUEBAABUpVOP2nP/5S1cThCNMbLZypZQf/75Z9WqVcstQQEAAMBzKn3MzU033STp1EaUoUOHWh7XUlpaqg0bNqhHjx7ujxAAAMDNOOamYpVOEMPCwiSdqiCGhIQoKCjI+V5AQIAuvfRSDR8+3P0RAgAAoFpVOkGcNWuWJCk6OloPPvgg08kAAODCZWzu33XsixXE09LS0qoiDgAAgGpTFZtKvGmTSqUSxEsuuURZWVmqW7euLr744nI3qZy2bt06twUHAACA6lepBHHAgAHOTSkDBw6syngAAACqHo/aq1ClEsTfTiszxQwAAODdXD4HcdeuXdq9e7fz9erVqzV69Gi9/PLLbg0MAACgqpw+5sbdl7dwOUG84447tHTpUklSbm6uEhMTtXr1aj366KOaMGGC2wMEAABA9XI5Qfz222/VrVs3SdLbb7+t2NhYffHFF3rzzTc1e/Zsd8cHAABQNYybLy/icoJ44sQJ54aVzz77TDfccIMkqW3bttq3b597owMAAEC1czlB7NChg2bMmKH//ve/WrJkifr16ydJ2rt3r+rXr+/2AAEAANyNNYgVczlBnDRpkl566SX17t1bgwcPVqdOnSRJH3zwgXPqGQAA4Lzm7ullL5tmdvlJKr1799bBgwdVUFCgunXrOtvvueceBQcHuzW4yooMP6wateweuTf+Z9eJBp4OAb/RILjI0yHgV+9t6uTpEPCrllrv6RCAC4LLCaIk+fv76+TJk1qxYoUkqU2bNoqOjnZnXAAAAFXI9uvl7jG9g8tTzEVFRRo2bJgaN26sK664QldccYUiIyN111136dixY1URIwAAAKqRywliamqqli1bpn//+986cuSIjhw5ovfff1/Lli3TX//616qIEQAAwL1Yg1ghl6eYFy5cqAULFqh3797OtmuuuUZBQUG67bbbNH36dHfGBwAAgGrmcoJ47NgxhYeHl2lv1KgRU8wAAODCUBUVPy+qILo8xZyQkKC0tDQdP37c2fbLL79o/PjxSkhIcGtwAAAAqH4uVxCnTJmipKQkNW3a1HkG4tdff63AwEAtXrzY7QECAAC4nbGdutw9ppdwOUHs2LGjtmzZojfffFObNm2SJA0ePFh33nmngoKC3B4gAACAuxlz6nL3mN7inM5BDA4O1vDhw90dCwAAAM4D55Qgbt68WS+88II2btwoSWrXrp1Gjhyptm3bujU4AACAKsEmlQq5vEll4cKF6tixo9auXatOnTqpU6dOWrdunWJjY7Vw4cKqiBEAAADVyOUK4kMPPaQxY8ZowoQJlva0tDQ99NBDuvnmm90WHAAAQJVgk0qFXK4g7tu3T8nJyWXa//SnP2nfvn1uCQoAAACe43KC2Lt3b/33v/8t075ixQpdfvnlbgkKAACgKtlM1VzewuUp5htuuEF///vftXbtWl166aWSpJUrV+qdd97R+PHj9cEHH1j6AgAA4MLicoJ43333SZJefPFFvfjii+W+J0k2m02lpaV/MDwAAIAqwC7mCrmcIDocjqqIAwAAoPqwSaVCLq9BBAAAgHc7p4OyAQAALmhMMVeICiIAAAAsqCACAADfQwWxQlQQAQAAYHFOFUSHw6GtW7dq//79ZXY1X3HFFW4JDAAAoMpQQayQywniypUrdccdd+inn36SMdbfBGcfAgAAXPhcThD/7//+T/Hx8froo4/UuHFj2Wzec+YPAADwEZyDWCGX1yBu2bJFTz31lNq1a6c6deooLCzMcgEAAKDypk2bpujoaAUGBqp79+5avXr1Gfu+8soruvzyy1W3bl3VrVtXiYmJZfobYzR27Fg1btxYQUFBSkxM1JYtW1yKyeUEsXv37tq6daurHwMAADhv2EzVXK6aP3++UlNTlZaWpnXr1qlTp05KSkrS/v37y+2fnZ2twYMHa+nSpcrJyVFUVJT69u2rPXv2OPs8/fTTev755zVjxgytWrVKtWrVUlJSko4fP17puFyeYr7//vv117/+Vbm5uYqNjVXNmjUt78fFxbk6JAAAQPU6TzapTJ48WcOHD1dKSookacaMGfroo480c+ZMPfzww2X6v/nmm5bXr776qhYuXKisrCwlJyfLGKOMjAw99thjGjBggCTpjTfeUHh4uBYtWqTbb7+9UnG5nCDefPPNkqRhw4Y522w2m4wxbFIBAAA+r6CgwPLabrfLbreX6VdSUqK1a9dqzJgxzjY/Pz8lJiYqJyenUvc6duyYTpw4oXr16kmStm/frtzcXCUmJjr7hIWFqXv37srJyal0gujyFPP27dvLXD/++KPzP10xffp0xcXFKTQ0VKGhoUpISNAnn3ziakgAAADnjaioKMv+jPT09HL7HTx4UKWlpQoPD7e0h4eHKzc3t1L3+vvf/67IyEhnQnj6c39kTOkcKojNmzd39SNn1LRpU02cOFGtW7eWMUavv/66BgwYoK+++kodOnRw230AAACqy65duxQaGup8XV710B0mTpyoefPmKTs7W4GBgW4d+5yepDJnzhz17NlTkZGR+umnnyRJGRkZev/9910a5/rrr9c111yj1q1b66KLLtI//vEP1a5dWytXrjyXsAAAACrFpirYpPLr2KdnRk9fZ0oQGzRoIH9/f+Xl5Vna8/LyFBERUWH8zzzzjCZOnKhPP/3Usv/j9OfOZczfcjlBnD59ulJTU3XNNdfoyJEjzjWHderUUUZGhqvDOZWWlmrevHkqKipSQkJCuX2Ki4tVUFBguQAAAC5EAQEB6tKli7KyspxtDodDWVlZZ8yFpFO7lJ944gllZmYqPj7e8l5MTIwiIiIsYxYUFGjVqlUVjvl7LieIL7zwgl555RU9+uij8vf3d7bHx8frm2++cXU4ffPNN6pdu7bsdrv+7//+T++9957at29fbt/09HTLnH5UVJTL9wMAAHAelO3uy0Wpqal65ZVX9Prrr2vjxo269957VVRU5NzVnJycbNnEMmnSJD3++OOaOXOmoqOjlZubq9zcXBUWFko6tXF49OjRevLJJ/XBBx/om2++UXJysiIjIzVw4MBKx+XyGsTt27fr4osvLtNut9tVVFTk6nBq06aN1q9fr/z8fC1YsEBDhgzRsmXLyk0Sx4wZo9TUVOfrgoICkkQAAHDBGjRokA4cOKCxY8cqNzdXnTt3VmZmpnOTyc6dO+Xn97963vTp01VSUqJbbrnFMk5aWprGjRsnSXrooYdUVFSke+65R0eOHNFll12mzMxMl9YpupwgxsTEaP369WU2q2RmZqpdu3auDqeAgAC1atVKktSlSxetWbNGU6ZM0UsvvVSm75m2iQMAALjkPDkHUZJGjhypkSNHlvtedna25fWOHTvOOp7NZtOECRM0YcKEcwtI55AgpqamasSIETp+/LiMMVq9erXeeustpaen69VXXz3nQE5zOBwqLi7+w+MAAACc0XmUIJ6PXE4Q7777bgUFBemxxx7TsWPHdMcddygyMlJTpkyp9OGLp40ZM0b9+/dXs2bNdPToUc2dO1fZ2dlavHixq2EBAADATVxOEAsKCnTnnXfqzjvv1LFjx1RYWKhGjRpJkrZu3eqcLq6M/fv3Kzk5Wfv27VNYWJji4uK0ePFiXX311a6GBQAAUGnn+uzks43pLVxOEK+99lp99tlnstvtCg4OVnBwsCRp8+bN6tOnj3bv3l3psV577TVXbw8AAIAq5vIxN7Vr19aNN96okydPOts2btyo3r17O5/TDAAAcF4zVXR5CZcTxHfffVf5+fm68847ZYzRt99+q969e2vw4MGaMmVKVcQIAACAauRyghgUFKSPPvpImzdv1m233aY+ffooOTlZkydPror4AAAA3I8KYoUqtQbx94+08/Pz0/z583X11Vfr5ptv1uOPP+7s89uHUwMAAODCU6kEsU6dOrLZyj4+xhijGTNm6KWXXpIxRjabzflsZgAAgPMVu5grVqkEcenSpVUdBwAAQPU5x2cnn3VML1GpBLFXr15VHQcAAADOEy6fgyhJR44c0WuvvaaNGzdKkjp06KBhw4YpLCzMrcEBAABUCR61VyGXdzF/+eWXatmypZ577jkdOnRIhw4d0uTJk9WyZUutW7euKmIEAABANXK5gviXv/xFN9xwg1555RXVqHHq4ydPntTdd9+t0aNHa/ny5W4PEgAAwJ3YpFIxlxPEL7/80pIcSlKNGjX00EMPKT4+3q3BAQAAoPq5PMUcGhqqnTt3lmnftWuXQkJC3BIUAABAleKg7Aq5nCAOGjRId911l+bPn69du3Zp165dmjdvnu6++24NHjy4KmIEAABANXJ5ivmZZ56RzWZTcnKyTp48KUmqWbOm7r33Xk2cONHtAQIAALhdFaxB9KYKossJYkBAgKZMmaL09HRt27ZNktSyZUsFBwe7PTgAAIAqwTE3FXJ5innYsGE6evSogoODFRsbq9jYWAUHB6uoqEjDhg2rihgBAABQjVxOEF9//XX98ssvZdp/+eUXvfHGG24JCgAAoEqxSaVClZ5iLigokDFGxhgdPXpUgYGBzvdKS0v18ccfq1GjRlUSJAAAAKpPpRPEOnXqyGazyWaz6aKLLirzvs1m0/jx490aHAAAQFXgoOyKVTpBXLp0qYwxuuqqq7Rw4ULVq1fP+V5AQICaN2+uyMjIKgkSAAAA1afSCWKvXr0kSdu3b1ezZs1ks9mqLCgAAAB4jsvH3DRv3rwq4gAAAMB5wuUEEQAA4ILHOYgVIkEEAAA+h00qFXP5HEQAAAB4N5criGlpaRo2bNh5tRYxxF6smnYvStsvUMH1jnk6BPzGlryGng4BAM5vpA5n5HIF8f3331fLli3Vp08fzZ07V8XFxVURFwAAADzE5QRx/fr1WrNmjTp06KBRo0YpIiJC9957r9asWVMV8QEAALgfj9qr0DmtQbz44ov1/PPPa+/evXrttde0e/du9ezZU3FxcZoyZYry8/PdHScAAACqyR/apGKM0YkTJ1RSUiJjjOrWraupU6cqKipK8+fPd1eMAAAAbnV6F7O7L29xTgni2rVrNXLkSDVu3Fh/+ctfdPHFF2vjxo1atmyZtmzZon/84x964IEH3B0rAAAAqoHLu5hjY2O1adMm9e3bV6+99pquv/56+fv7W/oMHjxYo0aNcluQAAAAbsVB2RVyOUG87bbbNGzYMDVp0uSMfRo0aCCHw/GHAgMAAKgqHJRdMZemmE+cOKHZs2eroKCgquIBAACAh7lUQaxZs6aOHz9eVbEAAABUD6aYK+TyJpURI0Zo0qRJOnnyZFXEAwAAAA9zeQ3imjVrlJWVpU8//VSxsbGqVauW5f13333XbcEBAABUCSqIFXI5QaxTp45uvvnmqogFAAAA5wGXE8RZs2ZVRRwAAADVhl3MFTung7JPnjypzz77TC+99JKOHj0qSdq7d68KCwvdGhwAAACqn8sVxJ9++kn9+vXTzp07VVxcrKuvvlohISGaNGmSiouLNWPGjKqIEwAAwH1Yg1ghlyuIo0aNUnx8vA4fPqygoCBn+4033qisrCy3BgcAAFAlTBVdXsLlCuJ///tfffHFFwoICLC0R0dHa8+ePW4LDAAAAJ7hcoLocDhUWlpapn337t0KCQlxS1AAAABViU0qFXN5irlv377KyMhwvrbZbCosLFRaWpquueYad8YGAADg9aZNm6bo6GgFBgaqe/fuWr169Rn7fvfdd7r55psVHR0tm81myclOGzdunGw2m+Vq27atSzG5nCA+++yz+vzzz9W+fXsdP35cd9xxh3N6edKkSa4OBwAAUP3OkzWI8+fPV2pqqtLS0rRu3Tp16tRJSUlJ2r9/f7n9jx07phYtWmjixImKiIg447gdOnTQvn37nNeKFStcisvlKeamTZvq66+/1rx587RhwwYVFhbqrrvu0p133mnZtAIAAOCLCgoKLK/tdrvsdnu5fSdPnqzhw4crJSVFkjRjxgx99NFHmjlzph5++OEy/bt27aquXbtKUrnvn1ajRo0KE8izcTlBPH3TP/3pT+d8UwAAAE+qyjWIUVFRlva0tDSNGzeuTP+SkhKtXbtWY8aMcbb5+fkpMTFROTk5fyiWLVu2KDIyUoGBgUpISFB6erqaNWtW6c+7nCC+8cYbFb6fnJzs6pAAAABeY9euXQoNDXW+PlP18ODBgyotLVV4eLilPTw8XJs2bTrn+3fv3l2zZ89WmzZttG/fPo0fP16XX365vv3220pvKHY5QRw1apTl9YkTJ3Ts2DEFBAQoODiYBBEAAJz/qvCg7NDQUEuCWN369+/v/DkuLk7du3dX8+bN9fbbb+uuu+6q1Bgub1I5fPiw5SosLNTmzZt12WWX6a233nJ1OAAAgOp3HmxSadCggfz9/ZWXl2dpz8vL+0PrB3+vTp06uuiii7R169ZKf+acnsX8e61bt9bEiRPLVBcBAABQvoCAAHXp0sXyJDqHw6GsrCwlJCS47T6FhYXatm2bGjduXOnPnNMmlXIHqlFDe/fudddwAAAAVcb26+XuMV2VmpqqIUOGKD4+Xt26dVNGRoaKioqcu5qTk5PVpEkTpaenSzq1seX77793/rxnzx6tX79etWvXVqtWrSRJDz74oK6//no1b95ce/fuVVpamvz9/TV48OBKx+VygvjBBx9YXhtjtG/fPk2dOlU9e/Z0dTgAAACfNWjQIB04cEBjx45Vbm6uOnfurMzMTOfGlZ07d8rP738Tvnv37tXFF1/sfP3MM8/omWeeUa9evZSdnS3p1NPtBg8erJ9//lkNGzbUZZddppUrV6phw4aVjsvlBHHgwIGW1zabTQ0bNtRVV12lZ5991tXhAAAAql8VblJx1ciRIzVy5Mhy3zud9J0WHR0tYyq+0bx5884tkN9weQ2iw+GwXKWlpcrNzdXcuXNdmtv+vYkTJ8pms2n06NHnPAYAAAD+uHNeg3jw4EEFBAS4ZRv3mjVr9NJLLykuLu4PjwUAAHA2VXlQtjdwqYJ45MgRjRgxQg0aNFB4eLjq1q2riIgIjRkzRseOHTunAAoLC3XnnXfqlVdeUd26dc9pDAAAALhPpSuIhw4dUkJCgvbs2aM777xT7dq1kyR9//33euGFF7RkyRKtWLFCGzZs0MqVK/XAAw9UatwRI0bo2muvVWJiop588skK+xYXF6u4uNj5+vfPOgQAAKiU82gN4vmo0gnihAkTFBAQoG3btpV5JMyECRPUt29f/fnPf9ann36q559/vlJjzps3T+vWrdOaNWsq1T89PV3jx4+vbMgAAABn5kUJnbtVeop50aJFeuaZZ8okh5IUERGhp59+WgsXLnSe53M2u3bt0qhRo/Tmm28qMDCwUjGMGTNG+fn5zmvXrl2VDR8AAACVVOkK4r59+9ShQ4czvt+xY0f5+fkpLS2tUuOtXbtW+/fv1yWXXOJsKy0t1fLlyzV16lQVFxfL39/f8hm73X7GB14DAABUFptUKlbpBLFBgwbasWOHmjZtWu7727dvV6NGjSp94z59+uibb76xtKWkpKht27b6+9//XiY5BAAAQPWodIKYlJSkRx99VEuWLFFAQIDlveLiYj3++OPq169fpW8cEhKijh07Wtpq1aql+vXrl2kHAABwKzapVMilTSrx8fFq3bq1RowYobZt28oYo40bN+rFF19UcXGx3njjjaqMFQAAANWg0gli06ZNlZOTo/vuu09jxoxxPubFZrPp6quv1tSpU9WsWbM/FMzvHycDAABQFViDWDGXnqQSExOjTz75RIcPH9aWLVskSa1atVK9evWqJDgAAABUv3N61F7dunXVrVs3d8cCAABQPViDWCGXHrUHAAAA73dOFUQAAIALGWsQK0aCCAAAfA9TzBViihkAAAAWVBABAIDvoYJYISqIAAAAsKCCCAAAfA6bVCpGBREAAAAWVBABAIDvYQ1ihaggAgAAwIIKIgAA8Dk2Y2Qz7i35uXs8TyJBBAAAvocp5goxxQwAAAALKogAAMDncMxNxaggAgAAwIIKIgAA8D2sQawQFUQAAABYUEEEAAA+hzWIFaOCCAAAAAsqiAAAwPewBrFCJIgAAMDnMMVcMaaYAQAAYEEFEQAA+B6mmCtEBREAAAAWXlFB3H6wnvyPBXo6DJ8XUafA0yHgN/LyQzwdAgCc17xpzaC7UUEEAACAhVdUEAEAAFxizKnL3WN6CSqIAAAAsKCCCAAAfA7nIFaMBBEAAPgejrmpEFPMAAAAsKCCCAAAfI7Ncepy95jeggoiAAAALKggAgAA38MaxApRQQQAAPCgadOmKTo6WoGBgerevbtWr159xr7fffedbr75ZkVHR8tmsykjI+MPj1keEkQAAOBzTh9z4+7LVfPnz1dqaqrS0tK0bt06derUSUlJSdq/f3+5/Y8dO6YWLVpo4sSJioiIcMuY5SFBBAAA8JDJkydr+PDhSklJUfv27TVjxgwFBwdr5syZ5fbv2rWr/vnPf+r222+X3W53y5jlIUEEAAC+5/Sj9tx9SSooKLBcxcXF5YZQUlKitWvXKjEx0dnm5+enxMRE5eTknNPXcteYJIgAAMDnVOUUc1RUlMLCwpxXenp6uTEcPHhQpaWlCg8Pt7SHh4crNzf3nL6Xu8ZkFzMAAIAb7dq1S6Ghoc7XZ5oKPp+RIAIAAN9ThcfchIaGWhLEM2nQoIH8/f2Vl5dnac/LyzvjBpTqGpMpZgAAAA8ICAhQly5dlJWV5WxzOBzKyspSQkKCR8ekgggAAHzOuR5Lc7YxXZWamqohQ4YoPj5e3bp1U0ZGhoqKipSSkiJJSk5OVpMmTZzrGEtKSvT99987f96zZ4/Wr1+v2rVrq1WrVpUaszJIEAEAADxk0KBBOnDggMaOHavc3Fx17txZmZmZzk0mO3fulJ/f/yZ89+7dq4svvtj5+plnntEzzzyjXr16KTs7u1JjVgYJIgAA8D2/OZbGrWOeg5EjR2rkyJHlvnc66TstOjpaphL3qWjMymANIgAAACyoIAIAAJ9zvqxBPF+RIAIAAN9ThcfceAOmmAEAAGBBBREAAPgcppgrRgURAAAAFlQQAQCA73GYU5e7x/QSVBABAABg4dEEcdy4cbLZbJarbdu2ngwJAAD4AlNFl5fw+BRzhw4d9Nlnnzlf16jh8ZAAAAB8msezsRo1aigiIsLTYQAAAB9iUxXsYnbvcB7l8TWIW7ZsUWRkpFq0aKE777xTO3fuPGPf4uJiFRQUWC4AAACXnX4Ws7svL+HRBLF79+6aPXu2MjMzNX36dG3fvl2XX365jh49Wm7/9PR0hYWFOa+oqKhqjhgAAMD7eTRB7N+/v2699VbFxcUpKSlJH3/8sY4cOaK333673P5jxoxRfn6+89q1a1c1RwwAALzB6YOy3X15C4+vQfytOnXq6KKLLtLWrVvLfd9ut8tut1dzVAAAAL7F42sQf6uwsFDbtm1T48aNPR0KAADwZhxzUyGPJogPPvigli1bph07duiLL77QjTfeKH9/fw0ePNiTYQEAAPg0j04x7969W4MHD9bPP/+shg0b6rLLLtPKlSvVsGFDT4YFAAC8nM0Y2dy869jd43mSRxPEefPmefL2AAAAKMd5tUkFAACgWjh+vdw9ppcgQQQAAD6HKeaKnVe7mAEAAOB5VBABAIDvqYpjabyngEgFEQAAAFZUEAEAgO8x5tTl7jG9BBVEAAAAWFBBBAAAPsdmTl3uHtNbUEEEAACABRVEAADge1iDWCEqiAAAALCggggAAHyOzXHqcveY3oIEEQAA+B6mmCvEFDMAAAAsqCACAADfw6P2KkQFEQAAABZUEAEAgM+xGSObm9cMuns8T6KCCAAAAAsqiAAAwPewi7lCVBABAABgQQURAAD4HiPJ3Qdbe08BkQQRAAD4HjapVIwpZgAAAFhQQQQAAL7HqAo2qbh3OE+igggAAAALKogAAMD3cMxNhbwiQfzy0vkKDaEY6mk3bOnn6RDwGwdXNfF0CPiVCfZ0BADgGq9IEAEAAFzikGSrgjG9BGU3AAAAWJAgAgAAn3P6HER3X+di2rRpio6OVmBgoLp3767Vq1dX2P+dd95R27ZtFRgYqNjYWH388ceW94cOHSqbzWa5+vVzbRkYCSIAAPA9pzepuPty0fz585Wamqq0tDStW7dOnTp1UlJSkvbv319u/y+++EKDBw/WXXfdpa+++koDBw7UwIED9e2331r69evXT/v27XNeb731lktxkSACAAB4yOTJkzV8+HClpKSoffv2mjFjhoKDgzVz5sxy+0+ZMkX9+vXT3/72N7Vr105PPPGELrnkEk2dOtXSz263KyIiwnnVrVvXpbhIEAEAgO+pwgpiQUGB5SouLi43hJKSEq1du1aJiYnONj8/PyUmJionJ6fcz+Tk5Fj6S1JSUlKZ/tnZ2WrUqJHatGmje++9Vz///LNLvx4SRAAAADeKiopSWFiY80pPTy+338GDB1VaWqrw8HBLe3h4uHJzc8v9TG5u7ln79+vXT2+88YaysrI0adIkLVu2TP3791dpaWmlvwPH3AAAAN9ThQdl79q1S6Ghoc5mu93u3vucxe233+78OTY2VnFxcWrZsqWys7PVp0+fSo1BBREAAMCNQkNDLdeZEsQGDRrI399feXl5lva8vDxFRESU+5mIiAiX+ktSixYt1KBBA23durXS34EEEQAA+B5HFV0uCAgIUJcuXZSVlfW/sBwOZWVlKSEhodzPJCQkWPpL0pIlS87YX5J2796tn3/+WY0bN650bCSIAAAAHpKamqpXXnlFr7/+ujZu3Kh7771XRUVFSklJkSQlJydrzJgxzv6jRo1SZmamnn32WW3atEnjxo3Tl19+qZEjR0qSCgsL9be//U0rV67Ujh07lJWVpQEDBqhVq1ZKSkqqdFysQQQAAD7njxxsXdGYrho0aJAOHDigsWPHKjc3V507d1ZmZqZzI8rOnTvl5/e/el6PHj00d+5cPfbYY3rkkUfUunVrLVq0SB07dpQk+fv7a8OGDXr99dd15MgRRUZGqm/fvnriiSdcWgtJgggAAHxPFW5ScdXIkSOdFcDfy87OLtN266236tZbby23f1BQkBYvXnxOcfwWU8wAAACwoIIIAAB8j8NINjdXEB1uHs+DqCACAADAggoiAADwPefRGsTzERVEAAAAWFBBBAAAPqgKKoiigggAAAAvRQURAAD4HtYgVogEEQAA+B6HkdunhDnmBgAAAN6KCiIAAPA9xnHqcveYXoIKIgAAACyoIAIAAN/DJpUKUUEEAACABRVEAADge9jFXCGPVxD37NmjP/3pT6pfv76CgoIUGxurL7/80tNhAQAA+CyPVhAPHz6snj176sorr9Qnn3yihg0basuWLapbt64nwwIAAN6ONYgV8miCOGnSJEVFRWnWrFnOtpiYGA9GBAAAfIJRFSSI7h3Okzw6xfzBBx8oPj5et956qxo1aqSLL75Yr7zyyhn7FxcXq6CgwHIBAADAvTyaIP7444+aPn26WrdurcWLF+vee+/VAw88oNdff73c/unp6QoLC3NeUVFR1RwxAADwCqenmN19eQmPJogOh0OXXHKJnnrqKV188cW65557NHz4cM2YMaPc/mPGjFF+fr7z2rVrVzVHDAAA4P08ugaxcePGat++vaWtXbt2WrhwYbn97Xa77HZ7dYQGAAC8mcMhyc2PxnPwqD236NmzpzZv3mxp++GHH9S8eXMPRQQAAACPVhD/8pe/qEePHnrqqad02223afXq1Xr55Zf18ssvezIsAADg7TjmpkIerSB27dpV7733nt566y117NhRTzzxhDIyMnTnnXd6MiwAAACf5vFH7V133XW67rrrPB0GAADwJVQQK+TxBBEAAKDa8SzmCnn8WcwAAAA4v1BBBAAAPscYh4xx77E07h7Pk6ggAgAAwIIKIgAA8D3GuH/NoBdtUqGCCAAAAAsqiAAAwPeYKtjFTAURAAAA3ooKIgAA8D0Oh2Rz865jL9rFTIIIAAB8D1PMFWKKGQAAABZUEAEAgM8xDoeMm6eYOSgbAAAAXosKIgAA8D2sQawQFUQAAABYUEEEAAC+x2EkGxXEM6GCCAAAAAsqiAAAwPcYI8ndB2VTQQQAAICXooIIAAB8jnEYGTevQTReVEEkQQQAAL7HOOT+KWYOygYAAICXIkEEAAA+xzhMlVznYtq0aYqOjlZgYKC6d++u1atXV9j/nXfeUdu2bRUYGKjY2Fh9/PHH1u9mjMaOHavGjRsrKChIiYmJ2rJli0sxkSACAAB4yPz585Wamqq0tDStW7dOnTp1UlJSkvbv319u/y+++EKDBw/WXXfdpa+++koDBw7UwIED9e233zr7PP3003r++ec1Y8YMrVq1SrVq1VJSUpKOHz9e6bhIEAEAgO8xjqq5XDR58mQNHz5cKSkpat++vWbMmKHg4GDNnDmz3P5TpkxRv3799Le//U3t2rXTE088oUsuuURTp0499bWMUUZGhh577DENGDBAcXFxeuONN7R3714tWrSo0nFd0JtUTu8WKij0nkWhF7ITRSWeDgG/UVpc+X9TRNUq5V/FzxsnzQlPhwBJJ3Xq7+DJXb8ndcLtj2I+/b0KCgos7Xa7XXa7vUz/kpISrV27VmPGjHG2+fn5KTExUTk5OeXeIycnR6mpqZa2pKQkZ/K3fft25ebmKjEx0fl+WFiYunfvrpycHN1+++2V+i4XdIJ49OhRSVLzS3Z4NhD86mVPBwAAFdrm6QBgcfToUYWFhVXrPQMCAhQREaEVuR+fvfM5qF27tqKioixtaWlpGjduXJm+Bw8eVGlpqcLDwy3t4eHh2rRpU7nj5+bmlts/NzfX+f7ptjP1qYwLOkGMjIzUrl27FBISIpvN5ulwzllBQYGioqK0a9cuhYaGejocn8bf4vzB3+L8wd/i/OINfw9jjI4eParIyMhqv3dgYKC2b9+ukpKqmfUyxpTJScqrHp7vLugE0c/PT02bNvV0GG4TGhp6wf6X3dvwtzh/8Lc4f/C3OL9c6H+P6q4c/lZgYKACAwM9dv/TGjRoIH9/f+Xl5Vna8/LyFBERUe5nIiIiKux/+j/z8vLUuHFjS5/OnTtXOjZWxgAAAHhAQECAunTpoqysLGebw+FQVlaWEhISyv1MQkKCpb8kLVmyxNk/JiZGERERlj4FBQVatWrVGccszwVdQQQAALiQpaamasiQIYqPj1e3bt2UkZGhoqIipaSkSJKSk5PVpEkTpaenS5JGjRqlXr166dlnn9W1116refPm6csvv9TLL5/aB2Cz2TR69Gg9+eSTat26tWJiYvT4448rMjJSAwcOrHRcJIjnAbvdrrS0tAtyjYK34W9x/uBvcf7gb3F+4e/hXQYNGqQDBw5o7Nixys3NVefOnZWZmencZLJz5075+f1vwrdHjx6aO3euHnvsMT3yyCNq3bq1Fi1apI4dOzr7PPTQQyoqKtI999yjI0eO6LLLLlNmZqZL0+o2401PlgYAAMAfxhpEAAAAWJAgAgAAwIIEEQAAABYkiAAAALAgQfSwadOmKTo6WoGBgerevbtWr17t6ZB80vLly3X99dcrMjJSNpvNpQeaw73S09PVtWtXhYSEqFGjRho4cKA2b97s6bB80vTp0xUXF+c8kDkhIUGffPKJp8OCpIkTJzqPMwGqAgmiB82fP1+pqalKS0vTunXr1KlTJyUlJWn//v2eDs3nFBUVqVOnTpo2bZqnQ/F5y5Yt04gRI7Ry5UotWbJEJ06cUN++fVVUVOTp0HxO06ZNNXHiRK1du1ZffvmlrrrqKg0YMEDfffedp0PzaWvWrNFLL72kuLg4T4cCL8YxNx7UvXt3de3aVVOnTpV06vT0qKgo3X///Xr44Yc9HJ3vstlseu+991w6UBRV58CBA2rUqJGWLVumK664wtPh+Lx69erpn//8p+666y5Ph+KTCgsLdckll+jFF1/Uk08+qc6dOysjI8PTYcELUUH0kJKSEq1du1aJiYnONj8/PyUmJionJ8eDkQHnl/z8fEmnEhN4TmlpqebNm6eioiKXHtcF9xoxYoSuvfZay/93AFWBJ6l4yMGDB1VaWuo8Kf208PBwbdq0yUNRAecXh8Oh0aNHq2fPnpanBKD6fPPNN0pISNDx48dVu3Ztvffee2rfvr2nw/JJ8+bN07p167RmzRpPhwIfQIII4Lw1YsQIffvtt1qxYoWnQ/FZbdq00fr165Wfn68FCxZoyJAhWrZsGUliNdu1a5dGjRqlJUuWuPS4NOBckSB6SIMGDeTv76+8vDxLe15eniIiIjwUFXD+GDlypD788EMtX75cTZs29XQ4PisgIECtWrWSJHXp0kVr1qzRlClT9NJLL3k4Mt+ydu1a7d+/X5dccomzrbS0VMuXL9fUqVNVXFwsf39/D0YIb8MaRA8JCAhQly5dlJWV5WxzOBzKyspifQ98mjFGI0eO1Hvvvaf//Oc/iomJ8XRI+A2Hw6Hi4mJPh+Fz+vTpo2+++Ubr1693XvHx8brzzju1fv16kkO4HRVED0pNTdWQIUMUHx+vbt26KSMjQ0VFRUpJSfF0aD6nsLBQW7dudb7evn271q9fr3r16qlZs2YejMz3jBgxQnPnztX777+vkJAQ5ebmSpLCwsIUFBTk4eh8y5gxY9S/f381a9ZMR48e1dy5c5Wdna3Fixd7OjSfExISUmYdbq1atVS/fn3W56JKkCB60KBBg3TgwAGNHTtWubm56ty5szIzM8tsXEHV+/LLL3XllVc6X6empkqShgwZotmzZ3soKt80ffp0SVLv3r0t7bNmzdLQoUOrPyAftn//fiUnJ2vfvn0KCwtTXFycFi9erKuvvtrToQGoYpyDCAAAAAvWIAIAAMCCBBEAAAAWJIgAAACwIEEEAACABQkiAAAALEgQAQAAYEGCCAAAAAsSRAAAAFiQIAJwm9mzZ6tOnTqeDsPthg4dqoEDB1bYJzs7WzabTUeOHKmWmACgKpEgAue58pKTBQsWKDAwUM8++2yV3NNbE71zNWXKFMsjF3v37q3Ro0db+vTo0cP5SDoAuNDxLGbgAvPqq69qxIgRmjFjhlJSUjwdjk+oTNIXEBCgiIiIaogGAKoeFUTgAvL000/r/vvv17x58yzJ4fvvv69LLrlEgYGBatGihcaPH6+TJ09KkoYNG6brrrvOMs6JEyfUqFEjvfbaa2XukZ2drZSUFOXn58tms8lms2ncuHGSpMOHDys5OVl169ZVcHCw+vfvry1btpwx3gMHDig+Pl433nijiouL5XA4lJ6erpiYGAUFBalTp05asGCB5d42m01ZWVmKj49XcHCwevTooc2bN5/xHjt27JDNZtO8efPUo0cPBQYGqmPHjlq2bJml37Jly9StWzfZ7XY1btxYDz/8sPN3JJ2qysbGxiooKEj169dXYmKiioqKJFmruEOHDtWyZcs0ZcoU5+9nx44d5U4xL1y4UB06dJDdbld0dHSZim90dLSeeuopDRs2TCEhIWrWrJlefvnlM35XAKg2BsB5bciQIWbAgAHmoYceMrVr1zafffaZ5f3ly5eb0NBQM3v2bLNt2zbz6aefmujoaDNu3DhjjDGff/658ff3N3v37nV+5t133zW1atUyR48eLXO/4uJik5GRYUJDQ82+ffvMvn37nP1uuOEG065dO7N8+XKzfv16k5SUZFq1amVKSkqMMcbMmjXLhIWFGWOM2blzp2nTpo0ZMmSIOXnypDHGmCeffNK0bdvWZGZmmm3btplZs2YZu91usrOzjTHGLF261Egy3bt3N9nZ2ea7774zl19+uenRo8cZfz/bt283kkzTpk3NggULzPfff2/uvvtuExISYg4ePGiMMWb37t0mODjY3HfffWbjxo3mvffeMw0aNDBpaWnGGGP27t1ratSoYSZPnmy2b99uNmzYYKZNm+b83qf/BsYYc+TIEZOQkGCGDx/u/P2cPHnSGfvhw4eNMcZ8+eWXxs/Pz0yYMMFs3rzZzJo1ywQFBZlZs2Y5Y2/evLmpV6+emTZtmtmyZYtJT083fn5+ZtOmTRX+MwEAVY0EETjPDRkyxAQEBBhJJisrq8z7ffr0MU899ZSlbc6cOaZx48bO1+3btzeTJk1yvr7++uvN0KFDz3jP3yZ6p/3www9Gkvn888+dbQcPHjRBQUHm7bfftnxu06ZNJioqyjzwwAPG4XAYY4w5fvy4CQ4ONl988YVl3LvuussMHjzYGPO/BPG3SfBHH31kJJlffvml3FhPJ4gTJ050tp04ccI0bdrU+Z0feeQR06ZNG2csxhgzbdo0U7t2bVNaWmrWrl1rJJkdO3aUe4/fJojGGNOrVy8zatQoS5/fJ4h33HGHufrqqy19/va3v5n27ds7Xzdv3tz86U9/cr52OBymUaNGZvr06eXGAQDVhSlm4AIQFxen6OhopaWlqbCw0PLe119/rQkTJqh27drOa/jw4dq3b5+OHTsmSbr77rs1a9YsSVJeXp4++eQTDRs2zKUYNm7cqBo1aqh79+7Otvr166tNmzbauHGjs+2XX37R5Zdfrptuusk5DStJW7du1bFjx3T11VdbYn3jjTe0bdu2Mt/3tMaNG0uS9u/fX2F8CQkJzp9r1Kih+Ph4Z1wbN25UQkKCMxZJ6tmzpwoLC7V792516tRJffr0UWxsrG699Va98sorOnz4sEu/n9/buHGjevbsaWnr2bOntmzZotLSUmfbb7+rzWZTRETEWb8rAFQ1EkTgAtCkSRNlZ2drz5496tevn44ePep8r7CwUOPHj9f69eud1zfffKMtW7YoMDBQkpScnKwff/xROTk5+te//qWYmBhdfvnlVRKr3W5XYmKiPvzwQ+3Zs8cSpyR99NFHlli///57yzpESapZs6bz59NJncPhqJJ4Jcnf319LlizRJ598ovbt2+uFF15QmzZttH379iq752m//a7Sqe9bld8VACqDBBG4QDRv3lzLli1Tbm6uJUm85JJLtHnzZrVq1arM5ed36r/i9evX18CBAzVr1izNnj37rLufAwICLFUuSWrXrp1OnjypVatWOdt+/vlnbd68We3bt3e2+fn5ac6cOerSpYuuvPJK7d27V5LUvn172e127dy5s0ycUVFRf/j3s3LlSufPJ0+e1Nq1a9WuXTtn7Dk5OTLGOPt8/vnnCgkJUdOmTSWdSsx69uyp8ePH66uvvlJAQIDee++9cu9V3u/n99q1a6fPP//c0vb555/roosukr+//zl9RwCoLhxzA1xAoqKilJ2drSuvvFJJSUnKzMzU2LFjdd1116lZs2a65ZZb5Ofnp6+//lrffvutnnzySedn7777bl133XUqLS3VkCFDKrxPdHS0CgsLlZWVpU6dOik4OFitW7fWgAEDNHz4cL300ksKCQnRww8/rCZNmmjAgAGWz/v7++vNN9/U4MGDddVVVyk7O1sRERF68MEH9Ze//EUOh0OXXXaZ8vPz9fnnnys0NPSsMZ3NtGnT1Lp1a7Vr107PPfecDh8+7JxGv++++5SRkaH7779fI0eO1ObNm5WWlqbU1FT5+flp1apVysrKUt++fdWoUSOtWrVKBw4ccCaY5f1+Vq1apR07dqh27dqqV69emT5//etf1bVrVz3xxBMaNGiQcnJyNHXqVL344ot/6HsCQLXw9CJIABX7/QYJY07tym3durW59NJLTX5+vsnMzDQ9evQwQUFBJjQ01HTr1s28/PLLls84HA7TvHlzc80111Tqvv/3f/9n6tevbyQ5d/seOnTI/PnPfzZhYWEmKCjIJCUlmR9++MH5md9vbjlx4oS56aabTLt27UxeXp5xOBwmIyPDtGnTxtSsWdM0bNjQJCUlmWXLlhljym70MMaYr776ykgy27dvLzfO05tU5s6da7p162YCAgJM+/btzX/+8x9Lv+zsbNO1a1cTEBBgIiIizN///ndz4sQJY4wx33//vUlKSjINGzY0drvdXHTRReaFF15wfvb3f4PNmzebSy+91AQFBTljKy/2BQsWmPbt25uaNWuaZs2amX/+85+WmJo3b26ee+45S1unTp2cv28A8BSbMb+ZcwHgtQoLC9WkSRPNmjVLN910k6fDcZsdO3YoJiZGX331lTp37uzpcADAKzDFDHg5h8OhgwcP6tlnn1WdOnV0ww03eDokAMB5jgQR8HI7d+5UTEyMmjZtqtmzZ6tGDf5rDwCoGFPMAAAAsOCYGwAAAFiQIAIAAMCCBBEAAAAWJIgAAACwIEEEAACABQkiAAAALEgQAQAAYEGCCAAAAIv/D+IBFNCnvvC2AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn8AAAIjCAYAAAByAn13AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTw0lEQVR4nO3df3zNdf/H8efZ2NnWfvht2GxCfv8oIiRkWSqlukpSRHRVCLu6qvVrqIx+iKtEVFTfXJSuUlcidhlhIhIV8jML8yNsNtk45/39w+VcnbbVOXOOs/k87rfb53Zr7/M+7/frnKleXu/P+/2xGWOMAAAAYAlBgQ4AAAAA5w/JHwAAgIWQ/AEAAFgIyR8AAICFkPwBAABYCMkfAACAhZD8AQAAWAjJHwAAgIWQ/AEAAFgIyR9QhnTt2lVdu3Y9r3MmJCTonnvucWvbtm2bevTooejoaNlsNn388cfnNaZAsNlsGj16dKDDkCRlZGTIZrNp3rx5gQ7FZdasWbLZbNq9e3egQwFwjkj+ABQxYMAAbdq0Sc8995zeffddtW3b9pzG++GHHzR69OhiE4fXXntNs2bNOqfxPbVgwYIyk+BdqPbv36/HHntM3bp1U2RkpGw2mzIyMgIdFoDfIPkD4ObXX39VZmam7r33Xg0bNkx33XWXYmNjz2nMH374QWPGjCkTyd+YMWOKfe3XX3/Vk08+eV7iuJBt3bpVEyZM0N69e9WiRYtAhwOgGCR/ANwcOnRIklSpUqXABnKehYaGqkKFCoEOo9xr06aNfvnlF/34449KTk4OdDgAikHyhwvK3r17de+996p27dqy2+2qV6+eHnjgARUWFkqSjhw5oocfflgtWrRQRESEoqKi1LNnT3377bdu45R0f9PZe7F+u4y1bds23XrrrYqJiVFoaKhiY2N1xx13KCcnx9Vn5syZuvrqq1WjRg3Z7XY1bdpUU6dOLfXnXLx4sa688kpVqlRJERERatSokR5//HG3PgUFBUpNTVWDBg1kt9sVFxenRx55RAUFBSWOO3r0aMXHx0uS/v73v8tmsykhIaHE/j/99JMefPBBNWrUSGFhYapatapuu+02t+9t1qxZuu222yRJ3bp1k81mc32HCQkJ+v7777Vs2TJX+2/veTx27JhGjhypuLg42e12NWjQQBMmTJDT6XT12b17t2w2m1588UVNnz5d9evXl91u1+WXX661a9e6+t1zzz2aMmWKJLnmstlsrteLu+fvm2++Uc+ePRUVFaWIiAh1795dq1evdutz9s/KypUrlZycrOrVq+uiiy7SzTff7EqkS8vpdOq5555TbGysQkND1b17d23fvr1Iv6+++krXXnutoqOjFR4eri5dumjlypVufTz5XZ31/fff6+qrr1ZYWJhiY2P17LPPun3nfyQyMlJVqlQp1ecFcH7w11xcMPbt26d27drp2LFjuu+++9S4cWPt3btX8+bN04kTJxQSEqKdO3fq448/1m233aZ69erpwIEDev3119WlSxf98MMPql27tldzFhYWKikpSQUFBRo+fLhiYmK0d+9e/fvf/9axY8cUHR0tSZo6daqaNWumG2+8URUqVNCnn36qBx98UE6nU0OHDvVqzu+//1433HCDWrZsqbFjx8put2v79u1u/7N3Op268cYbtWLFCt13331q0qSJNm3apJdfflk//vhjiRs4brnlFlWqVEmjRo1S3759dd111ykiIqLEWNauXatVq1bpjjvuUGxsrHbv3q2pU6eqa9eu+uGHHxQeHq6rrrpKDz30kP7xj3/o8ccfV5MmTSRJTZo00aRJkzR8+HBFREToiSeekCTVrFlTknTixAl16dJFe/fu1V//+lfVrVtXq1atUkpKivbv369Jkya5xTJ79mwdP35cf/3rX2Wz2fT888/rlltu0c6dO1WxYkX99a9/1b59+7R48WK9++67Hn3PnTt3VlRUlB555BFVrFhRr7/+urp27aply5apffv2bv2HDx+uypUrKzU1Vbt379akSZM0bNgwzZ0790/nKsn48eMVFBSkhx9+WDk5OXr++efVr18/ffXVV64+//nPf9SzZ0+1adNGqampCgoKcv1l48svv1S7du08/l1JUnZ2trp166bTp0/rscce00UXXaTp06crLCys1J8DQBljgAtE//79TVBQkFm7dm2R15xOpzHGmJMnTxqHw+H22q5du4zdbjdjx451tc2cOdNIMrt27XLru3TpUiPJLF261BhjzDfffGMkmQ8++OAPYztx4kSRtqSkJHPxxRe7tXXp0sV06dLlD8d6+eWXjSRz6NChEvu8++67JigoyHz55Zdu7dOmTTOSzMqVK11t8fHxZsCAAa6fd+3aZSSZF1544Q/jMKb4z5WZmWkkmXfeecfV9sEHH7h9b7/VrFmzYj/zM888Yy666CLz448/urU/9thjJjg42OzZs8ct3qpVq5ojR464+s2fP99IMp9++qmrbejQoaak/+xJMqmpqa6fe/fubUJCQsyOHTtcbfv27TORkZHmqquucrWd/bOSmJjo+nNmjDGjRo0ywcHB5tixY8XO90fO/jlr0qSJKSgocLVPnjzZSDKbNm0yxpz5c92wYUOTlJTkNveJEydMvXr1zDXXXOPW9nvF/a5GjhxpJJmvvvrK1Xbw4EETHR1d7L8Tf+SPfu8AAodlX1wQnE6nPv74Y/Xq1avYnalnl/fsdruCgs78sXc4HPrll19cy6br16/3et6zlb1FixbpxIkTJfb7bdUkJydHhw8fVpcuXbRz50635WFPnL0Xb/78+SUuxX3wwQdq0qSJGjdurMOHD7uuq6++WpK0dOlSr+YsyW8/16lTp/TLL7+oQYMGqlSpUqm+z9/64IMP1LlzZ1WuXNntMyQmJsrhcGj58uVu/fv06aPKlSu7fu7cubMkaefOnV7P7XA49MUXX6h37966+OKLXe21atXSnXfeqRUrVig3N9ftPffdd5/bMnLnzp3lcDj0008/eT3/WQMHDlRISIjbmNL/PtOGDRu0bds23Xnnnfrll19c31F+fr66d++u5cuXu/6MePq7WrBgga644gpXxVCSqlevrn79+pX6cwAoW1j2xQXh0KFDys3NVfPmzf+wn9Pp1OTJk/Xaa69p165dcjgcrteqVq3q9bz16tVTcnKyJk6cqPfee0+dO3fWjTfeqLvuusuVGErSypUrlZqaqszMzCJJYk5OjlvfP9OnTx+98cYbGjx4sB577DF1795dt9xyi/7yl7+4Ettt27Zp8+bNql69erFjHDx40OP5HA5HkXvXqlSpopCQEP36669KS0vTzJkztXfvXhlj3D7Xudi2bZs2btzo8WeoW7eu289nE8GjR496PfehQ4d04sQJNWrUqMhrTZo0kdPpVFZWlpo1a+aX+T0dc9u2bZLOHM1TkpycHFWuXNnj39VPP/1UZElbUrHfBYDyieQPljJu3Dg99dRTGjRokJ555hlVqVJFQUFBGjlypFsV7bcVnN/6bbJ41ksvvaR77rlH8+fP1xdffKGHHnpIaWlpWr16tWJjY7Vjxw51795djRs31sSJExUXF6eQkBAtWLBAL7/8ssc30p8VFham5cuXa+nSpfrss8+0cOFCzZ07V1dffbW++OILBQcHy+l0qkWLFpo4cWKxY8TFxXk8X1ZWlurVq+fWtnTpUnXt2lXDhw/XzJkzNXLkSHXo0MF1KPQdd9zh9ef6PafTqWuuuUaPPPJIsa9fcsklbj8HBwcX2++3SY4/+WP+Pxvz7Hf8wgsvqHXr1sX2PXvPpj9/VwDKF5I/XBCqV6+uqKgofffdd3/Yb968eerWrZvefPNNt/Zjx46pWrVqrp/PVliOHTvm1q+kJbwWLVqoRYsWevLJJ7Vq1Sp16tRJ06ZN07PPPqtPP/1UBQUF+uSTT9wqOeey9BoUFKTu3bure/fumjhxosaNG6cnnnhCS5cuVWJiourXr69vv/1W3bt3LzGR9VRMTIwWL17s1taqVStJZ77PAQMG6KWXXnK9dvLkySLf2x/FUNJr9evXV15enhITE0sZuedz/V716tUVHh6urVu3Fnlty5YtCgoK8iqB9pf69etLkqKiov70e/L0dxUfH++qKP5Wcd8FgPKJe/5wQQgKClLv3r316aef6uuvvy7y+tlKSXBwcJFKzAcffKC9e/e6tZ39n+pv7ytzOByaPn26W7/c3FydPn3ara1FixYKCgpyHalytnrz+2W2mTNnevUZzzpy5EiRtrNVn7Nz3n777dq7d69mzJhRpO+vv/6q/Px8j+cLDQ1VYmKi23U2OS7u+3zllVeKVEgvuugiSUWT6bOvFdd+++23KzMzU4sWLSry2rFjx4p87574ozh+Kzg4WD169ND8+fPdjkI5cOCAZs+erSuvvFJRUVFez+9rbdq0Uf369fXiiy8qLy+vyOu/Xa739Hd13XXXafXq1VqzZo3bOO+9956PowcQKFT+cMEYN26cvvjiC3Xp0sV1vMn+/fv1wQcfaMWKFapUqZJuuOEGjR07VgMHDlTHjh21adMmvffee2439UtSs2bNdMUVVyglJUVHjhxRlSpVNGfOnCIJx3/+8x8NGzZMt912my655BKdPn1a7777roKDg3XrrbdKknr06KGQkBD16tVLf/3rX5WXl6cZM2aoRo0a2r9/v9efc+zYsVq+fLmuv/56xcfH6+DBg3rttdcUGxurK6+8UpJ099136/3339f999+vpUuXqlOnTnI4HNqyZYvef/99LVq06Jwf2SZJN9xwg959911FR0eradOmyszM1JIlS4rcP9m6dWsFBwdrwoQJysnJkd1ud5172KZNG02dOlXPPvusGjRooBo1aujqq6/W3//+d33yySe64YYbdM8996hNmzbKz8/Xpk2bNG/ePO3evdutWuuJNm3aSJIeeughJSUlKTg4WHfccUexfZ999lnXeYoPPvigKlSooNdff10FBQV6/vnnS/V9zZo1SwMHDtTMmTOLPE+5NIKCgvTGG2+oZ8+eatasmQYOHKg6depo7969Wrp0qaKiovTpp59K8vx39cgjj+jdd9/VtddeqxEjRriOeomPj9fGjRs9iuvZZ5+VdOa4HEl69913tWLFCkniKSpAWRCobcaAP/z000+mf//+pnr16sZut5uLL77YDB061HVcxsmTJ83f/vY3U6tWLRMWFmY6depkMjMziz1iZceOHSYxMdHY7XZTs2ZN8/jjj5vFixe7HV2xc+dOM2jQIFO/fn0TGhpqqlSpYrp162aWLFniNtYnn3xiWrZsaUJDQ01CQoKZMGGCeeutt4ocneHJUS/p6enmpptuMrVr1zYhISGmdu3apm/fvkWORCksLDQTJkwwzZo1M3a73VSuXNm0adPGjBkzxuTk5Lj6nctRL0ePHjUDBw401apVMxERESYpKcls2bKlyJjGGDNjxgxz8cUXm+DgYLfvMDs721x//fUmMjLSSHL7/MePHzcpKSmmQYMGJiQkxFSrVs107NjRvPjii6awsPBP49Xvjm85ffq0GT58uKlevbqx2Wxux778vq8xxqxfv94kJSWZiIgIEx4ebrp162ZWrVrl1ufsUS+/P2Lo98cCGWPMK6+8YiSZhQsX/uH3eva9vz9C6OxnnTlzplv7N998Y2655RZTtWpVY7fbTXx8vLn99ttNenq6q483v6uNGzeaLl26mNDQUFOnTh3zzDPPmDfffNPjo14klXgBCDybMefpbmgAsLjbb79du3fvdltSBYDzjWVfADgPjDHKyMjQ//3f/wU6FAAWR+UPAADAQtjtCwAAYCEkfwAAAAGyfPly9erVS7Vr15bNZtPHH3/8p+/JyMjQZZddJrvdrgYNGmjWrFlezUnyBwAAECD5+flq1aqVpkyZ4lH/Xbt26frrr1e3bt20YcMGjRw5UoMHDy72TNSScM8fAABAGWCz2fTRRx+pd+/eJfZ59NFH9dlnn7k90eqOO+7QsWPHtHDhQo/mKde7fZ1Op/bt26fIyMhzfoQVAAA4P4wxOn78uGrXrq2goPO/CHny5EkVFhb6ZWxjTJGcxG63y263+2T8zMzMIo9zTEpK0siRIz0eo1wnf/v27SsTz9cEAADey8rKUmxs7Hmd8+TJk6oXH6Hsg44/71wKERERRR63mJqaqtGjR/tk/OzsbNWsWdOtrWbNmsrNzdWvv/6qsLCwPx2jXCd/kZGRkqSf1icoKoLbF3/v5ktaBDoEAACKOK1TWqEFrv+Pn0+FhYXKPujQT+sSFBXp29wh97hT8W12Kysry+35376q+vlKuU7+zpZVoyKCfP4LvBBUsFUMdAgAABT1390GgbxlKyLSpohI387v1H/zkqgot+TPl2JiYnTgwAG3tgMHDigqKsqjqp9UzpM/AACA0nAYpxw+3vLqME7fDliMDh06aMGCBW5tixcvVocOHTweg3IZAABAgOTl5WnDhg3asGGDpDNHuWzYsEF79uyRJKWkpKh///6u/vfff7927typRx55RFu2bNFrr72m999/X6NGjfJ4Tip/AADAcpwycsq3pb/SjPf111+rW7durp+Tk5MlSQMGDNCsWbO0f/9+VyIoSfXq1dNnn32mUaNGafLkyYqNjdUbb7yhpKQkj+ck+QMAAAiQrl276o+OXC7u6R1du3bVN998U+o5Sf4AAIDlOOWUr+/Q8/2I/sE9fwAAABZC5Q8AAFiOwxg5fPyEW1+P5y9U/gAAACyEyh8AALCcsrLbNxBI/gAAgOU4ZeSwaPLHsi8AAICFUPkDAACWY+VlXyp/AAAAFkLlDwAAWA5HvQAAAMASqPwBAADLcf738vWY5QGVPwAAAAuh8gcAACzH4Ydz/nw9nr+Q/AEAAMtxmDOXr8csD1j2BQAAsBAqfwAAwHLY8AEAAABLoPIHAAAsxymbHLL5fMzygMofAACAhVD5AwAAluM0Zy5fj1keUPkDAACwkDKR/E2ZMkUJCQkKDQ1V+/bttWbNmkCHBAAALmCO/97z5+urPAh48jd37lwlJycrNTVV69evV6tWrZSUlKSDBw8GOjQAAHCBIvkLoIkTJ2rIkCEaOHCgmjZtqmnTpik8PFxvvfVWoEMDAAC44AR0w0dhYaHWrVunlJQUV1tQUJASExOVmZlZpH9BQYEKCgpcP+fm5p6XOAEAwIXFaWxyGh8f9eLj8fwloJW/w4cPy+FwqGbNmm7tNWvWVHZ2dpH+aWlpio6Odl1xcXHnK1QAAIALQsCXfb2RkpKinJwc15WVlRXokAAAQDlk5Xv+ArrsW61aNQUHB+vAgQNu7QcOHFBMTEyR/na7XXa7/XyFBwAAcMEJaOUvJCREbdq0UXp6uqvN6XQqPT1dHTp0CGBkAADgQuZQkF+u8iDgT/hITk7WgAED1LZtW7Vr106TJk1Sfn6+Bg4cGOjQAAAALjgBT/769OmjQ4cO6emnn1Z2drZat26thQsXFtkEAgAA4CvGD7t9TTnZ7Rvw5E+Shg0bpmHDhgU6DAAAYBH+2KBRXjZ8lI/FaQAAAPhEmaj8AQAAnE8OEySH8W0NzGF8OpzfUPkDAACwECp/AADAcpyyyenjGphT5aP0R+UPAADAQqj8AQAAy2G3LwAAACyByh8AALAc/+z2LR/3/JH8AQAAyzmz4cO3y7S+Hs9fWPYFAACwECp/AADAcpwKkoOjXgAAAHCho/IHAAAsx8obPqj8AQAAWAiVPwAAYDlOBfF4NwAAAFz4qPwBAADLcRibHMbHj3fz8Xj+QvIHAAAsx+GHo14cLPsCAACgrKHyBwAALMdpguT08VEvTo56AQAAQFlD5Q8AAFgO9/wBAADAEqj8AQAAy3HK90ezOH06mv9Q+QMAALCQC6Ly13reIAWFhgY6jDLHTAp0BGVXg5GrAx0CACCA/PN4t/JRU7sgkj8AAABvOEyQHD4+6sXX4/lL+YgSAAAAPkHlDwAAWI5TNjnl6w0f5ePZvlT+AAAALITKHwAAsBzu+QMAAIAlUPkDAACW45/Hu5WPmlr5iBIAAAA+QeUPAABYjtPY5PT14918PJ6/UPkDAACwECp/AADAcpx+uOePx7sBAACUUU4TJKePj2bx9Xj+Uj6iBAAAgE9Q+QMAAJbjkE0OHz+Ozdfj+QuVPwAAAAuh8gcAACyHe/4AAABgCVT+AACA5Tjk+3v0HD4dzX+o/AEAAFgIlT8AAGA5Vr7nj+QPAABYjsMEyeHjZM3X4/lL+YgSAAAAPkHyBwAALMfIJqePL1PKDSRTpkxRQkKCQkND1b59e61Zs+YP+0+aNEmNGjVSWFiY4uLiNGrUKJ08edLj+Uj+AAAAAmTu3LlKTk5Wamqq1q9fr1atWikpKUkHDx4stv/s2bP12GOPKTU1VZs3b9abb76puXPn6vHHH/d4TpI/AABgOWfv+fP15a2JEydqyJAhGjhwoJo2bapp06YpPDxcb731VrH9V61apU6dOunOO+9UQkKCevToob59+/5ptfC3SP4AAAB8KDc31+0qKCgotl9hYaHWrVunxMREV1tQUJASExOVmZlZ7Hs6duyodevWuZK9nTt3asGCBbruuus8jo/dvgAAwHKcxian8e0hz2fHi4uLc2tPTU3V6NGji/Q/fPiwHA6Hatas6dZes2ZNbdmypdg57rzzTh0+fFhXXnmljDE6ffq07r//fq+WfUn+AAAAfCgrK0tRUVGun+12u8/GzsjI0Lhx4/Taa6+pffv22r59u0aMGKFnnnlGTz31lEdjkPwBAADLcShIDh/f/XZ2vKioKLfkryTVqlVTcHCwDhw44NZ+4MABxcTEFPuep556SnfffbcGDx4sSWrRooXy8/N133336YknnlBQ0J9/poDe87d8+XL16tVLtWvXls1m08cffxzIcAAAgEWcXfb19eWNkJAQtWnTRunp6f+Ly+lUenq6OnToUOx7Tpw4USTBCw4OliQZYzyaN6DJX35+vlq1aqUpU6YEMgwAAICASE5O1owZM/T2229r8+bNeuCBB5Sfn6+BAwdKkvr376+UlBRX/169emnq1KmaM2eOdu3apcWLF+upp55Sr169XEngnwnosm/Pnj3Vs2fPQIYAAAAsyKkgOX1cAyvNeH369NGhQ4f09NNPKzs7W61bt9bChQtdm0D27NnjVul78sknZbPZ9OSTT2rv3r2qXr26evXqpeeee87jOcvVPX8FBQVu26Vzc3MDGA0AAMC5GzZsmIYNG1bsaxkZGW4/V6hQQampqUpNTS31fOXqnL+0tDRFR0e7rt9vpQYAAPCEw9j8cpUH5Sr5S0lJUU5OjuvKysoKdEgAAADlSrla9rXb7T49KwcAAFiTPw95LuvKVeUPAAAA5yaglb+8vDxt377d9fOuXbu0YcMGValSRXXr1g1gZAAA4EJmTJCcxrc1MOPj8fwloMnf119/rW7durl+Tk5OliQNGDBAs2bNClBUAADgQueQTQ75dpnW1+P5S0CTv65du3p8GjUAAADOXbna8AEAAOALTuP7DRrOclLPKh+L0wAAAPAJKn8AAMBynH7Y8OHr8fylfEQJAAAAn6DyBwAALMcpm5w+3p3r6/H8hcofAACAhVD5AwAAluMwNjl8vNvX1+P5C8kfAACwHDZ8AAAAwBKo/AEAAMtxyub7Q57Z8AEAAICyhsofAACwHOOHo14MlT8AAACUNVT+AACA5TiNH+75KydHvVD5AwAAsBAqfwAAwHKsfM4fyR8AALAcln0BAABgCVT+AACA5Tj9cNQLhzwDAACgzKHyBwAALId7/gAAAGAJVP4AAIDlUPkDAACAJVD5AwAAlmPlyt8FkfwFFdoUFFQ+vvDzKXbpqUCHUGYdGdQh0CGUWVXeygx0CADgd1ZO/lj2BQAAsJALovIHAADgDSPfH8psfDqa/1D5AwAAsBAqfwAAwHK45w8AAACWQOUPAABYDpU/AAAAWAKVPwAAYDlWrvyR/AEAAMuxcvLHsi8AAICFUPkDAACWY4xNxseVOl+P5y9U/gAAACyEyh8AALAcp2w+f7ybr8fzFyp/AAAAFkLlDwAAWA67fQEAAGAJVP4AAIDlsNsXAAAAlkDlDwAAWI6V7/kj+QMAAJbDsi8AAAAsgcofAACwHOOHZV8qfwAAAChzqPwBAADLMZKM8f2Y5QGVPwAAAAuh8gcAACzHKZts8vFRLz4ez1+o/AEAAFgIlT8AAGA5Vj7nj+QPAABYjtPYZOMJH55LT09Xenq6Dh48KKfT6fbaW2+95fE4aWlp+te//qUtW7YoLCxMHTt21IQJE9SoUaPShAUAAIA/4fU9f2PGjFGPHj2Unp6uw4cP6+jRo26XN5YtW6ahQ4dq9erVWrx4sU6dOqUePXooPz/f27AAAAA8Zox/rvLA68rftGnTNGvWLN19993nPPnChQvdfp41a5Zq1KihdevW6aqrrjrn8QEAAODO6+SvsLBQHTt29EcsysnJkSRVqVKl2NcLCgpUUFDg+jk3N9cvcQAAgAublTd8eL3sO3jwYM2ePdvngTidTo0cOVKdOnVS8+bNi+2Tlpam6Oho1xUXF+fzOAAAAC5kXlf+Tp48qenTp2vJkiVq2bKlKlas6Pb6xIkTSxXI0KFD9d1332nFihUl9klJSVFycrLr59zcXBJAAADgNStX/rxO/jZu3KjWrVtLkr777ju312y20n3oYcOG6d///reWL1+u2NjYEvvZ7XbZ7fZSzQEAAIBSJH9Lly712eTGGA0fPlwfffSRMjIyVK9ePZ+NDQAAUBLO+Suln3/+WZL+sFr3R4YOHarZs2dr/vz5ioyMVHZ2tiQpOjpaYWFh5xIaAABAifxxNEt5OerF6w0fTqdTY8eOVXR0tOLj4xUfH69KlSrpmWeeKXLg85+ZOnWqcnJy1LVrV9WqVct1zZ0719uwAAAA4AGvK39PPPGE3nzzTY0fP16dOnWSJK1YsUKjR4/WyZMn9dxzz3k8likvKTIAALignKn8+XrDh0+H8xuvk7+3335bb7zxhm688UZXW8uWLVWnTh09+OCDXiV/AAAAOL+8Tv6OHDmixo0bF2lv3Lixjhw54pOgAAAA/MnKR714fc9fq1at9OqrrxZpf/XVV9WqVSufBAUAAAD/8Lry9/zzz+v666/XkiVL1KFDB0lSZmamsrKytGDBAp8HCAAA4Gvmv5evxywPvK78denSRT/++KNuvvlmHTt2TMeOHdMtt9yirVu3qnPnzv6IEQAAAD5SqnP+ateuzcYOAABQbln5nj+Pkr+NGzeqefPmCgoK0saNG/+wb8uWLX0SGAAAgN9YeN3Xo+SvdevWys7OVo0aNdS6dWvZbLZiz+iz2WxyOBw+DxIAAAC+4VHyt2vXLlWvXt31zwAAAOWaH5Z9VU6WfT3a8BEfHy+b7cwH+umnn1SnTh3Xo93OXnXq1NFPP/3k12ABAAAuNFOmTFFCQoJCQ0PVvn17rVmz5g/7Hzt2TEOHDlWtWrVkt9t1ySWXeHXiite7fbt161bsYc45OTnq1q2bt8MBAACcd2ce7+b7y1tz585VcnKyUlNTtX79erVq1UpJSUk6ePBgsf0LCwt1zTXXaPfu3Zo3b562bt2qGTNmqE6dOh7P6fVuX2OMqwr4W7/88osuuugib4cDAACwrIkTJ2rIkCEaOHCgJGnatGn67LPP9NZbb+mxxx4r0v+tt97SkSNHtGrVKlWsWFGSlJCQ4NWcHid/t9xyi6Qzmzruuece2e1212sOh0MbN25Ux44dvZocAAAgEPx51Etubq5bu91ud8ubziosLNS6deuUkpLiagsKClJiYqIyMzOLneOTTz5Rhw4dNHToUM2fP1/Vq1fXnXfeqUcffVTBwcEexelx8hcdHS3pTOUvMjJSYWFhrtdCQkJ0xRVXaMiQIZ4OBwAAcEGKi4tz+zk1NVWjR48u0u/w4cNyOByqWbOmW3vNmjW1ZcuWYsfeuXOn/vOf/6hfv35asGCBtm/frgcffFCnTp1SamqqR/F5nPzNnDlT0pnS4sMPP8wSLwAAKL+Mzfe7c/87XlZWlqKiolzNxVX9SsvpdKpGjRqaPn26goOD1aZNG+3du1cvvPCC75O/szwdGAAAoKwq7QaNPxtTkqKiotySv5JUq1ZNwcHBOnDggFv7gQMHFBMTU+x7atWqpYoVK7ot8TZp0kTZ2dkqLCxUSEjIn87rUfJ32WWXKT09XZUrV9all15a7IaPs9avX+/JkAAAAJYWEhKiNm3aKD09Xb1795Z0prKXnp6uYcOGFfueTp06afbs2XI6nQoKOnNoy48//qhatWp5lPhJHiZ/N910k6tkeTY4AACAcquMPN4tOTlZAwYMUNu2bdWuXTtNmjRJ+fn5rt2//fv3V506dZSWliZJeuCBB/Tqq69qxIgRGj58uLZt26Zx48bpoYce8nhOj5K/3y71suwLAADgG3369NGhQ4f09NNPKzs7W61bt9bChQtdm0D27NnjqvBJZzaTLFq0SKNGjVLLli1Vp04djRgxQo8++qjHc3p9z19WVpZsNptiY2MlSWvWrNHs2bPVtGlT3Xfffd4OBwAAcN7586gXbw0bNqzEZd6MjIwibR06dNDq1atLNZdUiid83HnnnVq6dKkkKTs7W4mJiVqzZo2eeOIJjR07ttSBAAAAwP+8Tv6+++47tWvXTpL0/vvvq0WLFlq1apXee+89zZo1y9fxAQAA+Ifx8VVOeJ38nTp1yrX5Y8mSJbrxxhslSY0bN9b+/ft9Gx0AAAB8yuvkr1mzZpo2bZq+/PJLLV68WNdee60kad++fapatarPAwQAAPC1s/f8+foqD7xO/iZMmKDXX39dXbt2Vd++fdWqVStJZ541d3Y5GAAAoEzz9ZJvOVr69Xq3b9euXXX48GHl5uaqcuXKrvb77rtP4eHhPg3OU9U2OFWhojMgc5dlhVGePeDZiiL2nQ50CGXWiVvaBzqEMiv8X18FOgQAOGdeJ3+SFBwcrNOnT2vFihWSpEaNGikhIcGXcQEAAPiR7b+Xr8cs+7xe9s3Pz9egQYNUq1YtXXXVVbrqqqtUu3Zt3XvvvTpx4oQ/YgQAAICPeJ38JScna9myZfr000917NgxHTt2TPPnz9eyZcv0t7/9zR8xAgAA+Bb3/Hnuww8/1Lx589S1a1dX23XXXaewsDDdfvvtmjp1qi/jAwAAgA95nfydOHHC9by536pRowbLvgAAoHzwR6WunFT+vF727dChg1JTU3Xy5ElX26+//qoxY8aoQ4cOPg0OAAAAvuV15W/y5MlKSkpSbGys64y/b7/9VqGhoVq0aJHPAwQAAPA5Yztz+XrMcsDr5K958+batm2b3nvvPW3ZskWS1LdvX/Xr109hYWE+DxAAAMDXjDlz+XrM8qBU5/yFh4dryJAhvo4FAAAAflaq5G/r1q165ZVXtHnzZklSkyZNNGzYMDVu3NinwQEAAPgFGz489+GHH6p58+Zat26dWrVqpVatWmn9+vVq0aKFPvzwQ3/ECAAAAB/xuvL3yCOPKCUlRWPHjnVrT01N1SOPPKJbb73VZ8EBAAD4hYU3fHhd+du/f7/69+9fpP2uu+7S/v37fRIUAAAA/MPr5K9r16768ssvi7SvWLFCnTt39klQAAAA/mQz/rnKA6+XfW+88UY9+uijWrduna644gpJ0urVq/XBBx9ozJgx+uSTT9z6AgAAoOzwOvl78MEHJUmvvfaaXnvttWJfkySbzSaHw3GO4QEAAPiBhXf7ep38OZ1Of8QBAABw/rDhAwAAAFZQqkOeAQAAyjULL/tS+QMAALAQKn8AAMB6qPwBAADACkpV+XM6ndq+fbsOHjxYZPfvVVdd5ZPAAAAA/MbClT+vk7/Vq1frzjvv1E8//SRj3D8lZ/sBAACUbV4nf/fff7/atm2rzz77TLVq1ZLNVj7OtAEAAHCx8Dl/Xid/27Zt07x589SgQQN/xAMAAAA/8nrDR/v27bV9+3Z/xAIAAHBe2Ix/rvLA68rf8OHD9be//U3Z2dlq0aKFKlas6PZ6y5YtfRYcAACAX7Dhw3O33nqrJGnQoEGuNpvNJmMMGz4AAADKOK+Tv127dvls8qlTp2rq1KnavXu3JKlZs2Z6+umn1bNnT5/NAQAAgP/xOvmLj4/32eSxsbEaP368GjZsKGOM3n77bd1000365ptv1KxZM5/NAwAAgDNK9YSPd999V506dVLt2rX1008/SZImTZqk+fPnezVOr169dN1116lhw4a65JJL9NxzzykiIkKrV68uTVgAAAAesckPGz4C/aE85HXyN3XqVCUnJ+u6667TsWPHXPf4VapUSZMmTSp1IA6HQ3PmzFF+fr46dOhQbJ+CggLl5ua6XQAAAPCc18nfK6+8ohkzZuiJJ55QcHCwq71t27batGmT1wFs2rRJERERstvtuv/++/XRRx+padOmxfZNS0tTdHS064qLi/N6PgAAANchz76+ygGvk79du3bp0ksvLdJut9uVn5/vdQCNGjXShg0b9NVXX+mBBx7QgAED9MMPPxTbNyUlRTk5Oa4rKyvL6/kAAACszOsNH/Xq1dOGDRuKbPxYuHChmjRp4nUAISEhrqeFtGnTRmvXrtXkyZP1+uuvF+lrt9tlt9u9ngMAAMAN5/x5Ljk5WUOHDtXJkydljNGaNWv0z3/+U2lpaXrjjTfOOSCn06mCgoJzHgcAAKBEJH+eGzx4sMLCwvTkk0/qxIkTuvPOO1W7dm1NnjxZd9xxh1djpaSkqGfPnqpbt66OHz+u2bNnKyMjQ4sWLfI2LAAAAHjA6+QvNzdX/fr1U79+/XTixAnl5eWpRo0akqTt27e7lnA9cfDgQfXv31/79+9XdHS0WrZsqUWLFumaa67xNiwAAACP+eNZvBfss32vv/56LVmyRHa7XeHh4QoPD5ckbd26Vd27d9fPP//s8Vhvvvmmt9MDAADgHHi92zciIkI333yzTp8+7WrbvHmzunbt6nruLwAAQJlm/HSVA14nf//617+Uk5Ojfv36yRij7777Tl27dlXfvn01efJkf8QIAAAAH/E6+QsLC9Nnn32mrVu36vbbb1f37t3Vv39/TZw40R/xAQAA+J6FK38e3fP3+8eoBQUFae7cubrmmmt066236qmnnnL1iYqK8n2UAAAA8AmPkr9KlSrJZiv6yBJjjKZNm6bXX39dxhjZbDbXs34BAADKKnb7/omlS5f6Ow4AAIDzxx/P4i0nz/b1KPnr0qWLv+MAAADAeeD1OX+SdOzYMb355pvavHmzJKlZs2YaNGiQoqOjfRocAACAX1j48W5e7/b9+uuvVb9+fb388ss6cuSIjhw5ookTJ6p+/fpav369P2IEAACAj3hd+Rs1apRuvPFGzZgxQxUqnHn76dOnNXjwYI0cOVLLly/3eZAAAAC+xIYPL3z99dduiZ8kVahQQY888ojatm3r0+AAAADgW14v+0ZFRWnPnj1F2rOyshQZGemToAAAAPzKwoc8e5389enTR/fee6/mzp2rrKwsZWVlac6cORo8eLD69u3rjxgBAADgI14v+7744ouy2Wzq37+/Tp8+LUmqWLGiHnjgAY0fP97nAQIAAPicH+75Ky+VP6+Tv5CQEE2ePFlpaWnasWOHJKl+/foKDw/3eXAAAAB+wVEvnhs0aJCOHz+u8PBwtWjRQi1atFB4eLjy8/M1aNAgf8QIAAAAH/E6+Xv77bf166+/Fmn/9ddf9c477/gkKAAAAL+y8IYPj5d9c3NzZYyRMUbHjx9XaGio6zWHw6EFCxaoRo0afgkSAAAAvuFx8lepUiXZbDbZbDZdcsklRV632WwaM2aMT4MDAADwBw559sDSpUtljNHVV1+tDz/8UFWqVHG9FhISovj4eNWuXdsvQQIAAMA3PE7+unTpIknatWuX6tatK5vN5regAAAA4B9eH/USHx/vjzgAAABwHnid/AEAAJR7Fj7nj+QPAABYjpU3fHh9zh8AAADKL68rf6mpqRo0aFCZuvcvev1+VQiyBzqMMie/WUygQyizQrPzAx1CmZXTKCrQIZRZ+X9pH+gQyqyL5n0V6BAA75WTSp2veV35mz9/vurXr6/u3btr9uzZKigo8EdcAAAA8AOvk78NGzZo7dq1atasmUaMGKGYmBg98MADWrt2rT/iAwAA8D0LP96tVPf8XXrppfrHP/6hffv26c0339TPP/+sTp06qWXLlpo8ebJycnJ8HScAAAB84Jw2fBhjdOrUKRUWFsoYo8qVK+vVV19VXFyc5s6d66sYAQAAfOrsbl9fX+VBqZK/devWadiwYapVq5ZGjRqlSy+9VJs3b9ayZcu0bds2Pffcc3rooYd8HSsAAADOkde7fVu0aKEtW7aoR48eevPNN9WrVy8FBwe79enbt69GjBjhsyABAAB8ikOePXf77bdr0KBBqlOnTol9qlWrJqfTeU6BAQAA+AuHPHvo1KlTmjVrlnJzc/0VDwAAAPzIq8pfxYoVdfLkSX/FAgAAcH5YeNnX6w0fQ4cO1YQJE3T69Gl/xAMAAGApU6ZMUUJCgkJDQ9W+fXutWbPGo/fNmTNHNptNvXv39mo+r+/5W7t2rdLT0/XFF1+oRYsWuuiii9xe/9e//uXtkAAAAOdXGan8zZ07V8nJyZo2bZrat2+vSZMmKSkpSVu3blWNGjVKfN/u3bv18MMPq3Pnzl7P6XXlr1KlSrr11luVlJSk2rVrKzo62u0CAACAZyZOnKghQ4Zo4MCBatq0qaZNm6bw8HC99dZbJb7H4XCoX79+GjNmjC6++GKv5/S68jdz5kyvJwEAAChL/Lnb9/cbY+12u+x2e5H+hYWFWrdunVJSUlxtQUFBSkxMVGZmZonzjB07VjVq1NC9996rL7/80us4S3XI8+nTp7VkyRK9/vrrOn78uCRp3759ysvLK81wAAAAF4y4uDi3VdG0tLRi+x0+fFgOh0M1a9Z0a69Zs6ays7OLfc+KFSv05ptvasaMGaWOz+vK308//aRrr71We/bsUUFBga655hpFRkZqwoQJKigo0LRp00odDAAAwHnhx3v+srKyFBUV5WourupXGsePH9fdd9+tGTNmqFq1aqUex+vkb8SIEWrbtq2+/fZbVa1a1dV+8803a8iQIaUOBAAA4LzxY/IXFRXllvyVpFq1agoODtaBAwfc2g8cOKCYmJgi/Xfs2KHdu3erV69errazD9WoUKGCtm7dqvr16//pvF4nf19++aVWrVqlkJAQt/aEhATt3bvX2+EAAAAsKSQkRG3atFF6errruBan06n09HQNGzasSP/GjRtr06ZNbm1PPvmkjh8/rsmTJysuLs6jeb1O/pxOpxwOR5H2n3/+WZGRkd4OBwAAcN6Vlce7JScna8CAAWrbtq3atWunSZMmKT8/XwMHDpQk9e/fX3Xq1FFaWppCQ0PVvHlzt/dXqlRJkoq0/xGvk78ePXpo0qRJmj59uiTJZrMpLy9Pqampuu6667wdDgAAwLL69OmjQ4cO6emnn1Z2drZat26thQsXujaB7NmzR0FBpdqfWyKvk7+XXnpJSUlJatq0qU6ePKk777xT27ZtU7Vq1fTPf/7Tp8EBAAD4RRk55FmShg0bVuwyryRlZGT84XtnzZrl9XxeJ3+xsbH69ttvNWfOHG3cuFF5eXm699571a9fP4WFhXkdAAAAAM4fr5M/6cyOkrvuusvXsQAAAJwXZeWev0DwOvl75513/vD1/v37lzoYAAAA+Fepzvn7rVOnTunEiRMKCQlReHg4yR8AACj7ytA9f+eb19tHjh496nbl5eVp69atuvLKK9nwAQAAygfjp6sc8Mne4YYNG2r8+PFFqoIAAAAoW0q14aPYgSpU0L59+3w1HAAAgN/Y/nv5eszywOvk75NPPnH72Rij/fv369VXX1WnTp18FhgAAAB8z+vk7+yz586y2WyqXr26rr76ar300ku+igsAAMB/2PDhOafT6XY5HA5lZ2dr9uzZqlWrVqkDGT9+vGw2m0aOHFnqMQAAAPDHSn3P3+HDhxUSEqKoqKhzDmLt2rV6/fXX1bJly3MeCwAA4M9Y+ZBnryp/x44d09ChQ1WtWjXVrFlTlStXVkxMjFJSUnTixIlSBZCXl6d+/fppxowZqly5cqnGAAAAgGc8rvwdOXJEHTp00N69e9WvXz81adJEkvTDDz/olVde0eLFi7VixQpt3LhRq1ev1kMPPeTRuEOHDtX111+vxMREPfvss3/Yt6CgQAUFBa6fc3NzPQ0fAADgfyx8z5/Hyd/YsWMVEhKiHTt2qGbNmkVe69Gjh+6++2598cUX+sc//uHRmHPmzNH69eu1du1aj/qnpaVpzJgxnoYMAABQsnKSrPmax8u+H3/8sV588cUiiZ8kxcTE6Pnnn9eHH36o5ORkDRgw4E/Hy8rK0ogRI/Tee+8pNDTUoxhSUlKUk5PjurKysjwNHwAAAPKi8rd//341a9asxNebN2+uoKAgpaamejTeunXrdPDgQV122WWuNofDoeXLl+vVV19VQUGBgoOD3d5jt9tlt9s9DRkAAKBYVt7w4XHyV61aNe3evVuxsbHFvr5r1y7VqFHD44m7d++uTZs2ubUNHDhQjRs31qOPPlok8QMAAMC58zj5S0pK0hNPPKHFixcrJCTE7bWCggI99dRTuvbaaz2eODIyUs2bN3dru+iii1S1atUi7QAAAD7Fho8/N3bsWLVt21YNGzbU0KFD1bhxYxljtHnzZr322msqKCjQO++8489YAQAAcI48Tv5iY2OVmZmpBx98UCkpKTLmTHprs9l0zTXX6NVXX1XdunXPKZiMjIxzej8AAIAnuOfPQ/Xq1dPnn3+uo0ePatu2bZKkBg0aqEqVKn4JDgAAAL5Vqse7Va5cWe3atfN1LAAAAOeHhe/58+rxbgAAACjfSlX5AwAAKM+45w8AAMBKWPYFAACAFVD5AwAA1kPlDwAAAFZA5Q8AAFiOlTd8UPkDAACwECp/AADAerjnDwAAAFZA5Q8AAFiOzRjZjG9Ldb4ez19I/gAAgPWw7AsAAAAroPIHAAAsh6NeAAAAYAlU/gAAgPVwzx8AAACsgMofAACwHO75AwAAgCVQ+QMAANZj4Xv+SP4AAIDlsOwLAAAAS6DyBwAArMfCy75U/gAAACzkgqj8FdapImeF0ECHUebYD58MdAhlVl69yECHUGaFHT4V6BDKLJuznPy1HoBHyss9er5G5Q8AAMBCLojKHwAAgFeMOXP5esxygMofAACAhVD5AwAAlmPlc/5I/gAAgPVw1AsAAACsgMofAACwHJvzzOXrMcsDKn8AAAAWQuUPAABYD/f8AQAAwAqo/AEAAMux8lEvVP4AAAAshMofAACwHgs/3o3kDwAAWA7LvgAAALAEKn8AAMB6OOoFAAAAVkDlDwAAWA73/AEAAMASqPwBAADrsfBRL1T+AAAALITKHwAAsBwr3/NH8gcAAKyHo14AAABgBVT+AACA5Vh52ZfKHwAAgIVQ+QMAANbjNGcuX49ZDlD5AwAAsJCAJn+jR4+WzWZzuxo3bhzIkAAAgBUYP13lQMCXfZs1a6YlS5a4fq5QIeAhAQAAXLACnmlVqFBBMTExgQ4DAABYiE1+2O3r2+H8JuD3/G3btk21a9fWxRdfrH79+mnPnj0l9i0oKFBubq7bBQAA4LWzz/b19VUOBDT5a9++vWbNmqWFCxdq6tSp2rVrlzp37qzjx48X2z8tLU3R0dGuKy4u7jxHDAAAUL4FNPnr2bOnbrvtNrVs2VJJSUlasGCBjh07pvfff7/Y/ikpKcrJyXFdWVlZ5zliAABwITh7yLOvr/Ig4Mu+v1WpUiVdcskl2r59e7Gv2+12RUVFuV0AAADl2ZQpU5SQkKDQ0FC1b99ea9asKbHvjBkz1LlzZ1WuXFmVK1dWYmLiH/YvTplK/vLy8rRjxw7VqlUr0KEAAIALWRk56mXu3LlKTk5Wamqq1q9fr1atWikpKUkHDx4stn9GRob69u2rpUuXKjMzU3FxcerRo4f27t3r8ZwBTf4efvhhLVu2TLt379aqVat08803Kzg4WH379g1kWAAAAOfFxIkTNWTIEA0cOFBNmzbVtGnTFB4errfeeqvY/u+9954efPBBtW7dWo0bN9Ybb7whp9Op9PR0j+cM6FEvP//8s/r27atffvlF1atX15VXXqnVq1erevXqgQwLAABc4GzGyObj3blnx/v9aSR2u112u71I/8LCQq1bt04pKSmutqCgICUmJiozM9OjOU+cOKFTp06pSpUqHscZ0ORvzpw5gZweAADA535/GklqaqpGjx5dpN/hw4flcDhUs2ZNt/aaNWtqy5YtHs316KOPqnbt2kpMTPQ4voAf8gwAAHDeOf97+XpMSVlZWW6bUour+vnC+PHjNWfOHGVkZCg0NNTj95H8AQAAy/Hnsq+nJ5JUq1ZNwcHBOnDggFv7gQMH/vTpZy+++KLGjx+vJUuWqGXLll7FWaZ2+wIAAFhFSEiI2rRp47ZZ4+zmjQ4dOpT4vueff17PPPOMFi5cqLZt23o9L5U/AABgPaU8muVPx/RScnKyBgwYoLZt26pdu3aaNGmS8vPzNXDgQElS//79VadOHaWlpUmSJkyYoKefflqzZ89WQkKCsrOzJUkRERGKiIjwaE6SPwAAgADp06ePDh06pKefflrZ2dlq3bq1Fi5c6NoEsmfPHgUF/W+hdurUqSosLNRf/vIXt3FK2lRSHJI/AABgPcacuXw9ZikMGzZMw4YNK/a1jIwMt593795dqjl+i3v+AAAALITKHwAAsBybOXP5eszygMofAACAhVD5AwAA1lOG7vk736j8AQAAWAiVPwAAYDk255nL12OWByR/AADAelj2BQAAgBVQ+QMAANZTRh7vFghU/gAAACyEyh8AALAcmzGy+fgePV+P5y9U/gAAACyEyh8AALAedvsCAADACqj8AQAA6zGSfH0oc/ko/JH8AQAA62HDBwAAACyByh8AALAeIz9s+PDtcP5C5Q8AAMBCqPwBAADrsfBRLxdE8lch/5QqBFPE/L3chpGBDqHMCs8uDHQIZVZhdMVAh1Bm2X8pCHQIAHDOLojkDwAAwCtOSTY/jFkOUC4DAACwECp/AADAcqx8zh/JHwAAsB4Lb/hg2RcAAMBCqPwBAADrofIHAAAAK6DyBwAArIfKHwAAAKyAyh8AALAeDnkGAACAFVD5AwAAlsMhzwAAAFbChg8AAABYAZU/AABgPU4j2XxcqXNS+QMAAEAZQ+UPAABYD/f8AQAAwAqo/AEAAAvyQ+VPVP4AAABQxlD5AwAA1mPhe/5I/gAAgPU4jXy+TMtRLwAAAChrqPwBAADrMc4zl6/HLAeo/AEAAFgIlT8AAGA9Ft7wQeUPAADAQqj8AQAA62G3b+Ds3btXd911l6pWraqwsDC1aNFCX3/9daDDAgAAuCAFtPJ39OhRderUSd26ddPnn3+u6tWra9u2bapcuXIgwwIAABc6C9/zF9Dkb8KECYqLi9PMmTNdbfXq1QtgRAAAwBKM/JD8+XY4fwnosu8nn3yitm3b6rbbblONGjV06aWXasaMGSX2LygoUG5urtsFAAAAzwU0+du5c6emTp2qhg0batGiRXrggQf00EMP6e233y62f1pamqKjo11XXFzceY4YAABcEM4u+/r6KgcCmvw5nU5ddtllGjdunC699FLdd999GjJkiKZNm1Zs/5SUFOXk5LiurKys8xwxAABA+RbQe/5q1aqlpk2burU1adJEH374YbH97Xa77Hb7+QgNAABcyJxOST5+HJuTx7v9qU6dOmnr1q1ubT/++KPi4+MDFBEAAMCFLaCVv1GjRqljx44aN26cbr/9dq1Zs0bTp0/X9OnTAxkWAAC40Fn4qJeAVv4uv/xyffTRR/rnP/+p5s2b65lnntGkSZPUr1+/QIYFAABwwQr4491uuOEG3XDDDYEOAwAAWImFK38BT/4AAADOO57tCwAAACug8gcAACzHGKeM8e3RLL4ez1+o/AEAAFgIlT8AAGA9xvj+Hr1ysuGDyh8AAICFUPkDAADWY/yw25fKHwAAAMoaKn8AAMB6nE7J5uPdueVkty/JHwAAsB6WfQEAAGAFVP4AAIDlGKdTxsfLvhzyDAAAgDKHyh8AALAe7vkDAACAFVD5AwAA1uM0ko3KHwAAAC5wVP4AAID1GCPJ14c8U/kDAABAGUPlDwAAWI5xGhkf3/Nnyknlj+QPAABYj3HK98u+HPIMAACAMobkDwAAWI5xGr9cpTFlyhQlJCQoNDRU7du315o1a/6w/wcffKDGjRsrNDRULVq00IIFC7yaj+QPAAAgQObOnavk5GSlpqZq/fr1atWqlZKSknTw4MFi+69atUp9+/bVvffeq2+++Ua9e/dW79699d1333k8J8kfAACwHuP0z+WliRMnasiQIRo4cKCaNm2qadOmKTw8XG+99Vax/SdPnqxrr71Wf//739WkSRM988wzuuyyy/Tqq696PGe53vBxdlfNaUdBgCMpm06fqhjoEMqs06cLAx1CmXX6lCPQIZRZwaf5b01JbOZUoENAOXJaZ/68BHJ37Gmd8vmjfc9+rtzcXLd2u90uu91epH9hYaHWrVunlJQUV1tQUJASExOVmZlZ7ByZmZlKTk52a0tKStLHH3/scZzlOvk7fvy4JGn595MCG0hZtTHQAQAAULLjx48rOjr6vM4ZEhKimJgYrcj27j45T0VERCguLs6tLTU1VaNHjy7S9/Dhw3I4HKpZs6Zbe82aNbVly5Zix8/Ozi62f3Z2tscxluvkr3bt2srKylJkZKRsNlugw1Fubq7i4uKUlZWlqKioQIdTZvC9lIzvpmR8NyXjuykZ303JytJ3Y4zR8ePHVbt27fM+d2hoqHbt2qXCQv+sABljiuQkxVX9AqlcJ39BQUGKjY0NdBhFREVFBfxfrLKI76VkfDcl47spGd9NyfhuSlZWvpvzXfH7rdDQUIWGhgZs/rOqVaum4OBgHThwwK39wIEDiomJKfY9MTExXvUvDhs+AAAAAiAkJERt2rRRenq6q83pdCo9PV0dOnQo9j0dOnRw6y9JixcvLrF/ccp15Q8AAKA8S05O1oABA9S2bVu1a9dOkyZNUn5+vgYOHChJ6t+/v+rUqaO0tDRJ0ogRI9SlSxe99NJLuv766zVnzhx9/fXXmj59usdzkvz5kN1uV2pqaplb2w80vpeS8d2UjO+mZHw3JeO7KRnfTdnUp08fHTp0SE8//bSys7PVunVrLVy40LWpY8+ePQoK+t9CbceOHTV79mw9+eSTevzxx9WwYUN9/PHHat68ucdz2kx5eQoxAAAAzhn3/AEAAFgIyR8AAICFkPwBAABYCMkfAACAhZD8+ciUKVOUkJCg0NBQtW/fXmvWrAl0SGXC8uXL1atXL9WuXVs2m82rZw9eyNLS0nT55ZcrMjJSNWrUUO/evbV169ZAh1UmTJ06VS1btnQdRNuhQwd9/vnngQ6rzBk/frxsNptGjhwZ6FDKhNGjR8tms7ldjRs3DnRYZcLevXt11113qWrVqgoLC1OLFi309ddfBzosBBDJnw/MnTtXycnJSk1N1fr169WqVSslJSXp4MGDgQ4t4PLz89WqVStNmTIl0KGUKcuWLdPQoUO1evVqLV68WKdOnVKPHj2Un58f6NACLjY2VuPHj9e6dev09ddf6+qrr9ZNN92k77//PtChlRlr167V66+/rpYtWwY6lDKlWbNm2r9/v+tasWJFoEMKuKNHj6pTp06qWLGiPv/8c/3www966aWXVLly5UCHhgDiqBcfaN++vS6//HK9+uqrks6czh0XF6fhw4frscceC3B0ZYfNZtNHH32k3r17BzqUMufQoUOqUaOGli1bpquuuirQ4ZQ5VapU0QsvvKB777030KEEXF5eni677DK99tprevbZZ9W6dWtNmjQp0GEF3OjRo/Xxxx9rw4YNgQ6lTHnssce0cuVKffnll4EOBWUIlb9zVFhYqHXr1ikxMdHVFhQUpMTERGVmZgYwMpQnOTk5ks4kOfgfh8OhOXPmKD8/36tHF13Ihg4dquuvv97tvzk4Y9u2bapdu7Yuvvhi9evXT3v27Al0SAH3ySefqG3btrrttttUo0YNXXrppZoxY0agw0KAkfydo8OHD8vhcLhO4j6rZs2ays7ODlBUKE+cTqdGjhypTp06eXVC+4Vs06ZNioiIkN1u1/3336+PPvpITZs2DXRYATdnzhytX7/e9Zgn/E/79u01a9YsLVy4UFOnTtWuXbvUuXNnHT9+PNChBdTOnTs1depUNWzYUIsWLdIDDzyghx56SG+//XagQ0MA8Xg3IMCGDh2q7777jvuTfqNRo0basGGDcnJyNG/ePA0YMEDLli2zdAKYlZWlESNGaPHixQoNDQ10OGVOz549Xf/csmVLtW/fXvHx8Xr//fctfbuA0+lU27ZtNW7cOEnSpZdequ+++07Tpk3TgAEDAhwdAoXK3zmqVq2agoODdeDAAbf2AwcOKCYmJkBRobwYNmyY/v3vf2vp0qWKjY0NdDhlRkhIiBo0aKA2bdooLS1NrVq10uTJkwMdVkCtW7dOBw8e1GWXXaYKFSqoQoUKWrZsmf7xj3+oQoUKcjgcgQ6xTKlUqZIuueQSbd++PdChBFStWrWK/KWpSZMmLIlbHMnfOQoJCVGbNm2Unp7uanM6nUpPT+ceJZTIGKNhw4bpo48+0n/+8x/Vq1cv0CGVaU6nUwUFBYEOI6C6d++uTZs2acOGDa6rbdu26tevnzZs2KDg4OBAh1im5OXlaceOHapVq1agQwmoTp06FTlG6scff1R8fHyAIkJZwLKvDyQnJ2vAgAFq27at2rVrp0mTJik/P18DBw4MdGgBl5eX5/Y37127dmnDhg2qUqWK6tatG8DIAmvo0KGaPXu25s+fr8jISNf9odHR0QoLCwtwdIGVkpKinj17qm7dujp+/Lhmz56tjIwMLVq0KNChBVRkZGSRe0IvuugiVa1alXtFJT388MPq1auX4uPjtW/fPqWmpio4OFh9+/YNdGgBNWrUKHXs2FHjxo3T7bffrjVr1mj69OmaPn16oENDIBn4xCuvvGLq1q1rQkJCTLt27czq1asDHVKZsHTpUiOpyDVgwIBAhxZQxX0nkszMmTMDHVrADRo0yMTHx5uQkBBTvXp10717d/PFF18EOqwyqUuXLmbEiBGBDqNM6NOnj6lVq5YJCQkxderUMX369DHbt28PdFhlwqeffmqaN29u7Ha7ady4sZk+fXqgQ0KAcc4fAACAhXDPHwAAgIWQ/AEAAFgIyR8AAICFkPwBAABYCMkfAACAhZD8AQAAWAjJHwAAgIWQ/AEAAFgIyR8An5k1a5YqVaoU6DB87p577lHv3r3/sE9GRoZsNpuOHTt2XmICgNIi+QPKuOISj3nz5ik0NFQvvfSSX+a8UJO40po8ebJmzZrl+rlr164aOXKkW5+OHTtq//79io6OPr/BAYCXKgQ6AADeeeONNzR06FBNmzZNAwcODHQ4luBJQhcSEqKYmJjzEA0AnBsqf0A58vzzz2v48OGaM2eOW+I3f/58XXbZZQoNDdXFF1+sMWPG6PTp05KkQYMG6YYbbnAb59SpU6pRo4befPPNInNkZGRo4MCBysnJkc1mk81m0+jRoyVJR48eVf/+/VW5cmWFh4erZ8+e2rZtW4nxHjp0SG3bttXNN9+sgoICOZ1OpaWlqV69egoLC1OrVq00b948t7ltNpvS09PVtm1bhYeHq2PHjtq6dWuJc+zevVs2m01z5sxRx44dFRoaqubNm2vZsmVu/ZYtW6Z27drJbrerVq1aeuyxx1zfkXSmmtqiRQuFhYWpatWqSkxMVH5+viT36us999yjZcuWafLkya7vZ/fu3cUu+3744Ydq1qyZ7Ha7EhISilRqExISNG7cOA0aNEiRkZGqW7eupk+fXuJnBQCfMADKtAEDBpibbrrJPPLIIyYiIsIsWbLE7fXly5ebqKgoM2vWLLNjxw7zxRdfmISEBDN69GhjjDErV640wcHBZt++fa73/Otf/zIXXXSROX78eJH5CgoKzKRJk0xUVJTZv3+/2b9/v6vfjTfeaJo0aWKWL19uNmzYYJKSkkyDBg1MYWGhMcaYmTNnmujoaGOMMXv27DGNGjUyAwYMMKdPnzbGGPPss8+axo0bm4ULF5odO3aYmTNnGrvdbjIyMowxxixdutRIMu3btzcZGRnm+++/N507dzYdO3Ys8fvZtWuXkWRiY2PNvHnzzA8//GAGDx5sIiMjzeHDh40xxvz8888mPDzcPPjgg2bz5s3mo48+MtWqVTOpqanGGGP27dtnKlSoYCZOnGh27dplNm7caKZMmeL63Gd/B8YYc+zYMdOhQwczZMgQ1/dz+vRpV+xHjx41xhjz9ddfm6CgIDN27FizdetWM3PmTBMWFmZmzpzpij0+Pt5UqVLFTJkyxWzbts2kpaWZoKAgs2XLlj/8MwEA54LkDyjjBgwYYEJCQowkk56eXuT17t27m3Hjxrm1vfvuu6ZWrVqun5s2bWomTJjg+rlXr17mnnvuKXHO3yZxZ/34449Gklm5cqWr7fDhwyYsLMy8//77bu/bsmWLiYuLMw899JBxOp3GGGNOnjxpwsPDzapVq9zGvffee03fvn2NMf9L/n6b4H722WdGkvn111+LjfVs8jd+/HhX26lTp0xsbKzrMz/++OOmUaNGrliMMWbKlCkmIiLCOBwOs27dOiPJ7N69u9g5fpv8GWNMly5dzIgRI9z6/D75u/POO80111zj1ufvf/+7adq0qevn+Ph4c9ddd7l+djqdpkaNGmbq1KnFxgEAvsCyL1AOtGzZUgkJCUpNTVVeXp7ba99++63Gjh2riIgI1zVkyBDt379fJ06ckCQNHjxYM2fOlCQdOHBAn3/+uQYNGuRVDJs3b1aFChXUvn17V1vVqlXVqFEjbd682dX266+/qnPnzrrllltcS6OStH37dp04cULXXHONW6zvvPOOduzYUeTznlWrVi1J0sGDB/8wvg4dOrj+uUKFCmrbtq0rrs2bN6tDhw6uWCSpU6dOysvL088//6xWrVqpe/fuatGihW677TbNmDFDR48e9er7+b3NmzerU6dObm2dOnXStm3b5HA4XG2//aw2m00xMTF/+lkB4FyQ/AHlQJ06dZSRkaG9e/fq2muv1fHjx12v5eXlacyYMdqwYYPr2rRpk7Zt26bQ0FBJUv/+/bVz505lZmbq//7v/1SvXj117tzZL7Ha7XYlJibq3//+t/bu3esWpyR99tlnbrH+8MMPbvf9SVLFihVd/3w2YXM6nX6JV5KCg4O1ePFiff7552ratKleeeUVNWrUSLt27fLbnGf99rNKZz6vPz8rAJD8AeVEfHy8li1bpuzsbLcE8LLLLtPWrVvVoEGDIldQ0Jl/xatWrarevXtr5syZmjVr1p/uEg4JCXGrTklSkyZNdPr0aX311Veutl9++UVbt25V06ZNXW1BQUF699131aZNG3Xr1k379u2TJDVt2lR2u1179uwpEmdcXNw5fz+rV692/fPp06e1bt06NWnSxBV7ZmamjDGuPitXrlRkZKRiY2MlnUm6OnXqpDFjxuibb75RSEiIPvroo2LnKu77+b0mTZpo5cqVbm0rV67UJZdcouDg4FJ9RgDwBY56AcqRuLg4ZWRkqFu3bkpKStLChQv19NNP64YbblDdunX1l7/8RUFBQfr222/13Xff6dlnn3W9d/DgwbrhhhvkcDg0YMCAP5wnISFBeXl5Sk9PV6tWrRQeHq6GDRvqpptu0pAhQ/T6668rMjJSjz32mOrUqaObbrrJ7f3BwcF677331LdvX1199dXKyMhQTEyMHn74YY0aNUpOp1NXXnmlcnJytHLlSkVFRf1pTH9mypQpatiwoZo0aaKXX35ZR48edS1tP/jgg5o0aZKGDx+uYcOGaevWrUpNTVVycrKCgoL01VdfKT09XT169FCNGjX01Vdf6dChQ67ksbjv56uvvtLu3bsVERGhKlWqFOnzt7/9TZdffrmeeeYZ9enTR5mZmXr11Vf12muvndPnBIBzFuibDgH8sd9vNjDmzO7Vhg0bmiuuuMLk5OSYhQsXmo4dO5qwsDATFRVl2rVrZ6ZPn+72HqfTaeLj4811113n0bz333+/qVq1qpHk2hV75MgRc/fdd5vo6GgTFhZmkpKSzI8//uh6z+83ipw6dcrccsstpkmTJubAgQPG6XSaSZMmmUaNGpmKFSua6tWrm6SkJLNs2TJjTNFNE8YY88033xhJZteuXcXGeXbDx+zZs027du1MSEiIadq0qfnPf/7j1i8jI8NcfvnlJiQkxMTExJhHH33UnDp1yhhjzA8//GCSkpJM9erVjd1uN5dccol55ZVXXO/9/e9g69at5oorrjBhYWGu2IqLfd68eaZp06amYsWKpm7duuaFF15wiyk+Pt68/PLLbm2tWrVyfd8A4A82Y36zDgLggpWXl6c6depo5syZuuWWWwIdjs/s3r1b9erV0zfffKPWrVsHOhwAKPNY9gUucE6nU4cPH9ZLL72kSpUq6cYbbwx0SACAACL5Ay5we/bsUb169RQbG6tZs2apQgX+tQcAK2PZFwAAwEI46gUAAMBCSP4AAAAshOQPAADAQkj+AAAALITkDwAAwEJI/gAAACyE5A8AAMBCSP4AAAAs5P8BdBQhI58UHFwAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAn8AAAIjCAYAAAByAn13AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABUaElEQVR4nO3df3zNdf/H8efZ2NnWbH4bNpv8/q2IkJBlqZTqiqSI6KoQdnVV69dQGf0QV35FRfXNRekq/ZBilxEmRUpXyM8szI+w2bBxzvv7h8u5Om2rc+YcZ/N53G+3z+3W3ud93u/XOaNevd6f9/tjM8YYAQAAwBKCAh0AAAAALhySPwAAAAsh+QMAALAQkj8AAAALIfkDAACwEJI/AAAACyH5AwAAsBCSPwAAAAsh+QMAALAQkj+gFOnatau6du16QeeMj4/XPffc49a2bds29ejRQ1FRUbLZbPrwww8vaEyBYLPZNGbMmECHIUlKT0+XzWbTwoULAx2Ky9y5c2Wz2bR79+5AhwLgPJH8AShk4MCB2rRpk5577jm9/fbbatu27XmN9+OPP2rMmDFFJg7Tp0/X3Llzz2t8Ty1evLjUJHgXq7S0NA0ePFgNGzZUeHi4Lr30Ug0ZMkT79+8PdGgA/qtcoAMAULqcPHlSGRkZeuKJJzR8+HCfjPnjjz9q7Nix6tq1q+Lj491emz59uqpWrVqo+ugPixcv1rRp04pMAE+ePKly5fhX4vl69NFHdeTIEd1+++1q0KCBdu7cqalTp+qTTz7Rxo0bFR0dHegQAcvj33QA3Bw6dEiSVLFixcAGcoGFhoYGOoSLwqRJk3TVVVcpKOh/C0vXXXedunTpoqlTp+rZZ58NYHQAJJZ9cZHZu3ev7r33XtWqVUt2u11169bVAw88oIKCAknSkSNH9PDDD6tFixaKiIhQZGSkevbsqe+++85tnOLubzp3L1Z6erqrbdu2bbrtttsUHR2t0NBQxcTE6I477lB2drarz5w5c3TNNdeoevXqstvtatq0qWbMmFHiz7l06VJdddVVqlixoiIiItSoUSM9/vjjbn3y8/OVkpKi+vXry263KzY2Vo888ojy8/OLHXfMmDGKi4uTJP3973+XzWYrVKn7rZ9//lkPPvigGjVqpLCwMFWpUkW333672/c2d+5c3X777ZKkbt26yWazub7D+Ph4/ec//9GKFStc7b+95/HYsWMaNWqUYmNjZbfbVb9+fU2cOFFOp9PVZ/fu3bLZbHrxxRc1a9Ys1atXT3a7XVdccYW+/vprV7977rlH06ZNkyTXXDabzfV6Uff8ffvtt+rZs6ciIyMVERGh7t27a+3atW59zv1ZWb16tZKSklStWjVdcskluuWWW1yJdEk5nU4999xziomJUWhoqLp3767t27cX6vfVV1/puuuuU1RUlMLDw9WlSxetXr3arY8nv6tz/vOf/+iaa65RWFiYYmJi9Oyzz7p953/k6quvdkv8zrVVrlxZmzdv9vzDA/AbKn+4aOzbt0/t2rXTsWPHdN9996lx48bau3evFi5cqBMnTigkJEQ7d+7Uhx9+qNtvv11169bVgQMH9Oqrr6pLly768ccfVatWLa/mLCgoUGJiovLz8zVixAhFR0dr7969+uSTT3Ts2DFFRUVJkmbMmKFmzZrppptuUrly5fTxxx/rwQcflNPp1LBhw7ya8z//+Y9uvPFGtWzZUuPGjZPdbtf27dvd/mPvdDp10003adWqVbrvvvvUpEkTbdq0SS+//LJ++umnYjdw3HrrrapYsaJGjx6tfv366frrr1dERESxsXz99ddas2aN7rjjDsXExGj37t2aMWOGunbtqh9//FHh4eG6+uqr9dBDD+kf//iHHn/8cTVp0kSS1KRJE02ePFkjRoxQRESEnnjiCUlSjRo1JEknTpxQly5dtHfvXv31r39VnTp1tGbNGiUnJ2v//v2aPHmyWyzz5s3T8ePH9de//lU2m03PP/+8br31Vu3cuVPly5fXX//6V+3bt09Lly7V22+/7dH33LlzZ0VGRuqRRx5R+fLl9eqrr6pr165asWKF2rdv79Z/xIgRqlSpklJSUrR7925NnjxZw4cP14IFC/50ruJMmDBBQUFBevjhh5Wdna3nn39e/fv311dffeXq8+9//1s9e/ZUmzZtlJKSoqCgINf/bHz55Zdq166dx78rScrKylK3bt105swZPfbYY7rkkks0a9YshYWFlfhz5ObmKjc3V1WrVi3xGAB8yAAXiQEDBpigoCDz9ddfF3rN6XQaY4w5deqUcTgcbq/t2rXL2O12M27cOFfbnDlzjCSza9cut77Lly83kszy5cuNMcZ8++23RpJ57733/jC2EydOFGpLTEw0l156qVtbly5dTJcuXf5wrJdfftlIMocOHSq2z9tvv22CgoLMl19+6dY+c+ZMI8msXr3a1RYXF2cGDhzo+nnXrl1GknnhhRf+MA5jiv5cGRkZRpJ56623XG3vvfee2/f2W82aNSvyMz/zzDPmkksuMT/99JNb+2OPPWaCg4PNnj173OKtUqWKOXLkiKvfokWLjCTz8ccfu9qGDRtmivvXniSTkpLi+rl3794mJCTE7Nixw9W2b98+U6FCBXP11Ve72s79WUlISHD9OTPGmNGjR5vg4GBz7NixIuf7I+f+nDVp0sTk5+e72qdMmWIkmU2bNhljzv65btCggUlMTHSb+8SJE6Zu3brm2muvdWv7vaJ+V6NGjTKSzFdffeVqO3jwoImKiiry74QnnnnmGSPJpKWlef1eAL7Hsi8uCk6nUx9++KF69epV5M7Uc8t7drvdtSTlcDj066+/upZNN2zY4PW85yp7n3/+uU6cOFFsv99WTbKzs3X48GF16dJFO3fudFse9sS5e/EWLVpU7FLce++9pyZNmqhx48Y6fPiw67rmmmskScuXL/dqzuL89nOdPn1av/76q+rXr6+KFSuW6Pv8rffee0+dO3dWpUqV3D5DQkKCHA6HVq5c6da/b9++qlSpkuvnzp07S5J27tzp9dwOh0NffPGFevfurUsvvdTVXrNmTd15551atWqVcnJy3N5z3333uS0jd+7cWQ6HQz///LPX858zaNAghYSEuI0p/e8zbdy4Udu2bdOdd96pX3/91fUd5eXlqXv37lq5cqXrz4inv6vFixfryiuvdFUMJalatWrq379/iT7DypUrNXbsWPXp08f15w9AYLHsi4vCoUOHlJOTo+bNm/9hP6fTqSlTpmj69OnatWuXHA6H67UqVap4PW/dunWVlJSkSZMm6Z133lHnzp1100036a677nIlhpK0evVqpaSkKCMjo1CSmJ2d7db3z/Tt21evvfaahgwZoscee0zdu3fXrbfeqr/85S+uxHbbtm3avHmzqlWrVuQYBw8e9Hg+h8NR6N61ypUrKyQkRCdPnlRqaqrmzJmjvXv3yhjj9rnOx7Zt2/T99997/Bnq1Knj9vO5RPDo0aNez33o0CGdOHFCjRo1KvRakyZN5HQ6lZmZqWbNmvllfk/H3LZtm6SzR/MUJzs7W5UqVfL4d/Xzzz8XWtKWVOR38We2bNmiW265Rc2bN9drr73m9fsB+AfJHyxl/PjxeuqppzR48GA988wzqly5soKCgjRq1Ci3KtpvKzi/9dtk8ZyXXnpJ99xzjxYtWqQvvvhCDz30kFJTU7V27VrFxMRox44d6t69uxo3bqxJkyYpNjZWISEhWrx4sV5++WWPb6Q/JywsTCtXrtTy5cv16aefasmSJVqwYIGuueYaffHFFwoODpbT6VSLFi00adKkIseIjY31eL7MzEzVrVvXrW358uXq2rWrRowYoTlz5mjUqFHq0KGD61DoO+64w+vP9XtOp1PXXnutHnnkkSJfb9iwodvPwcHBRfb7bZLjT/6Y/8/GPPcdv/DCC2rdunWRfc/ds+nP31VRMjMzXQeFL168WBUqVPD5HABKhuQPF4Vq1aopMjJSP/zwwx/2W7hwobp166bXX3/drf3YsWNuN6Ofq7AcO3bMrV9xS3gtWrRQixYt9OSTT2rNmjXq1KmTZs6cqWeffVYff/yx8vPz9dFHH7lVcs5n6TUoKEjdu3dX9+7dNWnSJI0fP15PPPGEli9froSEBNWrV0/fffedunfvXmwi66no6GgtXbrUra1Vq1aSzn6fAwcO1EsvveR67dSpU4W+tz+KobjX6tWrp9zcXCUkJJQwcs/n+r1q1aopPDxcW7duLfTali1bFBQU5FUC7S/16tWTJEVGRv7p9+Tp7youLs5VUfytor6L4vz666/q0aOH8vPzlZaWppo1a3r8XgD+xz1/uCgEBQWpd+/e+vjjj/XNN98Uev1cpSQ4OLhQJea9997T3r173drO/Uf1t/eVORwOzZo1y61fTk6Ozpw549bWokULBQUFuY5UOVe9+f0y25w5c7z6jOccOXKkUNu5qs+5Ofv06aO9e/dq9uzZhfqePHlSeXl5Hs8XGhqqhIQEt+tcclzU9/nKK68UqpBecsklkgon0+deK6q9T58+ysjI0Oeff17otWPHjhX63j3xR3H8VnBwsHr06KFFixa5HYVy4MABzZs3T1dddZUiIyO9nt/X2rRpo3r16unFF19Ubm5uodd/u1zv6e/q+uuv19q1a7Vu3Tq3cd555x2PYsrLy9P111+vvXv3avHixWrQoIE3HwnABUDlDxeN8ePH64svvlCXLl1cx5vs379f7733nlatWqWKFSvqxhtv1Lhx4zRo0CB17NhRmzZt0jvvvON2U78kNWvWTFdeeaWSk5N15MgRVa5cWfPnzy+UcPz73//W8OHDdfvtt6thw4Y6c+aM3n77bQUHB+u2226TJPXo0UMhISHq1auX/vrXvyo3N1ezZ89W9erVS/TIq3HjxmnlypW64YYbFBcXp4MHD2r69OmKiYnRVVddJUm6++679e677+r+++/X8uXL1alTJzkcDm3ZskXvvvuuPv/88/N+ZJsk3XjjjXr77bcVFRWlpk2bKiMjQ8uWLSt0/2Tr1q0VHBysiRMnKjs7W3a73XXuYZs2bTRjxgw9++yzql+/vqpXr65rrrlGf//73/XRRx/pxhtv1D333KM2bdooLy9PmzZt0sKFC7V7926vjw5p06aNJOmhhx5SYmKigoODdccddxTZ99lnn3Wdp/jggw+qXLlyevXVV5Wfn6/nn3++RN/X3LlzNWjQIM2ZM8cnTzQJCgrSa6+9pp49e6pZs2YaNGiQateurb1792r58uWKjIzUxx9/LMnz39Ujjzyit99+W9ddd51GjhzpOuolLi5O33///Z/G1L9/f61bt06DBw/W5s2b3c72i4iIUO/evc/7cwM4T4HaZgz4w88//2wGDBhgqlWrZux2u7n00kvNsGHDXMdlnDp1yvztb38zNWvWNGFhYaZTp04mIyOjyCNWduzYYRISEozdbjc1atQwjz/+uFm6dKnbkSU7d+40gwcPNvXq1TOhoaGmcuXKplu3bmbZsmVuY3300UemZcuWJjQ01MTHx5uJEyeaN954o9DRGZ4c9ZKWlmZuvvlmU6tWLRMSEmJq1apl+vXrV+hIlIKCAjNx4kTTrFkzY7fbTaVKlUybNm3M2LFjTXZ2tqvf+Rz1cvToUTNo0CBTtWpVExERYRITE82WLVsKjWmMMbNnzzaXXnqpCQ4OdvsOs7KyzA033GAqVKhgJLl9/uPHj5vk5GRTv359ExISYqpWrWo6duxoXnzxRVNQUPCn8ep3x7ecOXPGjBgxwlSrVs3YbDa3Y19+39cYYzZs2GASExNNRESECQ8PN926dTNr1qxx63PuqJffHzH0+2OBjDHmlVdeMZLMkiVL/vB7Pffe3x8hdO6zzpkzx63922+/NbfeequpUqWKsdvtJi4uzvTp08ftaBVvflfff/+96dKliwkNDTW1a9c2zzzzjHn99dc9OuolLi7OSCryiouL+8P3ArgwbMZcoLuhAcDi+vTpo927d7stqQLAhcayLwBcAMYYpaen6//+7/8CHQoAi6PyBwAAYCHs9gUAALAQkj8AAIAAWblypXr16qVatWrJZrPpww8//NP3pKen6/LLL5fdblf9+vU1d+5cr+Yk+QMAAAiQvLw8tWrVStOmTfOo/65du3TDDTeoW7du2rhxo0aNGqUhQ4YUeSZqcbjnDwAAoBSw2Wz64IMP/vA8zEcffVSffvqp2xOt7rjjDh07dkxLlizxaJ4yvdvX6XRq3759qlChwnk/wgoAAFwYxhgdP35ctWrVUlDQhV+EPHXqlAoKCvwytjGmUE5it9tlt9t9Mn5GRkahxzkmJiZq1KhRHo9RppO/ffv2lYrnawIAAO9lZmYqJibmgs556tQp1Y2LUNZBx593LoGIiIhCj1tMSUnRmDFjfDJ+VlaWatSo4dZWo0YN5eTk6OTJkwoLC/vTMcp08lehQgVJ0s8b4hUZwe2Lv3dLwxaBDgEAgELO6LRWabHrv+MXUkFBgbIOOvTz+nhFVvBt7pBz3Km4NruVmZnp9vxvX1X9fKVMJ3/nyqqREUE+/wVeDMrZygc6BAAACvvvboNA3rIVUcGmiAq+nd+p/+YlkZFuyZ8vRUdH68CBA25tBw4cUGRkpEdVP6mMJ38AAAAl4TBOOXy85dVhnL4dsAgdOnTQ4sWL3dqWLl2qDh06eDwG5TIAAIAAyc3N1caNG7Vx40ZJZ49y2bhxo/bs2SNJSk5O1oABA1z977//fu3cuVOPPPKItmzZounTp+vdd9/V6NGjPZ6Tyh8AALAcp4yc8m3pryTjffPNN+rWrZvr56SkJEnSwIEDNXfuXO3fv9+VCEpS3bp19emnn2r06NGaMmWKYmJi9NprrykxMdHjOUn+AAAAAqRr1676oyOXi3p6R9euXfXtt9+WeE6SPwAAYDlOOeXrO/R8P6J/cM8fAACAhVD5AwAAluMwRg4fP+HW1+P5C5U/AAAAC6HyBwAALKe07PYNBJI/AABgOU4ZOSya/LHsCwAAYCFU/gAAgOVYedmXyh8AAICFUPkDAACWw1EvAAAAsAQqfwAAwHKc/718PWZZQOUPAADAQqj8AQAAy3H44Zw/X4/nLyR/AADAchzm7OXrMcsCln0BAAAshMofAACwHDZ8AAAAwBKo/AEAAMtxyiaHbD4fsyyg8gcAAGAhVP4AAIDlOM3Zy9djlgVU/gAAACykVCR/06ZNU3x8vEJDQ9W+fXutW7cu0CEBAICLmOO/9/z5+ioLAp78LViwQElJSUpJSdGGDRvUqlUrJSYm6uDBg4EODQAAXKRI/gJo0qRJGjp0qAYNGqSmTZtq5syZCg8P1xtvvBHo0AAAAC46Ad3wUVBQoPXr1ys5OdnVFhQUpISEBGVkZBTqn5+fr/z8fNfPOTk5FyROAABwcXEam5zGx0e9+Hg8fwlo5e/w4cNyOByqUaOGW3uNGjWUlZVVqH9qaqqioqJcV2xs7IUKFQAA4KIQ8GVfbyQnJys7O9t1ZWZmBjokAABQBln5nr+ALvtWrVpVwcHBOnDggFv7gQMHFB0dXai/3W6X3W6/UOEBAABcdAJa+QsJCVGbNm2UlpbmanM6nUpLS1OHDh0CGBkAALiYORTkl6ssCPgTPpKSkjRw4EC1bdtW7dq10+TJk5WXl6dBgwYFOjQAAICLTsCTv759++rQoUN6+umnlZWVpdatW2vJkiWFNoEAAAD4ivHDbl9TRnb7Bjz5k6Thw4dr+PDhgQ4DAABYhD82aJSVDR9lY3EaAAAAPlEqKn8AAAAXksMEyWF8WwNzGJ8O5zdU/gAAACyEyh8AALAcp2xy+rgG5lTZKP1R+QMAALAQKn8AAMBy2O0LAAAAS6DyBwAALMc/u33Lxj1/JH8AAMByzm748O0yra/H8xeWfQEAACyEyh8AALAcp4Lk4KgXAAAAXOyo/AEAAMux8oYPKn8AAAAWQuUPAABYjlNBPN4NAAAAFz8qfwAAwHIcxiaH8fHj3Xw8nr+Q/AEAAMtx+OGoFwfLvgAAAChtqPwBAADLcZogOX181IuTo14AAABQ2lD5AwAAlsM9fwAAALAEKn8AAMBynPL90SxOn47mP1T+AAAALOSiqPy1W9dHweH2QIdR6pQbFRXoEEqt6MlrAh0CACCA/PN4t7JRU7sokj8AAABvOEyQHD4+6sXX4/lL2YgSAAAAPkHlDwAAWI5TNjnl6w0fZePZvlT+AAAALITKHwAAsBzu+QMAAIAlUPkDAACW45/Hu5WNmlrZiBIAAAA+QeUPAABYjtPY5PT14918PJ6/UPkDAACwECp/AADAcpx+uOePx7sBAACUUk4TJKePj2bx9Xj+UjaiBAAAgE9Q+QMAAJbjkE0OHz+Ozdfj+QuVPwAAAAuh8gcAACyHe/4AAABgCVT+AACA5Tjk+3v0HD4dzX+o/AEAAFgIlT8AAGA5Vr7nj+QPAABYjsMEyeHjZM3X4/lL2YgSAAAAPkHyBwAALMfIJqePL1PCDSTTpk1TfHy8QkND1b59e61bt+4P+0+ePFmNGjVSWFiYYmNjNXr0aJ06dcrj+Uj+AAAAAmTBggVKSkpSSkqKNmzYoFatWikxMVEHDx4ssv+8efP02GOPKSUlRZs3b9brr7+uBQsW6PHHH/d4TpI/AABgOefu+fP15a1JkyZp6NChGjRokJo2baqZM2cqPDxcb7zxRpH916xZo06dOunOO+9UfHy8evTooX79+v1ptfC3SP4AAAB8KCcnx+3Kz88vsl9BQYHWr1+vhIQEV1tQUJASEhKUkZFR5Hs6duyo9evXu5K9nTt3avHixbr++us9jo/dvgAAwHKcxian8e0hz+fGi42NdWtPSUnRmDFjCvU/fPiwHA6HatSo4dZeo0YNbdmypcg57rzzTh0+fFhXXXWVjDE6c+aM7r//fq+WfUn+AAAAfCgzM1ORkZGun+12u8/GTk9P1/jx4zV9+nS1b99e27dv18iRI/XMM8/oqaee8mgMkj8AAGA5DgXJ4eO7386NFxkZ6Zb8Fadq1aoKDg7WgQMH3NoPHDig6OjoIt/z1FNP6e6779aQIUMkSS1atFBeXp7uu+8+PfHEEwoK+vPPFNB7/lauXKlevXqpVq1astls+vDDDwMZDgAAsIhzy76+vrwREhKiNm3aKC0t7X9xOZ1KS0tThw4dinzPiRMnCiV4wcHBkiRjjEfzBjT5y8vLU6tWrTRt2rRAhgEAABAQSUlJmj17tt58801t3rxZDzzwgPLy8jRo0CBJ0oABA5ScnOzq36tXL82YMUPz58/Xrl27tHTpUj311FPq1auXKwn8MwFd9u3Zs6d69uwZyBAAAIAFORUkp49rYCUZr2/fvjp06JCefvppZWVlqXXr1lqyZIlrE8iePXvcKn1PPvmkbDabnnzySe3du1fVqlVTr1699Nxzz3k8Z5m65y8/P99tu3ROTk4AowEAADh/w4cP1/Dhw4t8LT093e3ncuXKKSUlRSkpKSWer0yd85eamqqoqCjX9fut1AAAAJ5wGJtfrrKgTCV/ycnJys7Odl2ZmZmBDgkAAKBMKVPLvna73adn5QAAAGvy5yHPpV2ZqvwBAADg/AS08pebm6vt27e7ft61a5c2btyoypUrq06dOgGMDAAAXMyMCZLT+LYGZnw8nr8ENPn75ptv1K1bN9fPSUlJkqSBAwdq7ty5AYoKAABc7ByyySHfLtP6ejx/CWjy17VrV49PowYAAMD5K1MbPgAAAHzBaXy/QcNZRupZZWNxGgAAAD5B5Q8AAFiO0w8bPnw9nr+UjSgBAADgE1T+AACA5Thlk9PHu3N9PZ6/UPkDAACwECp/AADAchzGJoePd/v6ejx/IfkDAACWw4YPAAAAWAKVPwAAYDlO2Xx/yDMbPgAAAFDaUPkDAACWY/xw1Iuh8gcAAIDShsofAACwHKfxwz1/ZeSoFyp/AAAAFkLlDwAAWI6Vz/kj+QMAAJbDsi8AAAAsgcofAACwHKcfjnrhkGcAAACUOlT+AACA5XDPHwAAACyByh8AALAcKn8AAACwBCp/AADAcqxc+bsokr9KCy5RufKhgQ6j1HGEOAMdQqm1a0KHQIdQatV9LCPQIQCA31k5+WPZFwAAwEIuisofAACAN4x8fyiz8elo/kPlDwAAwEKo/AEAAMvhnj8AAABYApU/AABgOVT+AAAAYAlU/gAAgOVYufJH8gcAACzHyskfy74AAAAWQuUPAABYjjE2GR9X6nw9nr9Q+QMAALAQKn8AAMBynLL5/PFuvh7PX6j8AQAAWAiVPwAAYDns9gUAAIAlUPkDAACWw25fAAAAWAKVPwAAYDlWvueP5A8AAFgOy74AAACwBCp/AADAcowfln2p/AEAAKDUofIHAAAsx0gyxvdjlgVU/gAAACyEyh8AALAcp2yyycdHvfh4PH+h8gcAAGAhVP4AAIDlWPmcP5I/AABgOU5jk40nfHguLS1NaWlpOnjwoJxOp9trb7zxhsfjpKam6l//+pe2bNmisLAwdezYURMnTlSjRo1KEhYAAAD+hNf3/I0dO1Y9evRQWlqaDh8+rKNHj7pd3lixYoWGDRumtWvXaunSpTp9+rR69OihvLw8b8MCAADwmDH+ucoCryt/M2fO1Ny5c3X33Xef9+RLlixx+3nu3LmqXr261q9fr6uvvvq8xwcAAIA7r5O/goICdezY0R+xKDs7W5JUuXLlIl/Pz89Xfn6+6+ecnBy/xAEAAC5uVt7w4fWy75AhQzRv3jyfB+J0OjVq1Ch16tRJzZs3L7JPamqqoqKiXFdsbKzP4wAAALiYeV35O3XqlGbNmqVly5apZcuWKl++vNvrkyZNKlEgw4YN0w8//KBVq1YV2yc5OVlJSUmun3NyckgAAQCA16xc+fM6+fv+++/VunVrSdIPP/zg9prNVrIPPXz4cH3yySdauXKlYmJiiu1nt9tlt9tLNAcAAABKkPwtX77cZ5MbYzRixAh98MEHSk9PV926dX02NgAAQHE456+EfvnlF0n6w2rdHxk2bJjmzZunRYsWqUKFCsrKypIkRUVFKSws7HxCAwAAKJY/jmYpK0e9eL3hw+l0aty4cYqKilJcXJzi4uJUsWJFPfPMM4UOfP4zM2bMUHZ2trp27aqaNWu6rgULFngbFgAAADzgdeXviSee0Ouvv64JEyaoU6dOkqRVq1ZpzJgxOnXqlJ577jmPxzJlJUUGAAAXlbOVP19v+PDpcH7jdfL35ptv6rXXXtNNN93kamvZsqVq166tBx980KvkDwAAABeW18nfkSNH1Lhx40LtjRs31pEjR3wSFAAAgD9Z+agXr+/5a9WqlaZOnVqoferUqWrVqpVPggIAAIB/eF35e/7553XDDTdo2bJl6tChgyQpIyNDmZmZWrx4sc8DBAAA8DXz38vXY5YFXlf+unTpop9++km33HKLjh07pmPHjunWW2/V1q1b1blzZ3/ECAAAAB8p0Tl/tWrVYmMHAAAos6x8z59Hyd/333+v5s2bKygoSN9///0f9m3ZsqVPAgMAAPAbC6/7epT8tW7dWllZWapevbpat24tm81W5Bl9NptNDofD50ECAADANzxK/nbt2qVq1aq5/hkAAKBM88Oyr8rIsq9HGz7i4uJks539QD///LNq167terTbuat27dr6+eef/RosAADAxWbatGmKj49XaGio2rdvr3Xr1v1h/2PHjmnYsGGqWbOm7Ha7GjZs6NWJK17v9u3WrVuRhzlnZ2erW7du3g4HAABwwZ19vJvvL28tWLBASUlJSklJ0YYNG9SqVSslJibq4MGDRfYvKCjQtddeq927d2vhwoXaunWrZs+erdq1a3s8p9e7fY0xrirgb/3666+65JJLvB0OAADAsiZNmqShQ4dq0KBBkqSZM2fq008/1RtvvKHHHnusUP833nhDR44c0Zo1a1S+fHlJUnx8vFdzepz83XrrrZLObuq45557ZLfbXa85HA59//336tixo1eTAwAABII/j3rJyclxa7fb7W550zkFBQVav369kpOTXW1BQUFKSEhQRkZGkXN89NFH6tChg4YNG6ZFixapWrVquvPOO/Xoo48qODjYozg9Tv6ioqIkna38VahQQWFhYa7XQkJCdOWVV2ro0KGeDgcAAHBRio2Ndfs5JSVFY8aMKdTv8OHDcjgcqlGjhlt7jRo1tGXLliLH3rlzp/7973+rf//+Wrx4sbZv364HH3xQp0+fVkpKikfxeZz8zZkzR9LZ0uLDDz/MEi8AACi7jM33u3P/O15mZqYiIyNdzUVV/UrK6XSqevXqmjVrloKDg9WmTRvt3btXL7zwgu+Tv3M8HRgAAKC0KukGjT8bU5IiIyPdkr/iVK1aVcHBwTpw4IBb+4EDBxQdHV3ke2rWrKny5cu7LfE2adJEWVlZKigoUEhIyJ/O61Hyd/nllystLU2VKlXSZZddVuSGj3M2bNjgyZAAAACWFhISojZt2igtLU29e/eWdLayl5aWpuHDhxf5nk6dOmnevHlyOp0KCjp7aMtPP/2kmjVrepT4SR4mfzfffLOrZHkuOAAAgDKrlDzeLSkpSQMHDlTbtm3Vrl07TZ48WXl5ea7dvwMGDFDt2rWVmpoqSXrggQc0depUjRw5UiNGjNC2bds0fvx4PfTQQx7P6VHy99ulXpZ9AQAAfKNv3746dOiQnn76aWVlZal169ZasmSJaxPInj17XBU+6exmks8//1yjR49Wy5YtVbt2bY0cOVKPPvqox3N6fc9fZmambDabYmJiJEnr1q3TvHnz1LRpU913333eDgcAAHDB+fOoF28NHz682GXe9PT0Qm0dOnTQ2rVrSzSXVIInfNx5551avny5JCkrK0sJCQlat26dnnjiCY0bN67EgQAAAMD/vE7+fvjhB7Vr106S9O6776pFixZas2aN3nnnHc2dO9fX8QEAAPiH8fFVRnid/J0+fdq1+WPZsmW66aabJEmNGzfW/v37fRsdAAAAfMrr5K9Zs2aaOXOmvvzySy1dulTXXXedJGnfvn2qUqWKzwMEAADwtXP3/Pn6Kgu8Tv4mTpyoV199VV27dlW/fv3UqlUrSWefNXduORgAAKBU8/WSbxla+vV6t2/Xrl11+PBh5eTkqFKlSq72++67T+Hh4T4NzlPlc8+oXLkzAZm7NDsZ59lhj1YUs6wg0CGUWnl/aR/oEEqtSxZ+FegQAOC8eZ38SVJwcLDOnDmjVatWSZIaNWqk+Ph4X8YFAADgR7b/Xr4es/Tzetk3Ly9PgwcPVs2aNXX11Vfr6quvVq1atXTvvffqxIkT/ogRAAAAPuJ18peUlKQVK1bo448/1rFjx3Ts2DEtWrRIK1as0N/+9jd/xAgAAOBb3PPnuffff18LFy5U165dXW3XX3+9wsLC1KdPH82YMcOX8QEAAMCHvE7+Tpw44Xre3G9Vr16dZV8AAFA2+KNSV0Yqf14v+3bo0EEpKSk6deqUq+3kyZMaO3asOnTo4NPgAAAA4FteV/6mTJmixMRExcTEuM74++677xQaGqrPP//c5wECAAD4nLGdvXw9ZhngdfLXvHlzbdu2Te+88462bNkiSerXr5/69++vsLAwnwcIAADga8acvXw9ZllQonP+wsPDNXToUF/HAgAAAD8rUfK3detWvfLKK9q8ebMkqUmTJho+fLgaN27s0+AAAAD8gg0fnnv//ffVvHlzrV+/Xq1atVKrVq20YcMGtWjRQu+//74/YgQAAICPeF35e+SRR5ScnKxx48a5taekpOiRRx7Rbbfd5rPgAAAA/MLCGz68rvzt379fAwYMKNR+1113af/+/T4JCgAAAP7hdfLXtWtXffnll4XaV61apc6dO/skKAAAAH+yGf9cZYHXy7433XSTHn30Ua1fv15XXnmlJGnt2rV67733NHbsWH300UdufQEAAFB6eJ38Pfjgg5Kk6dOna/r06UW+Jkk2m00Oh+M8wwMAAPADC+/29Tr5czqd/ogDAADgwmHDBwAAAKygRIc8AwAAlGkWXval8gcAAGAhVP4AAID1UPkDAACAFZSo8ud0OrV9+3YdPHiw0O7fq6++2ieBAQAA+I2FK39eJ39r167VnXfeqZ9//lnGuH9KzvYDAAAo3bxO/u6//361bdtWn376qWrWrCmbrWycaQMAAOBi4XP+vE7+tm3bpoULF6p+/fr+iAcAAAB+5PWGj/bt22v79u3+iAUAAOCCsBn/XGWB15W/ESNG6G9/+5uysrLUokULlS9f3u31li1b+iw4AAAAv2DDh+duu+02SdLgwYNdbTabTcYYNnwAAACUcl4nf7t27fLZ5DNmzNCMGTO0e/duSVKzZs309NNPq2fPnj6bAwAAAP/jdfIXFxfns8ljYmI0YcIENWjQQMYYvfnmm7r55pv17bffqlmzZj6bBwAAAGeV6Akfb7/9tjp16qRatWrp559/liRNnjxZixYt8mqcXr166frrr1eDBg3UsGFDPffcc4qIiNDatWtLEhYAAIBHbPLDho9AfygPeZ38zZgxQ0lJSbr++ut17Ngx1z1+FStW1OTJk0sciMPh0Pz585WXl6cOHToU2Sc/P185OTluFwAAADzndfL3yiuvaPbs2XriiScUHBzsam/btq02bdrkdQCbNm1SRESE7Ha77r//fn3wwQdq2rRpkX1TU1MVFRXlumJjY72eDwAAwHXIs6+vMsDr5G/Xrl267LLLCrXb7Xbl5eV5HUCjRo20ceNGffXVV3rggQc0cOBA/fjjj0X2TU5OVnZ2tuvKzMz0ej4AAAAr83rDR926dbVx48ZCGz+WLFmiJk2aeB1ASEiI62khbdq00ddff60pU6bo1VdfLdTXbrfLbrd7PQcAAIAbzvnzXFJSkoYNG6ZTp07JGKN169bpn//8p1JTU/Xaa6+dd0BOp1P5+fnnPQ4AAECxSP48N2TIEIWFhenJJ5/UiRMndOedd6pWrVqaMmWK7rjjDq/GSk5OVs+ePVWnTh0dP35c8+bNU3p6uj7//HNvwwIAAIAHvE7+cnJy1L9/f/Xv318nTpxQbm6uqlevLknavn27awnXEwcPHtSAAQO0f/9+RUVFqWXLlvr888917bXXehsWAACAx/zxLN6L9tm+N9xwg5YtWya73a7w8HCFh4dLkrZu3aru3bvrl19+8Xis119/3dvpAQAAcB683u0bERGhW265RWfOnHG1bd68WV27dnU99xcAAKBUM366ygCvk79//etfys7OVv/+/WWM0Q8//KCuXbuqX79+mjJlij9iBAAAgI94nfyFhYXp008/1datW9WnTx91795dAwYM0KRJk/wRHwAAgO9ZuPLn0T1/v3+MWlBQkBYsWKBrr71Wt912m5566ilXn8jISN9HCQAAAJ/wKPmrWLGibLbCjywxxmjmzJl69dVXZYyRzWZzPesXAACgtGK3759Yvny5v+MAAAC4cPzxLN4y8mxfj5K/Ll26+DsOAAAAXABen/MnSceOHdPrr7+uzZs3S5KaNWumwYMHKyoqyqfBAQAA+IWFH+/m9W7fb775RvXq1dPLL7+sI0eO6MiRI5o0aZLq1aunDRs2+CNGAAAA+IjXlb/Ro0frpptu0uzZs1Wu3Nm3nzlzRkOGDNGoUaO0cuVKnwcJAADgS2z48MI333zjlvhJUrly5fTII4+obdu2Pg0OAAAAvuX1sm9kZKT27NlTqD0zM1MVKlTwSVAAAAB+ZeFDnr1O/vr27at7771XCxYsUGZmpjIzMzV//nwNGTJE/fr180eMAAAA8BGvl31ffPFF2Ww2DRgwQGfOnJEklS9fXg888IAmTJjg8wABAAB8zg/3/JWVyp/XyV9ISIimTJmi1NRU7dixQ5JUr149hYeH+zw4AAAAv+CoF88NHjxYx48fV3h4uFq0aKEWLVooPDxceXl5Gjx4sD9iBAAAgI94nfy9+eabOnnyZKH2kydP6q233vJJUAAAAH5l4Q0fHi/75uTkyBgjY4yOHz+u0NBQ12sOh0OLFy9W9erV/RIkAAAAfMPj5K9ixYqy2Wyy2Wxq2LBhoddtNpvGjh3r0+AAAAD8gUOePbB8+XIZY3TNNdfo/fffV+XKlV2vhYSEKC4uTrVq1fJLkAAAAPANj5O/Ll26SJJ27dqlOnXqyGaz+S0oAAAA+IfXR73ExcX5Iw4AAABcAF4nfwAAAGWehc/5I/kDAACWY+UNH16f8wcAAICyy+vKX0pKigYPHlyq7v0rn1OgcuXIY38vJLd8oEMotcqdOBPoEEotmzPQEZRezs6XBTqEUivoy28DHQLgvTJSqfM1rzOmRYsWqV69eurevbvmzZun/Px8f8QFAAAAP/A6+du4caO+/vprNWvWTCNHjlR0dLQeeOABff311/6IDwAAwPcs/Hi3Eq2VXnbZZfrHP/6hffv26fXXX9cvv/yiTp06qWXLlpoyZYqys7N9HScAAAB84LxulDPG6PTp0yooKJAxRpUqVdLUqVMVGxurBQsW+CpGAAAAnzq329fXV1lQouRv/fr1Gj58uGrWrKnRo0frsssu0+bNm7VixQpt27ZNzz33nB566CFfxwoAAIDz5PVu3xYtWmjLli3q0aOHXn/9dfXq1UvBwcFuffr166eRI0f6LEgAAACf4pBnz/Xp00eDBw9W7dq1i+1TtWpVOZ2cFwEAAEonDnn20OnTpzV37lzl5OT4Kx4AAAD4kVeVv/Lly+vUqVP+igUAAODCsPCyr9cbPoYNG6aJEyfqzBmekAAAAHC+pk2bpvj4eIWGhqp9+/Zat26dR++bP3++bDabevfu7dV8Xt/z9/XXXystLU1ffPGFWrRooUsuucTt9X/961/eDgkAAHBhlZLK34IFC5SUlKSZM2eqffv2mjx5shITE7V161ZVr1692Pft3r1bDz/8sDp37uz1nF5X/ipWrKjbbrtNiYmJqlWrlqKiotwuAAAAeGbSpEkaOnSoBg0apKZNm2rmzJkKDw/XG2+8Uex7HA6H+vfvr7Fjx+rSSy/1ek6vK39z5szxehIAAIDSxJ+7fX+/MdZut8tutxfqX1BQoPXr1ys5OdnVFhQUpISEBGVkZBQ7z7hx41S9enXde++9+vLLL72Os0SHPJ85c0bLli3Tq6++quPHj0uS9u3bp9zc3JIMBwAAcNGIjY11WxVNTU0tst/hw4flcDhUo0YNt/YaNWooKyuryPesWrVKr7/+umbPnl3i+Lyu/P3888+67rrrtGfPHuXn5+vaa69VhQoVNHHiROXn52vmzJklDgYAAOCC8OM9f5mZmYqMjHQ1F1X1K4njx4/r7rvv1uzZs1W1atUSj+N18jdy5Ei1bdtW3333napUqeJqv+WWWzR06NASBwIAAHDB+DH5i4yMdEv+ilO1alUFBwfrwIEDbu0HDhxQdHR0of47duzQ7t271atXL1fbuYdqlCtXTlu3blW9evX+dF6vk78vv/xSa9asUUhIiFt7fHy89u7d6+1wAAAAlhQSEqI2bdooLS3NdVyL0+lUWlqahg8fXqh/48aNtWnTJre2J598UsePH9eUKVMUGxvr0bxeJ39Op1MOh6NQ+y+//KIKFSp4OxwAAMAFV1oe75aUlKSBAweqbdu2ateunSZPnqy8vDwNGjRIkjRgwADVrl1bqampCg0NVfPmzd3eX7FiRUkq1P5HvE7+evToocmTJ2vWrFmSJJvNptzcXKWkpOj666/3djgAAADL6tu3rw4dOqSnn35aWVlZat26tZYsWeLaBLJnzx4FBZVof26xvE7+XnrpJSUmJqpp06Y6deqU7rzzTm3btk1Vq1bVP//5T58GBwAA4Bel5JBnSRo+fHiRy7ySlJ6e/ofvnTt3rtfzeZ38xcTE6LvvvtP8+fP1/fffKzc3V/fee6/69++vsLAwrwMAAADAheN18ied3VFy1113+ToWAACAC6K03PMXCF4nf2+99dYfvj5gwIASBwMAAAD/KtE5f791+vRpnThxQiEhIQoPDyf5AwAApV8puufvQvN6+8jRo0fdrtzcXG3dulVXXXUVGz4AAEDZYPx0lQE+2TvcoEEDTZgwoVBVEAAAAKVLiTZ8FDlQuXLat2+fr4YDAADwG9t/L1+PWRZ4nfx99NFHbj8bY7R//35NnTpVnTp18llgAAAA8D2vk79zz547x2azqVq1arrmmmv00ksv+SouAAAA/2HDh+ecTqfb5XA4lJWVpXnz5qlmzZolDmTChAmy2WwaNWpUiccAAADAHyvxPX+HDx9WSEiIIiMjzzuIr7/+Wq+++qpatmx53mMBAAD8GSsf8uxV5e/YsWMaNmyYqlatqho1aqhSpUqKjo5WcnKyTpw4UaIAcnNz1b9/f82ePVuVKlUq0RgAAADwjMeVvyNHjqhDhw7au3ev+vfvryZNmkiSfvzxR73yyitaunSpVq1ape+//15r167VQw895NG4w4YN0w033KCEhAQ9++yzf9g3Pz9f+fn5rp9zcnI8DR8AAOB/LHzPn8fJ37hx4xQSEqIdO3aoRo0ahV7r0aOH7r77bn3xxRf6xz/+4dGY8+fP14YNG/T111971D81NVVjx471NGQAAIDilZFkzdc8Xvb98MMP9eKLLxZK/CQpOjpazz//vN5//30lJSVp4MCBfzpeZmamRo4cqXfeeUehoaEexZCcnKzs7GzXlZmZ6Wn4AAAAkBeVv/3796tZs2bFvt68eXMFBQUpJSXFo/HWr1+vgwcP6vLLL3e1ORwOrVy5UlOnTlV+fr6Cg4Pd3mO322W32z0NGQAAoEhW3vDhcfJXtWpV7d69WzExMUW+vmvXLlWvXt3jibt3765Nmza5tQ0aNEiNGzfWo48+WijxAwAAwPnzOPlLTEzUE088oaVLlyokJMTttfz8fD311FO67rrrPJ64QoUKat68uVvbJZdcoipVqhRqBwAA8Ck2fPy5cePGqW3btmrQoIGGDRumxo0byxijzZs3a/r06crPz9dbb73lz1gBAABwnjxO/mJiYpSRkaEHH3xQycnJMuZsemuz2XTttddq6tSpqlOnznkFk56efl7vBwAA8AT3/Hmobt26+uyzz3T06FFt27ZNklS/fn1VrlzZL8EBAADAt0r0eLdKlSqpXbt2vo4FAADgwrDwPX9ePd4NAAAAZVuJKn8AAABlGff8AQAAWAnLvgAAALACKn8AAMB6qPwBAADACqj8AQAAy7Hyhg8qfwAAABZC5Q8AAFgP9/wBAADACqj8AQAAy7EZI5vxbanO1+P5C8kfAACwHpZ9AQAAYAVU/gAAgOVw1AsAAAAsgcofAACwHu75AwAAgBVQ+QMAAJbDPX8AAACwBCp/AADAeix8zx/JHwAAsByWfQEAAGAJVP4AAID1WHjZl8ofAACAhVwUlT9nWDk5y10UH8WnQo+eCXQIpVZunbBAh1BqRW47HugQSi1TPjjQIQDwobJyj56vUfkDAACwEMplAADAeow5e/l6zDKAyh8AAICFUPkDAACWY+Vz/kj+AACA9XDUCwAAAKyAyh8AALAcm/Ps5esxywIqfwAAABZC5Q8AAFgP9/wBAADACqj8AQAAy7HyUS9U/gAAACyEyh8AALAeCz/ejeQPAABYDsu+AAAAsAQqfwAAwHo46gUAAABWQOUPAABYDvf8AQAAwBKo/AEAAOux8FEvVP4AAAAshMofAACwHCvf80fyBwAArIejXgAAAGAFVP4AAIDlWHnZl8ofAACAhVD5AwAA1uM0Zy9fj1kGUPkDAACwkIAmf2PGjJHNZnO7GjduHMiQAACAFRg/XWVAwJd9mzVrpmXLlrl+Llcu4CEBAABctAKeaZUrV07R0dGBDgMAAFiITX7Y7evb4fwm4Pf8bdu2TbVq1dKll16q/v37a8+ePcX2zc/PV05OjtsFAADgtXPP9vX1VQYENPlr37695s6dqyVLlmjGjBnatWuXOnfurOPHjxfZPzU1VVFRUa4rNjb2AkcMAABQtgU0+evZs6duv/12tWzZUomJiVq8eLGOHTumd999t8j+ycnJys7Odl2ZmZkXOGIAAHAxOHfIs6+vsiDgy76/VbFiRTVs2FDbt28v8nW73a7IyEi3CwAAoCybNm2a4uPjFRoaqvbt22vdunXF9p09e7Y6d+6sSpUqqVKlSkpISPjD/kUpVclfbm6uduzYoZo1awY6FAAAcDErJUe9LFiwQElJSUpJSdGGDRvUqlUrJSYm6uDBg0X2T09PV79+/bR8+XJlZGQoNjZWPXr00N69ez2eM6DJ38MPP6wVK1Zo9+7dWrNmjW655RYFBwerX79+gQwLAADggpg0aZKGDh2qQYMGqWnTppo5c6bCw8P1xhtvFNn/nXfe0YMPPqjWrVurcePGeu211+R0OpWWlubxnAE96uWXX35Rv3799Ouvv6patWq66qqrtHbtWlWrVi2QYQEAgIuczRjZfLw799x4vz+NxG63y263F+pfUFCg9evXKzk52dUWFBSkhIQEZWRkeDTniRMndPr0aVWuXNnjOAOa/M2fPz+Q0wMAAPjc708jSUlJ0ZgxYwr1O3z4sBwOh2rUqOHWXqNGDW3ZssWjuR599FHVqlVLCQkJHscX8EOeAQAALjjnfy9fjykpMzPTbVNqUVU/X5gwYYLmz5+v9PR0hYaGevw+kj8AAGA5/lz29fREkqpVqyo4OFgHDhxwaz9w4MCfPv3sxRdf1IQJE7Rs2TK1bNnSqzhL1W5fAAAAqwgJCVGbNm3cNmuc27zRoUOHYt/3/PPP65lnntGSJUvUtm1br+el8gcAAKynhEez/OmYXkpKStLAgQPVtm1btWvXTpMnT1ZeXp4GDRokSRowYIBq166t1NRUSdLEiRP19NNPa968eYqPj1dWVpYkKSIiQhERER7NSfIHAAAQIH379tWhQ4f09NNPKysrS61bt9aSJUtcm0D27NmjoKD/LdTOmDFDBQUF+stf/uI2TnGbSopC8gcAAKzHmLOXr8csgeHDh2v48OFFvpaenu728+7du0s0x29xzx8AAICFUPkDAACWYzNnL1+PWRZQ+QMAALAQKn8AAMB6StE9fxcalT8AAAALofIHAAAsx+Y8e/l6zLKA5A8AAFgPy74AAACwAip/AADAekrJ490CgcofAACAhVD5AwAAlmMzRjYf36Pn6/H8hcofAACAhVD5AwAA1sNuXwAAAFgBlT8AAGA9RpKvD2UuG4U/kj8AAGA9bPgAAACAJVD5AwAA1mPkhw0fvh3OX6j8AQAAWAiVPwAAYD0WPurlokj+8qLtKlfeHugwSp3IrccDHUKplRMXGegQSq8gFgSKU27vkUCHUGqdCXQAADx2USR/AAAAXnFKsvlhzDKA/8UHAACwECp/AADAcqx8zh/JHwAAsB4Lb/hg2RcAAMBCqPwBAADrofIHAAAAK6DyBwAArIfKHwAAAKyAyh8AALAeDnkGAACAFVD5AwAAlsMhzwAAAFbChg8AAABYAZU/AABgPU4j2XxcqXNS+QMAAEApQ+UPAABYD/f8AQAAwAqo/AEAAAvyQ+VPVP4AAABQylD5AwAA1mPhe/5I/gAAgPU4jXy+TMtRLwAAAChtqPwBAADrMc6zl6/HLAOo/AEAAFgIlT8AAGA9Ft7wQeUPAADAQqj8AQAA62G3b+Ds3btXd911l6pUqaKwsDC1aNFC33zzTaDDAgAAuCgFtPJ39OhRderUSd26ddNnn32matWqadu2bapUqVIgwwIAABc7C9/zF9Dkb+LEiYqNjdWcOXNcbXXr1g1gRAAAwBKM/JD8+XY4fwnosu9HH32ktm3b6vbbb1f16tV12WWXafbs2cX2z8/PV05OjtsFAAAAzwU0+du5c6dmzJihBg0a6PPPP9cDDzyghx56SG+++WaR/VNTUxUVFeW6YmNjL3DEAADgonBu2dfXVxkQ0OTP6XTq8ssv1/jx43XZZZfpvvvu09ChQzVz5swi+ycnJys7O9t1ZWZmXuCIAQAAyraA3vNXs2ZNNW3a1K2tSZMmev/994vsb7fbZbfbL0RoAADgYuZ0SvLx49icPN7tT3Xq1Elbt251a/vpp58UFxcXoIgAAAAubgGt/I0ePVodO3bU+PHj1adPH61bt06zZs3SrFmzAhkWAAC42Fn4qJeAVv6uuOIKffDBB/rnP/+p5s2b65lnntHkyZPVv3//QIYFAABw0Qr4491uvPFG3XjjjYEOAwAAWImFK38BT/4AAAAuOJ7tCwAAACug8gcAACzHGKeM8e3RLL4ez1+o/AEAAFgIlT8AAGA9xvj+Hr0ysuGDyh8AAICFUPkDAADWY/yw25fKHwAAAEobKn8AAMB6nE7J5uPduWVkty/JHwAAsB6WfQEAAGAFVP4AAIDlGKdTxsfLvhzyDAAAgFKHyh8AALAe7vkDAACAFVD5AwAA1uM0ko3KHwAAAC5yVP4AAID1GCPJ14c8U/kDAABAKUPlDwAAWI5xGhkf3/Nnykjlj+QPAABYj3HK98u+HPIMAACAUobkDwAAWI5xGr9cJTFt2jTFx8crNDRU7du317p16/6w/3vvvafGjRsrNDRULVq00OLFi72aj+QPAAAgQBYsWKCkpCSlpKRow4YNatWqlRITE3Xw4MEi+69Zs0b9+vXTvffeq2+//Va9e/dW79699cMPP3g8J8kfAACwHuP0z+WlSZMmaejQoRo0aJCaNm2qmTNnKjw8XG+88UaR/adMmaLrrrtOf//739WkSRM988wzuvzyyzV16lSP5yzTGz7O7apxnD4V4EhKpzOO/ECHUGo5CvgzU5wzDr6b4hgnf6eKc8acDnQIKEPO6Oyfl0Dujj2j0z5/tO+5z5WTk+PWbrfbZbfbC/UvKCjQ+vXrlZyc7GoLCgpSQkKCMjIyipwjIyNDSUlJbm2JiYn68MMPPY6zTCd/x48flyRtXPRsgCNBmfNdoAMAABw/flxRUVEXdM6QkBBFR0drVZZ398l5KiIiQrGxsW5tKSkpGjNmTKG+hw8flsPhUI0aNdzaa9SooS1bthQ5flZWVpH9s7KyPI6xTCd/tWrVUmZmpipUqCCbzRbocJSTk6PY2FhlZmYqMjIy0OGUGnwvxeO7KR7fTfH4borHd1O80vTdGGN0/Phx1apV64LPHRoaql27dqmgoMAv4xtjCuUkRVX9AqlMJ39BQUGKiYkJdBiFREZGBvwvVmnE91I8vpvi8d0Uj++meHw3xSst382Frvj9VmhoqEJDQwM2/zlVq1ZVcHCwDhw44NZ+4MABRUdHF/me6Ohor/oXhQ0fAAAAARASEqI2bdooLS3N1eZ0OpWWlqYOHToU+Z4OHTq49ZekpUuXFtu/KGW68gcAAFCWJSUlaeDAgWrbtq3atWunyZMnKy8vT4MGDZIkDRgwQLVr11ZqaqokaeTIkerSpYteeukl3XDDDZo/f76++eYbzZo1y+M5Sf58yG63KyUlpdSt7Qca30vx+G6Kx3dTPL6b4vHdFI/vpnTq27evDh06pKefflpZWVlq3bq1lixZ4trUsWfPHgUF/W+htmPHjpo3b56efPJJPf7442rQoIE+/PBDNW/e3OM5baasPIUYAAAA5417/gAAACyE5A8AAMBCSP4AAAAshOQPAADAQkj+fGTatGmKj49XaGio2rdvr3Xr1gU6pFJh5cqV6tWrl2rVqiWbzebVswcvZqmpqbriiitUoUIFVa9eXb1799bWrVsDHVapMGPGDLVs2dJ1EG2HDh302WefBTqsUmfChAmy2WwaNWpUoEMpFcaMGSObzeZ2NW7cONBhlQp79+7VXXfdpSpVqigsLEwtWrTQN998E+iwEEAkfz6wYMECJSUlKSUlRRs2bFCrVq2UmJiogwcPBjq0gMvLy1OrVq00bdq0QIdSqqxYsULDhg3T2rVrtXTpUp0+fVo9evRQXl5eoEMLuJiYGE2YMEHr16/XN998o2uuuUY333yz/vOf/wQ6tFLj66+/1quvvqqWLVsGOpRSpVmzZtq/f7/rWrVqVaBDCrijR4+qU6dOKl++vD777DP9+OOPeumll1SpUqVAh4YA4qgXH2jfvr2uuOIKTZ06VdLZ07ljY2M1YsQIPfbYYwGOrvSw2Wz64IMP1Lt370CHUuocOnRI1atX14oVK3T11VcHOpxSp3LlynrhhRd07733BjqUgMvNzdXll1+u6dOn69lnn1Xr1q01efLkQIcVcGPGjNGHH36ojRs3BjqUUuWxxx7T6tWr9eWXXwY6FJQiVP7OU0FBgdavX6+EhARXW1BQkBISEpSRkRHAyFCWZGdnSzqb5OB/HA6H5s+fr7y8PK8eXXQxGzZsmG644Qa3f+fgrG3btqlWrVq69NJL1b9/f+3ZsyfQIQXcRx99pLZt2+r2229X9erVddlll2n27NmBDgsBRvJ3ng4fPiyHw+E6ifucGjVqKCsrK0BRoSxxOp0aNWqUOnXq5NUJ7RezTZs2KSIiQna7Xffff78++OADNW3aNNBhBdz8+fO1YcMG12Oe8D/t27fX3LlztWTJEs2YMUO7du1S586ddfz48UCHFlA7d+7UjBkz1KBBA33++ed64IEH9NBDD+nNN98MdGgIIB7vBgTYsGHD9MMPP3B/0m80atRIGzduVHZ2thYuXKiBAwdqxYoVlk4AMzMzNXLkSC1dulShoaGBDqfU6dmzp+ufW7Zsqfbt2ysuLk7vvvuupW8XcDqdatu2rcaPHy9Juuyyy/TDDz9o5syZGjhwYICjQ6BQ+TtPVatWVXBwsA4cOODWfuDAAUVHRwcoKpQVw4cP1yeffKLly5crJiYm0OGUGiEhIapfv77atGmj1NRUtWrVSlOmTAl0WAG1fv16HTx4UJdffrnKlSuncuXKacWKFfrHP/6hcuXKyeFwBDrEUqVixYpq2LChtm/fHuhQAqpmzZqF/qepSZMmLIlbHMnfeQoJCVGbNm2UlpbmanM6nUpLS+MeJRTLGKPhw4frgw8+0L///W/VrVs30CGVak6nU/n5+YEOI6C6d++uTZs2aePGja6rbdu26t+/vzZu3Kjg4OBAh1iq5ObmaseOHapZs2agQwmoTp06FTpG6qefflJcXFyAIkJpwLKvDyQlJWngwIFq27at2rVrp8mTJysvL0+DBg0KdGgBl5ub6/Z/3rt27dLGjRtVuXJl1alTJ4CRBdawYcM0b948LVq0SBUqVHDdHxoVFaWwsLAARxdYycnJ6tmzp+rUqaPjx49r3rx5Sk9P1+effx7o0AKqQoUKhe4JveSSS1SlShXuFZX08MMPq1evXoqLi9O+ffuUkpKi4OBg9evXL9ChBdTo0aPVsWNHjR8/Xn369NG6des0a9YszZo1K9ChIZAMfOKVV14xderUMSEhIaZdu3Zm7dq1gQ6pVFi+fLmRVOgaOHBgoEMLqKK+E0lmzpw5gQ4t4AYPHmzi4uJMSEiIqVatmunevbv54osvAh1WqdSlSxczcuTIQIdRKvTt29fUrFnThISEmNq1a5u+ffua7du3BzqsUuHjjz82zZs3N3a73TRu3NjMmjUr0CEhwDjnDwAAwEK45w8AAMBCSP4AAAAshOQPAADAQkj+AAAALITkDwAAwEJI/gAAACyE5A8AAMBCSP4AAAAshOQPgM/MnTtXFStWDHQYPnfPPfeod+/ef9gnPT1dNptNx44duyAxAUBJkfwBpVxRicfChQsVGhqql156yS9zXqxJXElNmTJFc+fOdf3ctWtXjRo1yq1Px44dtX//fkVFRV3Y4ADAS+UCHQAA77z22msaNmyYZs6cqUGDBgU6HEvwJKELCQlRdHT0BYgGAM4PlT+gDHn++ec1YsQIzZ8/3y3xW7RokS6//HKFhobq0ksv1dixY3XmzBlJ0uDBg3XjjTe6jXP69GlVr15dr7/+eqE50tPTNWjQIGVnZ8tms8lms2nMmDGSpKNHj2rAgAGqVKmSwsPD1bNnT23btq3YeA8dOqS2bdvqlltuUX5+vpxOp1JTU1W3bl2FhYWpVatWWrhwodvcNptNaWlpatu2rcLDw9WxY0dt3bq12Dl2794tm82m+fPnq2PHjgoNDVXz5s21YsUKt34rVqxQu3btZLfbVbNmTT322GOu70g6W01t0aKFwsLCVKVKFSUkJCgvL0+Se/X1nnvu0YoVKzRlyhTX97N79+4il33ff/99NWvWTHa7XfHx8YUqtfHx8Ro/frwGDx6sChUqqE6dOpo1a1axnxUAfMIAKNUGDhxobr75ZvPII4+YiIgIs2zZMrfXV65caSIjI83cuXPNjh07zBdffGHi4+PNmDFjjDHGrF692gQHB5t9+/a53vOvf/3LXHLJJeb48eOF5svPzzeTJ082kZGRZv/+/Wb//v2ufjfddJNp0qSJWblypdm4caNJTEw09evXNwUFBcYYY+bMmWOioqKMMcbs2bPHNGrUyAwcONCcOXPGGGPMs88+axo3bmyWLFliduzYYebMmWPsdrtJT083xhizfPlyI8m0b9/epKenm//85z+mc+fOpmPHjsV+P7t27TKSTExMjFm4cKH58ccfzZAhQ0yFChXM4cOHjTHG/PLLLyY8PNw8+OCDZvPmzeaDDz4wVatWNSkpKcYYY/bt22fKlStnJk2aZHbt2mW+//57M23aNNfnPvc7MMaYY8eOmQ4dOpihQ4e6vp8zZ864Yj969KgxxphvvvnGBAUFmXHjxpmtW7eaOXPmmLCwMDNnzhxX7HFxcaZy5cpm2rRpZtu2bSY1NdUEBQWZLVu2/OGfCQA4HyR/QCk3cOBAExISYiSZtLS0Qq93797djB8/3q3t7bffNjVr1nT93LRpUzNx4kTXz7169TL33HNPsXP+Nok756effjKSzOrVq11thw8fNmFhYebdd991e9+WLVtMbGyseeihh4zT6TTGGHPq1CkTHh5u1qxZ4zbuvffea/r162eM+V/y99sE99NPPzWSzMmTJ4uM9VzyN2HCBFfb6dOnTUxMjOszP/7446ZRo0auWIwxZtq0aSYiIsI4HA6zfv16I8ns3r27yDl+m/wZY0yXLl3MyJEj3fr8Pvm78847zbXXXuvW5+9//7tp2rSp6+e4uDhz1113uX52Op2mevXqZsaMGUXGAQC+wLIvUAa0bNlS8fHxSklJUW5urttr3333ncaNG6eIiAjXNXToUO3fv18nTpyQJA0ZMkRz5syRJB04cECfffaZBg8e7FUMmzdvVrly5dS+fXtXW5UqVdSoUSNt3rzZ1Xby5El17txZt956q2tpVJK2b9+uEydO6Nprr3WL9a233tKOHTsKfd5zatasKUk6ePDgH8bXoUMH1z+XK1dObdu2dcW1efNmdejQwRWLJHXq1Em5ubn65Zdf1KpVK3Xv3l0tWrTQ7bffrtmzZ+vo0aNefT+/t3nzZnXq1MmtrVOnTtq2bZscDoer7bef1WazKTo6+k8/KwCcD5I/oAyoXbu20tPTtXfvXl133XU6fvy467Xc3FyNHTtWGzdudF2bNm3Stm3bFBoaKkkaMGCAdu7cqYyMDP3f//2f6tatq86dO/slVrvdroSEBH3yySfau3evW5yS9Omnn7rF+uOPP7rd9ydJ5cuXd/3zuYTN6XT6JV5JCg4O1tKlS/XZZ5+padOmeuWVV9SoUSPt2rXLb3Oe89vPKp39vP78rABA8geUEXFxcVqxYoWysrLcEsDLL79cW7duVf369QtdQUFn/4pXqVJFvXv31pw5czR37tw/3SUcEhLiVp2SpCZNmujMmTP66quvXG2//vqrtm7dqqZNm7ragoKC9Pbbb6tNmzbq1q2b9u3bJ0lq2rSp7Ha79uzZUyjO2NjY8/5+1q5d6/rnM2fOaP369WrSpIkr9oyMDBljXH1Wr16tChUqKCYmRtLZpKtTp04aO3asvv32W4WEhOiDDz4ocq6ivp/fa9KkiVavXu3Wtnr1ajVs2FDBwcEl+owA4Asc9QKUIbGxsUpPT1e3bt2UmJioJUuW6Omnn9aNN96oOnXq6C9/+YuCgoL03Xff6YcfftCzzz7reu+QIUN04403yuFwaODAgX84T3x8vHJzc5WWlqZWrVopPDxcDRo00M0336yhQ4fq1VdfVYUKFfTYY4+pdu3auvnmm93eHxwcrHfeeUf9+vXTNddco/T0dEVHR+vhhx/W6NGj5XQ6ddVVVyk7O1urV69WZGTkn8b0Z6ZNm6YGDRqoSZMmevnll3X06FHX0vaDDz6oyZMna8SIERo+fLi2bt2qlJQUJSUlKSgoSF999ZXS0tLUo0cPVa9eXV999ZUOHTrkSh6L+n6++uor7d69WxEREapcuXKhPn/72990xRVX6JlnnlHfvn2VkZGhqVOnavr06ef1OQHgvAX6pkMAf+z3mw2MObt7tUGDBubKK6802dnZZsmSJaZjx44mLCzMREZGmnbt2plZs2a5vcfpdJq4uDhz/fXXezTv/fffb6pUqWIkuXbFHjlyxNx9990mKirKhIWFmcTERPPTTz+53vP7jSKnT582t956q2nSpIk5cOCAcTqdZvLkyaZRo0amfPnyplq1aiYxMdGsWLHCGFN404Qxxnz77bdGktm1a1eRcZ7b8DFv3jzTrl07ExISYpo2bWr+/e9/u/VLT083V1xxhQkJCTHR0dHm0UcfNadPnzbGGPPjjz+axMREU61aNWO3203Dhg3NK6+84nrv738HW7duNVdeeaUJCwtzxVZU7AsXLjRNmzY15cuXN3Xq1DEvvPCCW0xxcXHm5Zdfdmtr1aqV6/sGAH+wGfObdRAAF63c3FzVrl1bc+bM0a233hrocHxm9+7dqlu3rr799lu1bt060OEAQKnHsi9wkXM6nTp8+LBeeuklVaxYUTfddFOgQwIABBDJH3CR27Nnj+rWrauYmBjNnTtX5crx1x4ArIxlXwAAAAvhqBcAAAALIfkDAACwEJI/AAAACyH5AwAAsBCSPwAAAAsh+QMAALAQkj8AAAALIfkDAACwkP8Hk/dRhDIPLRYAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "embed_dim = 8\n",
        "num_heads = 2\n",
        "batch_size = 1\n",
        "encoder_seq_length = 5\n",
        "decoder_seq_length = 7\n",
        "\n",
        "encoder_outputs = torch.randn(batch_size, encoder_seq_length, embed_dim)\n",
        "decoder_inputs = torch.randn(batch_size, decoder_seq_length, embed_dim)\n",
        "\n",
        "encoder_padding_mask = torch.zeros(batch_size, encoder_seq_length, dtype=torch.bool)\n",
        "encoder_padding_mask[:, -1] = True # The last encoder token is a padding tokens\n",
        "\n",
        "decoder_padding_mask = torch.zeros(batch_size, decoder_seq_length, dtype=torch.bool)\n",
        "decoder_padding_mask[:, -2:] = True # The last two decoder tokens are padding token\n",
        "\n",
        "\n",
        "cross_attention = MultiHeadAttention(embed_dim, num_heads, is_cross_attention=True)\n",
        "causal_attention = MultiHeadAttention(embed_dim, num_heads, is_causal_attention=True)\n",
        "\n",
        "cross_attention_out, cross_attention_weights = cross_attention(decoder_inputs, encoder_padding_mask, encoder_outputs)\n",
        "causal_attention_out, causal_attention_weights = causal_attention(decoder_inputs, decoder_padding_mask)\n",
        "\n",
        "# Make sure your outputs have the right hapes\n",
        "assert cross_attention_out.shape == (batch_size, decoder_seq_length, embed_dim)\n",
        "assert cross_attention_weights.shape == (batch_size, num_heads, decoder_seq_length, encoder_seq_length)\n",
        "assert causal_attention_out.shape == (batch_size, decoder_seq_length, embed_dim)\n",
        "assert causal_attention_weights.shape == (batch_size, num_heads, decoder_seq_length, decoder_seq_length)\n",
        "\n",
        "# Check that the attention weights are normalized\n",
        "assert torch.isclose(cross_attention_weights.sum(dim=-1), torch.tensor(1.0)).all()\n",
        "assert torch.isclose(causal_attention_weights.sum(dim=-1), torch.tensor(1.0)).all()\n",
        "\n",
        "# Check if the attention masking works\n",
        "assert torch.isclose(cross_attention_weights[:,:,:,-1], torch.tensor(0.0)).all()\n",
        "assert torch.isclose(causal_attention_weights[:,:,:,-2:], torch.tensor(0.0)).all()\n",
        "assert torch.isclose(causal_attention_weights[:,:,2,3:], torch.tensor(0.0)).all()\n",
        "\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_attention_matrix(attention_matrix, title):\n",
        "    \"\"\"Creates a new figure and plots the normalized attention weights as a heatmap.\n",
        "\n",
        "    This should provide a colorbar for the scale of the heatmap and label the axes \"query token position\" and \"key token position\".\n",
        "    Args:\n",
        "        attention_matrix: A numpy array of shape (number_of_query_tokens, number_of_key_tokens)\n",
        "        title: The title of the plot.\n",
        "    \"\"\"\n",
        "    # TODO Implement this function\n",
        "    plt.figure(figsize=(8, 6))\n",
        "    plt.imshow(attention_matrix, aspect='auto', cmap='viridis')\n",
        "    plt.colorbar()\n",
        "    plt.title(title)\n",
        "    plt.xlabel(\"Key token position\")\n",
        "    plt.ylabel(\"Query token position\")\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "\n",
        "plot_attention_matrix(cross_attention_weights[0,0].detach().numpy(), \"cross-attention, head 1\")\n",
        "plot_attention_matrix(cross_attention_weights[0,1].detach().numpy(), \"cross-attention, head 2\")\n",
        "plot_attention_matrix(causal_attention_weights[0,0].detach().numpy(), \"causal self-attention, head 1\")\n",
        "plot_attention_matrix(causal_attention_weights[0,1].detach().numpy(), \"causal self-attention, head 2\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Embedding + positional embeddings"
      ],
      "metadata": {
        "id": "etuvMKxaOQ4z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this section, we implement the `TransformerEmbeddings` class, which combines token embeddings and positional embeddings to form the final embeddings used in a transformer model. This is a crucial step in preparing the input sequences for the transformer architecture. Here is a detailed explanation of the class and its components:\n",
        "\n",
        "#### Class Initialization\n",
        "\n",
        "1. **Token Embeddings**:\n",
        "   - We use `nn.Embedding` to create an embedding matrix for the tokens. This matrix maps each token in the vocabulary to a dense vector of a specified `hidden_size`.\n",
        "   - `vocab_size`: The size of the vocabulary determines the number of rows in the token embedding matrix.\n",
        "   - `hidden_size`: The dimensionality of each embedding vector (the number of columns in the embedding matrix).\n",
        "\n",
        "2. **Positional Embeddings**:\n",
        "   - Transformers use positional embeddings to incorporate the order of tokens in the sequence since they do not inherently handle token positions.\n",
        "   - `nn.Embedding` is also used to create an embedding matrix for positions, mapping each position in the sequence to a dense vector of the same `hidden_size`.\n",
        "   - `max_sequence_length`: The maximum length of input sequences determines the number of rows in the positional embedding matrix.\n",
        "\n",
        "3. **Maximum Sequence Length**:\n",
        "   - We store the `max_sequence_length` to ensure that the positional embeddings can accommodate the longest possible input sequence.\n",
        "\n",
        "#### Methods\n",
        "\n",
        "1. **compute_logits**:\n",
        "   - This method computes the logits for the next token prediction based on the decoder's output.\n",
        "   - The decoder output, a tensor of shape `(batch_size, sequence_length, hidden_size)`, is multiplied by the transpose of the token embedding weights. This results in a tensor of shape `(batch_size, sequence_length, vocab_size)`, which contains the logits for each token in the vocabulary.\n",
        "\n",
        "2. **forward**:\n",
        "   - This method computes the embeddings for the input tokens by summing the token embeddings and positional embeddings.\n",
        "   - `input_ids`: A tensor of shape `(batch_size, sequence_length)` containing the token IDs.\n",
        "   - We create `position_ids`, a tensor of shape `(batch_size, sequence_length)`, where each position in the sequence is represented by its index (from 0 to `sequence_length - 1`).\n",
        "   - Token embeddings are obtained by passing `input_ids` through the token embedding layer.\n",
        "   - Positional embeddings are obtained by passing `position_ids` through the positional embedding layer.\n",
        "   - The final embeddings are the sum of token embeddings and positional embeddings, resulting in a tensor of shape `(batch_size, sequence_length, hidden_size)`.\n",
        "\n",
        "#### Example Workflow\n",
        "\n",
        "1. **Initialization**:\n",
        "   - The class is initialized with the vocabulary size, hidden size, and maximum sequence length.\n",
        "   - Example: `TransformerEmbeddings(vocab_size=5000, hidden_size=512, max_sequence_length=100)`\n",
        "\n",
        "2. **Forward Pass**:\n",
        "   - Given input token IDs, the `forward` method generates token embeddings and positional embeddings.\n",
        "   - It sums these embeddings to produce the final embeddings for the input sequence.\n",
        "\n",
        "3. **Logit Computation**:\n",
        "   - The `compute_logits` method takes the output from the decoder and computes the logits for the next token prediction using the token embedding weights.\n",
        "\n",
        "This implementation ensures that each token in the input sequence is represented by a combination of its token embedding and its position in the sequence, enabling the transformer model to effectively process and understand the input data."
      ],
      "metadata": {
        "id": "ZL6RAa9Z9CqI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d86pBrD71iIN"
      },
      "outputs": [],
      "source": [
        "class TransformerEmbeddings(nn.Module):\n",
        "    def __init__(self, vocab_size: int, hidden_size: int, max_sequence_length: int):\n",
        "        \"\"\"Defines the embedding layer with learnt positional embeddings.\n",
        "\n",
        "        This layer defines both the token embeddings and positional embeddings,\n",
        "        which are added together to form the final embedding.\n",
        "\n",
        "        Args:\n",
        "            vocab_size: The size of the vocabulary,\n",
        "                    used to define the size of the token embedding table.\n",
        "            hidden_size: The dimensionality of the embedding space for both token embeddings and positional embeddings.\n",
        "            max_sequence_length: The maximum sequence length of the input sequences,\n",
        "                    used to define the size of the position embedding table.\n",
        "\n",
        "        Note that this implementation does not use dropout on the embeddings\n",
        "        and uses learnt positional embeddings instead of sinusoidal embeddings.\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "\n",
        "        # TODO Initialize the module and its parameters here.\n",
        "        # You should use `nn.Embedding` for both token embeddings and positional embeddings\n",
        "        # Initialize token and position embeddings\n",
        "\n",
        "        self.token_embeddings = nn.Embedding(vocab_size, hidden_size)\n",
        "        self.position_embeddings = nn.Embedding(max_sequence_length, hidden_size)\n",
        "\n",
        "        # Store the maximum sequence length\n",
        "        self.max_sequence_length = max_sequence_length\n",
        "\n",
        "    def compute_logits(self, decoder_output: torch.FloatTensor) -> torch.FloatTensor:\n",
        "        \"\"\"Computes the logits for the next token prediction given the decoder output.\n",
        "\n",
        "        Args:\n",
        "            decoder_output: Tensor of shape (batch_size, sequence_length, hidden_size) - the output of the decoder.\n",
        "\n",
        "        Returns:\n",
        "            Tensor of shape (batch_size, sequence_length, vocab_size) containing the logits for the next token prediction.\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement this function\n",
        "        # Hint: you can access the weight parameter matrix via .weight of an nn.Embedding module:\n",
        "        # Example:\n",
        "        # ```embeddings = nn.Embedding(num_embeddings, embedding_dim)\n",
        "        # assert list(embeddings.weight.shape) == [num_embeddings, embedding_dim]```\n",
        "        # torch.matmul or F.linear may also be useful here.\n",
        "\n",
        "        # Compute the logits using the token embedding weights\n",
        "        logits = torch.matmul(decoder_output, self.token_embeddings.weight.T)\n",
        "        return logits\n",
        "    def forward(self, input_ids: torch.LongTensor) -> torch.FloatTensor:\n",
        "        \"\"\"Computes the embeddings for the input tokens.\n",
        "\n",
        "        Args:\n",
        "            input_ids: Tensor of shape (batch_size, sequence_length) containing the input token ids.\n",
        "\n",
        "        Returns:\n",
        "            Tensor of shape (batch_size, sequence_length, hidden_size) containing\n",
        "                    the sum of token embeddings and position embeddings for the input tokens.\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement the forward pass of the embedding layer.\n",
        "        # IMPORTANT: For full credit, you should not use python loops!\n",
        "        batch_size, sequence_length = input_ids.size()\n",
        "\n",
        "        # Create position ids from 0 to sequence_length - 1\n",
        "        position_ids = torch.arange(sequence_length, dtype=torch.long, device=input_ids.device)\n",
        "        position_ids = position_ids.unsqueeze(0).expand(batch_size, sequence_length)\n",
        "\n",
        "        # Get token and position embeddings\n",
        "        token_embeds = self.token_embeddings(input_ids)\n",
        "        position_embeds = self.position_embeddings(position_ids)\n",
        "\n",
        "        # Sum the token and position embeddings\n",
        "        embeddings = token_embeds + position_embeds\n",
        "        return embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The `TransformerBlock` class represents a single block of the transformer architecture, which can be used for both the encoder and decoder parts of a transformer model. Here's a detailed breakdown of its components and functionality:\n",
        "\n",
        "##### Class Initialization\n",
        "\n",
        "1. **Self-Attention**:\n",
        "   - The self-attention mechanism allows each token in the input sequence to attend to all other tokens. This is crucial for capturing dependencies regardless of their positions in the sequence.\n",
        "   - We use the `MultiHeadAttention` class to implement self-attention.\n",
        "   - If the block is part of the decoder (`is_decoder=True`), causal attention is applied to prevent a token from attending to future tokens.\n",
        "\n",
        "2. **Add & Norm Layer**:\n",
        "   - After the self-attention computation, we add the original input (residual connection) and apply layer normalization to stabilize and accelerate the training process.\n",
        "   - `nn.LayerNorm` is used for this purpose.\n",
        "\n",
        "3. **Cross-Attention (Decoder Only)**:\n",
        "   - If the block is part of the decoder, it includes a cross-attention mechanism, which allows the decoder to attend to the encoder's outputs.\n",
        "   - Another `MultiHeadAttention` layer is used for cross-attention, and a corresponding layer normalization is applied.\n",
        "\n",
        "4. **Feedforward Layer**:\n",
        "   - Each transformer block includes a feedforward neural network, which consists of two linear layers with a ReLU activation in between.\n",
        "   - Dropout is applied after the second linear layer to prevent overfitting.\n",
        "   - The feedforward network expands the dimensionality to `intermediate_size` and then projects it back to `hidden_size`.\n",
        "   - The result is then normalized using another layer normalization.\n",
        "\n",
        "##### Methods\n",
        "\n",
        "1. **Forward Pass**:\n",
        "   - This method defines the computations performed during the forward pass through the transformer block.\n",
        "\n",
        "2. **Self-Attention and Residual Connection**:\n",
        "   - The input `hidden_states` are passed through the self-attention layer.\n",
        "   - The output is added to the original `hidden_states` (residual connection).\n",
        "   - The result is then normalized using layer normalization.\n",
        "\n",
        "3. **Cross-Attention and Residual Connection (Decoder Only)**:\n",
        "   - If the block is a decoder block, the `hidden_states` are passed through the cross-attention layer, which attends to the encoder's outputs.\n",
        "   - The output is added to the original `hidden_states` (residual connection).\n",
        "   - The result is then normalized using layer normalization.\n",
        "\n",
        "4. **Feedforward Network and Residual Connection**:\n",
        "   - The `hidden_states` are passed through the feedforward network.\n",
        "   - The output is added to the original `hidden_states` (residual connection).\n",
        "   - The result is then normalized using layer normalization.\n",
        "\n",
        "##### Example Workflow\n",
        "\n",
        "1. **Initialization**:\n",
        "   - The `TransformerBlock` is initialized with specific parameters like `hidden_size`, `intermediate_size`, `num_attention_heads`, `hidden_dropout_prob`, and whether it is a decoder block (`is_decoder`).\n",
        "\n",
        "2. **Forward Pass**:\n",
        "   - During the forward pass, the input `hidden_states` and `padding_mask` are processed through self-attention, optional cross-attention, and the feedforward network with residual connections and normalization applied at each step.\n",
        "\n",
        "3. **Handling Encoder Outputs (Decoder Only)**:\n",
        "   - For decoder blocks, `encoder_outputs` and `encoder_padding_mask` are used to facilitate cross-attention, allowing the decoder to utilize information from the encoder.\n",
        "\n",
        "This implementation ensures that each token can effectively attend to relevant parts of the sequence, whether within the same layer (self-attention) or between encoder and decoder layers (cross-attention), while maintaining stable and efficient training through normalization and residual connections."
      ],
      "metadata": {
        "id": "2sA5txKv98xZ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CFW-eNNB1iIN"
      },
      "outputs": [],
      "source": [
        "class TransformerBlock(nn.Module):\n",
        "    def __init__(self,\n",
        "                 hidden_size: int,\n",
        "                 intermediate_size: int,\n",
        "                 num_attention_heads: int,\n",
        "                 hidden_dropout_prob: float,\n",
        "                 is_decoder: bool = False):\n",
        "        \"\"\"Defines a single Transformer block, which can be either for the encoder or the decoder.\n",
        "\n",
        "        Args:\n",
        "            hidden_size: The dimensionality of the input and output vectors of this layer.\n",
        "            intermediate_size: The intermediate size of the feedforward layers.\n",
        "            num_attention_heads: The number of attention heads.\n",
        "            hidden_dropout_prob: The dropout probability for the hidden states.\n",
        "            is_decoder: Whether this block is part of the decoder.\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "\n",
        "        self.is_decoder = is_decoder\n",
        "        self.self_attention = MultiHeadAttention(hidden_size, num_attention_heads, is_causal_attention=is_decoder)\n",
        "        #add&norm layer\n",
        "        self.self_attention_layer_norm = nn.LayerNorm(hidden_size)\n",
        "\n",
        "#if decoder use cross attention\n",
        "        if is_decoder:\n",
        "            self.cross_attention = MultiHeadAttention(hidden_size, num_attention_heads, is_cross_attention=True)\n",
        "            self.cross_attention_layer_norm = nn.LayerNorm(hidden_size)\n",
        "#feedForward layer\n",
        "        self.feedforward = nn.Sequential(\n",
        "            nn.Linear(hidden_size, intermediate_size),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(intermediate_size, hidden_size),\n",
        "            nn.Dropout(hidden_dropout_prob))\n",
        "        self.feedforward_layer_norm = nn.LayerNorm(hidden_size)\n",
        "\n",
        "    def forward(self,\n",
        "                hidden_states: torch.FloatTensor,\n",
        "                padding_mask: torch.BoolTensor,\n",
        "                encoder_outputs: Optional[torch.FloatTensor] = None,\n",
        "                encoder_padding_mask: Optional[torch.BoolTensor] = None) -> torch.FloatTensor:\n",
        "        \"\"\"Defines a single Transformer block, either for the encoder or the decoder.\n",
        "\n",
        "        Args:\n",
        "            hidden_states: Tensor of shape (batch_size, sequence_length, hidden_size) - the outputs from the previous layer.\n",
        "            padding_mask: Tensor of shape (batch_size, sequence_length) indicating which tokens are padding tokens.\n",
        "                    A `True` entry means that this token should be ignored for the purpose of attention.\n",
        "            encoder_outputs: Optional tensor of shape (batch_size, encoder_sequence_length, hidden_size),\n",
        "                    which are the output vectors of the encoder. This argument is only used by decoder blocks.\n",
        "            encoder_padding_mask: Optional tensor of shape (batch_size, encoder_sequence_length) indicating\n",
        "                    which encoder tokens are padding tokens. This argument is only used in decoder blocks.\n",
        "                    A `True` entry means that this token should be ignored for the purpose of attention.\n",
        "\n",
        "        \"\"\"\n",
        "        hidden_states = self.self_attention(hidden_states, padding_mask)[0] + hidden_states\n",
        "        hidden_states = self.self_attention_layer_norm(hidden_states)\n",
        "\n",
        "        if self.is_decoder:\n",
        "            hidden_states = self.cross_attention(hidden_states, encoder_padding_mask, encoder_outputs)[0] + hidden_states\n",
        "            hidden_states = self.cross_attention_layer_norm(hidden_states)\n",
        "\n",
        "        hidden_states = self.feedforward(hidden_states) + hidden_states\n",
        "        hidden_states = self.feedforward_layer_norm(hidden_states)\n",
        "        return hidden_states"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### EncoderDecoderModel Class Summary\n",
        "\n",
        "The `EncoderDecoderModel` class implements an encoder-decoder transformer model for neural machine translation (NMT). It consists of the following components:\n",
        "\n",
        "- **Initialization (`__init__` method):**\n",
        "  - **Source and Target Embeddings:** Utilizes `TransformerEmbeddings` to create embeddings for the source and target vocabularies.\n",
        "  - **Encoder Blocks:** A list of encoder blocks (`TransformerBlock`), initialized based on `num_encoder_layers`.\n",
        "  - **Decoder Blocks:** A list of decoder blocks (`TransformerBlock` with `is_decoder=True`), initialized based on `num_decoder_layers`.\n",
        "  - **Output Projection Layer:** A linear layer projecting hidden states to target vocabulary size.\n",
        "\n",
        "- **Encoder Forward Pass (`forward_encoder` method):**\n",
        "  - Computes embeddings for the source input IDs.\n",
        "  - Passes these embeddings through the encoder blocks.\n",
        "  - Returns the final hidden states of the encoder.\n",
        "\n",
        "- **Decoder Forward Pass (`forward_decoder` method):**\n",
        "  - Computes embeddings for the target input IDs.\n",
        "  - Passes these embeddings through the decoder blocks, utilizing cross-attention with encoder outputs.\n",
        "  - Projects the hidden states to produce logits for the target vocabulary.\n",
        "  - Returns the final logits for next token prediction.\n",
        "\n",
        "- **Full Model Forward Pass (`forward` method):**\n",
        "  - Combines the encoder and decoder forward passes.\n",
        "  - Processes input sequences and produces output logits for the target sequence.\n",
        "\n",
        "This class encapsulates the entire transformer model workflow, handling embedding lookups, multi-head attention, and the necessary transformations to generate translations."
      ],
      "metadata": {
        "id": "vyJ0erP5FhNi"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dC_EQSQ51iIN"
      },
      "outputs": [],
      "source": [
        "class EncoderDecoderModel(nn.Module):\n",
        "    def __init__(self,\n",
        "                 source_vocab_size: int,\n",
        "                 target_vocab_size: int,\n",
        "                 hidden_size: int,\n",
        "                 intermediate_size: int,\n",
        "                 num_attention_heads: int,\n",
        "                 num_encoder_layers: int,\n",
        "                 num_decoder_layers: int,\n",
        "                 max_sequence_length: int,\n",
        "                 hidden_dropout_prob: float):\n",
        "        \"\"\"A encoder-decoder transformer model which can be used for NMT.\n",
        "\n",
        "        Args:\n",
        "            source_vocab_size: The size of the source vocabulary.\n",
        "            target_vocab_size: The size of the target vocabulary.\n",
        "            hidden_size: The dimensionality of all input and output embeddings.\n",
        "            intermediate_size: The intermediate size in the feedforward layers.\n",
        "            num_attention_heads: The number of attention heads in each multi-head attention modules.\n",
        "            num_encoder_layers: The number of transformer blocks in the encoder.\n",
        "            num_decoder_layers: The number of transformer blocks in the decoder.\n",
        "            max_sequence_length: The maximum sequence length that this model can handle.\n",
        "            hidden_dropout_prob: The dropout probability in the hidden state in each block.\n",
        "        \"\"\"\n",
        "\n",
        "        super().__init__()\n",
        "        # TODO Register the input embedding modules and the encoder and decoder blocks.\n",
        "        # You should use the TransformerBlock and TransformerEmbeddings sub-modules.\n",
        "        #\n",
        "        # Hint: Check out `nn.ModuleList` to register a variable number of sub-modules.\n",
        "        # Define input embeddings for source and target tokens\n",
        "        self.source_embeddings = TransformerEmbeddings(source_vocab_size, hidden_size, max_sequence_length) #for enocder\n",
        "        self.target_embeddings = TransformerEmbeddings(target_vocab_size, hidden_size, max_sequence_length) #for decoder\n",
        "\n",
        "        # Define encoder blocks\n",
        "        self.encoder = nn.ModuleList([\n",
        "            TransformerBlock(hidden_size, intermediate_size, num_attention_heads, hidden_dropout_prob)\n",
        "            for _ in range(num_encoder_layers)\n",
        "        ])\n",
        "\n",
        "        # Define decoder blocks\n",
        "        self.decoder = nn.ModuleList([\n",
        "            TransformerBlock(hidden_size, intermediate_size, num_attention_heads, hidden_dropout_prob, is_decoder=True)\n",
        "            for _ in range(num_decoder_layers)\n",
        "        ])\n",
        "\n",
        "        # Define output projection layer\n",
        "        self.output_projection = nn.Linear(hidden_size, target_vocab_size)\n",
        "\n",
        "    def forward_encoder(self, input_ids: torch.LongTensor, padding_mask: torch.BoolTensor) -> torch.FloatTensor:\n",
        "        \"\"\"Implement the forward pass of the encoder.\n",
        "\n",
        "        Args:\n",
        "            input_ids: tensor of shape (batch_size, sequence_length) containing the input token ids to the encoder.\n",
        "            padding_mask: tensor of shape (batch_size, sequence_length) indicating which encoder tokens are padding tokens (`True`)\n",
        "                    and should be ignored in self-attention computations.\n",
        "\n",
        "        Returns:\n",
        "            Tensor of shape (batch_size, sequence_length, hidden_size) containing the output embeddings of the encoder.\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement this function\n",
        "        # Get source token embeddings\n",
        "        source_embeddings = self.source_embeddings(input_ids)\n",
        "\n",
        "        # Forward pass through encoder blocks\n",
        "        encoder_hidden_states = source_embeddings\n",
        "        for encoder_block in self.encoder:\n",
        "            encoder_hidden_states = encoder_block(encoder_hidden_states, padding_mask)\n",
        "\n",
        "        return encoder_hidden_states\n",
        "\n",
        "    def forward_decoder(self,\n",
        "                        input_ids: torch.LongTensor,\n",
        "                        padding_mask: torch.BoolTensor,\n",
        "                        encoder_outputs: torch.FloatTensor,\n",
        "                        encoder_padding_mask: torch.BoolTensor) -> torch.FloatTensor:\n",
        "        \"\"\"Implement the forward pass of the decoder.\n",
        "\n",
        "        Args:\n",
        "            input_ids: Tensor of shape (batch_size, sequence_length) containing the input token ids to the decoder.\n",
        "            padding_mask: Tensor of shape (batch_size, sequence_length) indicating which decoder tokens are padding tokens (`True`)\n",
        "                    and should be ignored in self-attention computations.\n",
        "            encoder_outputs: Tensor of shape (batch_size, encoder_sequence_length, hidden_size) containing the output embeddings of the encoder.\n",
        "            encoder_padding_mask: Tensor of shape (batch_size, encoder_sequence_length) indicating which encoder tokens are padding tokens (`True`)\n",
        "                    and should be ignored in cross-attention computations.\n",
        "\n",
        "        Returns:\n",
        "            Tensor of shape (batch_size, sequence_length, target_vocabulary_size)\n",
        "            containing the logits for predicting the next token in the target sequence.\n",
        "        \"\"\"\n",
        "\n",
        "        # TODO Implement this function\n",
        "\n",
        "        # Get target token embeddings\n",
        "        target_embeddings = self.target_embeddings(input_ids)\n",
        "\n",
        "        # Forward pass through decoder blocks\n",
        "        decoder_hidden_states = target_embeddings\n",
        "        for decoder_block in self.decoder:\n",
        "            decoder_hidden_states = decoder_block(decoder_hidden_states, padding_mask, encoder_outputs, encoder_padding_mask)\n",
        "\n",
        "        # Project to target vocabulary size\n",
        "        decoder_logits = self.output_projection(decoder_hidden_states)\n",
        "\n",
        "        return decoder_logits\n",
        "\n",
        "    def forward(self, encoder_input_ids, encoder_padding_mask, decoder_input_ids, decoder_padding_mask):\n",
        "        encoder_outputs = self.forward_encoder(encoder_input_ids, encoder_padding_mask)\n",
        "        decoder_logits = self.forward_decoder(decoder_input_ids, decoder_padding_mask, encoder_outputs, encoder_padding_mask)\n",
        "        return decoder_logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mbf1FkZteHIW"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "In this section, we train the seq2seq model on the parallel tokenized corpus.\n",
        "Before you start training models, you should implement and test the model and its sub-modules, especially the attention.\n",
        "\n",
        "First, we implement a `collate` function, which takes a list of examples from the dataset and forms a batch,\n",
        "consisting of padded encoder and decoder input ids, as well as encoder and decoder padding masks."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "The `collate_fn` function prepares a batch of variable-length sequences for neural network training by standardizing their lengths. It determines the longest sequence in the batch and initializes tensors filled with a padding token to match this length. It then populates these tensors with the actual sequence data and creates corresponding padding masks to indicate real tokens versus padding tokens. This process ensures uniform input shapes for efficient batch processing, allows the model to ignore padding during computations, and is especially critical for models using attention mechanisms to handle sequences correctly."
      ],
      "metadata": {
        "id": "5FdqZmrDADep"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VknfRSbb1iIN"
      },
      "outputs": [],
      "source": [
        "def collate_fn(examples: List[Dict[str, List[int]]]) -> Dict[str, torch.Tensor]:\n",
        "    \"\"\"Collates a list of variable length sequences from the dataset into a batch of pytorch tensors with padding.\"\"\"\n",
        "\n",
        "    # Determine the maximum sequence lengths for the encoder and decoder inputs in the batch\n",
        "    encoder_sequence_length = max(len(example[\"encoder_input_ids\"]) for example in examples)\n",
        "    decoder_sequence_length = max(len(example[\"decoder_input_ids\"]) for example in examples)\n",
        "    batch_size = len(examples)\n",
        "\n",
        "    # Initialize tensors for encoder input IDs and padding mask with the appropriate sizes\n",
        "    # Fill encoder_input_ids with the pad token ID and encoder_padding_mask with True (indicating padding)\n",
        "    encoder_input_ids = torch.full((batch_size, encoder_sequence_length),\n",
        "                                   fill_value=source_tokenizer.pad_token_id,\n",
        "                                   dtype=torch.int64)\n",
        "    encoder_padding_mask = torch.ones((batch_size, encoder_sequence_length),\n",
        "                                      dtype=torch.bool)\n",
        "\n",
        "    # Initialize tensors for decoder input IDs and padding mask with the appropriate sizes\n",
        "    # Fill decoder_input_ids with the pad token ID and decoder_padding_mask with True (indicating padding)\n",
        "    decoder_input_ids = torch.full((batch_size, decoder_sequence_length),\n",
        "                                   fill_value=target_tokenizer.pad_token_id,\n",
        "                                   dtype=torch.int64)\n",
        "    decoder_padding_mask = torch.ones((batch_size, decoder_sequence_length),\n",
        "                                      dtype=torch.bool)\n",
        "\n",
        "    # Fill the initialized tensors with actual input IDs and update padding masks\n",
        "    for i, example in enumerate(examples):\n",
        "        # Copy encoder input IDs to the tensor and set corresponding padding mask entries to False\n",
        "        encoder_input_ids[i, :len(example[\"encoder_input_ids\"])] = torch.tensor(example[\"encoder_input_ids\"])\n",
        "        encoder_padding_mask[i, :len(example[\"encoder_input_ids\"])] = False\n",
        "\n",
        "        # Copy decoder input IDs to the tensor and set corresponding padding mask entries to False\n",
        "        decoder_input_ids[i, :len(example[\"decoder_input_ids\"])] = torch.tensor(example[\"decoder_input_ids\"])\n",
        "        decoder_padding_mask[i, :len(example[\"decoder_input_ids\"])] = False\n",
        "\n",
        "    # Return the collated batch as a dictionary of tensors\n",
        "    return {\"encoder_input_ids\": encoder_input_ids,\n",
        "            \"encoder_padding_mask\": encoder_padding_mask,\n",
        "            \"decoder_input_ids\": decoder_input_ids,\n",
        "            \"decoder_padding_mask\": decoder_padding_mask}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mgp3PZL21iIN"
      },
      "source": [
        "Next, we provide a simple training loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FeZgvg2VkRQd"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "import math\n",
        "\n",
        "#compute the los using cross Entropy\n",
        "def compute_loss_per_token(model, batch):\n",
        "    logits = model(**batch)\n",
        "\n",
        "    valid_label_mask = ~(batch[\"decoder_padding_mask\"][:,1:])\n",
        "    labels = batch[\"decoder_input_ids\"][:,1:][valid_label_mask]\n",
        "    logits = logits[:,:-1][valid_label_mask]\n",
        "\n",
        "    return F.cross_entropy(logits, labels, reduction='none')\n",
        "\n",
        "\n",
        "def evaluate_perplexity(model, dataset, batch_size=32, device=\"cpu\"):\n",
        "    model.eval()\n",
        "\n",
        "    dev_loader = torch.utils.data.DataLoader(\n",
        "        dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_fn)\n",
        "\n",
        "    num_tokens = loss_sum = 0\n",
        "\n",
        "    # no_grad() signals backend to throw away all gradients\n",
        "    with torch.no_grad():\n",
        "        for batch in dev_loader:\n",
        "            # Move tensors in batch to device\n",
        "            for key in batch:\n",
        "                batch[key] = batch[key].to(device)\n",
        "\n",
        "            token_losses = compute_loss_per_token(model, batch)\n",
        "\n",
        "            loss_sum += token_losses.sum()\n",
        "            num_tokens += token_losses.numel()\n",
        "\n",
        "        dev_ppl = (loss_sum / num_tokens).exp().cpu().item()\n",
        "    return dev_ppl\n",
        "\n",
        "\n",
        "def train(model, training_dataset, validation_dataset,\n",
        "          batch_size=32, lr=1e-3, max_epoch=10, log_every=10, valid_niter=100,\n",
        "          model_path=\"model.pt\"):\n",
        "    model.train()\n",
        "\n",
        "    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "    print('Use device: %s' % device)\n",
        "\n",
        "    model = model.to(device)\n",
        "\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "\n",
        "    epoch = global_step = loss_sum = num_tokens = num_examples = 0\n",
        "    best_valid_perplexity = float('inf')\n",
        "    train_time = begin_time = time.time()\n",
        "    print('Beginning maximum likelihood training')\n",
        "\n",
        "\n",
        "    while True:\n",
        "        train_loader = torch.utils.data.DataLoader(\n",
        "            training_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)\n",
        "        epoch += 1\n",
        "        batches_per_epoch = len(train_loader)\n",
        "        for i, batch in enumerate(train_loader):\n",
        "            global_step += 1\n",
        "\n",
        "            # Move tensors in batch to device\n",
        "            for key in batch:\n",
        "                batch[key] = batch[key].to(device)\n",
        "#optimizer.zero_grad() ensures that the gradients are cleared before computing the new gradients reset the gradient\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            token_losses = compute_loss_per_token(model, batch)\n",
        "            total_loss = token_losses.sum()\n",
        "\n",
        "            loss = total_loss / batch_size\n",
        "\n",
        "#loss.backward() computes the gradients of the loss with respect to the model parameters.\n",
        "            loss.backward()\n",
        "\n",
        "#optimizer.step() updates the model parameters based on the computed gradients.\n",
        "\n",
        "            optimizer.step()\n",
        "\n",
        "            loss_sum += total_loss.cpu().item()\n",
        "            num_tokens += token_losses.numel()\n",
        "            num_examples += batch_size\n",
        "\n",
        "            if global_step % log_every == 0:\n",
        "                average_loss = loss_sum / num_examples\n",
        "                average_ppl = math.exp(loss_sum / num_tokens)\n",
        "                print(f\"epoch {epoch} ({i}/{batches_per_epoch}) | step {global_step} | \"\n",
        "                      f\"avg_nll={average_loss:.2f} avg_ppl={average_ppl:.2f} \"\n",
        "                      f\"speed={num_tokens / (time.time() - train_time):.2f} words/sec \"\n",
        "                      f\"time_elapsed={time.time() - begin_time:.2f} sec\")\n",
        "\n",
        "                train_time = time.time()\n",
        "                loss_sum = num_tokens = num_examples = 0.0\n",
        "\n",
        "            if global_step % valid_niter == 0:\n",
        "                print('Begin validation ...')\n",
        "                dev_perplexity = evaluate_perplexity(model, validation_dataset, batch_size=batch_size, device=device)\n",
        "\n",
        "                print(f\"validation: step {global_step} | dev_ppl={dev_perplexity}\")\n",
        "\n",
        "                if dev_perplexity < best_valid_perplexity:\n",
        "                    best_valid_perplexity = dev_perplexity\n",
        "                    print(f\"epoch {epoch} step {global_step}: save currently the best model to '{model_path}'\")\n",
        "                    torch.save(model.state_dict(), model_path)\n",
        "                    torch.save(optimizer.state_dict(), model_path + '.optim')\n",
        "                model.train()\n",
        "\n",
        "        if epoch == max_epoch:\n",
        "            print('Reached maximum number of epochs')\n",
        "            break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8NLae1r1iIO"
      },
      "source": [
        "Let's train a relatively small model architecture for 15 epochs.\n",
        "With a reasonable implementation, this should take about 16 minutes on CPU / 3 minutes on GPU and we should achieve a validation perplexity of below 10!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7j8KNRx21iIO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32eb0523-6c86-48b9-8ba5-a95ebf41e4b7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model architecture: EncoderDecoderModel(\n",
            "  (source_embeddings): TransformerEmbeddings(\n",
            "    (token_embeddings): Embedding(3200, 32)\n",
            "    (position_embeddings): Embedding(32, 32)\n",
            "  )\n",
            "  (target_embeddings): TransformerEmbeddings(\n",
            "    (token_embeddings): Embedding(3200, 32)\n",
            "    (position_embeddings): Embedding(32, 32)\n",
            "  )\n",
            "  (encoder): ModuleList(\n",
            "    (0-2): 3 x TransformerBlock(\n",
            "      (self_attention): MultiHeadAttention(\n",
            "        (query_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (key_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (value_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (out_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "      )\n",
            "      (self_attention_layer_norm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)\n",
            "      (feedforward): Sequential(\n",
            "        (0): Linear(in_features=32, out_features=128, bias=True)\n",
            "        (1): ReLU()\n",
            "        (2): Linear(in_features=128, out_features=32, bias=True)\n",
            "        (3): Dropout(p=0.1, inplace=False)\n",
            "      )\n",
            "      (feedforward_layer_norm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)\n",
            "    )\n",
            "  )\n",
            "  (decoder): ModuleList(\n",
            "    (0-2): 3 x TransformerBlock(\n",
            "      (self_attention): MultiHeadAttention(\n",
            "        (query_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (key_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (value_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (out_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "      )\n",
            "      (self_attention_layer_norm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)\n",
            "      (cross_attention): MultiHeadAttention(\n",
            "        (query_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (key_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (value_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "        (out_proj): Linear(in_features=32, out_features=32, bias=True)\n",
            "      )\n",
            "      (cross_attention_layer_norm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)\n",
            "      (feedforward): Sequential(\n",
            "        (0): Linear(in_features=32, out_features=128, bias=True)\n",
            "        (1): ReLU()\n",
            "        (2): Linear(in_features=128, out_features=32, bias=True)\n",
            "        (3): Dropout(p=0.1, inplace=False)\n",
            "      )\n",
            "      (feedforward_layer_norm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)\n",
            "    )\n",
            "  )\n",
            "  (output_projection): Linear(in_features=32, out_features=3200, bias=True)\n",
            ")\n",
            "Total number of trainable model parameters: 401536\n",
            "Use device: cpu\n",
            "Beginning maximum likelihood training\n",
            "epoch 1 (9/272) | step 10 | avg_nll=54.63 avg_ppl=2358.93 speed=1855.48 words/sec time_elapsed=1.21 sec\n",
            "epoch 1 (19/272) | step 20 | avg_nll=52.07 avg_ppl=1272.41 speed=1931.36 words/sec time_elapsed=2.42 sec\n",
            "epoch 1 (29/272) | step 30 | avg_nll=48.55 avg_ppl=772.99 speed=2009.21 words/sec time_elapsed=3.58 sec\n",
            "epoch 1 (39/272) | step 40 | avg_nll=44.56 avg_ppl=479.65 speed=2112.64 words/sec time_elapsed=4.68 sec\n",
            "epoch 1 (49/272) | step 50 | avg_nll=41.38 avg_ppl=314.97 speed=1407.05 words/sec time_elapsed=6.31 sec\n",
            "epoch 1 (59/272) | step 60 | avg_nll=38.03 avg_ppl=202.82 speed=1101.14 words/sec time_elapsed=8.39 sec\n",
            "epoch 1 (69/272) | step 70 | avg_nll=35.38 avg_ppl=135.50 speed=880.57 words/sec time_elapsed=11.02 sec\n",
            "epoch 1 (79/272) | step 80 | avg_nll=33.81 avg_ppl=105.78 speed=1919.45 words/sec time_elapsed=12.23 sec\n",
            "epoch 1 (89/272) | step 90 | avg_nll=31.68 avg_ppl=76.52 speed=2360.70 words/sec time_elapsed=13.22 sec\n",
            "epoch 1 (99/272) | step 100 | avg_nll=29.19 avg_ppl=60.06 speed=3463.73 words/sec time_elapsed=13.88 sec\n",
            "Begin validation ...\n",
            "validation: step 100 | dev_ppl=50.30553436279297\n",
            "epoch 1 step 100: save currently the best model to 'model.pt'\n",
            "epoch 1 (109/272) | step 110 | avg_nll=28.03 avg_ppl=49.83 speed=2614.43 words/sec time_elapsed=14.76 sec\n",
            "epoch 1 (119/272) | step 120 | avg_nll=26.84 avg_ppl=41.90 speed=3712.88 words/sec time_elapsed=15.38 sec\n",
            "epoch 1 (129/272) | step 130 | avg_nll=27.52 avg_ppl=41.95 speed=4194.74 words/sec time_elapsed=15.94 sec\n",
            "epoch 1 (139/272) | step 140 | avg_nll=26.12 avg_ppl=35.75 speed=3966.03 words/sec time_elapsed=16.53 sec\n",
            "epoch 1 (149/272) | step 150 | avg_nll=25.46 avg_ppl=32.96 speed=4079.11 words/sec time_elapsed=17.10 sec\n",
            "epoch 1 (159/272) | step 160 | avg_nll=25.33 avg_ppl=32.46 speed=4166.57 words/sec time_elapsed=17.66 sec\n",
            "epoch 1 (169/272) | step 170 | avg_nll=23.96 avg_ppl=28.05 speed=3851.49 words/sec time_elapsed=18.26 sec\n",
            "epoch 1 (179/272) | step 180 | avg_nll=25.11 avg_ppl=29.93 speed=3980.55 words/sec time_elapsed=18.85 sec\n",
            "epoch 1 (189/272) | step 190 | avg_nll=23.85 avg_ppl=26.90 speed=4170.61 words/sec time_elapsed=19.41 sec\n",
            "epoch 1 (199/272) | step 200 | avg_nll=23.19 avg_ppl=25.73 speed=3948.68 words/sec time_elapsed=19.98 sec\n",
            "Begin validation ...\n",
            "validation: step 200 | dev_ppl=22.024646759033203\n",
            "epoch 1 step 200: save currently the best model to 'model.pt'\n",
            "epoch 1 (209/272) | step 210 | avg_nll=22.30 avg_ppl=22.64 speed=2638.57 words/sec time_elapsed=20.85 sec\n",
            "epoch 1 (219/272) | step 220 | avg_nll=22.95 avg_ppl=23.58 speed=2791.46 words/sec time_elapsed=21.68 sec\n",
            "epoch 1 (229/272) | step 230 | avg_nll=23.36 avg_ppl=24.22 speed=2755.19 words/sec time_elapsed=22.54 sec\n",
            "epoch 1 (239/272) | step 240 | avg_nll=22.35 avg_ppl=21.26 speed=2541.85 words/sec time_elapsed=23.46 sec\n",
            "epoch 1 (249/272) | step 250 | avg_nll=23.04 avg_ppl=23.40 speed=2432.09 words/sec time_elapsed=24.42 sec\n",
            "epoch 1 (259/272) | step 260 | avg_nll=22.19 avg_ppl=21.22 speed=3413.81 words/sec time_elapsed=25.10 sec\n",
            "epoch 1 (269/272) | step 270 | avg_nll=21.25 avg_ppl=19.12 speed=3579.89 words/sec time_elapsed=25.75 sec\n",
            "epoch 2 (7/272) | step 280 | avg_nll=21.66 avg_ppl=19.93 speed=3621.07 words/sec time_elapsed=26.39 sec\n",
            "epoch 2 (17/272) | step 290 | avg_nll=20.35 avg_ppl=17.48 speed=3998.97 words/sec time_elapsed=26.95 sec\n",
            "epoch 2 (27/272) | step 300 | avg_nll=21.26 avg_ppl=18.04 speed=3837.02 words/sec time_elapsed=27.57 sec\n",
            "Begin validation ...\n",
            "validation: step 300 | dev_ppl=17.335166931152344\n",
            "epoch 2 step 300: save currently the best model to 'model.pt'\n",
            "epoch 2 (37/272) | step 310 | avg_nll=20.38 avg_ppl=17.40 speed=2683.54 words/sec time_elapsed=28.42 sec\n",
            "epoch 2 (47/272) | step 320 | avg_nll=21.09 avg_ppl=18.62 speed=4278.58 words/sec time_elapsed=28.96 sec\n",
            "epoch 2 (57/272) | step 330 | avg_nll=21.59 avg_ppl=18.42 speed=4155.05 words/sec time_elapsed=29.53 sec\n",
            "epoch 2 (67/272) | step 340 | avg_nll=20.60 avg_ppl=17.12 speed=4190.60 words/sec time_elapsed=30.08 sec\n",
            "epoch 2 (77/272) | step 350 | avg_nll=20.35 avg_ppl=16.25 speed=3782.28 words/sec time_elapsed=30.70 sec\n",
            "epoch 2 (87/272) | step 360 | avg_nll=20.92 avg_ppl=18.24 speed=3985.31 words/sec time_elapsed=31.28 sec\n",
            "epoch 2 (97/272) | step 370 | avg_nll=19.81 avg_ppl=16.08 speed=3952.32 words/sec time_elapsed=31.86 sec\n",
            "epoch 2 (107/272) | step 380 | avg_nll=19.34 avg_ppl=14.71 speed=3756.08 words/sec time_elapsed=32.47 sec\n",
            "epoch 2 (117/272) | step 390 | avg_nll=20.22 avg_ppl=15.81 speed=3834.80 words/sec time_elapsed=33.08 sec\n",
            "epoch 2 (127/272) | step 400 | avg_nll=20.23 avg_ppl=16.48 speed=3800.67 words/sec time_elapsed=33.69 sec\n",
            "Begin validation ...\n",
            "validation: step 400 | dev_ppl=14.879033088684082\n",
            "epoch 2 step 400: save currently the best model to 'model.pt'\n",
            "epoch 2 (137/272) | step 410 | avg_nll=20.97 avg_ppl=16.67 speed=2491.10 words/sec time_elapsed=34.65 sec\n",
            "epoch 2 (147/272) | step 420 | avg_nll=21.00 avg_ppl=17.42 speed=2717.68 words/sec time_elapsed=35.51 sec\n",
            "epoch 2 (157/272) | step 430 | avg_nll=19.29 avg_ppl=14.46 speed=2409.80 words/sec time_elapsed=36.47 sec\n",
            "epoch 2 (167/272) | step 440 | avg_nll=20.00 avg_ppl=15.60 speed=2536.56 words/sec time_elapsed=37.39 sec\n",
            "epoch 2 (177/272) | step 450 | avg_nll=19.49 avg_ppl=14.63 speed=3127.30 words/sec time_elapsed=38.14 sec\n",
            "epoch 2 (187/272) | step 460 | avg_nll=19.83 avg_ppl=15.38 speed=3948.43 words/sec time_elapsed=38.73 sec\n",
            "epoch 2 (197/272) | step 470 | avg_nll=19.84 avg_ppl=15.47 speed=3721.42 words/sec time_elapsed=39.35 sec\n",
            "epoch 2 (207/272) | step 480 | avg_nll=19.67 avg_ppl=14.86 speed=3816.44 words/sec time_elapsed=39.96 sec\n",
            "epoch 2 (217/272) | step 490 | avg_nll=18.76 avg_ppl=13.88 speed=4080.14 words/sec time_elapsed=40.52 sec\n",
            "epoch 2 (227/272) | step 500 | avg_nll=19.49 avg_ppl=14.81 speed=3677.29 words/sec time_elapsed=41.15 sec\n",
            "Begin validation ...\n",
            "validation: step 500 | dev_ppl=13.63797378540039\n",
            "epoch 2 step 500: save currently the best model to 'model.pt'\n",
            "epoch 2 (237/272) | step 510 | avg_nll=18.78 avg_ppl=12.79 speed=2684.33 words/sec time_elapsed=42.03 sec\n",
            "epoch 2 (247/272) | step 520 | avg_nll=18.58 avg_ppl=13.46 speed=4137.75 words/sec time_elapsed=42.58 sec\n",
            "epoch 2 (257/272) | step 530 | avg_nll=17.84 avg_ppl=12.17 speed=4076.67 words/sec time_elapsed=43.14 sec\n",
            "epoch 2 (267/272) | step 540 | avg_nll=18.61 avg_ppl=13.30 speed=3932.39 words/sec time_elapsed=43.73 sec\n",
            "epoch 3 (5/272) | step 550 | avg_nll=17.88 avg_ppl=12.15 speed=3764.83 words/sec time_elapsed=44.33 sec\n",
            "epoch 3 (15/272) | step 560 | avg_nll=18.25 avg_ppl=12.62 speed=3856.24 words/sec time_elapsed=44.93 sec\n",
            "epoch 3 (25/272) | step 570 | avg_nll=17.94 avg_ppl=11.71 speed=3877.02 words/sec time_elapsed=45.53 sec\n",
            "epoch 3 (35/272) | step 580 | avg_nll=18.11 avg_ppl=12.49 speed=3889.32 words/sec time_elapsed=46.12 sec\n",
            "epoch 3 (45/272) | step 590 | avg_nll=18.29 avg_ppl=12.40 speed=4145.87 words/sec time_elapsed=46.69 sec\n",
            "epoch 3 (55/272) | step 600 | avg_nll=17.62 avg_ppl=11.60 speed=4029.61 words/sec time_elapsed=47.26 sec\n",
            "Begin validation ...\n",
            "validation: step 600 | dev_ppl=12.10177230834961\n",
            "epoch 3 step 600: save currently the best model to 'model.pt'\n",
            "epoch 3 (65/272) | step 610 | avg_nll=17.15 avg_ppl=10.91 speed=2254.87 words/sec time_elapsed=48.27 sec\n",
            "epoch 3 (75/272) | step 620 | avg_nll=17.70 avg_ppl=11.37 speed=2615.35 words/sec time_elapsed=49.17 sec\n",
            "epoch 3 (85/272) | step 630 | avg_nll=18.14 avg_ppl=12.11 speed=2652.30 words/sec time_elapsed=50.04 sec\n",
            "epoch 3 (95/272) | step 640 | avg_nll=17.92 avg_ppl=11.64 speed=2436.82 words/sec time_elapsed=51.00 sec\n",
            "epoch 3 (105/272) | step 650 | avg_nll=17.87 avg_ppl=11.68 speed=3598.07 words/sec time_elapsed=51.65 sec\n",
            "epoch 3 (115/272) | step 660 | avg_nll=18.49 avg_ppl=12.28 speed=3770.51 words/sec time_elapsed=52.28 sec\n",
            "epoch 3 (125/272) | step 670 | avg_nll=17.10 avg_ppl=11.10 speed=4035.18 words/sec time_elapsed=52.84 sec\n",
            "epoch 3 (135/272) | step 680 | avg_nll=18.11 avg_ppl=11.72 speed=3950.60 words/sec time_elapsed=53.44 sec\n",
            "epoch 3 (145/272) | step 690 | avg_nll=16.94 avg_ppl=10.59 speed=4064.90 words/sec time_elapsed=54.00 sec\n",
            "epoch 3 (155/272) | step 700 | avg_nll=17.80 avg_ppl=11.56 speed=4090.87 words/sec time_elapsed=54.57 sec\n",
            "Begin validation ...\n",
            "validation: step 700 | dev_ppl=11.383760452270508\n",
            "epoch 3 step 700: save currently the best model to 'model.pt'\n",
            "epoch 3 (165/272) | step 710 | avg_nll=17.15 avg_ppl=10.73 speed=2591.25 words/sec time_elapsed=55.46 sec\n",
            "epoch 3 (175/272) | step 720 | avg_nll=18.04 avg_ppl=11.92 speed=3983.29 words/sec time_elapsed=56.05 sec\n",
            "epoch 3 (185/272) | step 730 | avg_nll=17.31 avg_ppl=11.15 speed=3930.71 words/sec time_elapsed=56.63 sec\n",
            "epoch 3 (195/272) | step 740 | avg_nll=17.81 avg_ppl=11.77 speed=4093.50 words/sec time_elapsed=57.20 sec\n",
            "epoch 3 (205/272) | step 750 | avg_nll=17.96 avg_ppl=11.70 speed=4033.55 words/sec time_elapsed=57.78 sec\n",
            "epoch 3 (215/272) | step 760 | avg_nll=17.76 avg_ppl=11.47 speed=3772.98 words/sec time_elapsed=58.39 sec\n",
            "epoch 3 (225/272) | step 770 | avg_nll=17.75 avg_ppl=11.44 speed=4187.92 words/sec time_elapsed=58.95 sec\n",
            "epoch 3 (235/272) | step 780 | avg_nll=17.90 avg_ppl=11.62 speed=3480.49 words/sec time_elapsed=59.62 sec\n",
            "epoch 3 (245/272) | step 790 | avg_nll=16.30 avg_ppl=9.62 speed=3742.70 words/sec time_elapsed=60.24 sec\n",
            "epoch 3 (255/272) | step 800 | avg_nll=16.99 avg_ppl=10.63 speed=4152.63 words/sec time_elapsed=60.79 sec\n",
            "Begin validation ...\n",
            "validation: step 800 | dev_ppl=10.335407257080078\n",
            "epoch 3 step 800: save currently the best model to 'model.pt'\n",
            "epoch 3 (265/272) | step 810 | avg_nll=16.82 avg_ppl=10.22 speed=1923.65 words/sec time_elapsed=61.99 sec\n",
            "epoch 4 (3/272) | step 820 | avg_nll=15.95 avg_ppl=9.56 speed=2666.68 words/sec time_elapsed=62.84 sec\n",
            "epoch 4 (13/272) | step 830 | avg_nll=16.22 avg_ppl=9.57 speed=2648.89 words/sec time_elapsed=63.71 sec\n",
            "epoch 4 (23/272) | step 840 | avg_nll=16.79 avg_ppl=9.94 speed=2976.98 words/sec time_elapsed=64.50 sec\n",
            "epoch 4 (33/272) | step 850 | avg_nll=16.95 avg_ppl=10.05 speed=4039.35 words/sec time_elapsed=65.08 sec\n",
            "epoch 4 (43/272) | step 860 | avg_nll=16.04 avg_ppl=9.42 speed=3322.03 words/sec time_elapsed=65.77 sec\n",
            "epoch 4 (53/272) | step 870 | avg_nll=16.21 avg_ppl=9.05 speed=3965.23 words/sec time_elapsed=66.36 sec\n",
            "epoch 4 (63/272) | step 880 | avg_nll=15.99 avg_ppl=9.22 speed=3920.95 words/sec time_elapsed=66.95 sec\n",
            "epoch 4 (73/272) | step 890 | avg_nll=16.30 avg_ppl=9.45 speed=3470.64 words/sec time_elapsed=67.62 sec\n",
            "epoch 4 (83/272) | step 900 | avg_nll=16.30 avg_ppl=9.68 speed=3787.42 words/sec time_elapsed=68.23 sec\n",
            "Begin validation ...\n",
            "validation: step 900 | dev_ppl=9.812176704406738\n",
            "epoch 4 step 900: save currently the best model to 'model.pt'\n",
            "epoch 4 (93/272) | step 910 | avg_nll=15.76 avg_ppl=8.53 speed=2513.39 words/sec time_elapsed=69.16 sec\n",
            "epoch 4 (103/272) | step 920 | avg_nll=15.94 avg_ppl=9.28 speed=3362.67 words/sec time_elapsed=69.84 sec\n",
            "epoch 4 (113/272) | step 930 | avg_nll=15.88 avg_ppl=8.86 speed=3919.93 words/sec time_elapsed=70.44 sec\n",
            "epoch 4 (123/272) | step 940 | avg_nll=14.97 avg_ppl=8.17 speed=3803.18 words/sec time_elapsed=71.04 sec\n",
            "epoch 4 (133/272) | step 950 | avg_nll=15.69 avg_ppl=8.59 speed=4351.33 words/sec time_elapsed=71.58 sec\n",
            "epoch 4 (143/272) | step 960 | avg_nll=14.34 avg_ppl=7.65 speed=4026.78 words/sec time_elapsed=72.14 sec\n",
            "epoch 4 (153/272) | step 970 | avg_nll=15.84 avg_ppl=8.69 speed=3851.05 words/sec time_elapsed=72.75 sec\n",
            "epoch 4 (163/272) | step 980 | avg_nll=16.20 avg_ppl=9.53 speed=3992.94 words/sec time_elapsed=73.32 sec\n",
            "epoch 4 (173/272) | step 990 | avg_nll=15.91 avg_ppl=9.04 speed=3983.34 words/sec time_elapsed=73.90 sec\n",
            "epoch 4 (183/272) | step 1000 | avg_nll=16.80 avg_ppl=10.21 speed=3420.20 words/sec time_elapsed=74.58 sec\n",
            "Begin validation ...\n",
            "validation: step 1000 | dev_ppl=9.220876693725586\n",
            "epoch 4 step 1000: save currently the best model to 'model.pt'\n",
            "epoch 4 (193/272) | step 1010 | avg_nll=16.06 avg_ppl=9.04 speed=1827.49 words/sec time_elapsed=75.86 sec\n",
            "epoch 4 (203/272) | step 1020 | avg_nll=15.62 avg_ppl=8.42 speed=2691.65 words/sec time_elapsed=76.73 sec\n",
            "epoch 4 (213/272) | step 1030 | avg_nll=15.84 avg_ppl=8.95 speed=2558.77 words/sec time_elapsed=77.63 sec\n",
            "epoch 4 (223/272) | step 1040 | avg_nll=15.86 avg_ppl=8.83 speed=3863.46 words/sec time_elapsed=78.24 sec\n",
            "epoch 4 (233/272) | step 1050 | avg_nll=15.99 avg_ppl=8.82 speed=3483.56 words/sec time_elapsed=78.91 sec\n",
            "epoch 4 (243/272) | step 1060 | avg_nll=15.76 avg_ppl=8.94 speed=3643.07 words/sec time_elapsed=79.54 sec\n",
            "epoch 4 (253/272) | step 1070 | avg_nll=15.29 avg_ppl=8.20 speed=3566.81 words/sec time_elapsed=80.20 sec\n",
            "epoch 4 (263/272) | step 1080 | avg_nll=14.40 avg_ppl=7.55 speed=3310.74 words/sec time_elapsed=80.88 sec\n",
            "epoch 5 (1/272) | step 1090 | avg_nll=15.20 avg_ppl=8.23 speed=3973.63 words/sec time_elapsed=81.46 sec\n",
            "epoch 5 (11/272) | step 1100 | avg_nll=15.09 avg_ppl=7.90 speed=4049.67 words/sec time_elapsed=82.04 sec\n",
            "Begin validation ...\n",
            "validation: step 1100 | dev_ppl=8.67812728881836\n",
            "epoch 5 step 1100: save currently the best model to 'model.pt'\n",
            "epoch 5 (21/272) | step 1110 | avg_nll=14.92 avg_ppl=7.65 speed=2708.06 words/sec time_elapsed=82.91 sec\n",
            "epoch 5 (31/272) | step 1120 | avg_nll=13.71 avg_ppl=6.81 speed=3428.35 words/sec time_elapsed=83.58 sec\n",
            "epoch 5 (41/272) | step 1130 | avg_nll=13.75 avg_ppl=6.81 speed=3780.59 words/sec time_elapsed=84.18 sec\n",
            "epoch 5 (51/272) | step 1140 | avg_nll=15.02 avg_ppl=7.55 speed=4298.72 words/sec time_elapsed=84.74 sec\n",
            "epoch 5 (61/272) | step 1150 | avg_nll=14.24 avg_ppl=7.05 speed=3982.97 words/sec time_elapsed=85.32 sec\n",
            "epoch 5 (71/272) | step 1160 | avg_nll=15.36 avg_ppl=8.30 speed=4186.24 words/sec time_elapsed=85.88 sec\n",
            "epoch 5 (81/272) | step 1170 | avg_nll=14.80 avg_ppl=7.43 speed=3966.76 words/sec time_elapsed=86.47 sec\n",
            "epoch 5 (91/272) | step 1180 | avg_nll=14.61 avg_ppl=7.60 speed=3899.82 words/sec time_elapsed=87.06 sec\n",
            "epoch 5 (101/272) | step 1190 | avg_nll=14.43 avg_ppl=7.46 speed=3435.51 words/sec time_elapsed=87.73 sec\n",
            "epoch 5 (111/272) | step 1200 | avg_nll=14.77 avg_ppl=7.55 speed=2657.01 words/sec time_elapsed=88.61 sec\n",
            "Begin validation ...\n",
            "validation: step 1200 | dev_ppl=8.296404838562012\n",
            "epoch 5 step 1200: save currently the best model to 'model.pt'\n",
            "epoch 5 (121/272) | step 1210 | avg_nll=14.38 avg_ppl=7.46 speed=1769.23 words/sec time_elapsed=89.91 sec\n",
            "epoch 5 (131/272) | step 1220 | avg_nll=14.37 avg_ppl=7.27 speed=3002.48 words/sec time_elapsed=90.68 sec\n",
            "epoch 5 (141/272) | step 1230 | avg_nll=14.23 avg_ppl=7.06 speed=4031.49 words/sec time_elapsed=91.26 sec\n",
            "epoch 5 (151/272) | step 1240 | avg_nll=13.64 avg_ppl=6.79 speed=3435.13 words/sec time_elapsed=91.92 sec\n",
            "epoch 5 (161/272) | step 1250 | avg_nll=13.10 avg_ppl=6.24 speed=3836.19 words/sec time_elapsed=92.52 sec\n",
            "epoch 5 (171/272) | step 1260 | avg_nll=13.43 avg_ppl=6.55 speed=3764.42 words/sec time_elapsed=93.13 sec\n",
            "epoch 5 (181/272) | step 1270 | avg_nll=14.54 avg_ppl=7.59 speed=3288.99 words/sec time_elapsed=93.82 sec\n",
            "epoch 5 (191/272) | step 1280 | avg_nll=13.85 avg_ppl=6.77 speed=3982.92 words/sec time_elapsed=94.41 sec\n",
            "epoch 5 (201/272) | step 1290 | avg_nll=14.30 avg_ppl=7.14 speed=4095.07 words/sec time_elapsed=94.97 sec\n",
            "epoch 5 (211/272) | step 1300 | avg_nll=14.21 avg_ppl=7.25 speed=3649.73 words/sec time_elapsed=95.60 sec\n",
            "Begin validation ...\n",
            "validation: step 1300 | dev_ppl=7.872757434844971\n",
            "epoch 5 step 1300: save currently the best model to 'model.pt'\n",
            "epoch 5 (221/272) | step 1310 | avg_nll=14.10 avg_ppl=7.15 speed=2590.48 words/sec time_elapsed=96.49 sec\n",
            "epoch 5 (231/272) | step 1320 | avg_nll=15.43 avg_ppl=8.09 speed=3893.54 words/sec time_elapsed=97.10 sec\n",
            "epoch 5 (241/272) | step 1330 | avg_nll=14.42 avg_ppl=7.17 speed=4073.32 words/sec time_elapsed=97.67 sec\n",
            "epoch 5 (251/272) | step 1340 | avg_nll=14.16 avg_ppl=7.14 speed=4077.03 words/sec time_elapsed=98.24 sec\n",
            "epoch 5 (261/272) | step 1350 | avg_nll=15.13 avg_ppl=8.00 speed=3845.85 words/sec time_elapsed=98.84 sec\n",
            "epoch 5 (271/272) | step 1360 | avg_nll=13.64 avg_ppl=6.75 speed=4048.30 words/sec time_elapsed=99.41 sec\n",
            "epoch 6 (9/272) | step 1370 | avg_nll=13.91 avg_ppl=6.60 speed=4011.06 words/sec time_elapsed=99.99 sec\n",
            "epoch 6 (19/272) | step 1380 | avg_nll=13.09 avg_ppl=5.94 speed=3550.18 words/sec time_elapsed=100.66 sec\n",
            "epoch 6 (29/272) | step 1390 | avg_nll=12.52 avg_ppl=5.76 speed=2730.10 words/sec time_elapsed=101.50 sec\n",
            "epoch 6 (39/272) | step 1400 | avg_nll=13.66 avg_ppl=6.41 speed=1502.61 words/sec time_elapsed=103.06 sec\n",
            "Begin validation ...\n",
            "validation: step 1400 | dev_ppl=7.63429594039917\n",
            "epoch 6 step 1400: save currently the best model to 'model.pt'\n",
            "epoch 6 (49/272) | step 1410 | avg_nll=13.74 avg_ppl=6.43 speed=1085.95 words/sec time_elapsed=105.24 sec\n",
            "epoch 6 (59/272) | step 1420 | avg_nll=13.91 avg_ppl=6.81 speed=2418.41 words/sec time_elapsed=106.20 sec\n",
            "epoch 6 (69/272) | step 1430 | avg_nll=12.35 avg_ppl=5.64 speed=4059.83 words/sec time_elapsed=106.76 sec\n",
            "epoch 6 (79/272) | step 1440 | avg_nll=12.75 avg_ppl=5.86 speed=3910.78 words/sec time_elapsed=107.35 sec\n",
            "epoch 6 (89/272) | step 1450 | avg_nll=13.68 avg_ppl=6.62 speed=4085.13 words/sec time_elapsed=107.92 sec\n",
            "epoch 6 (99/272) | step 1460 | avg_nll=12.22 avg_ppl=5.52 speed=3557.37 words/sec time_elapsed=108.56 sec\n",
            "epoch 6 (109/272) | step 1470 | avg_nll=13.93 avg_ppl=6.79 speed=3631.31 words/sec time_elapsed=109.20 sec\n",
            "epoch 6 (119/272) | step 1480 | avg_nll=13.07 avg_ppl=6.12 speed=3889.91 words/sec time_elapsed=109.79 sec\n",
            "epoch 6 (129/272) | step 1490 | avg_nll=13.34 avg_ppl=6.31 speed=3499.53 words/sec time_elapsed=110.46 sec\n",
            "epoch 6 (139/272) | step 1500 | avg_nll=13.47 avg_ppl=6.37 speed=3596.01 words/sec time_elapsed=111.10 sec\n",
            "Begin validation ...\n",
            "validation: step 1500 | dev_ppl=7.360482215881348\n",
            "epoch 6 step 1500: save currently the best model to 'model.pt'\n",
            "epoch 6 (149/272) | step 1510 | avg_nll=12.84 avg_ppl=5.97 speed=2579.59 words/sec time_elapsed=112.00 sec\n",
            "epoch 6 (159/272) | step 1520 | avg_nll=13.10 avg_ppl=6.07 speed=4145.93 words/sec time_elapsed=112.56 sec\n",
            "epoch 6 (169/272) | step 1530 | avg_nll=13.82 avg_ppl=6.69 speed=3844.51 words/sec time_elapsed=113.16 sec\n",
            "epoch 6 (179/272) | step 1540 | avg_nll=12.45 avg_ppl=5.67 speed=4130.25 words/sec time_elapsed=113.72 sec\n",
            "epoch 6 (189/272) | step 1550 | avg_nll=12.99 avg_ppl=6.02 speed=2860.90 words/sec time_elapsed=114.53 sec\n",
            "epoch 6 (199/272) | step 1560 | avg_nll=12.43 avg_ppl=5.58 speed=2866.45 words/sec time_elapsed=115.34 sec\n",
            "epoch 6 (209/272) | step 1570 | avg_nll=13.63 avg_ppl=6.50 speed=2583.36 words/sec time_elapsed=116.24 sec\n",
            "epoch 6 (219/272) | step 1580 | avg_nll=13.21 avg_ppl=6.18 speed=2976.04 words/sec time_elapsed=117.02 sec\n",
            "epoch 6 (229/272) | step 1590 | avg_nll=12.90 avg_ppl=6.12 speed=3864.00 words/sec time_elapsed=117.61 sec\n",
            "epoch 6 (239/272) | step 1600 | avg_nll=12.86 avg_ppl=5.90 speed=4056.79 words/sec time_elapsed=118.18 sec\n",
            "Begin validation ...\n",
            "validation: step 1600 | dev_ppl=6.998519420623779\n",
            "epoch 6 step 1600: save currently the best model to 'model.pt'\n",
            "epoch 6 (249/272) | step 1610 | avg_nll=13.20 avg_ppl=6.26 speed=2512.20 words/sec time_elapsed=119.10 sec\n",
            "epoch 6 (259/272) | step 1620 | avg_nll=12.97 avg_ppl=6.02 speed=3932.09 words/sec time_elapsed=119.69 sec\n",
            "epoch 6 (269/272) | step 1630 | avg_nll=12.77 avg_ppl=5.81 speed=4133.96 words/sec time_elapsed=120.25 sec\n",
            "epoch 7 (7/272) | step 1640 | avg_nll=11.97 avg_ppl=5.25 speed=3673.31 words/sec time_elapsed=120.88 sec\n",
            "epoch 7 (17/272) | step 1650 | avg_nll=11.05 avg_ppl=4.65 speed=4007.87 words/sec time_elapsed=121.45 sec\n",
            "epoch 7 (27/272) | step 1660 | avg_nll=11.99 avg_ppl=5.15 speed=4061.60 words/sec time_elapsed=122.03 sec\n",
            "epoch 7 (37/272) | step 1670 | avg_nll=11.89 avg_ppl=5.15 speed=3930.21 words/sec time_elapsed=122.62 sec\n",
            "epoch 7 (47/272) | step 1680 | avg_nll=12.41 avg_ppl=5.41 speed=3942.64 words/sec time_elapsed=123.22 sec\n",
            "epoch 7 (57/272) | step 1690 | avg_nll=11.55 avg_ppl=5.06 speed=3834.24 words/sec time_elapsed=123.81 sec\n",
            "epoch 7 (67/272) | step 1700 | avg_nll=12.64 avg_ppl=5.50 speed=3842.26 words/sec time_elapsed=124.43 sec\n",
            "Begin validation ...\n",
            "validation: step 1700 | dev_ppl=6.794333457946777\n",
            "epoch 7 step 1700: save currently the best model to 'model.pt'\n",
            "epoch 7 (77/272) | step 1710 | avg_nll=11.23 avg_ppl=4.87 speed=2602.77 words/sec time_elapsed=125.30 sec\n",
            "epoch 7 (87/272) | step 1720 | avg_nll=11.68 avg_ppl=5.04 speed=3855.97 words/sec time_elapsed=125.90 sec\n",
            "epoch 7 (97/272) | step 1730 | avg_nll=11.68 avg_ppl=4.95 speed=3836.62 words/sec time_elapsed=126.51 sec\n",
            "epoch 7 (107/272) | step 1740 | avg_nll=12.03 avg_ppl=5.42 speed=3241.56 words/sec time_elapsed=127.21 sec\n",
            "epoch 7 (117/272) | step 1750 | avg_nll=12.87 avg_ppl=5.79 speed=2714.67 words/sec time_elapsed=128.08 sec\n",
            "epoch 7 (127/272) | step 1760 | avg_nll=12.79 avg_ppl=5.72 speed=2699.48 words/sec time_elapsed=128.95 sec\n",
            "epoch 7 (137/272) | step 1770 | avg_nll=11.86 avg_ppl=5.14 speed=2677.46 words/sec time_elapsed=129.81 sec\n",
            "epoch 7 (147/272) | step 1780 | avg_nll=12.55 avg_ppl=5.58 speed=3357.19 words/sec time_elapsed=130.51 sec\n",
            "epoch 7 (157/272) | step 1790 | avg_nll=11.32 avg_ppl=4.85 speed=3869.27 words/sec time_elapsed=131.10 sec\n",
            "epoch 7 (167/272) | step 1800 | avg_nll=12.19 avg_ppl=5.40 speed=4137.99 words/sec time_elapsed=131.66 sec\n",
            "Begin validation ...\n",
            "validation: step 1800 | dev_ppl=6.693475723266602\n",
            "epoch 7 step 1800: save currently the best model to 'model.pt'\n",
            "epoch 7 (177/272) | step 1810 | avg_nll=11.49 avg_ppl=4.88 speed=2615.16 words/sec time_elapsed=132.55 sec\n",
            "epoch 7 (187/272) | step 1820 | avg_nll=12.00 avg_ppl=5.29 speed=3685.72 words/sec time_elapsed=133.17 sec\n",
            "epoch 7 (197/272) | step 1830 | avg_nll=12.52 avg_ppl=5.55 speed=3429.22 words/sec time_elapsed=133.86 sec\n",
            "epoch 7 (207/272) | step 1840 | avg_nll=12.00 avg_ppl=5.25 speed=3969.37 words/sec time_elapsed=134.44 sec\n",
            "epoch 7 (217/272) | step 1850 | avg_nll=12.14 avg_ppl=5.49 speed=3966.23 words/sec time_elapsed=135.01 sec\n",
            "epoch 7 (227/272) | step 1860 | avg_nll=12.80 avg_ppl=5.75 speed=3811.33 words/sec time_elapsed=135.63 sec\n",
            "epoch 7 (237/272) | step 1870 | avg_nll=11.04 avg_ppl=4.62 speed=3914.00 words/sec time_elapsed=136.22 sec\n",
            "epoch 7 (247/272) | step 1880 | avg_nll=11.69 avg_ppl=5.02 speed=4105.67 words/sec time_elapsed=136.78 sec\n",
            "epoch 7 (257/272) | step 1890 | avg_nll=11.32 avg_ppl=4.87 speed=3692.81 words/sec time_elapsed=137.40 sec\n",
            "epoch 7 (267/272) | step 1900 | avg_nll=11.50 avg_ppl=5.04 speed=3518.52 words/sec time_elapsed=138.05 sec\n",
            "Begin validation ...\n",
            "validation: step 1900 | dev_ppl=6.447020053863525\n",
            "epoch 7 step 1900: save currently the best model to 'model.pt'\n",
            "epoch 8 (5/272) | step 1910 | avg_nll=12.36 avg_ppl=5.30 speed=2575.60 words/sec time_elapsed=138.97 sec\n",
            "epoch 8 (15/272) | step 1920 | avg_nll=11.39 avg_ppl=4.63 speed=3986.65 words/sec time_elapsed=139.57 sec\n",
            "epoch 8 (25/272) | step 1930 | avg_nll=10.36 avg_ppl=4.27 speed=3875.68 words/sec time_elapsed=140.16 sec\n",
            "epoch 8 (35/272) | step 1940 | avg_nll=10.50 avg_ppl=4.33 speed=2616.71 words/sec time_elapsed=141.03 sec\n",
            "epoch 8 (45/272) | step 1950 | avg_nll=11.08 avg_ppl=4.53 speed=2833.08 words/sec time_elapsed=141.86 sec\n",
            "epoch 8 (55/272) | step 1960 | avg_nll=11.36 avg_ppl=4.73 speed=2608.57 words/sec time_elapsed=142.76 sec\n",
            "epoch 8 (65/272) | step 1970 | avg_nll=10.69 avg_ppl=4.37 speed=3046.31 words/sec time_elapsed=143.52 sec\n",
            "epoch 8 (75/272) | step 1980 | avg_nll=10.51 avg_ppl=4.31 speed=3968.78 words/sec time_elapsed=144.10 sec\n",
            "epoch 8 (85/272) | step 1990 | avg_nll=10.94 avg_ppl=4.51 speed=4103.14 words/sec time_elapsed=144.67 sec\n",
            "epoch 8 (95/272) | step 2000 | avg_nll=11.07 avg_ppl=4.69 speed=3903.54 words/sec time_elapsed=145.26 sec\n",
            "Begin validation ...\n",
            "validation: step 2000 | dev_ppl=6.4357500076293945\n",
            "epoch 8 step 2000: save currently the best model to 'model.pt'\n",
            "epoch 8 (105/272) | step 2010 | avg_nll=10.50 avg_ppl=4.21 speed=2695.40 words/sec time_elapsed=146.12 sec\n",
            "epoch 8 (115/272) | step 2020 | avg_nll=10.65 avg_ppl=4.45 speed=2681.84 words/sec time_elapsed=146.97 sec\n",
            "epoch 8 (125/272) | step 2030 | avg_nll=11.00 avg_ppl=4.64 speed=2928.48 words/sec time_elapsed=147.76 sec\n",
            "epoch 8 (135/272) | step 2040 | avg_nll=10.20 avg_ppl=4.16 speed=2705.27 words/sec time_elapsed=148.61 sec\n",
            "epoch 8 (145/272) | step 2050 | avg_nll=11.62 avg_ppl=4.89 speed=3305.29 words/sec time_elapsed=149.32 sec\n",
            "epoch 8 (155/272) | step 2060 | avg_nll=10.95 avg_ppl=4.55 speed=3985.63 words/sec time_elapsed=149.90 sec\n",
            "epoch 8 (165/272) | step 2070 | avg_nll=11.52 avg_ppl=4.94 speed=3907.07 words/sec time_elapsed=150.49 sec\n",
            "epoch 8 (175/272) | step 2080 | avg_nll=11.19 avg_ppl=4.69 speed=3758.17 words/sec time_elapsed=151.10 sec\n",
            "epoch 8 (185/272) | step 2090 | avg_nll=10.60 avg_ppl=4.38 speed=4125.85 words/sec time_elapsed=151.66 sec\n",
            "epoch 8 (195/272) | step 2100 | avg_nll=10.94 avg_ppl=4.51 speed=4027.96 words/sec time_elapsed=152.24 sec\n",
            "Begin validation ...\n",
            "validation: step 2100 | dev_ppl=6.302468776702881\n",
            "epoch 8 step 2100: save currently the best model to 'model.pt'\n",
            "epoch 8 (205/272) | step 2110 | avg_nll=11.20 avg_ppl=4.65 speed=2598.62 words/sec time_elapsed=153.13 sec\n",
            "epoch 8 (215/272) | step 2120 | avg_nll=10.59 avg_ppl=4.34 speed=2910.15 words/sec time_elapsed=153.93 sec\n",
            "epoch 8 (225/272) | step 2130 | avg_nll=11.13 avg_ppl=4.57 speed=2676.11 words/sec time_elapsed=154.80 sec\n",
            "epoch 8 (235/272) | step 2140 | avg_nll=11.08 avg_ppl=4.70 speed=2476.71 words/sec time_elapsed=155.73 sec\n",
            "epoch 8 (245/272) | step 2150 | avg_nll=11.23 avg_ppl=4.66 speed=2594.97 words/sec time_elapsed=156.63 sec\n",
            "epoch 8 (255/272) | step 2160 | avg_nll=10.67 avg_ppl=4.44 speed=3512.78 words/sec time_elapsed=157.29 sec\n",
            "epoch 8 (265/272) | step 2170 | avg_nll=11.43 avg_ppl=4.83 speed=3556.51 words/sec time_elapsed=157.94 sec\n",
            "epoch 9 (3/272) | step 2180 | avg_nll=10.23 avg_ppl=4.08 speed=4054.28 words/sec time_elapsed=158.51 sec\n",
            "epoch 9 (13/272) | step 2190 | avg_nll=9.04 avg_ppl=3.61 speed=3955.00 words/sec time_elapsed=159.08 sec\n",
            "epoch 9 (23/272) | step 2200 | avg_nll=9.84 avg_ppl=3.87 speed=4030.27 words/sec time_elapsed=159.66 sec\n",
            "Begin validation ...\n",
            "validation: step 2200 | dev_ppl=6.028085708618164\n",
            "epoch 9 step 2200: save currently the best model to 'model.pt'\n",
            "epoch 9 (33/272) | step 2210 | avg_nll=9.86 avg_ppl=3.97 speed=2531.27 words/sec time_elapsed=160.57 sec\n",
            "epoch 9 (43/272) | step 2220 | avg_nll=10.26 avg_ppl=4.12 speed=3947.69 words/sec time_elapsed=161.15 sec\n",
            "epoch 9 (53/272) | step 2230 | avg_nll=9.79 avg_ppl=3.87 speed=4020.61 words/sec time_elapsed=161.73 sec\n",
            "epoch 9 (63/272) | step 2240 | avg_nll=10.09 avg_ppl=4.03 speed=3966.94 words/sec time_elapsed=162.31 sec\n",
            "epoch 9 (73/272) | step 2250 | avg_nll=9.34 avg_ppl=3.75 speed=3948.98 words/sec time_elapsed=162.88 sec\n",
            "epoch 9 (83/272) | step 2260 | avg_nll=9.86 avg_ppl=3.89 speed=4001.25 words/sec time_elapsed=163.46 sec\n",
            "epoch 9 (93/272) | step 2270 | avg_nll=9.64 avg_ppl=3.73 speed=4129.27 words/sec time_elapsed=164.03 sec\n",
            "epoch 9 (103/272) | step 2280 | avg_nll=10.14 avg_ppl=4.10 speed=3962.58 words/sec time_elapsed=164.61 sec\n",
            "epoch 9 (113/272) | step 2290 | avg_nll=10.45 avg_ppl=4.19 speed=3921.92 words/sec time_elapsed=165.21 sec\n",
            "epoch 9 (123/272) | step 2300 | avg_nll=9.84 avg_ppl=3.88 speed=3739.30 words/sec time_elapsed=165.83 sec\n",
            "Begin validation ...\n",
            "validation: step 2300 | dev_ppl=5.939603328704834\n",
            "epoch 9 step 2300: save currently the best model to 'model.pt'\n",
            "epoch 9 (133/272) | step 2310 | avg_nll=9.83 avg_ppl=3.88 speed=2386.00 words/sec time_elapsed=166.80 sec\n",
            "epoch 9 (143/272) | step 2320 | avg_nll=9.88 avg_ppl=3.93 speed=2733.01 words/sec time_elapsed=167.65 sec\n",
            "epoch 9 (153/272) | step 2330 | avg_nll=9.90 avg_ppl=3.85 speed=2707.67 words/sec time_elapsed=168.52 sec\n",
            "epoch 9 (163/272) | step 2340 | avg_nll=10.08 avg_ppl=4.03 speed=2569.11 words/sec time_elapsed=169.42 sec\n",
            "epoch 9 (173/272) | step 2350 | avg_nll=10.32 avg_ppl=4.09 speed=3285.93 words/sec time_elapsed=170.13 sec\n",
            "epoch 9 (183/272) | step 2360 | avg_nll=9.77 avg_ppl=3.90 speed=3345.00 words/sec time_elapsed=170.82 sec\n",
            "epoch 9 (193/272) | step 2370 | avg_nll=9.95 avg_ppl=4.02 speed=3785.07 words/sec time_elapsed=171.42 sec\n",
            "epoch 9 (203/272) | step 2380 | avg_nll=10.32 avg_ppl=4.12 speed=4009.70 words/sec time_elapsed=172.01 sec\n",
            "epoch 9 (213/272) | step 2390 | avg_nll=9.37 avg_ppl=3.69 speed=4131.62 words/sec time_elapsed=172.56 sec\n",
            "epoch 9 (223/272) | step 2400 | avg_nll=10.57 avg_ppl=4.22 speed=4035.04 words/sec time_elapsed=173.14 sec\n",
            "Begin validation ...\n",
            "validation: step 2400 | dev_ppl=5.951430320739746\n",
            "epoch 9 (233/272) | step 2410 | avg_nll=10.09 avg_ppl=3.98 speed=2647.27 words/sec time_elapsed=174.03 sec\n",
            "epoch 9 (243/272) | step 2420 | avg_nll=10.66 avg_ppl=4.29 speed=4025.68 words/sec time_elapsed=174.61 sec\n",
            "epoch 9 (253/272) | step 2430 | avg_nll=10.07 avg_ppl=4.01 speed=4151.63 words/sec time_elapsed=175.17 sec\n",
            "epoch 9 (263/272) | step 2440 | avg_nll=10.27 avg_ppl=4.08 speed=3826.28 words/sec time_elapsed=175.78 sec\n",
            "epoch 10 (1/272) | step 2450 | avg_nll=9.58 avg_ppl=3.79 speed=4071.55 words/sec time_elapsed=176.34 sec\n",
            "epoch 10 (11/272) | step 2460 | avg_nll=8.97 avg_ppl=3.48 speed=3874.89 words/sec time_elapsed=176.94 sec\n",
            "epoch 10 (21/272) | step 2470 | avg_nll=9.68 avg_ppl=3.80 speed=3613.43 words/sec time_elapsed=177.58 sec\n",
            "epoch 10 (31/272) | step 2480 | avg_nll=8.99 avg_ppl=3.44 speed=4028.33 words/sec time_elapsed=178.16 sec\n",
            "epoch 10 (41/272) | step 2490 | avg_nll=8.83 avg_ppl=3.32 speed=3905.96 words/sec time_elapsed=178.76 sec\n",
            "epoch 10 (51/272) | step 2500 | avg_nll=9.03 avg_ppl=3.50 speed=3899.23 words/sec time_elapsed=179.35 sec\n",
            "Begin validation ...\n",
            "validation: step 2500 | dev_ppl=5.755066394805908\n",
            "epoch 10 step 2500: save currently the best model to 'model.pt'\n",
            "epoch 10 (61/272) | step 2510 | avg_nll=8.87 avg_ppl=3.46 speed=2092.89 words/sec time_elapsed=180.45 sec\n",
            "epoch 10 (71/272) | step 2520 | avg_nll=8.81 avg_ppl=3.40 speed=2937.71 words/sec time_elapsed=181.23 sec\n",
            "epoch 10 (81/272) | step 2530 | avg_nll=8.81 avg_ppl=3.37 speed=2577.23 words/sec time_elapsed=182.14 sec\n",
            "epoch 10 (91/272) | step 2540 | avg_nll=9.26 avg_ppl=3.55 speed=2598.57 words/sec time_elapsed=183.04 sec\n",
            "epoch 10 (101/272) | step 2550 | avg_nll=9.75 avg_ppl=3.79 speed=3604.91 words/sec time_elapsed=183.69 sec\n",
            "epoch 10 (111/272) | step 2560 | avg_nll=9.18 avg_ppl=3.49 speed=3850.13 words/sec time_elapsed=184.30 sec\n",
            "epoch 10 (121/272) | step 2570 | avg_nll=8.80 avg_ppl=3.35 speed=3855.64 words/sec time_elapsed=184.90 sec\n",
            "epoch 10 (131/272) | step 2580 | avg_nll=9.80 avg_ppl=3.93 speed=3882.78 words/sec time_elapsed=185.49 sec\n",
            "epoch 10 (141/272) | step 2590 | avg_nll=9.78 avg_ppl=3.76 speed=4110.40 words/sec time_elapsed=186.07 sec\n",
            "epoch 10 (151/272) | step 2600 | avg_nll=8.94 avg_ppl=3.45 speed=4084.84 words/sec time_elapsed=186.63 sec\n",
            "Begin validation ...\n",
            "validation: step 2600 | dev_ppl=5.730888366699219\n",
            "epoch 10 step 2600: save currently the best model to 'model.pt'\n",
            "epoch 10 (161/272) | step 2610 | avg_nll=8.87 avg_ppl=3.43 speed=2433.16 words/sec time_elapsed=187.58 sec\n",
            "epoch 10 (171/272) | step 2620 | avg_nll=9.21 avg_ppl=3.55 speed=3730.52 words/sec time_elapsed=188.20 sec\n",
            "epoch 10 (181/272) | step 2630 | avg_nll=8.88 avg_ppl=3.43 speed=4028.19 words/sec time_elapsed=188.78 sec\n",
            "epoch 10 (191/272) | step 2640 | avg_nll=8.94 avg_ppl=3.50 speed=4128.23 words/sec time_elapsed=189.33 sec\n",
            "epoch 10 (201/272) | step 2650 | avg_nll=8.74 avg_ppl=3.26 speed=3951.37 words/sec time_elapsed=189.93 sec\n",
            "epoch 10 (211/272) | step 2660 | avg_nll=8.81 avg_ppl=3.39 speed=3387.27 words/sec time_elapsed=190.61 sec\n",
            "epoch 10 (221/272) | step 2670 | avg_nll=8.74 avg_ppl=3.46 speed=3964.94 words/sec time_elapsed=191.18 sec\n",
            "epoch 10 (231/272) | step 2680 | avg_nll=9.70 avg_ppl=3.78 speed=3916.01 words/sec time_elapsed=191.77 sec\n",
            "epoch 10 (241/272) | step 2690 | avg_nll=8.70 avg_ppl=3.41 speed=3611.57 words/sec time_elapsed=192.40 sec\n",
            "epoch 10 (251/272) | step 2700 | avg_nll=8.61 avg_ppl=3.30 speed=3598.44 words/sec time_elapsed=193.04 sec\n",
            "Begin validation ...\n",
            "validation: step 2700 | dev_ppl=5.656057834625244\n",
            "epoch 10 step 2700: save currently the best model to 'model.pt'\n",
            "epoch 10 (261/272) | step 2710 | avg_nll=10.41 avg_ppl=4.13 speed=1882.23 words/sec time_elapsed=194.29 sec\n",
            "epoch 10 (271/272) | step 2720 | avg_nll=8.68 avg_ppl=3.33 speed=2799.21 words/sec time_elapsed=195.12 sec\n",
            "epoch 11 (9/272) | step 2730 | avg_nll=7.61 avg_ppl=2.87 speed=2528.07 words/sec time_elapsed=196.03 sec\n",
            "epoch 11 (19/272) | step 2740 | avg_nll=7.69 avg_ppl=2.97 speed=2776.37 words/sec time_elapsed=196.85 sec\n",
            "epoch 11 (29/272) | step 2750 | avg_nll=9.35 avg_ppl=3.53 speed=4045.91 words/sec time_elapsed=197.44 sec\n",
            "epoch 11 (39/272) | step 2760 | avg_nll=8.11 avg_ppl=3.07 speed=3894.02 words/sec time_elapsed=198.03 sec\n",
            "epoch 11 (49/272) | step 2770 | avg_nll=7.66 avg_ppl=2.84 speed=3995.36 words/sec time_elapsed=198.62 sec\n",
            "epoch 11 (59/272) | step 2780 | avg_nll=7.94 avg_ppl=3.04 speed=3856.59 words/sec time_elapsed=199.21 sec\n",
            "epoch 11 (69/272) | step 2790 | avg_nll=8.17 avg_ppl=3.14 speed=3536.51 words/sec time_elapsed=199.86 sec\n",
            "epoch 11 (79/272) | step 2800 | avg_nll=7.79 avg_ppl=2.98 speed=3795.97 words/sec time_elapsed=200.46 sec\n",
            "Begin validation ...\n",
            "validation: step 2800 | dev_ppl=5.755591869354248\n",
            "epoch 11 (89/272) | step 2810 | avg_nll=8.04 avg_ppl=3.06 speed=2634.32 words/sec time_elapsed=201.33 sec\n",
            "epoch 11 (99/272) | step 2820 | avg_nll=8.10 avg_ppl=3.02 speed=4106.04 words/sec time_elapsed=201.90 sec\n",
            "epoch 11 (109/272) | step 2830 | avg_nll=7.86 avg_ppl=2.98 speed=3794.76 words/sec time_elapsed=202.51 sec\n",
            "epoch 11 (119/272) | step 2840 | avg_nll=8.88 avg_ppl=3.35 speed=3672.82 words/sec time_elapsed=203.15 sec\n",
            "epoch 11 (129/272) | step 2850 | avg_nll=8.05 avg_ppl=3.04 speed=4099.49 words/sec time_elapsed=203.72 sec\n",
            "epoch 11 (139/272) | step 2860 | avg_nll=8.88 avg_ppl=3.37 speed=4033.09 words/sec time_elapsed=204.30 sec\n",
            "epoch 11 (149/272) | step 2870 | avg_nll=7.66 avg_ppl=2.92 speed=4001.84 words/sec time_elapsed=204.87 sec\n",
            "epoch 11 (159/272) | step 2880 | avg_nll=8.32 avg_ppl=3.17 speed=3624.13 words/sec time_elapsed=205.51 sec\n",
            "epoch 11 (169/272) | step 2890 | avg_nll=8.55 avg_ppl=3.25 speed=3538.75 words/sec time_elapsed=206.16 sec\n",
            "epoch 11 (179/272) | step 2900 | avg_nll=8.28 avg_ppl=3.13 speed=4082.69 words/sec time_elapsed=206.73 sec\n",
            "Begin validation ...\n",
            "validation: step 2900 | dev_ppl=5.716769695281982\n",
            "epoch 11 (189/272) | step 2910 | avg_nll=8.20 avg_ppl=3.11 speed=1734.23 words/sec time_elapsed=208.07 sec\n",
            "epoch 11 (199/272) | step 2920 | avg_nll=8.30 avg_ppl=3.17 speed=2792.47 words/sec time_elapsed=208.89 sec\n",
            "epoch 11 (209/272) | step 2930 | avg_nll=8.60 avg_ppl=3.26 speed=2402.13 words/sec time_elapsed=209.86 sec\n",
            "epoch 11 (219/272) | step 2940 | avg_nll=8.59 avg_ppl=3.26 speed=2738.45 words/sec time_elapsed=210.71 sec\n",
            "epoch 11 (229/272) | step 2950 | avg_nll=8.99 avg_ppl=3.42 speed=4118.52 words/sec time_elapsed=211.28 sec\n",
            "epoch 11 (239/272) | step 2960 | avg_nll=8.33 avg_ppl=3.13 speed=4041.41 words/sec time_elapsed=211.86 sec\n",
            "epoch 11 (249/272) | step 2970 | avg_nll=9.07 avg_ppl=3.43 speed=4240.95 words/sec time_elapsed=212.42 sec\n",
            "epoch 11 (259/272) | step 2980 | avg_nll=8.55 avg_ppl=3.28 speed=3890.79 words/sec time_elapsed=213.01 sec\n",
            "epoch 11 (269/272) | step 2990 | avg_nll=8.38 avg_ppl=3.25 speed=4111.59 words/sec time_elapsed=213.56 sec\n",
            "epoch 12 (7/272) | step 3000 | avg_nll=7.25 avg_ppl=2.78 speed=4202.35 words/sec time_elapsed=214.10 sec\n",
            "Begin validation ...\n",
            "validation: step 3000 | dev_ppl=5.743962287902832\n",
            "epoch 12 (17/272) | step 3010 | avg_nll=6.99 avg_ppl=2.66 speed=2707.88 words/sec time_elapsed=214.95 sec\n",
            "epoch 12 (27/272) | step 3020 | avg_nll=7.37 avg_ppl=2.76 speed=4095.52 words/sec time_elapsed=215.51 sec\n",
            "epoch 12 (37/272) | step 3030 | avg_nll=7.39 avg_ppl=2.83 speed=4050.82 words/sec time_elapsed=216.08 sec\n",
            "epoch 12 (47/272) | step 3040 | avg_nll=6.70 avg_ppl=2.61 speed=4124.12 words/sec time_elapsed=216.62 sec\n",
            "epoch 12 (57/272) | step 3050 | avg_nll=7.64 avg_ppl=2.88 speed=3856.66 words/sec time_elapsed=217.22 sec\n",
            "epoch 12 (67/272) | step 3060 | avg_nll=7.46 avg_ppl=2.84 speed=3897.07 words/sec time_elapsed=217.80 sec\n",
            "epoch 12 (77/272) | step 3070 | avg_nll=7.61 avg_ppl=2.88 speed=3819.26 words/sec time_elapsed=218.41 sec\n",
            "epoch 12 (87/272) | step 3080 | avg_nll=7.38 avg_ppl=2.73 speed=3581.44 words/sec time_elapsed=219.06 sec\n",
            "epoch 12 (97/272) | step 3090 | avg_nll=7.49 avg_ppl=2.82 speed=3472.49 words/sec time_elapsed=219.73 sec\n",
            "epoch 12 (107/272) | step 3100 | avg_nll=8.05 avg_ppl=2.99 speed=3824.52 words/sec time_elapsed=220.34 sec\n",
            "Begin validation ...\n",
            "validation: step 3100 | dev_ppl=5.56973123550415\n",
            "epoch 12 step 3100: save currently the best model to 'model.pt'\n",
            "epoch 12 (117/272) | step 3110 | avg_nll=7.30 avg_ppl=2.74 speed=1746.91 words/sec time_elapsed=221.67 sec\n",
            "epoch 12 (127/272) | step 3120 | avg_nll=7.66 avg_ppl=2.84 speed=2533.72 words/sec time_elapsed=222.60 sec\n",
            "epoch 12 (137/272) | step 3130 | avg_nll=7.19 avg_ppl=2.64 speed=2671.43 words/sec time_elapsed=223.49 sec\n",
            "epoch 12 (147/272) | step 3140 | avg_nll=7.61 avg_ppl=2.84 speed=3791.50 words/sec time_elapsed=224.11 sec\n",
            "epoch 12 (157/272) | step 3150 | avg_nll=7.32 avg_ppl=2.75 speed=4064.87 words/sec time_elapsed=224.67 sec\n",
            "epoch 12 (167/272) | step 3160 | avg_nll=7.21 avg_ppl=2.70 speed=3965.04 words/sec time_elapsed=225.26 sec\n",
            "epoch 12 (177/272) | step 3170 | avg_nll=7.68 avg_ppl=2.83 speed=4081.49 words/sec time_elapsed=225.84 sec\n",
            "epoch 12 (187/272) | step 3180 | avg_nll=7.47 avg_ppl=2.82 speed=4165.11 words/sec time_elapsed=226.39 sec\n",
            "epoch 12 (197/272) | step 3190 | avg_nll=7.67 avg_ppl=2.88 speed=3844.71 words/sec time_elapsed=227.00 sec\n",
            "epoch 12 (207/272) | step 3200 | avg_nll=8.07 avg_ppl=3.00 speed=3556.17 words/sec time_elapsed=227.66 sec\n",
            "Begin validation ...\n",
            "validation: step 3200 | dev_ppl=5.50862979888916\n",
            "epoch 12 step 3200: save currently the best model to 'model.pt'\n",
            "epoch 12 (217/272) | step 3210 | avg_nll=7.92 avg_ppl=3.00 speed=2470.41 words/sec time_elapsed=228.59 sec\n",
            "epoch 12 (227/272) | step 3220 | avg_nll=7.90 avg_ppl=2.93 speed=3702.72 words/sec time_elapsed=229.23 sec\n",
            "epoch 12 (237/272) | step 3230 | avg_nll=8.50 avg_ppl=3.17 speed=4173.49 words/sec time_elapsed=229.79 sec\n",
            "epoch 12 (247/272) | step 3240 | avg_nll=7.53 avg_ppl=2.85 speed=3874.48 words/sec time_elapsed=230.39 sec\n",
            "epoch 12 (257/272) | step 3250 | avg_nll=8.26 avg_ppl=3.13 speed=4042.26 words/sec time_elapsed=230.96 sec\n",
            "epoch 12 (267/272) | step 3260 | avg_nll=7.60 avg_ppl=2.88 speed=3988.83 words/sec time_elapsed=231.54 sec\n",
            "epoch 13 (5/272) | step 3270 | avg_nll=6.87 avg_ppl=2.62 speed=3733.47 words/sec time_elapsed=232.15 sec\n",
            "epoch 13 (15/272) | step 3280 | avg_nll=6.77 avg_ppl=2.55 speed=3431.22 words/sec time_elapsed=232.82 sec\n",
            "epoch 13 (25/272) | step 3290 | avg_nll=6.76 avg_ppl=2.52 speed=3638.06 words/sec time_elapsed=233.46 sec\n",
            "epoch 13 (35/272) | step 3300 | avg_nll=6.63 avg_ppl=2.49 speed=2885.44 words/sec time_elapsed=234.27 sec\n",
            "Begin validation ...\n",
            "validation: step 3300 | dev_ppl=5.488207817077637\n",
            "epoch 13 step 3300: save currently the best model to 'model.pt'\n",
            "epoch 13 (45/272) | step 3310 | avg_nll=6.53 avg_ppl=2.47 speed=1721.64 words/sec time_elapsed=235.61 sec\n",
            "epoch 13 (55/272) | step 3320 | avg_nll=6.45 avg_ppl=2.47 speed=2536.34 words/sec time_elapsed=236.51 sec\n",
            "epoch 13 (65/272) | step 3330 | avg_nll=6.96 avg_ppl=2.62 speed=3057.54 words/sec time_elapsed=237.27 sec\n",
            "epoch 13 (75/272) | step 3340 | avg_nll=6.09 avg_ppl=2.34 speed=4192.25 words/sec time_elapsed=237.82 sec\n",
            "epoch 13 (85/272) | step 3350 | avg_nll=6.39 avg_ppl=2.46 speed=4229.49 words/sec time_elapsed=238.36 sec\n",
            "epoch 13 (95/272) | step 3360 | avg_nll=7.33 avg_ppl=2.73 speed=3969.21 words/sec time_elapsed=238.95 sec\n",
            "epoch 13 (105/272) | step 3370 | avg_nll=6.58 avg_ppl=2.47 speed=3499.51 words/sec time_elapsed=239.61 sec\n",
            "epoch 13 (115/272) | step 3380 | avg_nll=6.63 avg_ppl=2.49 speed=3539.51 words/sec time_elapsed=240.27 sec\n",
            "epoch 13 (125/272) | step 3390 | avg_nll=7.04 avg_ppl=2.66 speed=3411.73 words/sec time_elapsed=240.94 sec\n",
            "epoch 13 (135/272) | step 3400 | avg_nll=7.14 avg_ppl=2.70 speed=3966.74 words/sec time_elapsed=241.52 sec\n",
            "Begin validation ...\n",
            "validation: step 3400 | dev_ppl=5.422985553741455\n",
            "epoch 13 step 3400: save currently the best model to 'model.pt'\n",
            "epoch 13 (145/272) | step 3410 | avg_nll=6.55 avg_ppl=2.50 speed=2602.22 words/sec time_elapsed=242.40 sec\n",
            "epoch 13 (155/272) | step 3420 | avg_nll=7.03 avg_ppl=2.67 speed=4149.67 words/sec time_elapsed=242.95 sec\n",
            "epoch 13 (165/272) | step 3430 | avg_nll=6.83 avg_ppl=2.58 speed=3956.24 words/sec time_elapsed=243.54 sec\n",
            "epoch 13 (175/272) | step 3440 | avg_nll=7.37 avg_ppl=2.76 speed=3789.91 words/sec time_elapsed=244.15 sec\n",
            "epoch 13 (185/272) | step 3450 | avg_nll=7.71 avg_ppl=2.88 speed=3963.38 words/sec time_elapsed=244.74 sec\n",
            "epoch 13 (195/272) | step 3460 | avg_nll=6.66 avg_ppl=2.55 speed=4013.93 words/sec time_elapsed=245.30 sec\n",
            "epoch 13 (205/272) | step 3470 | avg_nll=6.25 avg_ppl=2.39 speed=3944.27 words/sec time_elapsed=245.89 sec\n",
            "epoch 13 (215/272) | step 3480 | avg_nll=7.08 avg_ppl=2.62 speed=3659.98 words/sec time_elapsed=246.53 sec\n",
            "epoch 13 (225/272) | step 3490 | avg_nll=7.02 avg_ppl=2.63 speed=3540.79 words/sec time_elapsed=247.19 sec\n",
            "epoch 13 (235/272) | step 3500 | avg_nll=7.32 avg_ppl=2.66 speed=2807.35 words/sec time_elapsed=248.04 sec\n",
            "Begin validation ...\n",
            "validation: step 3500 | dev_ppl=5.582517147064209\n",
            "epoch 13 (245/272) | step 3510 | avg_nll=7.21 avg_ppl=2.67 speed=1962.17 words/sec time_elapsed=249.24 sec\n",
            "epoch 13 (255/272) | step 3520 | avg_nll=7.54 avg_ppl=2.77 speed=2692.37 words/sec time_elapsed=250.12 sec\n",
            "epoch 13 (265/272) | step 3530 | avg_nll=7.72 avg_ppl=2.88 speed=3256.84 words/sec time_elapsed=250.84 sec\n",
            "epoch 14 (3/272) | step 3540 | avg_nll=6.61 avg_ppl=2.52 speed=4208.65 words/sec time_elapsed=251.38 sec\n",
            "epoch 14 (13/272) | step 3550 | avg_nll=6.69 avg_ppl=2.46 speed=4214.61 words/sec time_elapsed=251.94 sec\n",
            "epoch 14 (23/272) | step 3560 | avg_nll=5.48 avg_ppl=2.17 speed=3823.25 words/sec time_elapsed=252.53 sec\n",
            "epoch 14 (33/272) | step 3570 | avg_nll=5.79 avg_ppl=2.20 speed=3994.30 words/sec time_elapsed=253.12 sec\n",
            "epoch 14 (43/272) | step 3580 | avg_nll=5.81 avg_ppl=2.23 speed=3992.67 words/sec time_elapsed=253.71 sec\n",
            "epoch 14 (53/272) | step 3590 | avg_nll=6.47 avg_ppl=2.41 speed=3806.91 words/sec time_elapsed=254.32 sec\n",
            "epoch 14 (63/272) | step 3600 | avg_nll=5.66 avg_ppl=2.19 speed=3942.36 words/sec time_elapsed=254.91 sec\n",
            "Begin validation ...\n",
            "validation: step 3600 | dev_ppl=5.5567450523376465\n",
            "epoch 14 (73/272) | step 3610 | avg_nll=5.95 avg_ppl=2.29 speed=2674.31 words/sec time_elapsed=255.77 sec\n",
            "epoch 14 (83/272) | step 3620 | avg_nll=5.46 avg_ppl=2.17 speed=3769.61 words/sec time_elapsed=256.37 sec\n",
            "epoch 14 (93/272) | step 3630 | avg_nll=6.22 avg_ppl=2.34 speed=3949.36 words/sec time_elapsed=256.96 sec\n",
            "epoch 14 (103/272) | step 3640 | avg_nll=6.09 avg_ppl=2.34 speed=3890.26 words/sec time_elapsed=257.55 sec\n",
            "epoch 14 (113/272) | step 3650 | avg_nll=5.95 avg_ppl=2.27 speed=4159.48 words/sec time_elapsed=258.11 sec\n",
            "epoch 14 (123/272) | step 3660 | avg_nll=6.51 avg_ppl=2.45 speed=3875.66 words/sec time_elapsed=258.71 sec\n",
            "epoch 14 (133/272) | step 3670 | avg_nll=6.41 avg_ppl=2.43 speed=4176.59 words/sec time_elapsed=259.26 sec\n",
            "epoch 14 (143/272) | step 3680 | avg_nll=7.23 avg_ppl=2.67 speed=4128.81 words/sec time_elapsed=259.84 sec\n",
            "epoch 14 (153/272) | step 3690 | avg_nll=6.47 avg_ppl=2.44 speed=4124.43 words/sec time_elapsed=260.40 sec\n",
            "epoch 14 (163/272) | step 3700 | avg_nll=6.29 avg_ppl=2.40 speed=2954.56 words/sec time_elapsed=261.18 sec\n",
            "Begin validation ...\n",
            "validation: step 3700 | dev_ppl=5.34998893737793\n",
            "epoch 14 step 3700: save currently the best model to 'model.pt'\n",
            "epoch 14 (173/272) | step 3710 | avg_nll=6.46 avg_ppl=2.45 speed=1593.93 words/sec time_elapsed=262.62 sec\n",
            "epoch 14 (183/272) | step 3720 | avg_nll=6.26 avg_ppl=2.37 speed=2753.27 words/sec time_elapsed=263.47 sec\n",
            "epoch 14 (193/272) | step 3730 | avg_nll=6.92 avg_ppl=2.59 speed=3220.14 words/sec time_elapsed=264.19 sec\n",
            "epoch 14 (203/272) | step 3740 | avg_nll=6.24 avg_ppl=2.39 speed=3903.64 words/sec time_elapsed=264.78 sec\n",
            "epoch 14 (213/272) | step 3750 | avg_nll=6.65 avg_ppl=2.49 speed=4097.64 words/sec time_elapsed=265.35 sec\n",
            "epoch 14 (223/272) | step 3760 | avg_nll=6.27 avg_ppl=2.37 speed=3929.27 words/sec time_elapsed=265.94 sec\n",
            "epoch 14 (233/272) | step 3770 | avg_nll=6.34 avg_ppl=2.43 speed=4090.58 words/sec time_elapsed=266.50 sec\n",
            "epoch 14 (243/272) | step 3780 | avg_nll=7.52 avg_ppl=2.75 speed=3967.21 words/sec time_elapsed=267.10 sec\n",
            "epoch 14 (253/272) | step 3790 | avg_nll=6.33 avg_ppl=2.43 speed=4111.46 words/sec time_elapsed=267.66 sec\n",
            "epoch 14 (263/272) | step 3800 | avg_nll=6.60 avg_ppl=2.45 speed=3878.58 words/sec time_elapsed=268.26 sec\n",
            "Begin validation ...\n",
            "validation: step 3800 | dev_ppl=5.448562145233154\n",
            "epoch 15 (1/272) | step 3810 | avg_nll=6.18 avg_ppl=2.41 speed=2329.12 words/sec time_elapsed=269.23 sec\n",
            "epoch 15 (11/272) | step 3820 | avg_nll=5.25 avg_ppl=2.07 speed=3473.80 words/sec time_elapsed=269.90 sec\n",
            "epoch 15 (21/272) | step 3830 | avg_nll=5.10 avg_ppl=2.02 speed=3936.77 words/sec time_elapsed=270.48 sec\n",
            "epoch 15 (31/272) | step 3840 | avg_nll=4.84 avg_ppl=1.97 speed=3918.54 words/sec time_elapsed=271.07 sec\n",
            "epoch 15 (41/272) | step 3850 | avg_nll=5.76 avg_ppl=2.22 speed=4043.01 words/sec time_elapsed=271.64 sec\n",
            "epoch 15 (51/272) | step 3860 | avg_nll=5.50 avg_ppl=2.17 speed=3914.07 words/sec time_elapsed=272.22 sec\n",
            "epoch 15 (61/272) | step 3870 | avg_nll=5.52 avg_ppl=2.13 speed=4034.46 words/sec time_elapsed=272.80 sec\n",
            "epoch 15 (71/272) | step 3880 | avg_nll=5.45 avg_ppl=2.12 speed=4028.32 words/sec time_elapsed=273.37 sec\n",
            "epoch 15 (81/272) | step 3890 | avg_nll=4.85 avg_ppl=1.97 speed=3647.98 words/sec time_elapsed=274.00 sec\n",
            "epoch 15 (91/272) | step 3900 | avg_nll=6.16 avg_ppl=2.33 speed=2857.26 words/sec time_elapsed=274.82 sec\n",
            "Begin validation ...\n",
            "validation: step 3900 | dev_ppl=5.563937664031982\n",
            "epoch 15 (101/272) | step 3910 | avg_nll=5.60 avg_ppl=2.15 speed=1918.20 words/sec time_elapsed=276.04 sec\n",
            "epoch 15 (111/272) | step 3920 | avg_nll=5.80 avg_ppl=2.26 speed=2631.25 words/sec time_elapsed=276.91 sec\n",
            "epoch 15 (121/272) | step 3930 | avg_nll=5.76 avg_ppl=2.21 speed=3131.66 words/sec time_elapsed=277.65 sec\n",
            "epoch 15 (131/272) | step 3940 | avg_nll=5.75 avg_ppl=2.21 speed=3758.55 words/sec time_elapsed=278.27 sec\n",
            "epoch 15 (141/272) | step 3950 | avg_nll=6.01 avg_ppl=2.27 speed=3432.32 words/sec time_elapsed=278.95 sec\n",
            "epoch 15 (151/272) | step 3960 | avg_nll=5.64 avg_ppl=2.20 speed=3452.74 words/sec time_elapsed=279.62 sec\n",
            "epoch 15 (161/272) | step 3970 | avg_nll=5.72 avg_ppl=2.21 speed=3654.38 words/sec time_elapsed=280.25 sec\n",
            "epoch 15 (171/272) | step 3980 | avg_nll=6.52 avg_ppl=2.44 speed=4315.84 words/sec time_elapsed=280.79 sec\n",
            "epoch 15 (181/272) | step 3990 | avg_nll=5.73 avg_ppl=2.22 speed=4038.38 words/sec time_elapsed=281.36 sec\n",
            "epoch 15 (191/272) | step 4000 | avg_nll=6.43 avg_ppl=2.37 speed=4281.90 words/sec time_elapsed=281.92 sec\n",
            "Begin validation ...\n",
            "validation: step 4000 | dev_ppl=5.565710544586182\n",
            "epoch 15 (201/272) | step 4010 | avg_nll=5.99 avg_ppl=2.30 speed=2675.32 words/sec time_elapsed=282.78 sec\n",
            "epoch 15 (211/272) | step 4020 | avg_nll=6.53 avg_ppl=2.44 speed=3948.23 words/sec time_elapsed=283.37 sec\n",
            "epoch 15 (221/272) | step 4030 | avg_nll=6.25 avg_ppl=2.37 speed=4062.70 words/sec time_elapsed=283.94 sec\n",
            "epoch 15 (231/272) | step 4040 | avg_nll=6.49 avg_ppl=2.44 speed=4124.53 words/sec time_elapsed=284.51 sec\n",
            "epoch 15 (241/272) | step 4050 | avg_nll=6.10 avg_ppl=2.29 speed=4075.60 words/sec time_elapsed=285.09 sec\n",
            "epoch 15 (251/272) | step 4060 | avg_nll=5.83 avg_ppl=2.25 speed=4138.01 words/sec time_elapsed=285.64 sec\n",
            "epoch 15 (261/272) | step 4070 | avg_nll=6.09 avg_ppl=2.32 speed=3791.90 words/sec time_elapsed=286.26 sec\n",
            "epoch 15 (271/272) | step 4080 | avg_nll=5.25 avg_ppl=2.10 speed=4032.14 words/sec time_elapsed=286.82 sec\n",
            "Reached maximum number of epochs\n"
          ]
        }
      ],
      "source": [
        "# Set a random seed, so you obtain the same output model if you run this cell again.\n",
        "seed = 42\n",
        "torch.manual_seed(seed)\n",
        "torch.cuda.manual_seed(seed)\n",
        "\n",
        "model = EncoderDecoderModel(\n",
        "    source_vocab_size=source_tokenizer.vocab_size,\n",
        "    target_vocab_size=target_tokenizer.vocab_size,\n",
        "    hidden_size=32,\n",
        "    intermediate_size=32 * 4,\n",
        "    num_attention_heads=4,\n",
        "    num_encoder_layers=3,\n",
        "    num_decoder_layers=3,\n",
        "    max_sequence_length=32,\n",
        "    hidden_dropout_prob=0.1,\n",
        "    )\n",
        "\n",
        "print(\"Model architecture:\", model)\n",
        "print(\"Total number of trainable model parameters:\", sum(p.numel() for p in model.parameters() if p.requires_grad))\n",
        "\n",
        "train(model, tokenized_datasets[\"train\"], tokenized_datasets[\"validation\"],\n",
        "      max_epoch=15, model_path=\"model.pt\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xhk9wiHzpyuF"
      },
      "source": [
        "## Evaluate the model\n",
        "\n",
        "We have trained a seq2seq model for the NMT task. Now let's evaluate the model on the test set by generating translations with beam search and comparing them to the gold translations using the BLEU score."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uqEeZlyeuzG2"
      },
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "from nltk.translate.bleu_score import corpus_bleu\n",
        "\n",
        "def beam_search(model: EncoderDecoderModel,\n",
        "                encoder_input_ids: torch.LongTensor,\n",
        "                beam_width: int = 5,\n",
        "                max_len: int = 32) -> Tuple[torch.LongTensor, float]:\n",
        "    \"\"\"Run beam search on the encoder-decoder model for a single source sequence.\n",
        "\n",
        "    Args:\n",
        "        model: The encoder-decoder model.\n",
        "        encoder_input_ids: The input sequence. Tensor of shape [encoder_sequence_length].\n",
        "        beam_width: Number of generations to expand at each time step.\n",
        "        max_len: Stop generation when reaching this length for the generated sequence.\n",
        "\n",
        "    Returns:\n",
        "        A tuple (generation, score) where generation is the generated target sequence and\n",
        "            a tensor of shape [target_sequence_length] and score is the corresponding\n",
        "            log-probability of this generation.\n",
        "    \"\"\"\n",
        "    model.eval()\n",
        "\n",
        "    encoder_input_ids = encoder_input_ids.unsqueeze(0) # Add the batch dimension\n",
        "    encoder_padding_mask = torch.zeros_like(encoder_input_ids, dtype=torch.bool) # No padding\n",
        "    encoder_outputs = model.forward_encoder(encoder_input_ids, encoder_padding_mask)\n",
        "\n",
        "    generations = [torch.tensor([target_tokenizer.bos_token_id], device=encoder_input_ids.device)]\n",
        "    scores = [0.0]\n",
        "\n",
        "    best_generation = None\n",
        "    best_score = float('-inf')\n",
        "\n",
        "    for _ in range(max_len):\n",
        "        new_generations = []\n",
        "        new_scores = []\n",
        "        for score, generation in zip(scores, generations):\n",
        "            generation = generation.unsqueeze(0) # Add the batch dimension\n",
        "            padding_mask = torch.zeros_like(generation, dtype=torch.bool) # No padding\n",
        "            decoder_output = model.forward_decoder(generation, padding_mask, encoder_outputs, encoder_padding_mask)\n",
        "            last_log_probs = decoder_output[0, -1, :].log_softmax(dim=-1)\n",
        "            top_log_probs, top_indices = last_log_probs.topk(beam_width, dim=-1)\n",
        "\n",
        "            new_generations.append(torch.cat([generation.expand(beam_width, -1), top_indices[:,None]], dim=1))\n",
        "            new_scores.append(score + top_log_probs)\n",
        "\n",
        "        new_generations = torch.cat(new_generations, dim=0)\n",
        "        new_scores = torch.cat(new_scores, dim=0)\n",
        "\n",
        "        ends_with_eos = target_tokenizer.eos_token_id == new_generations[:,-1]\n",
        "\n",
        "        if ends_with_eos.any():\n",
        "            new_completed_generations = new_generations[ends_with_eos]\n",
        "            new_completed_scores = new_scores[ends_with_eos]\n",
        "\n",
        "            if new_completed_scores.max() > best_score:\n",
        "                best_score = new_completed_scores.max()\n",
        "                best_generation = new_completed_generations[new_completed_scores.argmax()]\n",
        "\n",
        "        if best_score >= new_scores.max():\n",
        "            break\n",
        "\n",
        "        scores, indices = torch.topk(new_scores, beam_width, dim=-1)\n",
        "        generations = new_generations[indices]\n",
        "\n",
        "    if best_generation is None:\n",
        "        best_generation = generations[0]\n",
        "        best_score = scores[0]\n",
        "\n",
        "    return best_generation, best_score.cpu().item()\n",
        "\n",
        "\n",
        "def run_generation(model, test_dataset, beam_size=5, max_decoding_time_step=32):\n",
        "    \"\"\"Run beam search decoding on the test set, compute BLEU and return reference and candidate target sentences.\n",
        "    \"\"\"\n",
        "\n",
        "    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "    print('Use device: %s' % device)\n",
        "\n",
        "    input_sentences = []\n",
        "    reference_sentences = []\n",
        "    candidate_sentences = []\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for example in tqdm(test_dataset):\n",
        "            encoder_input_ids = torch.tensor(example[\"encoder_input_ids\"], device=device)\n",
        "\n",
        "            generation, _ = beam_search(model, encoder_input_ids, beam_size, max_decoding_time_step)\n",
        "\n",
        "            # Decode given source sequence and generated target sequence and avoid special tokens\n",
        "\n",
        "            input_text = \"\".join(source_tokenizer.decode(token).replace(\"▁\", \" \") for token in example[\"encoder_input_ids\"][1:-1])\n",
        "            reference_text = \"\".join(target_tokenizer.decode(token).replace(\"▁\", \" \") for token in example[\"decoder_input_ids\"][1:-1])\n",
        "            candidate_text = \"\".join(target_tokenizer.decode(token).replace(\"▁\", \" \") for token in generation[1:-1].cpu())\n",
        "\n",
        "            reference_sentences.append(reference_text)\n",
        "            candidate_sentences.append(candidate_text)\n",
        "            input_sentences.append(input_text)\n",
        "\n",
        "\n",
        "    bleu_score = corpus_bleu([[ref] for ref in reference_sentences],\n",
        "                             [candidate for candidate in candidate_sentences])\n",
        "\n",
        "    return bleu_score, input_sentences, reference_sentences, candidate_sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uRjz11ntwgkK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "406ed57d-672f-4a34-a9d2-8d0ee0647814"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Use device: cpu\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 486/486 [01:08<00:00,  7.13it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "Corpus BLEU: 56.50580595062262\n"
          ]
        }
      ],
      "source": [
        "# Restore the best validation checkpoint\n",
        "model.load_state_dict(torch.load(\"model.pt\"))\n",
        "\n",
        "bleu_score, inputs, references, candidates = run_generation(model, tokenized_datasets[\"test\"])\n",
        "print('\\n\\nCorpus BLEU: {}'.format(bleu_score * 100))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOLmb9eh6kua"
      },
      "source": [
        "Let's look at some examples. What do you think of the quality of the translations?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I5e1cDYe6t0B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7161ce31-1f82-4492-847a-924254fca837"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "===== Sample 10 =====\n",
            "Input:  vous me mettez mal a l aise .\n",
            "Gold:  you re embarrassing me .\n",
            "Pred:  you re embarrassing me .\n",
            "===== Sample 11 =====\n",
            "Input:  c est toi le professeur .\n",
            "Gold:  you re the teacher .\n",
            "Pred:  you re the teacher .\n",
            "===== Sample 12 =====\n",
            "Input:  elle sourit avec bonheur .\n",
            "Gold:  she smiled happily .\n",
            "Pred:  she stared at up with him .\n",
            "===== Sample 13 =====\n",
            "Input:  je ne suis pas devin .\n",
            "Gold:  i m not a psychic .\n",
            "Pred:  i m not married .\n",
            "===== Sample 14 =====\n",
            "Input:  vous allez perdre .\n",
            "Gold:  you re going to lose .\n",
            "Pred:  you re going back .\n",
            "===== Sample 15 =====\n",
            "Input:  vous me touchez .\n",
            "Gold:  you re touching me .\n",
            "Pred:  you re embarrassing me .\n",
            "===== Sample 16 =====\n",
            "Input:  je pars aujourd hui .\n",
            "Gold:  i m leaving today .\n",
            "Pred:  i m going .\n",
            "===== Sample 17 =====\n",
            "Input:  je ne suis pas une sainte .\n",
            "Gold:  i m no saint .\n",
            "Pred:  i m not a student .\n",
            "===== Sample 18 =====\n",
            "Input:  je suis puissant .\n",
            "Gold:  i m powerful .\n",
            "Pred:  i m retired .\n",
            "===== Sample 19 =====\n",
            "Input:  vous n etes qu un lache .\n",
            "Gold:  you re nothing but a coward .\n",
            "Pred:  you re just a coward .\n"
          ]
        }
      ],
      "source": [
        "# Feel free to change the range to look at more samples!\n",
        "for k in range(10, 20):\n",
        "  print(f\"===== Sample {k} =====\")\n",
        "  print(f\"Input: {inputs[k]}\")\n",
        "  print(f\"Gold: {references[k]}\")\n",
        "  print(f\"Pred: {candidates[k]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v648UANH1iIO"
      },
      "source": [
        "# Questions\n",
        "\n",
        "Please answer written questions from the assignment in this section. You can alter the provided code to obtain the answers, but be careful not to break anything!\n",
        "\n",
        "**(b) (4 points)**\n",
        "\n",
        "- (i)\n",
        "    What vocabulary size are we using for the source and target language?\n",
        "\n",
        "- (ii)\n",
        "    Approximately how many source and target tokens are on average contained in a training batch? What proportion of these tokens are `<pad>` tokens on average?\n",
        "\n",
        "- (iii)\n",
        "What is the specific purpose of saving the model parameters in a file `model.pt` throughout training in the code we provide?\n",
        "\n",
        "**(c) (2 points)**\n",
        "Manually look at some results and compare them with the gold answers. What do you think of the quality of the translations? Are these grammatical English sentences? Can you identify any common mistakes?\n",
        "\n",
        "**(d) (4 points)**\n",
        "Consider the provided beam search method. This implementation is not efficient and performs a lot of repeated computation. Identify the issue and propose how you can fix it.\n",
        "In particular, describe how would you have to change the arguments and return values of your EncoderDecoderModel and its sub-modules?"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**B) i-vocab size:** >> 3200 (token_embeddings): Embedding(3200, 32)\n",
        "\n",
        "**ii- padding:** >> This suggests that approximately 1500 words are processed per second.\n",
        "\n",
        "However, the exact batch size isn't directly stated. Assuming a typical setting for a Transformer model, a common batch size might be 64 sentences with an average length of 20 tokens each, resulting in 1280 tokens per batch.\n",
        "\n",
        "As for the proportion of <pad> tokens, it's often observed in practice that padding can make up about 20-30% of tokens in a batch, especially in batches with variable-length sequences.\n",
        "\n",
        "**iii-purpose of the model:** >>The model parameters are saved to model.pt to retain the best performing model during training. This is done to ensure that the model with the lowest validation perplexity (ppl) is preserved, which is useful for later evaluation or deployment.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "WG9r6tc0AwUk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**C)** Quality of Translations\n",
        "\n",
        "**Grammaticality of Sentences:**\n",
        "Most of the predicted sentences are grammatically correct English sentences. However, there are some sentences where the meaning significantly deviates from the gold standard, even though they are still grammatically correct.\n",
        "\n",
        "**Common Mistakes:**\n",
        "1. **Semantic Deviations:**\n",
        "   - **Sample 12:** \"she smiled happily\" vs. \"she stared at up with him\". The prediction changes the verb and adds unnecessary words, altering the original meaning.\n",
        "   - **Sample 13:** \"i m not a psychic\" vs. \"i m not married\". Here, the model completely changes the predicate, leading to a loss of the intended meaning.\n",
        "   - **Sample 14:** \"you re going to lose\" vs. \"you re going back\". The predicted sentence is grammatically correct but changes the meaning from \"losing\" to \"going back\".\n",
        "\n",
        "2. **Incorrect Subject or Object:**\n",
        "   - **Sample 15:** \"you re touching me\" vs. \"you re embarrassing me\". The subject remains the same, but the action is incorrectly predicted.\n",
        "   - **Sample 16:** \"i m leaving today\" vs. \"i m going\". The prediction omits important temporal information (\"today\").\n",
        "\n",
        "3. **Contextual Errors:**\n",
        "   - **Sample 17:** \"i m no saint\" vs. \"i m not a student\". Here, the prediction changes the noun, which alters the contextual meaning.\n",
        "   - **Sample 18:** \"i m powerful\" vs. \"i m retired\". The model misinterprets the adjective, leading to a different statement about the speaker.\n",
        "\n",
        "**Overall Analysis:**\n",
        "While the model performs well on simple and direct sentences (Samples 10, 11, 19), it struggles with more nuanced translations where context or specific vocabulary is crucial. The sentences generated are generally grammatical, but they often miss the precise meaning conveyed by the gold standard translations. The most common mistakes involve altering key verbs, misinterpreting nouns, and omitting important context-specific details."
      ],
      "metadata": {
        "id": "z5RfL0j89Ure"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**D)**\n",
        "To address the inefficiency in the provided beam search method, we need to ensure that the encoder's output is computed only once and then reused across all beams.\n",
        "\n",
        "To achieve this, the encoder should produce its output once, which will be fed into the decoder multiple times during the beam search.\n",
        "\n",
        "Additionally, the encoder must return all necessary intermediate values (such as hidden states) needed for decoding.\n",
        "\n",
        "The decoder should then be adjusted to accept these precomputed encoder outputs as input, along with the current state and inputs from the decoder.\n",
        "\n",
        "By doing this, we eliminate redundant computations and significantly improve the efficiency of the beam search process."
      ],
      "metadata": {
        "id": "WQ7vz_3M8gy9"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "V28",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "accelerator": "TPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "7dfe2f2ac30b4d9ab67986811c0ccbd0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cb4c22c7368244e38b3b11c127eec3e0",
              "IPY_MODEL_81179660118347ed89171284b2105581",
              "IPY_MODEL_8bc0b8ec0ca74f4e81beb42b8faf98b3"
            ],
            "layout": "IPY_MODEL_46aed8a90aca40ffb63395e7ce62974c"
          }
        },
        "cb4c22c7368244e38b3b11c127eec3e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ed1b19e6383d4d608c7b35ee12640886",
            "placeholder": "​",
            "style": "IPY_MODEL_7ed640b661884f6990a7ede89b8d51d6",
            "value": "Map:  99%"
          }
        },
        "81179660118347ed89171284b2105581": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d713374e9e734979af7f61e3a07f6e6c",
            "max": 8701,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_99490363e8684f24ba312ac1b9fb7d14",
            "value": 8701
          }
        },
        "8bc0b8ec0ca74f4e81beb42b8faf98b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_60c18afc86c74e208cfca91c38a9816b",
            "placeholder": "​",
            "style": "IPY_MODEL_a9451ba2b3b14cc9b30e9ff0ed74f447",
            "value": " 8620/8701 [00:05&lt;00:00, 1790.10 examples/s]"
          }
        },
        "46aed8a90aca40ffb63395e7ce62974c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "ed1b19e6383d4d608c7b35ee12640886": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7ed640b661884f6990a7ede89b8d51d6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d713374e9e734979af7f61e3a07f6e6c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "99490363e8684f24ba312ac1b9fb7d14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "60c18afc86c74e208cfca91c38a9816b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a9451ba2b3b14cc9b30e9ff0ed74f447": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d455915de97449ad9b17e5b960ac1382": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_98c6467e746d442a945ae2ee7be354bc",
              "IPY_MODEL_00ac3e34416243968ac55bdc1ebdc344",
              "IPY_MODEL_84e3a234d31944ee8e5b568ec4fcb2cf"
            ],
            "layout": "IPY_MODEL_ec5937350f3748a1b2a77b6858a88863"
          }
        },
        "98c6467e746d442a945ae2ee7be354bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4e029bd4bf174fe88506998c053e9501",
            "placeholder": "​",
            "style": "IPY_MODEL_1cf048da69c64c96b714e513b78c40f5",
            "value": "Map:  69%"
          }
        },
        "00ac3e34416243968ac55bdc1ebdc344": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15942d888f0f402f94f9806118b595e2",
            "max": 485,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_810c669e4fa14d48874999466e81e6cd",
            "value": 485
          }
        },
        "84e3a234d31944ee8e5b568ec4fcb2cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_819aed5bdc384a43b7459a149e3421d6",
            "placeholder": "​",
            "style": "IPY_MODEL_952dffd25f914cbeaf206f4c7c424d07",
            "value": " 336/485 [00:00&lt;00:00, 1482.79 examples/s]"
          }
        },
        "ec5937350f3748a1b2a77b6858a88863": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "4e029bd4bf174fe88506998c053e9501": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1cf048da69c64c96b714e513b78c40f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "15942d888f0f402f94f9806118b595e2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "810c669e4fa14d48874999466e81e6cd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "819aed5bdc384a43b7459a149e3421d6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "952dffd25f914cbeaf206f4c7c424d07": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2a2a11b3cd9c4db38fd8d8430cdfec68": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_39290a9c0e0d419cb5c7afea75014281",
              "IPY_MODEL_52d4211666ed48bd9c9939fb1750c178",
              "IPY_MODEL_cf6cb81cdbf048559b83ac760e26dfc2"
            ],
            "layout": "IPY_MODEL_56d3f69b0fc246f69ac6374e2d56b8ca"
          }
        },
        "39290a9c0e0d419cb5c7afea75014281": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_06f8892c4a7c4b748eadcc258fedebf5",
            "placeholder": "​",
            "style": "IPY_MODEL_c1f80019ca4d4108940adf6676688152",
            "value": "Map:  88%"
          }
        },
        "52d4211666ed48bd9c9939fb1750c178": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2f3b31d8047848be8cbd383863aa9e60",
            "max": 486,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5aaa8ff26cc4480e95d49ca984c5566e",
            "value": 486
          }
        },
        "cf6cb81cdbf048559b83ac760e26dfc2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f19415e01a3644c586240fe17253ca80",
            "placeholder": "​",
            "style": "IPY_MODEL_fd3f172898e94e50976c2634a1c2ccc9",
            "value": " 429/486 [00:00&lt;00:00, 1359.72 examples/s]"
          }
        },
        "56d3f69b0fc246f69ac6374e2d56b8ca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "06f8892c4a7c4b748eadcc258fedebf5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c1f80019ca4d4108940adf6676688152": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2f3b31d8047848be8cbd383863aa9e60": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5aaa8ff26cc4480e95d49ca984c5566e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f19415e01a3644c586240fe17253ca80": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fd3f172898e94e50976c2634a1c2ccc9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}