01-your-first-neural-network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: left; margin-bottom: 1em\" src=\"images/PRACE.png\" width=\"200\">\n",
    "<img style=\"float: right; margin-bottom: 1em\" src=\"images/surfsara.png\" width=\"150\">\n",
    "<hr style=\"clear: both\"/>\n",
    "\n",
    "# Your first neural network\n",
    "In this notebook you will build a small neural network to solve the classification problem introduced in the slides. You will see a number of terms appear that we have not yet discussed. We will introduce and explain these terms gradually during the day.\n",
    "\n",
    "We will use Python and the deep learning library [Keras](https://keras.io/) for building our network. We do not expect you to write much code today, so do not worry if you do not know Python.\n",
    "\n",
    "## Interacting with the notebook\n",
    "This notebook consists of a number of cells. Each cell can be selected by clicking on it, and can be executed by clicking on the 'Run' icon on the top of the page, or by pressing **Shift + Enter**. Try it with the cell directly below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('This is a Python cell')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the data\n",
    "Run the following cell once to download and extract the data. This can take a minute or two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!bash download.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data\n",
    "In the next cell, we load a subset of the [original data set](https://www.kaggle.com/emmarex/plantdisease) on plant disease classification. We have made the images smaller in order to keep the training time of the network manageable.\n",
    "\n",
    "**Run the next cell**. A star symbol (\\*) to the left of the cell indicates Python is executing the cell. A number will appear when it is finished. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "import lib\n",
    "\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "X, Y, labels = lib.dataset_plant_village_tomato_blight()\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot some of the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "lib.plot_examples(X[:16], Y[:16], labels);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Healthy examples will have 'healthy' in the name, diseased examples the name of the disease. Some of the diseases will have an obvious effect. Others less so.\n",
    "\n",
    "## Building a network\n",
    "Let's build our first network! It will be a very simple network that will map the input images directly to the output classes.\n",
    "\n",
    "We can define the network in Keras in a few lines of Python code. The next cell will build the network, or model, and show a summary of the network's layers and parameters. We will explain a few things about this network in the cell below.\n",
    "\n",
    "**Run the next cell.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have built a **sequential model** where data flows through the model from left to right: from input to output. The first layer is a `Flatten` layer, which will convert our 2D data into 1D data. This is a prerequisite for working with dense layers.\n",
    "\n",
    "The second layer is a `Dense` layer with two elements. The first element will correspond to the 'diseased' class, the second to the 'normal' class.\n",
    "\n",
    "Finally, we have an `Activation` layer with **softmax** activation. This layer applies a particular non-linear transformation to its inputs (the outputs of the dense layer), and makes sure that the output can be interpreted as a probability distribution.\n",
    "\n",
    "## Training the network\n",
    "In the next cell we **compile** the model with an optimizer. This optimizer is responsible for **training** the network. We will use an optimizer called Adam, which is quite robust and performant.\n",
    "\n",
    "After compiling the model, we train the model for a number of **epochs**. Each **epoch** the entire data set is sent through the network in a number of **batches**. After each batch, Keras updates the network to improve its performance.\n",
    "\n",
    "**Run the next cell, and read the cell below while it is training.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.optimizers import Adam\n",
    "\n",
    "model.compile(Adam(lr=0.0001), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=25, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss and accuracy\n",
    "While the network is training Keras will display some information about the progress. It will show the **loss** of the network and the **accuracy**.\n",
    "\n",
    "The loss is the error of the network: after each batch of data Keras will calculate 'how wrong' the network prediction is compared to the actual label. By modifying the parameters of the network Keras will try to minimize the loss.\n",
    "\n",
    "Since the loss is not easily interpretable we typically have a domain-specific or problem-specific **metric** to tell us how the current network performs. Since this is a classification problem, we can look at the **accuracy**. The accuracy is simply the proportion of percentage of images that are classified correctly. When it is 1 (or 100%), the network has perfectly classified all instances in the data set.\n",
    "\n",
    "### Training time\n",
    "Training a network can be quite time-consuming. We have defined a very simple network with a small data set. More complex problems with more data and more complex or larger images or data items, can take hours or days to train, even on a GPU.\n",
    "\n",
    "## Evaluating the network\n",
    "Keras has provided us with some updates on the network loss and accuracy during training, so we already have some sense of how our network performs.\n",
    "\n",
    "Let's plot the loss and accuracy during our training process and get a final estimate of our performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lib.plot_history(history)\n",
    "\n",
    "loss, accuracy = model.evaluate(X, Y, verbose=True)\n",
    "print('Final loss of model:     {:.4f}'.format(loss))\n",
    "print('Final accuracy of model: {:.4f} (error rate: {:.2f}%)'.format(accuracy, (1 - accuracy) * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is always a small degree of randomness involved in all deep learning computations. As a consequence, your resulting loss and accuracy may vary. For this problem, our test runs were around 0.95 accuracy.\n",
    "\n",
    "## Inspecting the network internals\n",
    "Although the network performs quite well, we can improve its performance even more. Before we do that, it may help us to know how the network comes to a classification. One way of doing this is by generating an image that would give us a maximal **activation** with respect to an output class. Remember: neural networks build feature detectors that activate on a given input.\n",
    "\n",
    "With some tricks from the [keras-vis](https://github.com/raghakot/keras-vis) package we can generate an image for the diseased and healthy class, each of which will maximally activate for that image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "lib.plot_activations_twoclass(model, labels);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reiterate: on the left we have the ideal input image for the network to maximally predict the diseased class. On the right the ideal input image to predict a healthy leaf.\n",
    "\n",
    "Although the network performs quite well, it is still a very simple network directly mapping the input to the output without any hidden layers. As such, these images look a bit strange. We can still see the outline of a leaf in both cases, though.\n",
    "\n",
    "Also, we can figure out that if the network detects some pink area around the edges of the leaf, the network will classify the leaf as diseased. Since pink is made up of red and yellow, this makes quite a lot of sense given the data set. Conversely, if the edges of the leaf are green (or turquoise, a mixture of green and blue), the network will predict the leaf as healthy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: improving network performance\n",
    "We have built a very primitive network. In practice, we will build networks with many hidden layers instead of directly mapping input to output. In the cell below we have added a hidden layer to the original network. It is still quite small, with only a few units in the hidden layer. Still, it will improve our performance quite a bit.\n",
    "\n",
    "**Improve the accuracy of the model to .97 or higher by increasing the number of units in the hidden layer.**\n",
    "\n",
    "**Note: try to keep the number of units in the hidden layer smaller than 256, to keep training time manageable.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "model.add(Dense(<FILL IN>, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=0.0001), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=25, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "When you are happy with the result, run the following cell to obtain the final accuracy of the model and the activation maps.\n",
    "\n",
    "**What is the difference in performance, and why? Do you see any changes in the activation maps? Write down your answer in the cell at the bottom of the notebook.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, accuracy = model.evaluate(X, Y, verbose=True)\n",
    "print('Final loss of model:     {:.4f}'.format(loss))\n",
    "print('Final accuracy of model: {:.4f} (error rate: {:.2f}%)'.format(accuracy, (1 - accuracy) * 100))\n",
    "\n",
    "lib.plot_activations_twoclass(model, labels);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}