Improving Network Intrusion Detection using a Denoising Autoencoder with Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<em>This notebook contains an excerpt from a study prepared by Radwan Diab, Mahmoud Aslan and Eiad Soufan; Supervised by Suhel Al-Hammoud PhD. </em>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving Network Intrusion Detection using a Denoising Autoencoder with Dropout\n",
    "\n",
    "This notebook presents an autoencoder based anomaly detection model for intrusion detection, we use the NSL-KDD dataset, this dataset is a benchmark for machine learning based intrusion detection, however , it suffers from several inefficiencies such as class imbalance, where for instance in the NSL-KDD training dataset only 0.04% of the samples belong to the u2r attack type making it severely underrepresented, the case is similar for the r2l and probe attack types whereas the majority of attack records are representing the DDOS attack type, this fact made it difficult for classifiers to detect these underrepresented types resulting in poor accuracy. Another issue is that this dataset is unrealistic, in reality most traffic in a network is benign and only a small percentage might be malicious, while in the NSL-KDD training set for example, attack samples compose 80% of the entire dataset which makes the models trained using this dataset ineffective in real life situations. Out autoencoder based approach attempts to overcome these problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies \n",
    "\n",
    "The notebook uses the following libraries :\n",
    "- NumPy\n",
    "- Pandas\n",
    "- matplotlib\n",
    "- Keras\n",
    "- scikit-learn\n",
    "- TensorFlow or Theano\n",
    "\n",
    "The following section makes the necessary imports :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import tensorflow as tf\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "import itertools\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.metrics import confusion_matrix,accuracy_score,recall_score,precision_score,f1_score\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from keras.layers import Input,Dropout,Dense\n",
    "from keras.models import Model\n",
    "from keras import regularizers\n",
    "from keras.utils.data_utils import get_file\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading the data\n",
    "The NSL-KDD dataset is available in a github __[repository](https://github.com/defcom17/NSL_KDD)__, we will be using the following files\n",
    "- __[KDDTrain+.csv](https://github.com/defcom17/NSL_KDD/blob/master/KDDTrain%2B.csv)__ for training the model.\n",
    "- __[KDDTest+.csv](https://github.com/defcom17/NSL_KDD/blob/master/KDDTest%2B.csv)__ for testing.\n",
    "\n",
    "We will use the utility function `get_file` from keras to download the files and we'll read them into a pandas dataframe afterward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# Downloading training and test sets to local drive\n",
    "try:\n",
    "    training_set_path = get_file('KDDTrain%2B.csv', origin='https://raw.githubusercontent.com/defcom17/NSL_KDD/master/KDDTrain%2B.csv')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "\n",
    "try:\n",
    "    test_set_path = get_file('KDDTest%2B.csv', origin='https://raw.githubusercontent.com/defcom17/NSL_KDD/master/KDDTest%2B.csv')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "training_df = pd.read_csv(training_set_path, header=None)\n",
    "testing_df = pd.read_csv(test_set_path, header=None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll take a look at the dataframes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "      <td>491</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>udp</td>\n",
       "      <td>other</td>\n",
       "      <td>SF</td>\n",
       "      <td>146</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>S0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>neptune</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>232</td>\n",
       "      <td>8153</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "      <td>normal</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>199</td>\n",
       "      <td>420</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0    1         2   3    4     5   6   7   8   9  ...    33    34    35  \\\n",
       "0   0  tcp  ftp_data  SF  491     0   0   0   0   0 ...  0.17  0.03  0.17   \n",
       "1   0  udp     other  SF  146     0   0   0   0   0 ...  0.00  0.60  0.88   \n",
       "2   0  tcp   private  S0    0     0   0   0   0   0 ...  0.10  0.05  0.00   \n",
       "3   0  tcp      http  SF  232  8153   0   0   0   0 ...  1.00  0.00  0.03   \n",
       "4   0  tcp      http  SF  199   420   0   0   0   0 ...  1.00  0.00  0.00   \n",
       "\n",
       "     36    37    38    39    40       41  42  \n",
       "0  0.00  0.00  0.00  0.05  0.00   normal  20  \n",
       "1  0.00  0.00  0.00  0.00  0.00   normal  15  \n",
       "2  0.00  1.00  1.00  0.00  0.00  neptune  19  \n",
       "3  0.04  0.03  0.01  0.00  0.01   normal  21  \n",
       "4  0.00  0.00  0.00  0.00  0.00   normal  21  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>REJ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>neptune</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>private</td>\n",
       "      <td>REJ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>neptune</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>tcp</td>\n",
       "      <td>ftp_data</td>\n",
       "      <td>SF</td>\n",
       "      <td>12983</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>normal</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>icmp</td>\n",
       "      <td>eco_i</td>\n",
       "      <td>SF</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>saint</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>tcp</td>\n",
       "      <td>telnet</td>\n",
       "      <td>RSTO</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.31</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.71</td>\n",
       "      <td>mscan</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0     1         2     3      4   5   6   7   8   9  ...    33    34    35  \\\n",
       "0   0   tcp   private   REJ      0   0   0   0   0   0 ...  0.04  0.06  0.00   \n",
       "1   0   tcp   private   REJ      0   0   0   0   0   0 ...  0.00  0.06  0.00   \n",
       "2   2   tcp  ftp_data    SF  12983   0   0   0   0   0 ...  0.61  0.04  0.61   \n",
       "3   0  icmp     eco_i    SF     20   0   0   0   0   0 ...  1.00  0.00  1.00   \n",
       "4   1   tcp    telnet  RSTO      0  15   0   0   0   0 ...  0.31  0.17  0.03   \n",
       "\n",
       "     36   37   38    39    40       41  42  \n",
       "0  0.00  0.0  0.0  1.00  1.00  neptune  21  \n",
       "1  0.00  0.0  0.0  1.00  1.00  neptune  21  \n",
       "2  0.02  0.0  0.0  0.00  0.00   normal  21  \n",
       "3  0.28  0.0  0.0  0.00  0.00    saint  15  \n",
       "4  0.02  0.0  0.0  0.83  0.71    mscan  11  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testing_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the CSV files don't contain a header we'll need to assign column names ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    " columns = [\n",
    "    'duration',\n",
    "    'protocol_type',\n",
    "    'service',\n",
    "    'flag',\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'land',\n",
    "    'wrong_fragment',\n",
    "    'urgent',\n",
    "    'hot',\n",
    "    'num_failed_logins',\n",
    "    'logged_in',\n",
    "    'num_compromised',\n",
    "    'root_shell',\n",
    "    'su_attempted',\n",
    "    'num_root',\n",
    "    'num_file_creations',\n",
    "    'num_shells',\n",
    "    'num_access_files',\n",
    "    'num_outbound_cmds',\n",
    "    'is_host_login',\n",
    "    'is_guest_login',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'serror_rate',\n",
    "    'srv_serror_rate',\n",
    "    'rerror_rate',\n",
    "    'srv_rerror_rate',\n",
    "    'same_srv_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_count',\n",
    "    'dst_host_srv_count',\n",
    "    'dst_host_same_srv_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate',\n",
    "    'dst_host_serror_rate',\n",
    "    'dst_host_srv_serror_rate',\n",
    "    'dst_host_rerror_rate',\n",
    "    'dst_host_srv_rerror_rate',\n",
    "    'outcome',\n",
    "    'difficulty'\n",
    "]\n",
    "training_df.columns = columns\n",
    "testing_df.columns = columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've assigned names to the columns let's take a deeper look into the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 125973 rows.\n",
      "Testing set has 22543 rows.\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set has {} rows.\".format(len(training_df)))\n",
    "print(\"Testing set has {} rows.\".format(len(testing_df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training set has 23 possible outcomes \n",
      "\n",
      "normal, neptune, warezclient, ipsweep, portsweep, teardrop, nmap, satan, smurf, pod, back, guess_passwd, ftp_write, multihop, rootkit, buffer_overflow, imap, warezmaster, phf, land, loadmodule, spy, perl.\n",
      "\n",
      "The testing set has 38 possible outcomes \n",
      "\n",
      "neptune, normal, saint, mscan, guess_passwd, smurf, apache2, satan, buffer_overflow, back, warezmaster, snmpgetattack, processtable, pod, httptunnel, nmap, ps, snmpguess, ipsweep, mailbomb, portsweep, multihop, named, sendmail, loadmodule, xterm, worm, teardrop, rootkit, xlock, perl, land, xsnoop, sqlattack, ftp_write, imap, udpstorm, phf.\n"
     ]
    }
   ],
   "source": [
    "training_outcomes=training_df[\"outcome\"].unique()\n",
    "testing_outcomes=testing_df[\"outcome\"].unique()\n",
    "print(\"The training set has {} possible outcomes \\n\".format(len(training_outcomes)) )\n",
    "print(\", \".join(training_outcomes)+\".\")\n",
    "print(\"\\nThe testing set has {} possible outcomes \\n\".format(len(testing_outcomes)))\n",
    "print(\", \".join(testing_outcomes)+\".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Preprocessing\n",
    "\n",
    "\n",
    "## Extracting the labels\n",
    "\n",
    "As depicted previously the testing set has an additional 15 attack types that are not available in training data hence we will need  more general labels to train the model for the classification task.\n",
    "\n",
    "\n",
    "The 37 attack types available in the dataset can be clustered into four general attack types\n",
    "\n",
    "- Denial of service attacks\n",
    "- Remote to Local attacks\n",
    "- User to Root\n",
    "- Probe attacks\n",
    "\n",
    "Our model will perform binary classification of the data to two classes indicating whether the traffic is normal or an Attack, however we will  use the four attack types to analyze the results and calculate performance metrics for each general attack type.\n",
    "\n",
    "The next section replaces the current outcome field with a Class field that has one of the following values :\n",
    "- Normal\n",
    "- Dos\n",
    "- R2L\n",
    "- U2R\n",
    "- Probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A list ot attack names that belong to each general attack type\n",
    "dos_attacks=[\"snmpgetattack\",\"back\",\"land\",\"neptune\",\"smurf\",\"teardrop\",\"pod\",\"apache2\",\"udpstorm\",\"processtable\",\"mailbomb\"]\n",
    "r2l_attacks=[\"snmpguess\",\"worm\",\"httptunnel\",\"named\",\"xlock\",\"xsnoop\",\"sendmail\",\"ftp_write\",\"guess_passwd\",\"imap\",\"multihop\",\"phf\",\"spy\",\"warezclient\",\"warezmaster\"]\n",
    "u2r_attacks=[\"sqlattack\",\"buffer_overflow\",\"loadmodule\",\"perl\",\"rootkit\",\"xterm\",\"ps\"]\n",
    "probe_attacks=[\"ipsweep\",\"nmap\",\"portsweep\",\"satan\",\"saint\",\"mscan\"]\n",
    "\n",
    "# Our new labels\n",
    "classes=[\"Normal\",\"Dos\",\"R2L\",\"U2R\",\"Probe\"]\n",
    "\n",
    "#Helper function to label samples to 5 classes\n",
    "def label_attack (row):\n",
    "    if row[\"outcome\"] in dos_attacks:\n",
    "        return classes[1]\n",
    "    if row[\"outcome\"] in r2l_attacks:\n",
    "        return classes[2]\n",
    "    if row[\"outcome\"] in u2r_attacks:\n",
    "        return classes[3]\n",
    "    if row[\"outcome\"] in probe_attacks:\n",
    "        return classes[4]\n",
    "    return classes[0]\n",
    "\n",
    "\n",
    "#We combine the datasets temporarily to do the labeling \n",
    "test_samples_length = len(testing_df)\n",
    "df=pd.concat([training_df,testing_df])\n",
    "df[\"Class\"]=df.apply(label_attack,axis=1)\n",
    "\n",
    "\n",
    "# The old outcome field is dropped since it was replaced with the Class field, the difficulty field will be dropped as well.\n",
    "df=df.drop(\"outcome\",axis=1)\n",
    "df=df.drop(\"difficulty\",axis=1)\n",
    "\n",
    "# we again split the data into training and test sets.\n",
    "training_df= df.iloc[:-test_samples_length, :]\n",
    "testing_df= df.iloc[-test_samples_length:,:]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the new labels "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training set has 5 possible outcomes \n",
      "\n",
      "Normal, Dos, R2L, Probe, U2R.\n",
      "\n",
      "The testing set has 5 possible outcomes \n",
      "\n",
      "Dos, Normal, Probe, R2L, U2R.\n"
     ]
    }
   ],
   "source": [
    "training_outcomes=training_df[\"Class\"].unique()\n",
    "testing_outcomes=testing_df[\"Class\"].unique()\n",
    "print(\"The training set has {} possible outcomes \\n\".format(len(training_outcomes)) )\n",
    "print(\", \".join(training_outcomes)+\".\")\n",
    "print(\"\\nThe testing set has {} possible outcomes \\n\".format(len(testing_outcomes)))\n",
    "print(\", \".join(testing_outcomes)+\".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Features\n",
    "\n",
    "For continuous features we use the `MinMaxScaler` provided by the scikit-learn library, we only allow the scaler to fit the training set values and then we use it to scale both the training and testing sets. The `minmax_scale_values` helper function does this task.\n",
    "\n",
    "As for the discrete features we use one hot encoding. The `encode_text` function achieves this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function for scaling continous values\n",
    "def minmax_scale_values(training_df,testing_df, col_name):\n",
    "    scaler = MinMaxScaler()\n",
    "    scaler = scaler.fit(training_df[col_name].reshape(-1, 1))\n",
    "    train_values_standardized = scaler.transform(training_df[col_name].reshape(-1, 1))\n",
    "    training_df[col_name] = train_values_standardized\n",
    "    test_values_standardized = scaler.transform(testing_df[col_name].reshape(-1, 1))\n",
    "    testing_df[col_name] = test_values_standardized\n",
    "    \n",
    "    \n",
    "#Helper function for one hot encoding\n",
    "def encode_text(training_df,testing_df, name):\n",
    "    training_set_dummies = pd.get_dummies(training_df[name])\n",
    "    testing_set_dummies = pd.get_dummies(testing_df[name])\n",
    "    for x in training_set_dummies.columns:\n",
    "        dummy_name = \"{}_{}\".format(name, x)\n",
    "        training_df[dummy_name] = training_set_dummies[x]\n",
    "        if x in testing_set_dummies.columns :\n",
    "            testing_df[dummy_name]=testing_set_dummies[x]\n",
    "        else :\n",
    "            testing_df[dummy_name]=np.zeros(len(testing_df))\n",
    "    training_df.drop(name, axis=1, inplace=True)\n",
    "    testing_df.drop(name, axis=1, inplace=True)\n",
    "    \n",
    "    \n",
    "sympolic_columns=[\"protocol_type\",\"service\",\"flag\"]\n",
    "label_column=\"Class\"\n",
    "for column in df.columns :\n",
    "    if column in sympolic_columns:\n",
    "        encode_text(training_df,testing_df,column)\n",
    "    elif not column == label_column:\n",
    "        minmax_scale_values(training_df,testing_df, column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>...</th>\n",
       "      <th>flag_REJ</th>\n",
       "      <th>flag_RSTO</th>\n",
       "      <th>flag_RSTOS0</th>\n",
       "      <th>flag_RSTR</th>\n",
       "      <th>flag_S0</th>\n",
       "      <th>flag_S1</th>\n",
       "      <th>flag_S2</th>\n",
       "      <th>flag_S3</th>\n",
       "      <th>flag_SF</th>\n",
       "      <th>flag_SH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>3.558064e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.057999e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.681203e-07</td>\n",
       "      <td>6.223962e-06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.442067e-07</td>\n",
       "      <td>3.206260e-07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 123 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration     src_bytes     dst_bytes  land  wrong_fragment  urgent  hot  \\\n",
       "0       0.0  3.558064e-07  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "1       0.0  1.057999e-07  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "2       0.0  0.000000e+00  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "3       0.0  1.681203e-07  6.223962e-06   0.0             0.0     0.0  0.0   \n",
       "4       0.0  1.442067e-07  3.206260e-07   0.0             0.0     0.0  0.0   \n",
       "\n",
       "   num_failed_logins  logged_in  num_compromised   ...     flag_REJ  \\\n",
       "0                0.0        0.0              0.0   ...            0   \n",
       "1                0.0        0.0              0.0   ...            0   \n",
       "2                0.0        0.0              0.0   ...            0   \n",
       "3                0.0        1.0              0.0   ...            0   \n",
       "4                0.0        1.0              0.0   ...            0   \n",
       "\n",
       "   flag_RSTO  flag_RSTOS0  flag_RSTR  flag_S0  flag_S1  flag_S2  flag_S3  \\\n",
       "0          0            0          0        0        0        0        0   \n",
       "1          0            0          0        0        0        0        0   \n",
       "2          0            0          0        1        0        0        0   \n",
       "3          0            0          0        0        0        0        0   \n",
       "4          0            0          0        0        0        0        0   \n",
       "\n",
       "   flag_SF  flag_SH  \n",
       "0        1        0  \n",
       "1        1        0  \n",
       "2        0        0  \n",
       "3        1        0  \n",
       "4        1        0  \n",
       "\n",
       "[5 rows x 123 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>...</th>\n",
       "      <th>flag_REJ</th>\n",
       "      <th>flag_RSTO</th>\n",
       "      <th>flag_RSTOS0</th>\n",
       "      <th>flag_RSTR</th>\n",
       "      <th>flag_S0</th>\n",
       "      <th>flag_S1</th>\n",
       "      <th>flag_S2</th>\n",
       "      <th>flag_S3</th>\n",
       "      <th>flag_SF</th>\n",
       "      <th>flag_SH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000047</td>\n",
       "      <td>9.408217e-06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.449313e-08</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.145093e-08</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 123 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration     src_bytes     dst_bytes  land  wrong_fragment  urgent  hot  \\\n",
       "0  0.000000  0.000000e+00  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "1  0.000000  0.000000e+00  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "2  0.000047  9.408217e-06  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "3  0.000000  1.449313e-08  0.000000e+00   0.0             0.0     0.0  0.0   \n",
       "4  0.000023  0.000000e+00  1.145093e-08   0.0             0.0     0.0  0.0   \n",
       "\n",
       "   num_failed_logins  logged_in  num_compromised   ...     flag_REJ  \\\n",
       "0                0.0        0.0              0.0   ...            1   \n",
       "1                0.0        0.0              0.0   ...            1   \n",
       "2                0.0        0.0              0.0   ...            0   \n",
       "3                0.0        0.0              0.0   ...            0   \n",
       "4                0.0        0.0              0.0   ...            0   \n",
       "\n",
       "   flag_RSTO  flag_RSTOS0  flag_RSTR  flag_S0  flag_S1  flag_S2  flag_S3  \\\n",
       "0          0            0          0        0        0        0        0   \n",
       "1          0            0          0        0        0        0        0   \n",
       "2          0            0          0        0        0        0        0   \n",
       "3          0            0          0        0        0        0        0   \n",
       "4          1            0          0        0        0        0        0   \n",
       "\n",
       "   flag_SF  flag_SH  \n",
       "0        0        0  \n",
       "1        0        0  \n",
       "2        1        0  \n",
       "3        1        0  \n",
       "4        0        0  \n",
       "\n",
       "[5 rows x 123 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testing_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we extract the values from the pandas dataframes as Numpy arrays, where :\n",
    "- `x` holds the features of the training dataset\n",
    "- `y` holds the classification of the training dataset to one of the five possible values\n",
    "- `x_test` holds the features of the testing dataset\n",
    "- `y_test` holds the classification of the testing dataset to one of the five possible values\n",
    "- `y0` holds the classification of the training dataset to one of two possible labels, 0 for normal traffic or 1 for an attack\n",
    "- `y0_test` holds the classification of the testing dataset to one of two possible labels, 0 for normal traffic or 1 for an attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y=training_df,training_df.pop(\"Class\").values\n",
    "x=x.values\n",
    "x_test,y_test=testing_df,testing_df.pop(\"Class\").values\n",
    "x_test=x_test.values\n",
    "y0=np.ones(len(y),np.int8)\n",
    "y0[np.where(y==classes[0])]=0\n",
    "y0_test=np.ones(len(y_test),np.int8)\n",
    "y0_test[np.where(y_test==classes[0])]=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(125973, 122)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22543, 122)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(125973,)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22543,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Model\n",
    "\n",
    "In order to avoid the imbalance of the samples representing each attack type in the training data, and to avoid the model’s inability to learn about new attack types by observing existing ones, we present an approach that utilizes autoencoders and reconstruction error to detect anomalies.\n",
    "\n",
    "## Architecture and Training\n",
    "\n",
    "\n",
    "In this approach we implemented a sparse autoencoder with dropout on the inputs, it consists of an input layer of 122 neurons due to the fact that the number of features for each sample is 122 followed by a dropout layer and a hidden layer of 8 neuron units so the hidden representation of the autoencoder has a compression ratio of 122/8 forcing it to learn interesting patterns and relations between the features, finally there is an output layer of 122 units, the activation of both the hidden layer and the output layer is the relu function.\n",
    "\n",
    "The autoencoder was trained to reconstruct its input, in other words it learns the identity function, the model was trained using only the samples labeled “Normal” in the training dataset allowing it to capture the nature of normal behavior, this was accomplished by training the model to minimize the mean squared error between its output and its input.\n",
    "\n",
    "The regularization constraints enforced over the autoencoder prevent it from simply copying the input to the output and overfitting the data, furthermore the dropout presented on the inputs makes the autoencoder a special case of a denoising autoencoder, this kind of autoencoders is trained to reconstruct the input from a distorted corrupted version of itself, forcing the autoencoder to learn even more properties of the data.\n",
    "\n",
    "The model is trained for 10 epochs using an Adam optimizer with a batch size of 100, furthermore we held out 10% of the normal training samples to validate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60608 samples, validate on 6735 samples\n",
      "Epoch 1/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0263 - val_loss: 0.0130\n",
      "Epoch 2/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0124 - val_loss: 0.0093\n",
      "Epoch 3/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0103 - val_loss: 0.0079\n",
      "Epoch 4/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0094 - val_loss: 0.0072\n",
      "Epoch 5/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0089 - val_loss: 0.0067\n",
      "Epoch 6/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0086 - val_loss: 0.0063\n",
      "Epoch 7/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0083 - val_loss: 0.0061\n",
      "Epoch 8/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0081 - val_loss: 0.0059\n",
      "Epoch 9/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0080 - val_loss: 0.0058\n",
      "Epoch 10/10\n",
      "60608/60608 [==============================] - 1s - loss: 0.0078 - val_loss: 0.0056\n"
     ]
    }
   ],
   "source": [
    "#Buildling and training the model\n",
    "\n",
    "def getModel():\n",
    "    inp = Input(shape=(x.shape[1],))\n",
    "    d1=Dropout(0.5)(inp)\n",
    "    encoded = Dense(8, activation='relu', activity_regularizer=regularizers.l2(10e-5))(d1)\n",
    "    decoded = Dense(x.shape[1], activation='relu')(encoded)\n",
    "    autoencoder = Model(inp, decoded)\n",
    "    autoencoder.compile(optimizer='adam', loss='mean_squared_error')\n",
    "    return autoencoder\n",
    "\n",
    "autoencoder=getModel()\n",
    "history=autoencoder.fit(x[np.where(y0==0)],x[np.where(y0==0)],\n",
    "               epochs=10,\n",
    "                batch_size=100,\n",
    "                shuffle=True,\n",
    "                validation_split=0.1\n",
    "                       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "The model performs anomaly detection by calculating the reconstruction error of samples, since the model was trained using normal data samples only the reconstruction error of samples that represent attacks should be relatively high compared to the reconstruction error of normal data samples, this intuition allows us to detect attacks by setting a threshold for the reconstruction error, if a data sample has a reconstruction error higher than the preset threshold then the sample is classified as an attack, otherwise it’s classified as normal traffic.\n",
    "\n",
    "\n",
    "For the choice of a threshold two values can be helpful for guiding the process, the model loss over the training data and over the validation data, we found by experiment that a choice around these values produces acceptable results, for our experiments we use the model loss over the training data as a threshold. \n",
    "\n",
    "\n",
    "Due to the nature of this approach it can only be used for 2-Class classification as it is purely for anomaly detection and not classification.\n",
    "\n",
    "The following section evaluates the performance over the testing dataset, the `calculate_losses` is a helper function that accepts the original features and the predicted features (the autoencoder's output) and returns the reconstruction loss of each data sample, afterwards each data sample is classified according to its reconstruction error and the preset threshold.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function that calculates the reconstruction loss of each data sample\n",
    "def calculate_losses(x,preds):\n",
    "    losses=np.zeros(len(x))\n",
    "    for i in range(len(x)):\n",
    "        losses[i]=((preds[i] - x[i]) ** 2).mean(axis=None)\n",
    "        \n",
    "    return losses\n",
    "\n",
    "# We set the threshold equal to the training loss of the autoencoder\n",
    "threshold=history.history[\"loss\"][-1]\n",
    "\n",
    "testing_set_predictions=autoencoder.predict(x_test)\n",
    "test_losses=calculate_losses(x_test,testing_set_predictions)\n",
    "testing_set_predictions=np.zeros(len(test_losses))\n",
    "testing_set_predictions[np.where(test_losses>threshold)]=1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "To evaluate the model we calculate the following performance metrics :\n",
    "- Accuracy\n",
    "- Recall\n",
    "- Precision\n",
    "- F1 Score\n",
    "- Detection rate for each of the five possible labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance over the testing data set \n",
      "\n",
      "Accuracy : 0.9032515636783037 , Recall : 0.959167770591444 , Precision : 0.88135471860232 , F1 : 0.918616366282324\n",
      "\n",
      "Normal Detection Rate : 0.17064881565396497\n",
      "Dos Detection Rate : 0.9392352016762703\n",
      "R2L Detection Rate : 0.9785898855666297\n",
      "U2R Detection Rate : 0.9701492537313433\n",
      "Probe Detection Rate : 1.0\n"
     ]
    }
   ],
   "source": [
    "accuracy=accuracy_score(y0_test,testing_set_predictions)\n",
    "recall=recall_score(y0_test,testing_set_predictions)\n",
    "precision=precision_score(y0_test,testing_set_predictions)\n",
    "f1=f1_score(y0_test,testing_set_predictions)\n",
    "print(\"Performance over the testing data set \\n\")\n",
    "print(\"Accuracy : {} , Recall : {} , Precision : {} , F1 : {}\\n\".format(accuracy,recall,precision,f1 ))\n",
    "\n",
    "#\n",
    "\n",
    "\n",
    "for class_ in classes:\n",
    "    print(class_+\" Detection Rate : {}\".format(len(np.where(np.logical_and(testing_set_predictions==1 , y_test==class_))[0])/len(np.where(y_test==class_)[0])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results performs well and is stable across many test runs and it outperforms many other approaches including other approaches we attempted as part of our study.\n",
    "\n",
    "The following section plots the confusion matrix, the `plot_confusion_matrix` helper function was adapted from the __[scikit-learn library documentation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html)__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV8AAAEmCAYAAADFmJOIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcV2Xd//HXe9jEAEEwM0AhBTdcQcIl0ixAQ8HSO9CUlF/kVvetd7llYaZ3ad253O5b0uIWLpmGRoa5Ai6sKgouCIosIqJoBsPn98e5Br+Mw2zOzPnOd97Px+M85nuuc51zrsMwn7nmc65zHUUEZmbWtMryboCZWUvk4GtmlgMHXzOzHDj4mpnlwMHXzCwHDr5mZjlw8LV6kdRe0l8kvSvpT5/iOMdI+ltDti0vkr4k6cW822HNgzzOt7RJOho4HdgJeA+YCVwYEY99yuMeC3wf2C8i1n3qhhY5SQH0iYgFebfFSoN7viVM0unApcD/AFsD2wJXASMa4PDbAS+1hMBbG5Ja590Ga2YiwksJLsAWwPvAUdXUaUcWnN9My6VAu7TtQGAx8N/AMmAJcHza9jPg38DadI6xwHnAHwqO3QsIoHVa/w7wClnv+1XgmILyxwr22w94Cng3fd2vYNvDwM+Bx9Nx/gZ028S1VbT/jIL2jwQOBV4CVgLnFNQfCDwJrEp1rwDapm2PpGtZk673WwXHPxN4C/h9RVnaZ/t0jr3T+ueBFcCBef/f8FIci3u+pWtfYDPg7mrq/BgYBOwJ7EEWgM4t2P45siDenSzAXimpS0SMJ+tN3x4RHSLixuoaIukzwOXAIRHRkSzAzqyi3pbA/aluV+A3wP2SuhZUOxo4Hvgs0Bb4YTWn/hzZv0F34KfA9cC3gf7Al4CfSvpCqlsOnAZ0I/u3Oxg4GSAiBqc6e6Trvb3g+FuS/RUwrvDEEfEyWWD+o6TNgd8CN0fEw9W011oQB9/S1RVYEdWnBY4Bzo+IZRGxnKxHe2zB9rVp+9qI+CtZr2/HerZnPdBPUvuIWBIRz1VR5+vA/Ij4fUSsi4hbgXnAYQV1fhsRL0XEh8AdZL84NmUtWX57LXAbWWC9LCLeS+d/DtgdICKeiYip6byvAdcCX67FNY2PiI9SezYSEdcD84FpwDZkv+zMAAffUvY20K2GXOTngYUF6wtT2YZjVAreHwAd6tqQiFhD9qf6icASSfdL2qkW7aloU/eC9bfq0J63I6I8fa4IjksLtn9Ysb+kvpLuk/SWpNVkPftu1RwbYHlE/KuGOtcD/YD/i4iPaqhrLYiDb+l6EvgXWZ5zU94k+5O5wraprD7WAJsXrH+ucGNEPBgRXyPrAc4jC0o1taeiTW/Us011cTVZu/pERCfgHEA17FPtUCFJHcjy6DcC56W0ihng4FuyIuJdsjznlZJGStpcUhtJh0i6OFW7FThX0laSuqX6f6jnKWcCgyVtK2kL4OyKDZK2lnR4yv1+RJa+KK/iGH8F+ko6WlJrSd8CdgHuq2eb6qIjsBp4P/XKT6q0fSnwhU/sVb3LgGci4v+R5bKv+dSttJLh4FvCIuI3ZGN8zwWWA4uAU4F7UpULgKeB2cAc4NlUVp9zTQZuT8d6ho0DZhnZqIk3yUYAfJl0M6vSMd4Ghqe6b5ONVBgeESvq06Y6+iHZzbz3yHrlt1fafh4wQdIqSf9R08EkjQCGkaVaIPs+7C3pmAZrsTVrfsjCzCwH7vmameXAwdfMLAcOvmZmOXDwNTPLQYufDKR9+/bRqVOnvJth1ejevXvNlSx3M2bMWBERWzXkMdNscrXxYEQMa8hzN7YWH3w7derE6NGj826GVeOCC+o1+s2aWMeOHSs/ndiUanoasei0+OBrZsVNqulBw2x2xubGOV8zK2plZWU1LjWRdJOkZZLmFpT9StI8SbMl3S2pc8G2syUtkPSipKEF5cNS2QJJZxWU95Y0TdJ8SbdLalvjddXpX8HMrAlJapDgC9xM9sRhoclAv4jYnWyO57PTOXcBRgG7pn2uktRKUivgSuAQssfeR6e6ABcBl0REH+AdsilYq+Xga2ZFTVKNS00i4hGyR9sLy/5WMGvfVKBH+jwCuC1NFfoqsIBsruuBwIKIeCUi/k02TekIZQ34CjAx7T+B6ie0Ahx8zazI1TL4dpP0dMEyrqbjVnICMCl97k42D0qFxalsU+VdgVUFgbyivFq+4WZmRa2WaYUVETGgPseX9GNgHfDHiqIqqgVVd1ajmvrVcvA1s6JV27TCpzj+GLKZ9A6Oj4dMLAZ6FlTrwcfzXFdVvgLoLKl16v0W1t8kpx3MrKg10A23T5A0jOw9e4dHxAcFm+4FRklqJ6k30AeYTvZC1z5pZENbspty96agPQU4Mu0/BvhzjddVr1abmTWRhrjhJulWsre77ChpsaSxZG+o7ghMljRT0jUA6f1+dwDPAw8Ap0REeerVngo8CLwA3FHwLsIzgdMlLSDLAVf7Ullw2sHMilxDpB0ioqrHWDcZICPiQuDCKsr/SvbGlcrlr5CNhqg1B18zK1oV43xLkYOvmRW1xrzhlicHXzMrau75mpk1scYeapYnB18zK2oOvmZmOXDawcwsB+75mpk1MQ81MzPLiXu+ZmY5cM/XzKyJeaiZmVlOHHzNzHLgtIOZWQ7c8zUza2IeamZmlhP3fM3McuCer5lZE/NQMzOznDj4mpnloFWrVnk3oVE4+JpZ0fJoBzOznDjtYGaWAwdfM7MmJsk5XzOzPDjna2aWg1JNO5TmrxQzKwkVox1qWmpxnJskLZM0t6BsS0mTJc1PX7ukckm6XNICSbMl7V2wz5hUf76kMQXl/SXNSftcrlr8xnDwNbOiVvGUW3VLLdwMDKtUdhbwUET0AR5K6wCHAH3SMg64OrVjS2A88EVgIDC+ImCnOuMK9qt8rk9w8DWzolVxw62mpSYR8QiwslLxCGBC+jwBGFlQ/rvITAU6S9oGGApMjoiVEfEOMBkYlrZ1iognIyKA3xUca5Oc8zWzotaIOd+tI2IJQEQskfTZVN4dWFRQb3Eqq658cRXl1XLwNbOiVsvRDt0kPV2wfl1EXFfPU1YV7aMe5dVy8DWzolbLnu+KiBhQx0MvlbRN6vVuAyxL5YuBngX1egBvpvIDK5U/nMp7VFG/Ws75mlnRqs3Ntk+RlrgXqBixMAb4c0H5cWnUwyDg3ZSeeBAYIqlLutE2BHgwbXtP0qA0yuG4gmNtknu+ZlbUGuIJN0m3kvVau0laTDZq4ZfAHZLGAq8DR6XqfwUOBRYAHwDHA0TESkk/B55K9c6PiIqbeCeRjahoD0xKS7UcfJuJL3/5ywwaNAiAJUuWcMstt9CpUyfGjBnD5ptvzuLFi/nDH/5AeXk5AwcO5PDDD+fdd98F4NFHH2Xq1Kl06dKFE044YcPYyEcffZQnnngiz8sqKSeddBIPPPAAW221FdOnT99Qfs0113DttdfSunVrhg4dygUXXMDChQsZMGAAffr0AWCfffbhsssu47333mPo0KEb9n3jjTcYNWoUF110UZNfTzFoqFnNImL0JjYdXEXdAE7ZxHFuAm6qovxpoF9d2uTg2wxsscUWDB48mF/+8pesXbuWMWPGsPfee7Pzzjvz8MMPM2PGDI466igGDRrE448/DsCMGTO48847NzrO6tWrufTSSykvL6dt27acddZZzJ07l9WrV+dxWSXnmGOO4Xvf+x7jxo3bUPbII49w//33M3XqVNq1a8fy5cs3bOvdu/cnfvl17Nhxo7IvfelLHHbYYY3f+CLmJ9wsV2VlZbRp04aysjLatm3L6tWr6dOnD7NmzQLgqaeeYrfddqv2GOXl5ZSXlwPQunXrkv1PnZcDDjiALl26bFR2ww03cPrpp9OuXTsAttpqq1ofb8GCBSxfvpz999+/QdvZ3DRizjdX7vk2A++++y5Tpkxh/PjxrF27lnnz5rFo0SI+/PBD1q9fD8CqVavYYostNuyz++67s/3227Ns2TLuueceVq1aBUDnzp0ZN24c3bp1495773Wvt5EtWLCAJ554gvPPP5927dpx4YUX0r9/fwAWLlzI/vvvT8eOHfnJT37yiSA7ceJEvvGNbzTb4NIQPKtZPUgK4DcR8d9p/YdAh4g4r7HOWUUbbgbui4iJTXXOxtC+fXv69evH+eefz4cffsjxxx/Pzjvv/Il6WaoK5s6dyzPPPEN5eTn77bcfRx99NFdddRWQBemLL76YTp06MXbsWGbOnMn777/fpNfTkqxbt45Vq1bxj3/8g2eeeYYxY8YwZ84cPve5z/H888/TtWtXZsyYwejRo5k+fTqdOnXasO/EiRO5/vrrc2x9cSjVWc0a86o+Ar4hqVt9dpbkXnnSt29fVq5cyZo1a1i/fj2zZ8+md+/etG/ffsN/zM6dO2/oxX7wwQcb0gtPPvkkPXv2/MQxV69ezVtvvcX222/fdBfSAnXv3p3DDz8cSQwYMICysjJWrFhBu3bt6Nq1KwB77bUXvXv3ZsGCBRv2mzNnDuvWrWOvvfbKq+lFo1TTDo0ZfNcB1wGnVd4gaTtJD6UZgx6StG0qv1nSbyRNAS6SdJ6kCZL+Juk1Sd+QdHGaPegBSW3Sfj+V9JSkuZKuq82MQs3JqlWr2G677WjTpg0Affr04a233mLBggXsscceQHa3fM6cOQAb9Z769evH0qVLgezGXcUx2rdvT+/evVm2bBnWeIYPH84///lPAObPn8+///1vunXrxvLlyzf8gnz11Vd5+eWX6dWr14b9/vSnP3HUUUdVdcgWpaFmNStGjd27vBKYLeniSuVXkE1cMUHSCcDlfDwRRV/gqxFRLuk8YHvgIGAX4EngmxFxhqS7ga8D9wBXRMT5AJJ+DwwH/rKpRkkaRzYDER07dmyQC21MCxcuZNasWfzwhz9k/fr1LF68mCeeeILnn3+e4447jkMPPZQ33niDqVOnAjB48GB23XVX1q9fzwcffMAtt9wCwNZbb83IkSOJCCQxZcoUlixZkuellZTjjz+eRx99lLfffpsdd9yRc845h2OPPZaTTz6ZgQMH0rZtW6699lok8cQTT3DBBRfQunVrWrVqxaWXXsqWW2654Vh33303Eyc262xZgymxvtQGqsgTNviBpfcjooOk84G1wIeknK+kFcA2EbE29V6XRES3lKOdEhET0jHOA9ZGxIWSytIxNouISMddGRGXSvomcAawObAl8H8R8cva5Hy33nrrGD16U0MArRhccMEFeTfBaqFjx47P1OMR32p16dIlDj74E0NxP+HOO+9s8HM3tqbIq14KPAv8tpo6hb8B1lTa9hFARKyXtDY+/m2xHmgtaTPgKmBARCxKAXuzBmm5meWqOed0a9LoyZL0+N0dwNiC4ieAUenzMcBjn+IUFYF2haQOwJGf4lhmVmR8w+3T+V+gcNTDD4DjJc0GjgX+s74HjohVwPXAHLL871PV72FmzUmpBt9GSztERIeCz0vJ8rEV668BX6lin+9UWj+vmmOeV/D5XODcmo5nZs1Pcw2uNfFYWjMrWn7CzcwsJ+75mpnlwMHXzCwHDr5mZk2soSZTL0YOvmZW1NzzNTPLgYOvmVkOHHzNzJpYc36CrSYOvmZW1HzDzcwsB+75mpnlwMHXzKyJOedrZpYTB18zsxyU6g230rwqMysZDTWZuqTTJD2X3nJ+q6TNJPWWNE3SfEm3S2qb6rZL6wvS9l4Fxzk7lb8oaWh9r8vB18yKVm0Cb22Cr6TuZG/QGRAR/YBWZK8yuwi4JCL6AO/w8evOxgLvRMQOwCWpHpJ2SfvtCgwDrpJUrwmHHXzNrKg14GuEWgPtJbUme7POErI36lS83XwCMDJ9HpHWSdsPVnaiEcBtEfFRRLwKLAAG1ue6HHzNrKiVlZXVuADdJD1dsIwrPEZEvAH8GnidLOi+CzwDrIqIdanaYqB7+twdWJT2XZfqdy0sr2KfOvENNzMrarXs2a6IiAHVHKMLWa+1N7AK+BNwSBVVo2KXTWzbVHmduedrZkWroXK+wFeBVyNieUSsBe4C9gM6pzQEQA/gzfR5MdAztaE1sAWwsrC8in3qxMHXzIpaAwXf14FBkjZPuduDgeeBKcCRqc4Y4M/p871pnbT9HxERqXxUGg3RG+gDTK/PdTntYGZFrSHG+UbENEkTgWeBdcAM4DrgfuA2SRekshvTLjcCv5e0gKzHOyod5zlJd5AF7nXAKRFRXp82OfiaWdFqyNcIRcR4YHyl4leoYrRCRPwLOGoTx7kQuPDTtmeTwVdSp+p2jIjVn/bkZmY1aYmPFz/HJ+/uVawHsG0jtsvMDGiBwTciem5qm5lZUynV4FurZIqkUZLOSZ97SOrfuM0yM/s451uLhyyanRpbLekK4CDg2FT0AXBNYzbKzKxCAz5eXFRqM9phv4jYW9IMgIhYWTHzj5lZY2uuwbUmtQm+ayWVkR6hk9QVWN+orTIzS1py8L0SuBPYStLPgP8AftaorTIzo2HH+RabGoNvRPxO0jNkz0YDHBURcxu3WWZmmRYbfJNWwFqy1ENp/kuYWVEq1bRDbUY7/Bi4Ffg82Qw+t0g6u7EbZmYGLXu0w7eB/hHxAYCkC8kmIf5FYzbMzKw5B9ea1Cb4LqxUrzXZZBRmZo2uxQVfSZeQ5Xg/AJ6T9GBaHwI81jTNM7OWriXecKsY0fAc2ZyXFaY2XnPMzDbW4nq+EXHjpraZmTWFFj3OV9L2ZBMH7wJsVlEeEX0bsV1mZkDp9nxr8yvlZuC3ZPP4HgLcAdzWiG0yM9ugxc5qBmweEQ8CRMTLEXEu2SxnZmaNqgHfXlx0ajPU7KP0ts+XJZ0IvAF8tnGbZWaWaa7BtSa1Cb6nAR2AH5DlfrcATmjMRpmZVWixwTcipqWP7/HxhOpmZk2ixQVfSXeT5vCtSkR8o1Fa1MR69uzJpZdemnczrBql+sNnNWupQ82uaLJWmJltQosLvhHxUFM2xMysKqX6l09t5/M1M8uFg6+ZWRNrzuN4a1LrZIqkdo3ZEDOzqrRq1arGpTYkdZY0UdI8SS9I2lfSlpImS5qfvnZJdSXpckkLJM2WtHfBccak+vMljanvddXmTRYDJc0B5qf1PST9X31PaGZWFw34hNtlwAMRsROwB/ACcBbwUET0AR5K65BNpdAnLeOAq1NbtgTGA18EBgLjKwJ2XdWm53s5MBx4GyAiZuHHi82sCTTU48WSOgGDgRsBIuLfEbEKGAFMSNUmACPT5xHA7yIzFegsaRtgKDA5IlZGxDvAZGBYfa6tNsG3LCIWViorr8/JzMzqqpbBt5ukpwuWcZUO8wVgOfBbSTMk3SDpM8DWEbEEIH2tmDqhO7CoYP/FqWxT5XVWmxtuiyQNBEJSK+D7wEv1OZmZWV1Iqm1Od0VEDKhme2tgb+D7ETFN0mV8nGKo8tRVlEU15XVWm57vScDpwLbAUmBQKjMza3QNlPNdDCwumC5hIlkwXprSCaSvywrq9yzYvwfwZjXldVZj8I2IZRExKiK6pWVURKyoz8nMzOqqIYJvRLxF9lf8jqnoYOB54F6gYsTCGODP6fO9wHFp1MMg4N2UlngQGCKpS7rRNiSV1Vlt3mRxPVV0qyOick7FzKzBNeDjxd8H/iipLdkb2I8n64DeIWks8DpwVKr7V+BQYAHZS4SPB4iIlZJ+DjyV6p0fESvr05ja5Hz/XvB5M+AINk44m5k1ioZ8yCIiZgJV5YUPrqJuAKds4jg3ATd92vbUZkrJ2wvXJf2ebHiFmVmja3ET61SjN7BdQzfEzKwqpfp4cW1yvu/wcc63DFhJ9UM0zMwaREudz5f07rY9yN7bBrA+5ULMzJpEqfZ8q/2VkgLt3RFRnhYHXjNrUg04t0NRqU3Od7qkvSPi2UZvjZlZgTo84dbsVPcOt9YRsQ44APiupJeBNWSP10VE7L2pfc3MGkpz7dnWpLqe73Syx+9GVlPHzKxRtcTgK4CIeLmJ2mJm9gktMfhuJen0TW2MiN80QnvMzDZokTlfoBXQgaqnUDMzaxItsee7JCLOb7KWmJlVoSUG39K8YjNrVlpi8P3ETD9mZk2pRT5eXN85Ks3MGlJL7PmameXOwdfMLAcOvmZmTaw5T5xTEwdfMytqDr5mZjlocaMdzMyKQan2fEvzV4qZWZFzz9fMipZvuJmZ5cQ5XzOzHLjna2aWAwdfM7MmVso539JMpphZyWjIV8dLaiVphqT70npvSdMkzZd0u6S2qbxdWl+QtvcqOMbZqfxFSUPre10OvmZW1Boy+AL/CbxQsH4RcElE9AHeAcam8rHAOxGxA3BJqoekXYBRwK7AMOAqSfV6z5GDr5kVtYYKvpJ6AF8HbkjrAr4CTExVJvDx29pHpHXS9oNT/RHAbRHxUUS8CiwABtbnuhx8zayo1TL4dpP0dMEyropDXQqcAaxP612BVRGxLq0vBrqnz92BRQBp+7up/obyKvapE99wM7OiVYee7YqIGFDNcYYDyyLiGUkHVhRXUTVq2FbdPnXinm8z1KtXL3bbbTf23HNPBgzI/r/96Ec/YqeddmL33XfniCOOYNWqVRvt8/rrr9OhQwd+/etf59HkknXjjTeydOlS5syZs6Hs4osv5oUXXmDWrFncddddbLHFFgDss88+zJgxgxkzZjBz5kxGjhy5YZ+hQ4cyb9485s+fz5lnnrmh/KCDDuKZZ55hzpw53HzzzSX7GvXqNFDaYX/gcEmvAbeRpRsuBTpLquiE9gDeTJ8XAz3T+VsDWwArC8ur2KdOHHybqSlTpjBz5kyefvppAL72ta8xd+5cZs+eTd++ffnFL36xUf3TTjuNQw45JI+mlrSbb76ZYcOGbVQ2efJk+vXrxx577MFLL73E2WefDcDcuXMZMGAAe+21F8OGDePaa6+lVatWlJWVceWVV3LIIYewyy67MHr0aHbeeWckMWHCBEaNGsVuu+3GwoULGTNmTB6XmauysrIal5pExNkR0SMiepHdMPtHRBwDTAGOTNXGAH9On+9N66Tt/4iISOWj0miI3kAfYHq9rqs+O1nxGTJkCK1bZ7/ABw0axOLFizdsu+eee/jCF77ArrvumlfzStajjz7KypUbv+5w8uTJlJeXAzB16lR69OgBwIcffrihfLPNNiP7WYaBAweyYMECXn31VdauXcttt93GiBEj6Nq1Kx999BHz58/fcNxvfvObTXVpRaOBRztUdiZwuqQFZDndG1P5jUDXVH46cBZARDwH3AE8DzwAnBIR5fU5sYNvMySJIUOG0L9/f6677rpPbL/ppps29HLXrFnDRRddxPjx45u6mQaccMIJTJo0acP6wIEDmTt3LnPmzOHEE0+kvLyc7t27s2jRx/dwFi9eTPfu3VmxYgVt2rShf//+ABx55JH07NnzE+ewuomIhyNiePr8SkQMjIgdIuKoiPgolf8rre+Qtr9SsP+FEbF9ROwYEZM2dZ6aNHnwlXSEpJC0U1rvJenogu17Sjr0Uxz/NUndGqKtxerxxx/n2WefZdKkSVx55ZU88sgjG7ZdeOGFtG7dmmOOOQaA8ePHc9ppp9GhQ4e8mttinXPOOaxbt44//vGPG8qmT59Ov3792GeffTj77LNp165dlT23il7xqFGjuOSSS5g2bRrvvfce69at+0TdUlabXm9zfQIuj9EOo4HHyPIu5wG9gKOBW9L2PYEBwF9zaFuz8PnPfx6Az372sxxxxBFMnz6dwYMHM2HCBO677z4eeuihDf8hp02bxsSJEznjjDNYtWoVZWVlbLbZZpx66ql5XkLJO+644xg+fDgHH3xwldvnzZvHmjVr6NevH4sXL96oR9ujRw/efDO7hzN16lQGDx4MZHn9vn37Nn7ji0ypzmpGRDTZAnQA3gD6AvNS2VSyMXQzyfIvrwPL0/q3yAYwPwHMSF93TPu1An4NzAFmA99P5a8B3YD2ZDmZ71bXpv79+0dz8v7778fq1as3fN53331j0qRJMWnSpNh5551j2bJlm9x3/Pjx8atf/aqpmtpgyIbyFO2y3XbbxZw5czasDx06NJ577rno1q3bRvV69eoVrVq1CiC23XbbeOONN6Jr167RqlWrePnll6NXr17Rpk2bmDlzZuyyyy4BxFZbbRVAtG3bNv7+97/HQQcdlPv1VrM8HQ0cM3bdddeYN29ejUtjnLuxl6bu+Y4EHoiIlyStlLQ3WSL7h5FyMJKWAgMi4tS03gkYHBHrJH0V+B/gm8A4oDewV9q2ZcF5OpANJ/ldRPyuciPSAOxxANtuu21jXWujWLp0KUcccQQA69at4+ijj2bYsGHssMMOfPTRR3zta18Dsptu11xzTZ5NbRFuueUWDjzwQLp168aiRYsYP378hnTC5MmTgaz3etJJJ3HAAQdw1llnsXbtWtavX8/JJ5/M22+/DcCpp57Kgw8+SKtWrbjpppt4/vnngWwI4fDhwykrK+Pqq69mypQpuV1rXpprWqEmiqjX+OD6nUy6H7g0IiZL+gHZeLn72Tj4foeNg29P4HKyIR0BtImInSTdCVwTEZMrneM1sp70xRHxR2owYMCAqBiuZcWpVH/4StAzUc2DDvWx2267xV133VVjvb59+zb4uRtbk/V8JXUlG9jcT1KQpQ2CmnO7PwemRMQRaWahhysOyaafLHkcOETSLdGUv13MzGqpKTPZR5KlAbaLiF4R0RN4lew5644F9d6rtL4FWZ4Y4DsF5X8DTqx4OqVS2uGnwNvAVQ16BWbW5BriIYti1JStHg3cXansTrJRD+skzZJ0GtkTJ7tIminpW8DFwC8kPU7WW65wA9nNudmSZpGNmCj0X8Bmki5uhGsxM/tUmiztEBEHVlF2+Saq71NpvXB8zU/SvuvInjw5vdIxexWsHl/XdppZcSnVnL9nNTOzoubga2bWxJrzE2w1cfA1s6Lm4GtmloNSDb7Nc4yGmVkz556vmRW1Uu35OviaWdGS1GwfoqhJaV6VmVmRc8/XzIqa0w5mZjlw8DUzy4GDr5lZDhx8zcyaWCk/XuzRDmZmOXDP18yKWqn2fB18zayo+SELMzNrMO75mllRK9W0g3u+Zla0KkY71LTU4jg9JU2R9IJSv8e5AAALvklEQVSk5yT9ZyrfUtJkSfPT1y6pXJIul7RA0mxJexcca0yqP1/SmPpem4OvmbUE64D/joidgUHAKZJ2Ac4CHoqIPsBDaR3gEKBPWsYBV8OGt6SPB74IDATGVwTsunLwNbOi1hCvjo+IJRHxbPr8HvAC0B0YAUxI1SYAI9PnEcDvIjMV6CxpG2AoMDkiVkbEO8BkYFh9rss5XzMrBd0kPV2wfl1EXFdVRUm9gL2AacDWEbEEsgAt6bOpWndgUcFui1PZpsrrzMHXzIpaLW+4rYiIAbU4VgfgTuC/ImJ1NceuakNUU15nTjuYWYsgqQ1Z4P1jRNyVipemdALp67JUvhjoWbB7D+DNasrrzMHXzIpWA452EHAj8EJE/KZg071AxYiFMcCfC8qPS6MeBgHvpvTEg8AQSV3SjbYhqazOnHYws6LWQON89weOBeZImpnKzgF+CdwhaSzwOnBU2vZX4FBgAfABcDxARKyU9HPgqVTv/IhYWZ8GOfiaWcmLiMeoOl8LcHAV9QM4ZRPHugm46dO2ycHXzIqan3AzM7MG456vmRW1Uu35OviaWVEr1eDrtIOZWQ7c8zWzouV3uJmZWYNyz9fMipp7vmZm1mDc8zWzouaer5mZNRj3fM2sqJVqz9fB18yKWqkGX6cdzMxy4J6vmRUtP2RhZmYNyj1fMytqpdrzdfA1s6JWqsHXaQczsxy452tmRa1Ue77K3hPXcklaDizMux0NrBuwIu9GWLVK8Xu0XURs1ZAHlPQA2b9VTVZExLCGPHdja/HBtxRJejoiBuTdDts0f4/MOV8zsxw4+JqZ5cDBtzRdl3cDrEb+HrVwzvmameXAPV8zsxw4+JqZ5cDB18wsBw6+LYyknSV9RVKbvNtiG1OpPsplVfLjxS3PKKAnUC7piYhYm3eDLBPp7rekfYFXI+KtnJtkjcg935bnZ8BrwLeAA9wDzp+kvSS1TZ93AC4A1uXbKmtsDr4tQOGfsxGxHrgQWIIDcLE4D/hLCsAvA+8C/waQVCapVY5ts0bi4FviJKngz9khkg4EOpP1rl4nC8D7OQA3PUllABExAngHuAPoSPaXyeZp23qgbU5NtEbkhyxaCEmnA0cAzwMdgBsiYoqkM4Hdgasj4rE829iSVPqluFVELJf0Z2AHoJzs+7SOLPC+CZwdER/m1mBrcL7h1gJI+ipwUER8SdIvgIHAaElExEWSTgMW5NvKlqUg8P4AGCDppIgYIeka4CvARWR/mXYBXnTgLT3u+Zagwl5VWt8R+Ag4EPg2cCxwCbANcGFE/C2PdrZ0kkYC5wKHRcSSgvI/Ae2BkRHhG28lyjnfElPpz9kvSupCNmzpNaAPWXphCTADmJUWy8f2wH0RsURSm4q8e0QcBSwFPp9r66xROe1QYgoC74nAj4DngL9Jug2YC0yQtDcwHDgiIpbm1tgWpPJfI8kbwGBJnSJidar3H8DiiBjb5I20JuXgWyIq9Xg/S3YTbSAwAPgaMBa4gmwY0xeBURHxSk7NbVEqfW++AbwHvA88SJYGOl7Si2T53R8Dh+XVVms6zvmWgEo/3KeQ/bl6QER8OZUdAnyV7Af+sohYmVtjW7B0c+1osrl8zwBOBl4Avg/0AjYDxkfEnLzaaE3HOd8SUBB4RwCjgWlAd0m3p+2TgEeANoDnD8iBpL2Aw8luevYAlgE3AF+MiHMi4mjgOAfelsM932asUo93AHA2MCkibkhPtT0LzIuI0anOZyJiTX4tbjkkdQa6RsTLknYne2JtJdkwslPSsL+zyZ5uGxsRf8ivtZYH53ybqSryiLuSPSV1kKSnImJWurH2iqSbI+I7DrxNQ1JroC8wXNI2ZK8+PyYiPkhB+ZZUdTnwG2BqPi21PDn4NlMFgfcrZDfTRgI7k93AOVzS+vQnbG9JvfNracuSfimuSzfQzgH2Bc6IiA9SldbA0DT2+qvA0Ih4I6fmWo6c823G0jwNJwGzI2JtRMwG/gx8Bjha0q4AEfFqfq1sOdKY6mFptS/ZHA1XAntLOgwgIq4A7gJmA99y4G253PNtRqoYK/oqWR6xj6TdI2J2RDyeBut/hWygvjWdNsD+kn4KEBH7SupGNsLhMEmrgFZk+d9b06Q51kI5+DYTlXK8h5FNurIKOBW4DPiPVGVORDwsaZrnA2gakj4XEW9FxDJJS4FdyHq3RMQKSX8h+36dCewBHOzAax7t0ExUBF9JJwPfBf4KfAP4LVnw/V+y+RtujIjn82tpyyJpJ7IZyC4DpgMTgf2BQ8lmjzslfd/6kD3g0sapBgPnfIuepG3TELFIT64dBRwdET8G9gO+BxxJNkF6K7Lxo9Z01gBPAm8B/w+4CuhE9vTaauAKSccCpwGrHXitgoNvEZO0NfDfwEmSOkTEMmAF6S0HEfEO2Q/17mmynB9FxIrcGtwCRcQish7v3sBQYArZrHE/B/4CbAl8B7giIv6VUzOtCDn4FrflwFNkjwsfnx6ceAW4LY0lBdgO6JFeNePpB5tQweuZzgSCbDzvm2Tzacwhe2z4DWCMU0FWmXO+RSjlB8si4sX0Az4cOASYGRHXSbqa7MbNbLJJco7xD3c+0venLfAT4AtkPeCzIuKelA9emv5CMduIg2+RkdSVrMe7guxNw+VkE7EcTfaKmSURca2kL5JNuL3Q43jzlx6aeBT4v4j4ed7tseLnoWZFJiLeTq/9+TtZWmgP4HayGcn+DeyWelu/jYiP8mupFUp/pZwJbCdp84In2syq5OBbhCLiH5KGApeTBd+tyR6aGEU2R++OwK1kQ8useDxJNvzPrEZOOxQxSV8ne9faoIhYmR5fbQNsnl4LZEXGvV6rLfd8i1hE3C9pPTBV0r4R8XbebbLqOfBabTn4FrmImCSpLfB3Sf39WKpZaXDaoZlID1m8n3c7zKxhOPiameXAT7iZmeXAwdfMLAcOvmZmOXDwNTPLgYOvfYKkckkzJc2V9CdJm3+KYx0o6b70+XBJZ1VTt3OaLL6u5zhP0g9rW16pzs2SjqzDuXpJmlvXNppV5uBrVfkwIvaMiH5k80mcWLhRmTr/34mIeyPil9VU6QzUOfiaNUcOvlaTR4EdUo/vBUlXAc8CPSUNkfSkpGdTD7kDgKRhkuZJeoyCuQ4kfUfSFenz1pLuljQrLfsBvwS2T73uX6V6P5L0lKTZkn5WcKwfS3pR0t/J5rqolqTvpuPMknRnpd78VyU9KuklScNT/VaSflVw7u992n9Is0IOvrZJacL2Q8gmBocsyP0uIvYie33OucBXI2Jv4GngdEmbAdcDhwFfAj63icNfDvwzIvYgmwP3OeAs4OXU6/6RpCFAH7LJhPYE+ksaLKk/2SRDe5EF931qcTl3RcQ+6XwvAGMLtvUCvgx8HbgmXcNY4N2I2Ccd/7uSetfiPGa14seLrSrtJc1Mnx8FbiR7m8bCiJiaygeRvaX38fRCh7Zks3rtBLwaEfMBJP0BGFfFOb4CHAcQEeXAu2nioEJD0jIjrXcgC8Ydgbsr5lGQdG8trqmfpAvIUhsdyN6xVuGO9Nj2fEmvpGsYAuxekA/eIp37pVqcy6xGDr5WlQ8jYs/CghRg1xQWAZMjYnSlenuSvVKnIQj4RURcW+kc/1WPc9wMjIyIWZK+AxxYsK3ysSKd+/sRURikkdSrjuc1q5LTDlZfU4H9Je0A2VSKkvoC84DekrZP9UZvYv+HgJPSvq0kdQLeI+vVVngQOKEgl9xd2RucHwGOkNReUkeyFEdNOgJLJLUBjqm07ShJZanNXwBeTOc+KdVHUl9Jn6nFecxqxT1fq5eIWJ56kLdKapeKz42IlySNA+6XtAJ4DOhXxSH+E7hO0liyVyWdFBFPSno8DeWalPK+OwNPpp73+8C3I+JZSbcDM4GFZKmRmvwEmJbqz2HjIP8i8E+ySetPjIh/SbqBLBf8bHpzyHJgZO3+dcxq5ol1zMxy4LSDmVkOHHzNzHLg4GtmlgMHXzOzHDj4mpnlwMHXzCwHDr5mZjn4/7v4VEZqJLVkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe55b6d3748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plotting confusion matrix\n",
    "\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Greys):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "   \n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "c = confusion_matrix(y0_test,testing_set_predictions)\n",
    "plot_confusion_matrix(c,[\"Normal\",\"Attack\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following violin plot shows the distribution of reconstruction loss values for all data samples in the testing set, it clearly shows that the  loss values of attacks are mostly higher then the threshold value, the opposite is true for normal samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD8CAYAAACGsIhGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmUXPV14PHvraWr91WtXS3JgAwSEhK0hBmwwYnDMnYgYwuz+Rhij8Fzgn1s4okhkzEKkzmJHTteAo6NbbCNk4CB47EcK4EYB+I4GEtCIJAEaKEltdZW73ttd/6o6lapVdVVXV2vXi33cw6HqlfvVf36qerd99vuT1QVY4wxZjoetwtgjDGm8FmwMMYYk5YFC2OMMWlZsDDGGJOWBQtjjDFpWbAwxhiTlgULY4wxaVmwMMYYk5YFC2OMMWn53C5ArsyZM0eXLVvmdjGMMaaobN++/ZSqtqbbr2SCxbJly9i2bZvbxTDGmKIiIgcz2c+aoYwxxqRlwcIYY0xaFiyMMcakVTJ9FsaY3AqFQnR2djI2NuZ2UUwOVFZWsnjxYvx+f1bHW7AwxiTV2dlJXV0dy5YtQ0TcLo6ZBVWlu7ubzs5Oli9fntV7WDOUMSapsbExWlpaLFCUABGhpaVlVrVECxbGmJQsUJSO2f5bWjOUMcY4aH/X0Kzf45zW2hyUZHasZmGMKUjd3d2sXbuWtWvXMn/+fBYtWsTatWtpbGxk5cqVOf+8559/ng984AMzOuaqq65KOhn4+9//PnffffcZ2zoPHeTyi95JNBqd3DYWivD+qy7j1ZdTTyh++vEfnfVebrCahTGmILW0tPDKK68AsGnTJmpra/nc5z5HR0dHRhf1cDiMz+f+JW6iVnBO6yqWL23j6Bs7uPLKKwF49j+3MzI8zAevuSrl8XPrKjmUj4KmYTULY0zRiUQifOITn2DVqlVcffXVjI6OArE7/T/90z/lyiuv5Otf/zpdXV186EMfYv369axfv55f//rXALzwwguTtZZ169YxODgIwNDQEBs3buT888/ntttuQ1UBeO6551i3bh2rV6/mYx/7GOPj42eV6dFHH2XFihVceeWVk58z1S233MLjjz8++fyffvI0v//BjQD87Gc/49JLL2XdunW8733v48SJE2cdf8cdd/DUU09NPq+tPd089dd//desX7+eNWvWcP/998/ofGbC/bBrjCl8n/kMxO/yc2btWvja17I6dO/evfzjP/4j3/nOd/jwhz/M008/zUc+8hEA+vr6eOGFFwC49dZb+exnP8sVV1zBoUOHuOaaa9izZw9f/vKXeeihh7j88ssZGhqisrISgB07drBr1y4WLlzI5Zdfzq9//Wva29u54447eO6551ixYgUf/ehH+bu/+zs+85nPTJbn2LFj3H///Wzfvp2Ghgbe+973sm7durPK/eEPf5h169bxt3/7t/h8Prb89Gn+9ruPAXDFFVfwm9/8BhHhu9/9Ll/60pf4yle+ktH5ePbZZ9m7dy+//e1vUVWuv/56/v3f/533vOc9WZ3fZCxYGGOKzvLly1m7di0Al1xyCR0dHZOv3XTTTZOPf/GLX7B79+7J5wMDAwwODnL55Zdzzz33cNttt/HBD36QxYsXA7Bhw4bJx2vXrqWjo4O6ujqWL1/OihUrALj99tt56KGHzggWL730EldddRWtra2TZXjrrbfOKvf8+fNZtWoVzz33HPPmzcPn87Piglj/S2dnJzfddBPHjh0jGAzOaD7Es88+y7PPPjsZoIaGhti7d68FC2NMnmVZA3BKIBCYfOz1eieboQBqamomH0ejUV588UWqqqrOOP7ee+/l/e9/P1u2bOFd73oXv/jFL5K+bzgcnmyKSifToakTTVHz5s3jA/EmKIBPfepT3HPPPVx//fU8//zzbNq06axjfT7fZAe5qhIMBicf33fffdx1110ZlSEb1mdhjClZV199NQ8++ODk84kO8/3797N69Wo+//nP097ezhtvvJHyPc4//3w6OjrYt28fAI899thkB/WESy+9lOeff57u7m5CoRBPPvlkyvf70Ic+xJYtW3jiiSf4wB+cDhb9/f0sWrQIgB/84AdJj122bBnbt28H4Kc//SmhUAiAa665hkceeYShodgw3SNHjnDy5MmUZciGo8FCRK4VkTdFZJ+I3Jvk9feIyMsiEhaRjQnb14rIiyKyS0R2ishNU481xph0vvGNb7Bt2zbWrFnDypUr+da3vgXA1772NS688EIuuugiqqqquO6661K+R2VlJY8++ig33ngjq1evxuPx8MlPfvKMfRYsWMCmTZu47LLLeN/73sfFF1+c8v0aGxt517vexbx581iydNnk9k2bNnHjjTfy7ne/mzlz5iQ99hOf+AQvvPACGzZs4KWXXpqsRV199dXceuutXHbZZaxevZqNGzdOdtrnimRaxZrxG4t4gbeA3wM6ga3ALaq6O2GfZUA98Dlgs6o+Fd++AlBV3SsiC4HtwAWq2pfq89rb29UWPzImd/bs2cMFF1zgdjFK2sSEvXxNukv2byoi21W1Pd2xTvZZbAD2qeqBeIEeB24AJoOFqnbEX4smHqiqbyU8PioiJ4FWIGWwMMYY4xwnm6EWAYcTnnfGt82IiGwAKoD9OSqXMcaYGXIyWCQbGjCjNi8RWQA8BvyhqkaTvH6niGwTkW1dXV1ZFtMYk4pTzdQm/2b7b+lksOgEliQ8XwwczfRgEakHfg78mar+Jtk+qvqwqraravvE+GZjTG5UVlbS3d1tAaMETKxnMTH5MBtO9llsBc4TkeXAEeBm4NZMDhSRCuAnwA9VNfUYNGOMYxYvXkxnZydWa3dO12AsbUjwVCDNnrM3sVJethwLFqoaFpG7gWcAL/CIqu4SkQeAbaq6WUTWEwsKTcDvi8ifq+oq4MPAe4AWEbkj/pZ3qGqO8w0YY1Lx+/1Zr6pmMrPp2y8C8MRda10uSXqOzuBW1S3AlinbvpDweCux5qmpx/0I+JGTZTPGGJM5m8FtjDEmLQsWxhhj0rJgYYwxJi0LFsYYY9KyYGGMMSYtCxbGGGPSsmBhjDEmLQsWxhhj0rJgYYwxJi0LFsYYY9KyYGGMMSYtCxbGGGPSsmBhjDEmLQsWxhjjomJZWsqChTHGmLQsWBhjjEnLgoUxxpi0LFgYY4xJy4KFMcaYtCxYGGOMScuChTHGmLQsWBhjjEnL0WAhIteKyJsisk9E7k3y+ntE5GURCYvIximv3S4ie+P/3e5kOY0xxkzPsWAhIl7gIeA6YCVwi4isnLLbIeAO4B+mHNsM3A9cCmwA7heRJqfKaowxZnpO1iw2APtU9YCqBoHHgRsSd1DVDlXdCUSnHHsN8K+q2qOqvcC/Atc6WFZjjHGFuF2ADDkZLBYBhxOed8a3OX2sMcaYHHMyWCQLmJnmzMroWBG5U0S2ici2rq6uGRXOGGNM5pwMFp3AkoTni4GjuTxWVR9W1XZVbW9tbc26oMYYY6bnZLDYCpwnIstFpAK4Gdic4bHPAFeLSFO8Y/vq+DZjjDEucCxYqGoYuJvYRX4P8GNV3SUiD4jI9QAisl5EOoEbgW+LyK74sT3A/yEWcLYCD8S3GWOMcYHPyTdX1S3AlinbvpDweCuxJqZkxz4CPOJk+YwxxmTGZnAbY4xJy4KFMcaYtCxYGGOMScuChTHGuCjTyWdus2BhjDEmLQsWxhjjJi2OuoUFC2OMcVFxhAoLFsYYYzJgwcIYY1xUJK1QFiyMMcZNRRIrLFgYY4xbFNAiqVpYsDDGGJeoqjVDGWOMmV5UIVok0cKChTHGuCSqSrQ4YoUFC2OMcUs0qlazMMYYM71wVAkXSdXCgoUxxrhAVYlEFVVlaDzsdnHSsmBhjDEuGBwPTzZB9QwFXS5NehYsjDHGBV2D46cfD425WJLMOLoGtyldN337xVm/xxN3XZaDkhhTnE4MnA4QvcMhxkIRKv1eF0s0PatZmJzbfWyA3UcH3C6GMQVrYCzEyHjkjG0nB8ZT7F0YrGZhsjJdreCmb79IVNVqDsakcLz/7Gano/2jtLVUu1CazDhasxCRa0XkTRHZJyL3Jnk9ICJPxF9/SUSWxbf7ReQHIvKaiOwRkfucLKcxxuSLqiYNFkNjYYYLeFSUY8FCRLzAQ8B1wErgFhFZOWW3jwO9qnou8FXgi/HtNwIBVV0NXALcNRFITHEoknlGxuRd30iIYDia9LXEfoxC42Qz1AZgn6oeABCRx4EbgN0J+9wAbIo/fgp4UESEWDLGGhHxAVVAELBG8CJSLLNSTe6lG/wwHo4Q8E3fkVvKTZjdw6n7JnqGg7yjNY+FmQEnm6EWAYcTnnfGtyXdR1XDQD/QQixwDAPHgEPAl1W1x8GymhwrkkmpJs92HxvgzeODbhfDVX0joZSv9Y+GiBboj8fJmoUk2Tb1LKTaZwMQARYCTcCvROQXE7WUyYNF7gTuBGhra5t1gU1uxJKjFeYX3jgv3eCH3pEgj9/5LmKNCOVncJp+CVUYCoapr/TnsUSZcbJm0QksSXi+GDiaap94k1MD0APcCvyLqoZU9STwa6B96geo6sOq2q6q7a2tBVp3K0ORaCyNgTHJCJRtoAhFokQi0/82xkKRaV93i5PBYitwnogsF5EK4GZg85R9NgO3xx9vBH6psWWjDgG/IzE1wLuANxwsq8mhUEQJR6KEI8k78UyZK9NAATCeomP7jH1Chfm7cSxYxPsg7gaeAfYAP1bVXSLygIhcH9/te0CLiOwD7gEmhtc+BNQCrxMLOo+q6k6nympyJxJVgpEoCpwYLOxJRsYd5TwTOJRBsCjULLSOTspT1S3AlinbvpDweIzYMNmpxw0l224K36Gekck1hTtODTO/vhKvp3zvJM3ZyrUJCmLNUOmkGlbrtnIO8ibHhsfDdJwannw+GoxwoGvIxRKZQlTGsYKRYPr+iJFgYU7Ms2BhciIUifJqZ99ZHdsHu0cKeqKRyT9PGUeLgbHUw2YnDI6FJ2vnhcRyQ2XIsqymFo5EeeVw31mJ0SbsOtqP1yPMqQ3kuWSmEJVrrAhFopwaSt+PFwxH6RkO0lJgvxerWeTI7mMD7D5WfpPMQ5EoOw730T/NRKNoFHZ29p2Rv9+Ur3KtWRzqGSGaYXdER/dw+p3yzGoWGUpXK5ioeZRq7SGZYDjKy4d6GRpL38Y6ETAuXNTAvPrKPJTOFKpyjBVD42EOziAA9A6HONI3yqLGKgdLNTNWszBZGQ9H2H4ws0AxQRVeP9KfNOOmKR/lFivGQhFeOdSXca1iwpvHB+gZLpzlVi1YmBmLRJWdnf1ZpVNWjfVhFNKPwORXOQ2dHQtFePlgb1azsqNRePVwH70F8luxYGFm7O1Tw9P2UaQzUcOwlCDlqVxCxcBYiK0dPRkNl00lElV2HO7lWP9oDkuWHQsWZkaiUeVwz8is3ycYjhbED8DkXzlULE4OjLG9ozcnqTuiUdh1ZIB9J4dcHVJrHdxmRoKRaM5qBKOzuOMyphCpKvu7hs+YnJorHaeGGRwLceGiBvze/N/nW83CzEjA58Hnzc2tYU3A7lXKkZRoQ9TEfCMnAsWE7qEgW9/ucWWWtwULMyMiQlvz7BeVr/R7bQhtuSrBWDExOrB7yPnO6JFghK0dvfSPZt9vmI2MgoWInCMigfjjq0Tk0yLS6GzRTKFa1lJDfVX2i7OIwMqF9ZZgsEyV2r96OBJlx6E+BmcwjHy2QhNznLIYkZitTGsWTwMRETmXWFrx5cA/OFYqU9A8HuGiJQ1UVUy/jnIq5y+op7mmIselMsYdu48NzGi+Ua5EIsrOw30ZZbLNhUyDRTS+PsV/A76mqp8FFjhXLFPoAj4vF7c1EfDPrCVzxby6gpqVasxs9I+EODngXhqbkWCEI735GVWY6S89JCK3EFvV7p/i2wpvkViTV1UVsYDh92X2NXpHaw1tLbPv7zCmUBwbcH/497E8ZUTINFj8IXAZ8H9V9W0RWQ78yLlimWJRE/CxdkkjnjTfpEVNVbyjtTY/hTImT8Jp1tPOSxlmmkckSxmNXVTV3cCnAUSkCahT1b9ysmCmeDRU+blgQT27jiTPuttY7eed8+ryXCpjnOfGfAe3ypBRsBCR54Hr4/u/AnSJyAuqeo+DZTNFZEFDFScHxs9KQ+7xxEY+eUp45JOtdVK+WusCOcloMNsy5EOmIalBVQeADwKPquolwPucK5YpRufOPbuZaVFjNdUV5T35rlzXOikHTdV+qrMcFZgLIrCwIT8DRjL9FftEZAHwYeB/OVgeU8RqAj6aa88cEru4qfRHPtlaJ+VLRFg6p4Y9R925GZjfUJn1EPaZyrRm8QDwDLBfVbeKyDuAvc4VyxSr1oSlIKsrvJbSw5S8BfWVVPrdqV0sa6nJ22dl2sH9JPBkwvMDwIecKpQpXnWVvoTHNrralD6PJ5YC560Tg3n93Ln1gbzejGWa7mOxiPxERE6KyAkReVpEFmdw3LUi8qaI7BORe5O8HhCRJ+KvvyQiyxJeWyMiL4rILhF5TUQskVARSLzDqpzhhD1jitX8hsq8p15fkKe+igmZ/pofBTYDC4FFwM/i21ISES/wEHAdsBK4RURWTtnt40Cvqp4LfBX4YvxYH7F5HJ9U1VXAVUB+s2aZrCTme/IVwLBCY/KhwuehsTp/NWmvR2jJc8qcTH/Nrar6qKqG4/99H2hNc8wGYJ+qHlDVIPA4cMOUfW4AfhB//BTwuxJbc/FqYKeqvgqgqt2qaosfFIHSHSBrzPTy2SRU6ffmfTh6psHilIh8RES88f8+AnSnOWYRcDjheWd8W9J94rmn+oEWYAWgIvKMiLwsIn+S7ANE5E4R2SYi27q6ujL8U4yTIgkredmyqaacBHz56+SeaU62XMj0Ez9GbNjsceAYsJFYCpDpJAt7U68eqfbxAVcAt8X//99E5HfP2lH1YVVtV9X21tZ0FR2TD6GE9Af5yoZpTCHI581R1IUbsYyChaoeUtXrVbVVVeeq6h8Qm6A3nU5gScLzxcDRVPvE+ykagJ749hdU9ZSqjgBbgIszKatb7B46ZiwUSfrYmFIXDOfv5iifnzVhNnWZdKk+tgLnichyEakAbibWSZ5oM7FMthCrrfxSYyuSPwOsEZHqeBC5Etg9i7KaPBkZPx0gRmyNbVNG8rnU6WgokvfaxWx6ZKbtXVHVsIjcTezC7wUeUdVdIvIAsE1VNxNbSOkxEdlHrEZxc/zYXhH5G2IBR4EtqvrzWZTVeYr17gIDY6cHrY0GIwTDUSoyTGFuTDEbzuPNkSqMhCLU5rFTfTaflDasqeoWYk1Iidu+kPB4DLgxxbE/wtKgFxVVpWf4zDWIe0eCtta2KXnj4QihPDcNDY+HCydYiMggyYOCAKWf9GcmrFZB/2jorLbUkwPjFixMyRsL5r8PId99gtMGC1W1RQhMxo70nb1qWNfQGMFwnTVFmZIWdGHkX747ue0XnCPlXrEYD0c4MXD28o7RaPIgUk5spFzpy3eqDzc+04KFyYmOUyOkWt3xYPdwec+5sGhxhlI8Hf506wo7wJfnz7RgYWZtaDxMZ2/q1cLCEWV/11AeS2QKWglGi5pA/lOU11bmN/2/BQszK6rKnmMDaJoLQGfPKH0jwel3MmVBSzBa+LyevF68RaChKr9LAFiwMLPy9qlh+kcySwi86+hAeTZHlXuHVpnI56i/ppoK/HnO6mzBwmSte2icA13DGe8/GozEayGld2dpMleq//wL8rimxaLG/M9csGBhsjI8Hua1I/0zPu7kwDj7ZxBgSkKJXhyzVao3C5V+b15qF5V+L3PrAul3zDELFmbGxkIRdhzqIxzJ7kffcWqYwz2pO8RLTSm20c9GKTdELm2pzstniAtjdS1YmBkZC0V4+VDvrGePvnl8kKNlPv+iXJVqzQJi687PrXfurj/g97jSBAUWLHKmdL/+p42FIrx8sPeMzLKzsfvowLRDbk1pSjUfp1S8o7XWsfdePqcm7yvkTcjvQF1TtEaCYV4+2JfzfDRvHBskElWWttTk9H1N4YqUcM0CoDbgY35DJcf7YxkNvvTMGyn3nWiOnW4fgD+55nyqKrwsbHAvJZ/VLExag2MhtnXMvukplb0nhth30ibtlYtyWG53+ZyanI+MWuZirQKsZpE7qu4kiHFY/2iIHYd6s+7MzlTHqWGiqqyYV3q5K8UmWkxSIKpKNKquXvicVhPwMbeukhMDY/zJNeen3G+iRjHdPhAbAbXA5ezNVrMwKQ2O5SdQTDjUPcK+k4N5+ay8Kt1r4oxNdG6Pu7AsaL4tnZO7kVFtzdWuB1cLFiap8XCEVw/35y1QTOg4NVJyWWotVpw2sRToeLj0l9ytr/TTWD37lBxer7Cw0f01YSxY5ExpXRLeOj6U98VVTn/2IKO2fndJmuiuGHXpu5Vvi5tmX7uYX1+JL8+pPZJxvwSlooRixcBYKOnaFPkSiSoHTlmHdymaGAlVLjcDrXUBvLNsPlrQ4H6tAixYmCROuhgoTpdhvGQmb5XGX5Eb0YlgUSY1C69HmFOb/SS9Cp8n79llU7FgYc4y6sJ6wlNFourKUpVOKJWglwsTE/LKpWYB0Fxbkf2xNRWupPZIxoKFOYvf5/6XU8Sd1cecUAbTCjI20Qw1UkbBomkWndxNNdkHmlxzdJ6FiFwLfB3wAt9V1b+a8noA+CFwCdAN3KSqHQmvtwG7gU2q+mUny2pOa66uoLNn+hFJuZqVmkpjtd/1oYK5ElVFS6lTK0vBcHSylhUMRwlHogXRceu0Kr8Xr0eymoxYGyicqXCO/UuJiBd4CLgOWAncIiIrp+z2caBXVc8Fvgp8ccrrXwX+2akymuTm1Aaorsj/MpGJ2ppLJ/2HqlpTFGc3PZVLv4WIUJXl78nt32EiJ8PWBmCfqh4AEJHHgRuI1RQm3ABsij9+CnhQRERVVUT+ADgAlNniB+7zeIQLFtTz8qHelAvV5GJWairz6itpdSFfv1OiWroL/szESCh85vNghLrKwui8dVqFb+b35R4PeV8NbzpOlmQRcDjheWd8W9J9VDUM9AMtIlIDfB74cwfLZ6bRVFPhSuqNukofFywovZQfFitgeEq24nLqt/Bl0aTqLbA+OydLk+zsTP3NpNrnz4Gvquq0g+1F5E4R2SYi27q6urIspkllSXM158x1Lt3yVLWVPta2NZZcO7aqjYiC2OqK0z03hc3JZqhOYEnC88XA0RT7dIqID2gAeoBLgY0i8iWgEYiKyJiqPph4sKo+DDwM0N7ebr9GByyfU4PPI7x53NmcTY3Vfi5a0lhQ1e7cUUpq1maWpgaHwbHyCRbZdG5HC2wYnZPBYitwnogsB44ANwO3TtlnM3A78CKwEfilxm7B3j2xg4hsAoamBgqTP0uaqwn4Pew6MuBIeum59QEuXNhQMqOfplLrsyAUiZ7V7DQSDJfNiKhwFr+bSLSwsvM69q8U74O4G3gG2AP8WFV3icgDInJ9fLfvEeuj2AfcA9zrVHkcV+IXg7l1lVy8tCmrjrrpLG2pZvWi0g0UEJtbUOoL/qTTNxI6a5tqLAV+Och2EuJYASVcdHQQr6puAbZM2faFhMdjwI1p3mOTI4UzM9ZQ5eeSpU28fKiX8dDsZ1efO7eWZXNKZ4hsMtH43WFUyjtYdA+Pp9gepGUW6TCKQTgSJZhlSvbh8QjVFYUx16L0638mp2oCPi5Z2oR/ljWMc8ogUAAMB8Mo2TVDlApV5eRA8mBRSjnAUplN38zgWOHUvCxYmBmrrvCxelFD1sfPrQ+wvAwCBZweHhpVLYvlRJM5NRRMeWc9ForQMxzMc4nyq28WTW2zOTbXLFiYrDTXVDA/i9TJHg8luXRqKolrgpTDgj/JHOqZfl7twXh6mFLVPZS8VpWJvpFgwdxkWLAwWcsmz35LTYBKf+GkMHBa4h11tu3WxaxnOEjv8PR3xz1DQfpGSrN2MRaKJO3cz1Q0Cl2D2QebXLJgYbKWTaqG2srC6KzLl/EyDhbRqGY8P+fN44Ml2XeRiyWCC2WZYQsWOVN6X/R0shnt6imQ3Pz5ktj0NJaDEWTF5O3u4YxnaQ+OhenoLq3mqFAkOpmBeTZ6hwuj5mXBIkfKL1SQ1eJE5XZ3nTjEuJz6LPpHQnScmlkO0ANdQyU17+Jg9zDhSG6uDHtPDrle87JgYbI2NTFcJkaC5ZPiASCU0DlZKiv/pRMMR3ntSP+MZ62rwutH+gmVwHnqHw1xMIc1pf6REIfTrDHjNAsWOVKONYtsZqWW03KacGZ+n2jxXwPTUlVeO9J/xiiwmRgNRnj9SL/rd9GzEY5E2XV05sEynX1dg67Ou7BgkSOFlvQrH4KRmV8QxsusGUrL7DZi78khemc5b6J7KMj+rmkTThcsVWXX0QFGsqh1pxONws5O92peFixypJxn6M5EuV08E5X63360b5RDOWp66Tg1wvH+sZy8Vz7t7xp2dKjraDDCzs5+V25OLVjkSKjM7pgBKrwzny8R8JXPHAtVPaPpqZTvJ/pHQ7xxfCCn77nn2EBBpbtI52jf6Iw79bPROxzkDYeXDEjGgkUOjIcjRFTLrnZRXzXzORP1ZbKMJpxd2wyXQMdtMqFIlNeP9Oe8TyYSVV7r7C+K89Y/kvtgOZ1c1uIyZcEiByYShRXKtPx8qa/0zzih4Jy6CodKU3hGp3TyTn1eKt48PujYwIWRYIS3ThR2/8V4OMLOI315H8Cw9+TgrPuHZsKCRQ5MdDhFi3gERzY8HplRyg+vV5hbN/MUIcWqf0qah/FQNOtRQoWqa3Dc8b6Fo32js8qv5LS9J4ZykrJ/plRjTXX56r+wYJEDvvjC6iW8fk9KS5qqM953cWMV3jI6SccHzr6IHivCTttUolFl74n8tJ2/dcL9SWnJ9I+EXO2IHwlGONybn+YoCxY5UBuItd2XWyoLgKoKL/Pq09cWRGLLs5aLk4NjZ9UsIDart1Rmch8fGDtrqVSnDI+HOZFiTQw3nRh0P/jn67xYsMiBqgovIlJWd82J2lrSB4F59ZVlk212JBhmz7Hkd9zhiPL6kfw1HTgpF3mPZqIzT3fQMzFQAOlJBkZDeal1WbDIEa9IWdakNHX7AAATQUlEQVQsILbcamP19KOclmYQUErBaDDCjkN90w6l7h0O8vpRd8bK58rweHhWK8Blo28kVHAZAPxe9y+hPq8gebj2uP+XlghPmZ/JRU1VKV+rr/Jnlc682AyMhdja0ZPRBe3kwDg7DvcVbR6k7iF3sqCeKrCO7qZq90f35asMZX6JM7nSWhtIGTDn1QfyWxgXdA2Os72jd0ZZdXuHg2zr6C3KEVJ9o+4Ei0LLSruwsRKf190WhWUt+Vmi2IJFjhTgQI288nk9KWsPTTXu33056XDPCDs7+7KaZzM8Hua3b/cU3EUwnaE8N0FNyHfTVzo+r4dVC7Nfj362ls2ppiFNE3CuOBosRORaEXlTRPaJyL1JXg+IyBPx118SkWXx7b8nIttF5LX4/3/HyXLmQrnNsUimpiL5jO7aFNtLwb6Tg/FV3rJ/j2A4yssHewt6LkGiaFRdm2A4GgoX3BDa1roA586tzfvnzqkLcE5r/j7XsWAhIl7gIeA6YCVwi4isnLLbx4FeVT0X+Crwxfj2U8Dvq+pq4HbgMafKmSuq5ZmmPFGF7+zquNcjeEp0lFjHqWE6TuVmhE4kqrza2cdAEeRCGg1FXKtJR6OFueLgsjk1nJPHgNFaF2DNooa8dGxPcPKWbwOwT1UPAIjI48ANwO6EfW4ANsUfPwU8KCKiqjsS9tkFVIpIQFUL9tZLoezbopK1wpRqjUtVeTvHSeOi0VgAWrO4Mafvm2vpahVfeuaNlK9NDLedbh+AP7nm/Gk/v6qi8IZhL59Tg88jvHVidjXNdBY0VnLB/Pq834Q52Qy1CDic8Lwzvi3pPqoaBvqBlin7fAjYUciBAgCrWSTtqFVNvr0UVMwwL1YmiiErr9vDVws5x9aS5mrWLG7E61Cn9zlza1m1sMGV2rqTNYtkf83U6+m0+4jIKmJNU1cn/QCRO4E7Adra2rIrpcmZVJ2eQ+PhkpuQJyKcP7+OVw735ewustLvZfmc/IxsmY10s7anqxVM1Cim2yft548XVif3VK11AdqXNvHK4b6c5YzyeGDlggbmzyAXW645GSw6gSUJzxcDR1Ps0ykiPqAB6AEQkcXAT4CPqur+ZB+gqg8DDwO0t7e7e2Nfms3yGRsLRVJeRPpGQsypLb3hsy21Ad7RWstdj22bdr9Mml4+f+35rF7c4EhtJdeGXV5HfbjAJuYlU1fpZ/2yZl493DfrEVx+n4eLFjfQ6PKcDie/mVuB80RkuYhUADcDm6fss5lYBzbARuCXqqoi0gj8HLhPVX/tYBlNjvSOpB5335PHNMr5NpOsu9OprvDRUFUcExeHXb6zL/SaxYRKv5f2Zc0012Z/ka+q8LJ+WZPrgQIcrFmoalhE7gaeAbzAI6q6S0QeALap6mbge8BjIrKPWI3i5vjhdwPnAv9bRP53fNvVqnrSqfLOWol25GZquoAwOBYiFIkWRGqEXIpGlQNdw2mbVDJpehkeD3NyYIy5GSRldFM4EnUlHXei0VCEaFSLYpSd1yOsXdzIrqMDnEiShXg6NQEf69oaC6YJ19EB8Kq6BdgyZdsXEh6PATcmOe4vgL9wsmy5NB6OoJT2spnpJMuwOkE1luyspUSaolSVU0NB9p0cyuld9s7OfubVj7O8tWYyk3GhGSmAzmXVWDkK9RxN5fEIFy6qR1FOZpghtjrg5eKljQU14KE4znaBmxgdUsyJ4WYjGtW0nZ7D4xFa8j9vKafCkSjH+sc43DPiWGruEwNjnBgYo7m2grbmalpqKvI6lj6dsQLpLxgromABsQERFy5sYEekL+3qdgG/h4vbmgoqUIAFi5yYGMoXKdOmqGAGyfAy2adQRaNKR/cwh3pGCEfy82/cMxSkZyhIdcDLuXNrC2aFwUL5dyzGBIwej7B6UQO/fbsn5XByEVizqHCanhKVViOyS0Lh2AWkTGMFBXTj64gdh3s50DWct0CRaGQ8ws7D/Rzszu0EwGwVyne8WCvxFT4PFy6qT/n6Oa21ecv1NFMWLHKg1C+W6VR4PWlTtAeKYEhoMqpaEOklCqEMgOsZVif4iqBzO5XG6goWNJ5dU6wOeGkr4NUki/MXXGAC/vJdgxti7bHVaZIFFlP7ciIRYfXiBvwuBrumGj/ntBbGZL1qf2H8O1YXYLqPmTintfasG6xz59YW9AivwviXL3ITF8pyXVYVoLHan3IGt8cTWwCpWNVX+tmwrJnXj/ZPO+rLCUuaqzmvgC4idZU+vF4h4kKT3AS/z1NUNx83ffvFpNuHg+HJCZtfefbNaX8jT9x1mSNlm4niOeMFrCreGVWuy6oCNNdU0NkzmvS1hqqKog+kVRVeLmlrorN3lAOnhhzvv6gJ+Hjn/DqaC2wtEI9HmFdXydG+5P/W+TCvPlBQI8SyVVPhK/ikkYksWOSA1xNbA7ecg0VTdQUiyTtAC+2Cly2PR2hrqWZBY+xiebhnNOdJEhur/bS1VNNaW7gXxKUt1RzrH3Wls9vjgaXNhdEkl6lCqBXkggWLHPFQ3h3dfq+HmoAvaVNUU4GO7siW3+thaUsNbc3VdA8HOdo3yqmhcaJZ9kH7fR4WNFSysLGqKJpXagI+2pqrOdidm7U8ZqKtuaYg05OXg8L/ZpqiUV+ZvN8i1XKrxU5EmFMbYE5tgPFwhKN9Y3T2jmScDqOh2s+Spmrm1gUKpk8iU+e01tI9HMzr8qr1VX7eUQRZeUuVBYscUWw9i2R3fAG/p+j7KzIR8MXSi7c1V9PZO8L+rqGUNY3qCi8r5tcVdSZej0e4aHEjv+3oIRR2flhvhc/DmsXurONgYmzobA6EI1GiqmWb7mOCP8kYfF+6CRglxusRlrbUcElbc9IgWVfpo31Zc1EHiglVFV7WLml0/GbA65WCSqhXrsrrl+yQoXgyuUiZB4tkHbLl2o/TUO1nSXPVWdvfOb+uKNasyFRDlZ+LHAwYXo+wbkljyTZlFhPRQpm/P0vtdXW67ZJLXPns0VCEt04M4pFY7pdyNRaKMhoKcyg+drytuRqPSNGs05Br4agyOBaaPB9Lm2toqPaX5DpZoYjGb5qmv54kfjfSE2oDvqQ1VpM78sIL21W1Pd1+pXOL46JwtLxzQ03QJBeKcq5sTb3b9nqlJAMFxJogYyO5cvUXCnWVFigKSel0cL/znfD883n/2GhU2fZWF3/5z3sA+H9/dHna1Bel6sDRfo71jZ212M9V72zFV2ILH2VCgI5DvXz+6Z0AfO/29dQVcO6f2fIDMhLko9/7bcp99ncNAbHRVNN57L9fiq9Ma6R5l2FbcXle1XKodyR4Rl9F91CQ6ubyPK3jKUbFjIejJR0sUqVzgFgq7VhKB+GPn3w15X13qUzcaqyumHYexIUZNtOWa9NlISvPq1oWUl0QRoJhxsPRyRwvdz22nbrK5Ke1VC4IqaSaXzAejlJT/IN/suL3eli1sAGR3DXQFLpS/56XKwsWsxCJKsEpd9PhaJRgJEpFCd9JpxJOMbEgXIQL1cyEXRxNObBgkaGpF4STA2PsOjYwmX0zsZ3e44Hz5taxpITbp5NJ1Zldxn3cxpQMCxYzEI5EOTE4zrG+UfqmSVUdjcKbxwfp7B1lUWMV8xsqS2psfSpeEZKdlXKda2FMKbFgkcbQeJje4SA98f9mMvFueDzMWycG2XtykKaaCpqrK2iqqaC+0lewGUVnI+D3JM3CWmgLzxtjZs7RYCEi1wJfB7zAd1X1r6a8HgB+CFwCdAM3qWpH/LX7gI8DEeDTqvqMk2WdMB6O0DMcpHsoFhym9klkQxV6hoL0DAWB2NKUTdUVNNdUMKc2UDJZNKsrvEkXB6opkb/PmHLmWLAQES/wEPB7QCewVUQ2q+ruhN0+DvSq6rkicjPwReAmEVkJ3AysAhYCvxCRFaqa28UDEhzqHuH4wBgDo86vhBaOKF2D43QNjvMmg1QHvMytC3BOa23R1DiSjQ4LhqNnrP71pWfewOsRvvOrA0nfwzqGjSkeTtYsNgD7VPUAgIg8DtwAJAaLG4BN8cdPAQ9K7Gp5A/C4qo4Db4vIvvj7pR7QPgujwQj7u4Zcy+00Mh7hYHCE5ppAUS8UVOHz4Pf6aUwcS18kwc8YMz0ng8Ui4HDC807g0lT7qGpYRPqBlvj230w5dpFTBa2q8PKeFa10D41zcnCc3pHgWSN7/nLLnmnf43B8SdEvP/tmyn3u+68XnLWtscpPa11sTYRi6gS3WoEx5cXJYJHslnLqrXuqfTI5FhG5E7gToK2tbablO4PXI8ytr2RufWXS17/5b/umPX7N4vQzU69c0ZpV2Ywxxm1OBotOYEnC88XA0RT7dIqID2gAejI8FlV9GHgYoL293dE2JLuTNsaUMyfbPbYC54nIchGpINZhvXnKPpuB2+OPNwK/1FjO9M3AzSISEJHlwHlA6uxkxhhjHOVYzSLeB3E38AyxobOPqOouEXkA2Kaqm4HvAY/FO7B7iAUU4vv9mFhneBj4IydHQhljjJle6Sx+1N6u27Ztc7sYxhhTVETEFj8yxhiTGxYsjDHGpGXBwhhjTFoWLIwxxqRlwcIYY0xaJTMaSkS6gIMuF2MOcMrlMhQKOxdnsvNxmp2LM7l9Ppaqatr0EiUTLAqBiGzLZAhaObBzcSY7H6fZuThTsZwPa4YyxhiTlgULY4wxaVmwyK2H3S5AAbFzcSY7H6fZuThTUZwP67MwxhiTltUsjDHGpGXBAhARFZGvJDz/nIhsynMZvi8iG/P5mbMhIhEReUVEdonIqyJyj4iU5fcp4Vy8LiI/E5HG+Pa1IvJi/BztFJGbEo55XkQKfgTMTInIMhF5fcq2TfHf1F+LyBvxc/GThPN0lYj0i8iO+Otfdqf0szflu/CkiFTP8Pghp8o2W2X5405iHPigiMzJ5uD4wk3lZlRV16rqKuD3gP8K3O9ymdwycS4uJJZq/4/i20eAj8bP0bXA1yYukGXqX4ELVXUN8BZwX8Jrv1LVdcA64AMicrkbBcyBxO9CEPhk4osSU5TX3aIstAPCxDqZPjv1BRFZKiLPxe+GnhORtvj274vI34jIvwFfjN89/UBEnhWRDhH5oIh8SUReE5F/ERF//LgviMjW+J3HwyKSbAnZoqKqJ4ktb3t3/MdQKSKPxv/2HSLyXgARWSUiv43fee0UkfPcLbkjXiS+XryqvqWqe+OPjwIngbJdW1dVn1XVcPzpb4itgDl1n1HgFeLnsMj9Cjg3XtvaIyLfBF4GlojILfHfx+si8sXEg0TkKyLycvx60xrfdk78OrJdRH4lIufn+4+xYHHaQ8BtIjJ1Me0HgR/G74b+HvhGwmsrgPep6h/Hn58DvB+4AfgR8G+quhoYjW8HeFBV18fvPKqADzjy1+SZqh4g9n2aS/zOOv633wL8QEQqid1lfV1V1wLtxJbPLRki4gV+l7NXhERENgAVwP58l6tAfQz456kbRaSJ2MqY/573EuVQvLXhOuC1+KZ3EruOrANCwBeB3wHWAutF5A/i+9UAL6vqxcALnK6tPwx8SlUvAT4HfDMvf0gCCxZxqjoA/BD49JSXLgP+If74MeCKhNeenLKC3z+raojYF8QL/Et8+2vAsvjj94rISyLyGrEvy6qc/RHum6glXUHsXKGqbxBLw7KC2F33n4rI54mlGBh1pZS5VyUirwDdQDOx5pZJIrKA2Pn4Q1WNulC+fEo1vHJyu4j8L2K1+b9PeP3dIrITOA78k6oed66Ijpr4LmwDDhFbDRTgoKr+Jv54PfC8qnbFa1p/D7wn/loUeCL++EfAFSJSC/wX4Mn4e38bWOD8n3ImCxZn+hrwcWLRPZXEH8PwlNfGAeIXhJCeHpccBXzxu+tvAhvjd93fASpzUXC3icg7gAixppakTWuq+g/A9cRqWs+IyO/kr4SOGo3XlpYSqz1M9FkgIvXAz4E/S7hYlLJuoGnKtmbiuY9E5HZitenbEn4fEOuzWAOsBv6HiKzNR2EdMNFnsVZVP6Wqwfj2xGvFTJqeldh1ui/hfdeq6gU5K3GGLFgkUNUe4MfEAsaE/yS+NjhwG/Afs/iIicBwKn63UDSjn6YTb1f9FrEmNiXWhHBb/LUVQBvwZjygHFDVbxBrqlnjUpEdoar9xGqmnxMRv4hUAD8h1vzwpLulyw9VHQKOicjvAohIM7HO/f8QkWuBzwPXq+pIiuPfAv4yvl+pegm4UkTmxJsubyHW5ASxa/LEdeFW4D/irR5vi8iNMNlJflG+C12Oo3jS+Qpwd8LzTwOPiMj/BLqAP8z2jVW1T0S+Q6xZqgPYOotyum2iuu0n1qTwGPA38de+CXwr3tQWBu5Q1XGJDR39iIiEiDU3POBCuR2lqjtE5FViNxhKrHmhRUTuiO9yh6q+En/88/i5AHhRVW/Mb2kd81HgITk9HP3PVXW/iDwDBIB/jY/r+I2qfjLJ8d8iFnCXq+rb+Sly/qjqMRG5D/g3YrWMLar60/jLw8AqEdkO9AMTw61vA/5ORP6M2G/uceDVfJbbZnAbY4xJy5qhjDHGpGXBwhhjTFoWLIwxxqRlwcIYY0xaFiyMMcakZcHCGGNMWhYsjDHGpGXBwhhjTFr/H7TZ6gbLyFkiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe55a66b358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.ylabel('Loss')\n",
    "plt.xticks(np.arange(0,5), classes)\n",
    "plt.violinplot([test_losses[np.where(y_test==class_)] for class_ in classes],np.arange(0,len(classes)),showmeans =True )\n",
    "plt.axhline(y=threshold,c='r',label=\"Threshold Value\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this approach we attempted to overcome the problems that exists in the KDD99 and NSL-KDD datasets, namely the class imbalance issue and the data being unrealistic, by avoiding the attacks data during training, the model was trained only using normal traffic, so it was not affected by the class imbalance of the dataset, in addition the fact that it only uses normal traffic data for training makes it more valuable in real world applications and more viable for use in real networks.\n",
    "\n",
    "Another strength of this approach is its simplicity, it consist of only a single hidden layer of 8 neurons making it very easy to train and especially suitable for online learning.\n",
    "During evaluation we avoided human manipulation of the threshold in order to achieve reproducible results without human interference, however in real networks when the system is deployed the network administrator can manually adjust the threshold which allows a tradeoff between sensitivity and specificity according to the network requirements which is a huge advantage over other existing approaches.\n",
    "In terms of detection rates our approach outperforms every method in the existing literature that we are aware of.\n",
    "\n",
    "The obvious limitation of our approach is that it can only differentiate between normal and attack traffic, so classifying attacks to the different attack types is not possible, future work can be done to overcome this limitation by building an ensemble of the model alongside others that extend its functionality in order to achieve 5-class classification."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3.5 with Spark 2.1",
   "language": "python",
   "name": "python3-spark21"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}