{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Linear Regression: Jupyter Notebook" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## `pandas.DataFrame`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using `read_csv()` from pandas, read data into dataframe. If your data happens to be in a M$ Excel file, then there is also a `read_excel()` function." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "dataset = pd.read_csv('./history_data.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Relationship Between `pandas.DataFrame` and `numpy.ndarray` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See how a DataFrame holds values using `numpy.ndarray`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([['New York', 'New York', nan, ..., 0.0, nan, 'Clear'],\n", " ['New York', 'New York', nan, ..., 0.0, nan, 'Clear'],\n", " ['New York', 'New York', nan, ..., 0.0, 87.77, 'Clear'],\n", " ...,\n", " ['New York', 'New York', nan, ..., 23.3, 74.96, 'Clear'],\n", " ['New York', 'New York', nan, ..., 14.3, 70.33, 'Clear'],\n", " ['New York', 'New York', nan, ..., 0.0, 84.26, 'Clear']],\n", " dtype=object)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.values" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "numpy.ndarray" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(dataset.values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For convenience, `pandas.DataFrame` provides many attributes from the underlying `numpy.ndarray`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Two dimensional array ..." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.ndim" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "... extending such and such cell in each direction ..." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(72, 16)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.values.shape" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(72, 16)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`DataFrame.describe()` is convenient for interactive use in a Jupyter notebook, just like many other methods." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Resolved AddressMaximum TemperatureMinimum TemperatureTemperatureWind ChillHeat IndexPrecipitationSnow DepthWind SpeedWind GustCloud CoverRelative Humidity
count0.072.00000072.00000072.0000000.022.00000072.0000000.072.00000065.00000072.00000038.000000
meanNaN82.99583370.68888976.287500NaN91.3272730.012222NaN12.69027820.4707693.36944472.513421
stdNaN5.9461065.5740235.313256NaN5.9945360.070695NaN6.0259808.3024237.21482512.665492
minNaN69.90000053.30000061.500000NaN82.4000000.000000NaN2.2000004.7000000.00000046.090000
25%NaN79.05000068.90000074.300000NaN87.0500000.000000NaN9.10000015.0000000.00000066.377500
50%NaN83.75000071.95000076.650000NaN89.7500000.000000NaN12.80000019.7000000.00000072.330000
75%NaN87.95000074.32500079.850000NaN95.9500000.000000NaN15.42500025.3000002.82500081.715000
maxNaN92.90000080.70000085.800000NaN101.6000000.470000NaN38.00000050.60000034.50000096.970000
\n", "
" ], "text/plain": [ " Resolved Address Maximum Temperature Minimum Temperature \\\n", "count 0.0 72.000000 72.000000 \n", "mean NaN 82.995833 70.688889 \n", "std NaN 5.946106 5.574023 \n", "min NaN 69.900000 53.300000 \n", "25% NaN 79.050000 68.900000 \n", "50% NaN 83.750000 71.950000 \n", "75% NaN 87.950000 74.325000 \n", "max NaN 92.900000 80.700000 \n", "\n", " Temperature Wind Chill Heat Index Precipitation Snow Depth \\\n", "count 72.000000 0.0 22.000000 72.000000 0.0 \n", "mean 76.287500 NaN 91.327273 0.012222 NaN \n", "std 5.313256 NaN 5.994536 0.070695 NaN \n", "min 61.500000 NaN 82.400000 0.000000 NaN \n", "25% 74.300000 NaN 87.050000 0.000000 NaN \n", "50% 76.650000 NaN 89.750000 0.000000 NaN \n", "75% 79.850000 NaN 95.950000 0.000000 NaN \n", "max 85.800000 NaN 101.600000 0.470000 NaN \n", "\n", " Wind Speed Wind Gust Cloud Cover Relative Humidity \n", "count 72.000000 65.000000 72.000000 38.000000 \n", "mean 12.690278 20.470769 3.369444 72.513421 \n", "std 6.025980 8.302423 7.214825 12.665492 \n", "min 2.200000 4.700000 0.000000 46.090000 \n", "25% 9.100000 15.000000 0.000000 66.377500 \n", "50% 12.800000 19.700000 0.000000 72.330000 \n", "75% 15.425000 25.300000 2.825000 81.715000 \n", "max 38.000000 50.600000 34.500000 96.970000 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extracting Input and Output Features from a `pandas.DataFrame`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Beware that algorithms expect a two dimensional array as the set of inputs. Using the column header (\"Minimum Temperature\") to index the dataframe gives a list-like type. **Wrong!!**" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "inputfeatures = dataset['Minimum Temperature']" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 53.3\n", "1 58.7\n", "2 60.2\n", "3 66.8\n", "4 68.3\n", " ... \n", "67 70.1\n", "68 72.2\n", "69 72.1\n", "70 75.5\n", "71 78.2\n", "Name: Minimum Temperature, Length: 72, dtype: float64" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "inputfeatures" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "pandas.core.series.Series" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(inputfeatures)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Correction**: index the dataframe *with a list of column headers*" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "inputfeatures = dataset[['Minimum Temperature']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sign of correctness: that one is made up more nicely by the notebook:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum Temperature
053.3
158.7
260.2
366.8
468.3
......
6770.1
6872.2
6972.1
7075.5
7178.2
\n", "

72 rows × 1 columns

\n", "
" ], "text/plain": [ " Minimum Temperature\n", "0 53.3\n", "1 58.7\n", "2 60.2\n", "3 66.8\n", "4 68.3\n", ".. ...\n", "67 70.1\n", "68 72.2\n", "69 72.1\n", "70 75.5\n", "71 78.2\n", "\n", "[72 rows x 1 columns]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "inputfeatures" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Likewise, the output features." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "outputfeatures = dataset[['Maximum Temperature']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting with `matplotlib`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fortunately (well, that was on purpose), our feature sets are one dimensional, so plotting the dataset im two dimensions makes sense. Multidimensional data analysis is not so straightforward - this is why they call it data **science**." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`pandas.DataFrame` interacts nicely with `matplotlib`. " ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAknElEQVR4nO3de3xU9bnv8c8DpBIvNYrIxngUbC0VBQI7Xqmt1jtapOq22qNFrVvb6lE5rW7avbulth5ztvZQbfdLa2sLVYsoCqXqab2waa1abRAUL/VCjUpERQS8RRvIs/+YSZwMM8nMZNas2/f9evHKzFpr1vr9ZsKT3zy/yzJ3R0RE0mNQ2AUQEZHaUuAXEUkZBX4RkZRR4BcRSRkFfhGRlFHgFxFJGQV+qQkzu87Mvht2OUREgV8GyMzazOzvZrZT3vblZuZmNgrA3b/m7t8v89zPmtmnzGxO9lzH5+2fnd1+xkDrkXfed3P+dZlZR87z/1nNa9VK9nM6POxySDQo8Es1vAic2v3EzMYBWw/khGb2CWCwuz+X3fQc8JWc/UOAk4FVA7lOIe6+bfc/4GXgCznbbq729QYq+17E/hpSOwr8Ug03khOUgenAr3IPyLbaf5B9fIiZrTazb5rZG2a2xszOzDvnscDdOc9/C3zGzHbIPj8aeAJ4LecanzCzJWa2zszeNLObzawhZ99bZjYp+3wXM1trZoeUWkkzG2RmM81sVfYat5rZjtl9o7LfPs40s1fMbL2Zfc3M9jWzJ8xsg5n9JOdcZ5jZg2b2EzPbaGZ/NbPDcvZvb2Y3ZN+bdjP7gZkNznvtbDNbB8zqp+43ArsBv81+a7mk+zPIq1/PtwIzm2VmC8zsJjN7GzijrzJJvCjwSzX8Gfi4me2VDQSnADf185p/ALYHGoGvAv+ZE9QBpgB35Tz/APhN9tyQ+UPT648LYMAVwC7AXsD/AGYBuPsq4F+Am8xsa+CXwFx3X1pyLeF/AdOAz2WvsR74z7xj9gf2BL4E/Aj4V+BwYG/gZDP7XN6xq4CdgEuBO7r/kABzgE3AJ4GJwJHA2Xmv/RswAri8n7qfTu9vLv9RYn2PBxYADcDNJZRJYkKBX6qlu9V/BPAM0N7P8Z3AZe7e6e53A+8CYwCygXlfYGnea34FfCXbkv0csCh3p7u/4O73uvuH7r4W+H/Z47r3/wx4AXgEGEkmKJfja8C/uvtqd/+QTGA9KS8N8n13/8Dd7wHeA+a5+xvu3g48QCZgdnsD+FH2PZgPPAsca2YjyPzhu8jd33P3N4DZfPRHD+BVd/+xu29y947+6l6hh919kbt3AR8voUwSE8rbSbXcCPwRGM2WLfFC1rn7ppzn7wPbZh8fBjyUDa493P1PZjacTMC+0907zKxnfzZgXg0cDGxHpmGzPu+6PwMWA+fkn78EuwMLzawrZ9tmMq3ubq/nPO4o8HzbnOft3nuVxJfItNh3B+qANTn1GwS8knNs7uNS616u3GuUUiaJCbX4pSrc/SUynbxTgDsGeLop9M7v57oJ+CaF/7j8H8CBce7+ceA0MikQAMxsWzLplxvI5MV3LHCOvrwCHOPuDTn/hmZb85VotNy/XJk8/KvZ63wI7JRznY+7+945x+Yvq9tn3Qsc/x45HfDZFN3wvGNyX1NKmSQmFPilmr4KfN7d3xvgeY6hd34/1zVk0kl/LLBvOzIpo41m1ghcnLf/aqDV3c/Onv+6Mst1HXC5me0OYGbD84eYlmln4AIzqzOzfyKTm7/b3dcA9wA/NLOPZzuVP5HXP5Cvv7q/DuyR8/w5YKiZHWtmdcC/AVsVO3mFZZKIUuCXqnH3Ve7eOpBzmNk+wLvu/nKRa7zl7vfnpUi6fQ+YBGwkE9h7vnlkA/TRwNezm/43MMnKG5d/NZk00T1m9g6ZTu39y3h9vkfIdAS/SaaD9iR3X5fd9xXgY8DTZFI2C8j0SxRTtO5ZVwD/lh1d9C133wh8A/g5mf6Y94DV9K3cMklEmW7EIlFiZpeQSSdcEnZZgmSZSWdnu/tnwi6LpI86dyVq2siM2ReRgCjwS6S4+61hl0Ek6ZTqERFJGXXuioikTCxSPTvttJOPGjUq7GKIiMTKsmXL3nT3/PkZ8Qj8o0aNorV1QKMERURSx8xeKrRdqR4RkZRR4BcRSRkFfhGRlIlFjr+Qzs5OVq9ezQcffBB2USRBhg4dyq677kpdXV3YRREJTGwD/+rVq9luu+0YNWoUvRc4FKmMu7Nu3TpWr17N6NGjwy6OSGBiG/g/+OADBX2pKjNj2LBhrF27NuyiyAAsWt7Olb9/llc3dLBLQz0XHzWGaRMbwy5WpMQ28AMK+lJ1+p2Kt0XL2/n2HSvp6NwMQPuGDr59x0oABf8c6twVkcS48vfP9gT9bh2dm7ny98+GVKJoUuAfADPjtNNO63m+adMmhg8fznHHHVfR+RYvXkxLS0u1ileWL37xizQ1NfHJT36S7bffnqamJpqamnjooYdCKU9f2tra+PWvfx12MSSCXt3QUdb2tIp1qqccQeT9ttlmG5588kk6Ojqor6/n3nvvpbGx8nNOnTqVqVOnDqhMlVq4cCEAS5cu5aqrruLOO+8MpRzdNm3axJAhhX89uwP/l7/85bLOuXnzZgYPHlyN4klE7dJQT3uBIL9LQ30IpYmuVLT4u/N+7Rs6cD7K+y1aXumtUj8yZcoU7rorc5fAefPmceqpp/bse/TRRznwwAOZOHEiBx10EM8+m/m6OXv2bM466ywAVq5cyT777MP777/PnDlzOP/88wE444wz+PrXv84BBxzAHnvswdKlSznrrLPYa6+9OOOMM3quse22H927e8GCBT37Sn19X9auXcuJJ57Ivvvuy7777suDDz4IwKxZs5g+fToHH3wwu+++O3fccQeXXHIJ48aN4+ijj6azsxPILLXRvX2//fbjhRde6Pe8p59+OpMnT+b000+nra2Ngw8+mEmTJjFp0qSebx8zZ87kgQceoKmpidmzZ/d63wCOO+44li5d2vP+fPOb32TChAk8/PDD3HTTTey33340NTVx7rnnsnlz77SAxNvFR42hvq73H/f6usFcfNSYkEoUTakI/EHm/U455RRuueUWPvjgA5544gn23/+jO/F9+tOf5oEHHmD58uVcdtllfOc73wHgwgsv5IUXXmDhwoWceeaZ/PSnP2Xrrbfe4tzr16/n4YcfZvbs2UydOpUZM2bw1FNPsXLlSlasWNFv2Qb6+gsvvJAZM2bwl7/8hdtvv52zzz67Z9+qVatYsmQJixcv5rTTTuPQQw9l5cqV1NfX9/whBNh+++1ZuXIl559/PhdddFG/53366ae57777mDdvHjvvvDP33nsvjz32GPPnz+eCCy4AoKWlhYMPPpgVK1YwY8aMPuvw3nvvsf/++/P4448zbNgw5s+fz4MPPsiKFSsYPHgwN998c7/vg8THtImNXHHCOBob6jGgsaGeK04Yp47dPKlI9QSZ9xs/fjxtbW3MmzePKVOm9Nq3ceNGpk+fzvPPP4+Z9bSEBw0axJw5cxg/fjznnnsukydPLnjuL3zhC5gZ48aNY8SIEYwbNw6Avffem7a2Npqamvos20Bff9999/H000/3PH/77bd59913ATjmmGOoq6tj3LhxbN68maOPPhqAcePG0dbW1vOa7m9Ap556ak+Q7uu8U6dOpb4+87W8s7OT888/vydIP/fcc32Wt5DBgwdz4oknAnD//fezbNky9t13XwA6OjrYeeedyz6nRNu0iY0K9P1IReAPOu83depUvvWtb7F06VLWrVvXs/273/0uhx56KAsXLqStrY1DDjmkZ9/zzz/Ptttuy6uvvlr0vFtttRWQ+UPR/bj7+aZNm4Deww/zZzGX8vq+dHV18ec//5mhQ4f2Wba6urqecuSfO7d83Y/7Ou8222zT83j27NmMGDGCxx9/nK6uroLHAwwZMoSurq6e57nvw9ChQ3vy+u7O9OnTueKKK/qtuxQX9jj5IK8fdt1qJRWpnqDzfmeddRaXXnppT4u628aNG3s6e+fMmdNr+wUXXMAf//hH1q1bx4IFCyq+9ogRI3jmmWfo6urq6aCtliOPPJIf//jHPc9LSQ/lmz9/fs/PAw88sKzzbty4kZEjRzJo0CBuvPHGnnz8dtttxzvvvNNz3KhRo1ixYgVdXV288sorPProowXPd9hhh7FgwQLeeOMNAN566y1eeqngqrVSRJD9ZWFfP+y61VIqAn/Qeb9dd921J/+c65JLLuHb3/42EydO7NUKnjFjBueddx6f+tSnuOGGG5g5c2ZPMCpXS0sLxx13HAcddBAjR46suA6FXHPNNbS2tjJ+/HjGjh3LddddV/Y51q9fz/jx47n66quZPXt2Wef9xje+wdy5c5kwYQJ//etfe74NjB8/nsGDBzNhwgRmz57N5MmTGT16NGPHjuWCCy5g0qRJBc83duxYfvCDH3DkkUcyfvx4jjjiCNasWVN2ndIs7HHyQV4/7LrVUizuudvc3Oz5N2J55pln2GuvvUIqkZSi+wY6O+20U9hFKYt+t4obPfMuCkUMA15sOTbW1w+7bkEws2Xu3py/PRUtfhGpjmL9YrUaJx/k9cOuWy0p8Etg2traYtfal76FPU4+yOuHXbdaivWoHnfXolpSVXFIfYapu18srJEvQV4/7LrVUmxz/C+++CLbbbcdw4YNU/CXquhej/+dd97RevySCMVy/LFt8e+6666sXr1aa6dLVXXfgUvCVep4+r6OS8uY/ErENvDX1dWpVSaSQKWuqd/XcYDW5e9DbAO/iCRTX+Ppc4N2f+PuSzlHVNT624kCv4hESqlra1WyBlcU1+UP465hGs4pIpFS6nj6vo6L05j8MGYMK/CLSKSUOp6+r+PiNCY/jLuGKdUjIpFS6nj6Uo6Lw6ieMO4aFttx/CIiSZCf44fMt5NqLCSZuHH8IiJJEMaM4UADv5ldCPwzmQXufubuPzKzHYH5wCigDTjZ3dcHWQ4RqUySJ0FFqW61vmtYYJ27ZrYPmaC/HzABOM7MPgnMBO539z2B+7PPRSRiknxjkiTXrRRBjurZC3jE3d93903AH4ATgOOBudlj5gLTAiyDiFQoyTcmSXLdShFkqudJ4HIzGwZ0AFOAVmCEu3ff9ug1YEShF5vZOcA5ALvttluAxRSRQmo1zDCMlEsYQyijJLAWv7s/A/xf4B7gd8AKYHPeMQ4Fb3qDu1/v7s3u3jx8+PCgiikiRdRiElRYKZc4TfAKQqATuNz9Bnf/R3f/LLAeeA543cxGAmR/VnazWREJVC0mQYWVconTBK8gBD2qZ2d3f8PMdiOT3z8AGA1MB1qyP38TZBlEpDy5qZft6+sYWjeIDe93BpKGCSvlEoebrgSZAgt6HP/t2Rx/J3Ceu28wsxbgVjP7KvAScHLAZRCREuVPJtrQ0Ul93WBmf6kpkKAYxqzVbrUeQlmOoBduCzrVc7C7j3X3Ce5+f3bbOnc/zN33dPfD3f2tIMsgIqUrlnqZtfipis+5aHk7k1uWMHrmXUxuWdIrf3/xUWOoG9T7Dnp1gyw1KZdigk6BaZE2EelRLMWyoaOzog7Xkjpv8++cqjupBp4CU+AXkR59pVgqaW3213K98vfP0rm598C+zs2emvH0xQQ96kiBX0R69JViqaS12V/LNe3j6YsJetSRAr+I9Jg2sZEdtq4ruK+S1mZ/Lde0j6cvZtrERq44YRyNDfUY0NhQX5XVOrsp8ItIL5d+Ye+qtTb7a7mmfTx9X6ZNbOTBmZ/nxZZjeXDm56s6AknLMotIL9Uc497fueIwnj6JdCMWEZGE0o1YRKQs5cwcjdLa9tI/BX4R2UI5M0eDnmUq1afOXRHZQrHx99/77ZYzeNO+tn0cKfCLyBaKjaNf//6WM3g1Fj9+FPhFZAvlzODVWPz4UeAXkS2UM4NXY/GL62uBujAp8IvIFqZNbKShvrQZvEHPMo2rKN/QXaN6RKSgWVP37jVaB4q35KO8tn1Y+ur0Dvu9UuAXiZiojImv1qzaqNSn1qLc6a3ALxIhURsTP9CWfNTqU0th3l2sP8rxi0RI0sbEJ60+5Yhyp7da/CIREuX0QCWSVp9yRHkBOgV+iaWk5o2jnB6oRNLqU66odnor1SOxE+VhcgMV5fRAJZJWn6RQ4JfYSXLeOGlj4pNWn6RQqkdiJ+l546imByqVtPokgQK/xE7a88ZJk9T+mihTqkdiR3nj5Ehyf02UKfBL7ChvnBxJ7q+JMqV6JJaUN46+UlI4Se+viSq1+EWk6kpN4Wgt/3Ao8IvEXBTXfC81hRNmf00U37daUapHJMaiughaqSmcsJY1iOr7VisK/CIxFtU138sZchtGf01U37daUeAXialFy9sLBleorHO0UGcsVNYav/ioMSXfxCUMae9UVuAXKUNUJht1pyqKKbdztFDq4+IFj4NDZ5f3bCs1HRLllSlBkwAV+EVKFKW8cKFURbdKWtaFzte52bc4rpx0SJSH3Eb9G0nQAh3VY2YzzOwpM3vSzOaZ2VAzm2NmL5rZiuy/piDLIFItUZps1FdKopLJbOWkOJKQDkn7JMDAWvxm1ghcAIx19w4zuxU4Jbv7YndfENS1RYIQpbxwsVRFY0N9RcGr2PmKHZsvKimwckT5G0nQgh7HPwSoN7MhwNbAqwFfTyQwUZpsVO3x74XOVzfYqBtk/V5D6+3ET2CB393bgauAl4E1wEZ3vye7+3Ize8LMZpvZVoVeb2bnmFmrmbWuXbs2qGKKlCxKi8NVO1VR6HxXnjSBK/9pQr/XiFIKTEpj7lt24FTlxGY7ALcDXwI2ALcBC4D7gdeAjwHXA6vc/bK+ztXc3Oytra2BlFOkHHFMaQRt9My7KBRFDHix5dhaF0dymNkyd2/O3x7kqJ7DgRfdfW22AHcAB7n7Tdn9H5rZL4FvBVgGkapKc164mLQPjYyjIHP8LwMHmNnWZmbAYcAzZjYSILttGvBkgGUQkYBFKQUmpQmsxe/uj5jZAuAxYBOwnExq5/+b2XAy3wRXAF8LqgwiEryoT9aSLQWW468m5fhFRMpXLMevZZlFRFJGSzaIRFSQI4iquSBb0GWV6lPgF4mgINcFKrgg222Pg320Pk8514vSGkZSGqV6RCKo2KSoi+avGPDdogouyNblWyzKVuokrIFO4ErznbDCoha/SAT1tf7PQFvU1V6QbSBrGOnbQjjU4heJoP4mPw1kSYRyJlaVcuxA1jDScg/hUOAXiaBCk6LyVboqaMEF2QYZdYP7X5Ct1POV+toorXiaJkr1iERQ7qSoYsslV7okQrEJV4W2lXrDlUpfq+UewqEJXCIRl58Hh0yLOgk3Dkly3aIgjEXaRKQKkrwkQpLrFmVq8YuIJJSWbBAREUCpHpFE0dIJUgoFfpGE0GQoKZVSPSIJoclQUioFfpGE0GQoKZUCv0hCDGTpBEmXkgK/mQ0tsG2n6hdHRCoV93vfapXO2im1xf8XMzug+4mZnQg8FEyRRKQS0yY2csUJ42hsqMeAxob62MyA7e6Ybt/QgfNRx7SCfzBKHdXzZeAXZrYU2AUYBnw+qEKJSGWmTWyMRaDP11fHdBzrE3UlBX53X2lmlwM3Au8An3X31YGWTERSQx3TtVVqjv8G4CJgPHAmcKeZnRdguUQkRbavrytruwxMqTn+lcCh7v6iu/8e2B+YFFyxRCRNzMrbLgNTaqrnR3nPNwJfDaJAIpI+G97vLGu7DEypqZ49zWyBmT1tZn/r/hd04UQkHTQHobZKTfX8ErgW2AQcCvwKuCmoQolIusR9DkLclDqcs97d7zczc/eXgFlmtgz49wDLJiIhq9Vqn7ohS22VGvg/NLNBwPNmdj7QDmwbXLFEJGy1Xu0zrnMQ4qjUwH8hsDVwAfB9MpO3pgdVKBGpnlJb7fnHvf/3TZpUlVCljur5S/bhu2TG8YtIDJTaai90XDGaVBV/fQZ+M1vc1353n1rd4ohINZW6FEKh44rRSJv466/FfyDwCjAPeATQdAqRGCl1KYRSW/EaaZMM/Q3n/AfgO8A+wNXAEcCb7v4Hd/9D0IUTkYEpdXx8seMa6utiudqn9K3PFr+7bwZ+B/zOzLYCTgWWmtn33P0ntSigiFTu4qPG9MrdQ+FW+8VHjeHi2x6ns8t7ttUNMmZN3VuBPitJN7Lvt3M3G/CPJRP0RwHXAAtLObmZzQDOBpzMej9nAiOBW8gs7bwMON3d/15B2aXKkvSLHRdBv+fljI/vynve2eXMWvxUr/PUosxRlLQb2Zu7F99p9isyaZ67gVvc/cmST2zWCPwJGOvuHWZ2a/Y8U4A73P0WM7sOeNzdr+3rXM3Nzd7a2lrqpaUC+b/YkGkZ6qt9cKL0nk+87B7WF1kXJ7dMUSpzLU1uWVJwpFNjQz0PzozurUnMbJm7N+dv7y/HfxqwJ5lx/A+Z2dvZf++Y2dslXHcIUG9mQ8jMA1hDZg7Aguz+ucC0EusgAepr9IcEI0rvebGgD73LFKUy11LS7hfQX46/4puxu3u7mV0FvAx0APeQSe1scPdN2cNWAwWbCWZ2DnAOwG677VZpMaRESfvFjoM4vefdZYpTmatpl4b6gi3+uA5trTiw98fMdgCOB0aTuV3jNsDRpb7e3a9392Z3bx4+fHhApZRuWh2x9qL0njf0c8OT7jJFqcy1lLRF5AIL/MDhwIvuvtbdO4E7gMlAQzb1A7ArmXV/JGRJ+8WOg2Lv+aGfHs7kliWMnnkXk1uW1OSG47Om7k3doMLTdHJ/D0r5PVm0vL3m5Q9anG9kX0ipa/VU4mXgADPbmkyq5zCgFfgv4CQyI3umA78JsAxSIq2OWHuF3vNDPz2c25e113z0SG5Z2jd0MNiMze405v0e9Pd7krTRL7mStIhcn6N6Bnxys+8BXyKzjv9yMkM7G8kE/R2z205z9w/7Oo9G9UhaxHX0SLe4lz9pio3qCbLFj7tfClyat/lvwH5BXlckrirpPI3SuPq0dv7GTZA5fhEpU7mdp92plfYNHTgfpVbCyquntfM3bhT4RSKk3E72oMfVl9tRq0EC8RBoqkdEylNuJ3uxdfP7Wk+/VJV01GqQQDwo8ItETDmjR7pH3xTaPlClruWfL0mjX5JKgV8kxgoF/b62lyPojtoodUqnjXL8IjHWWKTTtNj2cgTZURu1Tum0UeAXibEgO1ODPHdaF3uLCqV6ZECS8nU9rvUIsjM1yHNrvH+4FPilYrWenh9UcI77MgNBdqYGde6krXYZN0r1SMVq+XU9yJyw0g61p/H+4VLgl4rV8ut6kMFZaYfaS9pql3GjVI9UrJZf14MMzko7hEPj/cOjFr9UrJZf14McWljrtEMS16svJC31jCMFfqlYLb+uBxmca1mPtIxfT0s94yrQ9firRevxC8R3yGWutKxXn5Z6Rl0o6/GLVFMScsJp6UhOSz3jSqkekRpKy3r1aalnXCnwi9RQWsavp6WecaVUj0gNpWW9+rTUM67UuSsiklDFOneV6hERSRkFfhGRlFHgFxFJGQV+EZGU0ageiaUkzOIVCYsCv8RO3G+cIhI2pXokdnTjFJGBUeCX2NE6MCIDo8AvsaN1YEQGRoFfYkfrwIgMjDp3JXa0DozIwCjwSywlYW1+kbAo1SMikjJq8YuESBPRJAyBBX4zGwPMz9m0B/DvQAPwz8Da7PbvuPvdQZVDJKo0EU3CEliqx92fdfcmd28C/hF4H1iY3T27e5+CvqSVJqJJWGqV4z8MWOXuL9XoeiKRp4loEpZaBf5TgHk5z883syfM7BdmtkOhF5jZOWbWamata9euLXSISKxpIpqEJfDAb2YfA6YCt2U3XQt8AmgC1gA/LPQ6d7/e3ZvdvXn48OFBF1NiYNHydia3LGH0zLuY3LKERcvbwy7SgGgimoSlFqN6jgEec/fXAbp/ApjZz4A7a1AGibkkdoRqIpqEpRaB/1Ry0jxmNtLd12SffhF4sgZlkJjrqyM0zoFSE9EkDIEGfjPbBjgCODdn83+YWRPgQFvePpGC1BEqUj2BBn53fw8Ylrft9CCvKcm0S0M97QWCvDpCRcqnJRskFtQRKlI9WrJBYkEdoSLVo8AvZctdX2b7+jrMYMP7nYEHY3WEilSHAr+UJX9Y5YaOzp59SRhiKZIGyvFLWQoNq8yltWZEok+BX8pSyvBJDbEUiTYFfilLKcMnNcRSJNoU+KUshYZV5tIQS5HoU+eulCV/WGUtR/WISHUo8EvZNKxSJN6U6hERSRkFfhGRlFHgFxFJGQV+EZGUUeAXEUkZjeqRSMhd+E3DQkWCpcAvoUvi/XRFokypHgldX/fTFZHqU4tfylbttIzupytSW2rxS1m60zLtGzpwPkrLLFreXvE5iy3qpsXeRIKhwC9lCSIto/vpitSWUj0JUatRMUGkZXQ/XZHaUuBPgFqOitmloZ72AkF+oGkZLfwmUjtK9SRALUfFKC0jEn9q8SdALUfFKC0jEn8K/AkQVPqlGKVlROJNqZ4EUPpFRMqhFn8CKP0iIuVQ4E8IpV9EpFRK9YiIpIwCv4hIyijwi4ikjAK/iEjKqHO3ynQnKRGJusBa/GY2xsxW5Px728wuMrMdzexeM3s++3OHoMpQa0EsWSwiUm2BBX53f9bdm9y9CfhH4H1gITATuN/d9wTuzz6vukXL25ncsoTRM+9icsuSmgRf3UlKROKgVjn+w4BV7v4ScDwwN7t9LjCt2hcLq+WtO0mJSBzUKvCfAszLPh7h7muyj18DRhR6gZmdY2atZta6du3asi4WVstbd5ISkTgIPPCb2ceAqcBt+fvc3QEv9Dp3v97dm929efjw4WVdM6yWt9bMEZE4qEWL/xjgMXd/Pfv8dTMbCZD9+Ua1LxhWy3vaxEauOGEcjQ31GNDYUM8VJ4zTqB4RiZRaDOc8lY/SPACLgelAS/bnb6p9wYuPGtPrjlRQu5a31swRkagLNPCb2TbAEcC5OZtbgFvN7KvAS8DJ1b6uVqsUESnOMmn2aGtubvbW1tawiyEiEitmtszdm/O3a8kGEZGUUeAXEUkZBX4RkZRR4BcRSRkFfhGRlInFqB4zW0tm6GfYdgLeDLsQAUpy/ZJcN0h2/ZJcNwi2fru7+xZLH8Qi8EeFmbUWGhqVFEmuX5LrBsmuX5LrBuHUT6keEZGUUeAXEUkZBf7yXB92AQKW5PoluW6Q7PoluW4QQv2U4xcRSRm1+EVEUkaBX0QkZRT4izCzNjNbaWYrzKw1u22WmbVnt60wsylhl7NSZtZgZgvM7K9m9oyZHWhmO5rZvWb2fPbnDmGXs1JF6hf7z8/MxuSUf4WZvW1mFyXls+ujfrH/7ADMbIaZPWVmT5rZPDMbamajzewRM3vBzOZn71oYbDmU4y/MzNqAZnd/M2fbLOBdd78qrHJVi5nNBR5w959nf9G2Br4DvOXuLWY2E9jB3f8l1IJWqEj9LiIhnx+AmQ0G2oH9gfNIyGfXLa9+ZxLzz87MGoE/AWPdvcPMbgXuBqYAd7j7LWZ2HfC4u18bZFnU4k8hM9se+CxwA4C7/93dNwDHA3Ozh80FpoVRvoHqo35Jcxiwyt1fIiGfXZ7c+iXFEKDezIaQaYysAT4PLMjur8lnp8BfnAP3mNkyMzsnZ/v5ZvaEmf0irl+ngdHAWuCXZrbczH6evVvaCHdfkz3mNWBEaCUcmGL1g2R8ft1O4aPbmibls8uVWz+I+Wfn7u3AVcDLZAL+RmAZsMHdN2UPWw0EfqtABf7iPuPuk8jcLP48M/sscC3wCaCJzAf3w/CKNyBDgEnAte4+EXgPmJl7gGdygHHNAxarX1I+P7Lpq6nAbfn7Yv7ZAQXrF/vPLvvH6ngyDZNdgG2Ao8MoiwJ/Edm/zrj7G8BCYD93f93dN7t7F/AzYL8wyzgAq4HV7v5I9vkCMoHydTMbCZD9+UZI5RuogvVL0OcHmQbJY+7+evZ5Uj67br3ql5DP7nDgRXdf6+6dwB3AZKAhm/oB2JVMv0agFPgLMLNtzGy77sfAkcCT3f+xsr4IPBlG+QbK3V8DXjGzMdlNhwFPA4uB6dlt04HfhFC8AStWv6R8flmn0jsNkojPLkev+iXks3sZOMDMtjYz46P/d/8FnJQ9piafnUb1FGBme5Bp5UMmbfBrd7/czG4k81XTgTbg3Jy8aqyYWRPwc+BjwN/IjJoYBNwK7EZmGeyT3f2tsMo4EEXqdw0J+PyyjZGXgT3cfWN22zCS89kVql8i/u+Z2feALwGbgOXA2WRy+rcAO2a3nebuHwZaDgV+EZF0UapHRCRlFPhFRFJGgV9EJGUU+EVEUkaBX0QkZRT4RfpgZm5mN+U8H2Jma83szuzzqdlF0URiY0j/h4ik2nvAPmZW7+4dwBHkzKx098VkJk+JxIZa/CL9uxs4Nvs4f0bpGWb2k+zjOWZ2jZk9ZGZ/M7OTCpxLJHQK/CL9uwU4xcyGAuOBR/o4diTwGeA4oKUGZRMpmwK/SD/c/QlgFJnW/t39HL7I3bvc/WmSsTSyJJBy/CKlWUxmLfVDgGF9HJe7xooFWSCRSinwi5TmF2RumLHSzA4JuSwiA6LAL1ICd19NZnVPkdjT6pwiIimjzl0RkZRR4BcRSRkFfhGRlFHgFxFJGQV+EZGUUeAXEUkZBX4RkZT5byTfM45YCWjJAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "dataset.plot(x='Minimum Temperature', y='Maximum Temperature', style='o')\n", "plt.title('Min/Max Temperature')\n", "plt.xlabel('Min')\n", "plt.ylabel('Max')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Splicing: Split into Training and Test Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before creating the model (from an algorithm and a dataset), we prepare the dataset\n", "* 80% for training\n", "* 20% for testing/verification" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "import sklearn\n", "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "input_train, input_test, output_train, output_test = \\\n", " train_test_split(inputfeatures, outputfeatures, test_size=0.2, random_state=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating the Model: Algorithm + Training Data" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import LinearRegression" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Initially, the model *is* the algorithm" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "model = LinearRegression()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we feed it the training data" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "model = model.fit(input_train, output_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Model is complete; see the parameters of the linear interpolation (would need theory to better understand):" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.80189231]])" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.coef_" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([25.95355086])" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.intercept_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Verify the Model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We saved 20% of the dataset for verification.\n", "* Use the model to predict the output for the input test data.\n", "* Compare prediction to actual output test set" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "output_predicted = model.predict(input_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we (ab)use a `pandas.DataFrame` to nicely format actual output test data and predicted output side by side.\n", "\n", "**Note** that `input_test` is a `pd.DataFrame`, but `output_predicted` is a `numpy.ndarray`.\n", "\n", "**Reason**:`model.predict()` is happy with anything that supports indexing (thanks to duck typing - we gave it a `Dataframe`), but its output is always a `numpy.ndarray`" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ActualPredicted
080.083.609608
184.984.571879
291.186.736988
380.084.170933
478.278.798254
592.084.170933
678.275.189739
792.088.180394
885.483.449230
980.188.661530
1092.987.057745
1185.483.850176
1287.281.284120
1390.083.850176
1483.681.685067
\n", "
" ], "text/plain": [ " Actual Predicted\n", "0 80.0 83.609608\n", "1 84.9 84.571879\n", "2 91.1 86.736988\n", "3 80.0 84.170933\n", "4 78.2 78.798254\n", "5 92.0 84.170933\n", "6 78.2 75.189739\n", "7 92.0 88.180394\n", "8 85.4 83.449230\n", "9 80.1 88.661530\n", "10 92.9 87.057745\n", "11 85.4 83.850176\n", "12 87.2 81.284120\n", "13 90.0 83.850176\n", "14 83.6 81.685067" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame({'Actual': output_test.values.reshape((15,)),\n", " 'Predicted': output_predicted.reshape((15,))})\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparing the actual and predicted values, we can see that they are \"not far off\". Whatever this means - in a real data science world (this is only the surface), we would now have to use advanced statistical methods to actually measure the term \"not far off\".\n", "\n", "But this is left to data scientists. Our job is to create correct programs, and to keep those *maintainable*." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.7" } }, "nbformat": 4, "nbformat_minor": 4 }