%load_ext autoreload
%autoreload 2

import iml  # helper functions and classes (see iml/)
import numpy as np  # array processing
import pandas as pd  # dataframes, reading files
import matplotlib.pyplot as plt  # making plots

Part 1: Partial Derivatives

(b)

Compute the derivative $\partial L / \partial \mu.$ If $\lambda = 0,$ what values of $\mu$ minimize $L$?

If you want, you can help illustrate this problem by graphing $L$ for some arbitrary values $x(i)$.

def loss(mu, x):
    return np.abs(mu[:, None] - x).sum(axis=1)

mu = np.linspace(-3, 3, 1000)
x = np.array([-1, 0, 0.5, 2])  # x(1) = -1, x(2) = 0, ..., x(4) = 2
plt.plot(mu, loss(mu, x))

[<matplotlib.lines.Line2D at 0x11a070c80>]

In this case, the function is minimized when $\mu$ is on the interval $[0, 0.5].$ Your task: describe what will happen in general.

Part 2: Linear Regression

(b)

Extend the SimpleRegressor class from iml/simple_regressor.py, implementing the method fit(self, x, y, lr, steps) to implement gradient descent on mean squared error. Run fit() on the unnormalized data vectors x and y, and show the training process over 20 iterations with SimpleRegressor.show_training(). Find a learning rate that makes the loss decrease.

Let's do this problem with the simplification that we don't want to update the parameter $\theta_1$.

class Regressor(iml.SimpleRegressor):
    def __init__(self):
        # iml.SimpleRegressor.__init__() raises an exception
        self.weights_ = np.zeros(2)
        # self.weights_[0] is theta_0, self.weights_[1] is theta_1

    def predict(self, x):
        # this was already defined in iml.SimpleRegressor
        return self.weights_[0] + x * self.weights_[1]

    def fit(self, x, y, lr: float, steps: int = 20):
        # keeping copies of everything to visualize training later
        self.x_train_ = x.copy()
        self.y_train_ = y.copy()
        self.y_predicted_ = np.zeros((steps + 1, y.shape[0]))
        self.y_predicted_[0] = self.predict(x)

        for i in range(steps):
            # the recipe for gradient descent (on lecture slides) is:
            # for each parameter:
                # - take the gradient of the loss with respect to that parameter
                # - multiply that gradient by the "learning rate", subtract it from the
                # current value of that parameter
            # I'm denoting "lr" for the learning rate

            err = self.y_predicted_[i] - y
            theta_0_grad = 2 * err.mean()  # this is the gradient with respect to theta_0
            self.weights_[0] -= lr * theta_0_grad
            # in your solution you also need to update weights_[1]

            self.y_predicted_[i + 1] = self.predict(x)

x, y = iml.load_powerplant()
model = Regressor()
model.fit(x, y, lr=0.01)  # you need to choose a sufficiently small learning rate so that loss decreases
model.show_training()

Over the course of gradient descent, the graph of our model goes from being a horizontal line at 0 to a horizontal line at about 150. The loss function is clearly decreasing.

Part 3: k-Means and PCA

(b)

Investigate data/ulu.csv using PCA and explain how this dataset was produced. Using scikit-learn, run both $k$-means clustering and DBSCAN on this dataset. Which method produces more reasonable "clusters"?

In class, I showed that if you projected this three-dimensional dataset onto two dimensions using PCA, the data points lie on a rectangle.

from sklearn.decomposition import PCA
data = iml.load_ulu()
pca = PCA(n_components=2)
x = pca.fit_transform(data)
plt.scatter(x[:, 0], x[:, 1])

<matplotlib.collections.PathCollection at 0x11a38f6e0>

So, the points in our 3-dimensional dataset can be well approximated by their projections onto a plane. However, this isn't the whole story.

How much do our datapoints vary in the direction orthogonal to the plane? We can compute this by asking for the third principal component.

pca = PCA(n_components=3)
x = pca.fit_transform(data)
plt.hist(x[:, 2])  # x[:, 2] is the third principal component returned by PCA
plt.show()

I leave the rest of the analysis to you.

Documentation on scatter(): https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html.