Class on 23-10-2026

In this class I presented some ideas on the first homework assignment and left time for supervised work.

I promised to upload an example notebook with some hints on Part 2. Here goes: homework_1_example.html . This notebook also includes some things that may be helpful on Part 1 (b) and Part 3 (b). Below are some other notes from what was discussed on class.

Note on Grading

This course covers a lot of topics in quick succession, so my policy is to grade homework very leniently. It is enough to provide a correct answer and a brief justification to get full marks. Lacking a correct answer, I highly value honesty and evidence of critical thinking. You can email me with questions at christopher.gadzinski@uni.lu.

Part 1

Problem b

Remember that $$\frac{\partial}{\partial x}\lvert x \rvert = \operatorname{sign}(x) = \begin{cases} 1 & : x > 0 \\ -1 &: x < 0 \end{cases},$$ with the subtlety that this derivative doesn't exist when $x = 0.$ (Why not? Think about the graph of $\lvert x \rvert.$)

Because of this subtlety, to solve the minimization problem in Problem b, you can't apply the usual playbook of solving an equation $L'(\mu) = 0$. You can still make use of derivatives, but you need to think a little more!

You don't need to carefully justify your answer. It is enough to say where the function is minimized and give some intuition on how you got to this answer. (Remember the answer is slightly different in the cases of even and odd $N$; ideally you should discuss both.) You can also check the example notebook for a typical graph of the function $L.$

Problem c

This is a harder differentiation problem. I included it because differentiating the softmax function happens to be important in deep learning.

Hint 1: Remember that the derivative of the logarithm of $x$ is $1/x.$ So, for any function $f(x),$ $$\frac{\partial}{\partial x} \ln(f(x)) = \frac{1}{f(x)} \frac{\partial}{\partial x} f(x)$$ follows from the chain rule.

Hint 2: If you encounter an expression like $$\frac{\partial}{\partial z_j} \sum_{j = 1}^N e^{z_j},$$ remember that this is the same as $$\frac{\partial}{\partial z_j} (e^{z_1} + e^{z_2} + \dots + e_{z_N}) = \sum_{k = 1}^N \frac{\partial}{\partial z_j} e^{z_k}$$ where on the right-hand side I've replaced the index $j$ of the sum with $k.$

Hint 3: Note that $$\frac{\partial}{\partial z_j} e^{z_k} = \begin{cases} 0 & : j \neq k \\ e^{z_j} & : j = k. \end{cases}$$ If you like, you can write this as $\delta_{j k} e^{z_j},$ where $\delta_{j k}$ is defined to be zero when when indices $j$ and $k$ are not equal and $1$ otherwise. This is called the Kronecker delta.