Deep Learning · Quiz 1 · May 2026

**1. Key concept:** A neural network's decision boundary is non-linear only if its activations are non-linear and there is at least one non-linear hidden layer. **2. Counterexample:** A single neuron (perceptron or sigmoid), or a network whose layers all use linear/identity activations, gives a *linear* boundary. So 'always non-linear' is false. **3. Conclude:** The statement is False. Answer: B — False. A: **Incorrect:** The boundary is not always non-linear. A single neuron or a purely linear network has a linear boundary. B: **Correct:** A single neuron or a purely linear network has a linear boundary, so the statement is false.

**1. Statement I:** A single MP neuron has fixed unit weights (excitatory or inhibitory) and one threshold. It treats all excitatory inputs equally, so it cannot represent every linearly separable function. Example: \(f=x_1 \lor (x_2\land x_3)\) is linearly separable (\(2x_1+x_2+x_3\ge 2\)), but it is not symmetric in the inputs, so one MP neuron cannot compute it. Statement I is false. **2. Statement II:** A network of MP neurons (one or more MP neurons) can represent any Boolean function, so it can represent every linearly separable Boolean function. Statement II is true. **3. Conclude:** Only Statement II is true. Answer: B — Only Statement II is true. A: **Incorrect:** Statement I is false because one MP neuron cannot represent all linearly separable functions. B: **Correct:** Statement II is true and Statement I is false. C: **Incorrect:** Statement I is false, so both cannot be true. D: **Incorrect:** Statement II is true, so 'none' is wrong.

**1. Given:** \(L(w)=w^2\), \(w_0=2\). The gradient is \(2w\), so the update is \[w_{t+1}=w_t-\eta(2w_t)=(1-2\eta)w_t.\] **2. Case \(\eta=0.3\):** The factor is \(1-0.6=0.4\), so \(|w|\) shrinks. Loss at step 1: \((0.4\times2)^2=0.64\). This matches Graph X, which drops from 4 to about 0.64 and then to 0. **3. Case \(\eta=1.1\):** The factor is \(1-2.2=-1.2\), so \(|w|\) grows by 1.2 each step and the loss grows by \(1.44\) each step. After 10 steps, loss \(=4\times1.44^{10}\approx153\). This matches Graph Y. **4. Conclude:** X uses \(\eta=0.3\) and Y uses \(\eta=1.1\). Answer: B — X uses \(\eta=0.3\), Y uses \(\eta=1.1\). A: **Incorrect:** The large rate 1.1 would make the loss diverge, so it cannot be Graph X. B: **Correct:** Rate 0.3 converges (factor 0.4) and rate 1.1 diverges (factor -1.2).

**1. Given:** \(L=(w-5)^2\), \(\eta=0.04\). The curvature is \(\lambda=2\), so \(\eta\lambda=0.08\). The error \(e=w-5\) follows \[e_{t+1}=(1+\beta-0.08)\,e_t-\beta\,e_{t-1}.\] **2. Find the decay of each \(\beta\):** The characteristic equation is \(r^2-(0.92+\beta)r+\beta=0\). - \(\beta=0\): \(r=0.92\). Smooth, monotone decay (slow). - \(\beta=0.5\): real roots \(\approx0.77,\,0.65\). Faster monotone decay, no oscillation. - \(\beta=0.9\): complex roots with \(|r|=\sqrt{0.9}\approx0.95\). Oscillating, decaying. - \(\beta=0.99\): complex roots with \(|r|\approx0.995\). Large oscillations, decaying very slowly. **3. Match the curves:** S (smooth, slow) is \(\beta=0\). T (fastest, no oscillation) is \(\beta=0.5\). U (oscillates, then dies out) is \(\beta=0.9\). V (big long-lasting oscillation) is \(\beta=0.99\). Answer: A — S: \(\beta=0\), T: \(\beta=0.5\), U: \(\beta=0.9\), V: \(\beta=0.99\). A: **Correct:** S is the smooth vanilla curve, T is fastest, U oscillates and settles, V oscillates the longest. B: **Incorrect:** This reverses the order. The most oscillating curve V needs the largest beta, not 0. C: **Incorrect:** T (no oscillation, fastest) cannot be vanilla GD, and V cannot be 0.9 since it oscillates the longest. D: **Incorrect:** T has no oscillation, so it cannot be 0.9, and U oscillates more than 0.5 would.

**1. Given:** \(w=\begin{bmatrix}3\\4\end{bmatrix}\), \(x=\begin{bmatrix}4\\3\end{bmatrix}\), \(w_{new}=w-x\). **2. Compute \(w_{new}\):** \[w_{new}=\begin{bmatrix}3-4\\4-3\end{bmatrix}=\begin{bmatrix}-1\\1\end{bmatrix}\] **3. Use the angle formula:** \(\cos\theta=\dfrac{w_{new}\cdot x}{\|w_{new}\|\,\|x\|}\). \[w_{new}\cdot x=-4+3=-1,\quad \|w_{new}\|=\sqrt2,\quad \|x\|=5\] \[\cos\theta_{new}=\frac{-1}{5\sqrt2}\] **4. Conclude:** \(\theta_{new}=\cos^{-1}\!\left(\dfrac{-1}{5\sqrt2}\right)\). Answer: C — \(\cos^{-1}\!\left(\dfrac{-1}{5\sqrt2}\right)\). A: **Incorrect:** The dot product is -1, not 24. B: **Incorrect:** The dot product is -1, so the cosine is negative, not positive. C: **Correct:** The dot product is -1 and the norms are \(\sqrt2\) and 5, giving \(-1/(5\sqrt2)\). D: **Incorrect:** The vectors are not parallel, so the angle is not 0°.

**1. A:** SGD with \(B=1\) uses the gradient of a *single* example. This is a noisy estimate, not the true gradient of the total loss. False. **2. B:** One epoch means one pass over all \(N\) points. With batch size \(B\) there are \(\lceil N/B\rceil\) batches, so \(\lceil N/B\rceil\) updates. True. **3. C:** Momentum keeps accumulated velocity, so it can run past the minimum in a narrow valley and oscillate. True. **4. D:** For a smooth loss and small enough \(\eta\), a full-batch gradient step decreases the loss (descent lemma). True. **5. Conclude:** B, C and D are true. Answer: B, C, D — B, C, D are true. A: **Incorrect:** SGD uses one example, so it gives only a noisy estimate of the true gradient. B: **Correct:** One epoch has \(\lceil N/B\rceil\) batches, so that many updates. C: **Correct:** Accumulated velocity can carry the parameters past the minimum. D: **Correct:** With a small enough learning rate on a smooth loss, each full-batch step lowers the loss.

**1. Key idea:** All neurons have identical weights and bias, so they compute the same output and receive the same gradient. This is the *symmetry problem*. **2. A:** The gradients are identical, so after one step the weights stay identical. False. **3. B:** Same \(w\), same \(b\), same input \(\Rightarrow\) same output \(\sigma(w^Tx+b)\). True. **4. C:** The gradients are identical across neurons, but not zero in general. False. **5. D:** All \(n\) neurons compute the same function, so the layer acts like one neuron (with combined outgoing weight). True. **6. Conclude:** B and D are true. Answer: B, D — B, D. A: **Incorrect:** The gradients are identical, so the weights remain identical after the step. B: **Correct:** Identical weights, bias and input give identical outputs. C: **Incorrect:** The gradients are identical but generally non-zero. D: **Correct:** All neurons compute the same function, so one neuron is equivalent.

**1. Rule:** Output \(=1\) only if \(x_3=0\) and \(x_1+x_2\ge 2\). **2. Check inputs:** - \((1,1,0)\): \(x_3=0\), sum \(=2\ge2\), so output 1. A is true. - \((1,1,1)\): \(x_3=1\) (inhibitor active), so output 0. B is false. - \((1,0,0)\): sum \(=1<2\), so output 0. C is true. - Threshold 3: max excitatory sum is 2, so the neuron never fires. Input \((1,1,0)\) changes from 1 to 0. D is false. **3. Conclude:** A and C are correct. Answer: A, C — A, C. A: **Correct:** Inhibitory input is 0 and the sum of excitatory inputs is 2, so it fires. B: **Incorrect:** The inhibitory input is active, so the output is 0. C: **Correct:** The sum of excitatory inputs is 1, which is below 2, so the output is 0. D: **Incorrect:** With threshold 3 the neuron never fires, so the output of (1,1,0) changes from 1 to 0.

**1. Start:** \(w=(2,-1)\), \(b=0\). Predict \(+1\) if \(w^Tx+b\ge0\). **2. Point A \((3,1),+1\):** \(6-1+0=5\ge0\Rightarrow+1\). Correct, no update. **3. Point B \((2,0),-1\):** \(4\ge0\Rightarrow+1\). Wrong, update (first update): \[w=(2,-1)-(2,0)=(0,-1),\quad b=0-1=-1\] **4. Point C \((1,2),-1\):** \(0\cdot1-1\cdot2-1=-3<0\Rightarrow-1\). Correct, no update. **5. Point D \((4,4),+1\):** \(0-4-1=-5<0\Rightarrow-1\). Wrong, update (second update): \[w=(0,-1)+(4,4)=(4,3),\quad b=-1+1=0\] **6. Check options:** First update after B (true). Second update after D, not C (false). Final bias is 0, not -1 (false). Exactly two updates (true). **7. Conclude:** A and D are correct. Answer: A, D — A, D. A: **Correct:** The first mistake is at B. B: **Incorrect:** C is classified correctly; the second update happens at D. C: **Incorrect:** The final bias is -1+1 = 0. D: **Correct:** Updates happen only at B and D.

**1. A:** A perceptron is a hard step function, so its output jumps at the threshold. A sigmoid changes smoothly. True. **2. B:** The sigmoid is differentiable everywhere, so gradient-based learning works. True. **3. C:** For \(\sigma(kz)\) with large \(k\), the curve approaches a step function. True. **4. D:** One hidden layer of perceptrons (one for each input pattern) can represent any Boolean function. True. **5. E:** This is the universal approximation theorem. True. **6. F:** A single sigmoid neuron has a *linear* decision boundary \(w^Tx+b=0\). It cannot represent every non-linear boundary. False. **7. Conclude:** A, B, C, D and E are correct. Answer: A, B, C, D, E — A, B, C, D, E. A: **Correct:** Perceptron output jumps; sigmoid output varies smoothly. B: **Correct:** The sigmoid has a derivative everywhere, which allows gradient-based learning. C: **Correct:** Large weights make the sigmoid very steep, close to a step. D: **Correct:** A hidden layer of perceptrons can realise any Boolean function. E: **Correct:** This is the universal approximation theorem. F: **Incorrect:** A single sigmoid neuron has a linear boundary, so it cannot represent all non-linear boundaries.

**1. A:** \(\sigma(z)\approx\dfrac{1}{1+(1-z)}=\dfrac{1}{2-z}=\dfrac12\cdot\dfrac1{1-z/2}\approx\dfrac12+\dfrac z4\) for small \(z\). True (this equals the Taylor series of \(\sigma\) at 0). **2. B:** \(\sigma'(z)=\sigma(z)(1-\sigma(z))\), so \(\sigma'(0)=\tfrac12\cdot\tfrac12=\tfrac14\). True. **3. C:** A first-order approximation is valid only near \(z=0\). \(\sigma\) is bounded in \((0,1)\), but the line \(\tfrac12+\tfrac z4\) is unbounded. False. **4. D:** For large \(|z|\), the line goes outside \([0,1]\) while \(\sigma\) saturates, so the error grows. True. **5. E:** The sigmoid is smooth and differentiable, unlike the hard-threshold perceptron. True. **6. Conclude:** A, B, D and E are true. Answer: A, B, D, E — A, B, D, E. A: **Correct:** Expanding near 0 gives 1/2 + z/4. B: **Correct:** \(\sigma'(0)=\sigma(0)(1-\sigma(0))=1/4\). C: **Incorrect:** The sigmoid is not linear; the approximation is valid only near z = 0. D: **Correct:** The line is unbounded while the sigmoid saturates, so the error grows with |z|. E: **Correct:** The sigmoid is differentiable, so gradient-based learning works, unlike the step function.

**1. Given:** \(W_1=0\), \(b_1=0\), \(W_2=[100,100,100]\), \(b_2=0\), input \(x=(8,3000)\). **2. Hidden layer:** \(a_1=W_1x+b_1=0\), so each hidden output is \(h_1=\sigma(0)=0.5\). **3. Output layer (linear):** \[\hat y=W_2h_1+b_2=100(0.5)+100(0.5)+100(0.5)=150\] **4. Conclude:** The initial prediction does not depend on the input values here. Answer: 150.

**1. Given:** \(a_L=[a_1,0,0]\) and \(\hat y_1=0.5\). **2. Softmax:** \[\hat y_1=\frac{e^{a_1}}{e^{a_1}+e^0+e^0}=\frac{e^{a_1}}{e^{a_1}+2}\] **3. Solve:** \[\frac{e^{a_1}}{e^{a_1}+2}=0.5\Rightarrow e^{a_1}=0.5e^{a_1}+1\Rightarrow e^{a_1}=2\Rightarrow a_1=\ln2\approx0.69\] Answer: \(a_1=\ln 2\approx0.69\).

**1. Given:** \(x=2\), \(y=0.8\), \(w=b=0\), \(L=\tfrac12(f-y)^2\). **2. Forward pass:** \(f=\sigma(0\cdot2+0)=\sigma(0)=0.5\). **3. Chain rule:** \[\nabla_wL=(f-y)\,f(1-f)\,x\] **4. Substitute:** \[\nabla_wL=(0.5-0.8)(0.5)(0.5)(2)=(-0.3)(0.25)(2)=-0.15\] Answer: \(-0.15\).

**1. Setup:** With \(h_i=a_i\) in both hidden layers, \[a_3=W_3(W_2(W_1x+b_1)+b_2)+b_3=W_3W_2W_1x+(W_3W_2b_1+W_3b_2+b_3).\] **2. Meaning:** So \(a_3=W'x+b'\) with \(W'=W_3W_2W_1\). The hidden layers add no non-linearity, and the model is one softmax regression. **3. Check the others:** Softmax only normalises the scores; the boundaries remain linear. Gradients flow normally through linear layers. Nothing forces the weights to become identity matrices. Answer: B — The network is equivalent to a single softmax regression with \(W'=W_3W_2W_1\). A: **Incorrect:** The decision boundary is still linear, since softmax does not add curvature to the boundaries between classes. B: **Correct:** Stacked linear layers collapse into one linear map \(W'=W_3W_2W_1\), followed by softmax. C: **Incorrect:** Gradients flow fine through linear layers, so training works. D: **Incorrect:** Nothing pushes the weights toward identity matrices.

**1. Concept:** For softmax with cross-entropy loss \(\mathcal L=-\log\hat y_\ell\), the gradient with respect to the pre-activations is \[\frac{\partial\mathcal L}{\partial a_{L,i}}=\hat y_i-\mathbb 1[i=\ell].\] **2. Substitute for \(i=\ell\):** \[\frac{\partial\mathcal L}{\partial a_{L,\ell}}=\hat y_\ell-1=0.99-1=-0.01\] Answer: \(-0.01\).

**1. Concept:** Vanilla (full-batch) gradient descent computes the gradient over the whole dataset before each update. **2. Count:** One pass over all 100,000 clips (one epoch) gives exactly one update. Answer: 1.

**1. Given:** \(N=100{,}000\) clips, batch size \(B=50\). **2. Formula:** Updates per epoch \(=\lceil N/B\rceil\). **3. Substitute:** \[\frac{100000}{50}=2000\] Answer: 2000.