Deep Learning · Quiz 2 · Jan 2026

NAdam means Nesterov-accelerated Adaptive Moment Estimation. It combines Adam's adaptive scaling from first- and second-moment estimates with a Nesterov-style momentum update. Identify both components in the name rather than choosing an algorithm that supplies only one. **Answer: A.** A: **Correct:** Explicitly combines Nesterov momentum and Adam-style adaptive moments. B: **Incorrect:** Standard Adam uses moment estimates but not the NAdam modification. C: **Incorrect:** This description does not identify the Nesterov-corrected Adam algorithm asked for. D: **Incorrect:** Nesterov gradient descent alone does not provide adaptive second-moment scaling.

Dropout randomly masks units during training, and the mask is sampled again in later iterations. Different hidden layers may use different dropout probabilities; nothing requires one shared probability. Temporary zero activations do not force trained weights to become permanently zero. A unit dropped at iteration t may be active again at t+1 and receive a gradient then. **Answer: B.** A: **Incorrect:** Dropout creates temporary activation sparsity, not guaranteed sparsity of learned weights. B: **Correct:** Layer-specific dropout probabilities are valid hyperparameters. C: **Incorrect:** The next iteration uses a newly sampled mask, so the unit can be active and updated then. D: **Incorrect:** B is a valid statement.

AdaGrad scales updates using the accumulated sum of past squared gradients: ii. RMSprop uses an exponential moving average of squared gradients rather than a forever-growing sum: iii. Adam combines momentum-like first moments with RMSprop-like second moments and corrects their initial bias: i and iv. AdaDelta scales updates with gradient/update statistics to address unit consistency and reduce dependence on a manually chosen learning-rate scale: v. Thus 1–ii, 2–iii, 3–i,iv, 4–v. **Answer: C.** A: **Incorrect:** Assigns momentum/moment averaging to AdaGrad and other properties to the wrong algorithms. B: **Incorrect:** Misassigns RMSprop's behaviour and Adam's moment/bias-correction combination. C: **Correct:** All four mappings match the algorithms' update statistics. D: **Incorrect:** Does not match the accumulated-squares, moving-average and moment-correction properties.

Small early-layer updates can have different causes. Leaky ReLU supplies a small nonzero derivative on negative inputs, which can help when ordinary ReLU units are inactive. Increasing the learning rate can enlarge updates when gradients are small but nonzero; it cannot revive a strictly zero gradient and must be kept small enough for stable training. Replacing ReLU with saturating tanh does not generally cure vanishing gradients, and changing the number of output neurons changes the task rather than addressing this issue. **Answer: A, D.** A: **Correct:** Can maintain a gradient on the negative side where ReLU's derivative is zero. B: **Incorrect:** tanh can saturate and produce small derivatives, so it is not the general remedy intended. C: **Incorrect:** Output width normally corresponds to class count and does not directly repair early-layer gradients. D: **Correct:** Could enlarge nonzero updates if the current learning rate is too small; this is a conditional remedy, not a guarantee.

Write \(h_1=\max(0,w_1x+b_1)\) and \(\hat y=h_2=w_2h_1+b_2\). The output unit is linear. For \(L=\tfrac12(\hat y-y)^2\), the chain rule gives \[\frac{\partial L}{\partial w_1}=(\hat y-y)w_2\mathbf1_{w_1x+b_1>0}x.\] SGD changes w1 by minus the learning rate times this derivative. It is zero if the hidden preactivation is negative, if w2=0, or if prediction equals target. Positive x,w1,b1 only ensure an active ReLU; they do not ensure zero output error or zero w2. The derivative convention at exactly zero is not needed for the strict-negative claim. **Answer: B, C, D, F.** A: **Incorrect:** Applies ReLU to the output even though that unit is linear. B: **Correct:** The linear output is w2 times h1 plus b2. C: **Correct:** A strictly negative hidden preactivation makes the ReLU derivative zero. D: **Correct:** w2=0 blocks this gradient path into w1. E: **Incorrect:** Positive inputs and hidden parameters do not force the complete loss gradient to vanish. F: **Correct:** Zero residual makes every gradient in this squared-loss path zero.

A convolution without bias has \(k_hk_wC_{in}C_{out}\) learned weights. Each parallel layer sees the same 3-channel image and has 64 filters of size 5×5, so each has \(5\cdot5\cdot3\cdot64=4800\) parameters. Padding changes spatial output size, not filter shape or count. Therefore the difference is 4800-4800=0, even though the output resolutions differ. **Answer: 0.**

The required decrease is \(J(w_{t+1})\le J(w_t)-10\eta\). At eta=1.0 the cost increases by 5, failing. At 0.8 the actual decrease 4 is smaller than required 8, failing. At 0.4 the actual decrease 6 exceeds required 4, passing. At 0.1 the actual decrease 0.5 is smaller than required 1, failing. Hence 0.4 is the largest tested acceptable step. A smaller tested step need not satisfy this supplied table's condition automatically. **Answer: 0.4.**

The average prediction across the three fitted models is \(\bar f=(1.1+0.9+1.0)/3=1\). Bias at the fixed input is average prediction minus the true target: \(\bar f-y=1-1=0\). If the course uses squared bias in its decomposition, that too is zero. The individual prediction errors do not all have to be zero for the average bias to vanish. **Answer: 0.**

Use variation across fitted models around their mean prediction 1: \[\operatorname{Var}(\hat f)=\frac{(1.1-1)^2+(0.9-1)^2+(1-1)^2}{3}=\frac{0.01+0.01+0}{3}=0.006666\ldots.\] Correct to four decimal places, enter 0.0067. Here the three models represent the averaging distribution for the bias-variance calculation, so use denominator 3, rather than the n-1 correction for an unbiased sample-variance estimator. **Answer: 0.0067.**

First compute hidden preactivations: \(W^{(1)}x+b_1=(1-1+1,\,0+1+0)=(1,1)\). Both are positive, so Leaky ReLU leaves h=(1,1). Output logits are \(W^{(2)}h+b_2=(1,2,-1)\). Softmax gives the target class-2 probability \(p_2=e^2/(e^1+e^2+e^{-1})\approx0.705385\). With one-hot target (0,1,0), cross entropy is \(-\ln p_2\approx0.349012\), rounding to 0.35. Natural logarithms are required. **Answer: 0.35.**

The unregularised cross entropy from the forward pass is approximately 0.349012. The sum of squared entries in W1 is 4 and that in W2 is 4, hence the stipulated weight norm squared is 8. Biases are not part of this specified norm. With lambda=0.05, the penalty is \((0.05/2)\cdot8=0.20\). Therefore regularised loss is 0.349012+0.20=0.549012, rounding to 0.55. Use the original weights as requested. **Answer: 0.55.**

For \(L=\ell+\lambda\|w\|^2/2\) with lambda>0, the added term is nonnegative, so L>=ell. The SGD update is \(w\leftarrow w-\eta(\nabla\ell+\lambda w)=(1-\eta\lambda)w-\eta\nabla\ell\), showing the weight-decay term. The two gradient contributions can oppose each other, so their sum is not guaranteed larger in magnitude. Stronger regularisation favours smaller weight norms when comparing appropriately optimised solutions; it does not favour increasing them. **Answer: A, C.** A: **Correct:** The nonnegative squared-norm penalty cannot lower the objective at fixed weights. B: **Incorrect:** The regularisation gradient can cancel part of the data gradient, so update magnitude need not increase. C: **Correct:** The SGD update includes the shrinkage factor associated with L2 weight decay. D: **Incorrect:** A stronger norm penalty discourages large weights, rather than guaranteeing their increase.

Without biases, a convolution's parameter count is kernel area×input channels×output channels. Module A first reduces 128 channels to 32 using 1×1: \(1\cdot1\cdot128\cdot32=4096\). Its subsequent 3×3 layer therefore takes 32 channels and produces 128: \(3\cdot3\cdot32\cdot128=36864\). Total=4096+36864=40960. Spatial size 28×28 does not multiply the shared filter parameter count. **Answer: 40960.**

The two branches operate in parallel, so both receive all 128 original input channels. The 1×1 branch has \(1\cdot1\cdot128\cdot32=4096\) parameters. The 3×3 branch has \(3\cdot3\cdot128\cdot128=147456\). Concatenation adds no learned parameters. Thus total=4096+147456=151552. Unlike Module A, the large convolution is not preceded by the 32-channel bottleneck. **Answer: 151552.**

In Module A the 1×1 convolution reduces depth from 128 to 32 before the 3×3 convolution. The latter then needs one quarter as many input-channel weights and operations per output position. With stride 1 and same padding, spatial dimensions remain 28×28. Module B instead has parallel feature branches, and its 1×1 output is concatenated rather than feeding its 3×3 branch. Inception designs can also use bottlenecks; the distinction here is the two shown modules. **Answer: B.** A: **Incorrect:** Stride 1 and same padding preserve spatial size. B: **Correct:** Channel reduction makes the following 3×3 convolution cheaper. C: **Incorrect:** Parallel multiscale feature extraction describes the other module's branching, not this serial bottleneck's role. D: **Incorrect:** A 1×1 convolution has learned weights and changes channel depth; it is not merely a nonlinearity.