Deep Learning · Quiz 2 · Jan 2025

**1. Take the negative-input limit:** For \(\sigma(z)=1/(1+e^{-z})\), a very negative z makes e⁻ᶻ very large, so σ(z) approaches 0. **2. Calculate the activation derivative:** \[\sigma\prime(z)=\sigma(z)(1-\sigma(z)).\] As σ(z)→0, this product also approaches 0, making the neuron locally insensitive to changes in its input. **3. Connect to gradient flow:** This is negative-tail sigmoid saturation. Its small local derivative can shrink a backpropagated signal, though the full loss-gradient expression also depends on the chosen loss. **Answer: A — The output approaches 0 and the gradient approaches 0..** A: **Correct:** Both the output and its local derivative approach zero in the negative tail. B: **Incorrect:** Output 0.5 and derivative 0.25 occur at input zero. C: **Incorrect:** Output approaching one describes the positive tail instead. D: **Incorrect:** The sigmoid derivative never reaches one; its maximum is 0.25.

**1. Write Adam’s moment estimates:** Adam starts with m₀=v₀=0 and updates \(m_t=\beta_1m_{t-1}+(1-\beta_1)g_t\), \(v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2\). The initial zeros make early averages too small. **2. Show the initialization bias:** If gradients have a constant mean μ, then \(E[m_t]=(1-\beta_1^t)\mu\), not μ. A similar factor affects the second moment. **3. Correct both estimates:** Adam uses \[\hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t}.\] This removes the zero-initialization bias, especially during early iterations. **Answer: B — To correct the underestimated moving averages of the first and second .** A: **Incorrect:** The correction addresses biased moment estimates, not a direct rule capping the learning rate. B: **Correct:** Both moving averages are initially underestimated because they start from zero. C: **Incorrect:** It does not arbitrarily increase momentum; it normalizes the initialization effect. D: **Incorrect:** Adam can use a changing schedule, and its effective coordinate steps are not kept constant.

**1. Write a residual block:** A skip connection carries x around a learned transformation F and adds it to the branch output: \[y=x+F(x).\] Addition is elementwise when shapes match; a learned projection can align differing dimensions. **2. Explain the benefit:** The branch can learn a correction to x rather than rebuilding the entire mapping. The identity path also contributes an additive gradient path, since the Jacobian is I+∂F/∂x. **Answer: A — By incorporating element-wise addition of the input of a layer to its output.** A: **Correct:** It describes the standard residual addition with an identity or dimension-matching bypass. B: **Incorrect:** Concatenation joins features along an axis and is a different connection pattern. C: **Incorrect:** Elementwise multiplication is not the standard ResNet residual operation. D: **Incorrect:** The bypass preserves and adds the input rather than removing it.

**1. State the L2 penalty:** A typical regularized objective is \[J(W)=L_{data}(W)+\frac\lambda2\sum_jw_j^2,\qquad\lambda>0.\] Large weights increase the penalty quadratically. **2. Explain its effect:** The penalty discourages unnecessarily large coefficients and can reduce overfitting. Its gradient contribution λw tends to shrink weights toward zero but does not generally make them exactly zero. **3. Distinguish L1:** The sum of absolute values is L1 regularization, which more directly encourages exact sparsity. **Answer: C — To prevent overfitting by adding the sum of squared values of weights to the .** A: **Incorrect:** Absolute-value penalties define L1 rather than L2. B: **Incorrect:** Exact sparsity is more characteristic of L1, not the principal L2 mechanism. C: **Correct:** It identifies the sum-of-squared-weights penalty used to constrain fitting. D: **Incorrect:** L2 penalizes larger magnitudes, not small weights to increase complexity.

**1. Contrast a hard target:** A hard multiclass label is usually one-hot: one class receives probability 1 and all others 0. **2. Define a soft target:** A soft target assigns fractional probability mass across classes, for example (0.9,0.025,0.025,0.025,0.025). Label smoothing is one way to create such targets and discourage excessive certainty. **3. Apply the definition:** The important feature is a continuous or probabilistic target distribution rather than a single discrete class indicator. It is not necessarily equal to the model’s own prediction. **Answer: B — A target value that is continuous or probabilistic rather than discrete.** A: **Incorrect:** This describes a hard binary indicator rather than the requested soft target. B: **Correct:** Fractional target values or class probabilities are the defining distinction. C: **Incorrect:** Arbitrary random relabeling is not the definition of a soft distribution. D: **Incorrect:** Soft targets need not be copied from the current prediction.

**1. Find the mean prediction at each test point:** Across A,B,C, the four prediction triples are (1,3,2),(4,5,3),(7,6,5),(8,9,7). Their means are (2,4,6,8). **2. Measure variability across models:** At every test point the deviations from its mean are a permutation of (−1,0,1). Thus the finite-ensemble variance, using three models as the distribution, is \[Var_j=\frac{(-1)^2+0^2+1^2}{3}=\frac23.\] **3. Average across test points:** All four variances equal 2/3, so their mean is 2/3=0.666666…, giving 0.67. True labels are needed for bias or prediction error, not this between-model variance. **Answer: 0.67.** **Convention note:** Use the population variance across the three supplied models, with denominator 3. An unbiased sample-variance estimate would divide by 2 and is a different convention.

**1. Identify the moving-average rule:** \(v_t=\beta v_{t-1}+(1-\beta)g_t\), with β=0.9,v₋₁=0 and gradients g₀=5,g₁=3,g₂=4. This is the raw exponential average, not a bias-corrected Adam estimate. **2. Update sequentially:** \(v_0=0.9(0)+0.1(5)=0.5\). Then \(v_1=0.9(0.5)+0.1(3)=0.45+0.3=0.75\). **3. Compute the requested step:** \[v_2=0.9(0.75)+0.1(4)=0.675+0.4=1.075.\] Earlier averages must be included rather than replaced by the most recent gradient. **Answer: 1.075.**

**1. Count valid kernel positions:** A 4×4 input and a 3×3 kernel at stride 1 without padding give a 2×2 convolution map. The kernel is used unflipped, as in standard CNN cross-correlation. **2. Calculate the four sums:** The top-left window gives 1−2+0−1+1=−1; top-right gives 0+1+1−1+0=1; bottom-left gives 1+0+1+0+0=2; bottom-right gives 2+1+0+1+1=5. Therefore \[A=\begin{pmatrix}-1&1\\2&5\end{pmatrix}.\] No ReLU is specified before the averaging operation. **Answer: A — A with rows (−1,1), (2,5).** A: **Correct:** It matches the four kernel-window sums. B: **Incorrect:** Convolution sums products and does not divide them by nine. C: **Incorrect:** The convolution produces a 2×2 map rather than a single scalar. D: **Incorrect:** Its bottom-row and top-right signs do not match the dot products.

**1. Count valid kernel positions:** A 4×4 input and a 3×3 kernel at stride 1 without padding give a 2×2 convolution map. The kernel is used unflipped, as in standard CNN cross-correlation. **2. Calculate the four sums:** The top-left window gives 1−2+0−1+1=−1; top-right gives 0+1+1−1+0=1; bottom-left gives 1+0+1+0+0=2; bottom-right gives 2+1+0+1+1=5. Therefore \[A=\begin{pmatrix}-1&1\\2&5\end{pmatrix}.\] No ReLU is specified before the averaging operation. **3. Average the entire map:** A 2×2 average-pooling window covers all four entries, giving \[B=(-1+1+2+5)/4=7/4=1.75.\] **Answer: C — B=1.75.** A: **Incorrect:** It repeats the pre-pooling map rather than pooling it. B: **Incorrect:** It scales separate entries and does not average the 2×2 window. C: **Correct:** The single pooling window has arithmetic mean 1.75. D: **Incorrect:** It changes entry signs and does not compute the specified average.

**1. Count valid kernel positions:** A 4×4 input and a 3×3 kernel at stride 1 without padding give a 2×2 convolution map. The kernel is used unflipped, as in standard CNN cross-correlation. **2. Calculate the four sums:** The top-left window gives 1−2+0−1+1=−1; top-right gives 0+1+1−1+0=1; bottom-left gives 1+0+1+0+0=2; bottom-right gives 2+1+0+1+1=5. Therefore \[A=\begin{pmatrix}-1&1\\2&5\end{pmatrix}.\] No ReLU is specified before the averaging operation. **3. Compute the sigmoid derivative:** Pooling gives B=1.75 and ŷ=σ(1.75)≈0.851952802. Thus \(d\hat y/dB=\hat y(1-\hat y)\approx0.126129225\). **4. Differentiate the shared top-left kernel weight:** K₀₀ multiplies input values 1,0,1,2 across the four windows. Pooling averages their derivatives, giving \(\partial B/\partial K_{00}=(1+0+1+2)/4=1\). **5. Apply the supplied upstream loss gradient:** \[\frac{\partial L}{\partial K_{00}}=(-1)(0.126129225)(1)=-0.126129225\ldots.\] To three decimals, enter −0.126. **Answer: -0.126.**

**1. Define AdaGrad’s quantities:** For \(L(w)=w^3+2w\), the gradient is g(w)=3w²+2. The supplied rules are \(v_t=v_{t-1}+g_t^2\) and \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=0.5,v₋₁=0,ε=0. **2. Perform the first update:** At w₀=2, g₀=3(4)+2=14 and v₀=196. Therefore \(w_1=2-0.5(14)/14=1.5\). **3. Report the first updated parameter:** w₁=1.50. The first positive gradient is normalized by its own magnitude because no earlier squared gradients have accumulated. **Answer: 1.50.**

**1. Define AdaGrad’s quantities:** For \(L(w)=w^3+2w\), the gradient is g(w)=3w²+2. The supplied rules are \(v_t=v_{t-1}+g_t^2\) and \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=0.5,v₋₁=0,ε=0. **2. Perform the first update:** At w₀=2, g₀=3(4)+2=14 and v₀=196. Therefore \(w_1=2-0.5(14)/14=1.5\). **3. Accumulate the next squared gradient:** At w₁=1.5, g₁=3(1.5²)+2=8.75. Thus \[v_1=v_0+g_1^2=196+8.75^2=196+76.5625=272.5625\approx272.56.\] **Answer: 272.56.** **Convention note:** The question labels this accumulator v₁; computing it requires evaluating the gradient at w₁ after the first parameter update, following the explicit index convention.

**1. Define AdaGrad’s quantities:** For \(L(w)=w^3+2w\), the gradient is g(w)=3w²+2. The supplied rules are \(v_t=v_{t-1}+g_t^2\) and \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=0.5,v₋₁=0,ε=0. **2. Perform the first update:** At w₀=2, g₀=3(4)+2=14 and v₀=196. Therefore \(w_1=2-0.5(14)/14=1.5\). **3. Evaluate the gradient at the new point:** \[g_1=3w_1^2+2=3(1.5)^2+2=6.75+2=8.75.\] This is a derivative, not the new parameter or the squared-gradient accumulator. **Answer: 8.75.**

**1. Define AdaGrad’s quantities:** For \(L(w)=w^3+2w\), the gradient is g(w)=3w²+2. The supplied rules are \(v_t=v_{t-1}+g_t^2\) and \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=0.5,v₋₁=0,ε=0. **2. Perform the first update:** At w₀=2, g₀=3(4)+2=14 and v₀=196. Therefore \(w_1=2-0.5(14)/14=1.5\). **3. Update the accumulator at w₁:** g₁=8.75 and v₁=196+8.75²=272.5625, so √v₁≈16.509467. **4. Perform the second parameter step:** \[w_2=1.5-\frac{0.5(8.75)}{\sqrt{272.5625}}\approx1.23500053.\] To two decimal places, enter 1.24; retain the extra digits until the final rounding. **Answer: 1.24.** **Convention note:** This value is just above the rounding midpoint 1.235. Keeping the full square root gives 1.24 rather than deciding from a prematurely rounded intermediate.

**1. Determine the direct 1×1 branch shape:** A 28×28×256 input and 128 filters of size 1×1, stride 1 and padding 0 give \(N_{out}=(28-1)+1=28\). Each filter produces one channel. **2. Compute feature-map volume:** The branch output is 28×28×128, so \[V=28\cdot28\cdot128=784(128)=100352\text{ activations}.\] This counts activations, not parameters. **Answer: 100352.**

**1. Count one filter:** A 1×1 filter spans all 256 input channels, requiring 256 weights. **2. Count all filters without biases:** \[P=1\cdot1\cdot256\cdot128=32768.\] The 28×28 image positions reuse these weights and do not multiply the parameter count. **Answer: 32768.**

**1. Propagate through the 1×1 reduction:** Layer (1.1) maps 28×28×256 to 28×28×128 using 128 filters. **2. Propagate through the padded 3×3 layer:** Layer (1.2) has 192 filters, stride 1 and padding 1. Its side length is \((28+2-3)+1=28\), giving shape 28×28×192. **3. Multiply the dimensions:** \[V=28\cdot28\cdot192=784(192)=150528.\] **Answer: 150528.**

**1. Count reduction weights:** Layer (1.1) has \(1\cdot1\cdot256\cdot128=32768\) weights. **2. Use the reduced input depth for the next layer:** Layer (1.2) receives 128 channels, so its 192 filters of spatial size 3×3 have \(3\cdot3\cdot128\cdot192=221184\) weights. **3. Add both layers:** \[P=32768+221184=253952.\] Biases are explicitly excluded. The 1×1 bottleneck reduces the cost of the subsequent 3×3 operation. **Answer: 253952.**

**1. Apply the 1×1 reduction:** Layer (2.1) produces 28×28×32, preserving spatial size and setting channel count to 32. **2. Apply the padded 5×5 convolution:** With stride 1 and padding 2, the side length is \((28+4-5)+1=28\). The 96 filters give 96 output channels. **3. State the shape:** The branch output is 28×28×96. Padding exactly compensates for the four-pixel spatial reduction of a valid 5×5 convolution. **Answer: A — 28 X 28 X 96.** A: **Correct:** It preserves 28×28 and uses the final layer’s 96 filters. B: **Incorrect:** It ignores the stated padding 2 in the 5×5 layer. C: **Incorrect:** It uses an incorrect output channel count. D: **Incorrect:** The layers do not expand the spatial size to 32 or output 128 channels.

**1. Count the bottleneck:** The 32 filters in layer (2.1) each have 1×1×256 weights, giving 8192. **2. Count the 5×5 filters:** Layer (2.2) takes 32 input channels and has 96 filters, giving \(25\cdot32\cdot96=76800\) weights. **3. Sum:** \[P=8192+76800=84992.\] The 5×5 filters span the reduced depth 32, not the original depth 256. **Answer: 84992.**

**1. Calculate the pooling spatial output:** A 3×3 pool with stride 1 and padding 1 maps a side of 28 to \((28+2-3)+1=28\). **2. Preserve channel depth:** Max pooling operates independently within each input channel. It does not combine or create learned channels, so the original depth 256 is unchanged. **3. State the result:** The output of layer (3.1), before the later 1×1 projection, has shape 28×28×256. **Answer: D — 28 X 28 X 256.** A: **Incorrect:** 3×3 is the window size, not the complete output spatial size. B: **Incorrect:** Pooling does not collapse 256 channels to one. C: **Incorrect:** It incorrectly removes padding and uses the later branch projection’s 64 channels. D: **Correct:** Spatial size and channel depth both remain unchanged under these pooling settings. E: **Incorrect:** Input size, filter size, stride and padding are all given.

**1. Identify what max pooling does:** For each specified window it returns the largest activation. The window size, stride and padding are fixed hyperparameters. **2. Count learned coefficients:** There is no kernel weight, learned bias or trainable combination in a standard max-pooling operation. Consequently its trainable parameter count is 0, even though it produces many output activations. **Answer: 0.**

**1. List final branch outputs:** The direct 1×1 branch has 128 channels. The 1×1→3×3 branch ends with 192, the 1×1→5×5 branch with 96, and the pooling→1×1 branch with 64. **2. Concatenate along channels:** All branches have the same 28×28 spatial size. Filter concatenation joins channels rather than adding corresponding entries, so \[C_{out}=128+192+96+64=480.\] **3. State the full shape:** The concatenated module output is 28×28×480. **Answer: C — 28 X 28 X 480.** A: **Incorrect:** 192 is only the output depth of the 3×3 branch. B: **Incorrect:** 256 repeats the input depth rather than adding branch output depths. C: **Correct:** It sums all four output channel counts while keeping their common spatial size. D: **Incorrect:** All branch output shapes are determined by the stated padding and filter counts.