Deep Learning · Quiz 2 · Sep 2025

**1. Distinguish fit from generalization:** Training loss measures performance on examples used to learn the weights. Validation loss estimates performance on separate examples and is the relevant criterion for choosing regularization strength. **2. Apply the observed minimum:** The candidates are λ=0.001,0.01,0.1,1, and the lowest validation loss occurs at λ=0.1. Therefore choose 0.1 even though weaker regularization achieves lower training loss. **3. Explain why neither extreme is automatic:** Too little penalty can overfit; too much can underfit. The observed validation results identify the best supplied candidate rather than a universal best λ. **Answer: C — choose λ=0.1, the validation-loss minimum.** A: **Incorrect:** Lowest training loss alone can favor overfitting and ignores the validation evidence. B: **Incorrect:** A compromise is not sufficient when another supplied candidate has lower validation loss. C: **Correct:** λ=0.1 directly minimizes the observed validation loss. D: **Incorrect:** The strongest penalty can underfit and is not supported by the validation minimum.

**1. Compute the hidden activations:** The diagram’s incoming weights give \(a_1=1(2)+0.5(1)=2.5\) and \(a_2=-1(2)+1(1)=-1\). ReLU therefore gives h=(2.5,0). **2. Apply original-dropout test scaling:** With dropout probability p=0.5, the keep probability is q=0.5. Under the original convention, multiply hidden contributions by q at test time, giving output pre-activation \(z=0.5(2.5+0)=1.25\). **3. Apply the sigmoid:** \[\hat y=\frac1{1+e^{-1.25}}\approx0.77730.\] The closest listed option is 0.77; the alternatives 1.25 and 2.5 are pre-sigmoid values and cannot be sigmoid probabilities. **Answer: A — 0.77.** **Convention note:** The key uses original dropout: unscaled training activations and keep-probability scaling at test time. Inverted dropout scales during training and would give σ(2.5)≈0.924 at test time. Also, 0.7773 rounds conventionally to 0.78; 0.77 is selected only as the closest supplied option. A: **Correct:** It is the closest listed value to σ(1.25)≈0.7773 under original test-time scaling. B: **Incorrect:** It is the scaled output logit rather than the sigmoid output. C: **Incorrect:** It is the unscaled hidden contribution and exceeds the possible sigmoid range. D: **Incorrect:** It corresponds approximately to σ(2.5), obtained with inverted-dropout test behavior instead.

**1. Count a standard convolution’s weights:** A kₕ×k𝓌 filter spans all Cᵢₙ input channels. With Cₒᵤₜ filters and one bias per filter, \[P=(k_hk_wC_{in}+1)C_{out}.\] More input channels therefore increase the parameter count. **2. Separate filters from geometry:** Every filter generates one output channel, so the filter count determines output depth. Stride and padding determine how many spatial locations are evaluated but do not create additional filter weights. **Answer: A, B — The number of parameters depends on the depth of the input.; The number of filters determines the number of output channels..** A: **Correct:** A filter has a separate kernel slice for every input channel. B: **Correct:** Each filter produces one output feature map or channel. C: **Incorrect:** Changing stride reuses the same weights at different positions. D: **Incorrect:** Padding changes spatial evaluation, not the number of learned kernel entries.

**1. Define per-parameter adaptation:** An adaptive optimizer scales parameter i’s step using its own gradient history, rather than applying the same base learning-rate multiplier to every coordinate. **2. Compare AdaGrad and RMSProp:** AdaGrad accumulates \(G_{t,i}=\sum_{\tau\le t}g_{\tau,i}^2\) and divides its update by \(\sqrt{G_{t,i}}+\epsilon\). RMSProp replaces that unbounded sum by an exponential moving average of squared gradients, also giving coordinate-specific scaling. **3. Distinguish momentum methods:** Plain SGD and standard Nesterov momentum change the update direction using gradients or velocity, but they do not inherently maintain per-coordinate second-moment denominators. **Answer: C, D — Adagrad; RMSProp.** A: **Incorrect:** Plain SGD uses a common learning-rate schedule without coordinate-specific adaptive normalization. B: **Incorrect:** Standard Nesterov momentum adds look-ahead velocity, not per-parameter squared-gradient scaling. C: **Correct:** Its accumulated squared gradients produce a separate effective step size per parameter. D: **Correct:** Its moving average of squared gradients also adapts coordinate-specific step sizes.

**1. Identify overfitting:** An overfit model memorizes training-specific details and generalizes poorly. Useful countermeasures either expand representative data variation or constrain excessive reliance on particular parameters and features. **2. Explain the three direct controls:** Data augmentation creates label-preserving variations; dropout randomly removes units during training to discourage co-adaptation; L2 adds a penalty such as \(\lambda\|W\|_2^2/2\) that discourages large weights. **3. Separate optimization from regularization:** Adam changes how the objective is optimized. Merely switching SGD to Adam does not directly impose an overfitting control or guarantee better generalization. **Answer: A, B, C — Data augmentation; Dropout; L2 regularization.** A: **Correct:** Appropriate label-preserving variations help the model learn robust patterns. B: **Correct:** Randomly dropping units during training reduces dependence on specific hidden combinations. C: **Correct:** Penalizing weight magnitude constrains the fitted model. D: **Incorrect:** Adam is an optimizer, and switching to it alone is not a reliable regularization method.

**1. Write Adam’s moment estimates:** Adam starts with m₀=v₀=0 and updates \(m_t=\beta_1m_{t-1}+(1-\beta_1)g_t\), \(v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2\). The initial zeros make early averages too small. **2. Show the initialization bias:** If gradients have a constant mean μ, then \(E[m_t]=(1-\beta_1^t)\mu\), not μ. A similar factor affects the second moment. **3. Correct both estimates:** Adam uses \[\hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t}.\] This removes the zero-initialization bias, especially during early iterations. **Answer: B — To correct the underestimated moving averages of the first and second .** A: **Incorrect:** The correction addresses biased moment estimates, not a direct rule capping the learning rate. B: **Correct:** Both moving averages are initially underestimated because they start from zero. C: **Incorrect:** It does not arbitrarily increase momentum; it normalizes the initialization effect. D: **Incorrect:** Adam can use a changing schedule, and its effective coordinate steps are not kept constant.

**1. Interpret the residual pattern:** Systematic nonlinear structure in residuals means the linear model is missing a relationship. More data or scale changes alone do not make its prediction function nonlinear. **2. Add nonlinear representational capacity:** ReLU hidden layers can form piecewise-linear functions with many regions. Polynomial features of degree k>1 add terms such as x² and interactions, allowing nonlinear functions of the original inputs. **3. Check the remaining proposals:** Regularization usually restricts capacity. Standardization changes scale but preserves linearity. Multiple affine layers without nonlinear activations collapse into one affine transformation. **Answer: B, C — ReLU hidden layers and polynomial features.** A: **Incorrect:** Adding a penalty restricts coefficients rather than supplying the missing nonlinear features. B: **Correct:** ReLU hidden layers expand the set of nonlinear functions the model can represent. C: **Correct:** Polynomial powers and interactions allow curved relationships in the original variables. D: **Incorrect:** Standardization may help optimization but does not change a linear model into a nonlinear one. E: **Incorrect:** A composition of affine maps is still affine without nonlinear activations.

**1. Use convolutional parameter counting:** Ignoring biases, a convolution with kernel kₕ×k𝓌, Cᵢₙ input channels and Cₒᵤₜ filters has \(k_hk_wC_{in}C_{out}\) weights. Pooling has no learned weights. **2. Count Branches 1 and 2:** Branch 1 has \(1\cdot1\cdot192\cdot64=12288\). Branch 2 has a 1×1 reduction with \(192\cdot96=18432\), followed by a 3×3 layer with \(9\cdot96\cdot128=110592\), totaling 129024. **3. Count Branches 3 and 4:** Branch 3 has \(192\cdot16=3072\) reduction weights plus \(25\cdot16\cdot32=12800\), totaling 15872. Branch 4’s pool preserves 192 channels, so its 1×1 layer has \(192\cdot32=6144\). **4. Add all branches:** \[12288+129024+15872+6144=163328.\] The 28×28 spatial size changes computation and output shape, not the number of shared kernel weights. **Answer: 163328.** **Convention note:** With the stated no-padding operations the branches have different spatial sizes and cannot be directly concatenated as a normal Inception output. This does not change their individually well-defined parameter counts.

**1. Apply the output activation:** Let a₂ be the output pre-activation. Because the output uses ReLU, \(\hat y=\max(0,a_2)\). **2. Substitute into squared loss:** With \(L=(y-\hat y)^2\), \[L=\begin{cases}y^2,&a_2\le0,\\ (y-a_2)^2,&a_2>0.\end{cases}\] For a negative a₂ the output is zero; for a positive a₂ it equals a₂, not a fixed 1. **3. Match the supplied cases:** A gives the negative case and C gives the positive case. At a₂=0 the loss is also y², though the listed strict inequalities leave that boundary unstated. **Answer: A, C — loss y² for a₂<0 and (y−a₂)² for a₂>0.** A: **Correct:** For a₂<0, ReLU outputs zero and the loss is (y−0)². B: **Incorrect:** A positive pre-activation is passed through unchanged, not converted to 1. C: **Correct:** For a₂>0, the prediction equals a₂ and the stated loss follows. D: **Incorrect:** A and C correctly represent the two nonboundary cases.

**1. Trace the weight’s path:** w₁₂ connects x to the second hidden pre-activation a₁₂. That hidden activation connects through output weight w₂₂ to a₂, followed by the output ReLU. **2. Differentiate every link:** For squared loss, \(\partial L/\partial\hat y=-2(y-\hat y)\). ReLU contributes an indicator of a positive pre-activation. Thus \[\frac{\partial L}{\partial w_{12}}=-2(y-\hat y)\,\mathbf1(a_2>0)\,w_{22}\,\mathbf1(a_{12}>0)\,x.\] **3. Interpret zero-gradient cases:** A negative hidden or output pre-activation blocks this path. Omitting either indicator would incorrectly allow a gradient through an inactive ReLU. **Answer: B — −2(y−ŷ)·1(a₂>0)·w₂₂·1(a₁₂>0)·x.** **Convention note:** At a ReLU kink, use the conventional derivative value zero encoded by the strict-positive indicator. A: **Incorrect:** It gives only the output-loss derivative and omits the intervening layers and input. B: **Correct:** It includes the loss derivative, both ReLU gates, connecting weight and input. C: **Incorrect:** It omits both ReLU derivative gates. D: **Incorrect:** The complete chain-rule expression is already present in B.

**1. Perform the forward pass:** Initially w₁₁=1,w₁₂=0,w₂₁=w₂₂=1, with x=1 and y=0. Hidden ELU with α=1 gives h₁=ELU(1)=1 and h₂=ELU(0)=0. The output ReLU therefore receives a₂=1 and returns ŷ=1. **2. Differentiate ELU at zero:** \(ELU(z)=z\) for z≥0 and \(e^z-1\) for z<0. Both one-sided derivatives at zero equal 1. The output ReLU derivative at a₂=1 is also 1. **3. Compute and apply the gradient:** \[\frac{\partial L}{\partial w_{12}}=2(\hat y-y)\cdot1\cdot w_{22}\cdot ELU\prime(0)\cdot x=2.\] With η=0.2, \(w_{12,new}=0-0.2(2)=-0.4\). **Answer: -0.4.**

**1. Slide the kernel without flipping:** CNN convolution here means cross-correlation. A 4×4 input with a 3×3 kernel, stride 1 and no padding gives a 2×2 pre-activation map. The kernel’s nonzero positions are (1,1),(1,2),(2,2),(3,1),(3,3). **2. Calculate all four window sums:** The top-left sum is 2+0−1+0+2=3; top-right is 0+1−2+1−1=−1; bottom-left is 1−1+1+3+1=5; bottom-right is −1−2+2+0+2=1. Thus \[Z=\begin{pmatrix}3&-1\\5&1\end{pmatrix}.\] **3. Apply the convolutional ReLU:** Negative entries become zero, giving \[H=\begin{pmatrix}3&0\\5&1\end{pmatrix}.\] This activated map, rather than Z, enters the pooling layer. **Answer: B — pooling input matrix with rows (3,0), (5,1).** A: **Incorrect:** The top-left and bottom-left values do not match the kernel-window sums. B: **Correct:** It is the correctly computed convolution followed by ReLU. C: **Incorrect:** It leaves the negative pre-activation unchanged despite the intervening ReLU. D: **Incorrect:** Its top-left value 5 should be 3.

**1. Slide the kernel without flipping:** CNN convolution here means cross-correlation. A 4×4 input with a 3×3 kernel, stride 1 and no padding gives a 2×2 pre-activation map. The kernel’s nonzero positions are (1,1),(1,2),(2,2),(3,1),(3,3). **2. Calculate all four window sums:** The top-left sum is 2+0−1+0+2=3; top-right is 0+1−2+1−1=−1; bottom-left is 1−1+1+3+1=5; bottom-right is −1−2+2+0+2=1. Thus \[Z=\begin{pmatrix}3&-1\\5&1\end{pmatrix}.\] **3. Apply the convolutional ReLU:** Negative entries become zero, giving \[H=\begin{pmatrix}3&0\\5&1\end{pmatrix}.\] This activated map, rather than Z, enters the pooling layer. **4. Average the pooling window:** One 2×2 window covers the entire activated map. Average pooling divides the sum by four: \[\hat y=(3+0+5+1)/4=9/4=2.25.\] **Answer: 2.25.**

**1. Slide the kernel without flipping:** CNN convolution here means cross-correlation. A 4×4 input with a 3×3 kernel, stride 1 and no padding gives a 2×2 pre-activation map. The kernel’s nonzero positions are (1,1),(1,2),(2,2),(3,1),(3,3). **2. Calculate all four window sums:** The top-left sum is 2+0−1+0+2=3; top-right is 0+1−2+1−1=−1; bottom-left is 1−1+1+3+1=5; bottom-right is −1−2+2+0+2=1. Thus \[Z=\begin{pmatrix}3&-1\\5&1\end{pmatrix}.\] **3. Apply the convolutional ReLU:** Negative entries become zero, giving \[H=\begin{pmatrix}3&0\\5&1\end{pmatrix}.\] This activated map, rather than Z, enters the pooling layer. **4. Evaluate prediction and loss:** Average pooling gives ŷ=9/4=2.25, while the target is y=1/4=0.25. The specified squared loss is \[L=(\hat y-y)^2=(2.25-0.25)^2=2^2=4.00.\] There is no 1/2 factor in the given objective. **Answer: 4.00.**