Deep Learning · Quiz 2 · May 2025

**1. Read the model retained at epoch 50:** M₁ ends with low training and validation loss close together, suggesting low bias and low variance (C). M₂’s losses rise rather than settle, consistent with an excessively large learning rate (A). **2. Distinguish the remaining end states:** M₃ has very low training loss but its validation loss turns upward and ends much larger, the overfitting pattern of low bias and high variance (B). M₄ levels off with appreciable training loss and no increasing validation divergence, the intended underfitting/high-bias pattern (D). **3. Match the four assignments:** The most appropriate pairing is M₁→C, M₂→A, M₃→B, M₄→D. Stopping M₃ earlier could help, but the question explicitly retains the epoch-50 model. **Answer: A — M₁→C, M₂→A, M₃→B, M₄→D.** A: **Correct:** It matches convergence, divergence, overfitting and underfitting to the four shown curves. B: **Incorrect:** It swaps the well-fitted M₁ with underfitting and the overfit M₃ with low variance. C: **Incorrect:** M₁ is converging rather than diverging, and M₃ shows overfitting. D: **Incorrect:** M₂’s rising losses are not a low-bias/low-variance fit, and M₄ does not show the learning-rate instability.

**1. Define adaptive coordinate steps:** Adaptive methods use a parameter’s own gradient statistics to scale its update. This differs from giving every coordinate the same base learning-rate multiplier. **2. Identify Adam and RMSProp:** RMSProp divides by a root moving average of squared gradients. Adam also uses a second-moment denominator, together with a signed first-moment estimate and bias corrections. Both therefore adapt effective per-parameter step sizes. **3. Contrast ordinary momentum:** Standard momentum GD and Nesterov acceleration maintain a velocity or look-ahead direction but do not by themselves normalize each coordinate using its squared-gradient history. **Answer: A, B — ADAM; RMSprop.** A: **Correct:** Its first- and second-moment estimates give adaptive coordinate updates. B: **Correct:** Its squared-gradient moving average adapts each coordinate’s effective scale. C: **Incorrect:** Standard momentum changes the update direction, not through adaptive per-coordinate normalization. D: **Incorrect:** Standard Nesterov acceleration uses look-ahead momentum rather than an adaptive second-moment denominator.

**1. Express both weight shapes:** Let the input and output have d neurons and the hidden layer h=0.3d. The encoder has hd weights and the decoder has dh, with biases excluded. **2. Compare independent storage:** Untied matrices need \(n_1=hd+dh=2hd\) stored parameters. If the decoder equals the encoder transpose, it is determined by the encoder, so efficient storage needs only \(n_2=hd\). **3. Form the ratio:** \[\frac{n_1}{n_2}=\frac{2hd}{hd}=2.\] The 30% compression affects both counts equally and cancels. **Answer: 2.**

**1. Use cross-entropy for a soft target:** With smoothed label q and predicted probabilities p, \[L=-\sum_{j=1}^5q_j\ln p_j.\] All five target components contribute; the label is no longer strictly one-hot. **2. Substitute the values:** The true-class contribution is −0.9 ln(0.6). The four other classes each contribute −0.025 ln(0.1). Thus \[L=-0.9\ln0.6-4(0.025)\ln0.1.\] **3. Evaluate:** The terms are approximately 0.45974306 and 0.23025851, giving L=0.69000157. To two decimal places, enter 0.69. **Answer: 0.69.** **Convention note:** The export’s upper acceptance bound is printed as 74, apparently from a missing decimal point. The answer is independently computed using natural logarithms.

**1. Calculate spatial size:** A convolution output side is \[N_{out}=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1.\] Here N=64,P=2,F=5,S=1, so Nₒᵤₜ=(64+4−5)+1=64. **2. Calculate output depth:** Each of the 16 kernels spans all eight input channels and produces one output channel. Hence the output shape is 64×64×16. **Answer: A — 64 × 64 × 16.** A: **Correct:** It has the correct padded spatial size and one channel per filter. B: **Incorrect:** It does not apply the stated padding in the spatial calculation. C: **Incorrect:** It keeps input depth instead of using the 16 filters. D: **Incorrect:** Neither the stride-1 spatial size nor the output channel count matches. E: **Incorrect:** Stride 1 with padding 2 preserves 64 rather than halving it.

**1. Count one kernel across channels:** A 5×5 filter applied to an eight-channel input has 5×5×8=200 weights. The channel dimension must not be omitted. **2. Count all filters:** There are 16 such filters and no biases, so \[P=5\cdot5\cdot8\cdot16=200(16)=3200.\] Stride, padding and image area do not add separate learned coefficients. **Answer: 3200.**

**1. Identify what changes:** The input becomes 128×128×8, so its spatial area increases fourfold but its eight-channel depth is unchanged. **2. Use weight sharing:** The same 16 filters of shape 5×5×8 are applied at every spatial position. Therefore \(P=5\cdot5\cdot8\cdot16=3200\) remains unchanged. **3. Separate parameters from computation:** A larger image causes more filter applications and more activation storage. Those quantities can grow even when the learned weights do not. **Answer: D — No, it will remain the same..** A: **Incorrect:** Doubling a side length does not duplicate shared kernel weights. B: **Incorrect:** The same filter tensors are still required, so the parameter count does not halve. C: **Incorrect:** The number of spatial evaluations can quadruple, but the stored filter weights do not. D: **Correct:** Kernel sizes, input depth and filter count stay the same.

**1. Complete the square:** The unregularized objective is \[L_D=w_1^2+w_2^2-2w_1-2w_2+2=(w_1-1)^2+(w_2-1)^2.\] **2. Minimize both nonnegative terms:** Each square is zero exactly when its coordinate equals 1. Hence the unique minimizer is w*ᴰ=(1,1), and the minimum loss is zero. **Answer: A — (1, 1).** A: **Correct:** Both squared terms vanish at (1,1). B: **Incorrect:** It gives loss 2 rather than the attainable minimum zero. C: **Incorrect:** It gives loss 8, well above zero. D: **Incorrect:** Its second-coordinate square remains 4. E: **Incorrect:** Its first-coordinate square remains 4.

**1. Add the L2 penalty:** \(L=L_D+\frac\lambda2(w_1^2+w_2^2)\). Differentiation gives \[\frac{\partial L}{\partial w_j}=(2+\lambda)w_j-2.\] **2. Solve the optimum condition:** Setting the derivative to zero yields \(w_j^*=2/(2+\lambda)\). The Hessian (2+λ)I is positive for λ>0, so this stationary point is the unique minimum. **3. Use the specified minimizer:** If w*=(0.5,0.5), then 0.5=2/(2+λ), so 2+λ=4 and λ=2. **Answer: 2.**

**1. Compare the exact minimizers:** The unregularized solution is w*ᴰ=(1,1). With the specified L2 penalty, \(w^*=\frac2{2+\lambda}(1,1)\). **2. Compare norms:** \[\|w^*\|_2=\frac2{2+\lambda}\sqrt2.\] For every λ>0, the multiplier lies strictly between 0 and 1, so the regularized norm is strictly below the unregularized norm √2. **3. Answer the existence question:** No positive λ makes the regularized solution larger in L2 norm. In this explicit quadratic example the regularizer shrinks both coordinates by the same factor. **Answer: D — No, there is no such λ> 0 for which this happens..** A: **Incorrect:** No such λ exists, so even a single example cannot be supplied. B: **Incorrect:** The inequality fails for every positive λ. C: **Incorrect:** There are zero such values, not a nonempty finite collection. D: **Correct:** The exact shrinkage factor is less than one for all λ>0.

**1. Use the positive-weight condition:** The input is x=1, and all n weights equal k>0. Every hidden pre-activation stays positive, so all n−1 ReLU units act as the identity. **2. Propagate through every layer:** After the first hidden layer the value is k, after the second k², and after the last hidden layer kⁿ⁻¹. The final linear output multiplies once more by k. **3. State the result:** \[\hat y=\prod_{i=1}^nw_i=k^n.\] The output weight must be counted even though its activation is linear. **Answer: A — ŷ=kⁿ.** A: **Correct:** There are n multiplicative weights, all passed through active hidden units. B: **Incorrect:** No hidden ReLU is inactive because all preceding values and weights are positive. C: **Incorrect:** It omits the final output-layer multiplication by k.

**1. Differentiate before equating independent weights:** In the active region, \(\hat y=\prod_{i=1}^nw_i\), so \(\partial\hat y/\partial w_1=\prod_{i=2}^nw_i\). We differentiate one weight, not a tied common parameter. **2. Include the loss derivative:** With y=0 and \(L=\frac12(\hat y-y)^2\), \(\partial L/\partial\hat y=\hat y\). Therefore \[\frac{\partial L}{\partial w_1}=\hat y\prod_{i=2}^nw_i.\] **3. Evaluate all weights at k:** Substituting gives \(k^n k^{n-1}=k^{2n-1}\). An extra factor n would arise for a single tied parameter controlling all weights, which is not asked. **Answer: A — ∂L/∂w₁=k^(2n−1).** A: **Correct:** It multiplies the output kⁿ by the derivative kⁿ⁻¹. B: **Incorrect:** The ReLU path is active, so the gradient is not zero. C: **Incorrect:** It misses the reduction of one exponent when differentiating w₁. D: **Incorrect:** It omits the derivative of the network output. E: **Incorrect:** It omits the loss derivative factor kⁿ.

**1. Keep the same positive input x=1:** Every weight is nonzero. If all hidden-layer weights were positive, every hidden activation would remain positive. The nonzero final linear weight would then produce a nonzero output. **2. Explain how zero can arise:** A negative hidden-layer weight multiplies a previously positive activation to a negative value, and its ReLU sets that path to zero. Once zero appears, all later bias-free scalar layers retain zero. **3. Infer the necessary condition:** Thus ŷ=0 requires at least one negative hidden-layer weight wⱼ with j<n. It does not require every hidden weight to be negative, and it does not require any positive weight. **Answer: A — some hidden-layer weight wⱼ<0 with j<n.** **Convention note:** The passage fixes x=1. If x=0 were allowed instead, zero output would not imply a negative hidden weight. A: **Correct:** At least one hidden ReLU must block the initially positive signal, requiring a negative hidden weight. B: **Incorrect:** One negative hidden weight can suffice; all hidden weights need not be negative. C: **Incorrect:** All weights can be negative, with the first hidden ReLU already producing zero.

**1. Compute the four valid convolution windows:** The supplied 4×4 input and 3×3 kernel use stride 1 and no padding, so the output is 2×2. The kernel’s nonzero positions are (1,2),(1,3),(2,1),(2,3),(3,2), using one-based positions. **2. Show the window sums:** Top-left: 0+2+0+1+2=5. Top-right: 2+0+1+0+0=3. Bottom-left: 1+1+0+0+1=3. Bottom-right: 1+0+2+1+0=4. Every value is positive, so ReLU leaves them unchanged: \[H=\begin{pmatrix}5&3\\3&4\end{pmatrix}.\] **Answer: A — the 2×2 matrix with rows (5,3), (3,4).** A: **Correct:** It matches all four window sums and the unchanged positive ReLU outputs. B: **Incorrect:** Its top-right, bottom-left and bottom-right values do not match the convolution. C: **Incorrect:** A valid 4×4 by 3×3 convolution produces a 2×2 map, not 3×3. D: **Incorrect:** Its entries do not match the specified window products.

**1. Compute the four valid convolution windows:** The supplied 4×4 input and 3×3 kernel use stride 1 and no padding, so the output is 2×2. The kernel’s nonzero positions are (1,2),(1,3),(2,1),(2,3),(3,2), using one-based positions. **2. Show the window sums:** Top-left: 0+2+0+1+2=5. Top-right: 2+0+1+0+0=3. Bottom-left: 1+1+0+0+1=3. Bottom-right: 1+0+2+1+0=4. Every value is positive, so ReLU leaves them unchanged: \[H=\begin{pmatrix}5&3\\3&4\end{pmatrix}.\] **3. Take the max-pooling output:** The single 2×2 pooling window covers all four values. Its maximum is 5, so the scalar prediction is ŷ=5. Max pooling is a maximum operation, not an average or sum. **Answer: 5.**