Deep Learning · Quiz 2 · May 2026

**1. Identify what changes:** The polynomial degree remains 25, so the hypothesis class and its ability to represent the underlying relationship do not change. The change is an increase in independent training observations. **2. Explain variance reduction:** With few observations, fitted coefficients can change substantially between training samples. More representative data usually stabilizes these estimates and reduces this sampling variance. **3. Separate bias from noise:** Model bias is primarily associated with the chosen model class and learning procedure, so it stays approximately unchanged in the intended comparison. More data does not eliminate irreducible observation noise. **Answer: B — Variance will tend to decrease, while bias remains approximately the same..** **Convention note:** This is the usual bias–variance tendency with representative independent data and a fixed learning procedure, rather than an unconditional monotonicity theorem for every finite dataset. A: **Incorrect:** Keeping the degree fixed does not primarily remove approximation bias; more data principally stabilizes fitting. B: **Correct:** More independent data generally reduces sensitivity to the particular training sample while leaving model-class bias approximately unchanged. C: **Incorrect:** Neither a fixed model class nor additional observations guarantee zero bias, zero variance or zero irreducible noise. D: **Incorrect:** Training-set size can substantially affect estimation variance, especially for a flexible degree-25 model.

**1. Understand the cycle:** A triangular cyclical learning rate rises from \(\eta_{\min}=0.001\) to \(\eta_{\max}=0.1\) over \(\mu=500\) iterations, then falls over another 500. Its full period is therefore \(2\mu=1000\). **2. Locate the minimum:** At \(t=0,1000,2000,\ldots\), the triangular multiplier is zero, giving \(\eta_t=0.001\). At \(t=500,1500,\ldots\), that multiplier is one and the rate is 0.1. **3. Check repetition:** Because the floor term restarts the triangle every 1000 iterations, the minimum repeats rather than occurring only at initialization. **Answer: B — Minimum at t = 0, 1000, 2000, ….** A: **Incorrect:** The initial iteration is a minimum, but the same minimum recurs every full cycle. B: **Correct:** Every multiple of the full period 1000 reaches the minimum rate 0.001. C: **Incorrect:** The halfway points 500,1500,… are maxima, where the rate is 0.1. D: **Incorrect:** The rate varies along the triangle and equals its minimum only at cycle boundaries.

**1. Write the RMSProp accumulator:** With \(v_0=0\), \(\beta=0.9\), and gradients \(10,0,0\), use \(v_t=0.9v_{t-1}+0.1g_t^2\). The effective rate is \(\eta/\sqrt{v_t}\), because the given \(\epsilon=0\). **2. Trace all three iterations:** \(v_1=0.9(0)+0.1(10^2)=10\); \(v_2=0.9(10)+0.1(0)=9\); and \(v_3=0.9(9)+0.1(0)=8.1\). The old squared gradient is gradually forgotten. **3. Substitute and round:** \[\eta_{\mathrm{eff},3}=\frac{0.01}{\sqrt{8.1}}=0.00351364\ldots\approx0.0035.\] This is the rate multiplier; the actual third parameter update is zero since \(g_3=0\). **Answer: 0.0035.**

**1. Use AdaGrad’s cumulative sum:** Unlike RMSProp’s moving average, AdaGrad uses \(G_t=G_{t-1}+g_t^2\). For gradients 10,0,0 and \(G_0=0\), the successive sums are 100,100,100. **2. Calculate each effective rate:** With base rate 0.01 and zero numerical stabilizer, \(\eta_{\mathrm{eff},t}=0.01/\sqrt{100}=0.001\) at all three iterations. **3. Distinguish rate from movement:** The denominator does not decay between iterations, so the rate cannot increase here. The second and third updates are nevertheless zero because their gradients are zero. **Answer: C — The effective learning rate stays constant at 0.001.** A: **Incorrect:** AdaGrad retains the accumulated value 100; it does not forget it and increase the rate. B: **Incorrect:** The two additional squared gradients are zero, so the denominator does not grow after iteration 1. C: **Correct:** The accumulated squared gradient is 100 throughout, giving an unchanged effective rate of 0.001. D: **Incorrect:** Rates approach zero only if the cumulative squared gradients keep growing; they do not grow in this example.

**1. Substitute the ensemble size:** The supplied expression is \(\operatorname{MSE}=V/k+(k-1)C/k\). With \(V=12\) and \(k=2\), it becomes \(6+C/2\). **2. Minimize over covariance:** The coefficient of \(C\) is positive, so the smallest permitted covariance produces the smallest error. The allowed interval is \([-V,V]=[-12,12]\), whose lower endpoint is \(-12\). **3. Check the result:** At \(C=-12\), the expression is \(6-6=0\). This corresponds to two equal-variance errors cancelling through perfect negative correlation. **Answer: B — C = −V = −12.** **Convention note:** The calculation minimizes the MSE expression supplied in the question. A separate shared bias term, if present in a more general decomposition, would not be cancelled by averaging. A: **Incorrect:** Positive covariance reinforces shared errors rather than minimizing the expression. B: **Correct:** The lowest allowed covariance, −V = −12, minimizes the linear expression in C. C: **Incorrect:** Zero covariance gives MSE 6, which exceeds the zero value at C = −12. D: **Incorrect:** C = V/2 gives MSE 6 + 6/2 = 9, exceeding the zero at C = −V.

**1. Rearrange the expression:** Write \(V/k+(k-1)C_0/k=C_0+(V-C_0)/k\). This separates the shared covariance component from the part reduced by ensemble size. **2. Take the limit:** For fixed finite \(V\) and \(C_0>0\), \((V-C_0)/k\to0\) as \(k\to\infty\). Thus the limiting error is \(C_0\). **3. Interpret the floor:** Adding models averages away unshared variation, but correlated errors remain. An arbitrarily large ensemble does not remove the fixed positive covariance. **Answer: B — The limiting MSE is C₀.** A: **Incorrect:** The limit is zero only when the fixed covariance is zero, contrary to C₀ > 0. B: **Correct:** The shared covariance C₀ remains after the terms proportional to 1/k disappear. C: **Incorrect:** The individual-model variance V is not generally the limit; the shared covariance is. D: **Incorrect:** All terms have a finite limit, so the expression does not diverge.

**1. Represent retention:** Let \(I_j=1\) if unit \(j\) is retained and zero otherwise. Each of the ten units has retention probability 0.7, so \(\mathbb E[I_j]=0.7\). **2. Add expectations:** The total active count is \(N=I_1+\cdots+I_{10}\). By linearity, \(\mathbb E[N]=10(0.7)=7\). **3. Interpret the count:** Seven is an average over training passes, not a requirement that exactly seven units be active in every pass. With independent dropout, \(N\) has a binomial distribution. **Answer: 7.**

**1. Count choices per unit:** Each of ten units has two possible mask states: retained or dropped. A complete mask makes one such choice for every unit. **2. Apply the multiplication rule:** The number of masks is \(2\times2\times\cdots\times2=2^{10}=1024\). This includes all-retained and all-dropped masks. **3. Separate possibilities from probabilities:** Since retention probability 0.7 lies strictly between zero and one, every mask has positive probability. Masks with different active counts do not have equal probabilities, but this does not change their number. **Answer: C — 2¹⁰ = 1024 thinned subnetworks.** A: **Incorrect:** Ten counts units, not the independently selectable masks over all units. B: **Incorrect:** 10² is not the product of the two choices available to each of ten units. C: **Correct:** There are two states for each unit, giving 2¹⁰ = 1024 distinct masks. D: **Incorrect:** 10! counts permutations of ten objects; dropout selects subsets rather than orders.

**1. Identify the pre-activation:** Before applying Leaky ReLU, the neuron computes its weighted input sum. The two weights are 2 and −3, the corresponding inputs are 4 and 3, and there is no bias. **2. Multiply corresponding entries:** The first contribution is \(2(4)=8\), and the second is \((-3)(3)=-9\). **3. Add the contributions:** \(a=8-9=-1\). The leak coefficient is not used until the activation is applied. **Answer: -1.**

**1. Reconstruct the input sum:** With weights 2,−3 and inputs 4,3, the pre-activation is \(a=2(4)-3(3)=-1\). **2. Select the negative branch:** Leaky ReLU with \(\alpha=0.1\) is \(h(a)=a\) for nonnegative inputs and \(h(a)=0.1a\) for negative inputs. **3. Compute the output:** Since −1 is negative, \(h(-1)=0.1(-1)=-0.1\), already correct to one decimal place. **Answer: -0.1.**

**1. Compare negative branches:** Standard ReLU returns zero for a negative pre-activation and has derivative zero there. Leaky ReLU returns \(\alpha a\) and has derivative \(\alpha=0.1\). **2. Apply the chain rule:** A nonzero negative-branch derivative lets an upstream loss gradient reach the weights even when the neuron’s input sum is negative. This reduces the dead-neuron problem associated with a flat ReLU branch. **3. Identify the design purpose:** The relevant benefit is maintaining gradient flow, rather than bounding the output or guaranteeing a zero-centered distribution. **Answer: C — A small nonzero negative-side gradient reduces the dead-neuron problem.** A: **Incorrect:** Computational cheapness is not the defining advantage; Leaky ReLU introduces a negative-branch multiplication. B: **Incorrect:** Leaky ReLU is unbounded and does not restrict output to the stated finite interval. C: **Correct:** The small negative-side derivative preserves a learning path when the pre-activation is below zero. D: **Incorrect:** Leaky ReLU is not an odd saturating activation like tanh and does not ensure zero-centered outputs.

**1. Recall the derivative:** For \(\sigma(a)=1/(1+e^{-a})\), the derivative is \(\sigma(a)(1-\sigma(a))\). Its maximum is 0.25 at \(a=0\). **2. Evaluate the large positive input:** At \(a=15\), \(\sigma(15)\approx0.999999694\), so the factor \(1-\sigma(15)\) is only about \(3.059\times10^{-7}\). **3. Interpret the gradient:** The product is approximately \(3.059\times10^{-7}\), effectively zero at the scale of the options. The neuron is saturated, so backpropagated gradients are strongly attenuated. **Answer: B — Derivative approximately zero; the sigmoid neuron is saturated.** A: **Incorrect:** The derivative reaches 0.25 at zero pre-activation, not at the saturated value 15. B: **Correct:** The derivative is approximately zero because sigmoid is almost flat at a large positive input. C: **Incorrect:** The sigmoid output approaches one, but its derivative approaches zero; these are different quantities. D: **Incorrect:** Sigmoid is differentiable at every finite real input, including 15.

**1. Select the ReLU regime:** If every future pre-activation satisfies \(a<0\), then \(\operatorname{ReLU}(a)=0\) and \(h\prime(a)=0\) on all those examples. **2. Propagate the derivative:** The weight gradient through this neuron contains the factor \(h\prime(a)\). Multiplying by zero makes that data-loss contribution vanish for each input. **3. Explain the consequence:** Ordinary gradient descent through this path cannot move the neuron back into the active regime. It may remain dead under the stated assumption of permanently negative inputs. **Answer: B — Derivative 0; the neuron may remain dead.** **Convention note:** Other mechanisms such as changes in upstream features or separate regularization updates can alter weights. The zero-gradient conclusion concerns backpropagation of the data loss through this negative ReLU path. A: **Incorrect:** Negative-input ReLU has derivative zero, so it does not continue learning normally through that path. B: **Correct:** A zero local derivative blocks the loss gradient, producing the dead-ReLU behavior. C: **Incorrect:** The question fixes every future pre-activation below zero, so the derivative does not alternate with positive-input cases. D: **Incorrect:** The negative-side derivative is zero, not one-half.

**1. Inspect the input patches:** The 3×3 input has two all-zero rows and a bottom row of ones. The two top 2×2 patches contain only zeros; each bottom patch is \(\begin{bmatrix}0&0\\1&1\end{bmatrix}\). **2. Differentiate the patch sums:** In every patch, the input coefficients multiplying \(w_1,w_2\) are zero. In each bottom patch, the coefficients multiplying \(w_3,w_4\) are one, and the pre-activation is \(w_3+w_4\). **3. Include the activation gate:** If a bottom output is active and its upstream gradient is nonzero, it can send a gradient to \(w_3,w_4\). No activation or upstream gradient can overcome the zero input coefficients for \(w_1,w_2\). **Answer: C, D — w₃ and w₄ can receive nonzero gradients.** **Convention note:** “Can receive” means a nonzero gradient is possible; the bottom ReLUs must be active and their upstream contributions must not cancel. A: **Incorrect:** w₁ multiplies zero in every receptive field, so its data-loss gradient is zero. B: **Incorrect:** w₂ also multiplies zero in every receptive field. C: **Correct:** w₃ multiplies a one in each bottom patch and can receive gradient when the ReLU gate is active. D: **Correct:** w₄ likewise has nonzero input coefficients in the bottom patches.

**1. Identify active outputs:** The input is zero except for its bottom row of ones. The top two filter outputs have zero input coefficients, while each bottom output has pre-activation \(w_3+w_4>0\), so its ReLU derivative is one. **2. Differentiate each bottom output:** In each bottom patch, \(\partial a/\partial w_3=1\). The given upstream loss derivative is also one, so each bottom location contributes \(1\cdot1\cdot1=1\). **3. Sum shared-weight contributions:** A convolutional parameter is shared across output positions, so its gradient is the sum from all positions. The two top contributions are zero and the bottom contributions are one each: \(\partial L/\partial w_3=0+0+1+1=2\). **Answer: 2.**

**1. Use the convolution size formula:** For input width 48, kernel width 7, stride 1 and symmetric padding \(P\) on each side, \(O=\lfloor(48+2P-7)/1\rfloor+1\). **2. Require the original width:** Since all quantities are integers, \(O=42+2P\). Setting \(O=48\) gives \(2P=6\), hence \(P=3\). **3. Check both dimensions:** The input and kernel are square, so the same padding gives output height 48. The answer is three pixels on each side, or six additional pixels across each dimension in total. **Answer: 3.**

**1. Compute the spatial dimensions:** With input 48×48, a 7×7 kernel, no padding and stride one, each output side is \(\lfloor(48-7)/1\rfloor+1=42\). **2. Determine output depth:** Every filter spans all three input channels and produces one feature map. Four filters therefore produce four output channels. **3. Assemble the shape:** Width, height and depth are 42,42,4 respectively. Input depth controls the depth of each filter, not the number of output feature maps. **Answer: A — 42 × 42 × 4.** A: **Correct:** The spatial formula gives 42×42, and four filters produce depth four. B: **Incorrect:** 48×48 would require padding three on each side for this kernel and stride. C: **Incorrect:** Three is the input depth; the output depth equals the number of filters, four. D: **Incorrect:** The final +1 in the convolution formula gives 42, not 41. E: **Incorrect:** Only four filters are specified, so the output cannot have eight channels.

**1. Count one filter’s weights:** A 7×7 filter spanning three input channels contains \(7\cdot7\cdot3=147\) weights. **2. Add the bias:** Each filter has one shared scalar bias, giving \(147+1=148\) trainable parameters per filter. **3. Count four filters:** \[4(7\cdot7\cdot3+1)=4(148)=592.\] Spatial reuse across the 42×42 output positions does not create additional parameters. **Answer: 592.**

**1. Compare AdaGrad accumulators:** AdaGrad divides a base rate by the square root of a cumulative sum of squared gradients. Frequently active, dense features usually grow that sum faster; rarely active, sparse features grow it more slowly and retain larger rates for longer. **2. Compare RMSProp and AdaDelta:** RMSProp uses an exponentially weighted accumulator, which can decrease when recent gradients are small. Its effective rate can therefore increase. The original AdaDelta update uses a ratio of running update and gradient scales rather than requiring a manually chosen global initial rate. **3. Compare Adam and AdaMax:** Adam scales updates using an exponential second-moment estimate \(v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2\). AdaMax replaces this with the infinity-norm-based recurrence \(u_t=\max(\beta_2u_{t-1},|g_t|)\). This is the L₂-type versus L∞ distinction intended by the last statement. **Answer: B, E — B and E — sparse-feature behavior in AdaGrad and the Adam/AdaMax distinction.** **Convention note:** The sparse/dense comparison assumes comparable gradient magnitudes. Some software versions of AdaDelta expose an additional learning-rate multiplier; the statement is assessed using the original algorithm discussed in the course. A: **Incorrect:** For comparable nonzero-gradient magnitudes, dense features accumulate squared gradients more frequently and their rates decay faster. B: **Correct:** Sparse features have fewer nonzero gradient contributions, so their cumulative denominators grow more slowly. C: **Incorrect:** An exponential accumulator can shrink, making the RMSProp effective learning rate increase. D: **Incorrect:** Original AdaDelta was designed to remove the need to choose a global learning-rate scale; the claimed sensitivity is not its defining property. E: **Correct:** Adam uses a second-moment square-based scale, whereas AdaMax uses an infinity-norm-based maximum scale.