Deep Learning · Quiz 2 · Sep 2024

**1. Define the gradient and update:** For L(w)=w², gₜ=2wₜ. AdaGrad uses \(v_t=v_{t-1}+g_t^2\), \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=1,v₋₁=0,w₀=2,ε=0. **2. Compute the first step:** g₀=4 and v₀=16, so \(w_1=2-4/4=1\). **3. Report the requested first parameter:** The exact first update is w₁=1, written as 1.000 to three decimal places. **Answer: 1.000.**

**1. Define the gradient and update:** For L(w)=w², gₜ=2wₜ. AdaGrad uses \(v_t=v_{t-1}+g_t^2\), \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=1,v₋₁=0,w₀=2,ε=0. **2. Compute the first step:** g₀=4 and v₀=16, so \(w_1=2-4/4=1\). **3. Evaluate the next gradient and accumulator:** At w₁=1, g₁=2. Thus \[v_1=16+2^2=20.\] This accumulator includes the squared gradients evaluated at both w₀ and w₁. **Answer: 20.000.** **Convention note:** The question explicitly asks for v₁; follow the given index convention even though its wording says after the first iteration.

**1. Define the gradient and update:** For L(w)=w², gₜ=2wₜ. AdaGrad uses \(v_t=v_{t-1}+g_t^2\), \(w_{t+1}=w_t-\eta g_t/\sqrt{v_t}\), with η=1,v₋₁=0,w₀=2,ε=0. **2. Compute the first step:** g₀=4 and v₀=16, so \(w_1=2-4/4=1\). **3. Take the next AdaGrad step:** At w₁=1, g₁=2 and v₁=20. Therefore \[w_2=1-\frac2{\sqrt{20}}=1-0.4472135955=0.5527864045.\] To three decimals, enter 0.553. **Answer: 0.553.**

**1. Distinguish time schedules from gradient adaptation:** Exponential or step decay changes a shared learning-rate schedule according to the iteration count. An adaptive method instead uses current and past gradient statistics to choose coordinate-dependent scaling. **2. Identify the adaptive methods:** RMSProp uses a running mean of squared gradients. Adam combines a first-moment estimate with second-moment scaling. AdaDelta uses running squared-gradient and squared-update statistics. **3. Reject schedule-only descriptions:** Exponentially decayed GD and Nesterov with step decay can change rates over time, but the stated schedules do not adapt them from gradient history in the requested sense. **Answer: B, D, E — RMSProp; ADAM; Ada-Delta.** A: **Incorrect:** Its stated exponential schedule depends on time rather than gradient statistics. B: **Correct:** Its running squared-gradient estimate controls coordinate-wise update scales. C: **Incorrect:** Adding step decay to Nesterov momentum does not create gradient-history-based adaptive scaling. D: **Correct:** It uses gradient-moment estimates for adaptive coordinate updates. E: **Correct:** It normalizes updates using running gradient and update statistics.

**1. Define regularization:** Regularization changes the learning setup to reduce overfitting and improve generalization, for example by penalizing complexity or exposing the model to appropriate perturbations. **2. Apply the direct mechanisms:** Data augmentation introduces label-preserving variations; L2 penalizes squared weight magnitude; adding suitable input or output noise discourages overly brittle or overconfident fits. **3. Separate broad categories:** Activation functions define the represented nonlinearities, and optimizers define how parameters are fitted. Neither category is automatically a regularization technique merely by being present. **Answer: A, C, D — Dataset augmentation; L2 regularization; Adding Noise to the inputs or outputs.** A: **Correct:** Appropriate augmented examples encourage invariance and reduce memorization. B: **Incorrect:** An activation function is not inherently a regularizer, although its choice can affect capacity. C: **Correct:** It directly penalizes large weight magnitudes. D: **Correct:** Appropriately designed noise injection can act as regularization. E: **Incorrect:** An optimization method is not automatically an overfitting-control technique. F: **Incorrect:** Not every listed broad category is a regularizer.

**1. Identify the two ingredients:** Momentum averages signed gradients to create a persistent update direction. RMSProp rescales coordinates using a moving average of squared gradients. **2. Combine them in Adam:** Adam maintains both \(m_t=\beta_1m_{t-1}+(1-\beta_1)g_t\) and \(v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2\), applies bias corrections, and updates using \(\hat m_t/(\sqrt{\hat v_t}+\epsilon)\). Thus it combines a momentum-like numerator with RMSProp-like scaling. **Answer: B — Adam.** A: **Incorrect:** Nesterov acceleration uses momentum and look-ahead gradients, not the combined second-moment adaptation described. B: **Correct:** Its signed first-moment and squared-gradient second-moment estimates combine the two mechanisms. C: **Incorrect:** AdaGrad accumulates squared gradients without Adam’s signed first-moment estimate. D: **Incorrect:** AdaDelta adapts scales using running statistics but is not this named momentum-plus-RMSProp combination.

**1. Explain the optimization difficulty:** Parameters can have very different gradient scales or update frequencies. A single shared step size may be too large for one coordinate and too small for another. **2. Explain adaptation:** RMSProp and Adam use each coordinate’s gradient history to rescale its updates dynamically. This aims to make fitting more effective without manually assigning a separate fixed learning rate to every parameter. **3. Distinguish the goal from architecture and scheduling:** These algorithms do not principally change model size or layer count, and they are more than a predetermined global decay of the initial rate. **Answer: B — To adapt the learning rate for each parameter dynamically.** A: **Incorrect:** Adaptive optimization does not primarily add model parameters. B: **Correct:** Dynamic coordinate-specific scaling is the key mechanism. C: **Incorrect:** It does not principally remove hidden layers. D: **Incorrect:** A global time-only decay is distinct from adaptation based on gradient statistics.

**1. State the general Leaky ReLU:** For a chosen leak α>0, \[f(z)=\begin{cases}\alpha z,&z<0,\\z,&z>0.\end{cases}\] Its derivative away from zero is α for negative inputs and 1 for positive inputs. **2. Apply the conventional leak used by the key:** Taking α=0.01 gives derivative 0.01 on the negative side and 1 on the positive side, matching C. **3. Handle the boundary separately:** At zero the one-sided slopes differ when α≠1, so an implementation chooses a derivative convention. None of the listed options specifies this boundary. **Answer: C — 0.01 for negative inputs, 1 for positive inputs..** **Convention note:** The prompt does not specify α, so the derivative is not uniquely numerical. C follows the source’s α=0.01 convention; D is also a mathematically valid Leaky ReLU variant if α=0.05 is chosen. A: **Incorrect:** Zero negative slope describes ordinary ReLU rather than a leaky version. B: **Incorrect:** It reverses the positive and negative branch behavior. C: **Correct:** It is the intended answer under the usual α=0.01 convention. D: **Incorrect:** It would be valid for a different choice α=0.05, not the key’s intended α=0.01. E: **Incorrect:** Slope 1 everywhere is the identity function, corresponding to α=1 rather than the intended leak.

**1. Relate bias to model flexibility:** Bias is systematic error caused by a restricted model family. A rigid straight-line boundary has less ability to represent a curved class boundary than a suitably flexible model. **2. Compare the plotted shapes:** A is the simplest straight boundary. C bends to fit the broad nonlinear separation, while B has the most flexible, wavy boundary. In the intended capacity comparison, A has the highest bias, C intermediate, and B the lowest. **3. State the ordering:** A>C>B. This is a qualitative bias ordering from the illustrated model complexity, not a numerical estimate of expected prediction bias from a single training plot. **Answer: D — A > C > B.** A: **Incorrect:** It incorrectly assigns B more bias than the less flexible C. B: **Incorrect:** It gives the most flexible boundary the greatest bias. C: **Incorrect:** It does not reflect A’s stronger linear restriction. D: **Correct:** It orders the models from least to most flexible and therefore highest to lowest intended bias.

**1. Identify the mathematical penalty:** A term proportional to \(\sum_jw_j^2=\|w\|_2^2\) is a squared L2 penalty. A common convention writes \(\lambda\|w\|_2^2/2\). **2. Explain the contrast:** L1 uses absolute values, dropout randomly masks units during training, and batch normalization normalizes activations. They are distinct mechanisms from the explicit squared-weight term. **Answer: C — L2 Regularization.** A: **Incorrect:** Dropout uses stochastic unit masking rather than this explicit penalty. B: **Incorrect:** L1 penalizes absolute weight values. C: **Correct:** L2 is precisely the squared-weight penalty described. D: **Incorrect:** Batch normalization changes activation statistics rather than directly adding this penalty.

**1. Compute the third hidden pre-activations:** The given h₂ is (3,2,0,1,6)ᵀ and every W₃ entry equals 1. Each third-layer neuron therefore receives a₃=3+2+0+1+6=12. **2. Apply the specified rounding:** σ(12)≈0.999993856 is greater than 0.99, so the question instructs treating it as 1. Using that rounded value, the derivative σ(a₃)(1−σ(a₃)) becomes 1(0)=0. **3. Trace backpropagation:** This zero derivative blocks the signal into W₃ and therefore into W₂ and W₁. W₄ still receives the third-layer activations, all ones, and can have nonzero output-loss gradients. Thus the intended nonzero-gradient matrix is W₄ only. **Answer: D — W₄ only under the stated rounding rule.** **Convention note:** This uses the explicit saturation-rounding instruction and an ordinary nonstationary classification loss. In exact arithmetic σ′(12) is tiny but nonzero, so the earlier gradients would not be forced to be exactly zero. The unusual ReLU threshold printed in the question does not alter this downstream blocking argument. A: **Incorrect:** Its gradient is blocked by the rounded zero derivative at the third hidden layer. B: **Incorrect:** The same blocked signal prevents a nonzero gradient from reaching W₂. C: **Incorrect:** The rounded saturated sigmoid derivative makes its data gradient zero. D: **Correct:** It lies after the saturating hidden layer and still receives nonzero hidden activations. E: **Incorrect:** The supplied rounding rule and network placement determine the intended answer.

**1. Count dense input connections:** Flattening a 10×10 RGB image gives 10×10×3=300 inputs. Two output neurons require 300×2=600 weights, excluding biases. **2. Count shared convolution weights:** Three 2×2 filters each span three channels, so \(P_{conv}=2\cdot2\cdot3\cdot3=36\) weights. **3. Compare:** 36 is much smaller than 600 because each convolution filter shares its weights across positions. Including one bias per output neuron/filter would still leave convolution smaller. **Answer: A — CONV Layer.** A: **Correct:** It needs only 36 shared kernel weights versus 600 dense weights under the no-bias count. B: **Incorrect:** The fully connected layer has far more independent connections.

**1. Count FC weights:** The flattened image has 300 entries, so two fully connected neurons have \(300\cdot2=600\) weights. **2. Count CONV weights:** There are three filters of shape 2×2×3, giving \(2\cdot2\cdot3\cdot3=36\) weights. Stride 2 changes where they are applied but not this count. **3. Subtract:** \[P_{FC}-P_{CONV}=600-36=564.\] **Answer: 564.** **Convention note:** This follows the course’s no-bias counting convention reflected by the supplied key. If biases were included in both alternatives, the counts would be 602 and 39, giving a difference of 563. The local passage does not explicitly specify biases.

**1. Write the parameter-count formula:** A standard convolution has \[P=(k_hk_wC_{in}+b)C_{out},\] where b=1 if there is one bias per filter and b=0 if biases are excluded. **2. Identify dependencies:** Input depth Cᵢₙ determines each filter’s depth, and the number of filters Cₒᵤₜ determines how many such sets of weights are learned. **3. Exclude spatial hyperparameters:** Stride and padding affect the output grid but do not add new coefficients to the filters. **Answer: A, C — The total number of parameters depends on the depth of the input.; The total number of parameters depends on the number of filters (kernel)..** A: **Correct:** Each filter spans the full input channel depth. B: **Incorrect:** Padding inserts spatial border values rather than trainable filter coefficients. C: **Correct:** Each additional filter adds another kernel’s parameters. D: **Incorrect:** Stride changes evaluation positions, not the filter tensor size.

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Compute ConvLayer1 shape:** Its 32 filters of size 9×9 give side 128−9+1=120 and depth 32. Thus the output shape is 120×120×32. **3. Multiply for activation volume:** \[120\cdot120\cdot32=14400(32)=460800.\] **Answer: 460800.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Count one first-layer filter:** A 9×9 RGB kernel has 9×9×3=243 weights. **3. Count all 32 filters:** \[P_1=9\cdot9\cdot3\cdot32=243(32)=7776.\] Spatial output locations reuse the same parameters. **Answer: 7776.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Determine the first pooled shape:** ConvLayer1 gives 120×120×32. PoolLayer1 halves each spatial side, giving 60×60×32. **3. Compute pooled volume:** \[60\cdot60\cdot32=3600(32)=115200.\] Pooling reduces activation count by four while keeping 32 channels. **Answer: 115200.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Distinguish fixed selection from learned weights:** Max pooling selects a maximum from each fixed 2×2 window. It has no learned kernel coefficients or biases, so PoolLayer1 contains 0 trainable parameters. **Answer: 0.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Propagate to the second convolution:** ConvLayer1 gives 120×120×32, then PoolLayer1 gives 60×60×32. A valid 5×5 convolution maps side 60 to 60−5+1=56. **3. Include the 64 output filters:** ConvLayer2 therefore has shape 56×56×64 and volume \[56\cdot56\cdot64=3136(64)=200704.\] **Answer: 200704.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Use the actual input depth to ConvLayer2:** PoolLayer1 retains the first convolution’s 32 channels. Each second-layer 5×5 filter therefore has 25×32=800 weights. **3. Count its 64 filters:** \[P_2=5\cdot5\cdot32\cdot64=800(64)=51200.\] The original RGB depth 3 is no longer the relevant input depth. **Answer: 51200.**

**1. Use the architecture’s shape rules:** The input is 128×128×3. Convolutions use stride 1 without padding, so a kernel of side F maps N to N−F+1. Each 2×2 pool uses stride 2 without padding and preserves channel depth. Biases are excluded throughout. **2. Find the second pooling input:** The first convolution and pool give side lengths 120 then 60; ConvLayer2 then gives 56×56×64. **3. Pool and count:** The 2×2, stride-2 pool produces 28×28×64, giving \[28\cdot28\cdot64=784(64)=50176\text{ activations}.\] **Answer: 50176.**