Deep Learning · Quiz 2 · May 2024

**1. Connect feature activity to gradients:** For a linear input contribution, a sample’s derivative includes its feature value: \(\partial L/\partial w_j=\delta x_j\), where δ is the loss signal propagated to that unit. **2. Apply sparsity:** A sparse feature is zero in many samples, so its weight receives zero data gradient in those updates. A dense feature participates in more samples and typically receives useful updates more frequently. **3. State the likely effect under ordinary gradient descent:** The weights for sparse f₁ and f₃ tend to learn more slowly or receive fewer nonzero updates than the weight for dense f₂. Sparsity does not itself imply nonconvergence. **Answer: B — the sparse-feature weights typically update more slowly than the dense-feature weight.** **Convention note:** This describes typical nonzero data-gradient participation under ordinary updates, not a universal inequality between numerical update magnitudes. Momentum, weight decay, gradient scaling or adaptive optimizers can change the behavior. A: **Incorrect:** Sparse coordinates ordinarily receive fewer active data-gradient contributions, not automatically faster updates. B: **Correct:** Zero feature values suppress many per-example contributions, leading to the likely slower learning described. C: **Incorrect:** Feature sparsity can directly affect update frequency and gradient information. D: **Incorrect:** Sparse data can still be learned successfully; convergence failure is not implied.

**1. Write the effective coordinate rate:** AdaGrad keeps \(G_{t,j}=G_{t-1,j}+g_{t,j}^2\) and scales an update with \(\eta/(\sqrt{G_{t,j}}+\epsilon)\). **2. Use nonnegative accumulation:** Since g²≥0, G cannot decrease. Therefore its square-root denominator cannot decrease, and the effective coordinate learning rate cannot increase. **3. Allow flat periods:** If a coordinate’s gradient is zero, its accumulator and effective rate can remain unchanged. The correct property is monotonically nonincreasing, not necessarily strictly decreasing every iteration. **Answer: C — In each iteration, the learning rate monotonically decreases, although it may .** A: **Incorrect:** Its denominator grows or stays fixed, so the effective rate cannot monotonically increase. B: **Incorrect:** An increase is not possible under the standard fixed-base-rate rule. C: **Correct:** It correctly states nonincrease, including possible constant periods. D: **Incorrect:** It generally changes when nonzero squared gradients accumulate.

**1. State the RMSProp averaging rule:** \[v_t=\beta v_{t-1}+(1-\beta)g_t^2.\] The past accumulator receives weight β and the current squared gradient receives weight 1−β. **2. Compare high and low decay factors:** A β close to 1 retains a long history and assigns little weight to the current sample. A β close to 0 forgets history quickly and responds strongly to the current squared gradient. **3. Interpret the options:** The four individual statements describe these complementary high-β and low-β effects. Under that intended comparison, all are correct. **Answer: E — All of these.** **Convention note:** Higher and lower β refer to the usual regimes near 1 and near 0. A claim that current influence exceeds past influence specifically requires β<1/2, while the reverse requires β>1/2. A: **Incorrect:** High β gives the retained past accumulator more influence. B: **Incorrect:** Low β reduces the history coefficient and shortens the averaging memory. C: **Incorrect:** For β near one, the current contribution 1−β is small relative to the retained history. D: **Incorrect:** For β near zero, the current contribution dominates the retained history. E: **Correct:** It correctly combines the four intended high-versus-low β descriptions.

**1. Describe cosine annealing:** Within one cycle, a common schedule is \[\eta(\tau)=\eta_{min}+\tfrac12(\eta_{max}-\eta_{min})\left[1+\cos(\pi\tau/T)\right],\] where τ is elapsed cycle time and T the cycle length. It smoothly decreases the learning rate. **2. Explain the restart:** At a warm restart, the schedule returns to a higher learning rate while retaining the learned weights. This renews exploration after a low-rate refinement phase. **3. Match the distinctive purpose:** Periodic larger steps can help move away from an unfavorable basin and explore alternatives, making C the best explanation of the restart component. This is a possible optimization benefit, not a guarantee of escaping every sharp minimum. **Answer: C — Cosine annealing with warm restarts helps prevent the optimization process .** A: **Incorrect:** The schedule varies rather than remaining constant. B: **Incorrect:** It describes the cosine part but omits the distinctive reason for warm restarts. C: **Correct:** Returning to a higher rate renews exploration and can help escape an unfavorable local basin. D: **Incorrect:** It describes decay without explaining the periodic high-rate reset.

**1. Distinguish training from inference:** During training, dropout randomly masks units. During ordinary inference those random masks are disabled and all units are used. **2. Apply the original-dropout convention:** If training leaves surviving activations unscaled and a unit is retained with probability q, its expected training contribution is qh. At inference, multiply its full contribution by q to match that expected scale. **3. Match the option:** C describes all units active with keep-probability scaling, which is the original-dropout convention used by the source. The scaling applies to activations or equivalently to the corresponding outgoing weights. **Answer: C — All neurons are active, and their outputs are typically scaled by the keep .** **Convention note:** With inverted dropout, surviving training activations are divided by q and no additional inference scaling is needed, making D its rule. The source uses original dropout here. Monte Carlo dropout inference is not assumed. A: **Incorrect:** Ordinary inference does not sample the training dropout masks. B: **Incorrect:** All units are active in the intended deterministic inference pass, not a random subset. C: **Correct:** It correctly states the original-dropout rule: use all units and scale contributions by the keep probability. D: **Incorrect:** It is correct for inverted dropout, where scaling is already done during training, but not for the source’s original convention.

**1. Observe generalization during fitting:** Training loss can keep improving even after the model begins fitting noise. A separate validation score detects when further optimization stops improving unseen-example performance. **2. Apply early stopping:** Track the best validation checkpoint and stop after validation improvement ceases, often allowing a patience interval. Restore the best checkpoint rather than necessarily retaining the last one. **3. Explain regularization:** Limiting the optimization trajectory can reduce overfitting without directly adding a weight penalty or requiring a learning-rate schedule. **Answer: B — It stops training when the validation error stops decreasing and begins to .** A: **Incorrect:** Simply increasing epochs is not the protective mechanism; validation behavior determines when to stop. B: **Correct:** It describes stopping when validation performance stops improving or deteriorates. C: **Incorrect:** Learning-rate decay is a different training control. D: **Incorrect:** An explicit weight penalty is a different regularization mechanism.

**1. Interpret the observed parameter patterns:** An L1 penalty λΣ|wⱼ| can make some coefficients exactly zero, while an L2 penalty λΣwⱼ² mainly shrinks them continuously. The observed sparsity-versus-small-dense-weights pattern matches that distinction. **2. Explain the observed errors:** Removing many useful connections or features can restrict the L1 model too much and raise training error. On this particular task the denser shrunken L2 fit can therefore generalize better. **3. Avoid a universal ranking:** These results do not prove L1 is generally inferior. Which regularizer works best depends on the useful structure, regularization strength, dataset size and optimization. **Answer: A — L1 regularization induces sparsity, which can lead to feature selection, but .** A: **Correct:** It links L1-induced sparsity to possible feature removal and underfitting, matching the observations. B: **Incorrect:** It mentions weight shrinkage but a more-complex model is not the direct explanation for the observed L1 sparsity and error. C: **Incorrect:** The experiment supports a task-specific comparison, not a general inferiority theorem. D: **Incorrect:** Dataset size and signal structure can affect the relative performance and cannot be ruled out categorically.

**1. Take the negative-input limit:** For \(\sigma(z)=1/(1+e^{-z})\), a very negative z makes e⁻ᶻ very large, so σ(z) approaches 0. **2. Calculate the activation derivative:** \[\sigma\prime(z)=\sigma(z)(1-\sigma(z)).\] As σ(z)→0, this product also approaches 0, making the neuron locally insensitive to changes in its input. **3. Connect to gradient flow:** This is negative-tail sigmoid saturation. Its small local derivative can shrink a backpropagated signal, though the full loss-gradient expression also depends on the chosen loss. **Answer: A — The output approaches 0 and the gradient approaches 0..** A: **Correct:** Both the output and its local derivative approach zero in the negative tail. B: **Incorrect:** Output 0.5 and derivative 0.25 occur at input zero. C: **Incorrect:** Output approaching one describes the positive tail instead. D: **Incorrect:** The sigmoid derivative never reaches one; its maximum is 0.25.

**1. State the general Leaky ReLU:** For a chosen leak α>0, \[f(z)=\begin{cases}\alpha z,&z<0,\\z,&z>0.\end{cases}\] Its derivative away from zero is α for negative inputs and 1 for positive inputs. **2. Apply the conventional leak used by the key:** Taking α=0.01 gives derivative 0.01 on the negative side and 1 on the positive side, matching C. **3. Handle the boundary separately:** At zero the one-sided slopes differ when α≠1, so an implementation chooses a derivative convention. None of the listed options specifies this boundary. **Answer: C — 0.01 for negative inputs, 1 for positive inputs..** **Convention note:** The prompt does not specify α, so the derivative is not uniquely numerical. C follows the source’s α=0.01 convention; D is also a mathematically valid Leaky ReLU variant if α=0.05 is chosen. A: **Incorrect:** Zero negative slope describes ordinary ReLU rather than a leaky version. B: **Incorrect:** It reverses the positive and negative branch behavior. C: **Correct:** It is the intended answer under the usual α=0.01 convention. D: **Incorrect:** It would be valid for a different choice α=0.05, not the key’s intended α=0.01. E: **Incorrect:** Slope 1 everywhere is the identity function, corresponding to α=1 rather than the intended leak.

**1. Compare saturating activations:** Sigmoid and tanh approach bounded outputs in their tails, where their derivatives become tiny. This can weaken backpropagation across many layers. **2. Compare the rectified activations:** ReLU has slope 1 for positive inputs but exactly 0 for negative inputs, which can create dead units. Leaky ReLU replaces that negative zero slope with a small nonzero α. **3. Identify the intended choice:** Among the four listed functions, Leaky ReLU mitigates both bounded-tail saturation and the permanently flat negative-branch problem. Thus only activation 4 is the intended answer. **Answer: D — Only 4.** **Convention note:** This compares the usual activation-level problems, not an absolute guarantee about every deep network. Products of small slopes and weights can still make gradients vanish even with Leaky ReLU. A: **Incorrect:** Sigmoid has small derivatives in both saturation tails. B: **Incorrect:** Tanh also saturates in both tails. C: **Incorrect:** ReLU can have dead negative-side units. D: **Correct:** Leaky ReLU supplies a nonzero negative slope and a unit positive slope. E: **Incorrect:** Ordinary ReLU does not avoid its dead-unit problem. F: **Incorrect:** Both bounded activations have saturating tails.

**1. Examine sigmoid tails:** As z→−∞, σ(z)→0; as z→+∞, σ(z)→1. In both limits its derivative σ(z)(1−σ(z)) tends to zero. **2. Examine tanh tails:** Tanh approaches −1 and +1 in the two limits, and its derivative 1−tanh²(z) tends to zero. **3. Locate saturation:** Both functions therefore saturate for large negative and large positive inputs, not near input zero where their slopes are greatest. **Answer: E — Both at large positive values and at large negative values.** A: **Incorrect:** Near zero both functions have their steepest useful transition rather than saturation. B: **Incorrect:** A specific input value near 1 does not describe both saturation tails. C: **Incorrect:** This is true but omits the negative-input saturation regime. D: **Incorrect:** This is true but omits the positive-input saturation regime. E: **Correct:** It includes both large-magnitude input tails.

**1. Compare training and test error together:** A has train/test errors 0.30/0.25, B has 0.27/0.20, and C has 0.11/0.67. The smallest training error alone does not identify the best predictor on new data. **2. Diagnose C:** Its very low training error and much larger test error are the clearest overfitting pattern. C memorizes or fits training-specific detail that does not transfer well. **3. Evaluate B and A:** B has the lowest observed test error and a moderate training error, making it the best listed generalization tradeoff. A’s highest training error alone is insufficient to establish underfitting; its test error is also lower than its own training error, and no irreducible-error benchmark is supplied. **Answer: B, C — **Model C** shows signs of overfitting.; **Model B** strikes a balance between bias and variance, making it a potentially .** A: **Incorrect:** C’s test error 0.67 contradicts the claim that its low training error demonstrates best generalization. B: **Correct:** The large training–test discrepancy is strong evidence of overfitting. C: **Correct:** B has the smallest test error and is the best observed regularization tradeoff. D: **Incorrect:** A higher training error by itself does not prove underfitting without an adequacy benchmark or further evidence.

**1. Compute adjacent differences:** With X=(1,0,1,0,1,0,1,0) and kernel K=(1,−1), stride-1 valid CNN cross-correlation gives \[a=(1,-1,1,-1,1,-1,1).\] There are 8−2+1=7 output positions. **2. Sum before activation:** Four +1 entries and three −1 entries give Σaᵢ=4−3=1. ReLU is applied only in the next subquestion. **Answer: 1.** **Convention note:** As in the CNN convention used throughout the paper, the filter is not flipped. Mathematical convolution with a reversed kernel would give the opposite signs.

**1. Compute adjacent differences:** With X=(1,0,1,0,1,0,1,0) and kernel K=(1,−1), stride-1 valid CNN cross-correlation gives \[a=(1,-1,1,-1,1,-1,1).\] There are 8−2+1=7 output positions. **2. Apply ReLU and averaging:** ReLU gives h=(1,0,1,0,1,0,1), whose sum is 4. Averaging all seven entries gives ŷ=4/7. **3. Calculate the specified loss:** With y=1 and \(L=\tfrac12(\hat y-y)^2\), \[L=\tfrac12(4/7-1)^2=\tfrac12(9/49)=\frac9{98}=0.091836735\ldots.\] The reported answer 0.092 is a rounded approximation. **Answer: 0.092.**

**1. Count Conv1 and its output:** Six 3×3 RGB filters without bias use \(3\cdot3\cdot3\cdot6=162\) weights. Spatial side becomes 64−3+1=62; pooling halves it to 31, leaving 31×31×6. **2. Count Conv2 and the next pool:** Four 4×4 filters span six input channels, using \(4\cdot4\cdot6\cdot4=384\) weights. Side 31 becomes 28, then pooling gives 14×14×4. **3. Count dense connections:** Flattening yields 14×14×4=784 features. The 20-unit dense layer has 784×20=15680 weights, and the 10-class output has 20×10=200. **4. Sum all learned weights:** \[P=162+384+15680+200=16426.\] Pooling and softmax add no trainable parameters, and every bias is excluded. **Answer: 16426.**