Introduction to Deep Learning and Generative AI · Quiz 2 · Jan 2026

**1. Locate the operation:** model_0.train() places the module and its submodules in training mode. This changes the behavior of layers whose forward pass depends on whether training or evaluation is underway. **2. Explain the important layers:** Dropout randomly masks activations during training. BatchNorm normally uses batch statistics and updates its running estimates in training mode, whereas evaluation uses its configured evaluation behavior. **3. Separate mode from gradients:** train() does not reset weights, clear gradients or enable autograd. optimizer.zero_grad(), backward() and step() have those distinct gradient/update responsibilities. **Answer: D — It ensures layers like Dropout and BatchNorm behave correctly during training.** A: **Incorrect:** Autograd tracking is controlled separately by tensor requirements and gradient modes; train() is not its enable switch. B: **Incorrect:** The existing learned parameters remain in place when training mode is selected. C: **Incorrect:** optimizer.zero_grad() clears stored parameter gradients; train() does not. D: **Correct:** Training mode ensures layers such as Dropout and BatchNorm use their training-specific behavior.

**1. Read the input axes:** The tensor has shape \((2,3,8,6)\): batch two, channels three, height eight and width six. **2. Read the adaptive target:** nn.AdaptiveMaxPool2d(1) requests output spatial dimensions 1×1. The argument is a target output size, not a fixed pooling kernel width. **3. State the complete result:** Each sample/channel is reduced to one spatial maximum, so y has shape \((2,3,1,1)\). The requested spatial size alone is 1×1. **Answer: C — 1x1.** A: **Incorrect:** 2×3 are the batch/channel dimensions, not the requested spatial output. B: **Incorrect:** The adaptive target is one in each spatial direction, not two. C: **Correct:** The output_size=1 setting explicitly produces one row and one column per channel. D: **Incorrect:** A scalar target applies to both spatial dimensions and does not leave width two.

**1. Clamp each entry:** Clamping to [0,4] replaces negative values with zero and values above four with four. The rows become \((0,0,0.5)\) and \((1.2,3.5,4)\). **2. Sum along dim=1:** This sums the entries within each row: \(z=(0.5,8.7)\). It does not sum the two rows together. **3. Add the two-element bias:** \((0.5,8.7)+(1,-1)=(1.5,7.7)\). With the displayed precision, the output is tensor([1.5000, 7.7000]). **Answer: D — tensor([1.5000, 7.7000]).** A: **Incorrect:** This does not perform the required row sums and bias addition. B: **Incorrect:** The second row’s clamped sum is 8.7, so subtracting one gives 7.7 rather than 4.7. C: **Incorrect:** These are the row sums before the bias is added. D: **Correct:** It correctly clamps, sums each row and adds the corresponding bias value.

**1. Follow the dimensions:** The linear layers map image_size to hidden_size, then hidden_size to hidden_size, then hidden_size to one. These dimensions compose correctly for a flattened image batch. **2. Inspect activations:** LeakyReLU(0.2) is suitable for hidden discriminator layers. The last sigmoid maps the single logit to a real/fake probability between zero and one. **3. Match the intended formulation:** This is a valid conventional binary GAN discriminator when used with a probability-based binary cross-entropy loss. None of the three marked lines needs changing for that formulation. **Answer: D — The code will perfectly work as intended for a discriminator.** **Convention note:** Assume the enclosing class provides a forward method and flattened inputs of the stated width. With BCEWithLogitsLoss, the final sigmoid should instead be omitted; a WGAN critic uses another formulation. “Perfectly” does not guarantee successful GAN training. A: **Incorrect:** The negative slope 0.2 is a valid LeakyReLU choice for a hidden layer. B: **Incorrect:** A single output unit is appropriate for binary real/fake scoring. C: **Incorrect:** Sigmoid is appropriate when the following loss expects probabilities. D: **Correct:** The shown layer sequence is valid for the conventional sigmoid-plus-BCELoss discriminator.

**1. Relate the metrics:** With natural-log average cross-entropy H, perplexity is \(e^H\). The exponential function is strictly increasing. **2. Evaluate both models:** A has \(e^2\approx7.389\); B has \(e^{1.5}\approx4.482\). **3. Compare:** B has lower perplexity because its comparable average cross-entropy is lower. Changing the common logarithm base changes the conversion formula, not this ordering. **Answer: B — Model B.** **Convention note:** The comparison assumes the same tokenization, evaluation data and loss/logarithm convention. A: **Incorrect:** A’s higher cross-entropy gives higher, not lower, perplexity. B: **Correct:** The smaller cross-entropy 1.5 gives the smaller perplexity. C: **Incorrect:** Strict monotonicity means different comparable losses do not give equal perplexities. D: **Incorrect:** The supplied comparable average cross-entropy values suffice to rank perplexity.

**1. Include the activation derivative:** Under the question’s simplified approximation, the dominant per-step gradient factor is \(0.4\rho(W_{hh})\), not the recurrent eigenvalue magnitude alone. **2. Compute the three factors:** A has \(0.4(3.2)=1.28\); B has \(0.4(0.95)=0.38\); C has \(0.4(0.45)=0.18\). Values above one suggest growth and values below one suggest decay. **3. Explain repetition over twenty steps:** These factors are repeatedly multiplied along a long recurrent path. Approximate magnitudes \(1.28^{20}\), \(0.38^{20}\), and \(0.18^{20}\) respectively show substantial growth, severe attenuation and still stronger attenuation. Thus A is classified as exploding and B,C as vanishing. **Answer: C — Model A: Exploding, Model B: Vanishing, Model C: Vanishing.** **Convention note:** This is a diagnostic approximation, not an exact theorem from eigenvalues and an average tanh derivative. Actual gradients depend on products of input-dependent Jacobians, singular values, non-normality and gradient direction. A: **Incorrect:** B’s approximate factor is 0.38, far below one rather than near-stable. B: **Incorrect:** B’s factor is below one and therefore suggests vanishing, not exploding. C: **Correct:** The factors 1.28,0.38,0.18 give the intended exploding/vanishing/vanishing classification. D: **Incorrect:** A’s factor exceeds one, so the intended approximation predicts growth rather than near stability. E: **Incorrect:** Both B and C have strongly contracting factors, inconsistent with calling them near-stable.

**1. Decode the input:** batch_first=True makes x’s shape \((32,10,8)\) mean 32 sequences, ten time positions and eight input features. **2. Identify out:** nn.RNN returns the final layer’s hidden representation at every time position. The hidden width is 16, so the last dimension changes from eight to sixteen. **3. Distinguish h:** out therefore has shape \((32,10,16)\). The separate final hidden-state tensor h has shape \((2,32,16)\), because there are two recurrent layers. **Answer: A — (32, 10, 16).** A: **Correct:** The batch and time axes remain 32 and ten, and the output feature axis is hidden_size=16. B: **Incorrect:** This would correspond to sequence-first ordering, but batch_first is explicitly true. C: **Incorrect:** This omits the ten time positions present in out. D: **Incorrect:** This is the separate h tensor’s shape, not out’s shape. E: **Incorrect:** out does not expose an additional layer axis; it returns the last layer’s sequence outputs.

**1. Compute the prediction:** Linear activation gives \(\hat y=wx=0.2(5)=1\). The error is \(\hat y-y=1-3=-2\). **2. Differentiate with the chain rule:** For \(L=\tfrac12(\hat y-y)^2\), \(\partial L/\partial\hat y=\hat y-y\) and \(\partial\hat y/\partial w=x\). Hence \(\partial L/\partial w=(-2)(5)=-10\). **3. Apply gradient descent:** \(w_{new}=w-\eta\,\partial L/\partial w=0.2-0.05(-10)=0.7\). The negative gradient causes an increase in w, moving the prediction toward the target. **Answer: 0.7.**

**1. Use SAME output sizing:** Under the standard SAME-padding convention with stride s, each output side is \(\lceil N/s\rceil\). SAME does not mean preserving the original resolution when stride exceeds one. **2. Substitute:** With \(N=192\) and \(s=4\), \(\lceil192/4\rceil=48\). The square output is therefore 48×48. **3. Check compatibility:** The ordinary convolution formula with kernel three and zero padding also gives \(\lfloor(192-3)/4\rfloor+1=48\). The particular values happen not to require added padding to achieve this SAME output size. **Answer: B — 48 × 48.** **Convention note:** This uses the common framework-independent SAME sizing convention. PyTorch Conv2d padding="same" itself does not support arbitrary strided convolutions; the question does not name that API for this item. A: **Incorrect:** 47 omits the correct output-size calculation. B: **Correct:** The SAME convention gives ceil(192/4)=48 in each spatial dimension. C: **Incorrect:** 49 exceeds the target ceil(192/4). D: **Incorrect:** 50 is not produced by the stated dimensions and SAME sizing.

**1. Count the hidden recurrence:** Input-to-hidden weights contribute \(6(4)=24\); hidden-to-hidden weights contribute \(4(4)=16\); one hidden bias vector adds four. This subtotal is 44. **2. Count the output map:** Four hidden values feed five outputs, requiring \(4(5)=20\) weights and five output biases. This adds 25. **3. Add and explain sharing:** \(44+25=69\). The same recurrent parameters are reused at all ten word positions, so sentence length does not multiply the count. **Answer: C — 69.** **Convention note:** This abstract simple-RNN count uses one hidden bias vector. Some APIs, including PyTorch nn.RNN, store separate input-hidden and hidden-hidden biases; that different parameterization would add another four. No such API is specified here. A: **Incorrect:** 55 does not include all input, recurrent, output and bias parameters. B: **Incorrect:** 64 counts the full model except the five output biases. C: **Correct:** 24 input weights + 16 recurrent weights + 4 hidden biases + 20 output weights + 5 output biases = 69. D: **Incorrect:** The sequence length does not create independently learned parameters at each time step.

**1. Test smooth functions by curvature:** On an interval, a twice differentiable function is convex when its second derivative is nonnegative everywhere. For \(x^2\), \(f^{\prime\prime}=2\); for \(e^x\), \(f^{\prime\prime}=e^x>0\). Both are convex on the real line. **2. Reject sign-changing or negative curvature:** For \(x^4-3x^2\), \(f^{\prime\prime}=12x^2-6\), which is negative near zero. For \(\log x\), it is \(-1/x^2<0\). For \(\sin x\), it is \(-\sin x\), which changes sign. **3. Handle the nonsmooth function:** Absolute value is convex despite lacking a derivative at zero, since the triangle inequality gives \(|tx+(1-t)y|\le t|x|+(1-t)|y|\) for \(0\le t\le1\). **Answer: A, B, E — x², eˣ and |x|.** A: **Correct:** x² has constant positive second derivative two. B: **Correct:** eˣ has positive second derivative eˣ on its whole domain. C: **Incorrect:** Its second derivative is negative at zero, so it is not convex on all real numbers. D: **Incorrect:** ln x is strictly concave on x>0 rather than convex. E: **Correct:** The triangle inequality proves convexity of absolute value, including at its corner. F: **Incorrect:** The changing sign of −sin x prevents convexity on the whole real line.

**1. Read what is cleared:** reset_grad_modified() calls only d_optimizer.zero_grad(). Thus discriminator gradients from the previous iteration are erased; generator gradients are not. **2. Compute the second discriminator step:** Its stored gradient is only the new value 0.4. With SGD-style update rate 0.01, \(\Delta\theta_D=-0.01(0.4)=-0.004\). **3. Compute the accumulated generator step:** Its old gradient 0.5 remains and the new 0.2 is added, giving 0.7. The second update is \(\Delta\theta_G=-0.01(0.7)=-0.007\). **4. Interpret the time wording:** These are the individual update amounts executed in iteration two, not the sum of changes since the initial parameters. **Answer: A, F — θ_D ← θ_D - 0.004; θ_G ← θ_G - 0.007.** **Convention note:** The stated answer assumes the supplied scalar gradients and simple θ←θ−αg updates, as intended. Additional cross-network backward contributions are outside the simplified data given. A: **Correct:** The discriminator is cleared before the new backward pass, so its second step subtracts 0.004. B: **Incorrect:** Gradient descent subtracts the given positive gradient rather than adding 0.004. C: **Incorrect:** The discriminator’s old gradient is cleared, and the update sign is also wrong. D: **Incorrect:** The generator’s accumulated gradient is 0.7, not one. E: **Incorrect:** This would require clearing the old generator gradient, which the function does not do. F: **Correct:** The generator retains 0.5 and adds 0.2, producing the update −0.007.

**1. Restore gradient hygiene:** Discriminator gradients should be cleared before accumulating the current discriminator loss, so earlier iterations do not contaminate the step. **2. Correct the classification targets:** Real examples use real_labels; generated examples must use fake_labels when training the discriminator. Otherwise the code labels the fake batch as real. **3. Separate the computation graphs:** Passing fake_images.detach() prevents this discriminator backward pass from also accumulating gradients through the generator. Only the discriminator is being trained in this stage. **4. Fix the aggregate loss as well:** For the usual binary-cross-entropy setup, use the sum or average of the real and fake losses, not the displayed subtraction. The offered “ratio” correction is also wrong. **Answer: A, B, C — Clear discriminator gradients, correct fake targets and detach generated samples.** **Convention note:** The options omit a separate valid correction: replace d_loss_real − d_loss_fake with their sum/average for the conventional BCE setup. Detaching is important training-loop separation; without it, stepping only d_optimizer does not by itself update generator parameters, although generator gradients are unnecessarily computed. A: **Correct:** A zero_grad/reset call is needed for the intended independent discriminator step. B: **Correct:** The generated batch needs fake targets rather than the real targets used in the snippet. C: **Correct:** Detaching prevents unintended generator-gradient accumulation and graph retention during the discriminator-only step. D: **Incorrect:** A ratio is not the correct BCE aggregation; the real and fake losses should be added or averaged.

**1. Separate rate schedules from momentum:** A schedule changes the learning-rate scalar according to a rule such as step, exponential or cosine decay. Momentum instead maintains a velocity or gradient-history state. **2. Examine acceleration:** Suitable constant-step momentum or accelerated methods can converge and can improve speed on appropriate objectives. A decreasing learning rate is not a universal requirement for acceleration. **3. Combine the mechanisms:** Momentum/acceleration determines how an update direction uses history, while a schedule sets its scale over training. These mechanisms can be used together. **Answer: A, C, E — Step decay, exponential decay, and cosine decay are examples of learning rate ; Momentum-based methods can improve convergence speed even when the ; Learning rate scheduling and accelerated gradient methods can be used .** A: **Correct:** Step, exponential and cosine rules are standard learning-rate schedules. B: **Incorrect:** There are convergent accelerated methods with an appropriate constant step size, so “necessarily” is false. C: **Correct:** Momentum can accelerate convergence even with a constant learning rate in suitable problems. D: **Incorrect:** A schedule alone does not create an accumulator of past gradients; momentum and related optimizers do. E: **Correct:** The learning-rate schedule and history-based update rule address distinct aspects and can be combined.

**1. Count lengths:** The reference “machine learning is very fun” has five words. The prediction “machine learning is fun fun” also has five, so the brevity penalty is \(e^{1-5/5}=1\). **2. Clip unigram matches:** machine, learning and is match once each. fun occurs twice in the prediction but only once in the reference, so it contributes only one clipped match. Total matches are four of five candidate words. **3. Multiply:** Modified unigram precision is \(p_1=4/5=0.8\), giving BLEU-1 \(=1(0.8)=0.800\). Repeating a matching token cannot increase its clipped count beyond the reference count. **Answer: C — 0.800.** A: **Incorrect:** The duplicate fun cannot count twice against a single reference occurrence, so precision is below one. B: **Incorrect:** 0.779 is the shorter second prediction’s brevity-penalized score, not this equal-length case. C: **Correct:** Four clipped matches out of five and penalty one give 0.800. D: **Incorrect:** This is not the result of the specified clipped precision and equal-length penalty.

**1. Count and match words:** The candidate “machine learning is fun” has length four, while the reference length is five. All four candidate unigrams occur in the reference, so \(p_1=4/4=1\). **2. Apply the brevity penalty:** Because the candidate is shorter, \(BP=e^{1-r/c}=e^{1-5/4}=e^{-0.25}\approx0.778800783\). **3. Calculate BLEU-1:** \(BP\,p_1=0.778800783(1)\), which rounds to 0.779. The missing word affects the brevity penalty even though unigram precision is perfect. **Answer: B — 0.779.** A: **Incorrect:** Perfect unigram precision is reduced by the shorter candidate’s brevity penalty. B: **Correct:** The product is approximately 0.7788, or 0.779. C: **Incorrect:** 0.8 ignores the specified exponential brevity-penalty calculation. D: **Incorrect:** 0.606 would require a different length ratio.

**1. Identify the three GRU gates:** For input width I=20 and hidden width H=32, PyTorch stores three stacked input weight matrices with \(3HI=3(32)(20)=1920\) entries. **2. Count recurrent weights:** The hidden-to-hidden matrices contribute \(3H^2=3(32^2)=3072\). **3. Count both bias groups:** bias=True creates input-side and recurrent-side biases of size 3H each, totaling \(6H=192\). **4. Add the GRU subtotal:** \(1920+3072+192=5184\). The following Linear(32,10) is not part of the GRU-layer parameter count requested. **Answer: C — 5,184.** A: **Incorrect:** 4992 counts the GRU weight matrices but omits both bias groups. B: **Incorrect:** 4864 does not match the required three input/recurrent gate matrices and biases. C: **Correct:** The two weight groups and two bias groups total 5184. D: **Incorrect:** 5568 overcounts the specified single-layer, unidirectional GRU.

**1. Interpret the input:** With batch_first=True, x of shape \((4,32,20)\) contains four sequences, 32 time positions and twenty input features. **2. Use the hidden-state convention:** Unlike the sequence output, the final hidden state h always uses \((\text{layers}\times\text{directions},\text{batch},\text{hidden size})\). There is one layer, one direction and hidden size 32. **3. Substitute:** h has shape \((1,4,32)\). The later slicing of out and the fully connected layer do not change the already returned h tensor. **Answer: C — h has shape (1, 4, 32).** A: **Incorrect:** Twenty is input width, not hidden width. B: **Incorrect:** batch_first does not move h’s batch axis to the front. C: **Correct:** One layer/direction, batch four and hidden width 32 give (1,4,32). D: **Incorrect:** It both misorders the hidden-state axes and substitutes the input width.

**1. Calculate the logit and sigmoid:** \(z=0.5(1)-0.5(2)=-0.5\), so \(\hat y=1/(1+e^{0.5})=0.377540669\ldots\). **2. Combine sigmoid with BCE:** Binary cross-entropy has \(\partial L/\partial z=\hat y-y\). With y=1, this is \(-0.622459331\ldots\); the sigmoid derivative is already accounted for in this simplification. **3. Differentiate each weight:** Since \(\partial z/\partial w_i=x_i\), the gradients are \((-0.622459331)(1)=-0.622459331\) and \((-0.622459331)(2)=-1.244918662\). These match the intended approximate pair in D. **Answer: D — Gradients approximately (−0.62246, −1.24492).** **Convention note:** The option’s second entry −1.2450 uses rounding of an intermediate −0.6225. Direct evaluation rounds to −1.2449 at four decimals; the small rounding discrepancy does not change the intended option. A: **Incorrect:** These use −ŷ rather than ŷ−1 for the positive target. B: **Incorrect:** These reverse the signs of the correct positive-target BCE gradients. C: **Incorrect:** These use positive ŷ multiples, which do not give the derivative for y=1. D: **Correct:** The negative gradients approximately −0.6225 and −1.245 correspond to (ŷ−y)x.

**1. Recompute the forward result:** The initial logit is −0.5 and sigmoid output is 0.377540669. For target one, the BCE logit derivative is −0.622459331. **2. Obtain weight gradients:** Multiplying by inputs one and two gives \(g_1=-0.622459331\) and \(g_2=-1.244918662\). **3. Apply SGD separately:** \(w_1^{new}=0.5-0.1g_1=0.562245933\); \(w_2^{new}=-0.5-0.1g_2=-0.375508134\). **4. Round and check direction:** The displayed weights are approximately 0.5622 and −0.3755. Both increase because their gradients are negative, raising the logit for the positive target. **Answer: B — Updated weights approximately (0.5622, −0.3755).** A: **Incorrect:** These move both weights in the gradient-ascent direction rather than the specified descent direction. B: **Correct:** They match the rate-0.1 SGD results after rounding. C: **Incorrect:** The gradients are nonzero, so the weights do not remain unchanged. D: **Incorrect:** These changes are too large for the given rate and computed gradients.