Deep Learning · Quiz 1 · Sep 2025

**1. Apply the negative example:** With the fixed weights, the score is \(s=w_0-2x_1-x_2\). Input 00 has label 0, requiring \(w_0<0\), since zero predicts 1. **2. Apply the positive examples:** Input 01 needs \(w_0\ge1\), input 10 needs \(w_0\ge2\), and input 11 needs \(w_0\ge3\). **3. Compare the constraints:** No value is both negative and at least 3. Thus OR is linearly separable in general, but cannot be implemented with these fixed negative feature weights. **Answer: E — No possible bias w0.** A: **Incorrect:** Negative bias correctly labels 00 but wrongly labels every positive example. B: **Incorrect:** Every value in this range wrongly labels 00 as 1. C: **Incorrect:** These positive biases already misclassify 00; some also fail other examples. D: **Incorrect:** These biases classify all four points as 1, including the negative example 00. E: **Correct:** The required inequalities w0<0 and w0>=3 are incompatible.

**1. Match slow and fast stable descent:** A’s smooth but slow loss reduction indicates the smallest step size, 0.001. B converges faster without oscillation, indicating the moderate rate 0.01. **2. Identify excessive steps:** C repeatedly overshoots instead of stabilizing, which is characteristic of the largest proposed rate, 1.0. **Answer: B — A → 0.001 , B → 0.01 , C → 1.0.** **Convention note:** This is a behavioral inference for the displayed training runs; learning-rate values alone do not determine convergence independently of the loss geometry. A: **Incorrect:** This reverses the slow and faster stable curves A and B. B: **Correct:** A=0.001, B=0.01 and C=1.0 match the stated behaviors. C: **Incorrect:** The largest rate is assigned to the slow smooth curve instead of the oscillating one. D: **Incorrect:** The largest rate should explain C’s oscillation, not B’s stable convergence.

**1. Compare decision functions:** An SLP uses a single affine score and threshold, so its boundary is a hyperplane. An MLP composes affine maps with nonlinear hidden activations. **2. State the representational gain:** Nonlinear hidden features permit nonlinearly separable classifications such as XOR. This is an expressive advantage, not a guarantee of faster training or less overfitting. **Answer: A — An MLP can learn non-linear decision boundaries, allowing it to classify data .** A: **Correct:** Nonlinear hidden activations allow boundaries unavailable to a single perceptron. B: **Incorrect:** Extra hidden layers do not guarantee faster optimization. C: **Incorrect:** More parameters can increase overfitting; they do not automatically prevent it. D: **Incorrect:** MLPs can perform regression with suitable outputs and losses.

**1. Examine one full-batch update:** For \(L(\theta)=N^{-1}\sum_i\ell_i(\theta)\), batch GD computes \(\nabla L=N^{-1}\sum_i\nabla\ell_i\) across all N examples before updating. **2. Scale to billions of examples:** Each update then requires an expensive full-dataset pass. SGD or mini-batch SGD estimates this gradient from fewer examples and updates sooner. **Answer: D — Batch GD requires calculating the gradient of the loss function with respect to .** A: **Incorrect:** The gradient can be accumulated from streamed chunks without loading the whole dataset into RAM simultaneously. B: **Incorrect:** Full-batch gradients have less sampling noise than stochastic estimates, not more. C: **Incorrect:** Neither the trapping claim nor guaranteed SGD escape holds generally. D: **Correct:** The mandatory full-dataset gradient computation per update is the main computational bottleneck.

**1. Differentiate at the expansion point:** For \(L(w)=\ln(1+w)\), \(L(0)=0\), \(L\prime(0)=1\), and \(L\prime\prime(0)=-1\). **2. Substitute into second-order Taylor expansion:** \(L(w)\approx0+w-\tfrac12w^2\) near zero. The negative quadratic coefficient agrees with the logarithm’s concavity. **Answer: B — w minus w squared divided by 2.** A: **Incorrect:** This keeps only the constant term and omits the nonzero derivatives. B: **Correct:** The constant, linear and quadratic coefficients are 0,1,-1/2. C: **Incorrect:** The second derivative is negative, so the quadratic sign is wrong. D: **Incorrect:** ln(1+w) is finite at w=0; its Taylor polynomial is finite.

**1. Match the output to the target:** A car price is one continuous quantity, so a standard regression head uses one neuron with a linear output. **2. Match the loss and evaluation:** Replace categorical cross-entropy with a regression loss such as MSE, and evaluate using quantities such as MAE/RMSE rather than class accuracy. Softmax normalizes class probabilities and does not represent a free scalar price. **Answer: A, B, C — Change the activation function of the final output layer to a linear function.; Set the number of neurons in the output layer to one.; Change the loss function from Categorical Cross-Entropy to Mean Squared .** **Convention note:** These are standard conversions, rather than the only possible choices: MAE and other regression losses are valid, and a constrained positive-output head may also be appropriate for prices. A: **Correct:** A linear output is the standard unconstrained regression head. B: **Correct:** One output represents the single price target. C: **Correct:** MSE is a standard loss for continuous-value prediction. D: **Incorrect:** Classification accuracy is unsuitable for ordinary continuous-price regression. E: **Incorrect:** A single-neuron softmax is always 1, and a multi-neuron softmax is a probability vector.

**1. Write NOR’s requirement:** The output must be 1 only at 000 and must become 0 whenever any input is 1. **2. Assign inhibition:** Each of the three inputs must independently veto firing. Thus all three are inhibitory. Set the threshold to 0 so the all-zero input fires when no veto is active. **3. Check necessity:** If any input were not inhibitory, its singleton-one pattern would not veto firing and would violate NOR. Hence three inhibitory inputs are required. **Answer: 3.** **Convention note:** Use the stated MP-neuron convention: an active inhibitory input overrides the threshold computation. The displayed sum rule alone, without inhibition, is not a complete NOR implementation.

**1. Differentiate the loss:** \(L\prime(w)=2(w+3)\). Interpret the printed gradient in iteration t as evaluated at the previous parameter, so \(u_t=0.8u_{t-1}-L\prime(w_{t-1})\), \(w_t=w_{t-1}+0.2u_t\). **2. First iteration:** From \(w_0=5,u_0=0\), the gradient is 16, \(u_1=-16\), and \(w_1=5-3.2=1.8\). **3. Second iteration:** The new gradient is \(2(1.8+3)=9.6\). Thus \(u_2=0.8(-16)-9.6=-22.4\), and \(w_2=1.8+0.2(-22.4)=-2.68\). **Answer: -2.68.** **Convention note:** The source writes an unindexed gradient symbol. The calculation uses the usual explicit momentum update, evaluating the gradient at the current pre-update weight.

**1. Compute the hidden feature:** \(a_1=W_1x=1(2)=2\), so \(h_1=\sigma(2)=0.880797\approx0.88\). **Answer: 0.88.**

**1. Compute the hidden feature:** \(a_1=W_1x=1(2)=2\), so \(h_1=\sigma(2)=0.880797\approx0.88\). **2. Compute class probabilities:** The logits are \((-h_1,h_1,2h_1)^T\approx(-0.88,0.88,1.76)^T\). Softmax gives \(p\approx(0.047919,0.278972,0.673109)^T\), or (0.05,0.28,0.67) to two decimals. **3. Select the true class:** The one-hot label is (0,1,0), so \(L=-\ln p_2\). With the requested two-decimal intermediate probability, \(-\ln(0.28)=1.272966\), giving 1.27. **Answer: 1.27.** **Convention note:** Retaining full precision throughout gives 1.276645, which rounds to 1.28. The source requests two-decimal intermediate calculations; 1.27 follows that convention, and both values lie inside the scoring range.

**1. Compute the hidden feature:** \(a_1=W_1x=1(2)=2\), so \(h_1=\sigma(2)=0.880797\approx0.88\). **2. Compute class probabilities:** The logits are \((-h_1,h_1,2h_1)^T\approx(-0.88,0.88,1.76)^T\). Softmax gives \(p\approx(0.047919,0.278972,0.673109)^T\), or (0.05,0.28,0.67) to two decimals. **3. Differentiate the third output weight:** Softmax with cross-entropy gives \(\partial L/\partial a_{2,k}=p_k-y_k\). Therefore \(\partial L/\partial w_{231}=(p_3-0)h_1\approx0.67(0.88)=0.5896\approx0.59\). **Answer: 0.59.**

**1. Compute the hidden feature:** \(a_1=W_1x=1(2)=2\), so \(h_1=\sigma(2)=0.880797\approx0.88\). **2. Compute class probabilities:** The logits are \((-h_1,h_1,2h_1)^T\approx(-0.88,0.88,1.76)^T\). Softmax gives \(p\approx(0.047919,0.278972,0.673109)^T\), or (0.05,0.28,0.67) to two decimals. **3. Backpropagate to the hidden neuron:** Using two-decimal probabilities, \(\delta_1=[(-1)(0.05)+(1)(0.28-1)+(2)(0.67)]h_1(1-h_1)\). The bracket equals 0.57. **4. Include the input factor:** \(\partial L/\partial W_1=0.57(0.88)(0.12)(2)=0.120384\approx0.12\). Full-precision calculation gives approximately 0.121219 and the same final answer. **Answer: 0.12.**

**1. Identify a positive mistake:** P contains label-1 examples. Under the convention \(\hat y=1\) for \(w^Tx\ge0\), a positive example is wrong exactly when \(w^Tx<0\). **2. Fill the condition:** The weighted sum, including the bias feature if used, must be negative before applying the positive correction. **Answer: A — Sum of wi xi is less than zero.** A: **Correct:** The negative weighted score is exactly a false negative. B: **Incorrect:** The unweighted feature sum does not determine the model prediction. C: **Incorrect:** The weight sum ignores the actual example and is not a mistake condition. D: **Incorrect:** A negative weight sum alone does not imply a negative example-specific score.

**1. Correct the false negative:** A positive example with negative score needs a larger score. Adding its feature vector yields \(w^+=w+x\). **2. Check the direction:** \((w+x)^Tx=w^Tx+\|x\|^2\), so its score increases. This is the unit-learning-rate perceptron correction requested. **Answer: A — w equals w plus x.** **Convention note:** A perceptron mistake loss can express this as gradient descent with a unit rate, but the bare gradient option supplies neither that loss nor the rate. The direct update is the unambiguous intended choice. A: **Correct:** Adding x raises the positive example’s score by its squared norm. B: **Incorrect:** Subtracting x lowers the score and worsens this false negative. C: **Incorrect:** A generic gradient rule is not the specified fill-in; no loss or learning rate is defined in this pseudocode. D: **Incorrect:** A gradient with respect to x does not give the requested parameter-space update.

**1. Identify a negative mistake:** N contains label-0 examples. Because zero score is assigned label 1, a negative example is misclassified whenever \(w^Tx\ge0\). **2. Include the tie case:** Using a strict positive inequality would miss score-zero false positives; the supplied nonnegative test is correct. **Answer: A — Sum of wi xi is nonnegative.** A: **Correct:** The nonnegative weighted score is precisely a false positive under the stated threshold rule. B: **Incorrect:** The unweighted input sum does not determine prediction. C: **Incorrect:** The sum of weights alone ignores the example’s feature values. D: **Incorrect:** A negative weight sum is not the condition for a false positive.

**1. Correct the false positive:** For a negative example predicted positive, subtract its vector: \(w^+=w-x\). **2. Check the effect:** Its new score is \(w^Tx-\|x\|^2\), which moves it toward the negative side of the boundary. **Answer: B — w equals w minus x.** **Convention note:** With a suitably defined mistake loss and unit rate, the explicit perceptron correction can also be expressed as a gradient step. That additional definition is absent from the generic gradient option. A: **Incorrect:** Adding x increases the false-positive score. B: **Correct:** Subtracting x decreases the example’s score by its squared norm. C: **Incorrect:** No differentiable loss or step size is provided for this generic gradient expression; the explicit perceptron update is required. D: **Incorrect:** The gradient with respect to input x is not the specified weight correction.

**1. Backpropagate through the original output weights:** Let \(\delta_2=p-y\) for the standard softmax cross-entropy objective. The hidden error is \(\delta_1=(W_2^T\delta_2)\odot\sigma\prime(a_1)\). **2. Apply the zero initialization:** Initially \(W_2=0\), so \(\delta_1=0\) and \(\nabla_{W_1}L=\delta_1x^T=0\). A simultaneous SGD step therefore leaves W1 unchanged. **Answer: A — Yes.** **Convention note:** Assume the standard supervised classification loss, simultaneous parameter updates, and no weight decay or other regularization. The newly updated W2 can allow W1 to change on later steps. A: **Correct:** The initial zero output matrix blocks the hidden-layer gradient. B: **Incorrect:** W2 may change in this step, but gradients for the same step are computed from the original parameters.

**1. Evaluate the initial output:** Zero output weights and biases give equal logits, so K classes have probabilities \(p_k=1/K\). Sigmoid hidden activations are strictly positive for finite inputs. **2. Compute the output-weight gradient:** For a one-hot target and K>1, \(\nabla_{W_2}L=(p-y)h_1^T\ne0\). With a positive learning rate, the first SGD step changes W2 even though W1 remains unchanged. **Answer: B — No.** **Convention note:** This uses the usual softmax cross-entropy, one-hot class target, finite sigmoid activations and positive learning rate. Arbitrary unspecified losses or a zero learning rate would not imply this result. A: **Incorrect:** Zero initial weights do not force their own gradients to zero; the hidden activations are nonzero. B: **Correct:** The nonzero class error times the hidden feature produces a nonzero output-weight update.