Deep Learning · Quiz 1 · Jan 2025

**1. Locate the gradient evaluation:** Ordinary momentum uses the gradient at the current parameters, for example \(u_t=\beta u_{t-1}+\nabla L(w_t)\), followed by \(w_{t+1}=w_t-\eta u_t\). **2. Compare Nesterov’s lookahead:** In the corresponding NAG convention the gradient is evaluated at \(w_t-\eta\beta u_{t-1}\), anticipating the momentum movement. That evaluation point is the distinguishing feature. **Answer: A — Momentum uses the current gradient to update the velocity, while NAG uses a .** A: **Correct:** Current-point versus lookahead-point gradient evaluation is the standard distinction. B: **Incorrect:** NAG can also require only one gradient evaluation per step; a universal speed advantage is not implied. C: **Incorrect:** NAG’s defining change is the evaluation point, not an adaptive learning-rate schedule. D: **Incorrect:** Ordinary momentum does not inherently use an adaptive learning rate.

**1. Differentiate each candidate:** The derivatives of \(\log_{10}x,\ln(x^2),\sqrt x,x^3\) are respectively \(1/(x\ln10),2/x,1/(2\sqrt x),3x^2\) near x=1. **2. Compare at x=1:** Their slopes are approximately 0.4343,2,0.5,3. The greatest is 3, from x cubed. **Answer: D — x cubed.** A: **Incorrect:** Its slope at 1 is 1/ln10, about 0.4343. B: **Incorrect:** Its slope is 2, less than the cubic’s 3. C: **Incorrect:** Its slope is 0.5, less than 3. D: **Correct:** Its derivative 3x squared equals 3 at x=1, the largest candidate slope.

**1. Measure the update:** The printed rule is ordinary gradient descent, \(w_{t+1}=w_t-\eta\nabla L(w_t)\). Thus \(\|\Delta w\|=\eta\|\nabla L(w_t)\|\). **2. Compare surfaces at fixed learning rate:** A gentle surface has small gradient magnitude and hence a small step; a steep surface has large gradient magnitude and a larger step. **Answer: A, C — The weight update is tiny at a gentle loss surface.; The weight update is large at a steep loss surface..** **Convention note:** The source abbreviates the gradient as a derivative with respect to wt. “Tiny” and “large” are relative comparisons under the same positive learning rate; absolute step size also depends on eta. A: **Correct:** Small gradient magnitude produces a small update for fixed eta. B: **Incorrect:** Large gradient magnitude does not intrinsically produce a tiny step at the same eta. C: **Correct:** The step magnitude increases with gradient magnitude. D: **Incorrect:** A small gradient produces a smaller, rather than larger, step.

**1. Count batches:** Each epoch contains \(5000/50=100\) mini-batches. **2. Count parameter updates:** One update per mini-batch over 100 epochs gives \(100\times100=10000\) updates for each parameter. **Answer: 10000.**

**1. Cover all patterns:** Four binary features give 16 distinct vectors, so the dataset contains every binary assignment. Zero mistakes mean true labels equal the stated predictions. **2. Reduce the comparison:** Put \(A=x_1+x_2\) and \(B=x_3+x_4\). Each takes values 0,1,2 with multiplicities 1,2,1. A point is positive exactly when A>=B. **3. Count labels including ties:** For A=0 there is 1 positive pattern; for A=1 there are \(2(1+2)=6\); for A=2 there are \(1(1+2+1)=4\). Total positives: \(1+6+4=11\); negatives: \(16-11=5\). **Answer: 11.**

**1. Cover all patterns:** Four binary features give 16 distinct vectors, so the dataset contains every binary assignment. Zero mistakes mean true labels equal the stated predictions. **2. Reduce the comparison:** Put \(A=x_1+x_2\) and \(B=x_3+x_4\). Each takes values 0,1,2 with multiplicities 1,2,1. A point is positive exactly when A>=B. **3. Count labels including ties:** For A=0 there is 1 positive pattern; for A=1 there are \(2(1+2)=6\); for A=2 there are \(1(1+2+1)=4\). Total positives: \(1+6+4=11\); negatives: \(16-11=5\). **Answer: 5.**

**1. Use the unchanged parameter vector:** Under \(u_t=\beta u_{t-1}+\nabla f(w_t)\) and \(w_{t+1}=w_t-\eta u_t\), a nonzero learning rate and unchanged weights imply \(u_t=0\). **2. Solve for the gradient:** \(\nabla f(w_t)=-\beta u_{t-1}=-0.8(1,-1)=(-0.8,0.8)\). The rate 0.1 cancels when establishing zero displacement. **Answer: A — (−0.8, 0.8).** A: **Correct:** This exactly cancels the 0.8-scaled history vector. B: **Incorrect:** This incorrectly multiplies by the learning rate and reverses the required direction. C: **Incorrect:** The direction is right but the magnitude has an extra factor of 0.1. D: **Incorrect:** This is the history vector, not the gradient that cancels its contribution.

**1. Use the unchanged parameter vector:** Under \(u_t=\beta u_{t-1}+\nabla f(w_t)\) and \(w_{t+1}=w_t-\eta u_t\), a nonzero learning rate and unchanged weights imply \(u_t=0\). **2. Solve for the gradient:** \(\nabla f(w_t)=-\beta u_{t-1}=-0.8(1,-1)=(-0.8,0.8)\). The rate 0.1 cancels when establishing zero displacement. **3. Compare directions:** The gradient is a negative scalar multiple of the nonzero history vector. Their cosine is -1, so the angle is \(180^\circ\). **Answer: 180.**

**1. Trace the ordered examples:** With w=(0,0), (1,1) and (-1,1) both have score zero and are correctly predicted 1. At (1,-1), the zero score wrongly predicts 1 instead of 0. **2. Correct the only mistake:** For that negative example, \(w\leftarrow w-x=(0,0)-(1,-1)=(-1,1)\). **3. Verify convergence:** The three final scores are 0,2,-2, producing labels 1,1,0. No further update occurs, so the algorithm converges after one actual weight update. **Answer: C — (−1, 1).** **Convention note:** The question requires following this particular training trajectory, not merely finding any separating vector. That is why the otherwise valid separator (0,1) is not selected. A: **Incorrect:** It predicts 1 for all examples and misclassifies (1,-1). B: **Incorrect:** It separates the data, but it is not reached by the specified zero-initialized ordered updates. C: **Correct:** This is exactly the vector reached by subtracting the third example once. D: **Incorrect:** It gives zero score to (1,-1), incorrectly predicting 1.

**1. Trace the ordered examples:** With w=(0,0), (1,1) and (-1,1) both have score zero and are correctly predicted 1. At (1,-1), the zero score wrongly predicts 1 instead of 0. **2. Correct the only mistake:** For that negative example, \(w\leftarrow w-x=(0,0)-(1,-1)=(-1,1)\). **3. Verify convergence:** The three final scores are 0,2,-2, producing labels 1,1,0. No further update occurs, so the algorithm converges after one actual weight update. **Answer: 1.**

**1. Write the trained score:** The converged vector is \((-1,1)\), so \(s=-x_1+x_2\). **2. Match each correct classification:** A correctly classified point has either \(x_2-x_1\ge0,y=1\), or \(x_2-x_1<0,y=0\). These are the two accepted cases, with zero included in the positive side. **Answer: A, B — The two cases defined by the sign of x2 minus x1.** **Convention note:** A and B are alternative correct-classification cases; they are not asserted to hold simultaneously for the same point. A: **Correct:** This is precisely the positive-side condition with the matching label. B: **Correct:** This is precisely the negative-side condition with the matching label. C: **Incorrect:** The classifier uses x2-x1, not x1+x2. For example (-2,-1) is positive despite a negative sum. D: **Incorrect:** The input sum is unrelated to the trained separator; it is not the required negative condition.

**1. Compute the logit:** \(z=(0.5)(1)+(1)(0.5)+(0.5)(1)+1=2.5\). **Answer: 2.50.**

**1. Compute the logit:** \(z=(0.5)(1)+(1)(0.5)+(0.5)(1)+1=2.5\). **2. Apply sigmoid:** \(\hat y=1/(1+e^{-2.5})=0.9241418\ldots\), giving 0.92 to two decimals. **Answer: 0.92.**

**1. Compute the logit:** \(z=(0.5)(1)+(1)(0.5)+(0.5)(1)+1=2.5\). **2. Apply sigmoid:** \(\hat y=1/(1+e^{-2.5})=0.9241418\ldots\), giving 0.92 to two decimals. **3. Evaluate the specified squared loss:** For y=0, \(L=\frac12(0-\hat y)^2=0.4270191\ldots\), which rounds to 0.43. **Answer: 0.43.** **Convention note:** Keep full precision until the final rounding. Using the previously rounded prediction 0.92 gives 0.4232, or 0.42; both lie in the stated scoring range.

**1. Compute the logit:** \(z=(0.5)(1)+(1)(0.5)+(0.5)(1)+1=2.5\). **2. Apply sigmoid:** \(\hat y=1/(1+e^{-2.5})=0.9241418\ldots\), giving 0.92 to two decimals. **3. Backpropagate and update simultaneously:** For y=0, the logit derivative is \(d=(\hat y-y)\hat y(1-\hat y)=0.06478578\). With eta=2, \(w^+=w-2dx=(0.37042845,0.93521422,0.37042845)^T\), and \(b^+=1-2d=0.87042845\). **4. Recompute after the update:** The new logit is \(2.5-2d(\|x\|^2+1)=2.07889245\), since \(\|x\|^2=2.25\). Hence \(\hat y^+=0.88883465\) and \(L^+=\frac12(\hat y^+)^2=0.3950135\), rounding to 0.40. **Answer: 0.40.** **Convention note:** The bias must be updated along with the weights. Earlier rounding can produce 0.39, which the source range also accepts; full-precision intermediate arithmetic gives 0.40.

**1. Compute the first hidden layer:** The printed W1 rows applied to x=(0,1,2) give \(a_1=(3,2,1)^T\). Sigmoid yields \(h_1\approx(0.95,0.88,0.73)^T\) under the requested two-decimal intermediate convention. **Answer: 0.95.**

**1. Compute the first hidden layer:** The printed W1 rows applied to x=(0,1,2) give \(a_1=(3,2,1)^T\). Sigmoid yields \(h_1\approx(0.95,0.88,0.73)^T\) under the requested two-decimal intermediate convention. **2. Compute the second hidden layer:** The W2 rows are (0,0,1) and (1,1,0), so \(a_2=(0.73,1.83)^T\). Thus \(h_2\approx(0.67,0.86)^T\). **Answer: 0.67.** **Convention note:** Keeping full precision instead gives h21=0.6750375, which rounds to 0.68. The source explicitly requests two-decimal intermediate calculations; using h13=0.73 gives sigmoid(0.73)=0.674805, or 0.67. Both are within the scoring range.

**1. Compute the first hidden layer:** The printed W1 rows applied to x=(0,1,2) give \(a_1=(3,2,1)^T\). Sigmoid yields \(h_1\approx(0.95,0.88,0.73)^T\) under the requested two-decimal intermediate convention. **2. Compute the second hidden layer:** The W2 rows are (0,0,1) and (1,1,0), so \(a_2=(0.73,1.83)^T\). Thus \(h_2\approx(0.67,0.86)^T\). **3. Compute softmax outputs:** W3 has rows (0,1),(1,1),(1,0), giving logits \((0.86,1.53,0.67)^T\). Normalizing their exponentials gives \(p\approx(0.26,0.52,0.22)^T\). **Answer: 0.26.**

**1. Compute the first hidden layer:** The printed W1 rows applied to x=(0,1,2) give \(a_1=(3,2,1)^T\). Sigmoid yields \(h_1\approx(0.95,0.88,0.73)^T\) under the requested two-decimal intermediate convention. **2. Compute the second hidden layer:** The W2 rows are (0,0,1) and (1,1,0), so \(a_2=(0.73,1.83)^T\). Thus \(h_2\approx(0.67,0.86)^T\). **3. Compute softmax outputs:** W3 has rows (0,1),(1,1),(1,0), giving logits \((0.86,1.53,0.67)^T\). Normalizing their exponentials gives \(p\approx(0.26,0.52,0.22)^T\). **4. Use the true class:** The target is y=(1,0,0), so \(L=-\ln p_1\). With the two-decimal probability, \(-\ln(0.26)=1.347074\), giving 1.35. **Answer: 1.35.** **Convention note:** Retaining full precision throughout yields p1=0.2636125 and L=1.333275, or 1.33. The answer 1.35 follows the requested intermediate-rounding convention; both lie within the source range.

**1. Compute the first hidden layer:** The printed W1 rows applied to x=(0,1,2) give \(a_1=(3,2,1)^T\). Sigmoid yields \(h_1\approx(0.95,0.88,0.73)^T\) under the requested two-decimal intermediate convention. **2. Compute the second hidden layer:** The W2 rows are (0,0,1) and (1,1,0), so \(a_2=(0.73,1.83)^T\). Thus \(h_2\approx(0.67,0.86)^T\). **3. Compute softmax outputs:** W3 has rows (0,1),(1,1),(1,0), giving logits \((0.86,1.53,0.67)^T\). Normalizing their exponentials gives \(p\approx(0.26,0.52,0.22)^T\). **4. Differentiate the first output-layer weight:** The requested weight w311 joins h21 to class 1. For softmax cross-entropy, \(\partial L/\partial w_{311}=(p_1-y_1)h_{21}=(0.26-1)(0.67)=-0.4958\), or -0.50. **Answer: -0.50.**