Deep Learning · Quiz 1 · Sep 2024

**1. Apply the inhibitory veto:** The function is \((\neg x_1\land\neg x_2)\land(x_3\land x_4)\). The inhibitory inputs ensure any x1=1 or x2=1 forces output 0. **2. Set the excitatory threshold:** When both inhibitory inputs are zero, the neuron must fire only when x3=x4=1. Their sum is then 2; a threshold of 2 accepts this case and rejects sums 0 and 1. **Answer: 2.** **Convention note:** Under the standard integer MP-threshold convention theta=2. If arbitrary real thresholds were allowed with a >= firing rule, every theta in (1,2] would implement the same Boolean function; 2 is the canonical supplied answer.

**1. State the firing condition:** Neither inhibitory input may be active, and both excitatory inputs must be active. **2. Check the tuples:** Only \((x_1,x_2,x_3,x_4)=(0,0,1,1)\) satisfies all four requirements. Every other listed tuple contains an active inhibitory input. **Answer: D — Input (0,0,1,1).** A: **Incorrect:** Both inhibitory inputs are active, so output is forced to zero. B: **Incorrect:** x2=1 inhibits firing, and the excitatory inputs are also both zero. C: **Incorrect:** x2=1 inhibits firing and x3=0 prevents the AND condition. D: **Correct:** No inhibitor is active and the excitatory sum reaches 2.

**1. Exponentiate the logits:** For \(a=(1,1,2)^T\), softmax is \(p=(e,e,e^2)^T/(2e+e^2)\). **2. Normalize:** Equivalently \(p=(1,1,e)^T/(2+e)\approx(0.21194,0.21194,0.57612)^T\), giving (0.21,0.21,0.58) to two decimals. **Answer: D — Column vector (0.21,0.21,0.58).** A: **Incorrect:** The largest probability is misplaced on the first coordinate despite the third logit being largest. B: **Incorrect:** This is a matrix, whereas softmax of three logits returns a three-component vector. C: **Incorrect:** This matrix has entries exceeding the permitted probability normalization and is not the output vector. D: **Correct:** It preserves equality of the first two logits and assigns the greatest probability to the third.

**1. Exponentiate the logits:** For \(a=(1,1,2)^T\), softmax is \(p=(e,e,e^2)^T/(2e+e^2)\). **2. Normalize:** Equivalently \(p=(1,1,e)^T/(2+e)\approx(0.21194,0.21194,0.57612)^T\), giving (0.21,0.21,0.58) to two decimals. **3. Use the full Jacobian:** \(J_{ij}=p_i(\delta_{ij}-p_j)\), or \(J=\operatorname{diag}(p)-pp^T\). Its diagonal is positive and its off-diagonal entries are negative. **4. Evaluate and compare:** Approximately \(J=\begin{pmatrix}0.1670&-0.0449&-0.1221\\-0.0449&0.1670&-0.1221\\-0.1221&-0.1221&0.2442\end{pmatrix}\). C is the corresponding displayed matrix, using the paper’s approximate 0.16 diagonal values. **Answer: C — The symmetric 3 by 3 softmax Jacobian in option C.** **Convention note:** The source displays 0.16 for the first two diagonal entries; correct nearest-two-decimal rounding of the full-precision value 0.167022 is 0.17. C remains the sole option with the correct complete Jacobian structure; the printed entries use coarse/truncated approximations. A: **Incorrect:** A vector cannot represent the full three-by-three softmax derivative, and 0.68 is not its last diagonal derivative. B: **Incorrect:** The derivative must be symmetric with positive diagonal and negative off-diagonal elements; this arrangement fails. C: **Correct:** This is the correct Jacobian structure and the intended approximate values. D: **Incorrect:** This lists only approximate self-derivatives and omits all cross-coordinate derivatives.

**1. Write the score:** \(s=w_0+3x_1-x_2\), with class 1 for s>=0. **2. Enforce all four labels:** The negative examples require \(w_0<0\) for 00 and \(w_0<1\) for 01. The positives require \(w_0\ge-3\) for 10 and \(w_0\ge-2\) for 11. **3. Intersect the bounds:** The strongest lower bound is -2 and strongest upper bound is 0, strictly excluded. Thus \(w_0\in[-2,0)\). **Answer: A — Bias w0 in the interval [-2,0).** A: **Correct:** This is exactly the intersection of all four label constraints. B: **Incorrect:** Values below -2 wrongly classify 11, and much of the valid interval is omitted. C: **Incorrect:** Positive biases in this interval wrongly classify 00, and the valid endpoint -2 is omitted. D: **Incorrect:** Values between -3 and -2 wrongly classify 11. E: **Incorrect:** Option A already gives the exact valid range.

**1. Count batches per epoch:** \(20000/500=40\). **2. Count all updates:** Ten epochs give \(40\times10=400\) parameter updates. Batch size controls examples per update, not the number of parameters updated. **Answer: A — 400.** A: **Correct:** Forty batches each epoch times ten epochs gives 400. B: **Incorrect:** This counts training examples, not mini-batch updates. C: **Incorrect:** This does not follow from the batch or epoch counts. D: **Incorrect:** Forty is the update count for only one epoch.

**1. Compare evaluation locations:** Traditional momentum evaluates the gradient at the current parameters. NAG first forms a tentative position by moving along the existing momentum. **2. Apply the anticipatory correction:** NAG evaluates its gradient at that lookahead position, enabling a correction informed by where the momentum is taking the parameters. **Answer: C — NAG calculates the gradient at a look-ahead point..** A: **Incorrect:** Gradient history is also used by traditional momentum; this does not distinguish NAG. B: **Incorrect:** This reverses NAG’s defining lookahead evaluation rule. C: **Correct:** The gradient is evaluated at an anticipated momentum-shifted parameter position. D: **Incorrect:** NAG can also be applied to nonconvex objectives; convexity assumptions concern particular theoretical guarantees.

**1. Identify explicit schedules:** Step decay reduces the learning rate at chosen intervals, such as \(\eta_t=\eta_0\gamma^{\lfloor t/s\rfloor}\). Exponential decay continuously reduces it by a multiplicative factor, such as \(\eta_t=\eta_0e^{-kt}\). **2. Separate schedules from update algorithms:** Momentum accumulates gradient history, and SGD uses sampled gradients. Either can be combined with a learning-rate schedule, but neither name alone specifies rate adjustment. **Answer: A, D — Step decay; Exponential decay.** A: **Correct:** It explicitly changes the learning rate at selected training steps. B: **Incorrect:** Momentum describes gradient-history accumulation, not a learning-rate schedule. C: **Incorrect:** SGD describes stochastic gradient estimation; the rate can remain constant. D: **Correct:** It explicitly reduces the learning rate exponentially with training progress.

**1. Compute the current prediction:** \(w^Tx+b=2.5\), so \(\hat y=0.9241418\). For \(L=\frac12(y-\hat y)^2\) and y=0, \(d=\partial L/\partial z=\hat y^2(1-\hat y)=0.06478578\). **2. Update the requested weight vector:** With eta=2, \(w^+=w-2dx=(0.37042845,0.93521422,0.37042845)^T\). Holding the separately stated bias b=1 fixed gives \(z^+=2.5-2d\|x\|^2=2.20846401\). **3. Re-evaluate the loss:** \(\hat y^+=0.90100701\), so \(L^+=\frac12(0.90100701)^2=0.40590682\), rounded to 0.41. **Answer: 0.41.** **Convention note:** The question explicitly requests a weight-vector update and does not explicitly request a bias update; the main answer keeps b fixed. If bias is also trained, b+=0.87042845 and the loss is 0.39501351, or 0.40. Both interpretations are within the supplied 0.37–0.41 range. Earlier two-decimal rounding can also shift the last digit.

**1. Differentiate the quadratic:** \(L(w)=(w-4)^2\), so \(L\prime(w)=2(w-4)\). Use \(u_t=0.9u_{t-1}+L\prime(w_t)\), \(w_{t+1}=w_t-0.1u_t\), with the usual initial history zero. **2. First update:** At w0=0, the gradient is -8 and u0=-8, giving \(w_1=0-0.1(-8)=0.8\). **3. Second update:** The new gradient is \(2(0.8-4)=-6.4\). Hence \(u_1=0.9(-8)-6.4=-13.6\), and \(w_2=0.8-0.1(-13.6)=2.16\). **Answer: 2.16.** **Convention note:** The initial momentum history is not explicitly stated; the conventional zero initialization is used. A different initial history would change the answer.

**1. Compute the hidden layers:** The printed W1 applied to x=(1,0,1) gives \(a_1=(1,2)^T\), so \(h_1=(0.7310586,0.8807971)^T\). W2 gives \(a_2=(0.7310586,1.6118557)^T\), and \(h_2\approx(0.6750375,0.8336689)^T\). **2. Compute softmax:** The output logits are \((h_{21},h_{21},h_{21}+h_{22})^T\). Subtracting h21 from all logits gives \((0,0,h_{22})\), so \(p=(1,1,e^{h_{22}})^T/(2+e^{h_{22}})\approx(0.2324,0.2324,0.5352)^T\). **3. Select the target class:** For y=(0,0,1), \(L=-\ln p_3=\ln(2+e^{h_{22}})-h_{22}\approx0.625\), giving 0.63 to two decimals. **Answer: 0.63.**