Deep Learning · End Term · May 2025 · FN

**1. Read the MP neuron’s prediction rule:** Without inhibitory inputs, output 1 is produced when x₁+x₂ reaches the threshold θ. The four input sums are 0,1,1,2. **2. Check the threshold that achieves perfect classification:** At θ=1 the predictions are (0,1,1,1), exactly the OR truth table. The graph reaches NCC=4 at that threshold. **3. Verify the full count pattern:** Against OR, thresholds 0,1,2,3 give NCC counts 3,4,2,1 respectively, matching the shown step chart. **Answer: C — OR.** A: **Incorrect:** XOR has no threshold giving all four correct with purely excitatory equal-weight inputs. B: **Incorrect:** AND is perfect at threshold 2, not at threshold 1. C: **Correct:** OR gives the plotted correctness sequence 3,4,2,1. D: **Incorrect:** NAND does not match this threshold-count pattern. E: **Incorrect:** OR already supplies an exact match.

**1. Identify optimization’s objective:** During fitting, the optimizer adjusts parameters to reduce the training objective. In the unregularized comparison this means reducing training loss or error, so Statement 1 is true. **2. Identify regularization’s purpose:** Regularization restricts the fit or penalizes complexity to improve performance on unseen data. It often accepts a larger training error in exchange for less overfitting. **3. Evaluate the claimed consequence:** Preventing overfitting does not imply a reduction in training error. Statement 2 therefore combines a correct purpose with an incorrect consequence, making the statement false. **Answer: B — Statement 1 is true, but Statement 2 is false..** A: **Incorrect:** Statement 1 is true in the stated optimization-versus-regularization comparison. B: **Correct:** Optimization fits the training objective, whereas regularization does not guarantee smaller training error. C: **Incorrect:** It reverses the truth values of the two statements. D: **Incorrect:** The second statement incorrectly equates generalization improvement with reduced training error.

**1. Build the left rise:** The term \(2\sigma(5x)=2/(1+e^{-5x})\) rises from 0 to 2 near x=0. **2. Build the right fall by subtraction:** The term \(2\sigma(5(x-5))=2/(1+e^{-5x+25})\) rises near x=5. Subtracting it turns the earlier rise back down, leaving a plateau near 2 between the transitions. **3. Match the complete tower:** \[f(x)=\frac2{1+e^{-5x}}-\frac2{1+e^{-5x+25}}.\] Far left both terms are near 0; between 0 and 5 only the first is near 2; far right the two approximately cancel. **Answer: B — f(x)=2σ(5x)−2σ(5x−25).** A: **Incorrect:** It reverses the subtraction and has unit rather than height-2 scaling. B: **Correct:** It creates a positive height-2 plateau between rising edges centered at 0 and 5. C: **Incorrect:** It creates a negative tower by subtracting in the wrong order. D: **Incorrect:** It has the right orientation but a plateau height near 1 rather than 2.

**1. Derive one output-weight gradient:** With output logit z=Σⱼwⱼhⱼ, sigmoid output and binary cross-entropy, \(\partial L_i/\partial z_i=\hat y_i-y_i\). Therefore \[\frac{\partial L_i}{\partial w_j}=(\hat y_i-y_i)h_j^{(i)}.\] The sigmoid derivative is already accounted for by the cross-entropy simplification. **2. Average over the examples used in an update:** Full-batch GD uses \(\Delta w_j=-\eta N^{-1}\sum_{i=1}^N(\hat y_i-y_i)h_j^{(i)}\). Mini-batch GD replaces N by the selected batch size B and sums over that batch. **3. Specialize to stochastic GD:** A one-example update is \(\Delta w_j=-\eta(\hat y_i-y_i)h_j^{(i)}\). Thus A,B,C are the three correct update forms under their stated methods. **Answer: A, B, C — the full-batch mean, mini-batch mean and single-example SGD formulas.** A: **Correct:** It is the full-dataset mean gradient multiplied by −η. B: **Correct:** It is the mean over the B examples in the current mini-batch. C: **Correct:** It uses the single-example gradient appropriate to SGD. D: **Incorrect:** It incorrectly inserts another output sigmoid derivative after the BCE cancellation. E: **Incorrect:** It multiplies by B/N and sums over the dataset instead of averaging the selected batch. F: **Incorrect:** It describes a B-example mini-batch mean rather than the stated single-example SGD update.

**1. Project the input columns:** The sequence columns are x₁=(1,1)ᵀ and x₂=(1,0)ᵀ. With \(W_Q=W_K=\begin{pmatrix}2&-1\\1&-1\end{pmatrix}\), \[Q=K=W_QX=\begin{pmatrix}1&2\\0&1\end{pmatrix}.\] **2. Form pairwise query–key scores:** Because tokens are columns, use QᵀK: \[Q^TK=\begin{pmatrix}1&2\\2&5\end{pmatrix}.\] Each entry is the dot product of a query column and a key column. **3. Sum the four entries:** The requested unscaled score sum is 1+2+2+5=10. Scaling by √dₖ or applying softmax belongs to the following calculations, not this one. **Answer: 10.**

**1. Project the input columns:** The sequence columns are x₁=(1,1)ᵀ and x₂=(1,0)ᵀ. With \(W_Q=W_K=\begin{pmatrix}2&-1\\1&-1\end{pmatrix}\), \[Q=K=W_QX=\begin{pmatrix}1&2\\0&1\end{pmatrix}.\] **2. Form pairwise query–key scores:** Because tokens are columns, use QᵀK: \[Q^TK=\begin{pmatrix}1&2\\2&5\end{pmatrix}.\] Each entry is the dot product of a query column and a key column. **3. Scale and normalize each row:** With dₖ=2, divide each score by √2. Rowwise softmax gives \[A\approx\begin{pmatrix}0.330238&0.669762\\0.107042&0.892958\end{pmatrix}.\] For instance, the first row’s last entry is \(1/(1+e^{-1/\sqrt2})\); the second is \(1/(1+e^{-3/\sqrt2})\). **4. Sum the last column:** \(A_{12}+A_{22}\approx0.669762+0.892958=1.562720\). To two decimal places, enter 1.56. Columns need not sum to one because normalization is performed separately in each row. **Answer: 1.56.**

**1. Determine the padded window positions:** Add one zero border around the 4×4 input. A 3×3 kernel with stride 2 gives \(\lfloor(4+2-3)/2\rfloor+1=2\) positions per side. The kernel uses its four corners and center. **2. Evaluate the four windows:** Their nonzero input sums are 1+1=2, 3+1+2=6, 2+1=3, and 1+1+2+0+1=5. Every result is positive, so ReLU leaves \[H=\begin{pmatrix}2&6\\3&5\end{pmatrix}.\] **Answer: B — the matrix with rows (2,6), (3,5).** A: **Incorrect:** It does not match the padded stride-2 window sums. B: **Correct:** It gives the four correct activated convolution values. C: **Incorrect:** Its entries do not match the supplied kernel and padding. D: **Incorrect:** Its first-row values and bottom-right value differ from the calculated map.

**1. Determine the padded window positions:** Add one zero border around the 4×4 input. A 3×3 kernel with stride 2 gives \(\lfloor(4+2-3)/2\rfloor+1=2\) positions per side. The kernel uses its four corners and center. **2. Evaluate the four windows:** Their nonzero input sums are 1+1=2, 3+1+2=6, 2+1=3, and 1+1+2+0+1=5. Every result is positive, so ReLU leaves \[H=\begin{pmatrix}2&6\\3&5\end{pmatrix}.\] **3. Apply the single max-pooling window:** A 2×2, stride-2 pool covers the whole map once, returning \(\hat y=\max(2,6,3,5)=6\). **Answer: 6.**

**1. Determine the padded window positions:** Add one zero border around the 4×4 input. A 3×3 kernel with stride 2 gives \(\lfloor(4+2-3)/2\rfloor+1=2\) positions per side. The kernel uses its four corners and center. **2. Evaluate the four windows:** Their nonzero input sums are 1+1=2, 3+1+2=6, 2+1=3, and 1+1+2+0+1=5. Every result is positive, so ReLU leaves \[H=\begin{pmatrix}2&6\\3&5\end{pmatrix}.\] **3. Follow the maximizing convolution entry:** Max pooling uniquely selects the top-right value 6. Its ReLU derivative is 1, and the bottom-right kernel entry K₃₃ multiplies input X₂₄=2 in that padded window. **4. Differentiate the loss:** With target y=3, \(\partial L/\partial\hat y=\hat y-y=6-3=3\) for the specified half-squared loss. Thus \[\frac{\partial L}{\partial K_{33}}=3\cdot1\cdot1\cdot2=6.\] Other windows have zero max-pooling derivative here. **Answer: 6.**

**1. Select the good context vector:** From C, the column for good is h=(1,1)ᵀ. Dot it with W’s good,bad,ugly columns (1,−1),(0,1),(−1,0), giving logits (0,1,−1). **2. Normalize all vocabulary scores:** \[P(bad\mid good)=\frac{e^1}{e^0+e^1+e^{-1}}=\frac{2.718282}{1+2.718282+0.367879}.\] **3. Evaluate and round:** The probability is 0.665240956, giving 0.67 to two decimal places. **Answer: 0.67.** **Convention note:** The exported acceptance interval ends at 72, apparently from a missing decimal point. The softmax calculation independently determines the answer.

**1. Generate from good:** Its context vector (1,1) gives logits (0,1,−1), so bad has the highest probability and is word1. **2. Feed the selected word back:** The context vector of bad is (−1,−1), giving logits (0,−1,1). Thus word2 is ugly. The context vector of ugly is (1,−1), giving logits (2,−1,−1), so word3 is good. **3. Return exactly the three generated words:** The supplied seed good is not included in the requested sentence, which is bad ugly good. No sampling is performed because each step retains the maximum-probability word. **Answer: A — bad ugly good.** A: **Correct:** It follows the three successive argmax transitions good→bad→ugly→good. B: **Incorrect:** It incorrectly includes the seed as the first generated word. C: **Incorrect:** After bad, the highest-scoring next word is ugly, not bad. D: **Incorrect:** Good is not the highest-scoring output when the context is good. E: **Incorrect:** After ugly, the highest-scoring output is good rather than bad.

**1. Distinguish word and context parameters:** The context good selects h=(1,1)ᵀ from C. The output word embedding for good is W_good=(1,−1)ᵀ. Its logit is 0, and \(p_{good}=1/(1+e+e^{-1})=0.244728471\). **2. Differentiate cross-entropy for the target good:** The target indicator for good is 1, so \[\nabla_{W_{good}}L=(p_{good}-1)h=(-0.755271529,-0.755271529)^T.\] **3. Apply η=1 using the old forward values:** \[W_{good,new}=(1,-1)^T-\nabla L=(1.755271529,-0.244728471)^T.\] The closest listed vector is (1.76,−0.24). This is the output embedding, not the context embedding. **Answer: A — (1.76, -0.24).** A: **Correct:** It rounds the computed updated output word embedding. B: **Incorrect:** It moves in the wrong direction or by the wrong gradient. C: **Incorrect:** It fails to retain the original second component’s −1 starting value. D: **Incorrect:** It incorrectly makes the second component more negative despite the negative gradient there. E: **Incorrect:** It does not match the shared positive-coordinate increment from the gradient.

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Report the first hidden value:** The question asks for s₁, which is exactly 1/4=0.25. The second input does not affect this earlier state. **Answer: 0.25.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Report the second hidden value:** The two contributions to its pre-activation cancel exactly. A zero sigmoid logit gives s₂=0.5, not zero. **Answer: 0.5.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Apply the linear output weight:** \(\hat y=vs_2=2(1/2)=1\). The output neuron is linear, so no additional sigmoid is applied after multiplying by v. **Answer: 1.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Differentiate the recurrent path:** Since s₀=0, \(ds_1/dw=\sigma\prime(ux_1+ws_0)s_0=0\). At the second step, \[\frac{ds_2}{dw}=s_2(1-s_2)\left(s_1+w\frac{ds_1}{dw}\right)=\tfrac12\tfrac12\tfrac14=\tfrac1{16}.\] **3. Include output and loss derivatives:** ŷ=1,y=0 and v=2. Therefore \[\frac{\partial L}{\partial w}=(\hat y-y)v\frac{ds_2}{dw}=1\cdot2\cdot\tfrac1{16}=\tfrac18=0.125.\] Although w is shared across time, the earlier contribution is exactly zero here. **Answer: 0.125.**

**1. Write the first residual block:** Each skip connection adds its input to the ReLU output. At x=1 and w₁=1, \(r_1=x+\operatorname{ReLU}(w_1x)=1+1=2\). **2. Evaluate the second residual branch:** With w₂=−5, its ReLU receives −5(2)=−10 and outputs zero. **3. Include the bypass:** The final output is \(f(1)=r_1+\operatorname{ReLU}(w_2r_1)=2+0=2\). The skip path preserves the positive signal even though the second nonlinear branch is inactive. **Answer: 2.**