Deep Learning · End Term · May 2026 · FN

**1. Compute the affine input:** The input is \((3,-2)\), the weight vector is \((0.5,1.5)\), and the bias is 0.5. Thus \(z=0.5(3)+1.5(-2)+0.5=1.5-3+0.5=-1\). **2. Select the PReLU branch:** With \(\alpha=0.1\), PReLU returns \(z\) for \(z\ge0\) and \(\alpha z\) for \(z<0\). Here the input is negative. **3. Evaluate:** The activation is \(0.1(-1)=-0.1\), correct to one decimal place. The negative slope changes the activation, not the preceding weighted sum. **Answer: -0.1.**

**1. Set up the spatial size equation:** The input side is 300, kernel side 15 and padding zero. Hence \(96=\lfloor(300-15)/S\rfloor+1\), or \(\lfloor285/S\rfloor=95\). **2. Solve the floor constraint:** This requires \(95\le285/S<96\). For positive \(S\), it gives \(285/96<S\le285/95\), that is \(2.96875<S\le3\). The unique positive integer stride is 3. **3. Verify the full output:** Substitution gives \(285/3+1=96\) in both spatial dimensions. The ten filters produce the ten output channels; their depth six matches the six input channels. **Answer: 3.**

**1. Derive the recurrence:** For \(f=\tfrac12(25x^2+y^2)\) and rate 0.1, \(x_{t+1}=(1-0.1\cdot25)x_t=-1.5x_t\), whereas \(y_{t+1}=(1-0.1)y_t=0.9y_t\). **2. Test convergence factors:** Starting from one, \(x_t=(-1.5)^t\) changes sign and grows in magnitude because \(|-1.5|>1\). In contrast, \(y_t=0.9^t\to0\). **3. Identify the unstable coordinate:** Only the x-direction is unstable. Its higher curvature requires a smaller rate: \(|1-25\eta|<1\) requires \(0<\eta<0.08\), which 0.1 violates. **Answer: B — The x-coordinate fails to converge.** A: **Incorrect:** The y-coordinate contracts by the factor 0.9 and converges to zero. B: **Correct:** The x-coordinate is multiplied by −1.5, so its magnitude grows while its sign alternates. C: **Incorrect:** Only x diverges; y contracts rather than diverging equally. D: **Incorrect:** The coordinates have different curvatures and the chosen rate is too large for x.

**1. Calculate the initial gradient:** At \((x_0,y_0)=(1,1)\), the gradient of \(\tfrac12(25x^2+y^2)\) is \((25,1)\). **2. Build the AdaGrad sums:** With zero initial accumulators, \(v_x^{(1)}=25^2=625\) and \(v_y^{(1)}=1^2=1\). Each coordinate gets a separate denominator. **3. Calculate the effective rates:** Using the stated convention, \(\eta_x^{(1)}=0.1/\sqrt{625+10^{-8}}\approx0.004\), while \(\eta_y^{(1)}=0.1/\sqrt{1+10^{-8}}\approx0.100\). **4. Add and round:** Their sum is approximately \(0.004+0.100=0.104\), correct to three decimals. **Answer: 0.104.**

**1. Compare coordinate scales:** Initially the gradients are 25 along x and 1 along y. AdaGrad accumulates their squares, 625 and 1, and uses their square roots to scale the base rate. **2. Compute the reductions:** Ignoring only the negligible \(10^{-8}\) stabilizer, the x-rate becomes \(0.1/25=0.004\), a 25-fold reduction. The y-rate stays approximately \(0.1/1=0.1\). **3. Explain the benefit:** The steeper x-direction receives a smaller rate, countering the overshooting seen with vanilla gradient descent. This initial calculation illustrates adaptation without asserting that both rates stay fixed forever. **Answer: A — The initial effective learning rate is reduced 25-fold along x.** A: **Correct:** The initial x-gradient magnitude is 25, producing the 25-fold rate reduction. B: **Incorrect:** The y-gradient magnitude is one, so its initial effective rate is approximately unchanged. C: **Incorrect:** Five is not the square root of the y-accumulator, which equals one. D: **Incorrect:** The accumulators differ by coordinate, so the stated common 50-fold reduction is incorrect.

**1. Compose two affine layers:** Without a nonlinear activation, \(W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2)\). This is another affine map. **2. Extend to the full network:** Repeating the same substitution through all hidden layers gives \(W\prime x+b\prime\). Adding identity layers cannot create nonlinear dependence on the input; bottlenecks can even impose rank restrictions. **3. Explain the role of nonlinearity:** A nonlinear activation such as sigmoid prevents this affine collapse and permits nonlinear decision boundaries with a suitable architecture. Width alone cannot overcome the collapse when every activation is identity. **Answer: A, C — A and C — affine collapse with identity activations; nonlinearity enables nonlinear boundaries.** A: **Correct:** Composition of affine layers remains affine, regardless of the number of identity hidden layers. B: **Incorrect:** Extra identity layers do not exceed the function class of an unrestricted single affine map. C: **Correct:** Nonlinear activations allow a network to represent nonlinear input–output relationships and decision boundaries. D: **Incorrect:** Any finite stack of identity-activated affine layers still computes an affine map, even with many neurons.

**1. Count one head:** Each of its query, key and value matrices has shape \(512\times64\), so one head has \(3(512)(64)=98{,}304\) weights. **2. Count all eight heads:** Their combined projection weights are \(8(98{,}304)=786{,}432\). The concatenated head dimension is \(8(64)=512\). **3. Count the output projection:** The output matrix is \(512\times512\), contributing \(262{,}144\) weights. **4. Add the counts:** \(786{,}432+262{,}144=1{,}048{,}576\). Biases are excluded as explicitly requested. **Answer: 1048576.**

**1. Count a single transformation:** A linear transformation from a D-dimensional input to a D-dimensional output has a \(D\times D\) weight matrix and D biases, giving \(D^2+D\) parameters. **2. Count the three separate projections:** Queries, keys and values each have their own transformation. Their total is \(3(D^2+D)=3D^2+3D\). **3. Account for token sharing:** The same projection parameters are reused for every one of the N input vectors. Sequence length changes the amount of computation, not this parameter count. **Answer: B — 3D² + 3D learnable parameters.** A: **Incorrect:** 3D² counts the weights but omits the three length-D bias vectors. B: **Correct:** Three D-to-D affine projections contribute exactly 3D² + 3D parameters. C: **Incorrect:** Multiplying by N incorrectly assigns a separate set of learned weights to every token. D: **Incorrect:** The projection matrices act on feature dimensions D, not on the token-count dimension N.

**1. Identify the attention matrix:** With N queries and N keys, every query is compared against every key. The score matrix \(QK^T\) therefore has shape \(N\times N\). **2. Count normalized weights:** Applying a row-wise softmax changes the values but retains the same matrix shape. For \(N=100\), there are \(100(100)=10{,}000\) entries. **3. Separate feature dimension from token pairs:** The given \(D=16\) determines the length of query and key vectors used inside each dot product, rather than the number of attention weights. **Answer: 10000.**

**1. Flatten the complete input:** N vectors of dimension D give \(DN\) scalar inputs. The requested output also contains \(DN\) scalar values. **2. Count dense connections:** A fully connected map from \(DN\) inputs to \(DN\) outputs has \((DN)(DN)=D^2N^2\) weights. **3. Add output biases:** There is one bias per scalar output, hence \(DN\) biases. The total is \(D^2N^2+DN\). Unlike token-wise projection, this map directly mixes every flattened input with every output. **Answer: B — D²N² + DN learnable parameters.** A: **Incorrect:** D²N² counts only the connection weights and omits output biases. B: **Correct:** The square dense weight matrix plus DN output biases gives D²N² + DN. C: **Incorrect:** The bias count is the number of scalar outputs DN, not D². D: **Incorrect:** Only D biases would be insufficient for DN independently biased outputs.

**1. Compute query 1:** Token 1 is \((1,2,0)\), and multiplying by \(W_Q=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}\) gives \(q_1=(1,2)\). **2. Compute all keys:** Using \(W_K=\begin{bmatrix}1&1\\0&1\\1&0\end{bmatrix}\), the three token rows \((1,2,0),(0,1,1),(2,0,1)\) give \(k_1=(1,3),k_2=(1,1),k_3=(3,2)\). **3. Compare dot products:** The scores are \(q_1\cdot k_1=1+6=7\), \(q_1\cdot k_2=1+2=3\), and \(q_1\cdot k_3=3+4=7\). The smallest score, three, belongs to token 2. **Answer: 2.**

**1. Reconstruct the scores:** Multiplication by the supplied query and key matrices gives \(q_1=(1,2)\) and \(k_1=(1,3),k_2=(1,1),k_3=(3,2)\). Their dot products are 7,3,7. **2. Scale before softmax:** The key dimension is two, so the logits are \(7/\sqrt2,3/\sqrt2,7/\sqrt2\). The requested weight is \(\alpha_{13}=e^{7/\sqrt2}/(2e^{7/\sqrt2}+e^{3/\sqrt2})\). **3. Evaluate stably:** Divide numerator and denominator by \(e^{7/\sqrt2}\): \[\alpha_{13}=\frac{1}{2+e^{-4/\sqrt2}}=0.4856477146\ldots.\] To three decimals this is 0.486; tied largest scores split nearly all the weight. **Answer: 0.486.**

**1. Recompute the attention weights:** For query \(q_1=(1,2)\), the three scaled logits are \((7,3,7)/\sqrt2\). Their softmax gives \((\alpha_{11},\alpha_{12},\alpha_{13})\approx(0.485647715,0.028704571,0.485647715)\). **2. Compute value vectors:** Multiplying the three input rows by \(W_V=\begin{bmatrix}1&0\\1&1\\0&1\end{bmatrix}\) gives \(v_1=(3,2),v_2=(1,2),v_3=(2,1)\). **3. Form the weighted output:** \(z_1=\alpha_{11}(3,2)+\alpha_{12}(1,2)+\alpha_{13}(2,1)\approx(2.456943144,1.514352285)\). **4. Add the requested components:** \(z_{11}+z_{12}=3.971295429\ldots\), which rounds to 3.97. Retaining unrounded softmax weights avoids unnecessary error. **Answer: 3.97.**

**1. Identify the compression constraint:** A non-attention encoder must summarize every source token into one fixed-dimensional final state. A longer input places more distinct details and relationships into the same-size representation. **2. Relate compression to decoding:** Every decoder step depends on that same compressed context. Information lost during encoding cannot be recovered by choosing a later decoder position, making long inputs more difficult. **3. Explain why attention helps:** Attention allows a decoder to access different source-position representations at different steps. This directly addresses the fixed-context information bottleneck described here. **Answer: B — B — the fixed-dimensional context is an information bottleneck.** A: **Incorrect:** Vocabulary size counts available token types, not the maximum number of tokens the decoder can generate. B: **Correct:** A fixed-size context must carry more information as the source length grows, creating the described bottleneck. C: **Incorrect:** Teacher forcing supplies target history during training; it has no inherent rule forbidding longer source sequences. D: **Incorrect:** One-hot dimension depends on vocabulary size rather than sentence length; longer sequences contain more vectors.

**1. Read the stated loss carefully:** The formula is \(L=-\log\sigma(w^Tc)-\log\sigma(-w_{\mathrm{neg}}^Tc)\), with \(w=(1,0),c=(1,1),w_{\mathrm{neg}}=(0,1)\). Thus both unnegated dot products equal one. **2. Evaluate the two probabilities:** \(\sigma(1)=0.731058579\ldots\) and \(\sigma(-1)=0.268941421\ldots\). The positive example should score high, while the sampled negative should score low. **3. Add both loss terms:** \[-\log(0.731058579)-\log(0.268941421)=0.313261688+1.313261688=1.626523375.\] The three-decimal answer is 1.627. **Answer: 1.627.** **Convention note:** The solution follows the exact negative-sample term shown in this paper, involving w_neg and c. It does not silently replace that term with a differently parameterized skip-gram objective.

**1. Differentiate with respect to the requested vector:** The loss is \(-\log\sigma(w^Tc)-\log\sigma(-w_{\mathrm{neg}}^Tc)\). When updating w alone, c and the separately named negative vector are fixed, so the second term has derivative zero with respect to w. **2. Compute the gradient:** Since \(w^Tc=1\), \(\nabla_wL=(\sigma(1)-1)c=(-0.268941421,-0.268941421)\). **3. Perform the update:** With \(\eta=0.5\), \(w^{\mathrm{new}}=(1,0)-0.5\nabla_wL=(1.134470711,0.134470711)\). **4. Calculate cosine similarity:** \[\frac{w^{\mathrm{new}}\cdot c}{\|w^{\mathrm{new}}\|\|c\|}=\frac{1.134470711+0.134470711}{\sqrt{1.134470711^2+0.134470711^2}\sqrt2}=0.785423074\ldots.\] Rounded to three decimals, the result is 0.785. **Answer: 0.785.** **Convention note:** Only the word vector w is updated as requested. The negative term in this paper depends on w_neg and c, not on w; including a negative-sample derivative in the update of w would solve a different objective.

**1. Use the empirical PMI definition:** \(\operatorname{PMI}(w,c)=\log[P(w,c)/(P(w)P(c))]\), using natural logarithms and the common count normalizer \(N=10{,}000\) given by the problem. **2. Convert counts:** \(P(w)=50/10{,}000=0.005\), \(P(c)=40/10{,}000=0.004\), and \(P(w,c)=10/10{,}000=0.001\). **3. Compare observed co-occurrence with independence:** The independence product is \(0.005(0.004)=0.00002\). Their ratio is \(0.001/0.00002=50\). **4. Take the logarithm:** \(\log50=3.912023005\ldots\), so the two-decimal answer is 3.91. The positive PMI indicates more co-occurrence than expected under independence. **Answer: 3.91.**

**1. Check the supplied PMI:** With pair count 2, marginal counts 200 and 300, and common normalizer 10,000, the probability ratio is \((2\cdot10{,}000)/(200\cdot300)=1/3\). Its natural log is approximately −1.0986, matching the question. **2. Apply the transformation:** Positive PMI is defined by \(\operatorname{PPMI}(w,c)=\max(\operatorname{PMI}(w,c),0)\). Negative association values are clipped to zero. **3. Store the value:** \(\max(-1.0986,0)=0\). PPMI is not the absolute value of PMI, so the stored number is not positive 1.0986. **Answer: 0.**

**1. Consider gradient averaging:** A mini-batch gradient averages per-example gradients. With representative, approximately independent sampling, averaging four times as many examples reduces its sampling variance, approximately to one-quarter. **2. Count updates per epoch:** For a fixed dataset of size M, the number of updates is approximately \(M/B\). Increasing batch size fourfold therefore reduces updates per epoch by roughly a factor of four, up to the final-batch convention. **3. Separate variance from optimization quality:** A less noisy update does not guarantee a better local minimum or faster convergence when the learning rate is held fixed. The reliable intended comparison is lower variance and fewer updates. **Answer: B — Each update uses a gradient estimate with lower variance, while the number .** **Convention note:** The variance comparison assumes comparable representative sampling. For sampling without replacement, a finite-population correction applies, while the same direction of variance reduction holds. A: **Incorrect:** Larger batches generally lower gradient-estimate variance rather than raise it. B: **Correct:** Averaging more examples reduces sampling variance, and fewer batches fit into a fixed-size epoch. C: **Incorrect:** The number of updates per epoch decreases rather than increasing fourfold. D: **Incorrect:** A larger batch does not guarantee a better optimum; optimization and generalization depend on more than batch size.