Deep Learning · End Term · Sep 2025 · FN

**1. Compare representations:** A one-hot vector has one nonzero coordinate and treats different words as unrelated directions. For any two distinct words their one-hot dot product is zero, regardless of meaning. **2. Explain the learned alternative:** An embedding assigns a short dense vector learned from contexts or a prediction objective. Words used in similar ways can acquire nearby vectors, allowing semantic and usage relationships to influence later models. **3. Identify the main advantage:** Capturing those relationships is the key benefit. Neither automatic spelling correction nor greater sparsity is guaranteed. **Answer: B — Captures semantic relationships.** A: **Incorrect:** A lookup can be simple, but implementation simplicity is not the main representational benefit. B: **Correct:** Similar contexts can produce related dense vectors that encode semantic or usage structure. C: **Incorrect:** Spelling correction requires additional modeling and is not automatic. D: **Incorrect:** Embeddings are generally dense, whereas one-hot vectors are extremely sparse.

**1. Identify the recurrent dependency:** An RNN computes hₜ from hₜ₋₁, so hidden states within a sequence are inherently sequential during the forward pass. **2. Contrast Transformer self-attention:** All positions’ query, key and value projections can be formed together. Pairwise attention scores and positionwise feedforward operations are matrix computations that can be parallelized across positions during encoder training. **3. State the relevant advantage:** Better parallelization is the primary listed advantage. Full self-attention still stores a quadratic number of sequence-pair scores, so lower memory is not a general guarantee. **Answer: B — Parallelization during training.** A: **Incorrect:** Full attention can use substantial O(T²) score memory, so lower memory is not universal. B: **Correct:** It removes the RNN’s state-by-state dependency within encoder processing. C: **Incorrect:** Parameter count depends on architectural sizes, not simply on using attention instead of recurrence. D: **Incorrect:** Transformers contain several interacting components and are not necessarily simpler.

**1. List the OR targets:** For inputs (0,0),(0,1),(1,0),(1,1), OR outputs are (0,1,1,1). With no inhibitory input, the MP neuron predicts 1 when x₁+x₂≥θ. **2. Count correctness at each threshold:** At θ=0, predictions are (1,1,1,1), giving NCC=3. At θ=1, predictions exactly match OR, giving NCC=4. At θ=2, predictions are (0,0,0,1), so only (0,0) and (1,1) are correct, giving NCC=2. **3. Check θ=3:** Every input sum is below 3, so all predictions are 0 and only (0,0) is correct, giving NCC=1. Thus the listed correct pairs are θ=1,NCC=4 and θ=2,NCC=2. **Answer: B, E — θ=1 with NCC=4, and θ=2 with NCC=2.** A: **Incorrect:** At θ=0 the (0,0) case is misclassified, so NCC is 3 rather than 4. B: **Correct:** Threshold 1 gives exactly the OR truth table. C: **Incorrect:** At θ=1 all four cases are correct, not three. D: **Incorrect:** Threshold 2 correctly handles two cases, not one. E: **Correct:** At θ=2 exactly (0,0) and (1,1) are correct. F: **Incorrect:** Threshold 3 still correctly predicts (0,0), so NCC is 1 rather than 0.

**1. Define momentum precisely:** Momentum maintains a signed history of gradients or a velocity vector, so update directions can persist across steps. A moving average of squared gradients used only for scaling is a different mechanism. **2. Identify the two methods:** Nesterov Accelerated Gradient uses a momentum velocity and evaluates a look-ahead gradient. Adam combines the signed first-moment average \(m_t=\beta_1m_{t-1}+(1-\beta_1)g_t\) with a second-moment scaling estimate. **3. Separate the standard alternatives:** AdaGrad, basic RMSProp and AdaDelta adapt coordinate scales using squared-gradient or update statistics; their basic definitions do not require a signed momentum accumulator. **Answer: A, E — NAG; ADAM.** A: **Correct:** Nesterov acceleration explicitly uses a momentum velocity. B: **Incorrect:** Its basic form accumulates squared gradients for scaling, not signed gradient momentum. C: **Incorrect:** Basic RMSProp averages squared gradients; a separate optional momentum extension is not assumed. D: **Incorrect:** AdaDelta uses running squared-gradient and squared-update scales without standard signed momentum. E: **Correct:** Adam explicitly includes a first-moment gradient average, giving a momentum-like direction.

**1. Diagnose the source of the vanishing signal:** A deep chain multiplies many local derivatives and weight factors. Tanh saturation or inactive ReLU units can make the product too small to train earlier layers. **2. Keep useful local derivatives:** Leaky ReLU supplies a nonzero slope for negative pre-activations and can revive paths blocked by ordinary ReLU. Appropriate initialization, such as variance-aware Xavier or He schemes for the chosen activations, helps control signal magnitudes through depth. **3. Separate data quantity from gradient flow:** More examples may improve statistical estimation but do not directly change the problematic derivative chain in the architecture. **Answer: B, C — Replace ReLU activations with leaky ReLUs; Use appropriate weight initialization.** A: **Incorrect:** Increasing dataset size alone does not repair saturation or repeated derivative shrinkage. B: **Correct:** A negative-side slope can help paths blocked by dead ReLU units. C: **Correct:** Variance-aware initialization can reduce early signal shrinkage or explosion.

**1. Count scores for one head and sequence:** Each of T=12 query positions is compared with all 12 key positions. The attention score matrix therefore has T²=144 entries. **2. Include heads and batch:** With h=2 heads and B=4 sequences, \[N_{scores}=BhT^2=4\cdot2\cdot12^2=1152.\] This counts raw query–key scores, not the dimensions of the projected vectors. **Answer: 1152.**

**1. Identify dimensions:** The input has d=3 components, the LSTM hidden state has h=4, and the output has o=3. The input, forget and output gates and the candidate cell each have an input matrix, recurrent matrix and bias. **2. Count the four LSTM affine maps:** Each map has \(hd+h^2+h=4(3)+4^2+4=12+16+4=32\) parameters. Four maps give 128. **3. Add the output projection:** The output yₜ=Vhₜ+c has \(oh+o=3(4)+3=15\) parameters. Hence \[P=4h(d+h+1)+o(h+1)=128+15=143.\] **4. Explain sequence-length independence:** The same matrices and biases are reused at all five time steps, so T=5 increases computation but does not multiply the trainable parameter count. **Answer: 143.**

**1. Check the forward path:** Stride-1 valid CNN cross-correlation of the displayed X and K gives \(A=\begin{pmatrix}6&0\\3&1\end{pmatrix}\). Average pooling gives B=(6+0+3+1)/4=2.5, so the following ReLU is active and has derivative 1. **2. Differentiate with respect to the center weight:** In the four windows, the input entries multiplying the center kernel weight are 2,1,0,−1. Because the kernel is shared, all four contributions must be added. Thus \[\frac{\partial B}{\partial K_{11}}=\frac{2+1+0-1}{4}=0.5.\] **3. Chain the supplied upstream gradient:** \[\frac{\partial L}{\partial K_{11}}=\frac{\partial L}{\partial\hat y}\frac{\partial\hat y}{\partial B}\frac{\partial B}{\partial K_{11}}=(-3)(1)(0.5)=-1.50.\] The pooling division by four is essential. **Answer: -1.50.** **Convention note:** K₁₁ denotes the center entry as explicitly stated in the question, using zero-based kernel indices. CNN convolution uses the unflipped cross-correlation convention.

**1. Count the sentence positions:** There are seven distinct tokens. With window radius d=3, every center produces a directed center–context pair for each other position at distance at most three. **2. Count pairs by separation:** Separation 1 contributes 7−1=6 unordered position pairs, separation 2 contributes 5, and separation 3 contributes 4. Each pair appears in two directions as the center is exchanged. **3. Add:** \[N=2[(7-1)+(7-2)+(7-3)]=2(6+5+4)=30.\] Equivalently, the per-center counts are 3,4,5,6,5,4,3. Since the tokens are distinct, no repeated word pair reduces this count. **Answer: 30.**

**1. Use the center-word index:** networks has vocabulary index 5. Its one-hot vector has a 1 in that zero-based position and zero elsewhere. **2. Perform the embedding lookup:** With W_in shaped 8×2, \(h=W_{in}^Tx\) selects the row for index 5, the sixth row, which is (1,−2). **3. Preserve vector orientation:** Therefore \(h=(1,-2)^T\), a two-dimensional column vector. No average over context words occurs in a Skip-Gram center lookup. **Answer: B — h=(1,−2)ᵀ.** A: **Incorrect:** It is the row for index 4, models, rather than networks. B: **Correct:** It is exactly row index 5 of W_in. C: **Incorrect:** Its first component does not match the selected row. D: **Incorrect:** Its second component does not match the selected row.

**1. Retrieve the center embedding:** For networks, the input lookup gives h=(1,−2)ᵀ. **2. Retrieve the target output column:** learn has vocabulary index 3. Its W_out column is u_learn=(2,3)ᵀ. **3. Compute the unnormalized score:** \[z_{learn}=u_{learn}^Th=2(1)+3(-2)=2-6=-4.00.\] This is a logit; it can be negative and is not yet a probability. **Answer: -4.00.**

**1. Compute vocabulary scores:** Multiplying the eight output columns by h=(1,−2)ᵀ gives \[z=(-3,2,-4,-4,4,3,2.5,-5)^T.\] For example, models uses output column (0,−2)ᵀ and has score 0(1)+(−2)(−2)=4. **2. Use softmax ordering:** \(p_j=e^{z_j}/\sum_ke^{z_k}\). A common positive denominator and increasing exponential mean the highest logit also has the highest probability. **3. Return the index:** The largest score is 4 at zero-based index 4, corresponding to models. The question asks for the integer key, so enter 4 rather than the word itself. **Answer: 4.**

**1. Count the input lookup weights reached by the example:** W_in has 8×2 entries, but a single one-hot center selects only its networks row. Only those two input embedding weights receive data gradients in this iteration. **2. Count full-softmax output weights:** W_out has 2×8=16 entries. Cross-entropy uses all vocabulary probabilities, with logit derivative \(\partial L/\partial z_j=p_j-t_j\). All eight output columns participate, and both hidden coordinates are nonzero here. **3. Add the affected weights:** \[N_{updated}=2+16=18.\] Parameter sharing means the same weight is counted once even when used in several center–context terms. **Answer: 18.** **Convention note:** This uses the stated full softmax and ordinary data-gradient updates, without negative sampling, frozen parameters or additional weight decay that would update otherwise inactive input rows.