Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.
15 / 16 published solutions
Question 2 MSQ · 4.0 marks
Which statements are true about traditional attention used in sequence-to-sequence models?
6406537043567.
The decoder decides which encoder states are important.
6406537043568.
Attention weights are computed over all decoder hidden states.
6406537043569.
The context vector is a weighted sum of encoder states.
6406537043570.
Encoder hidden states are ignored after encoding.
Published solution
**1. Identify the attention query:** In traditional encoder-decoder attention, the decoder state at a generation step acts as the query. It is compared with encoder hidden states representing source positions.
**2. Normalize relevance scores:** Softmax converts source-position scores into weights \(\alpha_{tj}\) that sum to one.
**3. Build the context:** \[c_t=\sum_j\alpha_{tj}h_j^{enc}.\] The decoder uses this weighted source summary to predict the next output token. Thus encoder states remain available after encoding.
**Answer: A, C — Selected: The decoder decides which encoder states are important.; The context vector is a weighted sum of encoder states.**
A: **Correct:** The decoder’s current query determines relevance of the source states.
B: **Incorrect:** Cross-attention weights here are over encoder positions, not all decoder states.
C: **Correct:** The context is a weighted sum of encoder representations.
D: **Incorrect:** The decoder repeatedly consults the encoder representations.
Consider the text "learn easy math", consisting of three tokens: {learn,easy,math}.
The token embeddings are arranged in the input matrix
:
The query, key, and value projection matrices are given by:
Based on the above data, answer the given subquestions.
Compute the
of the final scaled dot-product attention and submit the sum of the diagonal
elements of
matrix.
Published solution
**1. Respect the source orientation:** Tokens are columns of \(X=\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}\). Both query and key projections are the 2-by-2 identity, so Q=K=X.
**2. Compute the raw dot products:**
\[Q^{\mathsf T}K=\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}.\]
Each diagonal entry is the squared norm of a token vector.
**3. Sum the diagonal:** \(1+1+2=4\). The question asks for the trace of the unscaled \(Q^{\mathsf T}K\), not of its scaled or softmax-transformed version. The printed attention formula has a nonstandard orientation; this requested intermediate matrix is nevertheless unambiguous.
**Answer: 4.**
Consider the text "learn easy math", consisting of three tokens: {learn,easy,math}.
The token embeddings are arranged in the input matrix
:
The query, key, and value projection matrices are given by:
Based on the above data, answer the given subquestions.
Choose the token pair with the least attention score.
6406537043572.
{"learn", "easy"}
6406537043573.
{"learn", "math"}
6406537043574.
{"easy", "math"}
6406537043575.
{"learn", "learn"}
6406537043576.
{"easy" , "easy"}
Published solution
**1. Compute token dot products:** The query/key vectors are learn=(1,0), easy=(0,1), math=(1,1), because both projections are identity.
**2. Compare the listed scores:** learn–easy is 0; learn–math is 1; easy–math is 1; learn–learn is 1; easy–easy is 1.
**3. Apply scaling:** All raw scores are divided by the same positive \(\sqrt2\), so their ordering is unchanged. Row softmax also preserves the within-row order; the learn–easy pair has the least relevant raw/scaled score among the listed pairs.
**Answer: A — Selected: {"learn", "easy"}.**
A: **Correct:** Its dot product is zero, the smallest listed score.
B: **Incorrect:** The dot product is one, larger than zero.
C: **Incorrect:** The dot product is one.
D: **Incorrect:** The squared norm of learn is one.
E: **Incorrect:** The squared norm of easy is one.
A Transformer model processes a sequence containing 6 tokens using Multi-Head Attention. The
model initially uses 4 attention heads. If the number of attention heads is doubled to 8, while the
sequence length remains unchanged, how many **additional** attention scores are computed across
all heads?
Published solution
**1. Count scores per head:** Each of six query positions compares with six key positions, producing \(6^2=36\) scores per head.
**2. Count extra heads:** The count rises from four to eight, adding four heads.
**3. Multiply:** \[\Delta N=(8-4)(6^2)=4(36)=144.\] The total rises from 144 to 288; the requested additional count is 144.
**Answer: 144.**
A batch of 8 sentences is passed through a Transformer Encoder consisting of 6 encoder layers
and 8 attention heads. After padding, each sentence has a maximum length of 32 tokens. The
model uses an embedding dimension of
. What is the total number of elements
(volume) in the output tensor produced by the final encoder layer for the entire batch?
Published solution
**1. Identify final output shape:** A standard encoder layer preserves batch size, sequence length and model width. The final output is \((B,T,d)\).
**2. Substitute:** The PDF gives \(B=8\), \(T=32\), \(d=128\). Hence \[N=8(32)(128)=32{,}768.\]
**3. Avoid double counting:** Six layers are applied successively; their outputs are not stacked into the final tensor. The eight attention heads recombine into width 128, rather than adding an extra factor of eight.
**Answer: 32768.**
Consider a masked multi-head attention layer in a transformer decoder. The source vocabulary is
of size
,
and context length
. To enforce autoregressive
generation, a causal mask is applied to the raw attention score before applying softmax. How
many entries in the causal mask are set to
?
Published solution
**1. Identify forbidden positions:** A causal mask blocks only future tokens; the diagonal and earlier positions remain visible.
**2. Sum the strict upper triangle:** For context length T, the number of blocked entries is \((T-1)+(T-2)+\cdots+1=T(T-1)/2\).
**3. Substitute:** \[N=128(127)/2=8128.\] Vocabulary size and hidden dimension do not affect one T-by-T causal mask. If a mask is broadcast across heads, these are the unique entries of the base mask.
**Answer: 8128.**
Consider a Transformer model with a hidden state dimension
. The positional
encoding for a given position,
and dimension
is defined by:
Compute the squared euclidean norm for the vector corresponding to
and enter the
value.
Published solution
**1. Pair positional components:** For each frequency, sinusoidal positional encoding has a sine and cosine of the same angle.
**2. Square and add a pair:** The identity \(\sin^2\theta+\cos^2\theta=1\) applies regardless of position, including position 3.
**3. Count pairs:** A 512-dimensional encoding has \(512/2=256\) pairs, so \[\|PE(3)\|_2^2=\sum_{i=0}^{255}1=256.\] The squared norm, not its square root, is requested.
**Answer: 256.**
Consider the following statements regarding the use of Teacher Forcing while training
autoregressive sequence-to-sequence models and select the appropriate option:
Statement 1: Teacher Forcing involves passing the input token to a larger Teacher model whose
output is then input as the next token
Statement 2: Teacher Forcing helps to prevent compounding of early incorrect predictions that
can destabilize training
6406537043581.
Statement 1 is True but Statement 2 is False
6406537043582.
Statement 1 is False but Statement 2 is True
6406537043583.
Both the statements are True
6406537043584.
Both the statements are False
Published solution
**1. Define teacher forcing:** During training, the decoder receives the actual previous target token as input, rather than necessarily feeding back its own last prediction.
**2. Evaluate Statement 1:** No larger teacher model is required. That description confuses teacher forcing with a teacher-student/distillation arrangement, so Statement 1 is false.
**3. Evaluate Statement 2:** Supplying correct previous tokens prevents errors in the model’s own generated prefixes from compounding through the training sequence. Statement 2 is true. At inference those true previous tokens are unavailable, which creates a train–inference exposure difference.
**Answer: B — Selected: Statement 1 is False but Statement 2 is True.**
A: **Incorrect:** Statement 1 is false and Statement 2 is true, reversing this option.
B: **Correct:** This matches the definition and the training-stability benefit.
C: **Incorrect:** The larger-teacher-model description is false.
D: **Incorrect:** The compounding-error benefit in Statement 2 is true.
Based on the above data, answer the given subquestions.
Compute the joint probability of the sentence :
(Provide answer correct upto 2 digits after the decimal)
Published solution
**1. Use the autoregressive chain rule:** A three-token sentence has probability \(P(w_1)P(w_2\mid w_1)P(w_3\mid w_1,w_2)\).
**2. Follow the “you eat apples” path:** The tree gives \(P(you)=0.4\), \(P(eat\mid you)=0.5\), \(P(apples\mid you,eat)=0.9\).
**3. Multiply:** \[P(you,eat,apples)=0.4(0.5)(0.9)=0.18.\] These are conditional probabilities along one path, so they are multiplied rather than added.
**Answer: 0.18.**
Based on the above data, answer the given subquestions.
Using the given probability tree calculate the conditional probability of predicting "apples" given
"like" [i.e P(apples | like)]. (Submit -1 if the provided information is insufficient)
Published solution
**1. Identify the conditioning difference:** The tree gives next-word probabilities after full histories, such as \(P(apples\mid i,like)=0.7\) and \(P(apples\mid you,like)=0.4\).
**2. Explain why “given like” is different:** To remove the earlier history, one must average over all histories ending in like, with their conditional weights. The model was trained on a larger vocabulary, and the tree is explicitly partial.
**3. Conclude:** Other histories leading to like, and their probabilities, are not supplied. The two displayed branches cannot be assumed to exhaust the conditioning event. Therefore the requested general probability is not identifiable, and the instructed response is -1.
**Answer: -1.**
Which of the following are components found within a GPT decoder layer?
6406537043592.
Causal (Masked) Multi-Head Self-Attention
6406537043593.
Position-wise Feed-Forward Network
6406537043594.
Multi Head (Cross) Attention
6406537043595.
Recurrent Neural Network (RNN) Layer
6406537043596.
Add & Norm Layer
Published solution
**1. Follow a standard GPT block:** Causal self-attention mixes current/past token information. A position-wise feed-forward network transforms each token representation.
**2. Preserve stable transformations:** Residual additions and layer normalization are the “Add & Norm” components, although exact pre-norm/post-norm ordering varies between architectures.
**3. Distinguish decoder-only design:** Standard GPT does not have a separate encoder to cross-attend to and does not contain an RNN layer. Its attention is causal self-attention.
**Answer: A, B, E — Selected: Causal (Masked) Multi-Head Self-Attention; Position-wise Feed-Forward Network; Add & Norm Layer.**
A: **Correct:** Causal multi-head self-attention is a core GPT block component.
B: **Correct:** A token-wise feed-forward network follows attention transformations.
C: **Incorrect:** Standard decoder-only GPT has no encoder cross-attention sublayer.
D: **Incorrect:** GPT uses Transformer blocks rather than recurrent layers.
E: **Correct:** Residual addition and normalization are standard block components.
The table below represents the conditional probability distribution over the vocabulary. Each
column corresponds to a decoding timestep and the values represent the conditional probability
of selecting a token at the timestep given the previously generated tokens
For all calculations, consider only the tokens mentioned in the first column as the Vocabulary
Based on the above data, answer the given subquestions.
Use greedy decoding to determine the most likely sequence of 6 tokens and compute the
probability of generating that sequence. Enter the value rounded off to 3 decimal places
Published solution
**1. Apply greedy selection at every column:** The largest listed probabilities are historic 0.38 at t=1, castle 0.40 at t=2, was 0.50 at t=3, majestic 0.35 at t=4, but 0.45 at t=5 and abandoned 0.40 at t=6.
**2. Multiply along the selected sequence:** \[P=0.38(0.40)(0.50)(0.35)(0.45)(0.40)=0.004788.\]
**3. Round:** To three decimal places this is 0.005. This uses the question’s supplied per-time distributions as the values along the selected history; a real autoregressive model can return different distributions for different prefixes.
**Answer: 0.005.**
The table below represents the conditional probability distribution over the vocabulary. Each
column corresponds to a decoding timestep and the values represent the conditional probability
of selecting a token at the timestep given the previously generated tokens
For all calculations, consider only the tokens mentioned in the first column as the Vocabulary
Based on the above data, answer the given subquestions.
Use exhaustive search strategy to identify the most probable sequence consisting of 4 tokens.
How many decoder runs are required in total to identify the most probable sequence containing 4
tokens?
Published solution
**1. Count the vocabulary:** The table contains seven tokens. Exhaustive search over four-token strings has \(7^4=2401\) possible completed sequences.
**2. Reuse shared prefixes:** One decoder evaluation of a prefix supplies the entire next-token distribution. To score every sequence efficiently, evaluate the empty prefix once, all 7 one-token prefixes, all 49 two-token prefixes and all 343 three-token prefixes.
**3. Add decoder runs:** \[1+7+7^2+7^3=1+7+49+343=400.\]
**Convention note:** This counts one next-token evaluation per distinct prefix with reuse/caching, as intended by the key. It is not the number of complete candidate strings or the number of batched hardware calls. Treating the fixed column table as globally prefix-independent would remove the need for this search; the question instead assumes ordinary autoregressive branching.
**Answer: 400.**
The table below represents the conditional probability distribution over the vocabulary. Each
column corresponds to a decoding timestep and the values represent the conditional probability
of selecting a token at the timestep given the previously generated tokens
For all calculations, consider only the tokens mentioned in the first column as the Vocabulary
Based on the above data, answer the given subquestions.
Use Top-k sampling at timestep t = 4 and k = 3. Compute the renormalized probability of the most
probable token from the candidate tokens and enter the value rounded off to three decimal places
Published solution
**1. Select the three largest values at t=4:** majestic has 0.35, abandoned 0.30 and historic 0.20. Their combined mass is \(0.35+0.30+0.20=0.85\).
**2. Renormalize within this restricted set:** The most probable candidate remains majestic, with new probability \[p'=0.35/0.85=0.4117647.\]
**3. Round:** To three decimal places the answer is 0.412. The excluded tokens receive zero sampling probability under top-k truncation.
**Answer: 0.412.**
The table below represents the conditional probability distribution over the vocabulary. Each
column corresponds to a decoding timestep and the values represent the conditional probability
of selecting a token at the timestep given the previously generated tokens
For all calculations, consider only the tokens mentioned in the first column as the Vocabulary
Based on the above data, answer the given subquestions.
Consider the sequence "the castle was abandoned". What is the minimum beam width required to
guarantee that this sequence is retained during beam search at timestep t = 4 ?
Published solution
**1. Compute the target prefix probability:** For “the castle was abandoned”, the table gives \(0.30(0.40)(0.50)(0.30)=0.018\).
**2. Rank competing four-token prefixes:** Four have larger products under the supplied distributions: historic castle was majestic (0.0266), historic castle was abandoned (0.0228), the castle was majestic (0.0210) and historic historic was majestic (0.01995).
**3. Check earlier pruning:** The target’s prefixes “the”, “the castle” and “the castle was” each rank second at their respective steps, so a width-five beam retains them. At step four the target ranks fifth; a width-four beam drops it because all four higher candidates arise from retained high-ranked earlier prefixes.
**4. Conclude:** The minimum width is 5 for this table and ordinary cumulative-probability beam search without an extra length penalty. This is a guarantee within the supplied distribution model, not across unknown prefix-dependent probabilities.
**Answer: 5.**
