A transformer layer with absolute positional encoding and another with ALiBi are compared on
sequences extended from length 128 to 512, without retraining.
Which numerical behavior is most likely for ALiBi in this extrapolation scenario?
6406536924746.
Attention scores remain numerically well-shaped because ALiBi biases scale
linearly with distance, not with a fixed maximum length.
6406536924747.
The model cannot compute positional information beyond 128 because ALiBi
stores one embedding per absolute position up to 128 only.
6406536924748.
ALiBi attention becomes undefined for positions beyond 128 because its
biases require precomputed lookup tables that stop at 128.
6406536924749.
Attention scores collapse to near zero for distant positions because ALiBi uses
fixed sinusoidal patterns tied to the original length 128.
A published solution is not available for this question yet.
Select the statements that correctly distinguish fine-tuning from zero-shot prompting for GPT-style
models.
6406536924703.
In Fine-Tuning, the weights of the pre-trained model (and often a new specific
head) are updated using backpropagation on a labeled dataset.
6406536924704.
In Zero-Shot Prompting, the model weights are **frozen** (not updated); the task
is formulated purely through the design of the input text (prompt).
6406536924705.
Zero-Shot Prompting generally achieves higher accuracy than Fine-Tuning
when abundant labeled data is available.
6406536924706.
Fine-Tuning typically requires a task-specific output layer (like a linear
classifier), whereas Zero-Shot Prompting relies on the model generating text directly.
A published solution is not available for this question yet.
.The model outputs the following sorted
probability distribution for the next token:
• "apple": 0.40
• "banana": 0.25
• "cherry": 0.15
• "date": 0.10
• "elderberry": 0.10
How many tokens will be included in the candidate pool from which the final token is sampled?
Enter the integer count.
A published solution is not available for this question yet.
sequences. The sequence length (context window) is
tokens.
Calculate the total number of **tokens** the model sees during pre-training.Enter your answer in
**Billions (B)**. (e.g., if the answer is 500 Billion, enter 500).
A published solution is not available for this question yet.
Suppose a language model outputs the following logits for a vocabulary of size 4:
You apply High Temperature Sampling with temperature
. After applying the softmax on
the temperature-scaled logits, what is the probability assigned to the token "apple"? (Give your
answer up to 2 decimal places).
First, apply temperature scaling with
:
A published solution is not available for this question yet.
Why do standard LLM architectures include explicit Positional Encodings (like RoPE or Sinusoidal)
despite the causal mask providing some implicit order?
6406536924716.
To reduce the memory footprint of the
and
matrices.
6406536924717.
To ensure that the model can only attend to future tokens.
6406536924718.
To provide a strong inductive bias that allows the model to learn structural
and distance-based relationships more efficiently than relying on masking alone.
6406536924719.
To fix a mathematical error in the original Dot-Product Attention formula.
A published solution is not available for this question yet.
In a transformer with relative positional encoding (RPE), attention logits between a query at
position 5 and keys at positions 3 and 7 depend on relative distances
where
is the
position of query and
is the position of key.
Suppose the learned scalar biases as per RPE are:
•
•
•
•
•
The base (content-only) attention logits are:
•
•
What are the final attention logits after incorporating RPE?
Which of the following statements about fine-tuning strategies such as gradual unfreezing and
multi-task learning in the context of the T5 model are correct?
6406536924712.
In gradual unfreezing, layers are progressively unfrozen starting from the
task-specific (top) layers toward the input (bottom) layers.
6406536924713.
Multi-task learning in T5 typically improves performance uniformly across all
tasks.
6406536924714.
Multi-task pre-training (as used in T5) followed by task-specific fine-tuning
often performs better than relying on multi-task learning alone.
6406536924715.
Gradual unfreezing can help mitigate catastrophic forgetting of pre-trained
representations during T5 fine-tuning.
A published solution is not available for this question yet.
