Calculate the KV Cache memory size in Megabytes for a single request (batch size = 1) with the
following parameters:
• Number of layers: 24
• Embedding dimension (d_model): 1024
• Sequence length (n): 2048 tokens
• Precision: 16-bit (2 bytes per element)
• Assume standard Multi-Head Attention (MHA) where total KV dimension per layer is 2 x d_model
(2 for K+V)
A published solution is not available for this question yet.
The standard Self-Attention mechanism has a time complexity of
, where
is the
sequence length and
is the embedding dimension. In Kernel-based Linear Attention methods
(like Linear Transformers), the associativity of matrix multiplication is exploited to compute
instead of
.
What is the resulting time complexity of this Linear Attention mechanism?
In a standard Transformer layer utilizing Rotary Positional Embeddings (RoPE), at which specific
stage of the forward pass is the rotary transformation applied?
6406536959459.
To the input embeddings
immediately before the Linear projections for
,
, and
.
6406536959460.
To the
and
vectors simultaneously using a coupled weight matrix within
the Attention head.
6406536959461.
To the
and
vectors independently, after their linear projections but
before the scaled dot-product calculation.
6406536959462.
To the
and
vectors after the linear projections to ensure the entire
latent space is rotationally invariant.
A published solution is not available for this question yet.
In a Transformer using ALiBi, the attention score is modified as:
Suppose a researcher proposes modifying ALiBi to:
and claims that:
“The model will still learn to focus on nearby tokens through training.”
Which of the following is the most accurate critique of this proposal?
6406536959467.
The modification will not affect attention behavior significantly because
softmax normalizes all scores, making additive biases irrelevant.
6406536959468.
The modification will cause instability because the attention scores will
become negative for nearby tokens and positive for distant tokens.
6406536959469.
The modification is equivalent to scaling (Q) and (K) vectors differently and
therefore does not fundamentally change the positional bias.
6406536959470.
The modification will bias attention toward distant tokens, and due to the
exponential nature of softmax, this effect can dominate learned
similarities, making it
difficult for training to recover locality.
A published solution is not available for this question yet.
In a feature-based approach using a pretrained transformer (like BERT) for a Multiple-Choice QA
task, which of the following statements regarding the methodology are correct? (Select all that
apply)
6406536959419.
The weights of the pretrained transformer are updated via backpropagation
during the training phase.
6406536959420.
The input must be formatted as multiple pairs, such as [CLS] Question [SEP]
Choice_n, creating a distinct representation for each candidate answer.
6406536959421.
The final prediction is determined by passing the fixed output representations
through a trainable linear layer and applying a Softmax function.
6406536959422.
The model generates the correct answer string token-by-token using an
autoregressive decoding strategy like Greedy Search.
A published solution is not available for this question yet.
A Transformer processes a sequence of length 5 using 6 identical self-attention layers.
Count the total number of allowed attention links across all layers and Choose the option
representing the correct pair
(A) BERT
(B) GPT
6406536959423.
BERT: 150, GPT: 90
6406536959424.
BERT: 90, GPT: 150
6406536959425.
BERT: 150, GPT: 150
6406536959426.
BERT: 90, GPT: 90
6406536959427.
BERT: 30, GPT: 36
6406536959428.
BERT: 36, GPT: 30
A published solution is not available for this question yet.
Consider the architecture of **BART**. Which of the following descriptions best match its structural
design?
6406536959429.
A Decoder-only model similar to GPT, but with bidirectional attention in the
first layer.
6406536959430.
An Encoder-only model similar to BERT, but trained with a causal masking
objective.
6406536959431.
An encoder-decoder model where the encoder processes a corrupted input
bidirectionally and the decoder autoregressively reconstructs the original sequence.
6406536959432.
A dual-encoder model where one encoder processes the context and another
processes the query.
A published solution is not available for this question yet.
In a transformer with relative positional encoding (RPE), attention logits between a query at
position 4 and keys at positions 2 and 6 depend on relative distances
where
is the
position of query and
is the position of key.
Suppose the learned scalar biases as per RPE are:
•
•
•
•
•
The base (content-only) attention logits are:
•
•
What are the final attention logits after incorporating RPE?
An engineer is designing a Transformer model capable of length extrapolation (handling
sequences longer than those seen during training) while minimizing the number of trainable
weights. They are evaluating Absolute Positional Encodings (APE), Relative Positional Encodings
(RPE), Rotary Positional Encodings (RoPE), and Attention with Linear Biases (ALiBi).Which of the
following statements correctly describes the parameterization and behavior of these techniques?
6406536959471.
RoPE is considered a learned encoding because the model must optimize the
rotation angles
as trainable parameters during the backpropagation phase.
6406536959472.
The engineer should choose APE or RPE to minimize weights, as both rely
exclusively on fixed sinusoidal functions without requiring a lookup table.
6406536959473.
RoPE and ALiBi are preferable for this use case because they utilize fixed
mathematical transformations, whereas standard APE and RPE typically require learning position-
specific embedding weights.
6406536959474.
ALiBi is the only technique among the four that requires the model to learn a
unique "slope" parameter for each attention head to determine the rate of penalty decay.
A published solution is not available for this question yet.
