Large Language Models · End Term · Sep 2025 · AN

A language model produces the following logits for the next token:

Before sampling, the model applies: • Temperature scaling with

• Top-K sampling with

Which tokens are eligible to be sampled?

A neural network layer produces activations of hidden dimension

. You consider two normalization strategies: 1. Batch Normalization 2. Layer Normalization Which option correctly states the number of trainable parameters added by each normalization method?

A sub-word vocabulary is being constructed using the WordPiece algorithm. The following tokens and their frequencies are extracted: • replay (frequency = 100) • replus (frequency = 3) The current vocabulary includes the tokens:

Two candidate merges are under consideration: • Merge A:

• Merge B:

Which of the above merges are most possible based on the WordPiece tokenizer?

For a 4-category classification task, how would you use BERT's output?

A hospital wants to extract disease mentions and treatment plans from unstructured clinical notes. Notes use heavy abbreviations like ``pt dx w/ CAD, rx aspirin'' For extracting specific disease names from text, which task should you fine-tune BERT for?

Consider a sequence of length

and a local block attention mechanism with

blocks. If the attention matrix is restricted to only the main diagonal blocks (non-overlapping), what is the computational complexity of the self-attention operation for one layer (ignoring linear projections)? Assume

be the model dimension.

Match the positional encoding method (Column A) with its property/behavior (Column B). **Column A (Methods)** 1. APE 2. RPE 3. RoPE 4. ALiBi 5. NoPE **Column B (Properties / Observations)** A. Works even on sequence lengths far beyond training length without retraining, because attention bias grows proportionally to token distance. B. Even if the model sees sequences only up to length 512 during training, it fails badly at length 1024, because positions above 512 have never had embeddings learned. C. Creates an implicit rotation such that the attention score between positions

and

depends only on the difference

, not the positions themselves. D. Uses a fixed-size bank of distance embeddings such that two attention pairs

and

with the same distance use the same embedding. E. The model becomes permutation-invariant, meaning it treats ''cat sat mat'' the same as ``mat cat sat''.

A 2D vector

is rotated by

using RoPE. What is the resulting vector?

If a model uses RoPE, increasing the sequence length from 100 to 1000 requires:

In a sequence of length

, how many different attention pairs produce a distance of

?

For a sequence of length

, how many Absolute Positional Encoding (APE) vectors are needed, and how many Relative Positional Encoding (RPE) vectors are required?

A model with positional encoding crashes when the input length exceeds the training max length. Which encoding is it most likely using?

If the unmodified attention score is

, then ALiBi changes it to:

Which of the following decoding method(s) can generate a different output every time you run the model with the same prompt?

Which of the following methods modifies/modify the attention scores AFTER computing

?

Consider a BART encoder with sequence length

and hidden size

(fixed). Now suppose the sequence length is doubled from

to

, keeping all other parameters unchanged. Which of the following statements are true?

You are implementing a custom Block Sparse Attention kernel. Given a sequence length

and block size

, you decide to compute the main diagonal blocks plus one random off-diagonal block for each block-row. How many total

block computations are performed?

Which of the following statements accurately describes the properties and computational complexities of the various sparse and block attention mechanisms?

In the standard Transformer with 6 Encoder Layers, how many distinct sets of transformation weight matrices (

) are learned in the entire Encoder stack, assuming 8 heads per layer?

Suppose you are working on prefix language modeling. The sequence length is 10 and the first two tokens represent the task-specific prefix. How many non-infinity elements are there in the mask for computing attention scores?

You are configuring a Large Language Model (LLM) for inference and need to estimate the memory footprint of the KV Cache. Calculate the KV cache memory required per token (in Bytes) given the following model architecture: • Number of Layers (

): 32 • Number of Heads (

): 16 • Head Dimension (

): 128 • Precision: 2 Bytes (FP16)

A team is developing a new large language model (LLM) for code generation, focusing on an autoregressive (AR) approach using the Causal Language Modeling (CLM) objective. They implement a standard Transformer decoder-only architecture.The team is training the model on a sequence of tokens

, where

is the total length. The CLM objective aims to maximize the following log-likelihood:

During training, they strictly enforce teacher forcing, meaning the ground-truth previous tokens are always provided as input to predict the current token. Based on the above setup, answer the given subquestions.

Which of the following describes the fundamental training mechanism of this model?

A team is developing a new large language model (LLM) for code generation, focusing on an autoregressive (AR) approach using the Causal Language Modeling (CLM) objective. They implement a standard Transformer decoder-only architecture.The team is training the model on a sequence of tokens

, where

is the total length. The CLM objective aims to maximize the following log-likelihood:

During training, they strictly enforce teacher forcing, meaning the ground-truth previous tokens are always provided as input to predict the current token. Based on the above setup, answer the given subquestions.

During the training phase on a sequence

of length

, how does the teacher forcing mechanism specifically relate to the calculation of the loss function

?

A team is developing a new large language model (LLM) for code generation, focusing on an autoregressive (AR) approach using the Causal Language Modeling (CLM) objective. They implement a standard Transformer decoder-only architecture.The team is training the model on a sequence of tokens

, where

is the total length. The CLM objective aims to maximize the following log-likelihood:

During training, they strictly enforce teacher forcing, meaning the ground-truth previous tokens are always provided as input to predict the current token. Based on the above setup, answer the given subquestions.

In the inference phase (when the model is used for generation), why is the teacher forcing technique not used, and what is the primary challenge this introduces?

A team is developing a new large language model (LLM) for code generation, focusing on an autoregressive (AR) approach using the Causal Language Modeling (CLM) objective. They implement a standard Transformer decoder-only architecture.The team is training the model on a sequence of tokens

, where

is the total length. The CLM objective aims to maximize the following log-likelihood:

During training, they strictly enforce teacher forcing, meaning the ground-truth previous tokens are always provided as input to predict the current token. Based on the above setup, answer the given subquestions.

Which component of the Transformer decoder is essential for enforcing the Causal Language Modeling constraint?