Large Language Models · Quiz 2 · Sep 2025

Suppose for a 2-component GMM, at some iteration you have

What is the responsibility

?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Which token will be generated in the first merge?

A study is comparing three architectures, all trained with the same compute budget: • **Model A (Encoder-Decoder):** A 12-layer encoder and a 12-layer decoder with separate parameters. Its total size is 220 million parameters. • **Model B (Decoder-Only):** A 24-layer decoder-only model. Its total size is also approximately 220 million parameters. • **Model C (Shared Enc-Dec):** A 12-layer encoder and a 12-layer decoder where the parameters are shared (

). Assuming that the 12-layer encoder and 12-layer decoder in Model A have an equal number of parameters, what is the approximate total parameter count (in millions) of Model C?

Consider a text corpus being processed using the Byte Pair Encoding (BPE) algorithm. The initial vocabulary contains

tokens (representing characters), and the corpus contains a total of

tokens. During BPE training, the following merge operations are performed. Each merge combines all occurrences of a particular token pair into a single new token, reducing the total token count accordingly: 1. **Merge 1:** Pair ('t', 'h') appears 24 times → merged into new token 'th' 2. **Merge 2:** Pair ('i', 'n') appears 18 times → merged into new token 'in' 3. **Merge 3:** Pair ('th', 'e') appears 10 times → merged into new token 'the' 4. **Merge 4:** Pair ('i', 's') appears 8 times → merged into new token 'is' What is the final total number of tokens in the corpus after all 4 merges are completed?

If Byte-Pair Encoding (BPE) starts with

unique characters and performs

merge operations, what will be the final vocabulary size?

In a 4-token sequence, a Transformer is calculating the attention for the third Token. Its Query vector (

) must be compared against the Key vectors (

) of all tokens. The Causal LM mask is applied to the raw scores before the softmax function. The raw dot-product scores are: •

•

•

•

After the Causal LM mask and the subsequent softmax function are applied, what will be the final attention probability assigned to Token 4?

Which of the following is/are **primarily** multilingual datasets? Select all that apply.

Which decoding method ensures the same response every time for a given prompt (assuming no randomness in model weights)? Select all that apply.

A data preprocessing pipeline removed data from the Common Crawl snapshot in three stages: 50% after language identification, 24% of the remaining after quality filtering, and 12% of what remained after deduplication. What percentage of the original Common Crawl snapshot remains in the final dataset? (Choose the closest number)

A data preprocessing pipeline needs to apply the following steps on raw Common Crawl data: 1. Fuzzy deduplication (using MinHash) 2. Toxicity detection (using ML-based classifiers) 3. PII removal (using regex patterns) 4. Exact deduplication (using hash-based matching) 5. Quality filtering (using several rule-based heuristics) 6. URL Filtering What is the most efficient sequence to apply these steps?

Consider the scaling law:

where

= model parameters,

= training tokens,

= scale constants,

= scaling exponents. A team trains an LLM and observes the following losses for their model: 1. Training Loss 2. Test Loss (same distribution as training) 3. Test Loss (different distribution from training) Which loss does

represent?

A company is building a sentiment analysis system for customer reviews. They have two

Which of the following statements best describes the fundamental difference between the unsupervised pre-training objective used in T5 (span corruption) and its supervised fine-tuning objectives such as translation or summarization?

A Transformer model is designed to handle both understanding and generation tasks. When given the input sequence

, its attention mask allows the following: • When processing

or

, the model can see both

and

. • When processing

, the model can see

and

but not

. This attention scheme is the defining characteristic of which training objective?

BART is an encoder-decoder model. During its denoising pre-training, what is fed as input to the decoder?

How does SentencePiece handle whitespace characters (like spaces) during tokenization?

For a layer with

features, how many learnable parameters do BatchNorm and LayerNorm have?

In a standard Encoder-Decoder Transformer, where do the Query

, Key

, and Value

inputs for the Decoder's Cross-Attention sub-layer come from?