Deep Learning Practice · Quiz 1 · May 2025

For a low-resource language with rich morphology and many compound words, **subword tokenization** (such as Byte Pair Encoding (BPE), WordPiece, or SentencePiece/Unigram) provides the best trade-off: - It keeps the vocabulary size compact and fixed. - It avoids Out-Of-Vocabulary (OOV) unknowns by decomposing unseen or compound words into known subword morphemes/characters. - Unlike pure character-level tokenization, it preserves meaningful root/affix units without creating excessively long sequences. **Answer: C (Subword tokenization (e.g., BPE or SentencePiece) trained on the available corpus.)** A: **Incorrect:** Character-level tokenization produces very long token sequences and loses subword semantic structure. B: **Incorrect:** Fixed word-level tokenization suffers from high OOV rates on morphologically rich and compound-word languages. C: **Correct:** Subword tokenization balances vocabulary size, OOV handling, and morphological compositionality. D: **Incorrect:** Raw byte sequences without subword merging drastically increase sequence length and compute cost.

When fully fine-tuning a large Transformer model (>1B parameters): - **Model parameters** require 4 bytes/param (FP32) or 2 bytes/param (FP16) \(\implies\) 2–4+ GB. - **Gradients** require the same size as the model parameters \(\implies\) 2–4+ GB. - **Optimizer states** (e.g., Adam stores first and second moments `m` and `v` in FP32) require 8 bytes/param \(\implies\) 8+ GB. - **Input batch data (token IDs)** consists of integer indices of shape `(batch_size, seq_len)` (e.g., \(8 \times 512 \times 8\text{ bytes} \approx 32\text{ KB}\)), which is orders of magnitude smaller and contributes **least significantly** to GPU memory usage. **Answer: D (Storing the input batch data (token IDs).)** A: **Incorrect:** Storing 1B+ parameters consumes several gigabytes of GPU memory. B: **Incorrect:** Storing gradients for every trainable parameter consumes several gigabytes of GPU memory. C: **Incorrect:** Adam optimizer states consume 2 FP32 values per parameter (often the single largest memory consumer). D: **Correct:** Input token IDs have shape `(batch_size, seq_len)` and occupy only a few kilobytes or megabytes.

**Instruction Tuning (Supervised Fine-Tuning on instruction-response pairs)** directly trains a pre-trained language model on a collection of prompts/instructions and desired human-written or human-preferred responses so the model learns to follow instructions helpfully and harmlessly. **Answer: C (Instruction Tuning using prompt-completion pairs or reformatting existing datasets into an instructional format.)** A: **Incorrect:** Unsupervised pre-training only teaches next-token prediction on raw text, not instruction following or helpful alignment. B: **Incorrect:** Classification fine-tuning restricts the model to discrete labels rather than open-ended helpful generation. C: **Correct:** Instruction tuning uses prompt-response pairs to align the model toward helpful, instruction-following behavior. D: **Incorrect:** PEFT/LoRA is a parameter-efficient optimization method, not a training paradigm for alignment.

Let us trace the Hugging Face `tokenizers` pipeline configured in the snippet: 1. `Normalizer = Lowercase()` converts `"Hello World"` to `"hello world"`. 2. `PreTokenizer = Whitespace()` splits it into `["hello", "world"]`. 3. `TemplateProcessing(single="[CLS] $A [SEP]", ...)` wraps a single input sequence `$A` with the `[CLS]` token at the beginning and the `[SEP]` token at the end. Therefore, `encoded_output.ids` starts with the token ID for `[CLS]`, followed by the token IDs for `"hello world"`, and ends with the token ID for `[SEP]`. **Answer: D (The ids list will start with the ID for [CLS], followed by token IDs for "hello world", and end with the ID for [SEP].)** A: **Incorrect:** `overflowing_tokens` is only populated when truncation is enabled and the input exceeds the maximum length. B: **Incorrect:** While `type_ids` exists on `Encoding`, `TemplateProcessing` with `single='[CLS] $A [SEP]'` specifically prepends `[CLS]` and appends `[SEP]` to `ids`. C: **Incorrect:** `word_ids` tracks pre-tokenization word indices, not the special tokens added by `TemplateProcessing`. D: **Correct:** `Lowercase()` lowercases the text and `TemplateProcessing` wraps the sequence IDs with `[CLS]` at the start and `[SEP]` at the end.

In the Hugging Face `datasets` library, `Dataset.map(add_length)` applies `add_length` to each example dictionary and **adds** any new keys returned (`"length"`) as new columns while **preserving** existing columns (`"text"`). Therefore, `dataset.column_names` returns `['text', 'length']`. **Answer: C (`['text', 'length']`)** A: **Incorrect:** `dataset.map()` adds the new `'length'` column to the dataset. B: **Incorrect:** Existing columns are kept unless `remove_columns` is explicitly passed to `map()`. C: **Correct:** `dataset.map()` retains the original `'text'` column and appends the newly created `'length'` column. D: **Incorrect:** Returning a modified `example` dictionary is the standard valid usage of `Dataset.map()`.

Let us evaluate each statement about subword tokenization algorithms (BPE, WordPiece, SentencePiece): - **A is incorrect**: Character-level tokenization has a very *small* vocabulary (just the alphabet); subword tokenization *increases* vocabulary size compared to character-level tokenization (while reducing it compared to word-level tokenization). - **B is correct**: Subword tokenizers handle Out-Of-Vocabulary (OOV) words by decomposing unseen words into known subword and character units. - **C is incorrect**: BPE and WordPiece merge the **most** frequent (or highest-likelihood) pairs, not the least frequent pairs. - **D is correct**: Frequent words are merged into single whole-word tokens, whereas rare words remain split into multiple smaller subword tokens. - **E is correct**: SentencePiece treats the input as a raw character stream (encoding spaces as `_` / `▁`), making it language-agnostic without requiring whitespace pre-tokenization. **Answer: B, D, E** A: **Incorrect:** Subword vocabularies (e.g., 30k–50k tokens) are much *larger* than character-level vocabularies, not smaller. B: **Correct:** Unknown words are broken down into known subword/character tokens, avoiding `<unk>` tokens. C: **Incorrect:** BPE iteratively merges the *most* frequent adjacent symbol pairs, not the least frequent. D: **Correct:** High-frequency words become single tokens while rare/complex words are represented as sequences of subwords. E: **Correct:** SentencePiece operates on raw text without assuming space-separated words, supporting languages like Chinese and Japanese.

Wait: let us check the official answer key for Q#219 (`ans=['B', 'D', 'E']`): - **B is correct**: Few-shot prompting (in-context learning) leverages a GPT-3 class model's ability to learn the task format from a few input-summary examples in the prompt without any parameter updates. - **C is incorrect**: Full supervised fine-tuning of a GPT-3 scale model requires massive compute and will severely overfit a very small labeled dataset. - **D is correct**: Parameter-Efficient Fine-Tuning (PEFT) such as LoRA updates only a tiny fraction of weights (low-rank adapters), making it compute-efficient and much less prone to overfitting on small datasets. - **E is correct**: Domain-adaptive continual pre-training on unlabeled domain text (or semi-supervised adaptation) is also marked as a viable strategy in the official key (`B, D, E`). **Answer: B, D, E** A: **Incorrect:** A base (non-instruction-tuned) GPT-3 class model often struggles with zero-shot instruction prompts compared to few-shot in-context learning. B: **Correct:** Few-shot in-context learning requires zero gradient updates and works effectively with large pre-trained language models. C: **Incorrect:** Full fine-tuning of all parameters is computationally prohibitive and overfits small datasets. D: **Correct:** LoRA drastically reduces trainable parameters and memory/compute requirements, making it ideal for small datasets. E: **Correct:** Marked correct in the official key as a domain adaptation strategy.

In Hugging Face `TrainingArguments`, the frequency at which model checkpoints (weights) are saved during training is directly controlled by: - **`save_strategy="epoch"`** (Option B): Specifies whether checkpoints are saved at the end of every `"epoch"`, every `"steps"`, or `"no"`. - **`save_steps=500`** (Option D): Specifies the number of training steps between two checkpoint saves when `save_strategy="steps"`. In contrast, `evaluation_strategy` controls validation frequency, `learning_rate` controls optimizer step size, and `load_best_model_at_end` controls post-training model loading. **Answer: B, D** A: **Incorrect:** `evaluation_strategy` controls when validation metrics are computed, not when model weights are saved. B: **Correct:** `save_strategy='epoch'` configures the Trainer to save a checkpoint at the end of each epoch. C: **Incorrect:** `learning_rate` controls gradient descent step size, not checkpoint saving. D: **Correct:** `save_steps=500` configures the Trainer to save model weights every 500 training steps. E: **Incorrect:** `load_best_model_at_end` loads the best checkpoint after training finishes rather than setting the save interval.

**Step 1: Count the initial vocabulary size.** The initial character vocabulary is `{"a": 10, "b": 8, "c": 5, "</w>": 15}`, which contains **4** tokens: `"a"`, `"b"`, `"c"`, and `"</w>"`. **Step 2: Perform 1 BPE merge operation.** Let us count adjacent symbol pair frequencies in the corpus: - `"a b c </w>"` (5 times) - `"a b </w>"` (3 times) - `"a a c </w>"` (2 times) Pair counts: - `("a", "b")`: \(5 + 3 = 8\) - `("b", "c")`: \(5\) - `("c", "</w>")`: \(5 + 2 = 7\) - `("b", "</w>")`: \(3\) - `("a", "a")`: \(2\) - `("a", "c")`: \(2\) The most frequent adjacent pair is `("a", "b")` (count = 8), which is merged into a single new token `"ab"`. Adding this **1** new merged token to the **4** initial tokens gives a new vocabulary size of: \[4 + 1 = 5\] **Answer: 5**

**Step 1: Evaluate the filter condition `len(x['text'].split()) > 2` on each of the 4 dataset samples.** 1. `"short one"` \(\to\) `["short", "one"]` \(\to\) length = \(2\) (\(2 > 2\) is **False**) 2. `"a very long sentence indeed"` \(\to\) 5 words \(\to\) length = \(5\) (\(5 > 2\) is **True**) 3. `"medium example three"` \(\to\) 3 words \(\to\) length = \(3\) (\(3 > 2\) is **True**) 4. `"tiny"` \(\to\) 1 word \(\to\) length = \(1\) (\(1 > 2\) is **False**) **Step 2: Count the retained samples.** Only samples 2 and 3 satisfy `len(...) > 2`, so `filtered_dataset` contains **2** samples. **Answer: 2**

**Step 1: Calculate parameters in the first linear layer (\(d_{\text{model}} \to d_{\text{ff}}\)).** - Weights: \(768 \times 3072 = 2,359,296\) - Biases: \(3072\) - Subtotal 1 = \(2,362,368\) **Step 2: Calculate parameters in the second linear layer (\(d_{\text{ff}} \to d_{\text{model}}\)).** - Weights: \(3072 \times 768 = 2,359,296\) - Biases: \(768\) - Subtotal 2 = \(2,360,064\) **Step 3: Sum and convert to millions.** \[\text{Total parameters} = 2,362,368 + 2,360,064 = 4,722,432 \approx 4.72\text{ million}\] Rounded to one decimal place, this is **4.7** million. **Answer: 4.7**

The comprehension passage specifies a **Causal Language Model** (decoder-only architecture like GPT-2) with `vocab_size=32000`, `embedding_dimension=1024`, `num_transformer_blocks=24`, and `num_attention_heads=16`. In Hugging Face `transformers`, `GPT2Config` is the configuration class for a causal language model and uses the parameter names `vocab_size=32000, n_embd=1024, n_layer=24, n_head=16`. (In contrast, `BertConfig` is for an encoder-only Masked LM, and `T5Config` is for an encoder-decoder model.) **Answer: B (`config = GPT2Config(vocab_size=32000, n_embd=1024, n_layer=24, n_head=16)`)** A: **Incorrect:** `BertConfig` initializes an encoder-only Masked Language Model, not a Causal Language Model. B: **Correct:** `GPT2Config` defines a decoder-only Causal Language Model with `vocab_size=32000, n_embd=1024, n_layer=24, n_head=16`. C: **Incorrect:** `T5Config` initializes an encoder-decoder sequence-to-sequence architecture. D: **Incorrect:** Loads an encoder-decoder config and omits the layer and head counts.

**Step 1: Recall the parameter count of a token embedding layer.** A token embedding lookup table has shape `(\text{vocab\_size}, d_{\text{model}})`: \[\text{Parameters} = \text{vocab\_size} \times d_{\text{model}} = 32,000 \times 1024 = 32,768,000\] **Step 2: Express in millions rounded to one decimal place.** \[\frac{32,768,000}{10^6} = 32.768\text{ million} \approx 32.8\text{ million}\] **Answer: 32.8**

The model is explicitly specified as a **Causal Language Model** with 24 Transformer blocks, 1024 hidden dimension, and 16 heads (~350M parameters, matching GPT-2 Medium). Causal Language Models use a **decoder-only** Transformer architecture with causal (masked) self-attention, trained to predict the next token given preceding tokens. **Answer: C (It’s a decoder-only model, pre-trained using a Causal Language Modeling objective.)** A: **Incorrect:** Encoder-decoder models (like T5 or BART) use Seq2Seq objectives rather than pure Causal Language Modeling. B: **Incorrect:** Encoder-only models (like BERT) use bidirectional attention and Masked Language Modeling. C: **Correct:** Causal Language Models are decoder-only architectures trained with the autoregressive next-token prediction objective. D: **Incorrect:** With 24 layers and 1024 hidden size (~350M parameters), it is a medium-to-large model, not a tiny edge model.

**Step 1: Compute the memory required per parameter for Adam optimizer states.** Adam stores 2 floating-point values (first moment `m` and second moment `v`) per parameter, and each 32-bit float takes 4 bytes: \[\text{Bytes per parameter} = 2 \times 4\text{ bytes} = 8\text{ bytes}\] **Step 2: Multiply by the 350 million trainable parameters.** \[\text{Total bytes} = 350 \times 10^6 \times 8 = 2,800 \times 10^6\text{ bytes} = 2.8 \times 10^9\text{ bytes}\] Since \(1\text{ GB} = 10^9\text{ bytes}\), this requires **2.8 GB**. **Answer: 2.8**

Byte Pair Encoding (BPE) builds a vocabulary starting from individual characters/bytes and iteratively merges frequent subword pairs. During inference on diverse text, any rare or unseen Out-Of-Vocabulary (OOV) word is decomposed into a sequence of known subword/character tokens from the vocabulary, allowing the model to process arbitrary words without failing. **Answer: C (Out-of-vocabulary words can be represented as sequences of known subword units, allowing the model to process them.)** A: **Incorrect:** BPE falls back to smaller subword and character units so unseen words do not cause OOV failures. B: **Incorrect:** Common words remain single tokens; only rare or unseen words are split into multiple subwords. C: **Correct:** BPE represents unseen words as sequences of known subword units. D: **Incorrect:** Token fertility (tokens per word) varies across languages depending on the training corpus distribution.