Deep Learning Practice · Quiz 1 · May 2026

**1. Diagnose unknown words:** An agglutinative language combines many meaningful pieces into productive word forms. A whole-word vocabulary cannot store every combination, so unseen forms become [UNK]. **2. Reuse smaller pieces:** A subword tokenizer such as a suitable SentencePiece or BPE model learns reusable parts. New words can then be represented as combinations of those pieces without adding every word to the vocabulary. **3. Compare interventions:** Batch size changes training computation, not coverage. Raising a frequency threshold excludes more rare forms, and storing every unnormalized form worsens vocabulary growth. Character coverage/byte fallback settings still matter for avoiding all unknown symbols. **Answer: A — Selected: Use SentencePiece or BPE.** A: **Correct:** Subword units improve coverage while keeping vocabulary size controlled. B: **Incorrect:** Changing batch size does not change which strings the vocabulary can represent. C: **Incorrect:** A larger minimum frequency generally excludes more rare word forms. D: **Incorrect:** Keeping every surface variation separately expands rather than controls the vocabulary.

**1. Read the options:** The tokenizer uses max_length=128, truncation=True and padding="max_length". The untruncated encoding has 220 positions. **2. Apply truncation:** Because 220 exceeds 128, the returned encoding is truncated to 128 total positions. There is no extra padding beyond that maximum. **3. Interpret the wording:** C states the intended output length. The 128 positions can include tokenizer-added special tokens, so “from the document” should not be read as guaranteeing 128 ordinary text tokens. Default right truncation retains the beginning plus required boundary special tokens. **Answer: C — Selected: The output contains exactly 128 tokens from the document.** A: **Incorrect:** Truncation is applied before the encoding is returned, so 220 positions are not retained. B: **Incorrect:** Padding does not extend a truncated sequence beyond max_length. C: **Correct:** The returned input_ids length is exactly 128 under these settings. D: **Incorrect:** The length is fixed by the explicit settings for this overlength input.

**1. Define LoRA:** Low-Rank Adaptation keeps selected pretrained weight matrices frozen and represents their trainable changes using smaller matrices. **2. Show the parameter saving:** For a weight \(W\in\mathbb R^{m\times n}\), a rank-r update \(BA\) uses \(r(m+n)\) trainable values instead of mn, when r is small. **3. Apply to the observation:** Updating only 0.15% is consistent with training adapters while most of the 13-billion-parameter backbone stays frozen. Exact fractions depend on target modules and rank. **Answer: A — Selected: LoRA updates low rank matrices while freezing most pretrained weights.** A: **Correct:** LoRA trains compact low-rank updates while freezing the base matrices. B: **Incorrect:** LoRA is not restricted to training token embeddings. C: **Incorrect:** Quantization is a separate technique, even when combined with LoRA. D: **Incorrect:** Attention layers remain present; selected projections receive low-rank updates.

**1. Use the unlabeled corpus for domain knowledge:** Continued language-model pretraining on the medical documents adapts representations and terminology without requiring manual question-answer labels. **2. Use reviewed examples for behavior:** Instruction tuning on the 300 reviewed pairs teaches the desired question-answer format and task response style. The small supervised set needs careful validation to avoid overfitting. **3. Combine complementary data:** This sequence uses both data resources for their appropriate purposes. A larger context window or batch size alone does not teach domain facts or instruction-following behavior. **Answer: B — Selected: Continue pretraining on the medical corpus and then instruction tune.** A: **Incorrect:** A new tokenizer alone does not adapt the model’s knowledge or response behavior. B: **Correct:** Domain pretraining followed by instruction tuning uses both the large unlabeled corpus and the reviewed examples. C: **Incorrect:** Context length changes input capacity, not domain knowledge by itself. D: **Incorrect:** Batch size is an optimization setting rather than a domain-adaptation objective.

**1. Identify fp32 storage:** One fp32 parameter occupies 4 bytes. **2. Multiply by parameter count:** \[M=7\times10^9(4)=28\times10^9\text{ bytes}.\] **3. Convert units:** Using decimal GB gives approximately 28 GB for weights alone. Real training/inference can require additional memory; that overhead is excluded here. **Answer: C — Selected: 28 GB.** A: **Incorrect:** 7 GB would use one byte per parameter. B: **Incorrect:** 14 GB corresponds to two-byte weight storage. C: **Correct:** Four bytes for each of seven billion weights gives 28 GB. D: **Incorrect:** 56 GB would use eight bytes per parameter.

**1. Explain sequence compression:** A larger vocabulary can contain more complete words and longer pieces, usually reducing how many tokens represent the same corpus text. **2. Explain the parameter cost:** With embedding width d, the token embedding matrix has Vd parameters. Increasing V from 16K to 128K multiplies this matrix size by eight at fixed d. **3. State the tradeoff:** Shorter typical sequences can reduce attention work, while a larger embedding/output vocabulary increases memory and vocabulary-related computation. Total efficiency is not guaranteed to improve for every input. **Answer: A — Selected: Shorter sequences and a larger embedding layer.** A: **Correct:** This captures the usual shorter-sequence versus larger-embedding tradeoff. B: **Incorrect:** The parameter direction is reversed. C: **Incorrect:** A larger vocabulary does not shrink the V-by-d embedding matrix. D: **Incorrect:** Vocabulary size affects sequence lengths and parameter storage.

**1. Identify the requested object:** The code wants a plain Python list containing integer token IDs. **2. Choose encode:** tokenizer.encode(text), with its default return format, directly returns that list, normally including the tokenizer’s default special tokens. **3. Distinguish related methods:** tokenize returns token strings. Calling the tokenizer or encode_plus returns a richer encoding containing input_ids and often masks, rather than only the list. **Answer: A.** A: **Correct:** encode returns the token-ID list requested by the code. B: **Incorrect:** __call__ returns a BatchEncoding with fields such as input_ids. C: **Incorrect:** tokenize returns string token pieces rather than integer IDs. D: **Incorrect:** encode_plus returns an encoding object with additional fields.

**1. Inspect the triangular mask:** torch.triu(..., diagonal=0) includes the main diagonal and all entries above it. True entries are replaced with negative infinity. **2. Track visibility:** The current token is blocked along with all future tokens. Only strictly earlier columns remain visible. The first row has no earlier column, so all its scores become negative infinity. **3. Explain the numerical result and correction:** Ordinary softmax over an all-negative-infinity row can produce NaNs. To allow each token to attend to itself and its past, use diagonal=1 so that only strictly future positions are blocked. **Answer: D.** A: **Incorrect:** The main diagonal should remain visible in standard causal self-attention. B: **Incorrect:** A boolean upper-triangular mask is valid for masked_fill when shape/device match. C: **Incorrect:** Valid softmax rows sum to one; the all-masked first row can instead become invalid. D: **Correct:** The diagonal is incorrectly blocked, and the first row can yield NaN attention weights.

**1. Compute effective batch size:** Two examples per device per microbatch and eight accumulation steps give \(2(8)=16\) examples per optimizer update on one device; multiply by device count when using data parallelism. **2. Separate backward and optimizer steps:** Gradients accumulate over eight microbatches before the optimizer updates. This simulates a larger batch while keeping microbatch activation storage small. **3. Interpret fp16:** Mixed precision reduces memory for suitable tensors and can speed supported hardware. Adam states can remain fp32, so their storage does not automatically halve. **Answer: A, B, D — Selected: Effective batch size is larger than 2; Memory usage is lower than FP32 training; Gradient accumulation can simulate larger batches.** A: **Correct:** The effective single-device batch is 16, exceeding two. B: **Correct:** Suitable lower-precision tensors generally reduce memory relative to an all-fp32 counterpart. C: **Incorrect:** The optimizer updates after accumulation, not after every microbatch. D: **Correct:** Accumulating gradients combines several microbatches into a larger effective batch. E: **Incorrect:** fp16 training does not automatically store Adam moments at half precision.

**1. Identify the padding problem:** GPT-2 does not define a separate padding token by default. A collator batching variable-length examples needs a valid padding token ID. **2. Explain the failure:** Without configuring a pad token, the padding operation raises an error. Setting ID 0 arbitrarily can repurpose a genuine vocabulary token rather than a dedicated padding symbol. **3. Use a standard convention carefully:** Reusing EOS as padding is common for GPT-2. Supply attention masks and mask padded label positions so padding is not scored. Adding a dedicated pad token and resizing embeddings is another option; EOS reuse also requires care to avoid masking genuine EOS labels indiscriminately. **Answer: A, C, D.** A: **Correct:** The variable-length batch requires padding, which fails when the tokenizer has no pad token configured. B: **Incorrect:** Decoder-only training can still batch variable-length sequences and require padding. C: **Correct:** ID 0 is a real GPT-2 vocabulary entry rather than a guaranteed pad token. D: **Correct:** EOS reuse is a common explicit padding convention. E: **Incorrect:** A separate [PAD] token is not defined by default in GPT-2.

**1. Reduce state storage:** Mixed precision stores suitable tensors in fewer bytes. Compatible quantization, often used with a frozen base model and adapters, can reduce weight/state memory. **2. Trade computation for activation memory:** Gradient checkpointing stores fewer activations and recomputes them during backpropagation. **3. Train fewer weights:** LoRA freezes the base and trains small adapters, reducing gradient and optimizer-state storage. These techniques have implementation/precision constraints, but each may help. Increasing vocabulary normally enlarges embeddings and output-related work. **Answer: A, B, C, D — Selected: Mixed precision training; Quantization; Gradient checkpointing; LoRA.** A: **Correct:** Lower-precision tensors can reduce memory demand. B: **Correct:** Compatible quantized training/adaptation can reduce weight memory; ordinary full updates of quantized weights are not assumed. C: **Correct:** Recomputation reduces saved activation storage at extra compute cost. D: **Correct:** Adapter training reduces the number of parameter gradients and optimizer states. E: **Incorrect:** A larger vocabulary usually increases embedding/output memory.

**1. Read the training format:** An instruction and an input sentence are followed by the textual answer “Positive”. The model learns to produce answer tokens in that task context. **2. Identify the output space:** A causal language model retains vocabulary-sized token predictions and can generate strings, rather than a single fixed class ID from a separate classification head. **3. Identify the objective:** The scored target positions use next-token prediction. This can teach instruction-following in the trained setting but does not guarantee generalization to arbitrary instructions. It is not identical to a separate logistic-regression classification objective. **Answer: A, B, D — Selected: The model learns to follow instructions; The output vocabulary remains open ended; Training becomes a next token prediction task.** A: **Correct:** The formatted examples teach the model to respond to the presented task instruction. B: **Correct:** The generative output head remains vocabulary-wide. C: **Incorrect:** A standard language-model head is not restricted to predefined class-ID logits. D: **Correct:** Training scores next-token targets over the selected sequence positions. E: **Incorrect:** A sequence of vocabulary predictions is not the same task formulation as fixed-class logistic regression.

**1. Identify text preprocessing outputs:** input_ids are vocabulary indices; attention_mask distinguishes valid input from padding. Some models/tokenizers also return token_type_ids for segment identity. **2. Identify alignment metadata:** Fast tokenizers can return offset_mapping when requested, mapping token spans to character positions in the original text. **3. Separate tokenizer from network:** Hidden states are learned contextual representations produced by a neural model forward pass, not by the tokenizer. Output availability depends on tokenizer/model support and requested flags. **Answer: A, B, C, E — Selected: input_ids; attention_mask; token_type_ids; offset_mapping.** A: **Correct:** Token IDs are a primary tokenizer output. B: **Correct:** An attention mask is commonly returned for padded encodings. C: **Correct:** Supported tokenizers can provide segment IDs, such as BERT sentence-type IDs. D: **Incorrect:** Hidden states are neural model outputs. E: **Correct:** Fast tokenizers support character-offset mappings when requested.

**1. Count token embeddings:** \(65{,}000(1536)=99{,}840{,}000\) parameters. **2. Count learned positional embeddings:** \(4096(1536)=6{,}291{,}456\) parameters. **3. Sum and round:** \[N=(65{,}000+4096)(1536)=106{,}131{,}456=106.131456\text{ million}.\] To one decimal place this is 106.1 million. The question excludes other model layers. **Answer: 106.1.**

**1. Identify four projection matrices:** Standard query, key, value and output projections each have shape \(d\times d\), with \(d=1536\). **2. Count weights:** \[N=4d^2=4(1536)^2=4(2{,}359{,}296)=9{,}437{,}184.\] **3. Convert units:** This is 9.437184 million, rounded to 9.4 million. Biases are excluded, and splitting attention into heads does not multiply this total again. **Answer: 9.4.**

**1. Count bytes per parameter:** Weight, gradient, first moment and second moment each occupy 4 bytes. Total storage is \(4+4+4+4=16\) bytes per parameter. **2. Multiply:** \[M=5\times10^9(16)=80\times10^9\text{ bytes}.\] **3. Convert:** Since \(1\text{ GB}=10^9\) bytes, the total is 80.0 GB. Activations and buffers are explicitly ignored. **Answer: 80.0.**

**1. Tokenize both sentences:** Standard bert-base-uncased encodes “Hi” as [CLS], hi, [SEP], giving length 3. “Hello beautiful world” becomes [CLS], hello, beautiful, world, [SEP], giving length 5. **2. Pad to the longest item:** With padding=True, the first sequence receives two padding positions to reach length 5. Its attention mask is [1,1,1,0,0]. **3. Count zeros:** The code selects the first attention mask and counts zeros, so it prints 2. Boundary special tokens are real attended positions, not padding. **Answer: 2.**

**1. Count tokens per microbatch:** \(8(4096)=32{,}768\) tokens, assuming all eight examples use the stated context length. **2. Count tokens per optimizer update:** Four accumulated microbatches give \(32{,}768(4)=131{,}072\) tokens. **3. Count all updates and convert:** \[131{,}072(10{,}000)=1{,}310{,}720{,}000\text{ tokens}=1310.72\text{ million}.\] Rounded to one decimal place, the answer is 1310.7 million. **Answer: 1310.7.**