Deep Learning Practice · Quiz 1 · Jan 2026

**1. Define subword tokenization:** A tokenizer can represent a word using smaller reusable pieces. Frequent words may remain whole; rarer words may be split into several pieces. **2. Explain how pieces are learned:** BPE learns merges from token-pair frequency. WordPiece also uses corpus statistics, commonly described through a likelihood-based selection criterion. Neither method directly understands the meaning of a word or guarantees a linguistic morpheme boundary. **3. Apply this distinction:** Statistical segmentation is the fundamental shared property. It helps represent unfamiliar words using known pieces, but does not make every word one token or guarantee perfect domain transfer. **Answer: B — Selected: They learn merge rules from corpus statistics (frequency/likelihood), not token semantics.** A: **Incorrect:** A statistically useful split can fall inside a morpheme. B: **Correct:** The token inventory/merges are learned from corpus statistics rather than semantic understanding. C: **Incorrect:** An unfamiliar word is typically represented by several pieces; some tokenizers can still produce unknown tokens. D: **Incorrect:** Unfamiliar domains can cause more fragmentation and longer sequences.

**1. Compare the vocabularies:** With 64,000 entries, Tokenizer B can store more complete words and longer pieces than a tokenizer with 8,000 entries. On comparable text, this generally reduces the number of pieces per sequence. **2. Count embedding parameters:** For vocabulary size \(V\) and embedding dimension \(d\), the token embedding matrix contains \(Vd\) parameters. Increasing \(V\) from 8,000 to 64,000 multiplies that matrix size by eight when \(d\) is unchanged. **3. Interpret the tradeoff:** Shorter sequences can reduce attention work, while larger embeddings and vocabulary-output computations increase parameter/memory costs. Total compute is not guaranteed to decrease for every dataset. **Answer: C — Selected: Larger vocabulary reduces sequence length but increases embedding parameters.** A: **Incorrect:** The word always is too strong; vocabulary-related computation and parameter storage increase. B: **Incorrect:** A smaller vocabulary does not guarantee better semantic alignment. C: **Correct:** This states the usual shorter-sequence versus larger-embedding tradeoff. D: **Incorrect:** Vocabulary size still affects token count, memory and output computation.

**1. Diagnose the failure:** Chemical names are fragmented because the tokenizer vocabulary poorly represents the target domain. This can make entity boundaries harder to learn and consume excessive context length. **2. Address the source:** Learn or adapt tokenization on a representative chemical corpus so that useful domain patterns are represented more efficiently. **3. Integrate it correctly:** A changed vocabulary must be matched with the model embedding/output layers and suitable continued training or initialization. Dropout, batch-size changes and label smoothing do not directly repair inappropriate token boundaries. **Answer: B — Selected: Retrain the tokenizer using an in-domain (chemical) corpus.** A: **Incorrect:** Dropout regularizes training but does not change segmentation. B: **Correct:** Domain-trained tokenization directly addresses excessive fragmentation of chemical terms. C: **Incorrect:** Batch size changes optimization/memory behavior, not the token vocabulary. D: **Incorrect:** Label smoothing alters target distributions and does not fix token splits.

**1. Read the settings:** max_length=12, padding="max_length" and truncation=True create a sequence of length 12 by truncating long inputs or padding short inputs. **2. Account for BERT special tokens:** The standard BERT tokenizer inserts [CLS] and [SEP]. These occupy positions within the 12-token budget, leaving fewer positions for ordinary text pieces. **3. Distinguish length from content:** Twelve total positions do not imply twelve non-padding text tokens. Truncation follows token positions, which can split a multi-piece word; padding and truncation are compatible settings. **Answer: B — Selected: Special tokens may consume a portion of the 12-token length budget.** A: **Incorrect:** Short inputs contain padding, so the non-padding count can be below 12. B: **Correct:** Special tokens are included in the configured total length. C: **Incorrect:** Subword truncation is not guaranteed to preserve whole words. D: **Incorrect:** Short inputs are still padded to 12 even when truncation is enabled.

**1. Identify the training setup:** The code loads a pretrained BERT masked-language-model head and updates it using an unlabeled domain corpus. Learning further from domain text is continued pretraining, rather than supervised classification or instruction tuning. **2. Explain the intended objective:** Proper masked language modeling hides selected input tokens and trains the model to recover their original identities. These targets come from the text itself, so manually supplied task labels are unnecessary. **3. Read the code carefully:** C names the intended training category. However, the printed code passes identical input_ids and labels without showing masking. By itself, that trains unmasked token reconstruction and permits trivial copying. A correct MLM implementation must mask/corrupt selected input positions and usually set unscored label positions to -100, for example using an MLM data collator. **Answer: C — Selected: This performs continual pretraining with MLM objective.** A: **Incorrect:** No externally supplied noisy classification labels are shown. B: **Incorrect:** A labels argument is required by many self-supervised losses; it does not establish instruction tuning. C: **Correct:** Continued domain pretraining with an intended MLM head is the closest description; actual masking is missing from the printed snippet. D: **Incorrect:** The MLM head predicts vocabulary tokens at positions, not a sequence-level class.

**1. Track tensor shapes:** For a sequence of \(N\) tokens, one attention head has query and key matrices of shape \(N\times d_k\). **2. Form pairwise scores:** Multiplying \(QK^{\mathsf T}\) gives an \(N\times N\) matrix. Every query token is compared with every key token, giving \(N^2\) pairwise scores and work proportional to \(N^2d_k\). **3. Contrast token-wise operations:** Embedding lookup, residual addition and a fixed-width feed-forward network process each position independently and scale linearly with \(N\), holding model dimensions fixed. **Answer: C.** A: **Incorrect:** Lookup returns one fixed-width vector per position, so it is linear in sequence length. B: **Incorrect:** Residual addition adds corresponding tensor elements and is linear in the number of positions. C: **Correct:** Pairwise query-key comparison produces the quadratic attention-score matrix. D: **Incorrect:** An FFN applies the same fixed-width transformations independently at each token.

**1. Count weights and gradients:** Four billion fp32 parameters need \(4\times10^9\times4=16\times10^9\) bytes, or 16 GB. Their gradients need another 16 GB. **2. Count AdamW states:** AdamW stores two moment values per trainable parameter. At fp32 this adds \(2\times16=32\) GB. **3. Add the minimum training storage:** \(16+16+32=64\) GB, already twice the 32-GB capacity before activations and temporary buffers. This assumes ordinary full training without sharding/offloading or state compression. **Answer: B — Selected: The optimizer states (32GB) plus the model weights (16GB) and gradients (16GB) exceed the 32GB VRAM limit.** A: **Incorrect:** Weights alone require 16 GB, not 32 GB, under the stated decimal units. B: **Correct:** Weights, gradients and two moment buffers already total 64 GB. C: **Incorrect:** Context length and integer indexing are unrelated to this memory calculation. D: **Incorrect:** fp32 uses four bytes per number, not eight.

**1. Define weight decay:** It shrinks trainable weights toward zero during optimization, discouraging unnecessarily large parameter magnitudes. **2. Show the effect:** In a simplified decoupled update, \[\theta_{t+1}=(1-\eta\lambda)\theta_t-\eta u_t,\] where \(\eta\) is the learning rate, \(\lambda\) the weight-decay coefficient and \(u_t\) the optimizer update direction. **3. Match the question:** The parameter controlling that magnitude penalty is weight_decay. This regularizes weight size; it does not directly constrain distance from pretrained weights unless that separate objective is added. **Answer: D — Selected: Weight Decay.** A: **Incorrect:** Accumulation combines gradients from microbatches; it does not impose a weight-size penalty. B: **Incorrect:** Batch size controls the examples per update. C: **Incorrect:** Checkpointing saves state or trades activation storage for recomputation. D: **Correct:** Weight decay explicitly discourages large weight magnitudes.

**1. Identify inference storage:** Inference needs model weights, temporary activations and possibly an autoregressive cache. It does not need optimizer moment buffers or parameter gradients. **2. Identify training storage:** Full fine-tuning stores trainable parameters, their gradients, optimizer states and activations needed for backpropagation. **3. Compare under similar conditions:** These extra training tensors explain why training usually requires substantially more memory, even if sequence length and batch size are the same. **Answer: C — Selected: Training requires storing gradients and optimizer states.** A: **Incorrect:** Inference batch size can be smaller or larger; it is not the inherent explanation. B: **Incorrect:** Optimizer states are used during training, not ordinary inference. C: **Correct:** Gradients and optimizer buffers add substantial training-only memory. D: **Incorrect:** Sequence length is a configurable choice, not the defining training/inference difference.

**1. Identify the available memory-saving method:** Mixed precision uses lower-precision formats such as fp16 or bf16 for suitable computations and tensors while retaining higher precision where needed for numerical stability. **2. Explain the saving:** Lower-precision activations and related temporary tensors require fewer bytes. Suitable GPUs can also execute those operations faster. Some training schemes retain fp32 master weights and optimizer states, so memory does not necessarily halve overall. **3. Match the alternatives:** Gradient clipping and warmup improve optimization stability. Less frequent disk checkpoints mainly change I/O, not the tensors required by a training step. **Answer: B — Selected: Mixed-precision training.** A: **Incorrect:** Clipping controls gradient magnitude and does not remove the main training buffers. B: **Correct:** Mixed precision reduces storage for suitable tensors without changing the network architecture. C: **Incorrect:** Warmup changes the learning-rate schedule. D: **Incorrect:** Saving fewer checkpoints reduces disk work, not the normal GPU state for training.

**1. Interpret input_ids:** These are integer vocabulary indices, not the actual word strings. With return_tensors="pt", they are returned as PyTorch tensors. **2. Interpret masks:** An attention_mask normally uses 1 for valid input positions and 0 for padding, preventing padded keys from being treated as real context. token_type_ids, when supported, identify segments such as sentence A and sentence B. **3. Account for model conventions:** Standard BERT preprocessing adds a leading [CLS] token and a separating [SEP] token. These special-token IDs are part of input_ids. **Answer: A, B, D — Selected: The input_ids tensor contains the numerical indices mapped from the vocabulary.; The attention_mask contains 0s for padding tokens and 1s for real tokens to prevent the model from attending to padding.; In models like BERT, the input_ids will typically begin with a special token index (e.g., [CLS]).** A: **Correct:** Each element indexes an entry in the tokenizer vocabulary. B: **Correct:** The usual mask distinguishes valid positions from padding positions. C: **Incorrect:** Token type IDs describe segments, not uppercase/lowercase character categories. D: **Correct:** BERT conventionally begins the encoded sequence with [CLS].

**1. Compute effective batch size:** The printed configuration uses one example per device per microbatch and eight gradient-accumulation steps. On one device, the effective batch size is \(1\times8=8\); with \(G\) devices it is \(8G\). **2. Interpret fp16:** Mixed precision reduces memory for suitable tensors and operations compared with an all-fp32 implementation, though optimizer states can remain fp32. **3. Interpret max_grad_norm=1.0:** Gradients are clipped to control their norm before an optimizer step. This limits extreme gradient-driven updates; it is not a universal hard bound on an Adam parameter-step norm. Accumulation changes when the optimizer steps, not the number of moment buffers per parameter. **Answer: A, B, C — Selected: Effective batch size is larger than 1.; Reduced memory usage compared to FP32 training.; Gradient clipping limits large parameter updates.** A: **Correct:** Eight microbatches contribute to each update, giving an effective batch larger than one. B: **Correct:** fp16 reduces storage for lower-precision tensors, subject to implementation overhead. C: **Correct:** Gradient clipping controls large gradients and thus helps prevent excessively large gradient-driven steps. D: **Incorrect:** Optimizer state count depends on trainable parameters and the optimizer, not accumulation steps.

**1. Encoder-only models:** Models such as BERT build contextual representations from both left and right context. Masked language modeling is a common pretraining objective. **2. Decoder-only models:** GPT-style models use causal attention so that a position cannot inspect future tokens. Generation proceeds autoregressively, adding tokens one at a time. **3. Encoder-decoder models:** An encoder represents the source and a decoder generates a target while attending to that source. This supports translation and other sequence-to-sequence tasks. Their masks differ across encoder self-attention, decoder self-attention and cross-attention. **Answer: A, B, C — Selected: Encoder-only models are commonly trained with Masked Language Modeling objectives.; Decoder-only models rely on causal masking and autoregressive inference.; Encoder-decoder models are well suited for sequence-to-sequence tasks.** A: **Correct:** MLM is a common objective for bidirectional encoder-only models. B: **Correct:** Causal masking supports left-to-right autoregressive generation. C: **Correct:** Separate source encoding and target decoding naturally support sequence-to-sequence tasks. D: **Incorrect:** The masks serve different visibility rules and are not identical across architectures.

**1. Compare prediction formats:** A sequence classifier predicts one of a fixed set of class IDs. The shown generative formatting presents an instruction, an input sentence and a textual answer. **2. Identify the learned output:** A causal language model predicts next tokens from its full vocabulary. Training can therefore teach it to express a class label as text, while its output head remains vocabulary-sized. **3. Explain the limits:** The objective is token-sequence likelihood rather than a separate fixed-class head. This does not guarantee correct behavior on unseen prompts. The code is schematic: a runnable Trainer setup must tokenize the text and supply labels/collation; whether prompt tokens are included in loss depends on label masking. **Answer: A, B, D.** A: **Correct:** The formatted example conditions prediction on an explicit task instruction and input. B: **Correct:** The language-model head retains vocabulary-wide token outputs, rather than only N class logits. C: **Incorrect:** A prompt format cannot guarantee perfect zero-shot generalization. D: **Correct:** The generative formulation trains next-token predictions over the chosen scored sequence positions.

**1. Factorize the sequence:** Causal language modeling uses \[p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_1,\ldots,x_{t-1}).\] The prediction distribution for target \(x_t\) can use only its earlier-token context. **2. Explain the loss and mask:** The loss term is \(-\log p(x_t\mid x_{<t})\). It necessarily uses \(x_t\) as the target, but its prediction cannot inspect future input tokens. In the common additive/blocked-position convention, strictly upper-triangular entries mark future positions as forbidden. An allowed-attention binary mask is instead lower triangular. **3. Interpret gradients correctly:** There is no forward attention from an earlier query to a future token. Later-token losses can still backpropagate through earlier context representations and shared parameters. Thus C is not correct if it means forbidden future-to-past attention; it should not be taken as a claim that all such gradient paths are absent. **Convention note:** A describes the prediction context, rather than claiming the target value is absent from its own loss. B describes the blocked-future mask convention. **Answer: A, B, D.** A: **Correct:** Under the intended context interpretation, the predictor for target t sees only tokens before t; the loss also uses the target itself. B: **Correct:** The strictly upper-triangular part marks blocked future keys in the common causal-mask convention. C: **Incorrect:** Future inputs cannot be attended to by earlier queries; later losses may nevertheless affect earlier representations through legitimate backward paths. D: **Correct:** The autoregressive product follows directly from left-to-right conditional prediction.

**1. Describe warmup:** During an initial number of steps, a warmup schedule raises the learning rate gradually from zero or a small value to its target level. **2. Explain its purpose:** Large early updates can destabilize parameters and optimizer statistics. Smaller initial steps help the training process settle before applying the full learning rate. **3. Describe the later schedule:** Warmup is commonly followed by linear or cosine decay. It does not remove optimizer states, and neither warmup nor decay alone guarantees convergence. **Answer: A, B, D — Selected: It involves linearly increasing the learning rate from 0 (or a small value) to the target maximum during the initial phase.; It helps prevent "divergence'' or instability caused by large gradients when the model weights are far from their optimal state.; After the warmup phase, the learning rate typically follows a decay schedule (like linear or cosine) to ensure convergence.** A: **Correct:** Linear warmup gradually increases the learning rate during the configured initial steps. B: **Correct:** Conservative early steps can reduce optimization instability. C: **Incorrect:** Moment buffers still exist for every trainable parameter. D: **Correct:** Decay after warmup is a standard schedule, though convergence is not guaranteed merely by choosing it.

**1. Identify the four stored values:** Every parameter requires a weight, a gradient, an Adam first moment and an Adam second moment. Each value occupies 4 bytes. **2. Compute bytes per parameter:** \(4\times4=16\) bytes. With \(1.2\times10^9\) parameters, \[M=1.2\times10^9\times16=19.2\times10^9\text{ bytes}.\] **3. Convert units:** The question defines \(1\text{ GB}=10^9\) bytes, so the total is 19.2 GB. Activations, buffers and other overhead are explicitly excluded. **Answer: 19.2.**

**1. Identify projection shapes:** Standard multi-head attention combines the heads into query, key and value projections, each with \(d\times d\) weights. The output projection also has \(d\times d\) weights. **2. Substitute the embedding dimension:** With \(d=768\), one matrix contains \(768^2=589{,}824\) weights. All four contain \[4d^2=4\times589{,}824=2{,}359{,}296.\] **3. Report in millions:** \(2{,}359{,}296/10^6=2.359296\), rounded to one decimal place, is 2.4 million. Twelve heads give a head size of 64; they split the same projected width rather than multiplying this parameter count by 12. Sequence length affects computation, not these weights. **Answer: 2.4.**

**1. Token embeddings:** A vocabulary of 32,000 and width 1,024 require \(32{,}000\times1{,}024=32{,}768{,}000\) parameters. **2. Learned positional embeddings:** Each of the 2,048 context positions has its own 1,024-dimensional vector, giving \(2{,}048\times1{,}024=2{,}097{,}152\) parameters. **3. Add and round:** \[32{,}768{,}000+2{,}097{,}152=34{,}865{,}152.\] This is 34.865152 million, or 34.9 million to one decimal place. No segment embeddings or output-head parameters are requested. **Answer: 34.9.**