Large Language Models · Quiz 1 · Jan 2026

**1. Set up the matrix multiplication:** In self-attention, the Query matrix \( Q \) is computed by multiplying the input sequence matrix \( X \) by the query projection matrix \( W_Q \): \[ Q = X W_Q \] **2. Check the dimensions:** - Input matrix: \( X \in \mathbb{R}^{T \times d_{\text{model}}} \) - Query projection matrix: \( W_Q \in \mathbb{R}^{d_{\text{model}} \times d_k} \) - Multiplying a \( (T \times d_{\text{model}}) \) matrix by a \( (d_{\text{model}} \times d_k) \) matrix cancels the inner dimension \( d_{\text{model}} \) and yields: \[ Q \in \mathbb{R}^{T \times d_k} \] **Answer: C — \( T \times d_k \).** A: **Incorrect:** \( d_{\text{model}} \times T \) is the transpose of the input shape \( X \), not the projected query matrix. B: **Incorrect:** \( d_k \times T \) is the shape of \( Q^T \), whereas \( Q = X W_Q \) has \( T \) rows and \( d_k \) columns. C: **Correct:** Multiplying \( X \in \mathbb{R}^{T \times d_{\text{model}}} \) by \( W_Q \in \mathbb{R}^{d_{\text{model}} \times d_k} \) produces \( Q \in \mathbb{R}^{T \times d_k} \). D: **Incorrect:** \( T \times d_{\text{model}} \) is the input shape before projecting into the \( d_k \)-dimensional query subspace.

**1. Examine the scaled dot-product attention formula:** \[ \text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V \] **2. Understand the variance of the dot product:** If the components of \( q \) and \( k \) are independent random variables with mean \( 0 \) and variance \( 1 \), their dot product \( q \cdot k = \sum_{i=1}^{d_k} q_i k_i \) has mean \( 0 \) and variance \( d_k \) (standard deviation \( \sqrt{d_k} \)). For large \( d_k \), the dot products grow large in magnitude, pushing the softmax function into saturation regions where one entry is near \( 1 \) and all others are near \( 0 \), causing vanishingly small gradients. **3. Conclude:** Dividing by \( \sqrt{d_k} \) rescales the variance of the dot products back to \( 1 \), preventing softmax saturation and preserving healthy gradients during backpropagation. **Answer: C — To prevent the dot product values from growing too large, which would push the softmax function into regions with extremely small gradients.** A: **Incorrect:** Dividing by a scalar \( \sqrt{d_k} \) does not change matrix dimensions. B: **Incorrect:** \( \sqrt{d_k} \) is a fixed constant scaling factor and does not change the number of trainable parameters. C: **Correct:** Scaling by \( 1/\sqrt{d_k} \) keeps the variance of \( QK^T \) near 1, preventing softmax saturation and vanishing gradients. D: **Incorrect:** Dividing by \( \sqrt{d_k} \) scales by the square root of the dimension, not by the \( L_2 \) norm of each individual vector.

**1. Understand the input-output interface of a Transformer Encoder layer:** Every Transformer Encoder layer takes a sequence of token representations of shape \( (T, d_{\text{model}}) \), applies multi-head self-attention and a position-wise feed-forward network (with residual connections and layer normalization), and outputs a contextualized representation of the exact same shape \( (T, d_{\text{model}}) \). **2. Apply to the 6-layer stack:** Because each of the \( N = 6 \) identical layers preserves both the sequence length \( T = 50 \) and the hidden dimension \( d_{\text{model}} = 512 \), the output of the 6th layer has shape \( 50 \times 512 \). **Answer: C — \( 50 \times 512 \) (One vector per token, preserving sequence length).** A: **Incorrect:** A Transformer encoder outputs contextualized vectors for all \( T = 50 \) tokens, not a single pooled \( 1 \times 512 \) vector. B: **Incorrect:** The output represents the \( T = 50 \) tokens of the final layer, not one vector per layer (\( 6 \times 512 \)). C: **Correct:** Each encoder layer preserves the shape \( T \times d_{\text{model}} = 50 \times 512 \) so layers can be stacked. D: **Incorrect:** Layers are applied sequentially rather than concatenated into \( (6 \times 50) \times 512 = 300 \times 512 \).

**1. Distinguish causal masking vs. padding masking in cross-attention:** In a Transformer encoder-decoder model: - **Causal (look-ahead) masking** is used only in the **decoder self-attention** layer so that target token \( t \) cannot attend to future target tokens \( > t \). - In the **cross-attention** layer, queries come from the decoder while keys and values come from the **encoder output**. Since the entire source sequence is already known and encoded, the decoder is allowed to attend to all positions of the source sequence (no causal mask). - However, when batching source sequences of different lengths, **padding masking** is applied in cross-attention so the decoder does not attend to `<PAD>` tokens in the source. **Answer: C — No causal masking is needed, but padding masking may be applied to ignore padded positions in the source sequence.** A: **Incorrect:** The encoder processes the full source input at once, so there are no 'future' source positions to hide with a causal mask. B: **Incorrect:** Causal masking is applied in decoder self-attention on the target sequence, not in encoder-decoder cross-attention. C: **Correct:** Cross-attention can view the entire source sentence (no causal mask), while using a padding mask to ignore `<PAD>` tokens. D: **Incorrect:** Padding masks are still needed in cross-attention whenever source batches contain padded sequences.

**1. Recall how BERT constructs input embeddings:** For every input token, BERT sums three embeddings: \[ E_{\text{input}} = E_{\text{token}} + E_{\text{position}} + E_{\text{segment}} \] where \( E_{\text{segment}} \in \{E_A, E_B\} \) is a learned embedding indicating whether a token belongs to Sentence A (segment ID 0) or Sentence B (segment ID 1). **2. Apply to single-sentence fine-tuning:** During single-sentence classification (such as sentiment analysis), the input has only one segment (`[CLS] sentence [SEP]`). To keep the input distribution consistent with pre-training, every token in the sequence is assigned segment ID 0, corresponding to segment embedding \( E_A \). **Answer: B — All tokens in the input sequence are assigned the same segment embedding (e.g., \( E_A \)).** A: **Incorrect:** Disabling segment embeddings would shift the magnitude and distribution of the input vectors away from what BERT saw during pre-training. B: **Correct:** Every token in a single-sentence input receives segment ID 0 (segment embedding \( E_A \)). C: **Incorrect:** Splitting a single sentence arbitrarily into \( E_A \) and \( E_B \) would falsely signal a two-sentence boundary to the model. D: **Incorrect:** \( E_A \) and \( E_B \) are learned during pre-training and reused (not re-initialized randomly) at fine-tuning time.

**1. Write the temperature-scaled softmax formula:** Given logits \( u = [2.0, 1.0, 0.5] \) and temperature \( T > 0 \), the probability of token \( i \) is: \[ P_i = \frac{\exp(u_i / T)}{\sum_j \exp(u_j / T)} = \frac{1}{\sum_j \exp\!\big((u_j - u_i)/T\big)} \] **2. Take the limit as \( T \to 0^+ \):** - For the maximum logit \( u_1 = 2.0 \), we have \( u_j - u_1 < 0 \) for all \( j \neq 1 \). As \( T \to 0^+ \), \( (u_j - u_1)/T \to -\infty \), so \( \exp((u_j - u_1)/T) \to 0 \), giving \( P_1 \to 1 \). - For all sub-optimal logits (\( u_2 = 1.0, u_3 = 0.5 \)), \( (u_1 - u_i)/T \to +\infty \), so \( P_2 \to 0 \) and \( P_3 \to 0 \). - Thus, \( P \to [1.0, 0.0, 0.0] \), a one-hot distribution concentrated on the argmax token (greedy decoding). **Answer: B — It approaches a "one-hot" distribution where the token with the highest logit has probability 1.0 (Greedy Search).** A: **Incorrect:** A uniform distribution is approached when \( T \to \infty \), not when \( T \to 0 \). B: **Correct:** As \( T \to 0^+ \), dividing by \( T \) magnifies logit differences infinitely, concentrating 100% of the probability mass on the highest logit (\( 2.0 \)). C: **Incorrect:** The original softmax distribution corresponds to \( T = 1 \). D: **Incorrect:** Mathematically, the limit as \( T \to 0^+ \) is well-defined and converges to the one-hot argmax distribution.

**1. Recall the role of special tokens in BERT:** - Every BERT input sequence begins with the special classification token `[CLS]`, while `[SEP]` marks the end (or separation) of segments. - During pre-training (Next Sentence Prediction) and sequence-level fine-tuning (such as spam or sentiment classification), the final-layer hidden state corresponding to the `[CLS]` token, denoted \( h_{\text{[CLS]}} \), aggregates sequence-wide representation via bidirectional self-attention. **2. Conclude:** Therefore, \( h_{\text{[CLS]}} \) is passed into the linear classification layer (`nn.Linear`) to predict the sequence label. **Answer: C — The final hidden state vector corresponding to the `[CLS]` token (\( h_{\text{[CLS]}} \)).** A: **Incorrect:** While mean pooling can be used in sentence embedding variants, standard BERT classification passes the `[CLS]` hidden state to the classification head. B: **Incorrect:** The `[SEP]` token marks sentence boundaries rather than serving as the sequence-level classification representation. C: **Correct:** In BERT, the final hidden state of the initial `[CLS]` token (\( h_{\text{[CLS]}} \)) is fed into the classification head. D: **Incorrect:** Concatenating all token vectors would depend on sequence length and is not how BERT sequence classification works.

**1. Analyze Multi-Head Attention vs. Single-Head Attention:** - **Option A:** By projecting \( Q, K, V \) into \( h \) lower-dimensional subspaces (\( d_k = d_{\text{model}} / h \)), multi-head attention enables the model to jointly attend to information from different representation subspaces at different positions. True. - **Option B:** Just as multiple convolutional filters in a CNN extract distinct spatial/channel features in parallel, multiple attention heads extract different relational patterns in parallel. True. - **Option C:** Because \( h \times d_k = d_{\text{model}} \), the combined \( Q, K, V \) projection matrices across all \( h \) heads have \( 3 \times d_{\text{model}}^2 \) parameters (plus the output projection \( W_O \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}} \)), so multi-head attention does **not** reduce parameter count compared to a single head of the same total dimension. False. - **Option D:** Different heads can specialize in distinct syntactic or semantic dependencies (e.g., coreference resolution linking a pronoun to different antecedents or predicates). True. **Answer: A, B, and D.** A: **Correct:** Multiple projection matrices project inputs into distinct representation subspaces simultaneously. B: **Correct:** Multiple attention heads play an analogous role to multiple feature kernels in a CNN. C: **Incorrect:** Splitting \( d_{\text{model}} \) into \( h \) heads of size \( d_{\text{model}}/h \) keeps the total projection parameter count the same (and adds an output projection \( W_O \)). D: **Correct:** Separate heads learn distinct attention distributions, capturing different syntactic and semantic links.

**1. Check matrix dimensions in \( Z = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V \):** Given \( Q, K, V \in \mathbb{R}^{T \times d_k} \) (since \( d_k = d_v \)): - **Option A:** Since \( Q \) is \( T \times d_k \) and \( K^T \) is \( d_k \times T \), their product \( QK^T \) has shape \( T \times T \). True. - **Option B:** The softmax operation is applied **row-wise** (independently for each query token so that the attention weights over all keys sum to 1 for each query), NOT globally across the entire \( T \times T \) matrix. False. - **Option C:** Multiplying the \( (T \times T) \) attention weight matrix by \( V \in \mathbb{R}^{T \times d_k} \) produces \( Z \in \mathbb{R}^{T \times d_k} \). True. - **Option D:** Entry \( (i, j) \) of \( QK^T \) is the dot product \( q_i \cdot k_j \), which measures the raw similarity/affinity score between query token \( i \) and key token \( j \). True. **Answer: A, C, and D.** A: **Correct:** Multiplying \( Q \in \mathbb{R}^{T \times d_k} \) by \( K^T \in \mathbb{R}^{d_k \times T} \) yields a \( T \times T \) matrix. B: **Incorrect:** Softmax is applied row-wise (along the key dimension for each query), not globally over all \( T^2 \) entries. C: **Correct:** Multiplying the \( T \times T \) attention weights by \( V \in \mathbb{R}^{T \times d_k} \) produces \( Z \in \mathbb{R}^{T \times d_k} \). D: **Correct:** Each entry \( (QK^T)_{ij} = q_i \cdot k_j \) is the raw attention compatibility score between token \( i \) and token \( j \).

**1. Recall how Cross-Attention (Encoder-Decoder Attention) works:** - In the cross-attention sub-layer of a Transformer decoder, the **Queries (\( Q \))** are projected from the previous sub-layer's hidden states in the **Decoder**. - The **Keys (\( K \))** and **Values (\( V \))** are projected from the final contextualized output representations of the **Encoder**. **2. Evaluate each statement:** - **Option A:** False, because \( K \) and \( V \) come from the Encoder, not the Decoder. - **Option B:** True, \( Q \) comes from the Decoder's hidden states. - **Option C:** False, \( Q, K, V \) all come from the same sequence only in *self-attention*, not in *cross-attention*. - **Option D:** True, cross-attention lets each decoding step attend to the relevant tokens of the encoded source sequence. **Answer: B and D.** A: **Incorrect:** In cross-attention, Keys (\( K \)) and Values (\( V \)) come from the Encoder output, not the Decoder. B: **Correct:** Queries (\( Q \)) are projected from the hidden states of the previous Decoder sub-layer. C: **Incorrect:** Having \( Q, K, V \) originate from the same sequence describes self-attention, not cross-attention. D: **Correct:** Cross-attention allows the Decoder to retrieve relevant information from the Encoder's representation of the input sequence.

**1. Identify the given parameters:** We have: - Model dimension: \( d_{\text{model}} = 128 \) - Token position (4th word, 0-based): \( \text{pos} = 3 \) - Embedding element index (6th element, 0-based): \( \text{index} = 5 \) **2. Match the index to the sinusoidal positional encoding formula:** The positional encoding formulas are: \[ PE_{(\text{pos}, 2i)} = \sin\!\left(\frac{\text{pos}}{10000^{2i / d_{\text{model}}}}\right), \quad PE_{(\text{pos}, 2i+1)} = \cos\!\left(\frac{\text{pos}}{10000^{2i / d_{\text{model}}}}\right) \] Since the index \( 5 \) is odd, we write \( 2i + 1 = 5 \implies 2i = 4 \) (so \( i = 2 \)), and use the cosine formula: \[ PE_{(3, 5)} = \cos\!\left(\frac{3}{10000^{4 / 128}}\right) = \cos\!\left(\frac{3}{10000^{1/32}}\right) \] **3. Calculate the numerical value (in radians):** Since \( 10000^{1/32} = (10^4)^{1/32} = 10^{1/8} = 10^{0.125} \approx 1.3335214 \): \[ \theta = \frac{3}{1.3335214} \approx 2.2496827 \text{ radians} \] \[ PE_{(3, 5)} = \cos(2.2496827) \approx -0.6279 \approx -0.63 \] **Answer: -0.63.**

**1. Set up the Bahdanau (additive) attention score formula:** We are given: \[ e_{tj} = \text{score}(s_{t-1}, h_j) = V_{\text{att}}^T \tanh(U_{\text{att}} s_{t-1} + W_{\text{att}} h_j) \] with \( s_{t-1} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} \), \( h_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \), \( V_{\text{att}} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \), and \( U_{\text{att}} = W_{\text{att}} = I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \). **2. Compute \( U_{\text{att}} s_{t-1} + W_{\text{att}} h_1 \):** \[ U_{\text{att}} s_{t-1} + W_{\text{att}} h_1 = \begin{bmatrix} 1 \\ 2 \end{bmatrix} + \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 2 \end{bmatrix} \] **3. Apply \( \tanh \) and multiply by \( V_{\text{att}}^T \):** Using \( \tanh(2) = \frac{e^2 - e^{-2}}{e^2 + e^{-2}} \approx 0.96402758 \): \[ e_{t1} = \begin{bmatrix} 1 & 1 \end{bmatrix} \begin{bmatrix} \tanh(2) \\ \tanh(2) \end{bmatrix} = 2 \tanh(2) \approx 2 \times 0.96402758 = 1.928055 \approx 1.93 \] **Answer: 1.93.**

**1. Compute the raw attention scores \( e_{t1}, e_{t2}, e_{t3} \):** Since \( U_{\text{att}} = W_{\text{att}} = I_2 \) and \( V_{\text{att}} = [1, 1]^T \), for each encoder state \( h_j \) we have \( e_{tj} = \text{sum}(\tanh(s_{t-1} + h_j)) \): - For \( h_1 = [1, 0]^T \): \( s_{t-1} + h_1 = [2, 2]^T \implies e_{t1} = \tanh(2) + \tanh(2) \approx 0.964028 + 0.964028 = 1.928055 \) - For \( h_2 = [0, 1]^T \): \( s_{t-1} + h_2 = [1, 3]^T \implies e_{t2} = \tanh(1) + \tanh(3) \approx 0.761594 + 0.995055 = 1.756649 \) - For \( h_3 = [1, 1]^T \): \( s_{t-1} + h_3 = [2, 3]^T \implies e_{t3} = \tanh(2) + \tanh(3) \approx 0.964028 + 0.995055 = 1.959082 \) **2. Apply the softmax function to find \( \alpha_{t1} \):** - \( \exp(e_{t1}) = \exp(1.928055) \approx 6.87613 \) - \( \exp(e_{t2}) = \exp(1.756649) \approx 5.79299 \) - \( \exp(e_{t3}) = \exp(1.959082) \approx 7.09281 \) - Sum of exponentials: \( 6.87613 + 5.79299 + 7.09281 = 19.76193 \) \[ \alpha_{t1} = \frac{6.87613}{19.76193} \approx 0.34795 \approx 0.35 \] **Answer: 0.35.**

**1. Compute all three attention weights \( \alpha_{t1}, \alpha_{t2}, \alpha_{t3} \):** From the softmax normalization in the previous step: - \( \alpha_{t1} \approx \frac{6.87613}{19.76193} \approx 0.34795 \) - \( \alpha_{t2} \approx \frac{5.79299}{19.76193} \approx 0.29314 \) - \( \alpha_{t3} \approx \frac{7.09281}{19.76193} \approx 0.35891 \) **2. Compute the context vector \( c_t = \sum_{j=1}^3 \alpha_{tj} h_j \):** \[ c_t = \alpha_{t1} \begin{bmatrix} 1 \\ 0 \end{bmatrix} + \alpha_{t2} \begin{bmatrix} 0 \\ 1 \end{bmatrix} + \alpha_{t3} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} \alpha_{t1} + \alpha_{t3} \\ \alpha_{t2} + \alpha_{t3} \end{bmatrix} \] **3. Sum the elements of \( c_t \):** Since \( \alpha_{t1} + \alpha_{t2} + \alpha_{t3} = 1 \), the sum of the two components of \( c_t \) is: \[ (\alpha_{t1} + \alpha_{t3}) + (\alpha_{t2} + \alpha_{t3}) = (\alpha_{t1} + \alpha_{t2} + \alpha_{t3}) + \alpha_{t3} = 1 + \alpha_{t3} \approx 1 + 0.35891 = 1.35891 \approx 1.36 \] **Answer: 1.36.**

**1. Set up the token embedding matrix dimensions:** A token embedding matrix maps each token in the vocabulary of size \( V = 40,000 \) to a dense vector of dimension \( d_{\text{model}} = 768 \). **2. Calculate the number of parameters:** \[ \text{Parameters} = V \times d_{\text{model}} = 40,000 \times 768 = 30,720,000 \] **Answer: A — 30,720,000.** A: **Correct:** Multiplying vocabulary size \( 40,000 \) by embedding dimension \( 768 \) gives \( 30,720,000 \). B: **Incorrect:** \( 40,768 = 40,000 + 768 \) adds the dimensions instead of multiplying them. C: **Incorrect:** \( 393,216 = 512 \times 768 \) is the number of parameters in the positional embedding matrix, not the token embedding matrix. D: **Incorrect:** \( 15,728,640,000 = 40,000 \times 768 \times 512 \) unnecessarily multiplies by the maximum sequence length.

**1. Set up the learned positional embedding matrix dimensions:** In GPT, each position up to the maximum sequence length \( L_{\max} = 512 \) has a learned embedding vector of dimension \( d_{\text{model}} = 768 \). **2. Calculate the number of parameters:** \[ \text{Parameters} = L_{\max} \times d_{\text{model}} = 512 \times 768 = 393,216 \] **Answer: B — 393,216.** A: **Incorrect:** Concatenates `512` and `768` as digits rather than multiplying them. B: **Correct:** Multiplying maximum sequence length \( 512 \) by \( d_{\text{model}} = 768 \) equals \( 393,216 \). C: **Incorrect:** Off by a factor of 10 (\( 3,932,160 = 10 \times 393,216 \)). D: **Incorrect:** Off by a factor of 10 (\( 39,321 \)).

**1. Find the per-head dimension \( d_k \):** With \( d_{\text{model}} = 768 \) split equally across \( h = 12 \) attention heads: \[ d_k = d_v = \frac{d_{\text{model}}}{h} = \frac{768}{12} = 64 \] **2. Count the parameters for Q, K, and V in ONE head:** Each projection matrix (\( W_Q^{(i)}, W_K^{(i)}, W_V^{(i)} \)) has shape \( d_{\text{model}} \times d_k = 768 \times 64 \): \[ \text{One projection matrix} = 768 \times 64 = 49,152 \] Combining all three projections (Q + K + V) for one head gives: \[ 3 \times 49,152 = 147,456 \] **Answer: D — 147,456.** A: **Incorrect:** \( 49,152 = 768 \times 64 \) is the parameter count for only a single projection (e.g., just \( W_Q^{(i)} \)), not Q + K + V combined. B: **Incorrect:** \( 196,608 = 4 \times 49,152 \) includes a fourth matrix of size \( 768 \times 64 \). C: **Incorrect:** \( 589,824 = 768 \times 768 \) is the size of a single projection across all 12 heads combined. D: **Correct:** Each head has \( d_k = 768 / 12 = 64 \), so \( Q + K + V \) for one head has \( 3 \times (768 \times 64) = 147,456 \) parameters.

**1. Determine the shape of the output projection matrix \( W_O \):** Concatenating the outputs of \( h = 12 \) heads, each of dimension \( d_v = 64 \), yields a vector of dimension \( 12 \times 64 = 768 = d_{\text{model}} \). The output projection matrix \( W_O \) maps this \( 768 \)-dimensional concatenated vector back to \( d_{\text{model}} = 768 \), so \( W_O \in \mathbb{R}^{768 \times 768} \). **2. Calculate the number of weight parameters:** \[ \text{Parameters in } W_O = 768 \times 768 = 589,824 \] **Answer: C — 589,824.** A: **Incorrect:** \( 768 \) is the dimension of the bias vector, not the \( 768 \times 768 \) weight matrix. B: **Incorrect:** \( 49,152 = 64 \times 768 \) only projects a single head rather than all 12 concatenated heads. C: **Correct:** The output projection weight matrix \( W_O \) has shape \( 768 \times 768 = 589,824 \) parameters. D: **Incorrect:** \( 294,912 \) is half of \( 768 \times 768 \).

**1. Identify the layers in a Transformer Feed-Forward Network (FFN):** A standard FFN consists of two linear transformations with a GELU activation in between: 1. First linear layer: maps \( d_{\text{model}} = 768 \to d_{\text{ff}} = 3,072 \) - Weights \( W_1 \): \( 768 \times 3,072 = 2,359,296 \) - Biases \( b_1 \): \( 3,072 \) 2. Second linear layer: maps \( d_{\text{ff}} = 3,072 \to d_{\text{model}} = 768 \) - Weights \( W_2 \): \( 3,072 \times 768 = 2,359,296 \) - Biases \( b_2 \): \( 768 \) **2. Sum all weight and bias parameters:** \[ \text{Total FFN parameters} = 2,359,296 + 3,072 + 2,359,296 + 768 = 4,718,592 + 3,840 = 4,722,432 \] **Answer: C — 4,722,432.** A: **Incorrect:** \( 2,359,296 = 768 \times 3,072 \) is the weight count of only one of the two linear layers (without biases). B: **Incorrect:** \( 4,718,592 = 2 \times 768 \times 3,072 \) counts both weight matrices but omits thetwo bias vectors (\( 3,072 + 768 = 3,840 \)). C: **Correct:** Summing \( W_1 (2,359,296) + b_1 (3,072) + W_2 (2,359,296) + b_2 (768) = 4,722,432 \). D: **Incorrect:** Roughly double the actual FFN parameter count.