Introduction to Deep Learning and Generative AI · End Term · Sep 2025 · FN

**1. Choose a Loss Function Robust to Outliers with Smooth Convergence:** - **Mean Squared Error (MSE)** and **Sum of Squared Errors (SSE)** square the residuals \( (y - \hat{y})^2 \), which heavily magnifies the loss and gradients of extreme outliers (such as rare luxury mansions). - **Binary Cross-Entropy Loss** is used for binary classification, not continuous house price regression. - **Huber Loss** behaves quadratically for small residuals (\( |e| \le \delta \)), ensuring smooth gradients and stable convergence near the optimum, and linearly for large residuals (\( |e| > \delta \)), preventing extreme outliers from dominating training. **Answer: Huber Loss (Option C).** A: **Incorrect:** MSE squares large errors, making the regression model highly sensitive to extreme price outliers. B: **Incorrect:** SSE also squares residuals and is equally sensitive to extreme outliers. C: **Correct:** Huber Loss combines quadratic behavior for small errors (smooth convergence) with linear behavior for large errors (robustness to outliers). D: **Incorrect:** Binary Cross-Entropy is a classification loss function for probabilities in \( [0, 1] \), not for continuous regression.

**1. Compute the First Hidden Layer Output:** - Input vector: \( x = [-6, 2, 1, -3, 2]^T \in \mathbb{R}^5 \). - All weights are initialized to \( 1 \) and biases are \( 0 \). - For each neuron in the 1st hidden layer, the pre-activation is the sum of the inputs: \[ z^{(1)}_j = \sum_{i=1}^5 1 \cdot x_i = -6 + 2 + 1 - 3 + 2 = -4 \] - Applying the ReLU activation function \( \text{ReLU}(z) = \max(0, z) \): \[ a^{(1)}_j = \max(0, -4) = 0 \quad \text{for all } 10 \text{ neurons in hidden layer 1} \] **2. Propagate Through Remaining Hidden Layers and Output Layer:** - Since \( a^{(1)} = \mathbf{0} \) and all biases are zero, every subsequent hidden layer \( l = 2, \dots, 80 \) receives \( z^{(l)} = 0 \) and outputs \( a^{(l)} = \text{ReLU}(0) = 0 \). - The output neuron receives pre-activation \( z_{\text{out}} = 0 \) and applies the **Sigmoid** activation function: \[ \hat{y} = \sigma(0) = \frac{1}{1 + e^0} = \frac{1}{2} = 0.5 \] **Answer: 0.5.**

**1. Evaluate Each Statement on Categorical Encoding:** - **Option A:** One-hot encoding produces **sparse**, high-dimensional binary vectors with no semantic similarity information (**False**). - **Option B:** Embedding layers contain **learnable parameters** trained via backpropagation rather than fixed manually-defined vectors (**False**). - **Option C:** For any two distinct categories \( i \neq j \), their one-hot vectors \( e_i \) and \( e_j \) have `1`s at different indices, so their dot product is \( e_i \cdot e_j = 0 \) (**True**). - **Option D:** The Euclidean distance between two distinct one-hot vectors \( e_i \) and \( e_j \) is \( \sqrt{(1 - 0)^2 + (0 - 1)^2} = \sqrt{2} \approx 1.414 \), not \( 1 \) (**False**). - **Option E:** Embedding layers learn dense lower-dimensional vectors where semantically similar categories are mapped to nearby vectors (**True**). **Answer: Options C and E.** A: **Incorrect:** One-hot vectors are sparse and orthogonal, capturing zero semantic relationship between categories. B: **Incorrect:** Embedding layer weights are learned automatically during training, not manually fixed. C: **Correct:** Distinct one-hot vectors have non-overlapping non-zero entries, so their dot product is always 0. D: **Incorrect:** The Euclidean distance between two distinct one-hot vectors is \( \sqrt{1^2 + (-1)^2} = \sqrt{2} \), not 1. E: **Correct:** Learned embedding layers map categories into a dense continuous space where related concepts have similar vector representations.

**1. Perform the Forward Pass Through the Network:** - Given: input \( x = 1 \), input-to-hidden weight \( w_1 = 2 \), hidden-to-output weight \( w_2 = 3 \), and zero biases. - **Hidden layer (ReLU activation):** \[ h = \text{ReLU}(w_1 \cdot x) = \max(0, 2 \times 1) = 2 \] - **Output layer (linear activation):** \[ \hat{y} = w_2 \cdot h = 3 \times 2 = 6 \] **Answer: 6.**

**1. Compute the Gradient \( \frac{\partial L}{\partial w_2} \) Using the Chain Rule:** - From Question 5, \( h = 2 \), \( \hat{y} = w_2 h = 6 \), ground truth \( y = 5 \), and loss \( L = \frac{1}{2}(\hat{y} - y)^2 \). - Applying the chain rule: \[ \frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_2} = (\hat{y} - y) \cdot h \] - Substituting \( \hat{y} = 6 \), \( y = 5 \), and \( h = 2 \): \[ \frac{\partial L}{\partial w_2} = (6 - 5) \times 2 = 1 \times 2 = 2 \] **Answer: 2.**

**1. Differentiate \( z = (x + y)^2 \) with \( y = e^x \) Using the Chain Rule:** - By the total derivative / chain rule: \[ \frac{dz}{dx} = \frac{\partial z}{\partial x} + \frac{\partial z}{\partial y} \frac{dy}{dx} = 2(x + y) \cdot \frac{d}{dx}(x + y) \] - Since \( y = e^x \), we have \( \frac{dy}{dx} = e^x \), so \( \frac{d}{dx}(x + y) = 1 + e^x \). - Therefore: \[ \frac{dz}{dx} = 2(x + y)(1 + e^x) \] **Answer: \( 2(x + y)(1 + e^x) \) (Option B).** A: **Incorrect:** The derivative of \( y = e^x \) with respect to \( x \) is \( e^x \), not \( x \). B: **Correct:** By the chain rule, \( \frac{dz}{dx} = 2(x + y)\left(1 + \frac{dy}{dx}\right) = 2(x + y)(1 + e^x) \). C: **Incorrect:** This forgets the derivative of the \( x \) term inside \( (x + y) \), which contributes \( +1 \). D: **Incorrect:** \( \frac{d}{dx}(e^x) = e^x \), not \( e^{2x} \).

**1. Analyze the Exponential Moving Average Velocity Update \( v_t = \alpha v_{t-1} + (1 - \alpha) \nabla L_t \):** - **Option A (`\alpha = 0.9`):** Retains \( 90\% \) of the previous velocity \( v_{t-1} \) and \( 10\% \) of the current gradient, making optimization smoother across steps but slower to change direction (**True**). - **Option B:** `\alpha = 0.9` retains (does not discard) \( 90\% \) of previous momentum (**False**). - **Option C (`\alpha = 0.1`):** Places \( 90\% \) weight on the current gradient \( \nabla L_t \) and only \( 10\% \) on past velocity, so it responds rapidly to gradient changes but is more susceptible to noisy gradients (**True**). - **Option D:** `\alpha = 0.1` provides very little smoothing (**False**). - **Option E:** Increasing \( \alpha \) from \( 0.5 \) to \( 0.95 \) increases the decay horizon \( \frac{1}{1 - \alpha} \), making the optimizer 'remember' more past gradients (**True**). - **Option F:** Setting \( \alpha = 0 \) gives \( v_t = \nabla L_t \), which reduces to vanilla SGD with **no** momentum (**False**). **Answer: Options A, C, and E.** A: **Correct:** With \( \alpha = 0.9 \), 90% of the previous velocity is preserved at each step, smoothing trajectory updates. B: **Incorrect:** \( \alpha = 0.9 \) keeps 90% of previous momentum and weights the new gradient by \( 1 - \alpha = 0.1 \). C: **Correct:** With \( \alpha = 0.1 \), 90% weight is given to the instantaneous gradient \( \nabla L_t \), reacting quickly but with higher sensitivity to noise. D: **Incorrect:** Heavy smoothing requires a large \( \alpha \) close to 1 (such as 0.9 or 0.99), not 0.1. E: **Correct:** A larger \( \alpha \) (0.95 vs 0.5) decays past gradients more slowly, extending the effective memory of past gradients. F: **Incorrect:** Setting \( \alpha = 0 \) eliminates the momentum term completely, reducing to plain SGD without momentum.

**1. Inspect the Original `EfficientNet-B0` Classifier Layer:** - As shown in the code comment, the original final layer is: `Linear(in_features=1280, out_features=1000, bias=True)` - This means the feature extractor outputs a **1280-dimensional** feature vector into `model.classifier[1]`, which then maps those 1280 features to 1000 ImageNet classes. **2. Identify the Error in `model.classifier[1] = nn.Linear(1000, 3)`:** - By replacing `model.classifier[1]` with `nn.Linear(1000, 3)`, the layer expects an input tensor of size `1000` instead of `1280`, which will cause a shape mismatch error during the forward pass. - It must be `nn.Linear(1280, 3)`. **Answer: The new nn.Linear layer's in_features is incorrect; it should be 1280 to match the features coming from the rest of the network, not 1000 (Option C).** A: **Incorrect:** Passing a 1280-dimensional feature vector into `nn.Linear(1000, 3)` raises a matrix multiplication shape mismatch `RuntimeError`. B: **Incorrect:** Replacing just `model.classifier[1]` is valid in PyTorch as long as `in_features=1280` matches. C: **Correct:** Because `model.classifier[1]` replaces the original `Linear(1280, 1000)` layer, its `in_features` must be `1280`, not `1000`. D: **Incorrect:** Keeping the dropout layer at `model.classifier[0]` is standard and valid.

**1. Order of Operations in `torchvision.transforms.Compose`:** - `Image.open("food.jpg")` returns a **PIL Image** object. - In `torchvision.transforms`, `T.Normalize(mean, std)` operates **only on PyTorch Tensors** (specifically float tensors scaled to `[0, 1]` produced by `T.ToTensor()`), not on PIL Images. - Because `T.Normalize(...)` is placed **before** `T.ToTensor()`, calling `tr(img)` passes a PIL Image into `T.Normalize`, which raises a `TypeError` (`img should be Tensor Image. Got <class 'PIL.Image.Image'>`). **Answer: The Normalize transform is applied before ToTensor, but it expects a tensor with values in the [0, 1] range as input (Option B).** A: **Incorrect:** Using ImageNet normalization statistics is standard practice when using ImageNet pre-trained models. B: **Correct:** `T.ToTensor()` must come before `T.Normalize()` because `Normalize` requires a float Tensor in `[0, 1]`, not a PIL Image. C: **Incorrect:** `T.ToTensor()` takes a PIL Image or NumPy `ndarray` as input, not a file path string. D: **Incorrect:** Calling `T.Normalize` on a PIL Image raises a `TypeError`.

**1. How an Inception Module Combines Parallel Branches:** - In an Inception module, all parallel branches preserve the same spatial height and width \( (H, W) \), and their output feature maps are **concatenated along the channel dimension**. - Given the 4 branches output \( 64, 128, 32, \) and \( 32 \) channels respectively: \[ \text{Total output channels} = 64 + 128 + 32 + 32 = 256 \] **Answer: 256.**

**1. Classify the Critical Point \( (0, 0) \) of \( f(x, y) = x^2 - y^2 \):** - **Gradient:** \( \nabla f(x, y) = (2x, -2y)^T \), which equals \( (0, 0)^T \) at \( (0, 0) \), so \( (0, 0) \) is a stationary point. - **Hessian Matrix:** \[ H = \begin{pmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{pmatrix} = \begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix} \] - Since the eigenvalues of \( H \) are \( \lambda_1 = 2 > 0 \) and \( \lambda_2 = -2 < 0 \) (or \( \det(H) = -4 < 0 \)), the function curves upward along the \( x \)-axis and downward along the \( y \)-axis. - Therefore, \( (0, 0) \) is a **saddle point**. **Answer: Saddle point (Option C).** A: **Incorrect:** Along the \( y \)-axis, \( f(0, y) = -y^2 < 0 = f(0, 0) \), so \( (0, 0) \) is not a local minimum. B: **Incorrect:** Along the \( x \)-axis, \( f(x, 0) = x^2 > 0 = f(0, 0) \), so \( (0, 0) \) is not a local maximum. C: **Correct:** Because the Hessian \( \begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix} \) has one positive and one negative eigenvalue (\( \det(H) = -4 < 0 \)), \( (0, 0) \) is a saddle point. D: **Incorrect:** Specifically, a stationary point where the Hessian is indefinite is classified as a saddle point.

**1. Desired Direction (High vs. Low) for Each Generative Evaluation Metric:** - **Perplexity (Language Modeling):** Measures uncertainty in predicting the next token (\( e^{\text{Cross-Entropy}} \)); **Low** is better. - **BLEU Score (Machine Translation / Text Similarity):** Measures \( n \)-gram precision overlap with reference text; **High** is better. - **FID Score (Fréchet Inception Distance for Image Quality):** Measures distribution distance between real and generated image features; **Low** is better. - **Inception Score (IS):** Measures clarity and diversity of generated images via KL divergence; **High** is better. - **CLIP Similarity (Text-Image Alignment):** Measures cosine similarity between text prompt and generated image embeddings; **High** is better. **Answer: Perplexity: Low, BLEU: High, FID: Low, IS: High, CLIP: High (Option A).** A: **Correct:** Good generative models minimize Perplexity and FID (distance/uncertainty metrics) while maximizing BLEU, IS, and CLIP similarity (overlap/quality/alignment scores). B: **Incorrect:** Perplexity and FID should be low, and IS should be high. C: **Incorrect:** BLEU, IS, and CLIP should be high, and FID should be low. D: **Incorrect:** Perplexity should be low, while BLEU and CLIP should be high.

**1. Calculate Unclipped and Clipped 1-Gram Counts:** - **Candidate sentence:** `"the the the cat"` (total length \( C = 4 \) words). - Candidate 1-gram counts: `"the"`: 3, `"cat"`: 1. - **Reference sentence:** `"the cat is on the mat"`. - Reference 1-gram counts: `"the"`: 2, `"cat"`: 1, `"is"`: 1, `"on"`: 1, `"mat"`: 1. **2. Compute Clipped 1-Gram Precision (\( p_1 \)):** - Clip each candidate word's count by its maximum occurrence count in the reference: - Clipped count for `"the"` = \( \min(3, 2) = 2 \). - Clipped count for `"cat"` = \( \min(1, 1) = 1 \). - Total clipped matches = \( 2 + 1 = 3 \). - Divide by total candidate words (\( 4 \)): \[ p_1 = \frac{3}{4} = 0.75 \] **Answer: 0.75 (Option B).** A: **Incorrect:** 1.0 (\( 4/4 \)) is the unclipped precision, which fails to clip the count of `"the"` to its reference count of 2. B: **Correct:** Clipping `"the"` to 2 and `"cat"` to 1 gives \( \frac{2 + 1}{4} = \frac{3}{4} = 0.75 \). C: **Incorrect:** \( 0.5 = 2/4 \) counts only unique word types instead of clipped token counts. D: **Incorrect:** \( 0.6 = 3/5 \) divides by the wrong denominator; precision divides by candidate length (4).

**1. Source of Queries (Q), Keys (K), and Values (V) in Transformer Cross-Attention:** - In an Encoder-Decoder Transformer's **cross-attention** layer, the Decoder attends to the Encoder's output representations: - **Queries (\( Q \)):** Come from the **Decoder** (the target sequence positions asking for relevant source context). - **Keys (\( K \)):** Come from the **Encoder** (the source sequence positions being matched against the queries). - **Values (\( V \)):** Come from the **Encoder** (the source sequence content aggregated by the attention weights). **Answer: | Q | Decoder | | K | Encoder | | V | Encoder | (Option A).** A: **Correct:** In cross-attention, Queries (`Q`) originate from the Decoder while Keys (`K`) and Values (`V`) originate from the Encoder. B: **Incorrect:** This reverses the roles of Encoder and Decoder. C: **Incorrect:** Keys (`K`) and Values (`V`) must come from the same sequence (the Encoder). D: **Incorrect:** Queries (`Q`) come from the Decoder, and Values (`V`) come from the Encoder.

**1. Evaluate Claim 1 and Claim 2 About RNN Encoder-Decoder Architectures:** - **Claim 1 is FALSE:** The encoder and decoder do **not** need to have the same number of layers or the same hidden size (if their hidden sizes differ, a simple linear projection layer maps the encoder's final state to the decoder's initial state). - **Claim 2 is FALSE:** Because the encoder processes the source language (English) and the decoder generates the target language (Hindi), they have completely separate vocabularies, grammatical structures, and **independent weight matrices**. **Answer: Both claims are FALSE - encoder and decoder can have different architectures (different hidden sizes, layers) and use separate, independent weights (Option D).** A: **Incorrect:** Neither identical architecture nor weight sharing is required between encoder and decoder. B: **Incorrect:** Claim 1 is also false; encoder and decoder can have different numbers of layers and hidden dimensions. C: **Incorrect:** Claim 2 is false; encoder and decoder use separate, independent weights. D: **Correct:** Both claims are false because the encoder and decoder are separate networks that can have different architectures and learn independent weights.

**1. Compute the Number of Parameters in an Embedding Lookup Table:** - An embedding layer (`nn.Embedding(num_embeddings=200, embedding_dim=5)`) stores a lookup matrix \( W \in \mathbb{R}^{200 \times 5} \), with one 5-dimensional vector for each of the 200 words in the vocabulary. - Total learnable parameters: \[ 200 \times 5 = 1000 \] **Answer: 1000 (Option C).** A: **Incorrect:** 200 is only the vocabulary size (number of rows). B: **Incorrect:** 5 is only the embedding dimension per word (number of columns). C: **Correct:** Each of the 200 words has 5 learnable embedding components, giving \( 200 \times 5 = 1000 \) parameters. D: **Incorrect:** The number of parameters is the product \( 200 \times 5 = 1000 \), not the sum \( 200 + 5 = 205 \).

**1. Understand Indexing in Sinusoidal Positional Encoding:** - The Transformer positional encoding formulas define **two** coordinates for each index \( i \): - Even coordinate \( 2i \): \( PE(p, 2i) = \sin\left(\frac{p}{10000^{2i/d}}\right) \) - Odd coordinate \( 2i + 1 \): \( PE(p, 2i + 1) = \cos\left(\frac{p}{10000^{2i/d}}\right) \) - Because each value of \( i \) generates a **pair** of dimensions (one sine at \( 2i \) and one cosine at \( 2i + 1 \)), \( i \) only needs to run from \( 0 \) to \( \frac{d}{2} - 1 \) to produce \( d \) total dimensions. - Running \( i \) from \( 0 \) to \( d - 1 \) would generate \( 2d \) dimensions instead of \( d \). **Answer: Because each i generates two dimensions (one sine and one cosine), so using (i = 0 to d−1) would produce 2d dimensions instead of d (Option B).** A: **Incorrect:** \( i \) takes integer values \( 0, 1, \dots, d/2 - 1 \); \( 2i \) is always even and \( 2i+1 \) is always odd. B: **Correct:** Each index \( i \) produces two vector entries (\( 2i \) and \( 2i+1 \)), so \( i \in [0, d-1] \) would produce \( 2d \) entries instead of \( d \). C: **Incorrect:** The upper bound on \( i \) is \( d/2 - 1 \), not \( \sqrt{d} \). D: **Incorrect:** Cosine terms occupy the odd indices \( 2i+1 \) alongside the sine terms at even indices \( 2i \), which is why \( i \) ranges up to \( d/2 - 1 \).

**1. Understand the 'Add and Norm' Step in a Transformer Block:** - In the standard Post-LN Transformer architecture (`Attention Is All You Need`), each sub-layer (Multi-Head Attention and Feed-Forward Network) is followed by a **residual connection ('Add')** and **Layer Normalization ('Norm')**: \[ x = \text{LayerNorm}(x + \text{SubLayer}(x)) \] - Notice that in Sub-layer 2 of the given pseudo-code, this exact pattern is shown as `output = self.layer_norm(x + ffn_output)`. - Therefore, blank (A) after Sub-layer 1 must be: `self.layer_norm(x + attention_output)` **Answer: self.layer_norm(x + attention_output) (Option B).** A: **Incorrect:** This omits the residual connection (`x + ...`), losing the direct gradient highway. B: **Correct:** Adds the residual input `x` to `attention_output` and applies `self.layer_norm(...)`, matching the 'Add and Norm' architecture. C: **Incorrect:** Normalizing before adding (`x + self.layer_norm(...)`) does not match the `self.layer_norm(x + ...)` pattern used in the block. D: **Incorrect:** Normalizing only `x` before adding leaves `attention_output` unnormalized.

**1. Apply the Causal (Look-Ahead) Mask at Position 1:** - The raw attention scores for Position 1 (the second token) across Positions 0, 1, 2, 3 are: \[ \text{Scores}_{\text{row } 1} = [3.0,\; 3.0,\; 9.0,\; 5.0] \] - Under the **Causal (Look-Ahead) Mask**, Position 1 can only attend to itself and earlier positions (**Position 0 and Position 1**); future positions (**Position 2 and Position 3**) are masked out by setting their scores to \( -\infty \): \[ \text{Masked Scores}_{\text{row } 1} = [3.0,\; 3.0,\; -\infty,\; -\infty] \] **2. Calculate the Softmax Probability for Attending to Position 0:** - Since \( e^{-\infty} = 0 \), the softmax probability that Position 1 attends to Position 0 is: \[ P(\text{Pos } 0) = \frac{e^{3.0}}{e^{3.0} + e^{3.0} + 0 + 0} = \frac{e^{3.0}}{2 e^{3.0}} = 0.5 \] **Answer: 0.5.**

**1. Apply the Closed-Form Diffusion Forward Noising Formula:** - In Denoising Diffusion Probabilistic Models (DDPM), the forward process at step \( t \) is given by: \[ x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon \] - Given \( x_0 = 100 \), \( \bar{\alpha}_t = 0.64 \), and \( \epsilon = 2 \): - \( \sqrt{\bar{\alpha}_t} = \sqrt{0.64} = 0.8 \) - \( \sqrt{1 - \bar{\alpha}_t} = \sqrt{1 - 0.64} = \sqrt{0.36} = 0.6 \) **2. Compute \( x_t \):** \[ x_t = (0.8 \times 100) + (0.6 \times 2) = 80 + 1.2 = 81.2 \] **Answer: 81.2 (Option A).** A: **Correct:** \( x_t = \sqrt{0.64}(100) + \sqrt{1 - 0.64}(2) = 80 + 1.2 = 81.2 \). B: **Incorrect:** 75.3 does not follow the DDPM forward noising equation. C: **Incorrect:** \( 68.1 \) uses un-square-rooted \( \bar{\alpha}_t = 0.64 \) instead of \( \sqrt{\bar{\alpha}_t} = 0.8 \). D: **Incorrect:** 40.9 swaps \( \bar{\alpha}_t \) and \( 1 - \bar{\alpha}_t \).

**1. Understand `torch.tril(x)` in PyTorch:** - `torch.tril(x)` returns the **lower triangular part** of the matrix `x` (keeping elements on and below the main diagonal \( i \ge j \)) and sets all elements strictly above the main diagonal (\( i < j \)) to `0`. - Given: \[ x = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix} \] - Zeroing out the entries above the main diagonal (`2`, `3`, and `6`) gives: \[ y = \begin{bmatrix} 1 & 0 & 0 \\ 4 & 5 & 0 \\ 7 & 8 & 9 \end{bmatrix} \] **Answer: \( \begin{bmatrix} 1 & 0 & 0 \\ 4 & 5 & 0 \\ 7 & 8 & 9 \end{bmatrix} \) (Option B).** A: **Incorrect:** This is the original matrix `x` without any triangular masking. B: **Correct:** `torch.tril(x)` retains the lower triangle (on and below the main diagonal) and zeroes out the upper-right entries. C: **Incorrect:** This zeroes out the first column instead of the upper triangle. D: **Incorrect:** This is the upper triangular matrix (`torch.triu(x)`), not the lower triangular matrix (`torch.tril(x)`).

**1. Purpose of `chunk_overlap` in Text Splitting for RAG:** - When splitting long documents into fixed-size chunks (e.g., `chunk_size=1000`), a sentence or concept at the boundary of a chunk might get cut in half. - Setting `chunk_overlap=200` repeats the last 200 characters of each chunk at the beginning of the next chunk, which **ensures continuity of context and information between adjacent chunks**. **Answer: Ensures continuity of information between chunks (Option B).** A: **Incorrect:** `chunk_overlap` intentionally introduces shared overlapping text between consecutive chunks rather than preventing repetition. B: **Correct:** Overlapping adjacent chunks preserves semantic context across chunk boundaries so sentences/ideas are not abruptly severed. C: **Incorrect:** Increasing overlap actually *increases* the total number of chunks needed to cover the document. D: **Incorrect:** `chunk_overlap` controls sliding window stride, not token filtering.

**1. Understand `vector_store.as_retriever(search_kwargs={"k": 4})`:** - In LangChain, calling `.as_retriever(search_kwargs={"k": 4})` configures a vector store retriever using similarity search (default) where `k=4` specifies the number of nearest neighbor documents to retrieve. - Therefore, for any input query, it **returns the top-4 most similar chunks** from the vector store. **Answer: A retriever that returns the top-4 most similar chunks (Option D).** A: **Incorrect:** Vector store retrieval ranks chunks by embedding similarity, not random selection. B: **Incorrect:** `"k": 4` controls the number of top results returned (`top-k`), not a metadata filter. C: **Incorrect:** The retriever searches across all indexed chunks in the vector store and returns the top 4 matches. D: **Correct:** `search_kwargs={"k": 4}` instructs the similarity search retriever to return the 4 chunks closest to the query embedding.

**1. Identify Dimensions for Token and Positional Embeddings in a Transformer:** - **`self.token_embedding = nn.Embedding(a, c)`:** Maps each token ID in the vocabulary to an embedding vector, so `a = vocab_size` and `c = embedding_dimension`. - **`self.position_embedding = nn.Embedding(b, c)`:** Maps each position index \( 0, 1, \dots, T-1 \) in the context window to an embedding vector of the **exact same dimension `c`** (so they can be added together: `token_emb + pos_emb`), so `b = context_length` and `c = embedding_dimension`. - Thus, `a, b, c` are **`vocab_size, context_length, embedding_dimension`**. **Answer: vocab_size,context_length,embedding_dimension (Option D).** A: **Incorrect:** Swaps `vocab_size` and `context_length`; `token_embedding` must be indexed by token ID (`vocab_size`). B: **Incorrect:** `position_embedding` is indexed by sequence position (`context_length`), not `vocab_size`. C: **Incorrect:** `nn.Embedding(num_embeddings, embedding_dim)` takes the count of items first and the vector dimension (`embedding_dimension`) second. D: **Correct:** `token_embedding` has shape `(vocab_size, embedding_dimension)` and `position_embedding` has shape `(context_length, embedding_dimension)`.

**1. Count Semantic Search Comparisons in a Vector Store:** - In a RAG system, semantic search operates at the **chunk level**: each chunk is embedded into a single vector in the vector database. - Regardless of how many pages (100) or lines (10,000) the original document had, since it was segmented into **10 chunks**, there are exactly **10 chunk embeddings** stored in the index. - When a query enters the RAG system, its embedding is compared against each of the **10** chunk embeddings, performing **10** semantic search matches. **Answer: 10.**

**1. Trace the Python Loop Extracting PDF Text:** - `raw_text = ""` initializes an empty string. - Inside the loop over `pdf_reader.pages`, `page_text = page.extract_text()` extracts the text of each page as a string, and `raw_text += page_text` concatenates it directly onto `raw_text` without inserting any delimiter or page break. - Therefore, `raw_text` is **a single long string containing the text of all pages combined, with no separators added**. **Answer: A single long string containing the text of all pages combined, with no separators added (Option B).** A: **Incorrect:** `raw_text` is initialized as a string `""` and concatenated with `+=`, not a list `.append()`. B: **Correct:** String concatenation `raw_text += page_text` joins every page's extracted string end-to-end into one string with no added separators. C: **Incorrect:** No page break characters are added in `raw_text += page_text`. D: **Incorrect:** `page_num` is never used to populate a dictionary.

**1. Relationship Between Key, Query, and Value Projections in Self-Attention:** - In self-attention: - `self.key = nn.Linear(n_embeddings, head_size, bias=False)` - `self.query = nn.Linear(n_embeddings, head_size, bias=False)` - `self.value = nn.Linear(n_embeddings, head_size, bias=False)` - Each `nn.Linear` call instantiates a **separate, independent weight matrix** (\( W_K, W_Q, W_V \)). - During training, these three matrices learn **independent linear projections** of the same input representation to serve distinct roles (matching keys, asking queries, and providing values). **Answer: They are independent learned transformations of the input (Option B).** A: **Incorrect:** `W_K`, `W_Q`, and `W_V` are three distinct `nn.Linear` instances and do not share weights. B: **Correct:** Each of `key`, `query`, and `value` is a separate `nn.Linear` layer that learns an independent linear transformation of the input. C: **Incorrect:** Key and Query also have separate weight matrices so that attention scores can be asymmetric. D: **Incorrect:** `nn.Linear` performs a linear matrix multiplication, not dropout.

**1. Execution Order of LangChain Expression Language (LCEL) Pipelines:** - In LCEL, the pipe operator `|` chains steps **sequentially**, whereas a dictionary literal `{"context": retriever | format_docs, "input": RunnablePassthrough()}` is automatically wrapped into a **`RunnableParallel`**. - When the chain is invoked, all branches inside the dictionary execute **in parallel**: - Branch 1: `retriever | format_docs` (fetches and formats documents sequentially within the branch) - Branch 2: `RunnablePassthrough()` (passes the user's query through unchanged) - Once both parallel branches finish, their dictionary output is piped sequentially into `prompt_template`, then `llm`, and finally `StrOutputParser()`. **Answer: retriever | format_docs runs in parallel with RunnablePassthrough() (Option B).** A: **Incorrect:** `retriever | format_docs` uses the pipe operator `|`, so `format_docs` runs *after* `retriever` finishes. B: **Correct:** A dict in LCEL creates a `RunnableParallel`, executing the `"context"` branch (`retriever | format_docs`) and the `"input"` branch (`RunnablePassthrough()`) concurrently. C: **Incorrect:** `prompt_template | llm` is sequential because `llm` needs the formatted prompt output from `prompt_template`. D: **Incorrect:** LCEL executes dictionary branches concurrently via `RunnableParallel`.