Mathematical Foundations of Generative AI · End Term · May 2025 · FN

**Step 1: Recall the definition of \(f\)-divergence.** \[D_f(P \parallel Q) = \int q(x)\,f\!\left(\frac{p(x)}{q(x)}\right) d\mu(x)\] **Step 2: Substitute \(f(u) = u \log u\) where \(u = \frac{p(x)}{q(x)}\).** \[D_f(P \parallel Q) = \int q(x) \left(\frac{p(x)}{q(x)} \log \frac{p(x)}{q(x)}\right) d\mu(x) = \int p(x) \log \frac{p(x)}{q(x)}\,d\mu(x) = \text{KL}(P \parallel Q)\] Also note that \(f(u) = u \log u\) is convex on \(\mathbb{R}_+\) (since \(f''(u) = 1/u > 0\)) and satisfies \(f(1) = 1 \log 1 = 0\). **Answer: B (\(f(u) = u \log u\))** A: **Incorrect:** \(f(u) = \frac{1}{2}(u - 1)^2\) yields the half Pearson \(\chi^2\)-divergence, not \(\text{KL}(P \parallel Q)\). B: **Correct:** Substituting \(f(u) = u \log u\) gives \(\int q(x) \frac{p(x)}{q(x)} \log \frac{p(x)}{q(x)} d\mu(x) = \text{KL}(P \parallel Q)\). C: **Incorrect:** \(f(u) = (\sqrt{u} - 1)^2\) yields the squared Hellinger distance. D: **Incorrect:** \(f(u) = -\log u + u - 1\) yields the reverse KL divergence \(\text{KL}(Q \parallel P)\).

**Step 1: Write the GAN value function \(V(D, G)\) as an integral over \(x\).** \[V(D, G) = \int \Big[ p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x)) \Big] dx\] **Step 2: Maximize the integrand pointwise with respect to \(y = D(x) \in (0, 1)\).** Take the derivative of \(g(y) = a \log y + b \log(1 - y)\) (where \(a = p_{\text{data}}(x)\) and \(b = p_g(x)\)) and set it to zero: \[\frac{dg}{dy} = \frac{a}{y} - \frac{b}{1 - y} = 0 \implies a(1 - y) = b y \implies y = \frac{a}{a + b}\] Therefore, the pointwise optimal discriminator is: \[D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}\] **Answer: B (\(D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}\))** A: **Incorrect:** A hard step function \(\mathbf{1}\{p_{\text{data}}(x) > p_g(x)\}\) makes \(\log D(x)\) or \(\log(1-D(x))\) diverge to \(-\infty\). B: **Correct:** Setting the derivative of \(p_{\text{data}}(x)\log D(x) + p_g(x)\log(1-D(x))\) to zero gives \(D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}\). C: **Incorrect:** The likelihood ratio \(p_g(x)/p_{\text{data}}(x)\) is not bounded in \([0, 1]\). D: **Incorrect:** \(D^*(x) = 1/2\) holds only at the global Nash equilibrium where \(p_g = p_{\text{data}}\), not for an arbitrary fixed \(G\).

**Step 1: Recall the output spatial size formula for `ConvTranspose2d`.** \[H_{\text{out}} = (H_{\text{in}} - 1) \times s - 2p + k\] With kernel \(k = 4\), stride \(s = 2\), and padding \(p = 1\): \[H_{\text{out}} = (H_{\text{in}} - 1) \times 2 - 2(1) + 4 = 2 H_{\text{in}} - 2 - 2 + 4 = 2 H_{\text{in}}\] Thus, each `ConvTranspose2d(kernel_size=4, stride=2, padding=1)` block **doubles** the spatial dimensions. **Step 2: Apply four blocks starting from \(4 \times 4\).** - After block 1: \(4 \times 2 = 8 \implies 8 \times 8\) - After block 2: \(8 \times 2 = 16 \implies 16 \times 16\) - After block 3: \(16 \times 2 = 32 \implies 32 \times 32\) - After block 4: \(32 \times 2 = 64 \implies 64 \times 64\) **Answer: C (64 × 64)** A: **Incorrect:** \(16 \times 16\) is the spatial size after only 2 blocks. B: **Incorrect:** \(32 \times 32\) is the spatial size after 3 blocks. C: **Correct:** Each block doubles the spatial resolution, so 4 blocks scale \(4 \times 4\) by \(2^4 = 16\) to \(64 \times 64\). D: **Incorrect:** \(128 \times 128\) would require 5 blocks.

In W-GAN (Arjovsky et al., 2017; Gulrajani et al., 2017), enforcing the Lipschitz constraint via crude element-wise **weight clipping** \(\|W_l\|_\infty \le c\) pushes critic weights toward the extreme clipping boundaries \(\pm c\), which **severely reduces the model capacity** of the critic (causing it to learn only very simple functions / underfit) and leads to **exploding or vanishing gradients (gradient pathology)** depending on the threshold \(c\). **Answer: B (Reduce capacity, causing underfitting and gradient pathology)** A: **Incorrect:** Weight clipping restricts the critic to simple functions, reducing rather than increasing capacity. B: **Correct:** Weight clipping forces weights to \(\pm c\), drastically reducing critic capacity (underfitting) and causing vanishing/exploding gradients. C: **Incorrect:** Weight clipping only enforces a rough \(K\)-Lipschitz bound with severe bias, which is why WGAN-GP was introduced. D: **Incorrect:** Clipping weights in \([-c, c]\) does not make the neural network \(T\) convex.

In **BiGAN (Bidirectional GAN)** (Donahue et al., 2017), the architecture includes a generator \(G : \mathcal{Z} \to \mathcal{X}\) and an encoder \(E : \mathcal{X} \to \mathcal{Z}\). The discriminator \(D\) operates on **joint data-latent pairs** \((x, z)\), distinguishing between: - Real data paired with its encoded latent representation: **\((x, E(x))\)** where \(x \sim p_{\text{data}}\) - Generated data paired with the prior latent sample: **\((G(z), z)\)** where \(z \sim p_Z\) **Answer: B (\((x, E(x))\) vs \((G(z), z)\))** A: **Incorrect:** Sampling \(x\) and \(z\) independently ignores the encoder \(E(x)\) and generator \(G(z)\) couplings. B: **Correct:** The BiGAN discriminator distinguishes joint encoder pairs \((x, E(x))\) from joint generator pairs \((G(z), z)\). C: **Incorrect:** Reverses the domains of \(E\) and \(G\); \(E\) takes \(x\) as input and \(G\) takes \(z\) as input. D: **Incorrect:** Does not involve either the encoder \(E\) or the generator \(G\).

**Step 1: Write the KL divergence between \(q(z|x) = \mathcal{N}(\mu, \text{diag}(\sigma^2))\) and \(p(z) = \mathcal{N}(0, I)\) in terms of \(\mu_i\) and \(\alpha_i = \log \sigma_i^2\).** Since \(\sigma_i^2 = e^{\alpha_i}\), the closed-form KL divergence is: \[\text{KL} = \frac{1}{2} \sum_{i=1}^d \left( \mu_i^2 + \sigma_i^2 - \log \sigma_i^2 - 1 \right) = \frac{1}{2} \sum_{i=1}^d \left( \mu_i^2 + e^{\alpha_i} - \alpha_i - 1 \right)\] **Step 2: Differentiate with respect to \(\mu_i\) and \(\alpha_i\).** \[\frac{\partial \text{KL}}{\partial \mu_i} = \frac{1}{2} (2 \mu_i) = \mu_i\] \[\frac{\partial \text{KL}}{\partial \alpha_i} = \frac{1}{2} \left( e^{\alpha_i} - 1 \right)\] **Answer: A (\(\partial \text{KL}/\partial \mu_i = \mu_i, \quad \partial \text{KL}/\partial \alpha_i = \frac{1}{2}(e^{\alpha_i} - 1)\))** A: **Correct:** Differentiating \(\frac{1}{2}(\mu_i^2 + e^{\alpha_i} - \alpha_i - 1)\) yields \(\mu_i\) and \(\frac{1}{2}(e^{\alpha_i} - 1)\). B: **Incorrect:** Neither derivative matches the partial derivatives of the Gaussian KL formula. C: **Incorrect:** Omits the \(1/2\) factor and the \(-1\) term from \(-\alpha_i\). D: **Incorrect:** Treats \(\mu_i^2\) and \(e^{\alpha_i}\) as constants.

**Step 1: Recall the per-sample Gaussian KL divergence formula for a batch `[B, d]`.** For each sample in the batch, summing over the latent dimension `d` (`dim=1`): \[\text{KL} = -\frac{1}{2} \sum_{i=1}^d \left(1 + \log(\sigma_i^2) - \mu_i^2 - \sigma_i^2\right)\] **Step 2: Translate to PyTorch with `mu` and `logvar` of shape `[B, d]`.** ```python kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp(), dim=1) ``` This sums along `dim=1` (the feature dimension `d`) to return a tensor of shape `[B]` containing the per-sample KL divergence. **Answer: A (`kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp(), dim=1)`)** A: **Correct:** Correctly applies `-0.5` and sums `1 + logvar - mu.pow(2) - logvar.exp()` along `dim=1` for per-sample KL. B: **Incorrect:** Uses `+0.5` instead of `-0.5` (flipping the sign) and sums over `dim=0` (batch dimension) instead of `dim=1`. C: **Incorrect:** Omits the `- logvar` term and sums over all elements into a single scalar instead of per-sample. D: **Incorrect:** Computes only the mean of `mu**2` and ignores `logvar` entirely.

In a Variational Autoencoder (VAE), **unconditional generation** (generating new samples at test time without an input image \(x\)) discards the encoder \(q_\phi(z \mid x)\) and follows the generative model: 1. Sample a latent vector \(z\) from the prior distribution: \(z \sim p(z)\) (typically \(\mathcal{N}(0, I)\)). 2. Pass \(z\) through the decoder to sample/generate data: \(x \sim p_\theta(x \mid z)\). **Answer: B (\(z \sim p(z)\), then \(x \sim p_\theta(x \mid z)\))** A: **Incorrect:** \(z \sim q_\phi(z \mid x)\) followed by \(x \sim p_\theta(x \mid z)\) is **reconstruction** of a given input \(x\), not unconditional generation. B: **Correct:** Unconditional generation samples \(z \sim p(z)\) from the prior and decodes \(x \sim p_\theta(x \mid z)\). C: **Incorrect:** Reverses the causal order; \(z\) must be sampled before \(x\) can be decoded. D: **Incorrect:** AVAE generates stochastic samples via \(z \sim p(z)\), not a point mass at 0.

In **VQ-VAE** vector quantization in PyTorch, given encoder output `z_e` of shape `[B, C, H, W]` and `codebook` of shape `[K, C]`: 1. Move channels `C` to the last dimension and flatten spatial/batch dimensions to `[N, C]` where \(N = B \cdot H \cdot W\): `z = z_e.permute(0, 2, 3, 1).reshape(-1, C)` 2. Compute pairwise squared Euclidean distances \(\|z - e_k\|^2 = \|z\|^2 - 2 z e_k^T + \|e_k\|^2\) of shape `[N, K]`: `dist = (z.pow(2).sum(1, keepdim=True) - 2 * z @ codebook.t() + codebook.pow(2).sum(1))` 3. Find the nearest codebook index for each spatial position: `idx = dist.argmin(1)` 4. Look up the quantized vectors, reshape back to `[B, H, W, C]`, and permute channels back to `[B, C, H, W]`: `z_q = codebook[idx].view(B, H, W, C).permute(0, 3, 1, 2)` **Answer: A** A: **Correct:** Properly flattens `z_e` to `[B*H*W, C]`, computes squared \(L_2\) distances, selects `argmin(1)`, and reshapes/permutes back to `[B, C, H, W]`. B: **Incorrect:** Multiplying `[B, C, H, W]` directly with `[K, C]` causes a tensor dimension mismatch and uses dot-product `argmax` instead of Euclidean distance. C: **Incorrect:** Picks random indices instead of nearest-neighbor quantization. D: **Incorrect:** Averages channels instead of quantizing against the codebook.

**Step 1: Compute \(\alpha_1, \alpha_2\) and \(\bar{\alpha}_2\).** Given \(\beta_1 = 0.01\) and \(\beta_2 = 0.02\): - \(\alpha_1 = 1 - \beta_1 = 1 - 0.01 = 0.99\) - \(\alpha_2 = 1 - \beta_2 = 1 - 0.02 = 0.98\) - \(\bar{\alpha}_2 = \alpha_1 \alpha_2 = 0.99 \times 0.98 = 0.9702\) **Step 2: Compute \(x_2 = \sqrt{\bar{\alpha}_2}\,x_0 + \sqrt{1 - \bar{\alpha}_2}\,\epsilon\).** Given \(x_0 = 2\) and \(\epsilon = -1\): - \(\sqrt{\bar{\alpha}_2} = \sqrt{0.9702} \approx 0.984987\) - \(\sqrt{1 - \bar{\alpha}_2} = \sqrt{1 - 0.9702} = \sqrt{0.0298} \approx 0.172627\) \[x_2 = 0.984987 \times 2 + 0.172627 \times (-1) = 1.969975 - 0.172627 = 1.797348 \approx 1.797\] **Answer: A (1.797)** A: **Correct:** \(\bar{\alpha}_2 = 0.9702\), so \(x_2 = 2\sqrt{0.9702} - \sqrt{0.0298} \approx 1.9700 - 0.1726 = 1.797\). B: **Incorrect:** Does not properly subtract \(\sqrt{1 - \bar{\alpha}_2}\). C: **Incorrect:** \(2.000\) is the un-noised initial value \(x_0\). D: **Incorrect:** Uses incorrect values for \(\sqrt{\bar{\alpha}_2}\) and \(\sqrt{1 - \bar{\alpha}_2}\).

In DDPM (Ho et al., 2020), the reverse process mean \(\mu_\theta(x_t, t)\) parameterized via the noise predictor \(\epsilon_\theta(x_t, t)\) is given by: \[\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\,\epsilon_\theta(x_t, t) \right)\] In PyTorch, adding `1e-20` inside `torch.sqrt(1 - abar_t + 1e-20)` prevents division by zero: ```python coef = beta_t / torch.sqrt(1 - abar_t + 1e-20) mu = (x_t - coef * eps_pred) / torch.sqrt(alpha_t) ``` **Answer: A** A: **Correct:** Implements \(\mu_\theta = \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \epsilon_\theta)\) with numerical stability. B: **Incorrect:** Multiplies by \(\sqrt{\alpha_t}\) instead of dividing, and omits \(\sqrt{1 - \bar{\alpha}_t}\). C: **Incorrect:** Omits all diffusion schedule scaling coefficients. D: **Incorrect:** Averages `x_t` and `eps_pred` instead of applying the reverse diffusion mean formula.

**Step 1: Write the conditional forward Gaussian density \(q(x_t \mid x_0)\).** Since \(x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon\) with \(\epsilon \sim \mathcal{N}(0, I)\), we have: \[q(x_t \mid x_0) = \mathcal{N}\!\left(x_t;\; \sqrt{\bar{\alpha}_t}\,x_0,\; (1 - \bar{\alpha}_t)I\right)\] \[\log q(x_t \mid x_0) = -\frac{\|x_t - \sqrt{\bar{\alpha}_t}\,x_0\|^2}{2(1 - \bar{\alpha}_t)} + C\] **Step 2: Take the gradient with respect to \(x_t\) and substitute \(x_t - \sqrt{\bar{\alpha}_t}\,x_0 = \sqrt{1 - \bar{\alpha}_t}\,\epsilon\).** \[\nabla_{x_t} \log q(x_t \mid x_0) = -\frac{x_t - \sqrt{\bar{\alpha}_t}\,x_0}{1 - \bar{\alpha}_t} = -\frac{\sqrt{1 - \bar{\alpha}_t}\,\epsilon}{1 - \bar{\alpha}_t} = -\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\,\epsilon\] **Answer: A (\(-\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\epsilon\))** A: **Correct:** Since \(x_t - \sqrt{\bar{\alpha}_t}x_0 = \sqrt{1-\bar{\alpha}_t}\epsilon\), the score \(-\frac{x_t - \sqrt{\bar{\alpha}_t}x_0}{1 - \bar{\alpha}_t}\) simplifies to \(-\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\epsilon\). B: **Incorrect:** Omits the scaling factor \(\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\). C: **Incorrect:** Uses \(1 - \bar{\alpha}_t\) in the denominator instead of \(\sqrt{1 - \bar{\alpha}_t}\). D: **Incorrect:** Replaces \(\sqrt{\bar{\alpha}_t}x_0\) with unscaled \(x_0\).

**Step 1: Substitute the given values into \(c_1\).** Given \(\bar{\alpha}_{t-1} = 0.95\), \(\bar{\alpha}_t = 0.91\), \(\alpha_t = 0.9578947368\), and \(\beta_t = 0.0421052632\): - \(1 - \bar{\alpha}_t = 1 - 0.91 = 0.09\) - \(\sqrt{\bar{\alpha}_{t-1}} = \sqrt{0.95} \approx 0.9746794\) \[c_1 = \frac{\sqrt{\bar{\alpha}_{t-1}}\,\beta_t}{1 - \bar{\alpha}_t} = \frac{0.9746794 \times 0.0421052632}{0.09} \approx \frac{0.0410391}{0.09} \approx 0.456\] **Step 2: Substitute the given values into \(c_2\).** - \(\sqrt{\alpha_t} = \sqrt{0.9578947368} \approx 0.978721\) - \(1 - \bar{\alpha}_{t-1} = 1 - 0.95 = 0.05\) \[c_2 = \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} = \frac{0.978721 \times 0.05}{0.09} \approx \frac{0.048936}{0.09} \approx 0.544\] **Answer: A (\(c_1 \approx 0.456, \; c_2 \approx 0.544\))** A: **Correct:** Direct evaluation yields \(c_1 \approx 0.456\) and \(c_2 \approx 0.544\). B: **Incorrect:** Does not match the numerical evaluation of \(c_1\) and \(c_2\). C: **Incorrect:** Swaps and miscalculates the values of \(c_1\) and \(c_2\). D: **Incorrect:** \(c_1\) and \(c_2\) are not equal to \(0.5\).

In **DDIM (Denoising Diffusion Implicit Models)** (Song et al., 2020), the family of forward processes indexed by \(\eta \ge 0\) is designed to be **non-Markovian** while preserving the exact same conditional marginals \(q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_t}x_0, (1 - \bar{\alpha}_t)I)\) as DDPM (so the same pretrained DDPM network can be used without retraining). When \(\eta = 0\), the random noise term in the reverse step vanishes completely, making the reverse sampler **deterministic** (an ODE solver) and **non-Markovian** while matching the \(q(x_t \mid x_0)\) marginals. **Answer: A (Deterministic and non-Markovian but matches \(q(x_t \mid x_0)\) marginals)** A: **Correct:** Setting \(\eta = 0\) in DDIM eliminates stochastic noise, giving a deterministic, non-Markovian sampler that shares the \(q(x_t \mid x_0)\) marginals of DDPM. B: **Incorrect:** Stochastic and Markovian describes standard DDPM (\(\eta = 1\)), not DDIM with \(\eta = 0\). C: **Incorrect:** DDIM only equals DDPM when \(\eta = 1\), not when \(\eta = 0\). D: **Incorrect:** \(\eta = 0\) is the standard deterministic DDIM sampler.

**Classifier-Free Guidance (CFG)** (Ho & Salimans, 2022) combines the unconditional noise prediction \(\epsilon_u = \epsilon_\theta(x_t, t, \varnothing)\) and the conditional noise prediction \(\epsilon_c = \epsilon_\theta(x_t, t, c)\) via the linear extrapolation: \[\tilde{\epsilon}_\theta = \epsilon_u + w(\epsilon_c - \epsilon_u)\] Because DDIM is parameterized directly in terms of the predicted noise \(\hat{\epsilon}\), CFG is seamlessly applied to DDIM by **replacing \(\epsilon_\theta\) with \(\epsilon_u + w(\epsilon_c - \epsilon_u)\) before each DDIM update step**. **Answer: A (Replacing \(\epsilon_\theta\) by \(\epsilon_u + w(\epsilon_c - \epsilon_u)\) before each DDIM step)** A: **Correct:** Substituting the guided noise estimate \(\epsilon_u + w(\epsilon_c - \epsilon_u)\) into each DDIM step directly implements CFG. B: **Incorrect:** CFG works with the existing conditional/unconditional diffusion network without training a separate guidance network. C: **Incorrect:** CFG works with both DDPM and deterministic samplers like DDIM. D: **Incorrect:** Diffusion models predict continuous noise/scores \(\epsilon\), not discrete token logits.

**Step 1: Keep the top \(k = 2\) largest probabilities.** The initial probability distribution is \((p_1, p_2, p_3) = \left(0.5, \frac{1}{3}, \frac{1}{6}\right)\). The top 2 probabilities are \(p_1 = 0.5 = \frac{3}{6}\) and \(p_2 = \frac{1}{3} = \frac{2}{6}\), while \(p_3 = \frac{1}{6}\) is zeroed out. **Step 2: Renormalize the top-2 probabilities so they sum to 1.** - Sum of top 2: \(0.5 + \frac{1}{3} = \frac{1}{2} + \frac{1}{3} = \frac{5}{6}\) - Renormalized \(P_1\): \[P_1 = \frac{0.5}{5/6} = \frac{1/2}{5/6} = \frac{3}{5} = 0.6\] **Answer: A (0.6)** A: **Correct:** Keeping the top 2 values \((1/2, 1/3)\) and dividing \(1/2\) by their sum \(5/6\) gives \(P_1 = 3/5 = 0.6\). B: **Incorrect:** \(0.5\) is the un-renormalized original probability. C: **Incorrect:** \(0.67\) would correspond to \(2/3\), which does not match \(\frac{0.5}{0.5 + 1/3}\). D: **Incorrect:** \(0.75\) would correspond to \(3/4\).

**Step 1: Compute the scaled dot-product attention scores \(s_i = \frac{Q \cdot K_i}{\sqrt{d_k}}\).** Given \(Q = [1, 0]\), \(K_1 = [1, 0]\), \(K_2 = [0, 1]\), and \(d_k = 2\): - \(s_1 = \frac{1(1) + 0(0)}{\sqrt{2}} = \frac{1}{\sqrt{2}} \approx 0.707107\) - \(s_2 = \frac{1(0) + 0(1)}{\sqrt{2}} = 0\) **Step 2: Apply Softmax to obtain the attention weights \(\alpha_1, \alpha_2\).** - \(e^{s_1} = e^{1/\sqrt{2}} \approx e^{0.707107} \approx 2.028115\) - \(e^{s_2} = e^0 = 1\) - \(\alpha_1 = \frac{2.028115}{2.028115 + 1} = \frac{2.028115}{3.028115} \approx 0.66976\) - \(\alpha_2 = 1 - \alpha_1 \approx 0.33024\) **Step 3: Compute the weighted sum of values \(V_1 = [2]\) and \(V_2 = [1]\).** \[\text{Attention}(Q, K, V) = \alpha_1 V_1 + \alpha_2 V_2 = 0.66976(2) + 0.33024(1) = 1.33952 + 0.33024 \approx 1.670\] **Answer: A (≈ 1.670)** A: **Correct:** With scaled scores \(1/\sqrt{2}\) and \(0\), \(\alpha_1 \approx 0.670\), giving \(2(0.670) + 1(0.330) \approx 1.670\). B: **Incorrect:** \(1.500\) corresponds to equal attention weights \(\alpha_1 = \alpha_2 = 0.5\). C: **Incorrect:** \(2.000\) corresponds to hard attention \(\alpha_1 = 1, \alpha_2 = 0\). D: **Incorrect:** Swaps \(V_1\) and \(V_2\).

In Multi-Head Attention (MHA) with model dimension \(d_{\text{model}} = 768\), \(h = 12\) heads, and per-head dimension \(d_k = d_v = 64\): - The outputs of all \(h = 12\) heads are concatenated along the feature dimension to form a vector of size \(h \times d_v = 12 \times 64 = 768\). - The output projection matrix \(W_O\) maps this concatenated \((h \cdot d_v)\)-dimensional vector back to \(d_{\text{model}}\), so \(W_O\) has shape \((h \cdot d_v) \times d_{\text{model}} = 768 \times 768\), containing: \[768 \times 768 = 589,824\text{ parameters}\] **Answer: A (768 × 768 = 589,824)** A: **Correct:** \(W_O\) maps the concatenated \(12 \times 64 = 768\) head outputs to \(d_{\text{model}} = 768\), so its size is \(768 \times 768 = 589,824\). B: **Incorrect:** \(768 \times 64 = 49,152\) is the size of a single head's query/key/value projection matrix \(W_Q^i, W_K^i, W_V^i\). C: **Incorrect:** \(64 \times 64\) is the per-head dimension squared, not the multi-head output projection. D: **Incorrect:** \(12 \times 64 = 768\) is the concatenated vector dimension, not the matrix parameter count.

**Step 1: Evaluate \(f(\text{LN}(x))\).** We are given \(x = (2, 0)\), \(\text{LN}(x) = (1, -1)\), and the sublayer \(f\) is the identity function \(f(u) = u\): \[f(\text{LN}(x)) = \text{LN}(x) = (1, -1)\] **Step 2: Add the residual connection \(x + f(\text{LN}(x))\).** \[y = x + f(\text{LN}(x)) = (2, 0) + (1, -1) = (2 + 1, \; 0 + (-1)) = (3, -1)\] **Answer: A ((3,−1))** A: **Correct:** Adding the input \(x = (2, 0)\) to \(f(\text{LN}(x)) = (1, -1)\) gives \((3, -1)\). B: **Incorrect:** \((2, 0)\) is just the input \(x\) without adding \(f(\text{LN}(x))\). C: **Incorrect:** \((1, -1)\) is \(f(\text{LN}(x))\) without the residual connection \(x\). D: **Incorrect:** Sum of coordinates is not zero.

**Step 1: Recall the discounted return \(G_1\) starting from time step \(t = 1\).** \[G_1 = r_1 + \gamma r_2 + \gamma^2 r_3\] **Step 2: Substitute \(r_1 = 1\), \(r_2 = 0\), \(r_3 = 2\), and \(\gamma = 0.9\).** \[G_1 = 1 + (0.9)(0) + (0.9)^2 (2) = 1 + 0 + 0.81 \times 2 = 1 + 1.62 = 2.62\] **Answer: A (2.62)** A: **Correct:** \(G_1 = 1 + 0.9(0) + 0.9^2(2) = 1 + 1.62 = 2.62\). B: **Incorrect:** Ignores the discounted future reward or uses wrong powers of \(\gamma\). C: **Incorrect:** Uses \(\gamma^1 = 0.9\) instead of \(\gamma^2 = 0.81\) for \(r_3\) (\(1 + 1.8 = 2.8\)). D: **Incorrect:** Does not properly sum \(r_1 + \gamma^2 r_3\).

In Reinforcement Learning from Human Feedback (RLHF) / policy gradient fine-tuning of an Autoregressive Language Model (AR-LM), text generation is formulated as a Markov Decision Process (MDP) where: - **State \(s_t\)**: The current **token prefix** (the prompt plus all tokens generated so far, \(x_{<t}\)). - **Action \(a_t\)**: Choosing the **next token** \(x_t\) from the vocabulary \(\mathcal{V}\). - **Transition**: Deterministically appending \(a_t\) to the prefix \(s_t\) to form \(s_{t+1} = (s_t, a_t)\). **Answer: A (State = token prefix, action = next token)** A: **Correct:** At step \(t\), the state is the token prefix generated so far, and the action is selecting the next token. B: **Incorrect:** Reverses the roles of state and action. C: **Incorrect:** The vocabulary is the action space, not the state. D: **Incorrect:** Framing autoregressive generation as an MDP is the foundation of RLHF (e.g., PPO for LLMs).

**Step 1: Compute the un-clipped term \(r A\).** Given \(r = 1.5\), \(\varepsilon = 0.2\), and \(A = 0.5\): \[r A = 1.5 \times 0.5 = 0.750\] **Step 2: Compute the clipped term \(\text{clip}(r, 1 - \varepsilon, 1 + \varepsilon)\,A\).** - The clipping interval is \([1 - 0.2, \; 1 + 0.2] = [0.8, \; 1.2]\). - Since \(r = 1.5 > 1.2\), \(\text{clip}(1.5, 0.8, 1.2) = 1.2\). - Multiplying by \(A = 0.5\): \[\text{clip}(r, 1 - \varepsilon, 1 + \varepsilon)\,A = 1.2 \times 0.5 = 0.600\] **Step 3: Take the minimum of the two terms.** \[\min(0.750, \; 0.600) = 0.600\] **Answer: A (0.600)** A: **Correct:** \(\min(1.5 \times 0.5, \; \text{clip}(1.5, 0.8, 1.2) \times 0.5) = \min(0.75, 0.60) = 0.600\). B: **Incorrect:** \(0.750\) is the un-clipped value \(r A\) before taking the minimum with the clipped term. C: **Incorrect:** \(0.500\) is just the advantage \(A\) (corresponding to \(r = 1\)). D: **Incorrect:** \(0.300\) is not the value of either term.