Mathematical Foundations of Generative AI · Quiz 1 · Sep 2025

**1. Recall the divergence form:** An f-divergence has the density-ratio form \(D_f(P\|Q)=\int q(x)f(p(x)/q(x))\,dx\), with an appropriate convex f. KL, reverse KL and total variation have such representations. **2. Identify what Wasserstein measures:** Wasserstein distance optimizes the cost of transporting probability mass between locations. It depends on the ground metric between points, rather than only their pointwise density ratios. **3. Distinguish disjoint point masses:** Two distinct point masses can be arbitrarily close geometrically while remaining mutually singular. Wasserstein distance shrinks with their separation, unlike an ordinary density-ratio f-divergence, illustrating the difference. **Answer: D — Wasserstein distance.** A: **Incorrect:** Forward KL is an f-divergence with f(t)=t log t. B: **Incorrect:** Reverse KL is an f-divergence, for example with f(t)=−log t under the corresponding argument order. C: **Incorrect:** Total variation is an f-divergence with f(t)=|t−1|/2 under its common normalization. D: **Correct:** Wasserstein distance uses optimal transport and a ground metric; it is not an ordinary f-divergence.

**1. Locate the supports:** \(p_{data}=\delta(x-2)\) puts all real mass at x=2, whereas \(p_g=\delta(x+4)\) puts generated mass at x=−4. Their supports are disjoint. **2. Optimize the discriminator:** Under the standard real-probability convention, an optimal discriminator has \(D^*(2)=1\) and \(D^*(-4)=0\). The supports can be classified perfectly. **3. Evaluate the full GAN value:** \[V(G)=\mathbb E_{data}\log D^*(x)+\mathbb E_g\log(1-D^*(x))=\log1+\log1=0.\] Equivalently, \(-\log4+2\operatorname{JSD}(P\|Q)=0\), since disjoint distributions have JSD \(\log2\). **Answer: B — 0.** **Convention note:** V(G) is interpreted as the usual GAN minimax value after optimizing the discriminator, as intended by the question. A: **Incorrect:** −log 2 is not the sum of the two perfectly classified log-likelihood terms. B: **Correct:** Both terms equal log 1 = 0 at the optimal discriminator. C: **Incorrect:** No probability 0.3 occurs in the stated point-mass problem. D: **Incorrect:** log 2 is the JSD here, not the full GAN value −log 4 + 2 JSD.

**1. Apply Fenchel duality:** For a convex f, \(f(t)\ge ut-f^*(u)\). Substituting a test function T and integrating gives the stated variational lower bound; optimizing over suitable T recovers the divergence under standard regularity conditions. **2. Identify the optimal test function:** Where \(P\ll Q\) and f is differentiable at the density ratio, the pointwise optimizer is \(T^*(x)=f\prime(p(x)/q(x))\). A restricted discriminator may fail to represent this optimizer exactly. **3. Check forward KL:** For \(f(t)=t\log t\), maximizing \(ut-t\log t\) yields \(t=e^{u-1}\), so \(f^*(u)=e^{u-1}\). The optimal test function is \(1+\log(p/q)\). **4. Inspect the support condition in D:** A positive-infinite density-ratio/test-function limit occurs where q vanishes while p is positive. The option instead says “where P doesn’t hold,” rather than specifying positive P mass. That support claim, as written, is the false part. **Answer: D — D — the stated support condition is incorrect.** **Convention note:** The phrase “where P doesn’t hold” is imprecise. If it was intended to mean q=0 while p>0, then the corrected D would also be true and the question would have no false option. The selection follows its supplied wording, without changing that phrase. A: **Incorrect:** This is the standard optimal variational function where the derivative and density ratio are defined, so it is not the false statement sought. B: **Incorrect:** Each admissible test function supplies a lower bound by Fenchel’s inequality, so it should not be selected as false. C: **Incorrect:** Restricting the discriminator class can prevent reaching the optimal test function and loosen the bound; this is true. D: **Correct:** Select this as false: the conjugate formula is correct, but the unboundedness condition must concern q=0 with p>0, not a region lacking P mass.

**1. Keep the stated sign convention:** The question defines a value to be maximized, \(V_D=\operatorname{mean}(\log D_{real})+\operatorname{mean}(\log(1-D_{fake}))\). Do not insert the negative sign used by a minimized binary-cross-entropy loss. **2. Compute the real term:** For 0.8 and 0.6, \((\log0.8+\log0.6)/2=(-0.22314355-0.51082562)/2=-0.36698459\). **3. Compute the fake term:** Fake scores 0.3 and 0.4 give complements 0.7 and 0.6. Their mean log is \((-0.35667494-0.51082562)/2=-0.43375028\). **4. Add:** \(V_D=-0.80073487\ldots\). The closest listed value is −0.8008. **Answer: A — −0.8008.** **Convention note:** None of the options is the exact four-decimal rounding, which is −0.8007. A differs by about 0.000065 and is clearly the intended nearest answer; this small option-rounding discrepancy is recorded explicitly. A: **Correct:** This is the closest option to the natural-log calculation −0.80073487; conventional four-decimal rounding itself gives −0.8007. B: **Incorrect:** It does not equal the sum of the two required mean-log terms. C: **Incorrect:** It is too negative for the supplied real and fake scores. D: **Incorrect:** It omits or miscalculates part of the two-term expression.

**1. Use the explicit linear-layer widths:** A dense layer from m inputs to n outputs has mn weights when biases are excluded. The specified chain is \(97\to125\to254\to782\). **2. Count each matrix:** The counts are \(97(125)=12{,}125\), \(125(254)=31{,}750\), and \(254(782)=198{,}628\). **3. Add only weights:** \(12{,}125+31{,}750+198{,}628=242{,}503\). No neuron biases are added. **Answer: C — 242,503.** **Convention note:** The narrative also mentions a 102-dimensional latent input and a 29×25 image (725 outputs), which conflict with the explicit 97-input/782-output layer chain. The answer counts the three explicitly specified linear layers; it does not claim those inconsistent dimensions form a valid end-to-end generator. A: **Incorrect:** 80,402 is not the sum of the three stated weight-matrix sizes. B: **Incorrect:** 99,402 does not account for the complete specified layer chain. C: **Correct:** 242,503 exactly equals 97×125 + 125×254 + 254×782. D: **Incorrect:** The displayed value is malformed and does not equal the computed weight count.

**1. Form the first reconstruction error:** \(E(G(z_1))-z_1=(2.1,-0.8)-(2,-1)=(0.1,0.2)\). Its Euclidean norm is \(\sqrt{0.1^2+0.2^2}=\sqrt{0.05}=0.2236068\ldots\). **2. Form the second error:** \((-1.2,0.6)-(-1,1)=(-0.2,-0.4)\), with norm \(\sqrt{0.04+0.16}=\sqrt{0.20}=0.4472136\ldots\). **3. Average the norms:** \((0.2236068+0.4472136)/2=0.3354102\ldots\), which rounds to 0.34. The formula averages L₂ norms, not squared norms or coordinate-wise errors. **Answer: C — 0.34.** A: **Incorrect:** 0.22 approximates the first norm alone and omits the second sample. B: **Incorrect:** 0.28 is not the average of the two Euclidean norms. C: **Correct:** The required mean is approximately 0.33541, which rounds to 0.34. D: **Incorrect:** 0.41 exceeds the calculated average and does not follow the supplied norm formula.

**1. State the feasibility constraints:** A transport plan T must have nonnegative entries, row sums \(P=(0.3,0.4,0.3)\), and column sums \(Q=(0.4,0.3,0.3)\). Total mass must be one. **2. Check the candidate in C:** \[T=\begin{bmatrix}0.2&0.1&0\\0.2&0.1&0.1\\0&0.1&0.2\end{bmatrix}.\] Its row sums are 0.3,0.4,0.3, and its column sums are 0.4,0.3,0.3. **3. Conclude feasibility:** All entries are nonnegative and both prescribed marginals match. Feasibility alone does not say this plan minimizes an unspecified transport cost. **Answer: C — C — the nonnegative plan with row marginal P and column marginal Q.** A: **Incorrect:** Its row sums are 0.3,0.4,0.4 and column sums 0.5,0.3,0.3, so even its total mass is 1.1. B: **Incorrect:** Its row sums are 0.3,0.4,0.4 and column sums 0.4,0.3,0.4, failing the required marginals. C: **Correct:** Its nonnegative entries have exactly the prescribed row and column sums. D: **Incorrect:** The diagonal gives row and column sums 0.4,0.4,0.2, matching neither required marginal.

**1. Interpret the first term:** \(\|\mu_r-\mu_g\|^2\) compares feature-space means. It measures displacement of the fitted distributions’ centres, termed “bias shift” by the option. **2. Interpret the trace term:** The covariance expression compares the fitted spreads and correlations. In one dimension it reduces to \((\sigma_r-\sigma_g)^2\), illustrating that it measures covariance/scale mismatch. **3. Keep the roles distinct:** The first term does not depend on covariance, and the second does not depend on the means. Neither is generically dominated by generator variance. **Answer: A — The first term measures bias shift; the second term measures covariance .** A: **Correct:** It correctly assigns the first term to mean shift and the second to covariance mismatch. B: **Incorrect:** The first term explicitly uses the two means, so it is not covariance-only. C: **Incorrect:** Either term can dominate depending on the two distributions; no general variance dominance follows. D: **Incorrect:** This reverses the roles of the two displayed terms.

**1. Identify the representation:** Standard FID evaluates statistics of features from a fixed pretrained image-recognition network rather than measuring raw pixel differences. **2. Identify the usual layer:** The conventional implementation extracts the 2048-dimensional pooled feature representation of Inception-v3, commonly called pool3 or its penultimate feature layer before the classification head. **3. Explain the comparison:** Means and covariance matrices of these real and generated image features are used to compute the Gaussian Fréchet distance. Using an arbitrary generator/discriminator layer would define a different evaluation representation. **Answer: D — The penultimate layer of an Inception-v3 network.** A: **Incorrect:** Generator outputs or logits are not the standard fixed Inception feature representation. B: **Incorrect:** Raw RGB values are not the usual FID feature space. C: **Incorrect:** A discriminator’s first convolution is not the conventional Inception-v3 pool3 representation. D: **Correct:** The pooled pre-classification Inception-v3 features are the standard FID representation.

**1. Use the Gaussian-distance interpretation:** FID is the squared Wasserstein-2 distance between the Gaussian distributions fitted to the two feature collections. It is therefore nonnegative. **2. Check equality:** Matching fitted means and covariances gives zero. Under the question’s Gaussian assumption this means the two fitted Gaussian distributions are identical. **3. Address singular covariance:** A covariance need only be positive semidefinite; a symmetric square-root formulation remains meaningful even for singular covariance. Strict positive definiteness is not required by the mathematical distance. **Answer: B — FID is nonnegative and vanishes for identical fitted Gaussian distributions.** **Convention note:** For arbitrary non-Gaussian image-feature distributions, zero FID only establishes equality of the measured means and covariances, not equality of the full distributions. Finite-precision implementations can also have numerical square-root issues distinct from mathematical undefinedness. A: **Incorrect:** Matching means and covariances gives zero, not a negative FID. B: **Correct:** FID is nonnegative and is zero for matching fitted Gaussian distributions. C: **Incorrect:** Singular positive-semidefinite covariance matrices do not inherently make the mathematical FID undefined. D: **Incorrect:** FID is a Gaussian Wasserstein-type distance, not KL divergence.

**1. Distinguish copying from mutation:** torch.clamp returns a clipped tensor. Assigning that result only to the loop variable p does not replace the registered parameter’s stored values. An in-place clamp_ mutates the original tensor. **2. Evaluate the listed code:** Iterating over D.parameters() and applying p.data.clamp_(-0.01,0.01) changes each critic parameter’s underlying values, giving the intended historical WGAN clipping implementation. **3. Explain scope:** Weight clipping constrains a parameter set to support a bounded-Lipschitz critic family. It does not calculate or certify that the resulting network’s exact Lipschitz constant equals one. **Answer: B — for p in D.parameters(): p.data.clamp_(-0.01, 0.01).** **Convention note:** The listed historical .data form performs the intended mutation, but modern code should prefer with torch.no_grad(): followed by p.clamp_(-0.01,0.01). Parameter clipping to this interval is a heuristic Lipschitz restriction, not an exact one-Lipschitz certificate. A: **Incorrect:** Rebinding the local variable p to a newly clipped tensor does not update the module’s registered parameter in place. B: **Correct:** The underscore method mutates the parameter storage, matching the requested in-place clipping behavior. C: **Incorrect:** D is a module and does not generally expose Tensor.clamp_ to clip all parameters automatically. D: **Incorrect:** Disabling requires_grad freezes parameters but does not clip their values.

**1. Compare the distributions:** Both Gaussian distributions have variance one and differ only in their means, 0.20 and 0.55. They are translations of the same distribution. **2. Give a transport coupling:** Pair \(X=0.20+Z\) and \(Y=0.55+Z\) using the same standard-normal Z. Then \(|X-Y|=0.35\) always, so \(W_1\le0.35\). **3. Match the lower bound:** Any coupling satisfies \(\mathbb E|X-Y|\ge|\mathbb E X-\mathbb E Y|=0.35\). The coupling attains the bound, so \(W_1=0.35\). **Answer: B — 0.35.** A: **Incorrect:** 0.20 is one mean, not the separation between the two translated distributions. B: **Correct:** The absolute mean difference is 0.35, which is the exact W₁ distance for equal-variance translates. C: **Incorrect:** 0.55 is the other mean rather than the transport distance. D: **Incorrect:** Adding the means does not give the translation distance.

**1. Explain saturation:** When real and generated distributions have disjoint supports, their Jensen–Shannon divergence is \(\log2\). Moving the generated support without creating overlap can leave that divergence unchanged. **2. Connect to generator learning:** A near-perfect discriminator can make the original minimax generator objective provide a very weak or vanishing gradient. A constant divergence over nearby parameter settings supplies no useful direction for improvement. **3. Demonstrate possible discontinuity:** Let \(P=\delta_0\) and \(Q_\theta=\delta_\theta\). JSD is zero at \(\theta=0\), but equals \(\log2\) for every nonzero θ, even as θ tends to zero. Thus continuity in generator parameters is not guaranteed. **Answer: A, B, C — JSD saturates when the supports of pdata and pg do not overlap.; Gradients for the generator can vanish when the discriminator becomes too ; JSD is not continuous w.r.t. the parameters of the generator distribution..** **Convention note:** C means JSD need not be continuous for parametrized generated distributions; it is not a claim that JSD is discontinuous for every model. Non-saturating losses alter the immediate generator-gradient behavior. A: **Correct:** Disjoint-support saturation can remove a useful distinction between generated distributions at different locations. B: **Correct:** An overly accurate discriminator can produce weak gradients for the original saturating minimax generator loss. C: **Correct:** JSD need not be continuous in generator parameters, as the moving-point-mass example demonstrates. D: **Incorrect:** An alleged universal unbiased-gradient guarantee is neither true nor a reason for instability.

**1. Trace the two joint samples:** Real-side pairs are \((x,E(x))\); generated-side pairs are \((G(z),z)\). In the displayed objective, only the first term explicitly depends on the encoder E. **2. Examine the ideal optimum:** If the two joint distributions match, their latent marginals also match. Consequently the aggregate encoded-data distribution equals the latent prior: \(E_{\#}p_{data}=p_z\). This is an idealized optimum statement, not a guarantee for any finite training run. **3. Identify what the discriminator learns:** It must check whether x and z form the appropriate pair, rather than merely whether their separate marginals look plausible. Joint matching can learn encoder–generator consistency without requiring an explicit reconstruction loss. **Answer: A, B, D — A, B and D — encoder gradient path, ideal latent marginal matching and joint discrimination.** **Convention note:** A assumes separate parameters for encoder and generator, as in the basic BiGAN setup, and concerns this displayed objective only. Additional losses or shared parameters would require a separate gradient analysis. A: **Correct:** For separate encoder/generator parameter sets, the second displayed term has no encoder E computation and gives it no direct gradient. B: **Correct:** Equality of the joint distributions implies equality of their latent marginals at the ideal optimum. C: **Incorrect:** An explicit reconstruction penalty is not a requirement of the basic BiGAN adversarial formulation. D: **Correct:** The discriminator receives pairs and can exploit their dependence, so joint statistics matter beyond marginal matching.

**1. Use the non-saturating objective:** The requested loss is \(-\operatorname{mean}(\ln D)\), with fake-sample discriminator outputs 0.4,0.55 and 0.5. It is not the saturating \(\log(1-D)\) objective. **2. Calculate the three log terms:** \(\ln0.4=-0.916290732\), \(\ln0.55=-0.597837001\), and \(\ln0.5=-0.693147181\). Their sum is −2.207274914. **3. Average and change sign:** \(L_G^{ns}=-(-2.207274914)/3=0.735758305\ldots\). A suitable reported value is 0.736. **Answer: 0.736.**