**Answer: A**
**Topic:** Principal Component Analysis — *PCA Reconstruction Error Formula*
### Step-by-Step Solution
**Step 1:** Recall that in Principal Component Analysis (PCA) on a centered dataset ($\sum_{i=1}^n x_i = 0$), the principal directions $w_1, w_2, \dots, w_d$ form an **orthonormal basis** of $\mathbb{R}^d$ ($w_j^T w_j = 1$ and $w_j^T w_k = 0$ for $j \ne k$).
**Step 2:** When we project a centered data point $x \in \mathbb{R}^d$ onto the 2D subspace spanned by the top two principal components $w_1$ and $w_2$, its orthogonal projection (reconstructed vector) is: $$\hat{x} = (x^T w_1)w_1 + (x^T w_2)w_2$$
**Step 3:** The squared reconstruction error for the point $x$ is the squared Euclidean norm of the residual vector $x - \hat{x}$: $$\|x - \hat{x}\|^2 = \|x - (x^T w_1)w_1 - (x^T w_2)w_2\|^2$$
**Step 4:** This matches Option **A**.
**Key Takeaway / Tip:** The scalar coordinate of $x$ along unit vector $w_j$ is $(x^T w_j)$, which must multiply the direction vector $w_j$ to form the vector projection $(x^T w_j)w_j$.
A: **Correct:** Option A (
Source diagram or notation unavailable. Please report this question for correction.
) matches our derived result under PCA Reconstruction Error Formula. The scalar coordinate of $x$ along unit vector $w_j$ is $(x^T w_j)$, which must multiply the direction vector $w_j$ to form the vector projection $(x^T w_j)w_j$.
B: **Incorrect:** Option B (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for PCA Reconstruction Error Formula (correct choice: A).
C: **Incorrect:** Option C (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for PCA Reconstruction Error Formula (correct choice: A).
D: **Incorrect:** Option D (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for PCA Reconstruction Error Formula (correct choice: A).
k1 is a valid kernel, but k2 is not a valid kernel.
6406532775016.
k2 is a valid kernel, but k1 is not a valid kernel.
6406532775017.
Neither k1 nor k2 is a valid kernel.
Published solution
**Answer: A**
**Topic:** Kernel Methods — *Closure Properties of Valid Mercer Kernels*
### Step-by-Step Solution
**Step 1:** Analyze $k_1(x, y) = x^T y + x_1 x_2 y_1 y_2$ for $x, y \in \mathbb{R}^2$: notice that $x^T y = x_1 y_1 + x_2 y_2$, so we can write $k_1(x, y) = \phi_1(x)^T \phi_1(y)$ explicitly using the feature map: $$\phi_1(x) = \begin{bmatrix} x_1 \\ x_2 \\ x_1 x_2 \end{bmatrix} \implies \phi_1(x)^T \phi_1(y) = x_1 y_1 + x_2 y_2 + x_1 x_2 y_1 y_2 = k_1(x, y)$$ Because $k_1$ is an explicit inner product in $\mathbb{R}^3$ (or the sum of the linear kernel $x^T y$ and the product kernel $(x_1 y_1)(x_2 y_2)$), $k_1$ is a valid kernel.
**Step 2:** Analyze $k_2(x, y) = k_1(x, y) + 1$: adding a positive constant $c = 1 > 0$ to a valid kernel always yields a valid kernel, corresponding to appending $1$ to the feature map: $$\phi_2(x) = \begin{bmatrix} x_1 \\ x_2 \\ x_1 x_2 \\ 1 \end{bmatrix}$$
**Step 3:** Therefore, **both $k_1$ and $k_2$ are valid kernels**, making **A** correct.
**Key Takeaway / Tip:** Valid kernels are closed under addition, multiplication, and addition of any non-negative constant $c \ge 0$.
A: **Correct:** Option A (Both k1 and k2 are valid kernels.) matches our derived result under Closure Properties of Valid Mercer Kernels. Valid kernels are closed under addition, multiplication, and addition of any non-negative constant $c \ge 0$.
B: **Incorrect:** Option B (k1 is a valid kernel, but k2 is not a valid kernel.) does not satisfy the conditions derived in the step-by-step analysis for Closure Properties of Valid Mercer Kernels (correct choice: A).
C: **Incorrect:** Option C (k2 is a valid kernel, but k1 is not a valid kernel.) does not satisfy the conditions derived in the step-by-step analysis for Closure Properties of Valid Mercer Kernels (correct choice: A).
D: **Incorrect:** Option D (Neither k1 nor k2 is a valid kernel.) does not satisfy the conditions derived in the step-by-step analysis for Closure Properties of Valid Mercer Kernels (correct choice: A).
**Answer: A, B**
**Topic:** Principal Component Analysis — *Covariance Matrix with Standard Basis Eigenvectors*
### Step-by-Step Solution
**Step 1:** We are given that the principal components (eigenvectors of the covariance matrix $C \in \mathbb{R}^{3 \times 3}$) are the standard basis vectors $e_1 = [1, 0, 0]^T$, $e_2 = [0, 1, 0]^T$, and $e_3 = [0, 0, 1]^T$ with eigenvalues $\lambda_1, \lambda_2, \lambda_3$.
**Step 2:** By the spectral decomposition of a symmetric matrix $C = U \Lambda U^T$, since the eigenvector matrix is $U = [e_1, e_2, e_3] = I_3$ (the $3 \times 3$ identity matrix), we have: $$C = I_3 \begin{bmatrix} \lambda_1 & 0 & 0 \\ 0 & \lambda_2 & 0 \\ 0 & 0 & \lambda_3 \end{bmatrix} I_3^T = \begin{bmatrix} \lambda_1 & 0 & 0 \\ 0 & \lambda_2 & 0 \\ 0 & 0 & \lambda_3 \end{bmatrix}$$ Thus, **$C$ is a diagonal matrix** (**A** is true and **C** is false).
**Step 3:** Because every covariance matrix $C$ is positive semi-definite ($v^T C v \ge 0$ for all $v$), its eigenvalues $\lambda_1, \lambda_2, \lambda_3$—which are also the diagonal entries (variances along each coordinate axis)—must be **non-negative** (**B** is true and **D** is false).
**Step 4:** Hence, **A and B** are correct.
**Key Takeaway / Tip:** The diagonal entries of any covariance matrix are the variances of the individual features, which are always non-negative ($\text{Var}(X_j) \ge 0$).
A: **Correct:** Option A (
Source diagram or notation unavailable. Please report this question for correction.
) matches our derived result under Covariance Matrix with Standard Basis Eigenvectors. The diagonal entries of any covariance matrix are the variances of the individual features, which are always non-negative ($\text{Var}(X_j) \ge 0$).
B: **Correct:** Option B (
Source diagram or notation unavailable. Please report this question for correction.
) matches our derived result under Covariance Matrix with Standard Basis Eigenvectors. The diagonal entries of any covariance matrix are the variances of the individual features, which are always non-negative ($\text{Var}(X_j) \ge 0$).
C: **Incorrect:** Option C (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Covariance Matrix with Standard Basis Eigenvectors (correct choice: A, B).
D: **Incorrect:** Option D (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Covariance Matrix with Standard Basis Eigenvectors (correct choice: A, B).
E: **Incorrect:** Option E (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Covariance Matrix with Standard Basis Eigenvectors (correct choice: A, B).
Kernel PCA with a polynomial kernel of degree 3 is run on a dataset of 100 data-points. Let ϕ be
the transformation corresponding to this kernel. If the largest eigenvalue of the centered kernel
matrix is 10, find the largest eigenvalue of the covariance matrix of the transformed dataset,
where ϕ is used to effect the transformation.
Published solution
**Answer: 0.1**
**Topic:** Kernel PCA — *Relationship Between Kernel Matrix and Covariance Matrix Eigenvalues*
### Step-by-Step Solution
**Step 1:** Let $X \in \mathbb{R}^{d \times n}$ be the centered data matrix with $n = 100$ data points. The sample covariance matrix is $C = \frac{1}{n} X X^T$, while the Gram (linear kernel) matrix is $K = X^T X$.
**Step 2:** Recall the fundamental algebraic property relating the non-zero eigenvalues of $X X^T$ and $X^T X$: both matrices share the exact same non-zero eigenvalues. Therefore, if $\mu_i$ is an eigenvalue of $K = X^T X$, the corresponding eigenvalue of $C = \frac{1}{n} X X^T$ is: $$\lambda_i(C) = \frac{\mu_i(K)}{n}$$
**Step 3:** Substitute the largest eigenvalue of $K$ ($\mu_{\max}(K) = 10$) and $n = 100$: $$\lambda_{\max}(C) = \frac{10}{100} = 0.1$$
**Key Takeaway / Tip:** Remember to divide the eigenvalue of the Gram matrix $K = X^T X$ by the sample size $n$ because the covariance matrix is defined as $C = \frac{1}{n} X X^T$.
**Answer: 0.8**
**Topic:** Gaussian Mixture Models — *Responsibility Normalization in EM Algorithm*
### Step-by-Step Solution
**Step 1:** In the Expectation-Maximization (EM) algorithm for a Gaussian Mixture Model (GMM) with $K = 3$ components, $\lambda_k^p = P(z_p = k \mid x_p)$ denotes the posterior responsibility that component $k$ takes for generating observation $x_p$.
**Step 2:** Because every data point $x_p$ must belong to one of the $K = 3$ mixture components, the responsibilities across all components for any fixed point $p$ form a valid probability distribution and sum to $1$: $$\lambda_1^p + \lambda_2^p + \lambda_3^p = 1$$
**Step 3:** Substitute the given sum $\lambda_1^p + \lambda_3^p = 0.2$: $$\lambda_2^p = 1 - (\lambda_1^p + \lambda_3^p) = 1 - 0.2 = 0.8$$
**Key Takeaway / Tip:** For any single data point $p$, the responsibilities $\lambda_k^p$ across all clusters $k = 1, \dots, K$ always sum to $1$.
Based on the above data, answer the given subquestions.
Find the variance of the dataset along the first principal component.
Published solution
**Answer: 3**
**Topic:** Principal Component Analysis — *Variance Along the First Principal Component*
### Step-by-Step Solution
**Step 1:** Write the given covariance matrix $C = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$.
**Step 2:** Recall that the variance of the dataset when projected onto the first principal component $w_1$ is equal to the largest eigenvalue $\lambda_1$ of the covariance matrix $C$ (i.e., $w_1^T C w_1 = \lambda_1$).
**Step 3:** Find the eigenvalues of $C$ from $\det(C - \lambda I) = 0$: $$(2 - \lambda)^2 - 1^2 = 0 \implies 2 - \lambda = \pm 1 \implies \lambda_1 = 3, \quad \lambda_2 = 1$$
**Step 4:** Thus, the variance along the first principal component is $\lambda_1 = 3$.
**Key Takeaway / Tip:** For any $2 \times 2$ symmetric matrix $\begin{bmatrix} a & b \\ b & a \end{bmatrix}$, the eigenvalues are immediately $a + b$ and $a - b$.
Based on the above data, answer the given subquestions.
Find the variance of the dataset along the x-axis.
Published solution
**Answer: 2**
**Topic:** Principal Component Analysis — *Variance Along a Coordinate Axis from Covariance Matrix*
### Step-by-Step Solution
**Step 1:** In a covariance matrix $C \in \mathbb{R}^{2 \times 2}$, the diagonal entry $C_{11}$ represents the variance of the first feature (along the x-axis, $e_1 = [1, 0]^T$), and $C_{22}$ represents the variance of the second feature (along the y-axis, $e_2 = [0, 1]^T$).
**Step 2:** Equivalently, project onto the unit x-axis vector $e_1 = [1, 0]^T$: $$\text{Var}(\text{x-axis}) = e_1^T C e_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = 2$$
**Key Takeaway / Tip:** The variance along the original $j$-th coordinate axis is simply the $j$-th diagonal element $C_{jj}$ of the covariance matrix.
**Answer: A**
**Topic:** Principal Component Analysis — *Change of Basis to Principal Component Coordinates*
### Step-by-Step Solution
**Step 1:** Find the normalized eigenvectors (principal components) $w_1$ and $w_2$ of $C = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$:
- For $\lambda_1 = 3$: $(C - 3I)w = 0 \implies w_1 = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ 1 \end{bmatrix}$
- For $\lambda_2 = 1$: $(C - 1I)w = 0 \implies w_2 = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ -1 \end{bmatrix}$ (or $\frac{1}{\sqrt{2}}[-1, 1]^T$)
**Step 2:** Compute the representation (coordinates) of $x = \begin{bmatrix} 2 \\ 2 \end{bmatrix}$ in the orthonormal basis $\{w_1, w_2\}$ by taking inner products with each basis vector:
- First coordinate: $c_1 = w_1^T x = \frac{1}{\sqrt{2}}(1(2) + 1(2)) = \frac{4}{\sqrt{2}} = 2\sqrt{2}$
- Second coordinate: $c_2 = w_2^T x = \frac{1}{\sqrt{2}}(1(2) - 1(2)) = 0$
**Step 3:** Thus, the representation of $[2, 2]^T$ in the principal component basis is $\begin{bmatrix} 2\sqrt{2} \\ 0 \end{bmatrix}$, which matches **A**.
**Key Takeaway / Tip:** Since $[2, 2]^T$ lies purely along the direction $[1, 1]^T$ of $w_1$, its second coordinate along the orthogonal direction $w_2$ must be $0$, and its norm $\sqrt{2^2 + 2^2} = 2\sqrt{2}$ is concentrated entirely in the first coordinate.
A: **Correct:** Option A (
Source diagram or notation unavailable. Please report this question for correction.
) matches our derived result under Change of Basis to Principal Component Coordinates. Since $[2, 2]^T$ lies purely along the direction $[1, 1]^T$ of $w_1$, its second coordinate along the orthogonal direction $w_2$ must be $0$, and its norm $\sqrt{2^2 + 2^2} = 2\sqrt{2}$ is concentrated entirely in the first coordinate.
B: **Incorrect:** Option B (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Change of Basis to Principal Component Coordinates (correct choice: A).
C: **Incorrect:** Option C (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Change of Basis to Principal Component Coordinates (correct choice: A).
D: **Incorrect:** Option D (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Change of Basis to Principal Component Coordinates (correct choice: A).
Based on the above data, answer the given subquestions.
If (x2, y2) is the mean of the second cluster after convergence, find the value of x2 + y2.
Published solution
**Answer: 9**
**Topic:** Clustering — *K-Means Converged Centroid Calculation*
### Step-by-Step Solution
**Step 1:** From Question 127, Cluster 2 consists of the three points in the first quadrant: $\{(5, 6), (4, 5), (3, 4)\}$.
**Step 2:** Compute the converged centroid $(x_2, y_2)$ of Cluster 2 by averaging the x- and y-coordinates: $$x_2 = \frac{5 + 4 + 3}{3} = \frac{12}{3} = 4, \quad y_2 = \frac{6 + 5 + 4}{3} = \frac{15}{3} = 5$$
**Step 3:** Check that no points change cluster assignment in the next iteration: every point in Cluster 1 is near $(-2, -3)$ and every point in Cluster 2 is near $(4, 5)$, so K-means has converged.
**Step 4:** Sum the coordinates of Cluster 2's final mean: $$x_2 + y_2 = 4 + 5 = 9$$
**Key Takeaway / Tip:** Averaging $(3,4), (4,5), (5,6)$ directly gives the middle point $(4,5)$, whose coordinate sum is $4 + 5 = 9$.
In the context of Bayesian estimation, consider a Beta prior for the parameter p of a Bernoulli
distribution:
The dataset has 8 ones and 5 zeros.
Based on the above data, answer the given subquestions.
What is the posterior?
6406532775040.
Beta(12, 8)
6406532775041.
Beta(11, 7)
6406532775042.
Br(0.5)
6406532775043.
Beta(8, 5)
Published solution
**Answer: A**
**Topic:** Bayesian Estimation — *Beta-Bernoulli Conjugate Prior and Posterior*
### Step-by-Step Solution
**Step 1:** In a Beta-Bernoulli Bayesian model, if the prior distribution over the success parameter $p$ is $\text{Beta}(\alpha, \beta)$ and we observe $k$ ones (successes) and $n - k$ zeros (failures) in $n$ independent Bernoulli trials, the conjugate posterior distribution is: $$p \mid D \sim \text{Beta}(\alpha + k, \; \beta + n - k)$$
**Step 2:** From the given data, the prior is $\text{Beta}(\alpha = 4, \beta = 3)$, and the 13 data points $[1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0]$ contain:
- Number of ones ($k$): $8$
- Number of zeros ($n - k$): $5$
**Step 3:** Update the Beta parameters: $$\alpha_{\text{post}} = 4 + 8 = 12, \quad \beta_{\text{post}} = 3 + 5 = 8$$
**Step 4:** Thus, the posterior distribution over $p$ is $\text{Beta}(12, 8)$, which corresponds to **A**.
**Key Takeaway / Tip:** With a $\text{Beta}(\alpha, \beta)$ conjugate prior, simply add the number of observed $1$s to $\alpha$ and the number of observed $0$s to $\beta$.
A: **Correct:** Option A (Beta(12, 8)) matches our derived result under Beta-Bernoulli Conjugate Prior and Posterior. With a $\text{Beta}(\alpha, \beta)$ conjugate prior, simply add the number of observed $1$s to $\alpha$ and the number of observed $0$s to $\beta$.
B: **Incorrect:** Option B (Beta(11, 7)) does not satisfy the conditions derived in the step-by-step analysis for Beta-Bernoulli Conjugate Prior and Posterior (correct choice: A).
C: **Incorrect:** Option C (Br(0.5)) does not satisfy the conditions derived in the step-by-step analysis for Beta-Bernoulli Conjugate Prior and Posterior (correct choice: A).
D: **Incorrect:** Option D (Beta(8, 5)) does not satisfy the conditions derived in the step-by-step analysis for Beta-Bernoulli Conjugate Prior and Posterior (correct choice: A).
In the context of Bayesian estimation, consider a Beta prior for the parameter p of a Bernoulli
distribution:
The dataset has 8 ones and 5 zeros.
Based on the above data, answer the given subquestions.
Published solution
**Answer: 0.6**
**Topic:** Bayesian Estimation — *Posterior Mean of a Beta Distribution*
### Step-by-Step Solution
**Step 1:** Recall that under squared error loss, the Bayesian estimator is the **posterior mean** $E[p \mid D]$.
**Step 2:** For a $\text{Beta}(\alpha_{\text{post}}, \beta_{\text{post}})$ distribution, the mean is given by: $$E[p \mid D] = \frac{\alpha_{\text{post}}}{\alpha_{\text{post}} + \beta_{\text{post}}}$$
**Step 3:** Substitute $\alpha_{\text{post}} = 12$ and $\beta_{\text{post}} = 8$ from Question 129: $$\hat{p}_{\text{Bayes}} = \frac{12}{12 + 8} = \frac{12}{20} = 0.6$$
**Key Takeaway / Tip:** Do not confuse the posterior mean $\frac{\alpha}{\alpha + \beta} = \frac{12}{20} = 0.6$ with the MAP estimate (posterior mode) $\frac{\alpha - 1}{\alpha + \beta - 2} = \frac{11}{18}$.
**Answer: A**
**Topic:** Maximum Likelihood Estimation — *Log-Likelihood of Poisson Distribution*
### Step-by-Step Solution
**Step 1:** Given the Poisson probability mass function $P(X = x_i \mid \lambda) = \frac{e^{-\lambda} \lambda^{x_i}}{x_i!}$, the likelihood function for $n$ i.i.d. observations $x_1, \dots, x_n$ is: $$L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda} \lambda^{x_i}}{x_i!}$$
**Step 2:** Take the natural logarithm of $L(\lambda)$ to convert the product into a sum: $$\ell(\lambda) = \log L(\lambda) = \sum_{i=1}^n \log\left(\frac{e^{-\lambda} \lambda^{x_i}}{x_i!}\right)$$
**Step 3:** Expand the logarithm of each term using $\log(ab/c) = \log a + \log b - \log c$: $$\ell(\lambda) = \sum_{i=1}^n \left[-\lambda + x_i \log(\lambda) - \log(x_i!)\right]$$
**Step 4:** This matches Option **A**.
**Key Takeaway / Tip:** Remember that $\log(\lambda^{x_i}) = x_i \log(\lambda)$ and $\log(e^{-\lambda}) = -\lambda$.
A: **Correct:** Option A (
Source diagram or notation unavailable. Please report this question for correction.
) matches our derived result under Log-Likelihood of Poisson Distribution. Remember that $\log(\lambda^{x_i}) = x_i \log(\lambda)$ and $\log(e^{-\lambda}) = -\lambda$.
B: **Incorrect:** Option B (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Log-Likelihood of Poisson Distribution (correct choice: A).
C: **Incorrect:** Option C (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Log-Likelihood of Poisson Distribution (correct choice: A).
D: **Incorrect:** Option D (
Source diagram or notation unavailable. Please report this question for correction.
) does not satisfy the conditions derived in the step-by-step analysis for Log-Likelihood of Poisson Distribution (correct choice: A).
Based on the above data, answer the given subquestions.
Consider a dataset that has 25 data-points. The data-point xi and its frequency is given in the
following table:
In case the table is not clear: the value 0 appears once in the dataset, the value 1 appears four
times in the dataset, and so on. Find the maximum likelihood estimate for the parameter λ of the
Poisson distribution given this dataset.
A published solution is not available for this question yet.