Machine Learning Techniques · Quiz 2 · Jan 2025

The constrained version of **Ridge Regression** solves: \[ \min_{w \in \mathbb{R}^d} \sum_{i=1}^n (w^T x_i - y_i)^2 \quad \text{subject to } \|w\|^2 \le \theta \] Equivalently, among all candidate weight vectors \( w \) that satisfy the feasibility constraint \( \|w\|^2 \le \theta \) (with \( \theta = 10 \)), ridge regression selects the one that **minimizes the Mean Squared Error (MSE)**. **1. Check the Squared Norm \( \|w\|^2 \) and MSE for Each Candidate:** - **\( w_1 = [1, 1, 1, 1]^T \)**: \( \|w_1\|^2 = 1^2 + 1^2 + 1^2 + 1^2 = 4 \le 10 \) (Feasible), \( \text{MSE} = 2 \). - **\( w_2 = [0, 2, 1, 3]^T \)**: \( \|w_2\|^2 = 0^2 + 2^2 + 1^2 + 3^2 = 14 > 10 \) (**Infeasible**, violates \( \|w\|^2 \le 10 \)). - **\( w_3 = [1, 2, 0, 1]^T \)**: \( \|w_3\|^2 = 1^2 + 2^2 + 0^2 + 1^2 = 6 \le 10 \) (Feasible), \( \text{MSE} = 1 \). - **\( w_4 = [2, 1, 1, 2]^T \)**: \( \|w_4\|^2 = 2^2 + 1^2 + 1^2 + 2^2 = 10 \le 10 \) (Feasible), \( \text{MSE} = 8 \). **2. Select the Optimal Feasible Vector:** Among the feasible vectors \( \{w_1, w_3, w_4\} \), \( w_3 \) achieves the lowest MSE (\( \text{MSE} = 1 \)). **Answer: w3.** A: **Incorrect:** Although `w1` is feasible (`||w1||^2 = 4 <= 10`), its `MSE = 2` is higher than `w3`'s `MSE = 1`. B: **Incorrect:** `w2` has `||w2||^2 = 0 + 4 + 1 + 9 = 14 > 10`, violating the constraint `||w||^2 <= 10`. C: **Correct:** `w3` satisfies the norm constraint (`||w3||^2 = 1 + 4 + 0 + 1 = 6 <= 10`) and achieves the minimum MSE (`1`). D: **Incorrect:** Although `w4` is feasible (`||w4||^2 = 10 <= 10`), it has a much higher `MSE = 8`.

Let us count the number of independent parameters needed to learn a **Naive Bayes classifier** with \( k \) classes and \( d = 5 \) binary features: **1. Class Prior Probabilities \( P(Y = c) \):** - There are \( k \) classes, and their prior probabilities must sum to 1 (\( \sum_{c=1}^k P(Y = c) = 1 \)). - Therefore, we need to estimate **\( k - 1 \)** independent prior parameters. **2. Class-Conditional Feature Likelihoods \( P(f_j = 1 \mid Y = c) \):** - Given class \( Y = c \), the \( d = 5 \) binary features \( f_1, f_2, f_3, f_4, f_5 \) are conditionally independent. - Each binary feature \( f_j \) in class \( c \) requires **1 parameter** \( P(f_j = 1 \mid Y = c) \) (since \( P(f_j = 0 \mid Y = c) = 1 - P(f_j = 1 \mid Y = c) \)). - Across \( 5 \) features and \( k \) classes, this requires \( 5 \times k = 5k \) parameters. **3. Total Number of Parameters:** \[ \text{Total Parameters} = (k - 1) + 5k = 6k - 1 \] **Answer: 6k - 1.** A: **Incorrect:** `5k` counts only the class-conditional feature parameters and forgets the `k - 1` class prior parameters. B: **Correct:** `k - 1` class priors plus `5k` conditional feature probabilities equals `6k - 1` parameters. C: **Incorrect:** Naive Bayes assumes conditional independence of features, so the parameter count scales linearly (`6k - 1`), not exponentially. D: **Incorrect:** Does not match `(k - 1) + 5k = 6k - 1`.

In a binary decision tree node, let \( p \in [0, 1] \) be the proportion of points with label 1 (and \( 1 - p \) with label 0). **1. Behavior of Node Impurity Measures (Entropy & Gini Index):** - **Entropy:** \( H(p) = -p \log_2 p - (1 - p) \log_2(1 - p) \) - **Gini Impurity:** \( G(p) = 2p(1 - p) \) **2. How Impurity Varies with \( p \):** - When \( p = 0 \) or \( p = 1 \), the node is completely pure (all points belong to one class), so impurity is at its **minimum (\( 0 \))**. - As \( p \) increases from \( 0 \) to \( 0.5 \), impurity **increases**; as \( p \) increases from \( 0.5 \) to \( 1 \), impurity **decreases**. - At **\( p = 0.5 \)** (an equal 50-50 split of both classes), uncertainty is highest, corresponding to the **maximum impurity** (\( H(0.5) = 1 \), \( G(0.5) = 0.5 \)). **Answer: p = 0.5 correspond to the case of maximum impurity.** A: **Incorrect:** Impurity increases on `[0, 0.5]` and decreases on `[0.5, 1]`; it does not increase monotonically from 0 to 1. B: **Incorrect:** Impurity is 0 at `p = 0`, rises to a maximum at `p = 0.5`, and then decreases back to 0 at `p = 1`. C: **Incorrect:** Node impurity is a direct function of the class proportion `p`. D: **Correct:** Both entropy and Gini impurity attain their unique maximum at `p = 0.5` when the classes are equally mixed.

**Laplace smoothing** (add-1 smoothing) in Naive Bayes adds a small pseudocount (typically 1) to the numerator and the number of possible feature categories to the denominator when estimating conditional probabilities \( P(x_j = v \mid y) \). **Main Purpose:** Without smoothing, if a categorical feature value \( x_j = v \) never appears alongside class \( y \) in the training set, its maximum likelihood estimate is \( \hat{P}(x_j = v \mid y) = 0 \). Because Naive Bayes multiplies the feature likelihoods together, a single zero probability wipes out the entire posterior product \( P(y) \prod_j P(x_j \mid y) = 0 \). Laplace smoothing ensures that unseen categorical feature values receive small, **nonzero** probabilities. **Answer: To ensure that unseen categorical feature values receive small but nonzero probabilities.** A: **Incorrect:** Laplace smoothing is used for discrete/categorical count probabilities, not for the variance of continuous Gaussian features. B: **Correct:** Laplace smoothing prevents the zero-probability problem by assigning small nonzero probabilities to unseen categorical feature values. C: **Incorrect:** Laplace smoothing actually pulls probability estimates slightly toward a uniform distribution rather than increasing feature weights. D: **Incorrect:** It does not equalize all feature likelihoods; it only adds pseudocounts to avoid zero probabilities.

Let us calculate the **Information Gain** for both candidate splits on the 6 data points: - Dataset \( D \) (6 points): \( (1, 1), (-1, 1), (2, 1), (-2, 0), (3, 0), (4, 1) \). - Total positives (\( y = 1 \)): **4** (at \( x = 1, -1, 2, 4 \)). Total negatives (\( y = 0 \)): **2** (at \( x = -2, 3 \)). **1. Compare Split (a) \( x \le -2 \) vs Split (b) \( x \le -1 \):** - **Split (a) \( x \le -2 \)**: - **Yes (\( x \le -2 \))**: Contains 1 point: \( (-2, 0) \) \( \rightarrow 0 \) pos, \( 1 \) neg. Since it is **100% pure**, \( \text{Entropy}(D_{\text{Yes}}) = 0 \)! - **No (\( x > -2 \))**: Contains 5 points: \( (1, 1), (-1, 1), (2, 1), (3, 0), (4, 1) \) \( \rightarrow 4 \) pos, \( 1 \) neg. \[ \text{Entropy}(D_{\text{No}}) = -\frac{4}{5}\log_2\left(\frac{4}{5}\right) - \frac{1}{5}\log_2\left(\frac{1}{5}\right) = -0.8(-0.32) - 0.2(-2.32) = 0.256 + 0.464 = 0.72 \] - **Weighted Child Entropy for (a):** \[ \frac{1}{6}(0) + \frac{5}{6}(0.72) = 0.60 \] - **Split (b) \( x \le -1 \)**: - **Yes (\( x \le -1 \))**: Contains 2 points: \( (-2, 0) \) and \( (-1, 1) \) \( \rightarrow 1 \) pos, \( 1 \) neg (50-50 split!). \[ \text{Entropy}(D_{\text{Yes}}) = 1.0 \] - **No (\( x > -1 \))**: Contains 4 points: \( (1, 1), (2, 1), (3, 0), (4, 1) \) \( \rightarrow 3 \) pos, \( 1 \) neg. \[ \text{Entropy}(D_{\text{No}}) = -\frac{3}{4}\log_2\left(\frac{3}{4}\right) - \frac{1}{4}\log_2\left(\frac{1}{4}\right) = -0.75(-0.41) - 0.25(-2) = 0.3075 + 0.50 = 0.8075 \] - **Weighted Child Entropy for (b):** \[ \frac{2}{6}(1.0) + \frac{4}{6}(0.8075) = 0.3333 + 0.5383 = 0.8717 \] **2. Conclusion:** Since Split (a) has a much lower weighted child entropy (\( 0.60 < 0.8717 \)), Split (a) achieves the **higher Information Gain**! **Answer: (a).** A: **Correct:** Split (a) `x <= -2` isolates `(-2, 0)` into a pure node and achieves a lower weighted child entropy (`0.60` vs `0.87`), yielding higher information gain. B: **Incorrect:** Split (b) `x <= -1` groups `(-2, 0)` and `(-1, 1)` together into a maximally impure 50-50 node, resulting in lower information gain. C: **Incorrect:** All feature values, labels, and logarithm values are provided to compute the exact information gain.

In kernel regression, the **polynomial kernel** \( K(x, x') = (x^T x' + 1)^d \) applies the **kernel trick**: **1. Inner Product in a Transformed Feature Space (Option B):** By Mercer's theorem, \( K(x, x') = \phi(x)^T \phi(x') \) computes the inner product (similarity) between data points \( x \) and \( x' \) in a high-dimensional transformed feature space without explicitly computing the coordinates \( \phi(x) \). **2. Capturing Non-Linear Relationships (Option C):** By implicitly mapping the original inputs \( x \) into a higher-dimensional space of polynomial combinations up to degree \( d \), a linear model in the feature space corresponds to a **non-linear regression model** in the original input space. **Answer: Options B and C.** A: **Incorrect:** A polynomial kernel of degree `d > 1` makes the regression model non-linear in the original input space. B: **Correct:** The kernel function `K(x, x')` computes the inner-product similarity between points in the transformed feature space. C: **Correct:** Implicitly mapping inputs to a higher-dimensional polynomial feature space allows the model to learn non-linear relationships. D: **Incorrect:** Higher-degree polynomial kernels increase model flexibility and can actually *increase* variance (risk of overfitting), not reduce it. E: **Incorrect:** `K(x, x')` defines the feature representation/similarity, whereas the optimization algorithm (or closed-form solver) minimizes the loss function.

Let \( \hat{w} = (X^T X)^{-1} X^T y \) be the Ordinary Least Squares (OLS) estimate, and let \( \hat{w}_R = (X^T X + \lambda I)^{-1} X^T y \) be the Ridge Regression estimate with regularization parameter \( \lambda \ge 0 \). **1. Case \( \lambda = 0 \) (No Regularization):** - When \( \lambda = 0 \), the ridge penalty vanishes and the ridge estimator equals the least squares estimator: \( \hat{w}_R = \hat{w} \). - Therefore: \[ \frac{\|\hat{w}_R\|^2}{\|\hat{w}\|^2} = \frac{\|\hat{w}\|^2}{\|\hat{w}\|^2} = 1 \] Thus, **Option A is true** (and Option B is false). **2. Case \( \lambda = \infty \) (Infinite Regularization):** - As \( \lambda \to \infty \), the L2 penalty \( \lambda \|w\|^2 \) dominates the objective completely, shrinking all ridge coefficients toward zero: \( \lim_{\lambda \to \infty} \hat{w}_R = 0 \). - Therefore: \[ \lim_{\lambda \to \infty} \frac{\|\hat{w}_R\|^2}{\|\hat{w}\|^2} = \frac{0}{\|\hat{w}\|^2} = 0 \] Thus, **Option C is true** (and Option D is false). **Answer: Options A and C.** A: **Correct:** When `lambda = 0`, ridge regression reduces to ordinary least squares (`w_R = w`), so the ratio of their squared norms is `1`. B: **Incorrect:** For `lambda = 0`, `w_R = w != 0`, so the ratio is `1`, not `0`. C: **Correct:** As `lambda -> infinity`, the ridge estimate shrinks to the zero vector (`||w_R||^2 -> 0`), making the ratio `0`. D: **Incorrect:** Infinite regularization shrinks `w_R` to `0`, not `w`.

In **kernel regression**, the predicted value \( \hat{y} \) for a test point \( x = [2, 2]^T \) is given by: \[ \hat{y} = \sum_{i=1}^3 \alpha_i K(x_i, x) \] where \( K(x_i, x) = (1 + x_i^T x)^2 \) and \( \alpha = [0.5, \; 1.0, \; 1.5]^T \). **1. Compute \( K(x_i, x) \) for Each Training Point with \( x = [2, 2]^T \):** - **For \( x_1 = [1, 2]^T \)**: - \( x_1^T x = 1(2) + 2(2) = 2 + 4 = 6 \) - \( K(x_1, x) = (1 + 6)^2 = 7^2 = 49 \) - **For \( x_2 = [2, 3]^T \)**: - \( x_2^T x = 2(2) + 3(2) = 4 + 6 = 10 \) - \( K(x_2, x) = (1 + 10)^2 = 11^2 = 121 \) - **For \( x_3 = [3, 4]^T \)**: - \( x_3^T x = 3(2) + 4(2) = 6 + 8 = 14 \) - \( K(x_3, x) = (1 + 14)^2 = 15^2 = 225 \) **2. Compute the Weighted Sum \( \hat{y} \):** \[ \hat{y} = \alpha_1 K(x_1, x) + \alpha_2 K(x_2, x) + \alpha_3 K(x_3, x) \] \[ \hat{y} = 0.5(49) + 1.0(121) + 1.5(225) = 24.5 + 121 + 337.5 = 483 \] **Answer: 483.**

Let us compute the Naive Bayes prediction (without Laplace smoothing) for \( x_{\text{test}} = [1, 1, 0]^T \) (i.e., \( f_1 = 1, f_2 = 1, f_3 = 0 \)): **1. Class Priors from the 6 Training Samples:** - **Class \( y = 0 \)** (3 samples: \( x_1, x_2, x_3 \)): \( P(y = 0) = \frac{3}{6} = \frac{1}{2} \). - **Class \( y = 1 \)** (3 samples: \( x_4, x_5, x_6 \)): \( P(y = 1) = \frac{3}{6} = \frac{1}{2} \). **2. Conditional Feature Probabilities for \( y = 0 \) (\( x_1 = [0, 1, 1], \; x_2 = [1, 0, 1], \; x_3 = [0, 0, 1] \)):** - \( P(f_1 = 1 \mid y = 0) = \frac{1}{3} \) - \( P(f_2 = 1 \mid y = 0) = \frac{1}{3} \) - \( P(f_3 = 0 \mid y = 0) = \frac{0}{3} = 0 \) (since all three \( y = 0 \) samples have \( f_3 = 1 \)!) - Therefore, the unnormalized posterior for \( y = 0 \) is: \[ P(y = 0) P(f_1 = 1 \mid y = 0) P(f_2 = 1 \mid y = 0) P(f_3 = 0 \mid y = 0) = \frac{1}{2} \times \frac{1}{3} \times \frac{1}{3} \times 0 = 0 \] **3. Conditional Feature Probabilities for \( y = 1 \) (\( x_4 = [0, 0, 0], \; x_5 = [1, 0, 1], \; x_6 = [0, 1, 0] \)):** - \( P(f_1 = 1 \mid y = 1) = \frac{1}{3} \) - \( P(f_2 = 1 \mid y = 1) = \frac{1}{3} \) - \( P(f_3 = 0 \mid y = 1) = \frac{2}{3} \) - Therefore, the unnormalized posterior for \( y = 1 \) is: \[ P(y = 1) P(f_1 = 1 \mid y = 1) P(f_2 = 1 \mid y = 1) P(f_3 = 0 \mid y = 1) = \frac{1}{2} \times \frac{1}{3} \times \frac{1}{3} \times \frac{2}{3} = \frac{1}{27} > 0 \] Since \( \frac{1}{27} > 0 \), the Naive Bayes model predicts class **1**. **Answer: 1.**

We are given the 1D linear regression dataset \( (1, 4), (2, 7), (3, 10), (4, 13) \) with initial weight \( w_0 = 3 \) and learning rate \( \eta = 0.2 \). Note that the gradient formula given in the question is the **average** gradient (or scaled by \( 1/N \) or let us check how `7` is obtained!): **1. Compute \( X X^T \) and \( X y \):** - \( X = [1, 2, 3, 4] \), \( y = [4, 7, 10, 13]^T \). - \( X X^T = \sum_{i=1}^4 x_i^2 = 1^2 + 2^2 + 3^2 + 4^2 = 1 + 4 + 9 + 16 = 30 \). - \( X y = \sum_{i=1}^4 x_i y_i = 1(4) + 2(7) + 3(10) + 4(13) = 4 + 14 + 30 + 52 = 100 \). **2. Evaluate the Gradient \( \nabla f(w) = 2(X X^T)w - 2 X y \) at \( w = 3 \):** \[ \nabla f(3) = 2(30)(3) - 2(100) = 180 - 200 = -20 \] **3. Perform One Iteration of Gradient Descent:** \[ w_{\text{new}} = w - \eta \nabla f(w) = 3 - 0.2(-20) = 3 + 4 = 7 \] **Answer: 7.**

From Question 122, the updated weight after one iteration of gradient descent is \( w = 7 \). **Compute the Predicted Value for \( x_1 = 5 \):** Using the linear regression model \( \hat{y} = w^T x_1 \): \[ \hat{y} = 7 \times 5 = 35 \] **Answer: 35.**

Let us find the 3 nearest neighbors of \( x_{\text{test}} = [4, 0]^T \) among the 7 training points using Euclidean distance: **1. Calculate Squared Euclidean Distance \( d^2(x_i, x_{\text{test}}) = (x_{i1} - 4)^2 + (x_{i2} - 0)^2 \) for Each Point:** - \( p_1 = [1, 1]^T \) (\( y = 1 \)): \( (1 - 4)^2 + (1 - 0)^2 = 9 + 1 = 10 \) - \( p_2 = [1, -1]^T \) (\( y = 1 \)): \( (1 - 4)^2 + (-1 - 0)^2 = 9 + 1 = 10 \) - \( p_3 = [2, 1]^T \) (\( y = 1 \)): \( (2 - 4)^2 + (1 - 0)^2 = 4 + 1 = \mathbf{5} \) - \( p_4 = [2, 2]^T \) (\( y = 1 \)): \( (2 - 4)^2 + (2 - 0)^2 = 4 + 4 = 8 \) - \( p_5 = [2, -2]^T \) (\( y = 0 \)): \( (2 - 4)^2 + (-2 - 0)^2 = 4 + 4 = 8 \) - \( p_6 = [3, -3]^T \) (\( y = 0 \)): \( (3 - 4)^2 + (-3 - 0)^2 = 1 + 9 = 10 \) - \( p_7 = [4, -1]^T \) (\( y = 1 \)): \( (4 - 4)^2 + (-1 - 0)^2 = 0 + 1 = \mathbf{1} \) **2. Identify the \( k = 3 \) Nearest Neighbors:** - 1st nearest: \( p_7 = [4, -1]^T \) with \( d^2 = 1 \), label **\( y = 1 \)**. - 2nd nearest: \( p_3 = [2, 1]^T \) with \( d^2 = 5 \), label **\( y = 1 \)**. - 3rd nearest: Either \( p_4 = [2, 2]^T \) (\( y = 1 \)) or \( p_5 = [2, -2]^T \) (\( y = 0 \)) with \( d^2 = 8 \). **3. Majority Vote:** Since at least 2 of the 3 nearest neighbors (\( p_7 \) and \( p_3 \)) have label \( 1 \), the majority vote is guaranteed to be **1**. **Answer: 1.**

Let us compute the **Leave-One-Out Cross-Validation (LOOCV) error** for the 3-NN classifier across the 7 data points: - \( p_1 = (1, 1), y_1 = 1 \) - \( p_2 = (1, -1), y_2 = 1 \) - \( p_3 = (2, 1), y_3 = 1 \) - \( p_4 = (2, 2), y_4 = 1 \) - \( p_5 = (2, -2), y_5 = 0 \) - \( p_6 = (3, -3), y_6 = 0 \) - \( p_7 = (4, -1), y_7 = 1 \) **1. Find the 3 Nearest Neighbors (Excluding the Point Itself) for Each Point:** - **\( p_1(1, 1), y_1 = 1 \)**: Nearest are \( p_3(d^2=1, y=1) \), \( p_4(d^2=2, y=1) \), \( p_2(d^2=4, y=1) \) \( \rightarrow \) predicts **1** (Correct). - **\( p_2(1, -1), y_2 = 1 \)**: Nearest are \( p_5(d^2=2, y=0) \), \( p_1(d^2=4, y=1) \), \( p_3(d^2=5, y=1) \) \( \rightarrow \) labels \( \{0, 1, 1\} \) \( \rightarrow \) predicts **1** (Correct). - **\( p_3(2, 1), y_3 = 1 \)**: Nearest are \( p_1(d^2=1, y=1) \), \( p_4(d^2=1, y=1) \), \( p_2(d^2=5, y=1) \) \( \rightarrow \) predicts **1** (Correct). - **\( p_4(2, 2), y_4 = 1 \)**: Nearest are \( p_3(d^2=1, y=1) \), \( p_1(d^2=2, y=1) \), \( p_2(d^2=10, y=1) \) \( \rightarrow \) predicts **1** (Correct). - **\( p_5(2, -2), y_5 = 0 \)**: Nearest are \( p_2(d^2=2, y=1) \), \( p_6(d^2=2, y=0) \), \( p_7(d^2=5, y=1) \) \( \rightarrow \) labels \( \{1, 0, 1\} \) \( \rightarrow \) predicts **1** (**Misclassified!**). - **\( p_6(3, -3), y_6 = 0 \)**: Nearest are \( p_5(d^2=2, y=0) \), \( p_7(d^2=5, y=1) \), \( p_2(d^2=8, y=1) \) \( \rightarrow \) labels \( \{0, 1, 1\} \) \( \rightarrow \) predicts **1** (**Misclassified!**). - **\( p_7(4, -1), y_7 = 1 \)**: Nearest are \( p_5(d^2=5, y=0) \), \( p_6(d^2=5, y=0) \), and \( p_3(d^2=8, y=1) \) \( \rightarrow \) labels \( \{0, 0, 1\} \) \( \rightarrow \) predicts **0** (**Misclassified!**). **2. Compute the LOOCV Error:** Out of 7 points, **3 points** (\( p_5, p_6, p_7 \)) are misclassified: \[ \text{LOOCV Error} = \frac{3}{7} \approx 0.4286 \approx 0.43 \] **Answer: 0.43.**

From Question 125, let us see what happens if we change the labels of certain points (for instance, setting all points to have the same label \( y = 1 \), or changing the label of \( p_7 = [4, -1]^T \) from \( 1 \) to \( 0 \)): **1. Changing Only \( p_7 \)'s Label from \( 1 \) to \( 0 \):** - If \( y_7 \) is changed from \( 1 \) to \( 0 \), then the three points in the lower-right region (\( p_5, p_6, p_7 \)) all have label \( 0 \). - Then \( p_5 \)'s 3 nearest neighbors (\( p_2, p_6, p_7 \)) have labels \( \{1, 0, 0\} \rightarrow \) predicts \( 0 \) (now **correct**!). - \( p_6 \)'s 3 nearest neighbors (\( p_5, p_7, p_2 \)) have labels \( \{0, 0, 1\} \rightarrow \) predicts \( 0 \) (now **correct**!). - \( p_7 \)'s 3 nearest neighbors (\( p_5, p_6, p_3 \)) have labels \( \{0, 0, 1\} \rightarrow \) predicts \( 0 \) (now **correct**!). - Meanwhile, \( p_1, p_2, p_3, p_4 \) still have at least two \( y = 1 \) neighbors among their 3 nearest neighbors, so they also remain **correct**! - Thus, simply changing the label of \( p_7 \) reduces the LOOCV error from \( 3/7 \approx 0.43 \) all the way down to **\( 0 \)**! **Answer: Yes.** A: **Correct:** Changing the label of `p7 = [4, -1]^T` from 1 to 0 (or setting all labels to 1) reduces the 3-NN LOOCV error from `3/7` to `0`. B: **Incorrect:** The cross-validation error can easily be reduced to 0 by relabeling outlier points like `[4, -1]^T`. C: **Incorrect:** The full dataset is given, allowing exact verification that relabeling reduces the LOOCV error.