Machine Learning Techniques · Quiz 2 · Sep 2024

**1. Treat columns as observations:** The feature vectors are \(x_1=(1,0)^T,x_2=(0,1)^T,x_3=(0,0)^T\). The Gram entry is \(K_{ij}=(1+x_i^Tx_j)^2\). **2. Calculate diagonal and cross terms:** \(K_{11}=K_{22}=4\), while \(K_{33}=1\). Distinct nonzero observations have dot product zero, and every dot product with \(x_3\) is zero, so every off-diagonal entry is 1. **3. Assemble the symmetric matrix:** \[K=\begin{pmatrix}4&1&1\\1&4&1\\1&1&1\end{pmatrix}.\] Symmetry is a necessary cross-check for this real-valued kernel. **Answer: D — the Gram matrix with rows (4,1,1), (1,4,1), (1,1,1).** A: **Incorrect:** It is nonsymmetric and puts an incorrect value on the first diagonal entry. B: **Incorrect:** It is nonsymmetric and has incorrect self-kernel entries. C: **Incorrect:** It assigns 4 to a cross-kernel whose dot product is zero and is nonsymmetric. D: **Correct:** Every entry follows directly from the stated polynomial kernel.

**1. Solve the kernel interpolation system:** With \(y=(2,1,2)^T\), solve \[\begin{pmatrix}4&1&1\\1&4&1\\1&1&1\end{pmatrix}\alpha=\begin{pmatrix}2\\1\\2\end{pmatrix}.\] Subtracting the third equation from the first gives \(3\alpha_1=0\); subtracting it from the second gives \(3\alpha_2=-1\). Therefore \(\alpha=(0,-1/3,7/3)^T\). **2. Compute similarities to the test point:** For \(x_*=(1,-1)^T\), the three dot products with the training observations are 1, −1 and 0. Hence \(k_*=(4,0,1)^T\). **3. Predict and round:** \[\hat y_*=k_*^T\alpha=4(0)+0(-1/3)+7/3=2.333\ldots.\] Rounding to two decimal places gives 2.33. **Answer: 2.33.**

**1. Use the binary-feature complement:** Conditional on class 1, the second feature has only outcomes 0 and 1. Their conditional probabilities sum to one. **2. Substitute the supplied probability:** \[P(f_2=0\mid y=1)=1-P(f_2=1\mid y=1)=1-0.05=0.95.\] The class prior is not needed for a probability already conditioned on the class. **Answer: 0.95.**

**1. Form the Naive Bayes scores:** Conditional independence allows \(s_c=P(y=c)P(f_1=1\mid c)P(f_2=0\mid c)\). Compare these unnormalized posterior scores because the evidence denominator is common. **2. Evaluate both classes:** \[s_0=0.6(0.25)(1-0.35)=0.0975,\qquad s_1=0.4(0.15)(1-0.05)=0.057.\] **3. Choose the larger posterior:** Since \(0.0975>0.057\), predict class 0. If normalized, the class-0 posterior is \(0.0975/(0.0975+0.057)\approx0.6311\). **Answer: 0.**

**1. Leave out the validation observation:** The held-out point is (2,2), leaving training points (1,2) and (0,1). Fit the affine model \(\hat y=w_0+w_1x\) only to those two points. **2. Solve the two equations:** The point (0,1) gives \(w_0=1\); then (1,2) gives \(1+w_1=2\), so \(w_1=1\). **3. Compute the requested norm:** \(\|w\|^2=w_0^2+w_1^2=1+1=2\). The held-out prediction would be 3, but its error is not the quantity asked. **Answer: 2.**

**1. Compare squared Euclidean distances:** Squaring preserves neighbor order. From test point (1,1,2), the squared distances to \(x_1,\ldots,x_5\) are respectively \[2,\ 17,\ 6,\ 5,\ 30.\] **2. Select the three nearest observations:** The order begins \(x_1,x_4,x_3\), with labels 0, 0 and 1. For example, the first squared distance is \((1-1)^2+(2-1)^2+(3-2)^2=2\). **3. Vote:** Two of the three neighbors vote for class 0, so the 3-nearest-neighbor prediction is 0. No distance or voting tie occurs. **Answer: 0.**

**1. Count the class prior:** In binary classification, one probability \(\pi=P(y=1)\) determines both priors because \(P(y=0)=1-\pi\). This contributes one independent parameter. **2. Count continuous-feature parameters:** The Gaussian study-hours feature has one mean per class, contributing two parameters. Its variance is shared between classes, contributing only one more. **3. Count binary-feature parameters:** Reviewing is Bernoulli, with one success probability for each class; complements are determined automatically. Thus the total is \[1+2+1+2=6.\] **Answer: 6.** **Convention note:** This is the intended Gaussian continuous-feature/Bernoulli binary-feature Naive Bayes model. Equal variances alone do not specify an arbitrary continuous probability distribution; the six-parameter count assumes Gaussian class-conditional study hours.

**1. Compare tree splits by the specified criterion:** In an entropy-based decision tree, the preferred split maximizes information gain, the parent entropy minus the weighted child entropy. A split with larger gain is better under this criterion. **2. Check the model-building statements:** KNN retains training observations for prediction; it does not dispense with them. A threshold outside the observed feature range can put every observation on one side, so a binary test does not guarantee two nonempty children. **3. Control model complexity:** Maximum depth is a hyperparameter that can be tuned using cross-validation. Setting it too low limits the patterns the tree can represent and can produce underfitting. **Answer: A, D, E — information-gain comparison, depth tuning and shallow-tree underfitting.** **Convention note:** A assumes better means better by entropy-based information gain, not every possible tree criterion. D is the intended recommendation to tune depth using cross-validation; its wording has to is not a mathematical claim that no other valid tuning method exists. A: **Correct:** Under the information-gain split criterion, a better split has greater gain. B: **Incorrect:** KNN predictions depend on stored training examples or an equivalent search structure. C: **Incorrect:** If every observed value is at most the threshold, the other child is empty. D: **Correct:** Cross-validation is an appropriate way to tune the depth hyperparameter. E: **Correct:** A shallow depth limit can prevent the tree from representing the true decision boundary.

**1. Interpret the requested minimum:** Including each query observation itself guarantees one same-class vote, but the other two nearest points still matter. Thus self-inclusion alone does not guarantee zero errors for every dataset. **2. Construct a dataset attaining zero:** Put all 50 class-0 observations in a small cluster and all 50 class-1 observations in another distant cluster. Every observation then has itself and two same-class points as its three nearest neighbors. **3. Establish minimality:** Every point is correctly classified in that construction, giving zero training errors. Error counts cannot be negative, so the minimum possible count is 0. **Answer: 0.**

**1. Relate k to local sensitivity:** Reducing k makes each prediction depend on fewer nearby observations, so an outlier can dominate a smaller neighborhood and create an isolated region or a more irregular boundary. **2. Decide the direction of smoothing:** Increasing k generally averages over more observations and smooths local fluctuations. The statement attributes that effect to decreasing k, so it is false. **Answer: B — False.** A: **Incorrect:** Smaller k generally increases sensitivity to outliers rather than smoothing the boundary. B: **Correct:** Decreasing k generally makes the boundary more local and irregular.

**1. Separate training fit from estimation risk:** Maximum likelihood under unit-variance Gaussian noise is ordinary least squares. Ridge introduces bias but can reduce the variance of the estimated coefficients, lowering their expected squared error. **2. Demonstrate that some positive penalty helps:** In one dimension let \(X^TX=s>0\) and the true coefficient be \(w_*\). The ridge estimator is \(\hat w_\lambda=(s+\lambda)^{-1}X^Ty\), with risk \[R(\lambda)=\frac{\lambda^2w_*^2+s}{(s+\lambda)^2}.\] At zero penalty, \(R(0)=1/s\) and \(R\prime(0)=-2/s^2<0\). Thus a sufficiently small positive penalty lowers risk. **3. Interpret the quantifier:** The claim is for some positive penalty, not every penalty. The same variance-reduction argument applies along the eigen-directions of a full-rank design. **Answer: A — True.** **Convention note:** MSE is interpreted as expected parameter-estimation error, as in the bias–variance comparison. Ordinary least squares already minimizes unpenalized training squared residuals; ridge cannot strictly improve that training objective. A: **Correct:** A suitable positive ridge penalty can improve expected coefficient MSE through the bias–variance tradeoff. B: **Incorrect:** It confuses the smallest training residual with the smallest expected estimation error.

**1. Write the likelihood penalty:** Unit-variance Gaussian errors give negative log likelihood \(\frac12\sum_i(y_i-w^Tx_i)^2\), apart from constants independent of w. **2. Write the prior penalty:** A Laplace density with scale \(b=2/\lambda\) is proportional to \(e^{-|w_j|/b}\). Independent coefficients therefore contribute \((\lambda/2)\sum_j|w_j|\) to the negative log prior. **3. Combine and rescale:** Maximizing the posterior is equivalent to minimizing the negative log posterior. Multiplying by 2 gives \[\min_w\;\sum_i(y_i-w^Tx_i)^2+\lambda\|w\|_1.\] This is the Lasso objective. **Answer: C — minimize squared residuals plus λ‖w‖₁.** A: **Incorrect:** A squared L2 penalty corresponds to a Gaussian prior, not the given Laplace prior. B: **Incorrect:** The negative-log-posterior expression must be minimized, and its penalty is not L2. C: **Correct:** The Laplace prior produces precisely the L1 penalty with coefficient λ after rescaling. D: **Incorrect:** Maximizing this error-plus-penalty expression has the wrong optimization direction.

**1. Identify what class entropy needs:** Binary label entropy is \(H(Y)=-p\log_2p-(1-p)\log_2(1-p)\). It depends only on class proportions, not on feature values or the total sample count. **2. Apply the given proportions:** A 60%/40% class split supplies \(p=0.6\), so the entropy is completely determined. **Answer: A — Yes.** A: **Correct:** The two class proportions provide everything needed for label entropy. B: **Incorrect:** Individual features and observations are unnecessary once the class distribution is known.

**1. Use Shannon entropy in bits:** \(H(Y)=-\sum_c p_c\log_2p_c\). The two class probabilities are 0.6 and 0.4. **2. Substitute and evaluate:** \[H=-0.6\log_2(0.6)-0.4\log_2(0.4)=0.442179\ldots+0.528771\ldots=0.970950\ldots.\] **3. Round:** To three decimal places the entropy is 0.971 bits. It is below 1 bit, as expected for a binary distribution that is not exactly balanced. **Answer: 0.971.**

**1. Scale the targets:** For least squares \(\hat w=(X^TX)^{-1}X^Ty\). Replacing y by 2y replaces \(\hat w\) by \(2\hat w\), because the expression is linear in y. **2. Scale all inputs:** Replacing X by 2X gives \[\hat w_{\rm new}=((2X)^T(2X))^{-1}(2X)^Ty=\tfrac12(X^TX)^{-1}X^Ty=\tfrac12\hat w.\] **3. Check prediction consistency:** Halving the coefficient vector compensates for doubling the feature vector, leaving fitted values unchanged. Doubling targets instead requires doubled fitted values. **Answer: A, C — doubling targets doubles coefficients; doubling inputs halves coefficients.** **Convention note:** Assume a unique least-squares solution, or use the minimum-norm solution. For a rank-deficient problem the corresponding entire solution sets scale in the same way. The input-scaling claim assumes every modeled input column is scaled, with no separately unscaled intercept. A: **Correct:** The least-squares coefficient vector is linear in the target vector. B: **Incorrect:** Doubling targets doubles the coefficients rather than halving them. C: **Correct:** A factor of two in all inputs is offset by a factor of one half in the coefficients. D: **Incorrect:** Doubling both inputs and coefficients would multiply predictions by four.