Machine Learning Techniques · Quiz 2 · Jan 2026

**1. Use the intercept normal equation:** With prediction \(\hat y_i=w_0+w_1x_i\), the squared-error objective is \(L=\sum_i(\hat y_i-y_i)^2\). At an unconstrained least-squares optimum, \[\frac{\partial L}{\partial w_0}=2\sum_i(\hat y_i-y_i)=0.\] **2. Test each strict-inequality scenario:** S₁ says every residual is strictly positive, so their sum is positive. S₂ says every residual is strictly negative, so their sum is negative. Neither can satisfy the intercept normal equation. **3. Conclude without requiring perfect fit:** Some residuals may be positive and others negative, or all may be zero. What is impossible is having all of them strictly on the same side when an intercept is fitted. **Answer: D — both S₁ and S₂ are impossible.** A: **Incorrect:** All-positive residuals contradict the zero residual sum. B: **Incorrect:** All-negative residuals also contradict the zero residual sum. C: **Incorrect:** Neither strict one-sided scenario is possible at the optimum. D: **Correct:** Both violate the necessary normal equation for the intercept.

**1. Choose by held-out performance:** Cross-validation selects the regularization strength using average validation error, not the smallest training error. The latter can favor an overfit model. **2. Compare the validation column:** For λ=0.001, 0.1, 1, 10 and 100, the validation MSEs are respectively 8.40, 4.20, 3.15, 3.20 and 11.90. The minimum is 3.15 at λ=1. **3. Interpret the tradeoff:** Very weak regularization gives low training error but poor validation error; very strong regularization underfits. Among the supplied candidates, λ=1 gives the best validation performance. **Answer: C — 1.0.** A: **Incorrect:** Its training MSE is smallest, but validation MSE 8.40 is much larger. B: **Incorrect:** Its validation MSE 4.20 exceeds 3.15. C: **Correct:** Its validation MSE 3.15 is the minimum among all candidates. D: **Incorrect:** Its validation MSE 3.20 is close but still larger; no standard-error rule is specified. E: **Incorrect:** Its validation MSE 11.90 is the largest.

**1. Translate zero training error:** Every training observation must receive its own exact label: \(h(x_i)=y_i\) for all i. An interpolating lookup rule can do this because all input locations are distinct. **2. Check why simpler rules are insufficient:** Predicting the sample mean works only when every label equals that mean; predicting zero works only when every label is zero. A linear model with d+1 coefficients need not interpolate more than d+1 arbitrary labeled points. **3. Select the guarantee:** The stated rule h(xᵢ)=yᵢ explicitly satisfies the required equality for every training point, regardless of the label pattern. **Answer: D — h(xᵢ)=yᵢ for every training observation.** **Convention note:** This guarantees interpolation on the training set, not accuracy on unseen inputs. A: **Incorrect:** A constant mean cannot fit arbitrary unequal labels exactly. B: **Incorrect:** A zero predictor cannot fit arbitrary nonzero labels exactly. C: **Incorrect:** A linear model is not guaranteed to interpolate arbitrary labels when n>d+1. D: **Correct:** It returns the exact training label at every distinct training input.

**1. Match the decision tree:** A standard tree splits one feature at a time, giving axis-aligned boundaries. Thus model 1 matches A. **2. Match the linear models:** Ridge and Lasso change the fitted coefficients through penalties, but their untransformed prediction surfaces remain linear or affine. With a classification threshold they give hyperplanes, so models 3 and 5 match B. **3. Match the flexible boundaries:** Gaussian Naive Bayes generally gives a quadratic boundary when class variances differ. A 1-nearest-neighbor rule can form a globally non-linear boundary from many Voronoi pieces. Thus models 2 and 4 belong to the non-linear category C. **Answer: B — 1→A; 3,5→B; 2,4→C.** **Convention note:** Use ordinary untransformed features and standard axis-aligned trees. Gaussian NB can become linear under equal class variances; 1-NN is piecewise linear under Euclidean distance but need not be one global hyperplane. The category non-linear/curved does not mean every 1-NN boundary piece is curved. A: **Incorrect:** Gaussian Naive Bayes and 1-NN are not generally single-hyperplane models, while Lasso remains linear. B: **Correct:** It assigns trees to axis-aligned geometry, Ridge/Lasso to hyperplanes, and Gaussian NB/1-NN to the non-linear category. C: **Incorrect:** Ridge is not restricted to axis-aligned boundaries, and Gaussian NB is not generally linear. D: **Incorrect:** It swaps the general geometry of Ridge/Lasso with Gaussian NB/1-NN.

**1. Determine the original vote:** The nearest three points to the origin occur at radii 1, 2 and 3, with labels +1, −1 and −1. The original 3-NN prediction is therefore −1. **2. Determine where an extra +1 changes the vote:** A new point closer than radius 3 enters the nearest three and displaces the farther negative point. Then the selected labels are +1,+1,−1 and the prediction changes. Outside radius 3, it does not enter the nearest three. **3. Calculate the allowed area:** Within the outer radius-4 disk, the prediction remains −1 throughout the annulus between radii 3 and 4. Its area is \[\pi(4^2-3^2)=7\pi.\] Whether the radius-3 boundary is included under a tie rule does not affect area. **Answer: A — 7π square units.** A: **Correct:** It is the area of the radius-3 to radius-4 annulus. B: **Incorrect:** It uses the wrong inner and outer squared-radius difference. C: **Incorrect:** It counts the entire outer disk, including locations that change the vote. D: **Incorrect:** It counts the radius-3 disk, where adding a positive point changes the majority.

**1. Check Dataset-1 using convex hulls:** Its opposite-label pairs occupy alternating corners of a square. The two class line segments cross at the origin, so no straight line can strictly separate the classes. **2. Check Dataset-2:** The positive point at the origin lies inside the convex hull of the four negative corner points. A half-plane containing the origin cannot exclude all four surrounding corners. **3. Give separating lines for the remaining datasets:** In Dataset-3, the green centers have x+y>0, while both red centers have x+y<0, so x+y=0 separates them. In Dataset-4, green centers are below y=0 and red centers are above it, so y=0 separates them. **Answer: C, D — Dataset-3; Dataset-4.** A: **Incorrect:** The alternating corner pattern is not linearly separable. B: **Incorrect:** The central positive point is surrounded by the negative-class convex hull. C: **Correct:** The line x+y=0 separates the shown point centers. D: **Correct:** The horizontal line y=0 separates its two classes.

**1. Count the parent labels:** The five observations (4,−1), (10,+1), (13,+1), (25,+1), (38,−1) contain three positives and two negatives. Hence p=3/5=0.6. **2. Apply binary entropy:** \[H=-0.6\log_2(0.6)-0.4\log_2(0.4)=0.970950594\ldots.\] **3. Round at the end:** To two decimal places, the parent entropy is 0.97 bits. **Answer: 0.97.**

**1. Apply the split x≤7:** The left child contains only (4,−1). The right child contains x values 10,13,25,38 with labels +1,+1,+1,−1. **2. Calculate right-child class proportions:** Three of four labels are positive, so p=3/4. Its entropy is \[H_R=-\tfrac34\log_2\tfrac34-\tfrac14\log_2\tfrac14=0.811278124\ldots.\] **3. Round:** The right-child entropy is 0.81 bits to two decimal places. **Answer: 0.81.**

**1. Compute parent and child entropies:** The parent has three positives and two negatives, giving Hₚ=0.970950594 bits. The left child is a single negative observation and has Hₗ=0; the right child has three positives and one negative, giving Hᵣ=0.811278124. **2. Weight by child sizes:** The split places 1/5 of the data on the left and 4/5 on the right. Its remaining weighted entropy is \[\tfrac15(0)+\tfrac45(0.811278124)=0.649022499.\] **3. Subtract from the parent:** \[IG=0.970950594-0.649022499=0.321928095\ldots\approx0.32.\] Entropies are weighted by sample proportions, not averaged equally between unequal-sized children. **Answer: 0.32.**

**1. Estimate the class prior:** Class 1 occurs in samples 1,3,4, so P(Y=1)=3/5. We need the joint score for X=(1,0,1,1), not the normalized posterior. **2. Estimate the four conditional factors:** Within those three class-1 samples, the needed probabilities are \[P(X_1=1\mid1)=\tfrac23,\quad P(X_2=0\mid1)=\tfrac23,\quad P(X_3=1\mid1)=\tfrac23,\quad P(X_4=1\mid1)=1.\] **3. Multiply under Naive Bayes independence:** \[P(X,Y=1)=\tfrac35\cdot\tfrac23\cdot\tfrac23\cdot\tfrac23\cdot1=\tfrac8{45}=0.177777\ldots.\] To two decimal places, enter 0.18. **Answer: 0.18.**

**1. Restrict to the augmented class-1 data:** Originally, samples 1,3,4 have class 1. Adding the all-ones class-1 sample increases this count from 3 to 4; the added class-0 sample does not enter this conditional denominator. **2. Count X₂=1 within that class:** Originally only sample 3 has X₂=1 among the class-1 samples. The new all-ones sample adds one more, so the numerator becomes 2. **3. Divide:** \[P(X_2=1\mid Y=1)=\frac24=0.5.\] Use the class-1 count 4 rather than the total augmented sample count 7. **Answer: 0.5.** **Convention note:** Follow the two explicitly specified synthetic observations. This all-ones augmentation is not the usual symmetric add-one smoothing of both outcomes of every Bernoulli feature; no additional pseudocounts are added.

**1. Count independent class probabilities:** A binary class requires one prior parameter, since P(Y=0)=1−P(Y=1). **2. Count feature probabilities conditional on class:** Each of four binary features needs one success probability in each of two classes, giving 4×2=8 parameters. Its failure probability is the complement, not an additional parameter. **3. Add:** The total is \(1+2(4)=9\). The number of training examples determines estimates, not the number of model parameters. **Answer: C — 9.** A: **Incorrect:** It omits one class-conditional probability per feature. B: **Incorrect:** It counts the eight feature parameters but omits the class prior. C: **Correct:** One prior plus eight class-conditional Bernoulli parameters gives nine. D: **Incorrect:** It treats both class priors as independent even though they sum to one.

**1. Expand the linear prediction:** If \(w^*=X\alpha=\sum_i\alpha_i x_i\), then \[(w^*)^Tx_{\rm test}=\sum_{i=1}^n\alpha_i\,x_i^Tx_{\rm test}.\] **2. Replace the inner product by the kernel:** In the kernel feature space, the inner product between training and test features is k(xᵢ,xₜₑₛₜ). Thus \[\hat y_{\rm test}=\sum_{i=1}^n\alpha_i k(x_i,x_{\rm test}).\] Each similarity is weighted by its learned coefficient. **Answer: C — Σᵢ αᵢ k(xᵢ,x_test).** A: **Incorrect:** It omits the learned coefficients αᵢ. B: **Incorrect:** It omits test-point similarities and would predict a constant for every test input. C: **Correct:** It retains both the learned coefficients and the training–test kernel similarities. D: **Incorrect:** The expansion is linear in αᵢ, not quadratic.

**1. Compute squared distances to (2,2):** For observations (2,3), (1,4), (4,2), (3,5), (5,2), (2,1), the squared distances are respectively \[1,5,4,10,9,1.\] Squared distances give the same neighbor ranking as Euclidean distances. **2. Read the three nearest labels:** The nearest observations are (2,3) with y=4.1, (2,1) with y=3.9, and (4,2) with y=5.5. **3. Average for regression:** \[\hat y=\frac{4.1+3.9+5.5}{3}=\frac{13.5}{3}=4.5.\] A KNN regressor averages numerical targets rather than taking a class majority. **Answer: 4.5.**