Deep Learning · Quiz 1 · May 2024

**1. Use the standard zero initialization:** With w=(0,0), every dot product is zero and the stated rule predicts class 1 when wᵀx≥0. Hence the first 50 positive samples x=(a,a) are already correctly classified and cause no updates. **2. Update on the first negative sample:** At x=(−a,−a), the initial prediction is still 1 but the target is 0. The perceptron update \(w\leftarrow w+\eta(y-\hat y)x\) changes w to (ηa,ηa), with η>0. **3. Verify convergence:** Positive samples then have dot product 2ηa²>0 and negative samples have −2ηa²<0. All later samples are correct, so there is exactly one update. **Answer: 1.** **Convention note:** The source does not explicitly state initial weights. The answer 1 uses the customary w=0 initialization reflected in the key. A different initialization can give a different update count, including zero for a separator that is already correct.

**1. Apply the exact threshold inequality:** The rule predicts 1 when the four-input sum is strictly greater than θ. The two positive examples each have sum 3, so they require θ<3. **2. Enforce the negative example:** Its input sum is 2 and the output must be 0. Since equality is assigned to class 0, it requires θ≥2. **3. Combine and minimize:** The admissible interval is \(2\le\theta<3\). Its minimum is θ=2; this gives 3>2 for both positive samples and 2≯2 for the negative sample. **Answer: 2.**

**1. Identify the output-weight derivative:** For a sigmoid output and binary cross-entropy, \(\partial L/\partial z=\hat y-y\), where z is the output pre-activation. **2. Include the selected hidden activation:** W₂₀₀ connects hidden neuron 1 to the output, so \[\frac{\partial L}{\partial W_{200}}=(\hat y-y)h_1.\] The first supplied hidden component is 1. **3. Substitute:** With target y=1 and ŷ=0.9, the gradient is (0.9−1)(1)=−0.1. An additional sigmoid derivative must not be multiplied after the BCE simplification. **Answer: -0.1.** **Convention note:** The supplied hidden vector contains a component 2, which is incompatible with a logistic hidden activation. This formal requested derivative uses only its stated first component 1 and the supplied output probability.

**1. Write a positive-example update:** If a positive x is misclassified, a perceptron update is w′=w+ηx. Its new score on that same sample is \[w\prime{}^Tx=w^Tx+\eta\|x\|^2.\] **2. Explain what is guaranteed:** The score moves in the correct direction, but a finite increment need not cross the decision threshold. For example, in one dimension with x=1,w=−10 and η=1, the new score is −9 and is still negative. **3. Conclude:** A correction step is not guaranteed to fix that individual example immediately. Eventual convergence for separable data concerns a sequence of updates, not necessarily one update. **Answer: B — NO.** A: **Incorrect:** The update improves the signed score but may leave it on the wrong side. B: **Correct:** A single finite step need not cross the boundary, as the explicit example shows.

**1. Compute the directional change:** For a small step εd, \[L(w+\epsilon d)=L(w)+\epsilon\nabla L(w)^Td+O(\epsilon^2).\] The first-order term is \(\epsilon\|\nabla L\|\|d\|\cos\theta\). **2. Use the angle:** For 90°<θ<180°, cosθ<0, so d is a descent direction and sufficiently small positive steps decrease the loss. **3. Compare equal-length local steps:** The most negative first-order change occurs at θ=180°. An oblique descent direction gives a smaller first-order reduction, explaining the intended option B. **Answer: B — The loss function will decrease, although not to the maximum possible extent..** **Convention note:** The decrease is a local statement for sufficiently small steps. Convexity alone does not guarantee decrease for an arbitrary finite step length; overshooting can increase even a convex loss. A: **Incorrect:** The local first-order change is negative, not positive, for an obtuse angle and a sufficiently small step. B: **Correct:** It describes the intended local, equal-step-length descent comparison. C: **Incorrect:** At a nonzero gradient and a strictly obtuse angle, the directional derivative is negative rather than zero. D: **Incorrect:** The directional derivative is determined, though a large finite step would require further step-size information.

**1. State the first-order approximation:** Near x₀, \[f(x_0+\Delta)\approx f(x_0)+\nabla f(x_0)^T\Delta.\] The gradient describes local change; higher-order terms measure curvature and approximation error. **2. Choose a local descent move:** For a fixed small step length, Δ opposite the gradient makes the linear term as negative as possible. Repeating such local updates is the basis of gradient descent. **3. Limit the claim:** The first-order approximation is not generally accurate over the whole input space and does not itself guarantee a global minimum for an arbitrary objective. **Answer: B — Gradient descent seeks to minimize the first-order term of the Taylor series .** A: **Incorrect:** It incorrectly treats a local first-order approximation as a global function model. B: **Correct:** It correctly links the gradient step to improving the local first-order approximation. C: **Incorrect:** Ordinary gradient descent does not compute the entire Taylor series or universally guarantee a global minimum.

**1. Construct hidden detectors:** There are 2ⁿ possible n-bit input patterns. A perceptron can detect one exact pattern using positive weights for its 1 bits, negative weights for its 0 bits and a suitable threshold. **2. Combine the required patterns:** Put a detector in one hidden layer for each pattern on which the desired Boolean function equals 1. One output perceptron implements OR over those detectors. **3. State a universal sufficient architecture:** One hidden layer with capacity for 2ⁿ detectors, followed by one output perceptron, can represent every Boolean truth table. Fewer detectors suffice for many particular functions. **Answer: D — one hidden layer with capacity for 2ⁿ perceptrons.** **Convention note:** The question’s at least wording is imprecise: 2ⁿ is a sufficient worst-case construction size, not a necessary hidden-unit lower bound for each Boolean function. A: **Incorrect:** The stated n-neuron-per-layer size is not the general truth-table construction being asked for. B: **Incorrect:** n hidden layers are not needed; one hidden layer with enough pattern detectors suffices. C: **Incorrect:** Only n hidden detectors are not sufficient for the general one-hidden-layer truth-table construction. D: **Correct:** One hidden layer with up to 2ⁿ pattern detectors is a universal sufficient construction.

**1. Build the left rise:** The term \(2\sigma(5x)=2/(1+e^{-5x})\) rises from 0 to 2 near x=0. **2. Build the right fall by subtraction:** The term \(2\sigma(5(x-5))=2/(1+e^{-5x+25})\) rises near x=5. Subtracting it turns the earlier rise back down, leaving a plateau near 2 between the transitions. **3. Match the complete tower:** \[f(x)=\frac2{1+e^{-5x}}-\frac2{1+e^{-5x+25}}.\] Far left both terms are near 0; between 0 and 5 only the first is near 2; far right the two approximately cancel. **Answer: B — f(x)=2σ(5x)−2σ(5x−25).** A: **Incorrect:** It reverses the subtraction and has unit rather than height-2 scaling. B: **Correct:** It creates a positive height-2 plateau between rising edges centered at 0 and 5. C: **Incorrect:** It creates a negative tower by subtracting in the wrong order. D: **Incorrect:** It has the right orientation but a plateau height near 1 rather than 2.

**1. Describe a single threshold neuron:** Its output is decided by whether a weighted input sum exceeds a threshold. It can implement linearly separable rules such as AND and OR, but its decision boundary is a single hyperplane. **2. Explain XOR’s obstruction:** The two XOR-positive corners lie diagonally opposite in the binary square, as do the negative corners. No single line separates the two classes, so one such neuron cannot exactly represent XOR. **3. Explain why a network helps:** Combining multiple threshold units creates intermediate features and multiple decision regions. For example, hidden units can detect separate XOR-positive patterns and an output unit can OR them. **Answer: A, D — The MP neuron model can represent a wide range of boolean functions (not ; The representation power of the MP neuron model increases when multiple .** A: **Correct:** A single threshold unit represents many Boolean rules, but not every truth table. B: **Incorrect:** A single MP/threshold neuron cannot fit every arbitrary Boolean truth table. C: **Incorrect:** XOR is not linearly separable, so changing a single threshold boundary cannot represent it. D: **Correct:** Networks can combine intermediate decisions and exceed one neuron’s representational power.

**1. Count the input-to-hidden connections:** The diagram has three input neurons and two hidden neurons, fully connected, giving 3×2=6 weights. **2. Count hidden-to-output connections:** Two hidden neurons connect to three output neurons, giving 2×3=6 more weights. **3. Add without biases:** \[P=3(2)+2(3)=6+6=12.\] Activation functions themselves do not add learned coefficients. **Answer: 12.**

**1. Compute output logits:** With h₁=(0.5,1)ᵀ and \(W_2=\begin{pmatrix}1&0\\1&1\\1&0\end{pmatrix}\), the pre-activation vector is z=W₂h₁=(0.5,1.5,0.5)ᵀ. **2. Apply three-class softmax:** Subtract 0.5 from all logits, leaving (0,1,0) without changing the probabilities. Hence \[\hat y=\frac{(1,e,1)^T}{2+e}\approx(0.211942,0.576117,0.211942)^T.\] **3. Use the one-hot target:** y=(0,1,0)ᵀ selects the second probability. Natural-log cross-entropy is \[L=-\ln\frac{e}{2+e}=\ln(2+e)-1=0.551444714\ldots\approx0.55.\] **Answer: 0.55.**

**1. Compute output logits:** With h₁=(0.5,1)ᵀ and \(W_2=\begin{pmatrix}1&0\\1&1\\1&0\end{pmatrix}\), the pre-activation vector is z=W₂h₁=(0.5,1.5,0.5)ᵀ. **2. Apply three-class softmax:** Subtract 0.5 from all logits, leaving (0,1,0) without changing the probabilities. Hence \[\hat y=\frac{(1,e,1)^T}{2+e}\approx(0.211942,0.576117,0.211942)^T.\] **3. Differentiate with respect to probabilities, not logits:** For \(L=-\sum_iy_i\ln\hat y_i\), \(\partial L/\partial\hat y_i=-y_i/\hat y_i\). With y=(0,1,0), the gradient is \((0,-1/\hat y_2,0)^T\). **4. Sum its entries:** \[\sum_i\frac{\partial L}{\partial\hat y_i}=-\frac{2+e}{e}=-(1+2e^{-1})=-1.735758882\ldots\approx-1.74.\] Using the logit derivative ŷ−y instead would answer a different question. **Answer: -1.74.**

**1. Identify the decay frequency:** The learning rate is decreased by 10% after each epoch, not after each mini-batch. Thus the update count 200 within the first epoch does not trigger 200 decays. **2. Apply one completed epoch’s reduction:** Starting from η₀=0.1, the beginning of epoch 2 uses \[\eta_1=0.1(1-0.10)=0.1(0.9)=0.09.\] **Answer: A — 0.09.** A: **Correct:** One 10% reduction gives 0.09. B: **Incorrect:** It does not follow the specified 10% epoch reduction. C: **Incorrect:** It is far smaller than one 10% reduction of 0.1. D: **Incorrect:** It would require a much more aggressive decay than specified.

**1. Count mini-batches per epoch:** One complete pass through 1000 examples with batches of 20 uses 1000/20=50 updates. **2. Divide total updates by that count:** \[\text{epochs}=10000/50=200.\] Equivalently, the model processes 10000×20=200000 example presentations, or 200 full passes. **Answer: D — 200 epochs.** A: **Incorrect:** Five epochs would require only 250 mini-batches. B: **Incorrect:** Ten epochs would require only 500 mini-batches. C: **Incorrect:** Twenty epochs would require only 1000 mini-batches. D: **Correct:** 10000 updates divided by 50 per epoch equals 200 epochs.

**1. Express a batch gradient as an average:** For independent examples with individual gradient covariance Σ, a B-example mean has covariance approximately Σ/B. **2. Compare the batch sizes:** Increasing B from 32 to 128 multiplies the sample count by four, so the idealized gradient variance falls to about one quarter. Finite-population sampling without replacement also lowers variance as batch size grows. **3. State the likely effect:** The per-update gradient estimate becomes less noisy. This statement concerns estimate variance, not a guarantee that every other training or generalization metric improves. **Answer: B — Decreases the variance.** A: **Incorrect:** A larger average generally reduces sampling noise rather than increasing it. B: **Correct:** More examples per gradient estimate usually reduce its variance. C: **Incorrect:** Batch size directly affects sampling variance. D: **Incorrect:** No such reversal is implied by the stated change.

**1. Define a level set:** A contour at level c is the set \(\{x:f(x)=c\}\). Every point on that contour has the same function value. **2. Relate contours to gradients:** For a differentiable function with nonzero gradient, the gradient is perpendicular to a contour and points toward increasing values. The contour itself is not the gradient or a line of maximum increase. **Answer: B — Lines connecting points where the function has the same output value.** A: **Incorrect:** Rapid increase concerns gradient magnitude and direction, not the definition of a contour. B: **Correct:** Equal output values define a contour or level set. C: **Incorrect:** Contours exist at many values, not only maxima and minima. D: **Incorrect:** The gradient is a vector perpendicular to the contour where the level set is smooth.