Deep Learning · Quiz 1 · May 2025

**1. Compose the layers:** With no biases and all activations linear, \(f(x)=W_4W_3W_2W_1x\). Every matrix product remains a linear map. **2. Collapse to one weight vector:** The product has shape 1 by 10, so it equals \(w^T\) for some \(w\in\mathbb R^{10}\). Hence \(f(x)=w^Tx\), regardless of the three hidden layers. **Answer: A — True.** A: **Correct:** Composition of linear maps is linear and the dimensions match a single 10-component weight vector. B: **Incorrect:** Additional linear layers alone do not introduce nonlinearity. C: **Incorrect:** The weights need not be numerically known to establish this structural property.

**1. Use monotonicity:** The standard sigmoid \(\sigma(t)\) is strictly increasing. The plot places \(f(x)=\sigma(x+a)\) above \(g(x)=\sigma(x+b)\) at the same finite x. **2. Compare the shifts:** Thus \(x+a>x+b\), yielding \(a>b\). Equivalently, the midpoint of f is farther left, at x=-a. **Answer: A — a > b.** A: **Correct:** A larger additive shift makes the sigmoid higher at every fixed x and moves its midpoint left. B: **Incorrect:** A smaller a would put f below g. C: **Incorrect:** Equal shifts would give coincident curves, unlike the plot.

**1. Read the original geometry:** In D1, positives lie above the parabola \(x_2=x_1^2\), and negatives lie below it on both sides. The displayed positive points lie within the surrounding negative-point convex hull, so one straight line cannot separate the original classes. **2. Transform the features:** Put \(z_1=x_1^2,z_2=x_2\). The curved boundary becomes the straight line \(z_2=z_1\); the shown positive and negative points satisfy \(z_2-z_1>0\) and \(z_2-z_1<0\), respectively. **3. Apply perceptron convergence:** A finite strictly separable D2 admits a convergent perceptron run under standard repeated training. D1 has no separating hyperplane and cannot reach an all-correct perceptron solution. **Answer: A, B — D2 is linearly separable; The perceptron learning algorithm will converge on D2.** A: **Correct:** The transformed line z2=z1 separates the classes. B: **Correct:** Finite linearly separable data satisfy the perceptron convergence theorem. C: **Incorrect:** The original coordinates require a curved separator for these displayed points. D: **Incorrect:** No perceptron weight vector can correctly classify all the original nonseparable data.

**1. Count Boolean input patterns:** Ten binary inputs have \(2^{10}=1024\) possible assignments. **2. Construct a hidden detector for each pattern:** A threshold neuron can detect one exact assignment by summing matched bits, using negative weights for positions requiring zero and adjusting its threshold. **3. Combine the relevant patterns:** The output perceptron ORs the detectors for assignments where f=1. Allocating one detector for every possible assignment therefore gives the sufficient universal bound \(k=1024\). **Answer: 1024.** **Convention note:** “Sufficient” is an upper bound, not a claim that every Boolean function needs 1024 hidden neurons. Individual functions can need far fewer.

**1. Count mini-batches per epoch:** \(1000/25=40\), with no partial batch. **2. Count updates:** Each mini-batch causes one update of every trainable parameter. Across ten epochs this gives \(40\times10=400\) updates per parameter. **Answer: 400.**

**1. Substitute the threshold:** With \(\theta=1\), the output is 1 whenever \(x_1+\cdots+x_5\ge1\), meaning at least one input is active. **2. Identify the function and separator:** This is five-input OR. The hyperplane \(\sum_i x_i=0.5\) separates the all-zero input from every other binary input, so it is linearly separable. **Answer: A, B — h is the OR function; h is a linearly separable Boolean function.** A: **Correct:** At least one active bit is exactly the OR condition. B: **Correct:** A linear threshold separates zero active bits from one or more. C: **Incorrect:** AND would require all five bits, corresponding to threshold 5. D: **Incorrect:** The function is specifically OR. E: **Incorrect:** The explicit separating hyperplane proves separability.

**1. Convert the count:** There are 32 input patterns, with 26 output zeros, so exactly 6 must satisfy \(\sum_i x_i\ge\theta\). **2. Count Hamming-weight groups:** Five bits have \(\binom5k\) assignments of weight k. For \(\theta=4\), the number of ones is \(\binom54+\binom55=5+1=6\). **3. Check uniqueness:** Threshold 3 gives 16 ones, whereas threshold 5 gives only 1. Hence the required positive integer threshold is 4. **Answer: 4.**

**1. Count the upper half:** A threshold of 3 accepts assignments with 3,4 or 5 ones. **2. Sum the counts:** \(\binom53+\binom54+\binom55=10+5+1=16\), leaving 16 zeros. **3. Use symmetry:** With an odd number of bits there is no tied midpoint; complementing each bit pairs a weight at most 2 with a weight at least 3. Thus theta=3 splits the 32 patterns equally. **Answer: 3.**

**1. Compute the hidden values:** Both input weights equal 1 and \(x=-\ln3\). Therefore \(h_1=h_2=\sigma(-\ln3)=1/(1+3)=1/4\). **2. Apply the linear output:** \(\hat y=8h_1-4h_2=8/4-4/4=1\). **Answer: 1.**

**1. Compute the hidden values:** Both input weights equal 1 and \(x=-\ln3\). Therefore \(h_1=h_2=\sigma(-\ln3)=1/(1+3)=1/4\). **2. Apply the linear output:** \(\hat y=8h_1-4h_2=8/4-4/4=1\). **3. Differentiate the chosen path:** For \(L=\frac12(\hat y-y)^2\), \(\partial L/\partial w_1=(\hat y-y)w_3h_1(1-h_1)x\). The residual is \(-1/\ln3\). **4. Cancel and simplify:** \(\left(-\frac1{\ln3}\right)8\left(\frac14\right)\left(\frac34\right)(-\ln3)=8(3/16)=1.5\). **Answer: 1.5.**

**1. Count input-to-hidden weights:** Twenty inputs connect to ten neurons, giving \(20\times10=200\). **2. Count remaining connections:** The second hidden layer adds \(10\times10=100\), and the five-class output adds \(10\times5=50\). **3. Add without biases:** The prompt explicitly ignores biases, so the total is \(200+100+50=350\). **Answer: 350.**

**1. Use equality of output weights:** With every weight equal and no biases, each output neuron receives the same weighted sum of the second hidden layer. All five logits are therefore identical for any input. **2. Apply softmax:** Equal logits produce \(p_k=1/5\) for every class. The unknown common weight and input do not affect that conclusion. **3. Evaluate cross-entropy:** For any one-hot class label, \(L=-\ln(1/5)=\ln5=1.6094379\ldots\), which rounds to 1.61. **Answer: 1.61.**

**1. Compute the gradient:** For \(f(x,y)=x^2+y^2\), \(\nabla f=2(x,y)\). **2. Apply one update:** With \(\eta=0.2\), \((x_{t+1},y_{t+1})=(1-0.4)(x_t,y_t)=0.6(x_t,y_t)\). **3. Apply the second update:** \((x_2,y_2)=0.6^2(x_0,y_0)=0.36(x_0,y_0)\). Thus alpha=0.36. **Answer: 0.36.**