Deep Learning Practice · Quiz 2 · Jan 2025

**1. Read the waveform length:** torch.randn(12000) creates one audio waveform with 12,000 sample values. The sampling rate 16,000 means samples per second; it does not change the number of samples to 16,000. **2. Follow preprocessing:** The processor normalizes the waveform and creates the requested tensor batch. There is only one waveform, so padding to the batch maximum does not extend it. **3. State the axes:** input_values has shape (batch size, sample count), hence (1, 12000). The waveform duration would be \(12000/16000=0.75\) seconds. **Answer: C — (1, 12000).** A: **Incorrect:** The sampling rate is not the waveform length; the input contains only 12,000 samples. B: **Incorrect:** This output is a processed waveform, before the model’s convolutional sequence reduction. Its length is known from the input. C: **Correct:** There is one waveform containing 12,000 samples. D: **Incorrect:** return_tensors="pt" requests a PyTorch tensor and is supported.

**1. Distinguish a path from audio samples:** The string audio.wav identifies a file location. It does not contain the numeric waveform that the Wav2Vec2 feature extractor expects. **2. Correct the preprocessing sequence:** Load/decode the audio with an audio library, convert it to a suitable mono numeric waveform, and resample to the model’s required rate if necessary. Then pass the waveform array and its actual sampling rate to processor. **3. Retain the tensor request:** return_tensors="pt" is valid for a PyTorch model. Changing the returned container type or indexing the result cannot fix an undecoded input path. **Answer: D — processor cannot process raw audio file paths, only arrays of audio samples..** A: **Incorrect:** PyTorch tensor output is valid; changing to NumPy does not load the file. B: **Incorrect:** Removing the batch dimension does not repair the invalid input. Models normally expect batched input_values. C: **Incorrect:** The waveform’s sampling_rate is appropriately supplied when processing the waveform. D: **Correct:** The processor call requires decoded numeric samples rather than the file-path string shown.

**1. Define word error rate:** \[\mathrm{WER}=\frac{S+D+I}{N},\] where S is the number of substituted words, D deleted words, I inserted words, and N the number of reference words. Counts are accumulated over the supplied reference/hypothesis pairs. **2. Align the first pair:** Reference hello world becomes hypothesis hello word. There is one substitution, world to word, and 2 reference words. **3. Align the second pair:** Reference this is a test becomes this test. The words is and a are deleted, giving 2 deletions and 4 reference words. **4. Compute and format:** Total errors are \(1+2=3\) and reference length is \(2+4=6\). Thus \[\mathrm{WER}=3/6=0.50.\] The formatting instruction prints WER: 0.50. **Answer: B — WER: 0.50.** A: **Incorrect:** 0.25 undercounts the three word-level edit operations. B: **Correct:** One substitution plus two deletions over six reference words gives 0.50. C: **Incorrect:** 0.33 would account for only two errors among six words. D: **Incorrect:** 0.67 would require four errors among six reference words.

**1. Identify the input and output:** Whisper receives an audio representation and generates a sequence of text tokens. Its encoder represents audio; its decoder predicts the output text. **2. State the conditional objective:** With audio X and transcript tokens y, training maximizes \[P(y\mid X)=\prod_t P(y_t\mid y_{<t},X).\] The decoder uses previous transcript tokens together with encoded audio. **3. Classify the full architecture:** This is sequence-to-sequence learning. The decoder is autoregressive, but calling the entire speech-conditioned system only a decoder-only causal language model omits the audio encoder and conditional sequence mapping. **Answer: B — Sequence-to-Sequence Learning (S2S).** A: **Incorrect:** Its decoder is causal, but the complete audio-to-text training task is encoder-decoder sequence-to-sequence learning. B: **Correct:** The model learns a conditional mapping from an audio sequence to a text sequence. C: **Incorrect:** The principal objective does not reconstruct randomly masked text tokens. D: **Incorrect:** Whisper’s supervised transcription objective differs from Wav2Vec2’s contrastive pretraining objective.

**1. Define a logit:** A logit is a model’s raw score before probability normalization. At each output time step, an ASR head produces one score for each token in its configured output vocabulary. **2. Distinguish scores from probabilities:** Softmax converts scores z into probabilities, \[p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}.\] The .logits field contains z, not the normalized p values. **3. Identify the labels:** A CTC vocabulary can contain characters or other tokenizer units and a blank symbol. It is not necessarily a phoneme inventory. **Answer: D — Raw unnormalized scores for each token in the vocabulary..** A: **Incorrect:** Logits are not normalized probabilities, and the vocabulary is not necessarily phonemic. B: **Incorrect:** This would describe softmax-normalized logits rather than the raw .logits field. C: **Incorrect:** The raw-score part is correct, but restricting every vocabulary item to a phoneme is unjustified. D: **Correct:** The output contains raw scores over the configured token vocabulary.

**1. Prepare the paired training examples:** Decode audio into numeric waveform arrays and ensure the required sampling rate. Prepare the matching transcript for each waveform. **2. Prepare model inputs and targets:** Process the waveform into input_values. Tokenize the transcript into target token IDs. Padding and target masking allow variable-length examples to be batched. **3. Train the end-to-end recognizer:** Wav2Vec2 learns waveform features and contextual representations; its CTC head is fine-tuned on the speech-text pairs. A separate fixed MFCC or spectrogram-to-phoneme pipeline is not required. **Answer: C — Load data, prepare waveform arrays and tokenized labels, then train on speech-text pairs.** A: **Incorrect:** A spectrogram CNN pipeline is a different setup from the standard raw-waveform Wav2Vec2 workflow. B: **Incorrect:** Wav2Vec2 learns its own waveform features rather than requiring the stated MFCC-to-phoneme pipeline. C: **Correct:** It includes the dataset, numeric speech input, tokenized transcript labels and paired training. D: **Incorrect:** Speech is not converted into text embeddings before recognition in this workflow.

**1. Separate preprocessing from modeling:** Wav2Vec2Processor combines waveform preprocessing and text tokenization. The feature extractor normalizes numeric samples and pads a batch when requested. The neural Wav2Vec2Model consumes those input_values and computes learned audio representations. **2. Explain masked pretraining:** Wav2Vec2 masks portions of latent audio features and learns from the surrounding context, using a contrastive objective in its original pretraining procedure. **3. Compare with GPT:** The contextual audio encoder is not restricted to left-to-right next-token prediction. Its masked representation learning is therefore different from GPT’s causal language-model objective. **Answer: B, D — The Wav2Vec2Processor takes a raw audio waveform and converts it into ; Like BERT, Wav2Vec2 applies masked learning to the audio representations..** A: **Incorrect:** The processor/feature extractor prepares input_values; Wav2Vec2Model consumes them to compute neural representations. B: **Correct:** The processor performs the indicated waveform normalization and requested padding. C: **Incorrect:** The model is a deep neural network, but its processor is preprocessing/tokenization logic rather than another deep network. D: **Correct:** Its pretraining masks latent audio representations, giving a broad analogy with BERT’s masked learning. E: **Incorrect:** The contextual encoder does not use GPT-style strictly causal left-to-right pretraining.

**1. Separate transcription from diarization:** Transcription answers what was said. Diarization assigns who spoke at each time. Accurate words can coexist with incorrect speaker boundaries or identities. **2. Analyze short segments:** Very short segments contain little speaker-specific evidence. Their embeddings can fluctuate and be clustered inconsistently, producing false speaker changes. **3. Analyze timestamp alignment:** Even correct diarization segments can be assigned to the wrong words if their times and the transcript times are misaligned. Aligning both timelines is therefore necessary when merging results. **Answer: A, C — The speaker segments are too short, causing the clustering model to ; The timestamps from Whisper do not align with the diarization output, leading .** **Convention note:** The word definite is too strong: the observations alone do not prove either cause. A and C identify plausible mechanisms to investigate, as intended by the key; diagnostic audio/timestamps would be needed to establish the actual cause. A: **Correct:** Short segments can cause unstable speaker embeddings and clustering, making this a valid potential cause. B: **Incorrect:** Whisper does not itself provide general speaker diarization, but a separate diarization pipeline does not assign labels randomly by definition. C: **Correct:** Misalignment between word/transcript times and speaker intervals can introduce spurious switches in the merged output. D: **Incorrect:** The prompt says the words are accurate, and diarization normally uses acoustic evidence rather than needing misrecognized words.

**1. Read the model construction:** WhisperModel("large-v2") loads the specified pretrained Whisper model using faster-whisper. **2. Follow the function:** transcribe returns an iterable of recognized segments and metadata. The list comprehension iterates over those segments and returns their text fields. **3. Interpret the beam size:** beam_size = 3 controls the search width used during decoding. It can affect the selected transcript and computation, but cannot guarantee that speech recognition is error-free. **Answer: A, B — It loads a large pre-trained Whisper model for transcription.; It splits the audio into segments and returns the transcribed text..** A: **Correct:** large-v2 specifies a large pretrained Whisper model. B: **Correct:** The function consumes the returned segments and collects their transcribed text. C: **Incorrect:** No speaker-embedding, clustering or diarization step appears in the code. D: **Incorrect:** A beam-search setting cannot guarantee perfect recognition on every recording.

**1. Define the task:** Speaker diarization partitions an audio recording into speech intervals and groups intervals by speaker identity. It can operate without knowing the words or language. **2. Explain the conventional pipeline:** Voice activity detection locates speech, speaker embedding extraction converts speech segments into vectors representing voice characteristics, and clustering groups similar vectors into speaker identities. **3. Distinguish optional tasks:** Transcription and language identification may be added to an application, but they are not required to decide who spoke when. **Answer: A, B, D — Voice Activity Detection (VAD) to remove non-speech segments.; Speaker Embedding Extraction (e.g., ECAPA-TDNN, x-vectors) to represent ; Clustering (e.g., Agglomerative Clustering, K-Means) to group similar speaker .** **Convention note:** These are the standard components of the embedding-and-clustering pipeline intended here. Some end-to-end diarization architectures integrate or replace separate VAD/clustering modules. A: **Correct:** VAD identifies speech regions so silence/background-only intervals do not contaminate the conventional pipeline. B: **Correct:** Speaker embeddings provide the representations compared to identify the same voice across segments. C: **Incorrect:** Speech-to-text answers what was said; diarization can work without it. D: **Correct:** Clustering groups segment representations that correspond to the same speaker. E: **Incorrect:** Language identification is not necessary for assigning speaker identities.

**1. Identify the task:** SpeechT5ForTextToSpeech generates speech conditioned on tokenized text and a speaker embedding. The text determines linguistic content and the speaker vector conditions the desired voice. **2. Explain the reduction factor:** The standard speech decoder has reduction_factor = 2. Its training decoder inputs use every second target spectrogram frame, reducing the number of decoder time steps, while the output projection predicts two frames per decoder step. **3. Distinguish the model variant:** The text-to-speech variant synthesizes audio features from text; it is not Whisper-style audio transcription. Its architecture is not inherently limited to English, although a specific checkpoint’s language ability depends on its data and tokenizer. **Answer: A, B — It requires both text and speaker embeddings as inputs.; The decoder input is compressed by a factor of 2 during training..** **Convention note:** The factor of 2 describes the default configuration. Changing reduction_factor changes this behavior. A: **Correct:** Text inputs and speaker-conditioning embeddings are supplied in the normal TTS workflow. B: **Correct:** The default reduction factor of 2 reduces decoder input time steps during training. C: **Incorrect:** ForTextToSpeech is a synthesis model, not the speech-to-text variant. D: **Incorrect:** English-only capability is not an inherent architectural restriction; adaptation and suitable data can support other languages.

**1. Read the units:** The loaded mono waveform contains N = 320,000 samples, and librosa.load was asked to produce a sampling rate \(f_s=16000\) samples per second. **2. Use the duration relation:** Sample count equals duration multiplied by sampling rate, so \[T=\frac{N}{f_s}=\frac{320000}{16000}=20\text{ seconds}.\] **3. Check the interpretation:** The array dimension counts samples rather than milliseconds or bytes. Dividing by samples per second gives seconds. **Answer: 20.**

**1. Identify the representation size:** Each speaker is represented by a 512-component x-vector. The question assumes one stored vector for each of the 100 speakers. **2. Count all stored components:** \[N=100\times512=51200\text{ scalar embedding entries}.\] **3. Clarify what is counted:** This is the storage count for the assumed speaker-vector table. It is not the total number of trainable weights in the pretrained speaker-recognition neural network. **Answer: 51200.** **Convention note:** The source informally calls these embedding parameters. The multiplication counts one stored 512-dimensional representation per speaker; actual model-weight count cannot be inferred from speaker count alone.

**1. Read the tensor convention:** PyTorch Conv1d uses (batch, channels, sequence length). The input has shape (1, 64, 100), and the convolution produces 128 channels. **2. Compute convolution length:** For dilation d, padding p, kernel k and stride s, \[L_{\mathrm{out}}=\left\lfloor\frac{L_{\mathrm{in}}+2p-d(k-1)-1}{s}\right\rfloor+1.\] Here d = 1, p = 2, k = 5, s = 1, so \(L_{\mathrm{out}}=\lfloor(100+4-4-1)/1\rfloor+1=100\). The shape is (1, 128, 100). **3. Apply max pooling:** With kernel 2, stride 2 and no padding, the length becomes \(\lfloor(100-2)/2\rfloor+1=50\). Pooling preserves batch and channel counts, giving (1, 128, 50). **Answer: C — conv_out: (1, 128, 100), pool_out: (1, 128, 50).** A: **Incorrect:** The max-pooling operation halves the length rather than preserving 100. B: **Incorrect:** 96 would result from the convolution without its specified padding. C: **Correct:** The padding preserves convolution length, then pooling reduces it to 50. D: **Incorrect:** 102 does not follow from the specified convolution padding and kernel.

**1. Use the length formula:** At fixed dilation 1, \[L_{\mathrm{out}}=\left\lfloor\frac{L+2p-k}{s}\right\rfloor+1.\] This shows how padding p, kernel k and stride s affect output length when the other settings remain fixed. **2. Compare convolution changes:** More padding adds input positions, so it does not reduce length. A larger kernel uses more positions per output and can reduce the number of placements. A larger stride spaces output positions farther apart and reduces their count. **3. Compare pooling changes:** With the given input length 100 and pooling stride 2, increasing pooling kernel from 2 to 3 changes the pooled length from 50 to \(\lfloor97/2\rfloor+1=49\). Thus increasing pooling size is also a reduction mechanism. **Answer: B, C, D — Increasing kernel size; Increasing stride; Increasing maxpool size.** **Convention note:** Other settings are held fixed. Flooring can make a particular small change leave a length unchanged; the selected settings are the mechanisms that reduce length rather than increase it. A: **Incorrect:** Increasing padding increases or preserves the number of valid output positions. B: **Correct:** A larger kernel reduces the numerator of the length formula when padding is fixed. C: **Correct:** A larger stride samples fewer output positions. D: **Correct:** A larger pooling window reduces valid window placements with the other settings fixed.

**1. Identify the model direction:** The snippet loads SpeechT5ForTextToSpeech, so the task maps text into speech, conditioned on a target voice. **2. Identify the conditioning inputs:** Text is tokenized into input IDs. Speaker embeddings provide voice information. These are the relevant input combination among the choices. **3. Include the training target:** Fine-tuning also needs matching target speech, normally converted into spectrogram labels. Text and speaker embeddings describe the conditioning inputs; they are not the complete supervised training record by themselves. **Answer: C — A combination of text and speaker embeddings..** A: **Incorrect:** Raw waveforms alone do not provide the text content that the TTS model must learn to speak. B: **Incorrect:** A speaker embedding identifies voice characteristics but does not specify the words. C: **Correct:** Tokenized text plus speaker embeddings supplies the linguistic and voice conditioning. D: **Incorrect:** The displayed model does not use MFCC features as its primary text-conditioning input.

**1. Identify the adaptation problem:** A low-resource language may contain characters or pronunciations that are poorly represented by the pretrained text vocabulary. Training data may also cover only a small range of speakers or dialects. **2. Consider text and voice coverage:** A suitable phoneme representation can help express pronunciations when supported by the tokenizer/model setup. Representative speaker embeddings and audio across dialects can improve the diversity of voice conditioning. **3. Reject unrelated changes:** Uppercasing is not a universal improvement; text must match the tokenizer vocabulary. Increasing transformer hidden size changes the architecture and is not a necessary language-preprocessing step. **Answer: A, B — Use phoneme-based tokenization instead of direct text tokenization.; Train on speaker embeddings that include multiple dialects..** **Convention note:** A and B are the beneficial strategies intended by the source, not universal requirements. SpeechT5 can be adapted with appropriate character/text tokenization without a phoneme frontend, and multiple dialects are not required for every task. Actual preprocessing depends on tokenizer coverage and the chosen data. A: **Correct:** Phoneme-based input can be an appropriate adaptation strategy when its vocabulary and pronunciation pipeline are supported. B: **Correct:** Diverse speaker/dialect coverage can improve representative voice conditioning, although embeddings do not replace language-specific text/audio training. C: **Incorrect:** Case normalization must follow the tokenizer; blanket uppercasing is not a general accuracy rule. D: **Incorrect:** Increasing hidden size is neither preprocessing nor inherently required for a new language.

**1. Compute the effective batch:** Each device processes 8 examples per forward/backward batch. Accumulating gradients across 2 such batches before an optimizer step gives \[B_{\mathrm{effective}}=8\times2=16\] examples per device per optimizer update. With D data-parallel devices the total is 16D. **2. Interpret the training settings:** Three epochs means three passes through the training data, not a guarantee of perfect synthesis. The learning rate 3e-5 is a plausible fine-tuning rate; no general rule requires it to exceed 0.1. **3. Locate checkpoints:** save_strategy = "epoch" requests checkpoint saving after each epoch, and output_dir sets the directory under which Trainer stores those checkpoints. **Answer: A, D — Gradient accumulation gives an effective batch of 16 per device; epoch checkpoints use output_dir.** A: **Correct:** Gradient accumulation doubles the effective batch without retaining activations for a simultaneously doubled microbatch; there can still be modest bookkeeping overhead. B: **Incorrect:** Epoch count specifies training duration, not guaranteed output quality. C: **Incorrect:** A learning rate above 0.1 is not required and can destabilize fine-tuning. D: **Correct:** Together with save_strategy="epoch", the specified output directory holds the end-of-epoch checkpoints.