In the Masked Language Modeling (MLM) objective of BERT, 15% of tokens are chosen for
prediction. Of these chosen tokens, 80% are replaced with
, 10% are replaced with a
random word, and 10% are left unchanged. What is the primary theoretical motivation for
**keeping 10% of the tokens unchanged**?
6406536642429.
To reduce the computational cost of the softmax layer during training.
6406536642430.
To prevent the network from overfitting to the
token.
6406536642431.
To mitigate the mismatch between pre-training (where
appears) and
fine-tuning (where
does not appear), ensuring the model creates meaningful
representations for non-masked words.
6406536642432.
To act as a regularizer similar to Dropout.
A published solution is not available for this question yet.
Why is the standard GPT architecture (Decoder-only with causal masking) generally **unsuitable** for
the Masked Language Modeling (MLM) objective as implemented in BERT?
6406536642433.
GPT models are too small to learn bidirectional contexts.
6406536642434.
The causal mask in GPT prevents the model from attending to future tokens,
making it impossible to use right-side context to predict a masked token.
6406536642435.
GPT does not have positional embeddings, which are required for MLM.
6406536642436.
GPT uses ReLU activation, while BERT uses GELU, which is required for MLM.
A published solution is not available for this question yet.
In the T5 (Text-to-Text Transfer Transformer) framework, every NLP task is cast as a text
generation problem.
If you use T5 for a **Semantic Textual Similarity (STS-B)** task, where the goal is to predict a
similarity score (e.g., 3.8) between two sentences, how does the model output this score?
6406536642452.
It outputs a single scalar value from a regression head on top of the encoder.
6406536642453.
It generates the string "3.8" token-by-token using the decoder.
6406536642454.
It outputs a class label corresponding to a bucketed score range (e.g., "High
Similarity").
6406536642455.
T5 cannot be used for regression tasks like STS-B.
A published solution is not available for this question yet.
When constructing training datasets for large language models (LLMs), which of the following best
describes the key factors that must be balanced to achieve strong and reliable performance?
6406536642468.
Model depth, number of parameters, and learning rate
6406536642469.
Scale, diversity, and quality of the training data
6406536642470.
Vocabulary size, tokenization method, and batch size
6406536642471.
Compute budget, optimizer choice, and hardware efficiency
A published solution is not available for this question yet.
training sample.
In **Layer Normalization**, normalization is performed **across features for each individual**
**sample**.
Let
be the number of neurons (features) in the hidden layer.
For a given sample, the layer normalization statistics are computed as:
where
is a small constant for numerical stability.
Layer Normalization is applied **independently to each sample** in the given data above.
For **Sample 2**, after applying Layer Normalization. **What is the maximum value in the**
**normalized output vector**
**?**
(ignore the learnable parameters
and
, and assume
):
A published solution is not available for this question yet.
**Sentence Corpus:**
"the key to artificial intelligence has always been the representation"
**Pre-processing Instructions:**
• Convert all text to lowercase.
• Append the end-of-word symbol
to every word (e.g., "the" becomes
).
• The **Initial Vocabulary** is defined as the set of all unique characters present in the corpus plus
the
symbol.
Based on the above data, answer the given subquestions.
Which of the following character pairs occurs with the **highest frequency** across the entire
sentence before any BPE merges are performed?
**Sentence Corpus:**
"the key to artificial intelligence has always been the representation"
**Pre-processing Instructions:**
• Convert all text to lowercase.
• Append the end-of-word symbol
to every word (e.g., "the" becomes
).
• The **Initial Vocabulary** is defined as the set of all unique characters present in the corpus plus
the
symbol.
Based on the above data, answer the given subquestions.
Calculate the **total vocabulary size** immediately after the first merge operation is completed.
**Note:** The vocabulary size includes all individual base characters/symbols plus the newly created
merge token.
A published solution is not available for this question yet.
Consider a small BERT-like model with the following configuration:
• Embedding dimension (
) = 128
• Vocabulary size (
) = 5000
• Maximum sequence length (
) = 64
• Number of segment types = 2
Calculate the **total number of parameters** in the **Embedding Layer** (sum of Token embeddings,
Position embeddings, and Segment embeddings).
Enter the exact integer value.
A published solution is not available for this question yet.
To obtain high-quality training text from raw web data for large language models, which of the
following mechanisms are typically included in the pre-processing pipeline?
6406536642472.
Tokenizing text into subword units using Byte Pair Encoding (BPE)
6406536642473.
Deduplicating content at line, paragraph, and document levels
6406536642474.
Detecting and filtering toxic content such as hate speech and profanity
6406536642475.
Identifying the language of web pages
6406536642476.
Assessing the quality of content to remove low-value or spam text
6406536642477.
Detecting and removing Personally Identifiable Information (PII)
6406536642478.
Fine-tuning the model using reinforcement learning from human feedback
(RLHF)
A published solution is not available for this question yet.
Which of the following statements correctly explain the importance of deduplication during
preprocessing of large-scale datasets used for training Deep Learning or Large Language Models?
6406536642479.
Deduplication reduces the risk of overfitting by preventing repeated samples
from dominating the gradient updates.
6406536642480.
Deduplication guarantees that the trained model will achieve higher accuracy
on all downstream tasks.
6406536642481.
Deduplication helps avoid data leakage between training and evaluation sets,
leading to more reliable performance metrics.
6406536642482.
Deduplication eliminates the need for regularization techniques such as
dropout and weight decay.
A published solution is not available for this question yet.
