Deep Learning Practice · Quiz 1 · Sep 2024

Choose the Hugging Face module that helps us train a tokenizer from scratch on a specific dataset.

A dataset contains 10 billion words ( separated by a single white space). Suppose we use a pre- trained tokenizer that has a vocabulary of size 10,000 to tokenize the dataset, then the number of tokens in the dataset will always be greater than or equal to the number of words in the dataset. The statement is

Which of the following tokenization algorithms can be applied to languages that do not have any word delimiters?

Consider the Wikipedia dataset scraped from the web that contains 2 billion words. A team decided to use the BPE tokenization algorithm with the varying vocabulary size from 2K to 52K, then the statement that increasing the number of merges will increase the size of the vocabulary is

Suppose that we pre-train a Causal Language Model. Choose the data collator function from the Hugging Face library that is suitable for this task

Based on the above data, answer the given subquestions.

Enter the number of parameters in the embedding layer of the model in millions. For example, if the answer is 1234567. Then enter it as 1.23

Based on the above data, answer the given subquestions.

Enter the context length.

Based on the above data, answer the given subquestions.

Enter the number of tokens (in millions) processed by the model after 1000 steps. Enter the answer to 2 decimal places. For example, if your answer is 123456789, then enter it as 123.45.

Based on the above data, answer the given subquestions.

How many steps does it take to complete one epoch of training? Enter the answer in thousands (round down to an integer). For example, if your answer is 1234567.89, then enter it as 1234567.