Every technology studied in this course, from speech synthesis and speech recognition through machine translation and summarization, depends on a single preparatory step that is easy to overlook. Before any model can learn, the raw material of language, whether typed text or transcribed speech, must be converted into a sequence of discrete units that a computer can count, compare, and predict. Week 1 is devoted to that step. It introduces text normalization and tokenization, the processes that turn a stream of characters into the tokens on which every subsequent module rests.

The week opens with the vocabulary of corpora, drawing the distinction between word types and word tokens and showing how simple counts already expose structure in language. It then treats regular expressions as the practical instrument for locating and extracting patterns in text. From there the focus moves to normalization proper, which covers case folding, tokenization, lemmatization, and stemming, and to the deceptively hard question of what counts as a word in writing systems that do not separate words with spaces. The final part of the week presents subword tokenization and works carefully through the Byte-Pair Encoding algorithm, which learns its vocabulary from letter statistics rather than from a fixed dictionary and therefore copes gracefully with words it has never seen.

The decisions made this week are not merely technical housekeeping. The way text is segmented sets the size of a model’s vocabulary, governs its handling of rare and unknown words, and shapes the quality of every downstream task. A tokenizer that is well matched to a language and a task is one of the quiet reasons a system works, and a poorly chosen one is a common reason a system fails.

Learning Objectives

By the end of Week 1 you will be able to:

1. Distinguish word types from word tokens and compute basic frequency statistics for a corpus.
2. Read and construct regular expressions to find and extract patterns in text.
3. Describe the stages of text normalization and explain when case folding, lemmatization, and stemming are each appropriate.
4. Explain why word level tokenization struggles with unknown words and how subword tokenization resolves the problem.
5. Trace the Byte-Pair Encoding algorithm by hand, learning an ordered set of merges from a small corpus and applying those merges to new text.
6. Segment text into sentences and account for the ambiguities that make the task harder than it first appears.
7. Explain how the same tokenization idea applies to audio as to text, through discrete audio tokens.

Reading and Preparation

The anchor text for this week is Jurafsky and Martin, Speech and Language Processing, third edition draft of January 2026, Chapter 2, titled Regular Expressions, Text Normalization, and Edit Distance. Concentrate on the sections covering words and corpora, text normalization, and tokenization, including the treatment of Byte-Pair Encoding. The minimum edit distance material that closes the same chapter is reserved for Week 2 and may be set aside for now.

In the Laboratory

The accompanying laboratory session puts these ideas into practice in Python. You will normalize and tokenize real text, compare the output of a word level tokenizer against a subword tokenizer, and implement the Byte-Pair Encoding merge loop so that the algorithm you trace by hand in the lecture becomes running code you can inspect and extend. The exercise closes the gap between the statistics on the slide and the behaviour of a working tokenizer.

Looking Ahead

Week 2 takes the tokens produced here and puts them to work. It builds n-gram language models, the first probabilistic account of how likely a sequence of words is, and it defines minimum edit distance, a measure of how far apart two strings are. Every module that follows, from automatic speech recognition to machine translation, consumes tokenized input, so the foundation laid this week is drawn upon continuously for the remainder of the course.

Downloads

Lecture_Note

Presentation

Model Examination Questions with Answers