Course Description

EAI 820 covers eight language technologies, namely speech synthesis, speech recognition, language identification, machine translation, summarization, spell checking, grammar checking, and mobile application development. Although these appear distinct, they rest on one small set of ideas. Six of these recur throughout everything that follows. The first is to turn the input into a sequence of tokens. The second is to represent those tokens as vectors. The third is to predict the next token. The fourth is to learn by gradient descent on a loss. The fifth is to mix information across positions using attention. The sixth is to pretrain a model and then adapt it. The Introduction and Weeks 1 to 4 install these ideas in plain code, while Weeks 5 to 8 present the same ideas rearranged for text technologies, speech synthesis, speech recognition, and on-device deployment.

The course also splits along a practical line. Everything from the Introduction through Week 5 runs offline, in plain Python with NumPy, on any laptop, requiring no downloads, no network, and no graphics processor. From Week 6 onward, however, you need audio libraries and a GPU. This split is deliberate, since it keeps environment problems out of the weeks in which the conceptual material is hardest.

There is a clear reason for covering so much text before any speech. Modern speech systems are built from the same components as text systems, because a recogniser generates text conditioned on audio, and a synthesiser generates audio conditioned on text. Weeks 1 to 5 are therefore what make Weeks 6 to 8 comprehensible rather than merely operational. Throughout the course, the governing engineering question is not only how something works, but also what it costs and whether it can run on the device in front of you. It is precisely this question that sets EAI 820 apart from a general course in language technology.

Course Learning Outcomes

On successful completion of the course, a student will be able to:

• Describe the human–machine speech-and-language loop and place any named technology within it, consuming and producing the right representations at each stage.
• Turn text and audio into sequences of tokens (byte-pair encoding, discrete audio tokens), represent tokens as vectors, and compute similarity with the cosine measure.
• Build and rigorously evaluate counting language models and classifiers, using perplexity, precision, recall and the F-measure, with proper held-out data and significance testing.
• Explain and implement scaled dot-product attention and the transformer block, and distinguish encoder-only, decoder-only and encoder–decoder configurations.
• Design and evaluate automatic translation, summarization, spell-checking and grammar-checking systems, and recognise each as conditional generation or classification over shared machinery.
• Explain the classical speech-synthesis pipeline — text normalization, the letter-to-sound cascade, prosody — and build and evaluate modern neural text-to-speech.
• Extract speech features, build and evaluate speech-recognition systems, and relate language, speaker and wake-word recognition as one encoder with different heads.
• Compress and deploy a speech-and-language pipeline on a constrained device using quantization, pruning, distillation and streaming, and reason about latency, memory and energy.
• Reason about data, bias, ethics, safety and evaluation across every language and speech technology in the course.

Structure at a Glance