AI Training and Inference Systems: From First Principles to Self-Design
COMPSCI 2650
Subject & Catalog Number
Course Information
Description
The systems that train, fine-tune, and serve large models are the engines of the AI revolution, and their performance comes down to two goals: minimizing data movement across the memory hierarchy and maximizing GPU and CPU utilization. This course teaches the first principles through which system designs and implementations achieve these goals: modularity and abstractions, breaking problems into smaller pieces, parallelism, overlapping computation with data movement, and navigating vast design spaces of tradeoffs. These principles are timeless: the same ideas that shaped decades of systems evolution shape today's AI systems and will shape whatever comes next. On the training side, topics include GPU kernels, compilation of tensor programs, the parallelism and memory optimizations that fit massive models onto limited hardware, and fine-tuning methods that adapt models by updating only a small fraction of their parameters. On the inference side, topics include KV-cache management, batching, and splitting the serving pipeline across specialized machines. Underneath both sits storage: the data layouts and retrieval pipelines that feed models, where every bit stored and moved bounds the space of possible designs. Agentic applications are the workloads driving these designs, from serving agents efficiently to fine-tuning models for custom agents. We take the principles to their logical conclusion: self-designing systems that automatically synthesize their own architecture for a given workload, hardware, and performance target. The class is research-oriented, built on recent and classical papers and interactive discussion, with a semester-long systems project integrated with DASlab research.
Course Notes
This course was previously numbered CS 265.
Recommended Prep
CS 1650 (formerly CS 165) or permission of instructor.
Available for Harvard Cross Registration
