BUILD AI: Modules
Expected Outline for this AI Engineering Series
Here’s my expected roadmap for the BUILD AI series. It’s liable to change as we go along, ofc, but changes should be reflected here, so it will always reflect the current “course” structure. I’ll also use this as a master TOC, linking to each section as they are published and go live.
Module I: Foundation Model Training Infrastructure
Unit 1: Distributed Training from Scratch
Tensor parallelism: Code walkthrough of splitting attention heads across GPUs
Pipeline parallelism: Implementing forward/backward passes across model layers
Data parallelism with gradient synch: AllReduce implementation details
Mixed precision training: FP16/BF16 memory layouts and gradient scaling
Gradient checkpointing: Memory-compute tradeoffs with implementation
ZeRO optimizer states: Sharding optimizer memory across devices
Communication backends: NCCL setup, topology awareness, bandwidth optimization
Unit 2: Data Pipeline Engineering
Streaming from object storage: S3/GCS integration with prefetching
Data Loading Optimizations: AsyncDataLoader, Memory Pinning & NUMA
Tokenization at scale: Parallelized preprocessing with SentencePiece
Data deduplication: MinHash, bloom filters, exact match algorithms
Quality filtering implementation: Perplexity-based filtering, language detection
Dynamic batching: Variable sequence length handling
Checkpointing and resumption: Stateful data pipeline recovery
Unit 3: Memory and Compute Optimization
Attention mechanisms at scale: FlashAttention implementation walkthrough
KV-cache optimization: Memory layouts for inference serving
Activation recomputation strategies: Selective checkpointing algorithms
Memory mapping for large datasets: Efficient data loading from object storage
CUDA kernels for transformers: Custom implementations for layer norms, attention
Quantization techniques: INT8/INT4 inference with calibration datasets
Model sharding strategies: Loading 70B models across multiple GPUs
KV-Cache Sharding & Distributed Inference: Run large (400B) models locally
Module II: Training Large Models
Unit 4: Transformer Implementation Deep Dive
Multi-head Attention: Why Everyone’s Implementation Looks Identical
Positional Encodings: Length Extrapolation and the Context Problem
Unit 5: Training Loop Engineering
Learning rate scheduling: Warmup, cosine annealing, restart strategies
Gradient clipping: Per-parameter vs global norm clipping
Loss computation: Cross-entropy optimization, label smoothing
Logging and monitoring: Weights & Biases integration, custom metrics
Checkpoint management: Efficient serialization, incremental saves
Training stability: Loss spikes, gradient explosion detection and recovery
Hyperparameter sweeps: Population-based training, Bayesian optimization
Unit 6: Evaluation and Benchmarking
Evaluation harness implementation: Multi-GPU evaluation, caching strategies
Benchmark datasets: Loading, preprocessing MMLU, HellaSwag, etc.
Perplexity computation: Sliding window approaches, memory efficiency
Generation evaluation: BLEU, ROUGE implementation details
A/B testing frameworks: Statistical significance, multiple comparisons
Performance profiling: CUDA profiling, bottleneck identification
Module III: Fine-tuning and Alignment
Unit 7: Parameter-Efficient Fine-tuning
LoRA implementation: Low-rank matrix decomposition, merge strategies
QLoRA details: 4-bit quantization with LoRA adapters
Prefix tuning: Learnable prefix tokens, initialization strategies
Adapter layers: Architecture variants, placement strategies
Multi-task fine-tuning: Task routing, interference mitigation
Catastrophic forgetting: Regularization techniques, replay buffers
Unit 8: RLHF Implementation
Reward model training: Pairwise ranking loss, data collection strategies
PPO for language models: Action spaces, value function approximation
KL penalty computation: Reference model comparison, dynamic scaling
Experience replay: Buffer management, sampling strategies
Constitutional AI: Self-critique training loops, principle selection
Direct preference optimization: DPO loss implementation, comparison to RLHF
Unit 9: Instruction Tuning
Dataset construction: Alpaca-style formatting, quality control
Multi-turn conversation handling: Context management, turn boundaries
System message implementation: Special token handling, role management
Safety filtering: Content moderation integration, refusal training
Tool use training: Function calling, API integration patterns
Chain-of-thought prompting: Training data generation, verification
Module IV: Deployment and Serving
Unit 10: Inference Optimization
KV-cache management: Memory pools, attention mask optimization
Batching strategies: Continuous batching, sequence bucketing
Speculative decoding: Draft model selection, acceptance criteria
Parallel sampling: Beam search, nucleus sampling implementation
Memory bandwidth optimization: Weight loading, activation memory layout
Model parallelism for serving: Load balancing, fault tolerance
Dynamic quantization: Runtime precision adjustment
Unit 11: Production Serving Architecture
Load balancing: Request routing, queue management
Auto-scaling: Demand prediction, cold start optimization
Model versioning: A/B testing, canary deployments
Monitoring and alerting: Latency tracking, quality metrics
Rate limiting: Token bucket, sliding window implementations
Caching strategies: Response caching, embedding caches
Multi-tenancy: Resource isolation, fair share scheduling
Unit 12: Edge Deployment
Model compression: Pruning, distillation for mobile deployment
ONNX conversion: Graph optimization, operator fusion
Quantization for edge: INT8 calibration, dynamic range estimation
Memory mapping: Model loading on resource-constrained devices
Batch size optimization: Latency vs throughput tradeoffs
Power consumption: Energy-aware inference scheduling
Framework integration: Core ML, TensorFlow Lite deployment
Module V: Advanced Topics
Unit 13: Custom Training Frameworks
PyTorch distributed: Process groups, communication backends
JAX/Flax implementation: Functional transformations, SPMD parallelism
DeepSpeed integration: ZeRO stages, pipeline scheduling
FairScale usage: FSDP, activation checkpointing
Custom CUDA operators: Memory-efficient attention implementations
Profiling and debugging: Memory leaks, deadlock detection
Framework comparison: Performance benchmarking across frameworks
Unit 14: Research Infrastructure
Experiment tracking: MLflow, Weights & Biases advanced usage
Hyperparameter optimization: Optuna, Ray Tune integration
Distributed computing: Slurm job scheduling, cloud orchestration
Data versioning: DVC, dataset lineage tracking
Model registries: Versioning, governance, access control
Reproducibility: Environment management, seed handling
Cost optimization: Spot instances, preemption handling
Unit 15: Multimodal Extensions
Vision-language models: CLIP-style contrastive training
Image tokenization: DALL-E style discrete VAE implementation
Cross-modal attention: Architecture modifications for multimodal input
Video processing: Temporal modeling, frame sampling strategies
Audio integration: Speech tokenization, alignment techniques
Unified embeddings: Joint training strategies, modality fusion
Evaluation metrics: Multimodal benchmarking, human evaluation