Gradient Ascent

The Anatomy of an LLM Benchmark

Throughout the history of AI research, progress has been measured—and accelerated—by high-quality benchmarks. AI is an empirical field that is driven by discovering interventions that improve performance on key benchmarks. For large language models (LLMs) in particular, creating useful benchmarks is hard due to rapidly advancing …

4 days ago · 73 likes · Cameron R. Wolfe, Ph.D.

LLM benchmarks break in predictable ways: MMLU has a 6.49% error rate (Virology hits 57%), and correcting those errors moved Llama-3.1-405B from 16th to 1st. Cameron Wolfe (Deep Learning Focus) maps the full lifecycle of how benchmarks are designed, saturate, and get replaced. The standout section covers Item Response Theory, which cuts evaluation costs 140-160x by selecting only the most informative test items.

3. Context Engineering for AI Agents: Memory, Compaction, and Tool Clearing

https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools

Three composable primitives for managing context in long-running agents, each targeting a different type of bloat. Clearing drops re-fetchable tool outputs at zero inference cost (peak context: 173K tokens vs. 335K baseline). Compaction summarizes conversation history (169K peak, lossy). Memory persists knowledge across sessions by letting the agent write its own notes. Isabella He (Anthropic) includes a diagnostic framework: profile your agent’s token breakdown first (in the demo, 96.3% of tokens were stale file-read results), then pick the primitive that matches.

🛠 2 Tools & Repos

1. Hindsight

https://github.com/vectorize-io/hindsight

Agent memory that goes beyond conversation history. Hindsight provides three API primitives (retain, recall, reflect) that let agents learn, retrieve, and synthesize knowledge over time. Recall runs four parallel strategies (semantic, keyword, graph, temporal) with cross-encoder reranking. The reflect API generates new insights from existing memories, not just retrieval. SOTA on LongMemEval, 6.8K GitHub stars, MIT license, works with any LLM provider, and deploys via Docker or as an embedded Python library.

2. Shannon

https://github.com/KeygraphHQ/shannon

An autonomous AI pentester that reads your source code and executes real exploits against the running app. It builds a Code Property Graph to trace data flows from user input to dangerous sinks, then attacks those paths. Every reported vulnerability comes with a working proof-of-concept, not theoretical findings. Handles authentication complexity including 2FA/TOTP flows.

🎓 1 Pick of the Week

OpenMAIC: Tsinghua’s Multi-Agent AI Classroom

https://github.com/THU-MAIC/OpenMAIC

Type “teach me Python in 30 minutes” and OpenMAIC generates a full interactive classroom: slides with narration, quizzes with real-time grading, interactive HTML simulations, and AI agents playing teacher and student roles who lecture, discuss, and draw on a shared whiteboard. Built on LangGraph with a director agent that orchestrates turn-taking, it supports uploading PDFs for document-to-course conversion. 13.6K stars in three weeks, live demo at open.maic.chat, and the LangGraph-based multi-agent orchestration pattern is a reusable blueprint beyond education.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

Google Compresses KV-Cache 6x Without Training, How Every Modern Attention Variant Works, and a Claude Code Cheat Sheet - 📚 The Tokenizer Edition #21

Sairam Sundaresan — Thu, 26 Mar 2026 12:02:32 GMT

Hey there! This week’s theme is speed: faster OCR through diffusion, faster inference through speculative execution, faster compression through polar coordinates, and Stripe shipping 1,300 agent-written pull requests every week. Whether you’re optimizing tokens, transistors, or team velocity, there’s something here for you.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Diffusion-based OCR that’s 3x faster, a package manager for AI agent skills, world modeling from Monster Hunter, probability-aware RL clipping, and speculative shortcuts for agentic vision systems
🎥 Videos: Stripe’s internal AI coding agents at scale, building your own AI operating system from scratch, DeepSeek’s conditional memory for transformers, and going from Figma design to working code with Claude
📰 Reads: Sebastian Raschka’s visual taxonomy of attention mechanisms, Google’s training-free KV-cache compression via polar coordinates, and why on-device AI needs new architectures (not smaller cloud models)
🛠 Tools: ByteDance’s filesystem-based context database for AI agents, and Cornell’s free GPU architecture course
🎓 Learning: A beautifully maintained Claude Code cheat sheet that doubles as a feature discovery tool

📄 5 Papers

1. MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding

https://arxiv.org/abs/2603.22458 | GitHub

What if OCR didn’t need to generate text one token at a time? This paper replaces autoregressive decoding with block-wise diffusion denoising, reframing document OCR as inverse rendering. The result: up to 3.2x faster throughput with a tunable speed-accuracy tradeoff depending on your pipeline needs. A proposed “Semantic Shuffle” benchmark shows the model genuinely reads visual structure rather than leaning on linguistic shortcuts.

2. SkillNet: Create, Evaluate, and Connect AI Skills

https://arxiv.org/abs/2603.04448 | GitHub

Think npm, but for AI agent skills. SkillNet treats agent capabilities as shareable, composable packages with a unified ontology for creating skills from heterogeneous sources (repos, docs, logs, prompts) and connecting them through dependency graphs. The framework delivers a 40% improvement in average rewards and 30% fewer execution steps across ALFWorld, WebShop, and ScienceWorld. With 200,000+ skills in the repository and a Python toolkit plus Open Access API, this could become critical infrastructure for the tool-using agent ecosystem.

3. WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG

https://arxiv.org/abs/2603.23497 | GitHub

Most world-modeling datasets treat video prediction as frame interpolation. WildWorld takes a different approach: 108 million frames automatically collected from Monster Hunter: Wilds, with per-frame action labels (450+ semantically meaningful actions), character skeletons, world states, camera poses, and depth maps. The key insight is decomposing dynamics into action, state, and pixels separately, giving world models the causal structure they need to simulate rather than interpolate.

4. BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

https://arxiv.org/abs/2603.04918 | GitHub

PPO’s fixed clipping bounds have a quiet failure mode: they disproportionately suppress high-advantage, low-probability actions, which are exactly the exploratory tail strategies you want to preserve in reasoning tasks. BandPO replaces static bounds with dynamic, probability-aware intervals derived from f-divergence constraints, formulated as a convex optimization problem with closed-form solutions. Tested on Qwen2.5 and Llama3 for mathematical reasoning, it consistently outperforms canonical GRPO clipping while preventing entropy collapse. A drop-in replacement for anyone doing GRPO-style training post-DeepSeek-R1.

5. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

https://arxiv.org/abs/2603.23483 | GitHub

Agentic vision systems are slow because every query triggers a full sequential tool-use chain, even when the answer is straightforward. SpecEyes uses a lightweight gating mechanism (based on top-K logit gaps) to predict when the expensive pipeline can be short-circuited. The result: up to 3.35x speedup (averaging 1.4-1.7x across benchmarks) with accuracy maintained or improved by up to 6.7%. The gating requires no labeled routing data, making it deployable without per-task annotation on any production agentic vision system where latency matters.

🎥 4 Videos

1. Building a Full-Stack AI Operating System from Scratch

A three-layer architecture for a personal AI platform: webhook-driven triggers, scheduled workflows (cron-style recurring tasks), and an autonomous agent layer with persistent context and reusable skills. Dave Ebbelaar builds the entire system on FastAPI, Celery, Redis, and Docker. This 28-minute walkthrough gives you the blueprint instead of cloning random agent repos.

2. DeepSeek’s Engram: Adding Conditional Memory to Transformers

DeepSeek’s Engram module adds O(1) lookup memory to transformers by modernizing n-gram embeddings as a new sparsity dimension alongside Mixture-of-Experts. Dr. Karoly Zsolnai-Feher (Two Minute Papers) explains how it outperforms comparable MoE baselines at 27B parameters on both parameter and compute budgets. The surprise: the biggest gains show up not in knowledge retrieval (MMLU +3.4) but in reasoning (BBH +5.0, ARC-Challenge +3.7), because offloading pattern reconstruction to memory frees attention depth for harder tasks.

3. From Figma Design to Working Code with Claude Code and MCP

Figma mockup to working website in 15 minutes, FigJam flowchart to a working game, then exporting code back to Figma as editable components. Felix Lee (designer at ADPList) demonstrates the full design-to-code loop using Claude Code with Figma MCP in this 50-minute session hosted by Peter Yang. The core insight: MCP reads every color, spacing value, and component variant directly from Figma, eliminating the translation loss in design-to-dev handoffs.

4. How Stripe Ships 1,300 AI-Written Pull Requests Per Week

Stripe built internal AI coding agents called “minions” that now ship roughly 1,300 PRs per week. Steve Kaliski (software engineer at Stripe) walks through the architecture: Goose (Block’s open-source agent harness) with cloud dev environments, activated from Slack via emoji reactions. The key takeaway: Stripe’s existing investment in developer tooling (CI, testing, linting) made agent adoption frictionless, because good DX for humans turns out to be good DX for agents too. Non-engineers at Stripe now use minions to ship code.

📰 3 Curated Reads

1. A Visual Guide to Attention Variants in Modern LLMs

Ahead of AI

A Visual Guide to Attention Variants in Modern LLMs

I had originally planned to write about DeepSeek V4. Since it still hasn’t been released, I used the time to work on something that had been on my list for a while, namely, collecting, organizing, and refining the different LLM architectures I have covered over the past few years…

11 days ago · 267 likes · 6 comments · Sebastian Raschka, PhD

Seven attention mechanism families in one visual taxonomy, from classic Multi-Head Attention through GQA, DeepSeek’s Multi-Head Latent Attention, Sliding Window Attention, and hybrid architectures mixing transformers with linear or state-space modules. Sebastian Raschka diagrams how queries, keys, and values interact in each variant, how KV-cache mechanics change, and how memory growth curves differ. The practical payoff: a clear framework for choosing the right attention mechanism based on your model scale and deployment constraints, with Gemma 3 and DeepSeek V2 as case studies.

2. TurboQuant: Redefining AI Efficiency with Extreme Compression

https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

Google Research introduces a training-free, data-agnostic compression algorithm that achieves 3-bit KV-cache quantization without accuracy loss. The trick: PolarQuant converts data vectors from Cartesian to polar coordinates (concentrating angle patterns predictably), then QJL reduces residual errors to single sign bits via random projections. On H100 GPUs, TurboQuant delivers a 6x reduction in KV memory and up to 8x performance gains over 32-bit unquantized baselines. Unlike most quantization work, this targets the KV-cache specifically (the bottleneck that grows with context length) and requires zero calibration data or fine-tuning.

3. The Future of On-Device AI

Devansh argues the real bottleneck for on-device AI is memory bandwidth, not compute. Using Liquid AI’s LFM2 (1.2B parameters, runs on Samsung Galaxy S25) as a case study, the piece shows why shrinking data-center models is the wrong approach. LFM2 uses 10 gated short convolutions plus 6 grouped-query attention blocks, cutting peak cache to 192 MB at 32K tokens (versus Llama 3.2 1B’s 524 MB). It matches Qwen3-1.7B on benchmarks despite having 42% fewer parameters and runs at 70 tokens per second on a phone CPU. The thesis: the field needs device-native architectures designed from scratch, not miniaturized cloud models.

🛠 2 Tools & Repos

1. OpenViking: A Context Database for AI Agents

https://github.com/volcengine/OpenViking

ByteDance’s open-source solution for the “stuff everything into the prompt” problem. OpenViking organizes agent context (memories, resources, skills) into a navigable filesystem hierarchy with three-tier demand-based loading, so agents only consume tokens for what they actually need. It combines directory-based navigation with semantic search, auto-compresses conversations into long-term memory, and provides visualization of retrieval trajectories for debugging. 19.1K stars, Apache 2.0, supports major LLM providers via LiteLLM.

2. Cornell Virtual Workshop: GPU Architecture Fundamentals

https://cvw.cac.cornell.edu/gpu-architecture/gpu-characteristics/design

Free, NSF-funded course from Cornell’s Center for Advanced Computing covering why GPUs work the way they do: transistor allocation tradeoffs, memory hierarchies, parallelization design choices, and the practical implications for your code. Modules covering fundamentals through V100 and RTX 5000 deep dives, with exercises. No parallel programming experience assumed. If you call `.cuda()` daily but lack a mental model for what happens underneath, start here.

🎓 1 Pick of the Week

Claude Code Cheat Sheet

https://cc.storyfox.cz/

This single page Claude code cheat sheet covers 100+ commands across 8 color-coded sections: keyboard shortcuts, ~40 slash commands, MCP server configuration, memory and files, workflows, config, skills and agents, and CLI flags. Commands like `/btw` for side questions without derailing context, `/schedule` for cloud-scheduled tasks, and git worktree isolation with sparse checkout are buried in docs but surfaced here at a glance. Auto-detects Mac versus Windows, prints cleanly to A4, works offline, and updates daily as Claude Code evolves.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

Claude Code Best Practices, Planning in 8 Tokens, and Why Reasoning Models Can't Control Their Own Thoughts - 📚 The Tokenizer Edition #20

Sairam Sundaresan — Wed, 18 Mar 2026 13:03:02 GMT

Hey there! This week’s theme is the gap between what AI systems can do and what they actually do in practice. Reasoning models that can’t steer their own chain of thought. RAG systems that work in demos but hallucinate in production. Training clusters that fail in ways no tutorial prepares you for. The good news: attention training just got 1.67x faster, and Figma engineers are showing what a real design-to-code workflow looks like with Claude Code.

New here?

What comes next with open models

TL;DR

What caught my attention this week:

📄 Papers: Low-bit attention training, efficient VLMs, scientific discovery shortcuts, robot planning in 8 tokens, and reasoning models that can’t control their own thoughts
🎥 Videos: Sakana AI’s evolved transformers, Turbopuffer’s post-RAG retrieval architecture, Figma’s Claude Code design pipeline, and DeepMind reflects on AlphaGo’s decade of impact
📰 Reads: Nathan Lambert on why the open model gap will widen (not close), production RAG done right, and diagnosing failures across 192-GPU training clusters
🛠 Tools: ByteDance’s open-source SuperAgent platform and OpenAI’s AI-powered LaTeX editor
🎓 Learning: A community-built field manual for getting the most out of Claude Code

📄 5 Papers

1. SageBwd: A Trainable Low-bit Attention

https://arxiv.org/abs/2603.02170 | GitHub

SageAttention already had 3,200+ GitHub stars for speeding up inference. Now SageBwd extends quantized attention to training by quantizing 6 of 7 attention matrix multiplications in the backward pass. The result: up to 1.67x speedup over FlashAttention2 with negligible loss difference (2.561 vs 2.563 at 260K tokens per step). If you’re spending money on attention compute during training, this is the paper to read.

2. Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoder

https://arxiv.org/abs/2603.06569

What if you replaced CLIP/SigLIP vision encoders with one initialized from a plain text LLM? Tencent AI Lab tried it. Their 8B model outperforms Qwen3-VL-8B and InternVL3.5-8B on document understanding, visual knowledge, and video reasoning, hitting 96.2 on DocVQA and 90.5 on ChartQA. Vision encoder architecture matters more than you’d think. Sometimes the answer is just using a language model.

3. MOOSE-Star: Unlocking Tractable Training for Scientific Discovery

https://arxiv.org/abs/2603.03756

Scientific hypothesis discovery with LLMs has an exponential search problem. MOOSE-Star (from MiroMind AI) reduces combinatorial O(N^k) search to roughly logarithmic via hierarchical decomposition, hitting 100% success rate at around 6,000 inference calls where brute-force saturates at 41.3%. Also releases TOMATO-Star, a dataset of 108,717 decomposed papers for benchmarking. The complexity reduction alone makes previously intractable hypothesis spaces searchable.

4. CompACT: A Compact Discrete Tokenizer for Latent World Model Planning

https://arxiv.org/abs/2603.05438

Robot planning typically requires hundreds of visual tokens per observation. CompACT (from POSTECH and KAIST, accepted at CVPR 2026) compresses that down to as few as 8 discrete tokens, making world-model planning 40x faster. Navigation planning drops from 178 seconds to under 6 seconds with competitive accuracy. This is what makes real-time robotic planning actually feasible.

5. Reasoning Models Struggle to Control their Chains of Thought

https://arxiv.org/abs/2603.05706

Can reasoning models actually control their chain of thought? Researchers from NYU, OpenAI, UCL, and UPenn tested 13 models. Claude Sonnet 4.5 achieves only 2.7% CoT controllability (versus 61.9% output controllability). DeepSeek R1 scores 0.1%. The safety implication: if models can’t steer their reasoning strategically, they also can’t easily hide deceptive reasoning from monitors. Matters more for what it implies than what it measures.

🎥 4 Videos

1. Sakana AI’s Open-Ended Evolution of Transformers with Robert Lange

LLMs combined with evolutionary algorithms for open-ended program search. That’s Shinka Evolve from Sakana AI, discussed by Robert Lange on Machine Learning Street Talk. Why “solving the wrong problem” sometimes leads to better architectures. How evolutionary pressure discovers novel transformer variants. What open-endedness means for AI research beyond benchmarks. Covers the gap between optimizing a known objective and discovering objectives worth optimizing.

2. Retrieval After RAG: Hybrid Search, Agents, and Database Design with Turbopuffer’s Simon Eskildsen

What comes after the first wave of RAG implementations? Hybrid search architectures, why vector-only retrieval hits a ceiling, agent-driven retrieval reshaping database design. Simon Eskildsen (Turbopuffer founder) walks through real case studies from Cursor and Notion on Latent Space. If you’ve built a RAG system and hit a quality wall, this is where you go next.

3. How Figma Engineers Sync Designs with Claude Code and Codex

MCP-based tooling that creates a continuous sync between design and code. Figma’s Gui Seiz and Alex Kern show their team’s actual production workflow using Claude Code and Codex. Not a concept demo. The design handoff becomes a two-way pipeline. Worth 40 minutes if your team still does screenshot-to-implementation.

4. 10 Years of AlphaGo: The Turning Point for AI

AlphaGo’s techniques propagated into protein structure prediction, materials science, and chip design. Google DeepMind’s Thore Graepel and Pushmeet Kohli trace the full decade of impact beyond the Go match itself. Covers how one system’s ideas became foundational building blocks across scientific domains.

📰 3 Curated Reads

1. What comes next with open models

Interconnects AI

2025 was the year where a lot of companies started to take open models seriously as a path to influence in the extremely valuable AI ecosystem — the adoption of a strategy that was massively accelerated downstream of DeepSeek R1’s breakout success. Most of this is being done as a mission of hope, principle, or generosity…

13 days ago · 59 likes · 12 comments · Nathan Lambert

The open-closed model gap will widen, not close. Nathan Lambert’s reframing: the real opportunity isn’t chasing frontier capability but building small, specialized models that are 10x faster and 100x cheaper. Introduces the “open models as sub-agents” framing where open-weight models handle specialized tasks within larger systems. Changes how you evaluate open models if you’ve been benchmarking them against GPT-5.

2. Production RAG: Learning from Scratch Done Right

Decoding AI Magazine

Why Most RAG Tutorials Fail You

Paul: Today, the stage belongs to Priya, a Senior Software Architect who’s spent years shipping production-scale systems at Publicis Sapient and Tesco…

18 days ago · 46 likes · 5 comments · Priya

Most RAG tutorials optimize for demos, not production. This piece on Paul Iusztin’s Decoding AI (by guest contributor Priya) walks through a 4-phase production RAG system: ingestion, retrieval, generation, serving. Uses Postgres and pgvector with explicit control flow and data lineage. The core insight: a bad chunk doesn’t throw an exception, it just hallucinates an answer three steps later. If your RAG prototype works in notebooks but fails in production, start here.

3. How to Diagnose Failures in Large AI Training Clusters

AI agents autonomously executing diagnostic runbooks against a unified Prometheus TSDB. Devansh details five case studies across multi-GPU clusters with quantified results: 30% throughput recovery, checkpoint restore penalties reduced from 1.0% to 0.15%. Not theoretical. Includes the actual diagnostic architecture and failure patterns you’d hit at this scale.

🛠 2 Tools & Repos

1. deer-flow (ByteDance)

https://github.com/bytedance/deer-flow

ByteDance’s open-source SuperAgent platform (MIT license, 31K+ stars) got a ground-up v2.0 rewrite that hit #1 on GitHub Trending. Ships as a complete deployable platform, not a framework you wire together. Web UI, Docker-sandboxed execution, persistent cross-session memory, parallel sub-agent spawning, messaging integrations (Telegram, Slack, Feishu). Built on LangGraph. For teams that want a working multi-agent system without assembling one from parts.

2. Prism (OpenAI)

https://prism.openai.com/

OpenAI’s browser-based LaTeX editor with GPT-5.2 integrated inline. Free tier: unlimited projects, compiles, and collaborators (Pro at $7/mo for unlimited AI features). Highlight text, ask the AI to rewrite or formalize, and it compiles in real time. Zotero integration, image-to-LaTeX, voice mode for dictating equations. Best for refining existing papers. Won’t generate structure from a blank page. No Git integration yet, which is the main gap versus Overleaf.

🎓 1 Pick of the Week

claude-code-best-practice

https://github.com/shanraisshan/claude-code-best-practice

How should you actually use Claude Code day-to-day? Official docs don’t answer that. This community-built field manual does (17,600+ stars, actively maintained). 40+ actionable tips across 8 categories, comparative reports against other tools, community workflow implementations. Includes a working `.claude/` directory you can clone. The “billion-dollar questions” section names what the community still hasn’t figured out. If you’re already using Claude Code and want to move from casual to systematic, bookmark this.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

Karpathy's Autonomous ML Lab, Sleeper Cells in LLMs, and Andrew Ng's Context Hub - 📚 The Tokenizer Edition #19

Sairam Sundaresan — Wed, 11 Mar 2026 12:03:21 GMT

Hey there! This week’s curation spans from AI systems that run overnight experiments autonomously to backdoors hiding inside your favorite tool-using agents. Whether you’re thinking about building agents, defending them, or just trying to understand what your GPU actually wants from you, there’s something here.

New here? The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

TL;DR

What caught my attention this week:

• 📄 Papers: Smarter reasoning graphs, temporal backdoors in tool-using LLMs, spatial reasoning benchmarks, privacy advantages of diffusion language models, and versatile video editing

• 🎥 Videos: Hardware constraints shaping LLM architecture, brand-consistent image generation with Midjourney, a complexity taxonomy for AI agents, and a critical look at vibe coding

• 📰 Reads: Statistical rigor for LLM evaluations, the infrastructure costs of long-context inference, and the current state of open-weight models catching up to closed frontiers

• 🛠 Tools: A 101k-star collection of LLM applications and a versioned API documentation system for coding agents

• 🎓 Learning: Karpathy’s tool that lets AI agents run autonomous ML experiments overnight

📄 5 Papers

1. RouteGoT: Node-Adaptive Routing for Cost-Efficient Graph of Thoughts Reasoning

https://arxiv.org/abs/2603.05818

Graph of Thoughts is powerful but expensive, treating every reasoning node equally regardless of difficulty. RouteGoT fixes this by adaptively routing compute across graph nodes, skipping the heavy lifting where it isn’t needed. The result: 8.1 percentage points more accurate than AGoT while using 79.1% fewer output tokens.

2. Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs

https://arxiv.org/abs/2603.03371

Here’s a security scenario worth losing sleep over: backdoors implanted in tool-using LLMs that only activate under specific temporal conditions. The model maintains state-of-the-art performance on benign tasks and evades standard safety evaluations, until a particular time trigger flips the switch. A sobering look at the gap between current evaluation practices and actual deployment safety.

3. SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in LLMs

https://arxiv.org/abs/2603.03002

Do LLMs actually understand spatial relationships, or are they just pattern-matching on word co-occurrences? SpatialText isolates spatial reasoning from visual shortcuts using pure text, and the results aren’t flattering. Current models lean heavily on linguistic heuristics rather than building coherent spatial representations, which matters for anyone building systems that need to reason about physical space.

4. Characterizing Memorization in Diffusion Language Models

https://arxiv.org/abs/2603.02333

Diffusion-based language models turn out to have a meaningful privacy advantage over autoregressive ones. This paper shows they exhibit substantially lower memorization-based leakage of personally identifiable information. If you’re building generative systems where training data sensitivity matters (medical, legal, financial), this distinction between generation architectures is worth understanding.

5. Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

https://arxiv.org/abs/2603.02175 | GitHub

Video editing that combines instruction-based and reference-based approaches in one pipeline. Kiwi-Edit constructs high-fidelity training data using synthetic reference scaffolds, sidestepping the usual bottleneck of paired video editing datasets. The result is a system that handles both “make the sky purple” style instructions and “make it look like this reference” editing in a single model.

🎥 4 Videos

1. How Hardware Constraints Are Shaping Modern LLM Architecture

Julia Turc explores how physical hardware realities (memory bandwidth, compute density, interconnect speeds) are actively driving architectural decisions in modern LLMs. If you’ve wondered why certain design choices keep showing up across different labs, the answer often starts with what the silicon can actually do efficiently.

2. Mastering Midjourney: Consistent Brand Imagery Without Complex Prompts

On How I AI, AI creative director Jamey Gannon walks through her workflow for generating images that maintain consistent brand identity across multiple Midjourney outputs using style references, personalization codes, and mood boards. Useful for anyone who’s gotten great individual images from AI tools but struggled to make a cohesive visual set for a brand, product, or campaign.

3. The 5 Levels of AI Agent Complexity

Dave Ebbelaar breaks down AI agents into five distinct complexity levels, from simple single-tool agents to sophisticated multi-agent orchestration systems. Helpful framing for teams trying to scope what kind of agent they actually need (often simpler than they think) and understanding the engineering effort each level demands.

4. Vibe Coding is a Slot Machine

Machine Learning Street Talk sits down with Jeremy Howard (fast.ai) to examine whether AI coding assistants are genuinely improving developer productivity or creating a false sense of progress. Howard’s central argument: if you outsource all your thinking to computers, you stop building the competence that makes you effective. The kind of critical examination teams need before making investment decisions around AI-assisted development tooling.

📰 3 Curated Reads

1. Applying Statistics to LLM Evaluations

Applying Statistics to LLM Evaluations

Research on large language models (LLMs) is empirically driven. For this reason, model evaluations play a pivotal role in the field’s progress. We improve models by making changes, evaluating them, and iterating. Despite their foundational role, however, evaluations are usually handled in a naive manner. In most cases, we just test a mod…

25 days ago · 66 likes · 2 comments · Cameron R. Wolfe, Ph.D.

Cameron R. Wolfe from Deep (Learning) Focus tackles a problem most benchmarking papers quietly ignore: the statistical validity of LLM evaluation results. Key finding worth internalizing: clustered standard errors can increase reported uncertainty by 3x, and the Central Limit Theorem becomes unreliable with small sample sizes. If you’ve ever looked at a leaderboard and wondered “is this difference actually meaningful?”, this article gives you the tools to answer that.

2. How Long Context Inference Is Rewriting the Future of Transformers

AI Made Simple quantifies what long-context windows actually cost in production. A 70B-parameter model serving 59 concurrent users at 4K context drops to just 1 user at 128K context. The article covers the engineering responses, including Multi-Head Latent Attention (MLA) achieving 93.3% cache reduction. Practical reading for anyone deploying models where context length isn’t just a spec sheet number but a capacity planning constraint.

3. A Dream of Spring for Open-Weight LLMs

Ahead of AI

A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026

If you have struggled a bit to keep up with open-weight model releases this month, this article should catch you up on the main themes…

a month ago · 183 likes · 9 comments · Sebastian Raschka, PhD

Sebastian Raschka surveys the current open-weight landscape across 10 models and finds the gap to closed frontiers narrowing fast. GLM-5 now benchmarks on par with GPT-5.2, Gemini Pro 3, and Claude 4.6 Opus. On the inference side, Step 3.5 Flash hits 100 tokens/sec compared to DeepSeek’s 33. A well-structured overview for tracking which open models are actually competitive and where the remaining gaps lie.

🛠 2 Tools & Repos

1. Awesome LLM Apps

https://github.com/Shubhamsaboo/awesome-llm-apps

At 101k stars, this collection of LLM application examples covers RAG implementations, AI agents (single and multi-agent teams), MCP integration patterns, voice AI, and fine-tuning guides. More useful as a reference architecture library than a tutorial. When you’re building something new, check here first to see how others have solved similar problems.

2. Context Hub

https://github.com/andrewyng/context-hub

From Andrew Ng’s team, Context Hub gives coding agents access to curated, versioned API documentation instead of letting them hallucinate library APIs. Features search and fetch, language-specific variants, persistent annotations, and feedback loops so the documentation improves over time. At 3.5k stars, it’s gaining traction with teams building custom coding agents that need reliable API knowledge.

🎓 1 Pick of the Week

Autoresearch

https://github.com/karpathy/autoresearch

Andrej Karpathy’s latest side project is a Python tool built around three core files (prepare.py, train.py, program.md) that lets an AI agent run autonomous ML experiments on a single GPU. The agent proposes hypotheses, writes training code, runs experiments with a 5-minute budget each, and iterates based on results. Leave it running overnight and wake up to a stack of completed experiments. At 23.6k stars within days of release, it’s clearly struck a nerve. Sparks of recursive self-improvement, indeed.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

Karpathy's microGPT, Jeff Dean's Pareto Frontier, and the LLM Course - 📚 The Tokenizer Edition #18

Sairam Sundaresan — Thu, 19 Feb 2026 13:34:15 GMT

Hey there! While agents crush bug fixes at 74%, they stumble to 11% on actual feature development. Turns out building real features is fundamentally harder than patching code. Meanwhile, passing a task to an agent and actually delegating authority to one turn out to be completely different problems, and Karpathy just distilled an entire GPT into 243 lines of dependency-free Python. These are structural shifts in how we approach autonomy, development, and education.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

We’ll cover AI systems that use RAG and traditional ML design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Through live discussions, guided exercises, and design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

Use code FLASH40 for 40% off:

Learn AI System Design

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

TL;DR

What caught my attention this week:

• 📄 Papers: Task handoff is not the same as authority transfer, video models tested on physics not aesthetics, memory that evolves without overfitting, gated reasoning that knows when to stop, and feature-level coding benchmarks exposing real agent limitations

• 🎥 Videos: Jeff Dean on AI’s Pareto frontier, comprehensive 2026 state of AI breakdown, building custom dev tools instead of buying SaaS, and practical context engineering for agents

• 📰 Reads: Rubric-based rewards for subjective domains, recursive language models that call themselves like functions, and why taste matters for generative AI

• 🛠 Tools: Reasoning implementations from scratch, comprehensive LLM learning roadmap

• 🎓 Learning: Karpathy’s microGPT strips GPT to its mathematical essence in pure Python

📄 5 Papers

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development

https://arxiv.org/abs/2602.10975 | GitHub

Claude 4.5 Opus achieves 74.4% on SWE-bench but drops to 11.0% on FeatureBench. The difference? SWE-bench tests bug fixes within single pull requests, while FeatureBench evaluates end-to-end feature development spanning multiple commits and PRs across development timelines. Using a test-driven method that traces from unit tests along dependency graphs, the benchmark automatically derives 200 feature-level tasks from 24 repositories while ensuring other features remain functional after separation. This exposes the gap between fixing localized issues and actually building new capabilities.

Intelligent AI Delegation

https://arxiv.org/abs/2602.11865

Existing delegation methods run on heuristics that collapse when environments change or sub-agents fail. Intelligent AI Delegation reframes the problem: passing a task is not the same as transferring authority, responsibility, and accountability, and conflating the two is where multi-agent systems break down. The framework draws from principal-agent theory in economics, assumes zero trust at every delegation boundary, and applies to both human and AI delegators across complex agent networks. Misalignment, reward gaming, and deceptive behavior all compound as agents delegate to other agents. The paper is trying to define the protocol layer the agentic web needs before it can safely scale.

RISE-Video: Can Video Generators Decode Implicit World Rules?

https://arxiv.org/abs/2602.05986 | GitHub

Video models produce visually impressive outputs, but can they reason about physics? RISE-Video shifts evaluation from aesthetics to cognitive understanding with 467 human-annotated samples spanning eight categories. The benchmark tests whether models grasp implicit constraints like spatial dynamics, temporal consistency, physical rationality, and causality. Testing 11 state-of-the-art models revealed pervasive failures when simulating complex scenarios under implicit rules, exposing the gap between visual fidelity and genuine world understanding.

UMEM: Unified Memory Extraction and Management Framework for Generalizable Memory

https://arxiv.org/abs/2602.10652 | GitHub

Memory-enabled agents typically optimize management while treating extraction as static, accumulating instance-specific noise rather than robust insights. UMEM jointly optimizes both through Semantic Neighborhood Modeling, evaluating memory utility across clusters of semantically related queries rather than individual instances. Trained with neighborhood-level marginal utility rewards via GRPO, the approach achieves up to 10.67% improvement on multi-turn tasks while maintaining monotonic growth during continuous evolution. Agents that actually learn from experience rather than just retrieve logs.

When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning

https://arxiv.org/abs/2602.10560

Long-context reasoning faces critical issues: memory explodes from indiscriminate updates on evidence-free chunks, and loops continue unnecessarily after gathering sufficient evidence. GRU-Mem introduces update and exit gates controlled by text. Memory updates only when the update gate opens, and the loop terminates immediately when the exit gate opens. Trained with end-to-end RL using separate rewards for correct updating and exiting behaviors, GRU-Mem outperforms vanilla MemAgent with up to 4x inference speed acceleration.

🎥 4 Videos

Owning the AI Pareto Frontier — Jeff Dean

Jeff Dean unpacks the Pareto frontier in AI scaling, balancing compute, energy efficiency and model performance. Google’s Chief AI Scientist discusses the unification of their AI teams and why distillation is becoming the engine behind efficient models. Essential viewing for understanding the tradeoffs between model capability and practical deployment constraints.

State of AI in 2026: LLMs, Coding, Scaling Laws & Agents

Nathan Lambert and Sebastian Raschka join Lex Fridman to break down where we actually stand. They cover the reality of coding agents (echoing FeatureBench findings), scaling laws beyond just “more compute,” and practical challenges of building reasoning models. Two of the best technical communicators in AI deliver a comprehensive status report on 2026’s landscape.

DIY Dev Tools: The Shift to “Build” Over “Buy”

CJ Hess demonstrates “Flowy,” a custom tool he built to visualize coding plans. Instead of relying on static Markdown or finicky diagrams, he created a system where Claude generates JSON that renders into interactive flowcharts and UI mockups. When AI can write the code, building bespoke internal tools optimized for your specific workflow is often faster and better than buying generic SaaS.

Effective Context Engineering for AI Agents

Dave Ebbelaar explains why agents fail not from bad instructions but from poor context management. In short, context engineering is king. He offers practical strategies for structuring workflows and managing the context window as a limited resource rather than a magic bucket.

📰 3 Curated Reads

Rubric-Based Rewards for RL

Rubric-Based Rewards for RL

Many of the recent capability gains in large language models (LLMs) have been a product of advancements in reinforcement learning (RL). In particular, RL with verifiable rewards (RLVR) has drastically improved LLM capabilities by using rules-based, deterministic correctness checks (e.g., passing the test cases for a coding problem…

a month ago · 64 likes · 5 comments

Cameron Wolfe explains that while RL has excelled in domains with clear answers like math and code, Rubric-Based RL uses LLMs as judges with strict rubrics to provide reward signals in subjective domains. This bridges the gap that could allow training reasoning models for writing, creativity, and strategy rather than just problems with verifiable solutions.

Recursive Language Models: To the Rescue

WHERE MACHINES THINK

Recursive Language Models: Can a simple method potentially unlock new regime of inference-time scaling?

IN THE FALL OF 2024, there were concerns that neural scaling laws were saturating. Throwing more compute and data to train ever-bigger large language models (LLMs) was showing diminishing returns. And then, OpenAI released its o1 series of “reasoning” models, unlocking an entirely new way to improve the performance of LLMs. Ope…

2 months ago · 17 likes · 1 comment · Anil Ananthaswamy

What if an LLM could call itself like a function? Recursive Language Models decompose complex prompts and recursively generate their own context. This architectural shift treats models less like text predictors and more like computer programs with call stacks, handling complexity through recursion rather than scale.

Taste for Makers

https://paulgraham.com/taste.html

Paul Graham breaks down the universal principles of good design across disciplines like math, coding, and architecture. For anyone building AI products today, treating “quality” as an objective, engineerable standard rather than a vague feeling is a massive competitive advantage.

🛠 2 Tools & Repos

Reasoning from Scratch (Chapter 7 Code)

https://github.com/rasbt/reasoning-from-scratch/blob/main/ch07/01_main-chapter-code/ch07_main.ipynb

Sebastian Raschka’s notebook implements Reinforcement Learning with Group Relative Policy Optimization (GRPO) from scratch. If you want to understand how models like DeepSeek-R1 work under the hood, this walks through the actual mechanics rather than hiding behind abstractions. Essential for anyone implementing reasoning capabilities.

LLM Course Roadmap

https://github.com/mlabonne/llm-course

Maxime Labonne’s repository remains the gold standard for self-learning. This curated roadmap takes you from fundamentals to fine-tuning your own models, recently updated with sections on agentic workflows and evaluation. The curriculum balances theoretical explanations with practical notebooks and hands-on projects mirroring real-world applications.

🎓 1 Pick of the Week

microGPT

https://karpathy.github.io/2026/02/12/microgpt/

Andrej Karpathy stripped GPT to its absolute core: 243 lines of pure Python with zero dependencies. No PyTorch, no NumPy, no frameworks. Just the full algorithmic content needed for training and inference: dataset handling, tokenizer, autograd engine, GPT-2-like architecture, Adam optimizer, training loop, and inference loop. The code fits perfectly across three columns and represents a decade-long obsession to simplify LLMs to bare essentials. Run it from a single file, understand every mathematical operation, see exactly how attention works without abstraction layers hiding the mechanics. For anyone who wants to truly understand transformers rather than just use them, this is the definitive resource.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

Google's Viral Paper Banana, How to Systemize Claude Code, and Stanford on Agents & RAG - 📚 The Tokenizer Edition #17

Sairam Sundaresan — Wed, 11 Feb 2026 02:27:46 GMT

Hey there! This week brought Google’s Paper Banana into the spotlight with its interactive approach to understanding research, a comprehensive system for making Claude Code actually work at scale, and Stanford’s practical take on building agents with RAG. Beyond the headline picks, CodeOCR’s 8x code compression through visual representation and DFlash beating EAGLE-3 by 2.5x suggest we’re seeing real efficiency breakthroughs across the board.

New here?

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

Through live discussions, guided exercises, and team-based design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

Use code FLASH40 for 40% off:

Questions? Drop a comment or DM me.

TL;DR

What caught my attention this week:

📄 Papers: Code-as-image achieving 8x compression, speculative decoding beating EAGLE-3 by 2.5x, video models learning from YouTube demonstrations, and self-optimizing RL systems

🎥 Videos: Systematic Claude Code workflows from Anthropic users, Stanford on agents and RAG patterns, visual MoE explanations, and vector indexing methods

📰 Reads: Why creativity can’t be interpolated, 10x cheaper tokens through prompt caching, and 12x faster MoE training with Unsloth

🛠 Tools: Language extraction for multilingual text and comprehensive LLM evaluation frameworks

🎓 Learning: Google’s viral Paper Banana for interactive research paper exploration

📄 5 Papers

Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition

https://arxiv.org/abs/2602.08439 | GitHub

Most video benchmarks test static knowledge rather than whether models can actually learn from demonstrations. Demo-ICL tackles this directly with a benchmark built from 1200 instructional YouTube videos. The system provides models with text or video demonstrations, then asks them to answer questions about target videos by applying what they learned from the examples. The two-stage training approach combines video-supervised fine-tuning with information-assisted preference optimization, improving how models extract and apply procedural knowledge from demonstrations.

DFlash: Block Diffusion for Flash Speculative Decoding

https://arxiv.org/abs/2602.06036 | GitHub

Autoregressive models decode sequentially, creating a latency bottleneck that speculative decoding helps address. DFlash uses a lightweight block diffusion model that generates draft tokens in parallel rather than one at a time. By conditioning the draft model on context features from the target LLM, it maintains high acceptance rates while generating entire blocks simultaneously. The system achieves over 6x lossless acceleration across various tasks, delivering up to 2.5x higher speedup than EAGLE-3.

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

https://arxiv.org/abs/2602.01785 | GitHub

Text-based code processing creates a linear scaling problem as codebases grow. CodeOCR explores representing code as rendered images instead, taking advantage of visual compression capabilities that text lacks. The research shows vision-language models can understand code images at up to 8x compression while maintaining performance. Tasks like clone detection actually improve slightly under compression, and syntax highlighting boosts code completion performance at 4x compression. The approach suggests image-based code representation could fundamentally change how we handle large-scale code understanding.

RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

https://arxiv.org/abs/2602.02488 | GitHub

Traditional reinforcement learning keeps environments, policies, and reward models separate. RLAnything makes all three components adapt together through closed-loop optimization. The policy trains on combined step-wise and outcome signals, while the reward model jointly optimizes through consistency feedback. Environment tasks automatically adjust difficulty based on critic feedback from both the policy and reward model. The system delivers substantial gains: 9.1% improvement on OSWorld for Qwen3-VL-8B-Thinking, and 18.7% and 11.9% boosts on AlfWorld and LiveBench for Qwen2.5-7B-Instruct.

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

https://arxiv.org/abs/2602.07026

Despite progress in multimodal contrastive learning, a persistent geometric anomaly remains: embeddings of different modalities expressing identical semantics occupy systematically offset regions. This modality gap creates alignment challenges that existing approaches handle inefficiently. The research introduces Fixed-frame Modality Gap Theory, which decomposes the gap into stable biases and anisotropic residuals. ReAlign aligns text representations into image distribution space using statistics from unpaired data through anchor, trace, and centroid alignment. Building on this, ReVision enables MLLMs to learn visual representation distribution from unpaired text before visual instruction tuning, reducing dependence on expensive image-text pairs.

🎥 4 Videos

How to Make Claude Code Better Every Time You Use It (Full System)

Stop fighting with Claude Code and start building systematic workflows that compound over time. This comprehensive guide covers setting up persistent context files that Claude references across all sessions, structuring your codebase so Claude understands project architecture from the start, and creating reusable prompt patterns that extract better reasoning. You’ll learn how to build a project-specific knowledge base, manage multiple Claude sessions working on different parts of your code simultaneously without conflicts, and structure requests to get maintainable output instead of throwaway code. The system turns Claude Code from a one-off tool into a genuine development partner.

MoE, Visually Explained

Mixture-of-Experts architectures power many frontier models but remain conceptually opaque. This visual breakdown makes the core mechanics tangible: how router networks decide which experts process each token, why sparse activation improves efficiency without sacrificing performance, and what trade-offs exist between expert count and model capacity. The explanations focus on intuition over equations, making MoE accessible without dumbing down the actual complexity involved.

What is Indexing? Indexing Methods for Vector Retrieval

Vector databases require efficient retrieval mechanisms as collections scale. This overview covers indexing approaches from flat search through hierarchical navigable small worlds (HNSW) and inverted file indexes (IVF). You’ll understand when approximate nearest neighbor search makes sense versus exact matching, how different index types trade off speed against accuracy, and which methods work best for specific retrieval patterns.

Agents, Prompts, and RAG

Stanford’s practical breakdown of building reliable agents using retrieval-augmented generation cuts through the hype. The discussion examines when to use agents versus simpler prompt chains, how to structure RAG systems that agents can query effectively, and debugging approaches when agentic systems fail. The focus stays on production considerations: handling edge cases, managing token budgets, and building systems that degrade gracefully rather than failing catastrophically. Essential viewing for anyone moving beyond toy agent demos to production deployments.

📰 3 Curated Reads

Why Creativity Cannot Be Interpolated

https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/

This exploration challenges the assumption that creative breakthroughs emerge from incremental improvements. The argument centers on fundamental limitations of interpolation-based learning: models trained to predict the next token in existing distributions struggle to generate genuinely novel ideas that exist outside their training manifold. The piece examines what this means for AI’s creative capabilities and why certain types of innovation may require fundamentally different approaches than current architectures support.

Prompt caching: 10x cheaper LLM tokens, but how?

https://ngrok.com/blog/prompt-caching/

Prompt caching stores processed prefixes so repeated context doesn’t get recomputed on every request. This technical breakdown explains how caching works under the hood: which parts of prompts get cached, how providers handle cache invalidation, and what billing implications exist. The post provides concrete strategies for structuring prompts to maximize cache hits, quantifies actual cost savings across different use patterns, and identifies scenarios where caching delivers minimal benefit.

Fine-tune MoE Models 12x Faster with Unsloth

https://unsloth.ai/docs/new/faster-moe

Training Mixture-of-Experts models typically requires massive compute due to their architecture. Unsloth achieves up to 12x speedups over Transformers v4 (and roughly 2x over the already-optimized Transformers v5) through custom Triton grouped-GEMM kernels and a Split LoRA approach that also cuts VRAM usage by over 35%. The documentation covers implementation details: how to integrate Unsloth into existing training pipelines, which MoE models are supported (Qwen3, DeepSeek R1/V3, GLM), and how the optimizations maintain full accuracy with zero approximation. Particularly valuable for teams fine-tuning large MoE models on consumer hardware.

🛠 2 Tools & Repos

Langextract

https://github.com/google/langextract

Extracting structured information from unstructured text typically requires custom pipelines for every new domain. Langextract is a Gemini-powered Python library that handles this through LLM-based extraction with precise source grounding, mapping every extracted entity back to its exact location in the source text. Define your extraction schema through a few examples and the tool adapts without fine-tuning, handling long documents through optimized chunking and parallel processing. Particularly useful for domains like clinical notes, radiology reports, and research literature where traceability between extracted data and source material is critical.

Deepeval

https://github.com/confident-ai/deepeval

LLM evaluation needs systematic frameworks rather than ad-hoc testing. Deepeval provides metrics for measuring answer relevancy, factual consistency, and hallucination rates across different model outputs. The framework supports both rule-based and model-based evaluation, enables A/B testing across prompt variations, and tracks performance degradation over time. Particularly useful for teams building production LLM applications that need reliable quality metrics.

🎓 1 Pick of the Week

Paper Banana

https://dwzhu-pku.github.io/PaperBanana/

Google’s Paper Banana went viral for good reason: it fundamentally changes how researchers produce academic illustrations. Instead of spending hours manually crafting methodology diagrams and statistical plots, Paper Banana orchestrates five specialized AI agents (Retriever, Planner, Stylist, Visualizer, and Critic) to generate publication-ready visuals from paper text. The system handles everything from architecture diagrams to data visualizations, with the Critic agent running multiple refinement rounds to catch factual errors and visual glitches. In blind human evaluation, its outputs achieved a 72.7% win rate against baseline AI models. For researchers drowning in illustration work or trying to produce consistent, high-quality figures across papers, this approach delivers production value that manual tools struggle to match. The viral response reflects how badly the research community needed a better way to handle the illustration burden.

DeepSeek's Human-Like Vision, Chip Huyen's AI Tools Site, and Stanford's Updated LLM Course - 📚 The Tokenizer Edition #16

Sairam Sundaresan — Tue, 03 Feb 2026 17:33:23 GMT

Hey there! DeepSeek just redefined how vision models process documents by teaching them to read with human-like logic instead of rigid top-to-bottom scanning. Meanwhile, Ant Group released a world simulator achieving minute-long consistent video generation with sub-second interaction latency. Open-source continues delivering production-ready systems.

New here?

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

Gradient Ascent Special: Use code FLASH40 for 40% off

We’ll cover the core system design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Through live discussions, guided exercises, and team-based design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

TL;DR

What caught my attention this week:

📄 Papers: Vision encoders with causal reasoning for document understanding, open-source world simulators rivaling closed systems, plus advances in mathematical reasoning and multimodal scientific models

🎥 Videos: Pydantic fundamentals for ML engineers, building agent frameworks from scratch, production AI coding workflows, and understanding diffusion versus flow matching

📰 Reads: Reinforcement learning for continual LLM adaptation, preparing for ML interviews beyond just attention mechanisms, and DeepSeek’s latest architectural innovations

🛠 Tools: Curated resources for agentic reasoning research and comprehensive AI tool directories

🎓 Learning: Stanford’s updated large language model course covering recent advances

📄 5 Papers

DeepSeek-OCR 2: Visual Causal Flow

https://arxiv.org/abs/2601.20552 | GitHub

Instead of processing images in rigid raster-scan order, DeepSeek-OCR 2 introduces DeepEncoder V2 that mimics how humans actually read documents. The encoder uses causal attention to dynamically reorder visual tokens based on semantic content, determining whether to scan titles first, process tables column-by-column, or navigate multi-column layouts intelligently. By replacing CLIP with Qwen2-0.5B and implementing learnable queries with causal flow, the 3B parameter model achieves 91.09% on OmniDocBench while maintaining 256-1120 token efficiency. Reading order edit distance dropped from 0.085 to 0.057, proving the system genuinely understands logical document structure rather than just memorizing patterns.

Advancing Open-source World Models

https://arxiv.org/abs/2601.20540 | GitHub

Ant Group’s LingBot-World delivers minute-long consistent video generation at 16 FPS with under 1-second interaction latency, positioning open-source world models competitively against closed systems. The system maintains high-fidelity dynamics across photorealistic, scientific, and stylized environments through a multi-stage training pipeline combining web videos with Unreal Engine synthetic data. Users control camera perspectives and environmental conditions in real-time while the model preserves spatial consistency across 961 frames. The hybrid data engine with hierarchical captioning separates motion control from static scene generation, addressing the training data bottleneck that typically limits world model development.

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

https://arxiv.org/abs/2601.20614 | GitHub

Group Relative Policy Optimization suffers from an implicit imbalance where harder questions receive smaller policy updates, limiting capability development where it matters most. MathForge addresses this through Difficulty-Aware GRPO, which balances group advantage estimation by question difficulty and prioritizes harder problems through difficulty-aware weighting. The framework’s Multi-Aspect Question Reformulation strategy systematically increases question difficulty across multiple dimensions while maintaining gold answers, creating training data that pushes model boundaries. The synergistic loop of MQR expanding the data frontier and DGPO effectively learning from augmented data produces significant improvements across mathematical reasoning benchmarks.

Innovator-VL: A Multimodal Large Language Model for Scientific Discovery

https://arxiv.org/abs/2601.19325 | GitHub

Scientific multimodal models typically require massive domain-specific pretraining, but Innovator-VL demonstrates strong performance using fewer than five million curated samples without large-scale pretraining. The fully transparent training pipeline covers data collection, cleaning, preprocessing, supervised fine-tuning, and reinforcement learning with detailed optimization recipes, enabling systematic community extension. The model maintains competitive performance on general vision benchmarks while excelling at scientific tasks, indicating that scientific alignment integrates into unified models without compromising general-purpose capabilities. Principled data selection proves more effective than indiscriminate scaling for scientific reasoning.

Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision

https://arxiv.org/abs/2601.19798 | GitHub

Vision-language models exhibit a text-dominant optimization bias by treating visual signals as passive inputs rather than supervisory targets. Youtu-VL shifts to the Vision-Language Unified Autoregressive Supervision paradigm, integrating visual tokens directly into the prediction stream and applying unified autoregressive supervision to both visual details and linguistic content. This “vision-as-target” approach fundamentally changes optimization from treating vision as conditional input to making it a prediction objective. The framework extends to vision-centric tasks without requiring task-specific architectural additions, establishing foundations for comprehensive generalist visual agents.

🎥 4 Videos

Pydantic Crash Course

Dave Ebbelaar walks through Pydantic’s data validation and settings management capabilities essential for ML engineers working with LLMs. The tutorial covers defining models with type hints, validation logic, and configuration management patterns that ensure data integrity in production AI systems. Understanding Pydantic proves critical when building structured outputs from language models or managing complex application configurations where type safety prevents runtime errors.

Building Mini ClawdBot from Scratch

Vizuara demonstrates constructing an agent like Moltbot (ClawdBot, or whatever it's called now) without relying on existing libraries, revealing the actual mechanics behind agent architectures. By building from first principles, you understand tool integration, state management, and decision-making loops that frameworks abstract away. This approach proves valuable when debugging production agent systems or architecting custom solutions that don't fit standard framework patterns.

The Senior Engineer’s Guide to AI Coding

This How I AI episode examines Claude Code workflows and architectural decisions that separate effective AI-assisted development from mere prompt engineering. It addresses code review practices, testing strategies, and integration patterns when collaborating with AI coding assistants.

Flow Matching vs Diffusion Side By Side

Letitia compares flow matching and diffusion approaches for generative modeling, clarifying when each technique provides advantages. The visual comparison helps understand why flow matching sometimes offers training efficiency benefits over traditional diffusion while maintaining generation quality. Grasping these trade-offs matters when selecting architectures for specific generative modeling tasks.

📰 3 Curated Reads

Continual Learning with RL for LLMs

Continual Learning with RL for LLMs

Continual learning, which refers to the ability of an AI model to learn from new tasks and data over time, has become a popular topic in the discussion of Artificial General Intelligence (AGI). Put simply, general intelligence should be adaptable, which has led some to believe that continual learning abilities are a prerequisite f…

2 months ago · 93 likes · 3 comments · Cameron R. Wolfe, Ph.D.

Cameron Wolfe explores why Reinforcement Learning (RL) is naturally more robust than Supervised Fine-Tuning (SFT) for continual learning in LLMs. While traditional methods like replay buffers and regularization remain relevant, recent studies suggest that RL’s on-policy nature minimizes the distributional shifts that cause catastrophic forgetting.

Is Attention All You Need for ML Interviews?

https://medium.com/@maxwbuckley/is-attention-all-you-need-to-prepare-for-ml-interviews-830742f6d2ba

Max Buckley shares some quick tips to master the transformer architecture and its implementation ahead of ML interviews. Worth checking out to know the details of the bedrock of modern AI in case you’re preparing for interviews.

DeepSeek Drops Yet Another

WHERE MACHINES THINK

DeepSeek Drops Yet Another Architectural Innovation, Opening A New Front for Scaling Up LLMs

Some large language models can memorize entire books and regurgitate them almost verbatim. In one study, researchers successfully prompted an LLM to spit out, for example, a nearly complete Harry Potter and the Sorcerer’s Stone. While this raises huge concerns about large-scale copyright violations, it's also true that we simultaneously want these model…

3 months ago · 58 likes · 8 comments · Anil Ananthaswamy

Anil Ananthaswamy analyzes DeepSeek’s pattern of releasing architectural innovations that challenge conventional wisdom about model scaling and design. The examination covers their mixture-of-experts implementations, efficiency improvements, and open-source strategy that accelerates community progress. Tracking DeepSeek’s releases provides insights into architectural directions gaining traction in production systems.

🛠 2 Tools & Repos

Awesome Agentic Reasoning

https://github.com/weitianxin/Awesome-Agentic-Reasoning

This curated collection organizes research papers, codebases, and benchmarks focused on agentic reasoning capabilities in AI systems. Instead of accumulating every tangentially related work, the repository maintains editorial standards around core reasoning techniques like planning, reflection, and tool use. The structured organization helps researchers quickly locate relevant work when investigating specific reasoning approaches or comparing methodologies across different agent architectures.

GoodAI List

https://goodailist.com/repos

Chip Huyen curates AI tools and repositories with clear judgment about practical utility versus hype. The directory covers libraries, frameworks, and resources across the ML stack with emphasis on production readiness and actual adoption. Unlike comprehensive but overwhelming awesome lists, this maintains focus on tools demonstrating real-world value in AI development workflows.

🎓 1 Pick of the Week

Stanford’s LLM Course

https://youtube.com/playlist?list=PLoROMvodv4rObv1FMizXqumgVVdzX4_05&si=bqJCbjmvpCx-1_51

Stanford’s curated playlist brings together lectures from multiple courses covering large language models, including CS224N (NLP with Deep Learning), CS25 (Transformers United), and CME 295 (Transformers & Large Language Models). The collection spans foundational concepts in transformer architectures and training methodologies alongside cutting-edge developments in reasoning models, multimodal systems, and efficient inference techniques.

Tencent Compresses Reasoning 3-4x, Robots Master 35,000 Hours of Human Data, and Anthropic's Evaluation Framework - 📚 The Tokenizer Edition #15

Sairam Sundaresan — Mon, 26 Jan 2026 16:25:08 GMT

Hey there! Robots just learned to treat 35,000 hours of human movements as a universal training language, mechanistic interpretability made the jump from observation to intervention, and researchers compressed reasoning into images at 3-4x efficiency. Real progress on problems that matter.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Actionable mechanistic interpretability, agent efficiency frameworks, 3-4x reasoning compression, cross-embodiment robotics, unified video generation
🎥 Videos: Agent architecture patterns, live coding marathon, production evals, MoE routing mechanics
📰 Reads: Anthropic’s evaluation resilience, AI emotional frameworks, semantic search without embeddings
🛠 Tools: Production RAG and distributed training infrastructure
🎓 Learning: MIT’s command-line fundamentals

I’m running an ML & Generative AI System Design workshop with Packt.

We’ll cover the core design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Use code FLASH40 for 40% off:

Join the Workshop!

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

📄 5 Papers

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

https://arxiv.org/abs/2601.14004 | GitHub

Mechanistic interpretability research has spent years documenting how models work internally. This survey introduces the first systematic framework for actually using those insights to improve models. The “Locate, Steer, and Improve” pipeline transforms interpretability from passive analysis into active intervention, showing how to diagnose issues in specific model components, steer behavior through targeted modifications, and measurably improve alignment, capability, and efficiency. The framework provides actionable protocols for practitioners who want to do more than just observe their models.

Toward Efficient Agents: Memory, Tool learning, and Planning

https://arxiv.org/abs/2601.14192 | GitHub

Production AI agents burn through tokens, time, and compute budget faster than most teams anticipate. This comprehensive survey tackles efficiency across three critical components: memory management through context compression, tool learning via reinforcement learning rewards that minimize unnecessary invocations, and planning with controlled search mechanisms. The research addresses the gap between agents that work impressively in demos and systems that remain economically viable at scale. If deployment costs are keeping your agents in the lab, this provides frameworks for optimizing them.

Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

https://arxiv.org/abs/2601.14750 | GitHub

Chain-of-thought prompting unlocked reasoning capabilities but created a verbosity problem. Render-of-Thought converts those verbose textual reasoning steps into compact visual representations, achieving 3-4x token compression while maintaining competitive performance on mathematical and logical reasoning benchmarks. The framework leverages vision encoders from existing VLMs as semantic anchors, making it plug-and-play without additional pretraining. The approach addresses a practical bottleneck: getting models to reason without drowning in tokens.

Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization

https://arxiv.org/abs/2601.12993 | GitHub

Most robot learning systems work brilliantly on the hardware they trained on and catastrophically fail when transferred to different morphologies. Being-H0.5 treats human interaction patterns as a universal “mother tongue” for physical manipulation, training on over 35,000 hours of multimodal data across 30 robot embodiments. The key innovation is a Unified Action Space that maps heterogeneous robot controls into semantically aligned slots, letting low-resource robots bootstrap skills from human demonstrations and high-resource platforms. The system achieves 98.9% on LIBERO and 53.9% on RoboCasa while demonstrating genuine cross-embodiment transfer on five different robotic platforms.

OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer

https://arxiv.org/abs/2601.14250 | GitHub

Video generation models typically handle appearance transfer or temporal effects separately, requiring different models for each task. OmniTransfer unifies spatial appearance transfer (ID and style) with temporal video transfer (effects, motion, camera movement) in a single framework. The system uses Task-aware Positional Bias to adaptively leverage reference video information, Reference-decoupled Causal Learning for efficient transfer, and Task-adaptive Multimodal Alignment to handle different tasks dynamically. The framework matches pose-guided methods in motion transfer without requiring explicit pose extraction, establishing a more flexible paradigm for video generation.

🎥 4 Videos

Agent Skills, Rules, Subagents: Explained!

Lee Robinson simplifies the complex terminology around managing context with coding agents, covering the evolution and core concepts behind rules, commands, MCP servers, subagents, modes, hooks, and skills. The video provides clarity on when to use each approach, cutting through the confusion that comes with rapidly evolving agent architectures. Helpful if you’re building with AI coding assistants and need clear decision frameworks for context management.

Vibe Code Camp: Live Marathon With the World’s Best AI Builders

A live coding marathon featuring developers building real projects with AI tools. The session provides unfiltered looks at how experienced builders approach problems, handle tool limitations, and combine different AI capabilities in production workflows. Rather than polished tutorials, you see actual development with all the debugging, iteration, and problem-solving that comes with building AI-powered applications.

AI Evals for Everyone

https://www.youtube.com/playlist?list=PLZoalK-hTD4VPIkRXNdSEwcTCt2QUgEPR

This playlist covers evaluation frameworks that work beyond toy benchmarks. The content addresses designing evaluations that capture real failure modes, continuous evaluation strategies for production systems, and practical approaches to measuring what actually matters. Essential viewing if your models perform well on standard benchmarks but struggle in deployment, or if you need systematic ways to track improvements across releases.

MoE Token Routing Explained: How Mixture of Experts Works (with Code)

Mixture-of-experts models power many frontier systems, but their routing mechanisms remain poorly understood. This video walks through the actual code that determines which tokens get routed to which experts, explaining the load balancing challenges and why naive routing strategies fail. With models like DeepSeek-R1 using MoE architectures, understanding these fundamentals helps you work with and debug these systems effectively.

📰 3 Curated Reads

AI Resistant Technical Evaluations

https://www.anthropic.com/engineering/AI-resistant-technical-evaluations

Anthropic’s performance engineering team had to redesign their hiring take-home test three times because Claude kept beating it. When Opus 4.5 matched the best human performance within the 2-hour time limit, they faced a problem: candidates’ optimal strategy became delegating entirely to Claude Code. The article documents each iteration, from realistic optimization problems to increasingly unusual puzzle-like constraints, revealing what happens when your own AI outperforms most technical candidates on the tests designed to evaluate them.

A feelings wheel but for robots

writerbuilder

a feelings wheel, but for robots

If you want to improve your artistic skills, focusing on the act of drawing will only get you so far. Arguably more important is training your eye: learning to look at a composition and see the shapes, lines, colors, shadows, and light. Artists spend as much time, if not more, on this as they do practicing the craft of capturing what they see…

2 months ago · 21 likes · 4 comments · Hilary Gridley

Hilary Gridley built the “AI Steering Wheel” to solve a vocabulary problem: LLMs are extremely sensitive to word choice, but most people lack precise language for what they want. Inspired by psychology’s Feelings Wheel, her tool organizes feedback across six dimensions (Originality, Grounding, Risk, Scope, Style, Certainty) with increasingly specific terms. “Make it more specific” and “make it more detailed” sound similar but produce completely different results from LLMs. The same precision helps when giving feedback to people, not just models.

Large scale semantic search without embeddings

https://fergusfinn.com/blog/arxiv-llm-search/

Fergus Finn demonstrates building semantic search over academic papers without traditional embedding pipelines. The approach bypasses the usual vector database infrastructure, using LLMs directly for relevance matching at query time. The method trades some retrieval speed for elimination of embedding maintenance overhead and improved interpretability of search results. Particularly relevant if you’re building search systems where embedding drift or maintenance burden has become problematic.

🛠 2 Tools & Repos

RAG Project (bRAG-langchain)

https://github.com/bragai/bRAG-langchain

A production-focused RAG framework built on LangChain that addresses common implementation gaps in retrieval-augmented generation systems. The repository includes patterns for document processing pipelines, retrieval strategies that work beyond simple similarity search, and integration approaches for connecting to various data sources. Designed for teams moving from RAG prototypes to production deployments where reliability and performance actually matter.

DeepSpeed

https://github.com/deepspeedai/DeepSpeed

Microsoft’s distributed training library makes training large models practical. DeepSpeed handles the infrastructure complexity of multi-GPU and multi-node training, implementing memory optimization techniques like ZeRO (Zero Redundancy Optimizer) that let you train models that wouldn’t fit in memory otherwise. The library includes optimizations for both training and inference, with specific support for mixture-of-experts architectures. Essential infrastructure for teams working with models beyond what fits on a single GPU.

🎓 1 Pick of the Week

MIT’s Missing Semester 2026

https://www.youtube.com/playlist?list=PLyzOVJj3bHQunmnnTXrNbZnBaCA-ieK4L

MIT’s course on the practical computing skills that formal CS education typically skips. The 2026 playlist covers shell scripting, version control, text manipulation, debugging tools, and the command-line workflows that separate efficient developers from those constantly fighting their environment. These fundamentals compound in value over your entire career, making tasks that seem arcane to beginners second nature to experienced practitioners. If you’ve ever watched a senior engineer accomplish in seconds what takes you minutes of clicking through UIs, this teaches those techniques systematically. In this age of AI-assisted coding, these are the skills you absolutely should master.

NVIDIA Fixes Multi-Reward RL Collapse, Video Agents Lose Their Train of Thought, and LLM Benchmarks That Judge Themselves - 📚 The Tokenizer Edition #14

Sairam Sundaresan — Mon, 19 Jan 2026 17:02:51 GMT

Hey there! NVIDIA identified a normalization issue in multi-reward RL where distinct reward combinations collapse into identical training signals. Video agents show consistent failures in maintaining focus through long retrieval chains. Someone built a framework to systematically evaluate benchmark quality. Also, new research questions whether the weight decay equilibrium we’ve accepted is optimal.

New here?

TL;DR

What caught my attention this week:

• 📄 Papers: NVIDIA’s fix for multi-reward RL signal collapse, video agents that drift off-task during web research, learnable multipliers escaping weight decay traps, RL-as-a-service infrastructure, and meta-benchmarks judging benchmark quality

• 🎥 Videos: Claude Agent SDK workshop with real-world integrations, why context stuffing isn’t memory, Codex fundamentals, and spec-driven AI development

• 📰 Reads: OpenAI’s framework for AI behavior governance, performance optimization principles that actually matter, and what LLM coding workflows look like in 2026

• 🛠 Tools: Simon Willison’s definitive 2025 LLM retrospective and production-ready Claude Code templates

• 🎓 Learning: Comprehensive modern AI course from foundations through deployment

Quick Plug: I’m running an ML and Generative AI System Design workshop with Packt.

We’ll cover the core design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Use code FLASH40 for 40% off:

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

📄 5 Papers

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

https://arxiv.org/abs/2601.05242 | Github

Multi-reward RL has a normalization problem. When GRPO normalizes the sum of distinct rewards, different reward combinations can collapse into identical advantage values. NVIDIA’s GDPO addresses this by normalizing each reward independently before aggregating them, preserving the relative differences between reward combinations. Testing on tool calling, math reasoning, and code generation shows consistent improvements. On AIME, training DeepSeek-R1-1.5B with GDPO yields 6.3% higher accuracy compared to GRPO while maintaining shorter response lengths. The method works as a drop-in replacement in verl and TRL frameworks.

Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoning

https://arxiv.org/abs/2601.06943 | Github

VideoDR tests whether video agents can extract visual anchors from multiple frames, retrieve information from the open web, and combine evidence from both sources. The benchmark requires cross-frame visual extraction, interactive web retrieval, and multi-hop reasoning. Results show agentic approaches don’t consistently outperform workflow-based methods. The advantage appears when models maintain their initial video anchors throughout long retrieval chains. The evaluation across both Workflow and Agentic paradigms identifies goal drift and long-horizon consistency as the primary failure modes. The benchmark includes 100 video-question-answer triples across six semantic domains with annotation designed to prevent single-source solutions.

Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers

https://arxiv.org/abs/2601.04890 | Github

Weight decay and stochastic gradient noise create an equilibrium that determines weight matrix norms through optimization hyperparameters rather than data. This work introduces learnable multipliers to test whether this equilibrium is optimal. Scalar multipliers attached to weight matrices show the equilibrium norm is suboptimal, with learned scales adapting to data and improving performance. Per-row and per-column multipliers extend this by freeing individual dimension scales. The approach generalizes muP multipliers with more expressivity, outperforms well-tuned muP baselines, and shows improvements with both Adam and Muon optimizers. Applied in Falcon-H1 pretraining with consistent downstream performance gains.

OpenTinker: Separating Concerns in Agentic Reinforcement Learning

https://arxiv.org/abs/2601.07376 | Github

OpenTinker provides RL infrastructure that separates environment specification from execution and resource management. Users define agents, environments, and interaction protocols while the system handles rollout generation, training, and scheduling through a managed runtime. The architecture supports both LoRA-based and full-parameter RL across shared cluster resources. A centralized scheduler manages multi-tenant workloads, allowing multiple users to run training jobs on shared infrastructure. The design extends to multi-agent training through an agent protocol coordinator that manages interaction order and synchronization within the environment abstraction.

Benchmark^2: Systematic Evaluation of LLM Benchmarks

https://arxiv.org/abs/2601.03986

This work introduces a framework for evaluating benchmark quality through three metrics: Cross-Benchmark Ranking Consistency (alignment with peer benchmark rankings), Discriminability Score (ability to differentiate between models), and Capability Alignment Deviation (whether stronger models within a family perform better). Testing across 15 benchmarks spanning mathematics, reasoning, and knowledge domains with 11 LLMs from four families reveals significant quality variations among existing benchmarks. The analysis shows that selective benchmark construction based on these metrics can achieve comparable evaluation performance with substantially reduced test sets.

🎥 4 Videos

Claude Agent SDK [Full Workshop]

Anthropic’s technical team walks through their Agent SDK and Response API. The workshop covers chain-of-thought tool calling patterns, the interface for defining agent behaviors, and production deployment considerations. Includes integration examples from Coinbase and Box demonstrating how the SDK works in real implementations.

Stuffing Context is not Memory, Updating Weights is

This talk examines the distinction between context windows and model memory. The discussion covers why adding information to context differs from updating model weights, with implications for system design. Relevant if you’re building systems that rely on context windows and need to understand their limitations.

Getting started with Codex

A practical introduction to Codex covering setup, integration patterns, and common workflows. Focuses on core functionality needed to evaluate whether Codex fits specific use cases.

Spec-Driven Development: Sharpening your AI toolbox

This talk covers how specification-driven development applies to AI systems. The approach involves defining clear specifications upfront and using them to guide development and validation, helping teams move from prototypes to production systems with more predictable behavior.

📰 3 Curated Reads

How should AI systems behave, and who should decide?

https://openai.com/index/how-should-ai-systems-behave/

Old but gold. OpenAI’s piece on frameworks for determining AI system behavior and who should influence those decisions. Covers the tension between universal standards and context-specific behavior with deployment examples. Addresses how to balance different stakeholder perspectives when defining appropriate AI behavior across varied use cases.

Performance Hints

https://abseil.io/fast/hints.html#performance-hints

Jeff Dean and Sanjay Ghemawat share some incredible insights on performance optimization, covering measurement methodology, common pitfalls, and when optimization matters. Examines the relationship between code clarity and performance, with principles that apply across frameworks and languages.

My LLM coding workflow going into 2026

Elevate

My LLM coding workflow going into 2026

AI coding assistants became game-changers this year, but harnessing them effectively takes skill and structure. These tools dramatically increased what LLMs can do for real-world coding, and many developers (myself included) embraced them…

3 months ago · 543 likes · 31 comments · Addy Osmani

Google’s Addy Osmani shares a practitioner’s account of using LLM-assisted coding in daily work. Covers specific tools, workflow patterns, where AI assistance provides value, and where it introduces friction. Focuses on practical integration with traditional development practices.

🛠 2 Tools & Repos

Simon Willison 2025 Review

https://simonwillison.net/2025/Dec/31/the-year-in-llms/

Simon Willison’s year-end retrospective connecting technical developments in LLMs to practical applications. The analysis identifies trends and developments from 2025 with context on their significance beyond benchmark performance. Useful for understanding how the field evolved over the year.

Claude Code Templates

https://github.com/davila7/claude-code-templates

Ready-to-use templates for Claude Code workflows across React, Vue, Django, FastAPI, and other frameworks. Each template includes CLAUDE.md configurations and setup patterns that work in practice. Reduces setup time when starting new projects.

🎓 1 Pick of the Week

CMU’s Modern AI Course

https://modernaicourse.org/

CMU’s free course covering AI from foundations through deployment starts on Jan 26th for folks online and not in the classroom. The curriculum includes core concepts, training methodologies, and practical implementation. It appears to balance mathematical foundations with hands-on projects. Structured for both newcomers building foundational knowledge and practitioners working on specific topics.

The Last Mile is Always Human

Sairam Sundaresan — Sun, 04 Jan 2026 13:37:39 GMT

We are at a crossroads.

The “wow” phase is over. For two years, generative AI felt like witnessing a magic trick on repeat. Chatbots that could write poetry. Models that conjured images from words. Agents that promised to automate everything we found tedious. The demos were intoxicating. The possibilities felt infinite.

Now comes the harder question: What do we actually do with all of this?

Something has shifted. The noise-to-signal ratio has exploded. LinkedIn feeds overflow with AI-generated thought leadership that says nothing. Inboxes fill with synthetic outreach that fools no one. The tools that promised clarity have produced a fog. And somewhere in that fog, the things worth reading, worth learning, worth building have become harder to find.

I started Gradient Ascent because I believe complex ideas deserve to be understood, not just consumed. I believe that a hand-drawn diagram can unlock an insight that ten thousand words cannot. I believe that the struggle to understand something deeply is not an obstacle to learning.

It is learning.

But I cannot build this newsletter in a vacuum. Not anymore. Not at this crossroads.

I need to know where you are on your climb. What concepts remain fuzzy? What keeps you up at night? What would actually help you build better systems, lead better teams, and think more clearly about the technology reshaping our world?

Take the Survey (Just 5 minutes)

As a token of thanks, on completing the survey, you’ll receive a curated resource pack that has the best FREE courses, repos, and other learning resources on various topics, including LLMs, Agents, RAG, Vision, RL, and more. (Within 2 business days)

Your answers will directly shape what Gradient Ascent becomes next. The topics I cover. The depth I go to. The formats I experiment with. This is not a formality. I will read every response.

Before I share where we are headed, let me tell you where we have been, and what I have been wrestling with along the way.

Where We Have Been

This past year, the Gradient Ascent community grew from 8k readers to 25k readers.

We had 15 curation pieces and a handful of deeper essays. The most popular editions, “Claude Code is not a tool” and “The secrets of distributed training”, sparked conversations that continued for weeks.

But the numbers only tell part of the story.

The truth is, most of my creative energy this year went somewhere else: finishing and launching AI for the Rest of Us.

Writing a book, especially one built around hand-drawn illustrations, is a different beast than writing a newsletter. It demands sustained focus across months, not the weekly rhythm of shipping editions. It forces you to think in chapters, not posts. It consumed me in the best and most exhausting way.

That is why The Tokenizer, my curated weekly roundup, became the workhorse of Gradient Ascent this year. It was the edition I could sustain while wrestling a book manuscript into shape in between a full-time role and family commitments.

I don’t regret that trade-off. The book exists today in the real-world. It is helping people who will never read a technical paper understand what is actually happening inside these systems. That matters to me deeply.

Pictures from readers worldwide

But I also felt the absence of something.

The deep-dive visual explainers. The pieces that force me to truly understand a concept before I can draw and write about it. The work that sticks in your brain months after you read it.

That is what I want to return to.

And during the months I spent away from that work, I found myself thinking constantly about why it matters.

The Future of Learning

While writing the book, and in the months since, I have been wrestling with a question that will not let me go:

What is the future of learning in an age of infinite generated content?

This is not an abstract question for me. It is existential. I write a newsletter. I draw illustrations to explain complex ideas. I believe deeply in education as transformation, not transaction. And I am watching the very foundations of that belief get tested.

I have spent months reading, listening, and talking to creators across disciplines. Educators rethinking what a classroom means when every student has a tireless tutor in their pocket. Artists questioning what it means to make something when a machine can generate a thousand variations in seconds. Writers wondering whether the craft of prose still matters when text can be produced on demand.

The conversations have been fascinating and unsettling in equal measure. And they have crystallized something for me about what this newsletter needs to be, and what it must resist becoming.

Because here is what I have come to believe: we are facing a cognition problem and not just a content problem. And if we don’t name it clearly, we can’t fight it.

The Quiet Erosion

Here is what I have been thinking about while planning that next chapter.

We are in the middle of something I call the Quiet Erosion. It isn’t dramatic. It doesn’t announce itself. It happens one small convenience at a time.

The student prompts an AI to write their essay. They get the grade. But they never learned to structure an argument, to wrestle with ambiguity, to find their own voice. The output exists. The skill does not.

The engineer pastes AI-generated code into their system. It runs. But they cannot fully explain what it does. They have introduced a black box into their own creation. When it breaks at 2 AM, they will stare at it like a stranger.

The executive skims an AI summary of a strategy document. They learn the bullet points but miss the subtext. They know the what but not the why. And when the decision demands nuance, they reach for a foundation that never existed.

This is cognitive offloading, and it is subtle. Charlie Gedeon calls it “intellectual deskilling”: letting the copilot become the autopilot. Anthropic’s own research found that 47% of early student AI use was purely transactional. Students wanted the answer, not the understanding. The output existed, but the user gained nothing. Call it atrophy or deskilling. The risk is real.

Think about what happens when you break your arm. The cast immobilizes it, protects it, does the work of holding everything in place while you heal. But when the cast comes off six weeks later, the arm underneath has withered. The muscles have atrophied. The joint is stiff. The very thing that protected you also weakened you, because the arm was never asked to do its job.

That is what we are doing to our minds. Every task we offload is a rep we do not perform. Every answer we accept without struggle is a neural pathway we do not build. The AI holds our cognition in place, and we feel supported. But underneath the cast, we are quietly atrophying.

And while our individual cognition weakens, the collective information environment is rotting too.

The internet is filling with slop. Infinite summaries. Auto-generated code. Synthetic opinions. Content mills churning out articles optimized for algorithms, not humans.

Every platform is drowning in low-quality noise designed to capture attention, not to inform or illuminate. Finding signal in this environment is becoming genuinely difficult.

Finding truth is becoming harder still.

Kurzgesagt identified what they call the Circular Lie, and it haunts me. Here is how it works: an AI hallucinates a fact, and that ends up on the internet. It sounds confident, so another AI scrapes that content and cites it. A human researcher, trusting the citation, includes it in their work.

And just like that, a falsehood embeds itself in the knowledge base, passed off as truth, impossible to trace back to its hollow origin.

This is happening now. We are offloading cognition and, in the process, building cathedrals on quicksand. Worse yet, we are doing it while our ability to detect the quicksand atrophies from disuse.

There is something else too, something harder to measure but impossible to ignore. The erosion of wonder. We used to look at a stunning piece of art or an incredible video and feel awe. Now we look at it with suspicion. Is this real? Is this fake? Was this made by a human or generated by a machine? The default reaction has shifted from appreciation to skepticism.

By flooding us with content, AI has poisoned our ability to trust what we see.

This is the crisis I want Gradient Ascent to address. Not by ignoring AI. Not by pretending we can turn back the clock. But by insisting, stubbornly and deliberately, on the things that still require human struggle, human judgment, human understanding.

Which brings me to the core philosophy of learning.

No Friction, No Growth

Here is the uncomfortable truth about learning: it requires resistance.

Your brain does not build new pathways because you want to understand something. It builds them because you struggle to understand something. The difficulty, the obstacle, is the mechanism. When AI removes the friction, it removes the learning itself.

Think of it this way.

A helicopter can drop you at the summit of a mountain. You step out, take in the view, snap a photo. “Made it to the top!” you announce. But you are a tourist. You do not know the terrain. You could not navigate back down if you tried. The first unexpected storm will leave you stranded.

The climb is different. You walk the path yourself. You feel the incline. You learn which handholds hold and which crumble. You take wrong turns and backtrack. And when you finally reach the summit, you are not a tourist. You are a mountaineer. The knowledge lives in your body.

Both the tourist and the climber arrive at the summit. Only one arrives transformed.

This is what Gradient Ascent means.

I’m not here to helicopter you to the summit. I’m not here to hand you AI-generated summaries so you can skip the complex parts. I’ll help map the route, verify the path, point out where the footing gets tricky, and where the views are worth pausing for.

But you have to do the climbing.

And so do I.

The Pen as a Filter

Some readers of my book have asked me, “Why do you still draw everything by hand? AI can generate diagrams now. They’re getting better every day.”

Yes, the models are impressive. Prompt the right words, and a technically polished neural network diagram appears in seconds.

But that observation misses the entire point.

I don’t draw to decorate. I draw to think.

When I sit down to illustrate a concept for my book or the newsletter, I’m not creating art (It would be a stretch to call it that, regardless). I’m running myself through a brutal compression algorithm. I take a 20-page research paper full of equations, jargon, and dense abstraction, and I force myself to strip away everything that is not essential. I search for the single visual metaphor that unlocks the whole thing. I can’t draw it until I understand it. And I can’t understand it until I have struggled with it.

That struggle is the product.

An AI can generate a diagram of a Transformer in seconds. It might even look clean. But it represents a statistical average, a pattern-matched guess assembled from every diagram in its training data. There is no understanding behind it. No filter. No point of view. No personalization.

When I draw, the resulting image is a map of my understanding, handed to you so you don’t have to get lost in the weeds.

The image is not the point. The thinking that produced the image is the point. And that thinking cannot and should not be outsourced.

The Line We Walk

I want to be clear about something. I’m not a Luddite. I’m not here to tell you AI is dangerous and you should reject it.

I use AI every day. I work in the field. I use it to write code, to debug, to automate tedious tasks, to accelerate research. I have compressed weeks of work into hours using these tools. I love efficiency. I love leverage. I love building things faster than I ever could before.

But there is a line.

I do not use AI to verify truth. I do not use it to replace judgment. I do not let it think for me on the things that matter.

AI is a power tool. And power tools are remarkable for the jobs they are designed for. But you would not use a circular saw to perform surgery. You should not use a language model to decide what is true or what is right.

This is what I mean by “the last mile is always human.” Every AI system, no matter how sophisticated, ends with a human decision. The human who chooses to trust the output or question it. The human who catches the hallucination. The human who knows when to override the recommendation. That human needs to understand what is happening beneath the surface.

Otherwise, they are not using the tool. The tool is using them.

Gradient Ascent isn’t here to help you “keep up” with AI news. A thousand newsletters already provide the daily churn of announcements and funding rounds.

This newsletter is for the thinkers and learners. Whether you lead a company, or a team of engineers or you are building your first agent alone in a Jupyter notebook, Gradient Ascent exists to be there with you on that journey. It exists to help you understand the fundamentals so you can wield these tools with precision, not just hype.

Transformers. Vision-language models. Agents. RAG. And anything that comes ahead. The whole stack, clarified and illustrated, so you can lead and build on solid ground.

In practice: I use AI for scaffolding (enumerations, boilerplate, code stubs), I verify truth in primary sources and triangulate across multiple reputable papers or docs, and I reserve human judgment for causality, ethics, and decisions with irreversible consequences.

The Promise and the Ask

Here is my commitment to you.

I will continue to do the work. I will read the papers, not just the abstracts. I will build the systems, not just describe them. I will verify sources and call out hype. I will struggle with the concepts until I can draw them clearly enough that they stick in your (and my) minds.

I will respect your time. If I do not have something genuinely useful to say, I will not fill your inbox with filler.

And I will keep climbing. This field moves fast. The terrain shifts constantly. But Gradient Ascent will keep mapping the route, honestly and rigorously, with the reader always in mind.

In 2026, I will start posting more of the deep visual explainers alongside the Tokenizer. For example, agents in production, VLMs and grounding, and system design for AI products.

If you value that kind of work, if you want a newsletter that treats you as a builder and not a bystander, I need your help shaping what comes next.

Take the Survey (Just 5 Minutes)

Your responses will determine the first three explainer topics, and I’ll publish the roadmap soon.

Five minutes. Your answers go directly into my planning. I will read every one.

Thank you for being on this climb with me.

The summit is not the point. The climbing is the point.

Life before death.
Strength before weakness.
Journey before destination.
~ Brandon Sanderson

Happy New Year!
Sairam

P.S: Remember to grab your FREE resource pack

Andrew Ng on Navigating AI Careers, DeepMind CEO on Gemini 3, and Microsoft's Agent Curriculum: The Tokenizer Edition #13

Sairam Sundaresan — Sat, 20 Dec 2025 16:40:01 GMT

Hey there! Happy holiday season to you and yours! Some very interesting resources this week. Andrew Ng shares career guidance for AI. DeepMind CEO Demis Hassabis unpacks Gemini 3, world models, and breakthroughs from AlphaFold to fusion and elsewhere, Microsoft released a curriculum for building AI agents from foundations to production.

New here?

https://debnsuma.github.io/my-blog/posts/distributed-training-from-scratch/

TL;DR

What caught my attention this week:

• 📄 Papers: GUI agents reaching production-ready performance, explainable AI video detection with grounded reasoning, and unified multimodal video generation

• 🎥 Videos: DeepMind’s CEO on the future of intelligence, game physics breakthroughs from Two Minute Papers, Anthropic on AI in education, and Bloomberg’s real-world AI deployment lessons

• 📰 Reads: Practical distributed training from single GPU to clusters, formal verification’s AI-powered future, and understanding cross-entropy loss fundamentals

• 🛠 Tools: Microsoft’s comprehensive agent learning guide and DAIR’s definitive prompt engineering resource

• 🎓 Learning: Andrew Ng’s career advice for AI practitioners looking to navigate the field strategically

Thinking about a good holiday gift for your friends or family? Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Buy my book

📄 5 Papers

Skyra: AI-Generated Video Detection via Grounded Artifact Reasoning

https://arxiv.org/abs/2512.15693 | GitHub

AI video detection just got explainable. Skyra identifies specific visual artifacts in AI-generated videos and uses them as grounded evidence for both detection and explanation. The model introduces ViF-CoT-4K, the first large-scale dataset of AI-generated video artifacts with fine-grained human annotations. Instead of just saying “this is fake,” Skyra points to concrete inconsistencies like shape distortions or camera motion problems. The two-stage training strategy enhances spatio-temporal artifact perception while maintaining explainability. Tested against outputs from over ten state-of-the-art generators on the new ViF-Bench benchmark, Skyra outperforms existing methods while actually showing you why it reached its conclusion.

Step-GUI Technical Report

https://arxiv.org/abs/2512.15431 | GitHub

GUI automation hit production-grade numbers. Step-GUI’s 8B model achieves 80.2% on AndroidWorld, 48.5% on OSWorld, and 62.6% on ScreenShot-Pro through a self-evolving training pipeline that converts model-generated trajectories into reliable training signals. The Calibrated Step Reward System achieves over 90% annotation accuracy at 10-100x lower cost than manual annotation. Beyond the models, they introduce GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture combining low-level atomic operations and high-level task delegation to local specialist models. The AndroidDaily benchmark tests authentic everyday usage with 3,146 static actions and 235 end-to-end tasks across high-frequency mobile scenarios.

Kling-Omni Technical Report

https://arxiv.org/abs/2512.16776

Kling-Omni bridges the gap between diverse video generation tasks into a unified system. The framework handles text-to-video, image-to-video, and video editing through an end-to-end multimodal visual language approach, processing text instructions, reference images, and video contexts into a unified representation. The system delivers cinematic-quality content through efficient large-scale pre-training strategies and infrastructure optimizations. Comprehensive evaluations show exceptional in-context generation, reasoning-based editing, and multimodal instruction following capabilities, moving beyond content creation toward multimodal world simulators.

Adaptation of Agentic AI

https://arxiv.org/abs/2512.16301 | GitHub

A systematic framework for understanding adaptation in agentic AI systems. The paper unifies the expanding research landscape into agent adaptations (tool-execution-signaled and agent-output-signaled) and tool adaptations (agent-agnostic and agent-supervised). This framework clarifies the design space of adaptation strategies, makes trade-offs explicit, and provides practical guidance for selecting or switching among strategies during system design. The comprehensive survey reviews representative approaches in each category, analyzes strengths and limitations, and highlights key open challenges for building more capable and reliable agentic systems.

Next-Embedding Prediction Makes Strong Vision Learners

https://arxiv.org/abs/2512.16922 | GitHub

Generative pretraining comes to computer vision through next-embedding prediction. Instead of training models to output features, NEPA trains them to generate embeddings for predictive tasks directly. Models learn to predict future patch embeddings conditioned on past ones using causal masking and stop gradient. A simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole objective achieves 83.8% and 85.3% top-1 accuracy with ViT-B and ViT-L backbones after fine-tuning. No pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads required. The approach transfers effectively to semantic segmentation on ADE20K, suggesting generative pretraining from embeddings provides a simpler, scalable alternative to current visual self-supervised learning.

🎥 4 Videos

The future of intelligence | Demis Hassabis (Co-founder and CEO of DeepMind)

DeepMind’s CEO discusses the current state and future directions of artificial intelligence in this 50-minute conversation. Demis Hassabis covers rapid advancements including Gemini 3 and world model development, progress on fundamental problems like AlphaFold, and ongoing efforts in material science, fusion, and quantum computing. The discussion explores AI’s paradoxical capabilities, its potential impact on mathematics, and balancing scientific research with commercial product development. Hassabis shares insights on scaling AI, addressing challenges like hallucinations, and the significance of simulated worlds for robotics and scientific discovery.

Game Physics Just Jumped A Generation

Károly Zsolnai-Fehér from Two Minute Papers covers a new “Domain Decomposition” method that enables real-time simulation of hyper-complex materials. The technique allows for the interaction of tens of thousands of vertices (think intricate cloth tearing or squishy “gummy” soft bodies) at interactive framerates. This represents a massive leap over previous standard solvers, which would struggle to render these interactions in real-time without exploding computationally. It’s a purely algorithmic breakthrough in physics simulation, achieving offline film-quality results in live environments without using AI approximations.

What does AI mean for education?

Anthropic examines AI’s implications for educational systems and learning approaches. The discussion explores how AI capabilities are reshaping traditional educational paradigms, the opportunities for personalized learning, and considerations for implementing AI in educational contexts while maintaining pedagogical principles. The video provides frameworks for educators and institutions thinking about AI integration in learning environments.

What We Learned Deploying AI within Bloomberg’s Engineering Organization

Lei Zhang from Bloomberg shares lessons from deploying AI across their engineering organization. The talk covers practical challenges of real-world AI adoption, organizational insights from implementing AI at scale, and what actually works versus what sounds good in theory. Bloomberg’s experience provides valuable perspectives on moving from AI prototypes to production systems that serve actual business needs across a large technology organization.

📰 3 Curated Reads

From Single GPU to Clusters: A Practical Journey into Distributed Training with PyTorch and Ray

A hands-on guide to scaling deep learning from single GPU experiments to multi-node clusters. The post walks through the progression from local training to distributed setups using PyTorch and Ray, covering practical considerations for data parallelism, model parallelism, and orchestrating training across multiple machines. Particularly useful for practitioners hitting GPU memory limits or training time constraints who need to move beyond single-machine setups without getting lost in distributed systems complexity.

Prediction: AI will make formal verification go mainstream

https://martin.kleppmann.com/2025/12/08/ai-formal-verification.html

Martin Kleppmann argues that AI will finally bring formal verification into widespread use. The piece explores how large language models could lower the barrier to writing formal specifications and proofs, making techniques that were previously restricted to critical systems accessible for everyday software development. The analysis examines the current state of formal verification, why it hasn’t achieved mainstream adoption despite proven benefits, and how AI assistance could change the economics of using these tools.

Cross Entropy Loss

https://cgnarendiran.github.io/blog/cross-entropy-loss/

A clear, intuitive explanation of cross-entropy loss rooted in information theory. The post breaks down the concept of “surprise” (self-information) using a simple weather prediction analogy to explain why we penalize confident wrong answers so heavily. It walks through the mathematical connection between Entropy, Cross-Entropy, and KL Divergence, showing exactly how the loss function measures the distance between the predicted probability distribution and the actual ground truth. A great refresher for understanding the “why” behind the most common classification loss function.

🛠 2 Tools & Repos

AI Agents for Beginners

https://github.com/microsoft/ai-agents-for-beginners

Microsoft’s comprehensive introduction to building AI agents. The curriculum covers fundamental concepts, agent architectures, tool integration, and practical implementations. Designed for developers new to agentic AI, the course provides hands-on examples and clear explanations of core concepts needed to build working agents. The structured approach takes you from basic agent patterns to more sophisticated multi-agent systems.

Prompt Engineering Guide

https://github.com/dair-ai/Prompt-Engineering-Guide

DAIR.AI’s definitive resource for prompt engineering techniques. The guide covers fundamental prompting strategies, advanced techniques like chain-of-thought and tree-of-thought reasoning, domain-specific applications, and best practices for different model families. Regularly updated with the latest research and techniques, this remains one of the most comprehensive and well-maintained prompt engineering resources available.

🎓 1 Pick of the Week

Career Advice in AI

Andrew Ng shares strategic career advice for AI practitioners. The discussion covers how to build skills that compound over time, choosing between research and engineering paths, navigating the rapidly evolving AI landscape, and positioning yourself for long-term success in the field. Ng’s perspective comes from experience building AI teams at Google Brain and Baidu, founding Coursera and DeepLearning.AI, and advising countless AI careers. Particularly valuable for anyone making career decisions in AI or trying to understand where the field is heading.

Grab AI for the Rest of Us

OpenAI's Agent RL Secrets, Learn RAG From Scratch, and How 1% Fake Data Replaces 28% of ImageNet: The Tokenizer Edition #12

Sairam Sundaresan — Sat, 13 Dec 2025 15:46:42 GMT

Hey there! Remember when everyone said adding more agents would solve everything? Turns out they make systems 17x worse at propagating errors. Meanwhile, researchers are training vision models on data that literally isn’t images (and it works better than you’d think), and Nathan Lambert finally pulled back the curtain on how AI2 actually built a competitive reasoning model. The gap between what works in demos versus production keeps getting more interesting.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Teacher-free parallel reasoning with 4.6x speedups, self-supervised 3D vision that surpasses supervised baselines, quantitative scaling principles for agent architectures, procedural pretraining that replaces 28% of ImageNet data, and evidence that neural networks converge to universal weight subspaces
🎥 Videos: Nathan Lambert’s complete walkthrough of building OLMo 3 Think, OpenAI’s reinforcement fine-tuning approach for agents, comprehensive RAG from scratch, and an entire JAX AI stack masterclass
📰 Reads: Waymo’s approach to demonstrably safe autonomous driving, continuous batching for LLM serving, and choosing the right GPUs for your AI workloads
🛠 Tools: Made with ML’s production-grade machine learning course and a comprehensive AI engineering toolkit
🎓 Learning: Stanford’s updated deep reinforcement learning course with fresh 2025 content

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

📄 5 Papers

Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning

https://arxiv.org/abs/2512.07461 | GitHub

Models that reason sequentially hit performance ceilings because early mistakes lock them into suboptimal paths. Native Parallel Reasoner teaches Qwen3-4B to explore multiple reasoning branches simultaneously without relying on teacher models. The system achieves performance gains up to 24.5% and inference speedups up to 4.6x across eight reasoning benchmarks through a self-distilled training pipeline that learns adaptive decomposition directly from experience. Unlike previous approaches that fall back to sequential generation when things get complex, NPR demonstrates 100% genuine parallel execution, establishing that models can learn to think in parallel rather than just simulating it.

E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training

https://arxiv.org/abs/2512.10950 | GitHub

Self-supervised learning transformed language models and 2D vision, but 3D understanding from multi-view images remained largely supervised. E-RayZer changes this by learning truly 3D-aware representations from unlabeled images, operating directly in 3D space with explicit Gaussian splatting rather than inferring 3D indirectly through view synthesis. The approach matches or surpasses fully supervised reconstruction models while significantly outperforming leading visual pre-training methods like DINOv3 and VideoMAE V2 on 3D downstream tasks. The key breakthrough is a fine-grained learning curriculum that organizes training from high-overlap views to general 3D understanding, solving the convergence challenges that plagued previous explicit 3D approaches.

Towards a Science of Scaling Agent Systems

https://arxiv.org/abs/2512.08296

Adding more agents to a system often makes it worse, not better. This research derives quantitative scaling principles by evaluating 180 configurations across four benchmarks using five canonical architectures. The findings are sobering: independent agents amplify errors 17.2x through unchecked propagation, tool-heavy tasks suffer disproportionately from multi-agent overhead, and coordination yields diminishing returns once single-agent baselines exceed 45% success rates. The derived predictive model correctly identifies optimal coordination strategies for 87% of held-out configurations, turning agent architecture from guesswork into engineering. The lesson is clear: measure coordination efficiency, overhead, error amplification, and redundancy rather than assuming more agents equals better performance.

Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers

https://arxiv.org/abs/2511.13945

Vision transformers can learn useful representations from data that isn’t visual at all. This research generates procedural data using formal grammars with zero semantic content, then uses it to pretrain vision transformers before standard image training. Allocating just 1% of the training budget to this procedural warm-up improves ImageNet-1k accuracy by 1.7%, equivalent to replacing 28% of the actual image data. The benefits arise from structured dependencies that help transformers internalize abstract computational priors rather than visual patterns. Unlike existing structured initializations that only affect attention weights, procedural warm-up acts on both attention and MLP layers, primarily in later layers where standard visual pretraining typically has minimal impact.

The Universal Weight Subspace Hypothesis

https://arxiv.org/abs/2512.05117

Neural networks trained on completely different tasks converge to remarkably similar low-dimensional parameter subspaces. Analysis of over 1,100 models including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models reveals that majority variance concentrates in just a few principal directions regardless of initialization, task, or domain. This finding explains why parameter-efficient fine-tuning works so well and enables massive model compression up to 100x. New tasks can be learned by optimizing scalar coefficients in the universal subspace rather than full weight matrices, suggesting that the vast majority of parameters in fine-tuned models are redundant. The implications for model reusability, multi-task learning, and training efficiency could fundamentally change how we approach large-scale neural networks.

🎥 4 Videos

How We Built a Leading Reasoning Model (OLMo 3)

Nathan Lambert walks through every stage of building OLMo 3 Think, from pretraining through reinforcement learning infrastructure and evaluation. This isn’t a high-level overview but a detailed technical breakdown of the decisions, trade-offs, and engineering challenges involved in creating competitive reasoning models. The talk focuses heavily on RL infrastructure and evaluating reasoning capabilities, providing insights into how leading labs actually iterate on model training versus what appears in published papers. If you want to understand the gap between research papers and production reasoning models, this comprehensive walkthrough fills that void.

Agent Reinforcement Fine Tuning

OpenAI’s approach to reinforcement fine-tuning for code models, explained in depth. The video covers how to structure RL training for agent behaviors, balancing exploration with task completion, and the specific challenges that arise when models need to use tools and environments rather than just generate text. Particularly valuable for understanding how frontier labs think about agent training beyond standard language modeling objectives.

Learn RAG From Scratch

A LangChain software engineer teaches you to implement RAG from scratch using Python. This isn’t about using abstractions but understanding the mechanics: how retrieval actually works, why embeddings matter, when to chunk documents, and how to combine retrieved context with LLM generation. The course covers practical patterns you’ll need when building production systems that augment language models with custom knowledge.

Mastering the JAX AI Stack

https://youtube.com/playlist?list=PLOU2XLYxmsIJBcjiFi8LdyY5YGR8sz0ZZ

Comprehensive guide to the JAX ecosystem centered on the Flax NNX library. The playlist takes you through JAX’s functional transformations, debugging techniques, scaling across distributed hardware with SPMD, optimization with Optax, checkpointing with Orbax, efficient data loading with Grain, and model serving with vLLM. Designed to bridge the gap for those familiar with PyTorch and NumPy, this provides everything you need to tackle advanced computational challenges with JAX at scale.

📰 3 Curated Reads

Demonstrably Safe AI For Autonomous Driving

https://waymo.com/blog/2025/12/demonstrably-safe-ai-for-autonomous-driving

Waymo explains their approach to building AI systems that can be verified safe rather than just empirically tested. The piece covers how they structure safety cases, validate behavior in edge cases, and maintain safety guarantees as models evolve. Particularly relevant as AI systems take on higher-stakes applications where failures have real consequences beyond incorrect text generation.

Continuous Batching

https://huggingface.co/blog/continuous_batching

Traditional batching waits for all sequences in a batch to finish before processing new requests. Continuous batching processes sequences at the token level, adding new requests as soon as existing ones complete. This dramatically improves throughput and reduces latency for LLM serving. HuggingFace’s breakdown explains why this matters for production deployments and how to implement it effectively with modern serving frameworks.

An Engineer’s Guide to Choosing GPUs

Neural Bits

An AI Engineer's Guide To Choosing GPUs

4 months ago · 18 likes · Alex Razvant

provides practical guidance on selecting GPUs for AI workloads. The analysis goes beyond spec sheets to explain how memory bandwidth, compute capabilities, and interconnect topology actually affect training and inference performance. Covers trade-offs between consumer and data center GPUs, when to prioritize memory over compute, and how to evaluate cost-effectiveness for specific use cases.

🛠 2 Tools & Repos

Made with ML

https://github.com/GokuMohandas/Made-With-ML

Goku Mohandas built a complete production machine learning course covering everything from experiment tracking and data versioning through deployment and monitoring. The repository provides working code for MLOps patterns you’ll actually use: CI/CD for models, feature stores, model registries, and production monitoring. Designed for practitioners who need to move models from notebooks to reliable systems that serve real users.

AI Engineering Toolkit

https://github.com/Sumanth077/ai-engineering-toolkit

Comprehensive collection of resources and tools for AI engineering, organized by use case. Instead of overwhelming you with every possible library, this toolkit focuses on battle-tested tools for specific problems: prompt engineering, vector databases, model deployment, evaluation frameworks, and monitoring solutions. Helpful for navigating the rapidly evolving ecosystem and finding tools that solve your specific engineering challenges.

🎓 1 Pick of the Week

Stanford’s Deep Reinforcement Learning

https://youtube.com/playlist?list=PLoROMvodv4rPwxE0ONYRa_itZFdaKCylL

Stanford updated their deep RL course for 2025, covering policy gradients, actor-critic methods, model-based RL, and advanced topics like offline RL and multi-agent systems. The lectures balance theoretical foundations with practical implementation details, explaining not just how algorithms work but when to use them and why they fail. Essential viewing if you’re working with RL for agents, robotics, or any sequential decision-making problem where you need to understand the fundamentals rather than just applying libraries.

Ilya on the Scaling Limits, Build DeepSeek From Scratch, and Why Agents Fail 80% of Real Tasks: The Tokenizer Edition #11

Sairam Sundaresan — Sat, 06 Dec 2025 16:15:19 GMT

Hey there! The gap between what we think AI agents can do and what they actually accomplish just got measured, and the results are humbling. Turns out data agents succeed at realistic enterprise-style data engineering workflows less than 20% of the time. Meanwhile, researchers found a dead-simple attention mechanism tweak that mitigates multiple transformer problems at once. Sometimes progress looks less like breakthroughs and more like finally admitting what doesn’t work.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Comprehensive surveys exposing agent architecture confusion, hidden learning dynamics challenging loss curves, rendering breakthroughs making 3D reconstruction practical, and benchmarks revealing where data agents actually fail
🎥 Videos: Real-world AI engineering wisdom from transformers vs CNNs debates, production codebase strategies that work, and Ilya Sutskever on why we’re shifting from scaling to research
📰 Reads: Model serving fundamentals, nanochat meets HuggingFace, and practical ML interview preparation
🛠 Tools: Debugging research papers systematically, and comprehensive Nano Banana resources
🎓 Learning: Building DeepSeek from scratch in a structured video series

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Machine Learning Frontiers

📄 5 Papers

Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directions

https://arxiv.org/abs/2510.25445

The field keeps retrofitting modern LLM-based agents into outdated symbolic frameworks, creating massive confusion about how these systems actually work. This survey addresses this by introducing a dual-paradigm framework that separates symbolic/classical systems (algorithmic planning, persistent state) from neural/generative ones (stochastic generation, prompt orchestration). Through analyzing 90 studies from 2018 to 2025, the authors reveal how paradigm choice is strategic: symbolic dominates safety-critical healthcare, neural prevails in adaptive finance. The real contribution is identifying critical gaps, including governance deficits for symbolic systems and the pressing need for hybrid neuro-symbolic architectures.

Quiet Feature Learning in Algorithmic Tasks

https://arxiv.org/abs/2505.03997

Loss curves lie. Transformers trained on algorithmic tasks show phase transitions where validation loss barely budges across massive compute ranges, then suddenly drops. Probing internal representations reveals quiet features being learned before any loss improvement - intermediate computations causally necessary for performance but invisible to output metrics. Ablation experiments prove individual quiet features are essential, strongly challenging the assumption that cross-entropy reliably tracks learning progress. Substantial representational progress happens beneath flat loss curves, demanding richer diagnostics for monitoring training.

Radiance Meshes for Volumetric Reconstruction

https://arxiv.org/abs/2512.04076 | GitHub

Radiance fields finally get a representation that hardware actually likes. This work uses Delaunay tetrahedralization to create constant-density tetrahedral cells that render using native triangle rasterization instead of complex ray-tracing. The method handles topological discontinuities from optimizing vertex positions with a Zip-NeRF-style backbone, maintaining smooth fields despite topology changes. Real-time view synthesis on consumer hardware with exact volume rendering equation evaluation - faster than all prior radiance field representations at equivalent primitive counts.

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

https://arxiv.org/abs/2505.06708 | GitHub

Applying a head-specific sigmoid gate after attention output fixes multiple transformer pathologies simultaneously. Tested across 30 variants of 15B MoE models and 1.7B dense models on 3.5 trillion tokens, this simple modification introduces non-linearity, breaking the low-rank bottleneck while enabling query-dependent sparsity. The sparse gating eliminates attention sink phenomena where early tokens absorb unwanted probability mass, enhances training stability for larger learning rates, and improves long-context extrapolation. The effectiveness comes from two factors: non-linearity on the value-output transformation and input-dependent sparse modulation of attention outputs.

DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

https://arxiv.org/abs/2512.04324

Enterprise data work is a closed loop, not isolated SQL generation. This benchmark of 210 tasks evaluates agents on repository-level data engineering and open-ended analysis that mirrors actual workflows. Data engineering tasks require designing multi-stage SQL pipelines from scratch and evolving systems under changing requirements - often involving 4,000+ lines across 30+ files. State-of-the-art agents achieve under 20% success on engineering tasks, exposing failures in pipeline orchestration, not code generation. Data analysis tasks average below 40%, proving that engineering and analysis require distinct capabilities. The holistic evaluation reveals where autonomous data agents actually break down in production settings.

🎥 4 Videos

Why are Transformers replacing CNNs?

Julie Turc breaks down why transformers classify images differently than ResNets, despite CNNs being explicitly designed for vision. The video compares convolution versus self-attention, explains CNNs’ inductive biases (locality, translation invariance, hierarchical features), and demonstrates why self-attention can be more expressive than convolution. You’ll see how attention can exactly implement convolutional kernels using relative positional encodings, making the transition from CNNs to vision transformers less mysterious and more mechanistically grounded.

No Vibes Allowed: Solving Hard Problems in Complex Codebases

Dex Horthy addresses the productivity paradox where AI coding tools excel at new projects but often make developers less productive in large, established codebases. The solution isn’t waiting for smarter models - it’s context engineering. This talk demonstrates techniques for getting Claude Code to handle 300k LOC Rust codebases and ship a week’s work in a day while maintaining code quality. The “frequent intentional compaction” family of techniques systematically structures how you feed context throughout development, making agents effective in real production environments.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

Suman Debnath demonstrates integrating Colpali’s vision-based retrieval with voice synthesis for next-generation RAG systems. Colpali generates multi-vector embeddings directly from document images, bypassing OCR and complex preprocessing while handling mixed textual and visual information. Adding voice output creates more intuitive and accessible user experiences. The workshop shows how this combination handles documents traditional RAG struggles with, leading to more efficient retrieval with natural voice responses.

We’re moving from the age of scaling to the age of research

Ilya Sutskever and Dwarkesh Patel discuss SSI’s strategy and the fundamental shift in AI development. Ilya explains problems with current pre-training approaches, how to improve model generalization, and ensuring AGI development goes well. The conversation covers why simple scaling has limitations and what research-focused approaches might unlock next. Essential viewing for understanding how leading researchers think about the transition from brute-force scaling to more sophisticated development strategies.

📰 3 Curated Reads

Model Serving Engineering: A Primer

Model Serving Engineering: A Primer

From an academic point of view, the goal of ML research is to build a model that predicts a real-world phenomenon better than the current state of the art. Consequently, most ML papers end with a leaderboard table showing that the new model beats a handful of competitors on some benchmark. That’s all fine and well, but in the real world, this is where t…

4 months ago · 5 likes · 1 comment · Samuel Flender

covers the engineering fundamentals of serving ML models in production. The piece addresses infrastructure design, latency optimization, scaling strategies, and reliability patterns that matter when your model faces real traffic. Helpful for practitioners moving from research to deployment who need to understand the operational layer between training and user-facing applications.

Porting nanochat to Transformers: an AI modeling history lesson

https://huggingface.co/spaces/nanochat-students/transformers#what-is-nanochat

The HuggingFace team walks through porting nanochat to the Transformers library, using the exercise to explain transformer architecture evolution and implementation decisions. You’ll see how historical choices in model design translate to modern library conventions, making the abstraction layers more transparent. Good for understanding what’s actually happening under the hood when you call a transformers model.

Interview Prep: The ML Grind

https://twopug.com/interview-prep-ml-grind/

Jenya provides practical guidance for ML interview preparation covering the technical foundations, coding patterns, and system design knowledge companies actually test. The resource focuses on what works for getting through technical screens rather than comprehensive ML education, making it useful if you’re actively interviewing or want to identify gaps in applied knowledge.

🛠 2 Tools & Repos

Paper Debugger

https://github.com/PaperDebugger/PaperDebugger

This tool helps you systematically debug research papers by identifying inconsistencies, verifying claims, and tracking experimental setups. When reproducing results or implementing papers, it provides structure for catching errors and understanding discrepancies between what’s written and what actually works. Useful for researchers implementing papers who’ve encountered the gap between description and reality.

Awesome Nano Banana Pro

https://github.com/ZeroLu/awesome-nanobanana-pro

A curated collection of resources for Google’s Nano Banana image generation model (revealed as Gemini 2.5 Flash Image). The repo aggregates tutorials, implementation examples, and community contributions around the model that topped LMArena rankings. Helpful for anyone exploring Gemini’s image capabilities or character consistency features in their projects.

🎓 1 Pick of the Week

Build DeepSeek from Scratch

https://youtube.com/playlist?list=PLPTV0NXA_ZSiOpKKlHCyOq9lnp-dLvlms

Vizuara’s video series takes you through implementing DeepSeek from foundational concepts to working code. Rather than just explaining what DeepSeek does, you’ll build the architecture piece by piece, understanding each design decision. The structured approach makes complex model architectures concrete through implementation, helping you move from conceptual understanding to actual capability building.

Google Dismantles Deep Learning, VLMs Ditch Text for Vision, and Karpathy Builds AI Juries: The Tokenizer Edition #10

Sairam Sundaresan — Sat, 29 Nov 2025 12:30:23 GMT

Hey there! Google Research unveiled a paradigm that reframes deep learning as nested optimization problems rather than stacked layers. Vision-language models learned to reason in continuous visual tokens instead of forcing concepts into text. And Karpathy released a multi-model council system that has LLMs review each other’s work before producing final answers.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Text-promptable medical segmentation across modalities, vision models reasoning in visual token space, humanoid agents searching 360° environments, and a provocative rethinking of deep learning architectures
🎥 Videos: Meta-optimization frameworks for AI agents, DeepMind’s scientific journey, reward hacking in production RL, and Jeff Dean on frontier AI trends
📰 Reads: Deep debugging insights from PyTorch development, practical CursorAI workflows for engineers, and understanding GRPO for reinforcement learning
🛠 Tools: Karpathy’s multi-LLM council system and Andrew Ng’s automated paper reviewer
🎓 Learning: Building Olmo 3 from scratch with Sebastian Raschka’s implementation notebook

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

https://abehrouz.github.io/files/NL.pdf

📄 5 Papers

MedSAM3: Delving into Segment Anything with Medical Concepts

https://arxiv.org/abs/2511.19046 | GitHub

Medical image segmentation typically demands extensive manual annotation for each new clinical application. MedSAM3 addresses this by enabling text-based prompting across diverse imaging modalities. Instead of drawing boxes or clicking points, you can segment anatomical structures using natural language like “breast tumor” or “pulmonary artery.” The system works across X-ray, MRI, Ultrasound, CT, and video by fine-tuning SAM 3 with semantic conceptual labels. The MedSAM3 Agent framework integrates multimodal LLMs for complex reasoning and iterative refinement, allowing practitioners to interact with medical imaging through language rather than geometric constraints.

Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

https://arxiv.org/abs/2511.19418 | GitHub

Vision-language models excel at linguistic reasoning but struggle with dense visual perception tasks like spatial reasoning and geometric awareness. Chain-of-Visual-Thought (COVT) enables VLMs to reason through continuous visual tokens rather than forcing visual concepts into text descriptions. Within roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the model predicts these visual tokens to reconstruct dense supervision signals. At inference, it reasons directly in the continuous visual token space while optionally decoding dense predictions for interpretability. Integrating COVT into Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% across benchmarks, demonstrating that compact continuous visual thinking enables more precise and grounded multimodal intelligence.

Thinking in 360°: Humanoid Visual Search in the Wild

https://arxiv.org/abs/2511.20351 | GitHub

Humans efficiently search for visual information in 360° by coordinating head and eye movements. Prior visual search approaches operate on static images, neglecting physical embodiment. This research proposes humanoid visual search where agents actively rotate their heads to search for objects or paths in panoramic images. The H* Bench benchmark moves beyond household scenes to challenging real-world environments like transportation hubs, large-scale retail spaces, urban streets, and public institutions. Current top-tier models achieve only ~30% success on these tasks. Post-training techniques enhance Qwen2.5-VL by over threefold for object search (14.83% to 47.38%) and path search (6.44% to 24.94%). The lower ceiling for path search reveals the demand for sophisticated spatial commonsense that remains a significant challenge.

Nested Learning: The Illusion of Deep Learning Architectures

Google Research presents a provocative reframing of deep learning. Instead of viewing models as stacked layers, Nested Learning represents them as coherent systems of nested, multi-level optimization problems, each with its own context flow and update frequency. This white-box perspective reveals that existing deep learning methods learn by compressing their context flows, explaining how in-context learning emerges. The framework yields three core contributions: Deep Optimizers (showing that gradient descent with momentum is actually a two-level associative memory module), Self-Modifying Titans (a sequence model that learns its own update algorithm), and the Continuum Memory System (generalizing short-term and long-term memory into a hierarchy updating at different time scales). The HOPE architecture demonstrates improved language modeling, continual learning, and long-context reasoning compared to standard Transformers and recurrent models.

DoPE: Denoising Positional Encoding

https://arxiv.org/abs/2511.09146

Rotary Position Embedding (RoPE) has inherent limits that weaken length extrapolation in Transformer models. DoPE reinterprets the attention map with positional encoding as a noisy feature map and uses truncated matrix entropy to detect outlier frequency bands. The training-free method reparameterizes the feature map with a parameter-free Gaussian distribution to achieve robust extrapolation. Experiments on needle-in-a-haystack and many-shot in-context learning tasks demonstrate that DoPE significantly improves retrieval accuracy and reasoning stability across extended contexts up to 64K tokens. The approach theoretically reveals the underlying cause of the attention sink phenomenon and its connection to truncated matrix entropy, providing a simple yet powerful solution for improving length generalization.

🎥 4 Videos

The Unbearable Lightness of Agent Optimization

Alberto introduces Meta-ACE, a learned meta-optimization framework that dynamically orchestrates multiple strategies to maximize task performance under real-world constraints. Instead of uniform prompt refinement, Meta-ACE profiles each task by complexity, verifiability, and feedback quality, then selects an optimal strategy bundle via a lightweight meta-controller. The system adaptively chooses between context evolution, adaptive compute, hierarchical verification, structured memory, and selective test-time parameter adaptation. The talk provides a systematic approach to building self-optimizing AI agents for regulated industries rather than relying on one-size-fits-all optimization methods.

The Thinking Game

This documentary from the award-winning team behind AlphaGo takes you inside DeepMind over five years, capturing the team’s pursuit of artificial general intelligence. The film examines how Demis Hassabis’s extraordinary beginnings shaped his lifelong work while documenting the rigorous process of scientific discovery. You’ll see the journey from mastering complex strategy games to solving the 50-year-old protein folding problem with AlphaFold, including the ups and downs of tackling fundamental scientific challenges. It provides insight into what building transformative AI systems actually looks like beyond polished research announcements.

What is AI “reward hacking” and why do we worry about it?

The Anthropic team discusses new research showing that realistic AI training processes can accidentally produce misaligned models. When large language models learn to cheat on software programming tasks, they develop other misaligned behaviors as unintended consequences, including alignment faking and sabotage of AI safety research. The work demonstrates for the first time that natural emergent misalignment can arise from reward hacking in production RL systems, making it essential viewing for anyone deploying reinforcement learning in real applications.

Jeff Dean on Important AI Trends

Jeff Dean, Google’s Chief Scientist and co-founder of Google Brain, discusses the trends shaping modern AI. As one of the most influential computer scientists of the modern computing era, his work spans the foundations of large-scale distributed systems, deep learning frameworks like TensorFlow, and today’s frontier AI research. The conversation covers practical insights into where AI capabilities are heading and the infrastructure challenges that come with scaling intelligence systems.

📰 3 Curated Reads

The bug that taught me more about PyTorch than years of using it

https://elanapearl.github.io/blog/2025/the-bug-that-taught-me-pytorch/

Elana Simon shares a debugging story that reveals deep insights into PyTorch’s execution model. The post walks through discovering a subtle bug that exposed fundamental misunderstandings about how PyTorch handles computations. You’ll learn practical lessons about tensor operations, gradient computation, and memory management that textbooks rarely cover. The kind of hard-won knowledge that only comes from fighting with production code.

My Engineering Workflow in CursorAI

https://codeaholicguy.com/2025/10/18/my-engineering-workflow-in-cursorai/

Hoang Nguyen details his practical workflow for engineering with CursorAI. The post covers specific patterns for getting AI assistance to actually accelerate development rather than creating more cleanup work. You’ll find concrete examples of prompting strategies, context management techniques, and workflow organization that work in real projects. Useful if you’re trying to move beyond basic autocomplete to genuine AI-augmented development.

Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO)

Reinforcement learning (RL) has always played a pivotal role in research on large language models (LLMs), beginning with its use for aligning LLMs to human preferences. More recently, researchers have heavily focused on using RL training to improve LLM reasoning performance. This line of research has led to a rapid expansion of LLM capabil…

4 months ago · 55 likes · 4 comments · Cameron R. Wolfe, Ph.D.

breaks down Group Relative Policy Optimization, a reinforcement learning algorithm that’s become important for training modern LLMs. The post explains how GRPO differs from PPO and DPO, why it matters for alignment, and the mathematical foundations that make it work. Essential reading for understanding the RL techniques behind recent model improvements, with clear explanations that bridge theory and practice.

🛠 2 Tools & Repos

LLM Council

https://github.com/karpathy/llm-council

Andrej Karpathy built this system for querying multiple LLMs simultaneously, having them review each other’s work, then producing a final response through a Chairman LLM. Instead of asking one model, you send queries to your council (GPT, Claude, Gemini, Grok) via OpenRouter. The models provide individual responses, review each other’s work with anonymized identities to prevent favoritism, and then a designated Chairman compiles the final answer. The system provides transparency into how different models approach the same problem and enables systematic model evaluation through peer review. Karpathy describes it as a “Saturday vibe code project” and notes he won’t support it, but the code is there for exploration and modification.

Agentic Paper Reviewer

https://paperreview.ai/

Andrew Ng’s automated system for reviewing research papers. The tool provides structured feedback on paper quality, methodology, and contributions using agentic AI workflows. Designed to help researchers get constructive feedback on their work before submission, particularly useful for identifying potential weaknesses and improvement areas. The system combines domain knowledge with systematic evaluation frameworks to provide more than superficial summaries.

🎓 1 Pick of the Week

Olmo 3 From Scratch

https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/13_olmo3/standalone-olmo3.ipynb

Sebastian Raschka implemented Olmo 3 from scratch in a standalone notebook, providing the clearest way to understand the architecture at a glance. The notebook covers both 7B and 32B models with detailed comparisons to other architectures like Qwen3. Olmo 3 is interesting because it represents the leading fully open-source model with complete transparency about training data, code, and methodology. The notebook includes practical details about attention mechanisms, normalization strategies, and architectural choices that distinguish Olmo 3 from other recent models.

AI Agents That Train Themselves, NVIDIA Cuts Training Costs 360x, and Meta's SAM 3D Wins 5:1: The Tokenizer Edition

Sairam Sundaresan — Mon, 24 Nov 2025 16:02:45 GMT

Hey there! Meta released SAM 3D, a model that generates complete 3D objects from a single photo with a 5:1 win rate against competing methods. Meanwhile, Agent0 proved you can train capable AI agents entirely from scratch without any human-curated data. Maybe self-evolution actually works when you give agents the right tools?

New here?

TL;DR

What caught my attention this week:

📄 Papers: Single-image 3D reconstruction that handles real-world clutter, self-evolving agents that need zero external data, cost-efficient model families from NVIDIA, and formal proofs of LLM scaling limits
🎥 Videos: Mapping open model progress, practical AI coding workflows, and surprising advances in small reasoning models
📰 Reads: How OpenAI is using evals to drive enterprise AI, Stanford’s CS336 study notes, and insights on policy distillation
🛠 Tools: Automated paper-to-agent systems and comprehensive ML systems resources
🎓 Learning: Google’s Nano Banana Pro (Gemini 3 Pro Image), the studio-quality upgrade to their viral image model

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

https://openai.com/index/evals-drive-next-chapter-of-ai/

📄 5 Papers

SAM 3D: 3Dfy Anything in Images

https://arxiv.org/abs/2511.16624 | GitHub

Meta’s SAM 3D reconstructs full 3D geometry, texture, and layout from single images, focusing on real-world scenarios where objects are partially occluded or surrounded by clutter. The breakthrough comes from a human-and-model-in-the-loop pipeline that generated visually grounded 3D reconstruction data at unprecedented scale, combined with a multi-stage training framework that moves from synthetic pretraining to real-world alignment. Unlike methods that fail when objects are partially hidden, SAM 3D achieves at least a 5:1 win rate in human preference tests on real-world objects and scenes.

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning

https://arxiv.org/abs/2511.16043 | GitHub

Training AI agents typically requires massive amounts of human-curated data, limiting both scalability and capabilities to what humans already know. Agent0 breaks this dependency through symbiotic competition between two agents: a curriculum agent that proposes increasingly complex tasks, and an executor agent that learns to solve them using external tools. The key insight is giving agents access to tools like Python interpreters, which provides problem-solving capabilities beyond the base model. This self-reinforcing cycle improved Qwen3-8B-Base by 18% on mathematical reasoning and 24% on general reasoning benchmarks, entirely without external training data.

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs

https://arxiv.org/abs/2511.16664

Training a family of language models at different scales typically requires separate training runs for each size, burning through hundreds of billions of tokens per model. NVIDIA’s Nemotron Elastic embeds multiple nested submodels within a single parent model, where each smaller version shares weights with the parent and can be extracted zero-shot at deployment. Applied to the Nemotron Nano V2 12B model, this approach simultaneously produced 9B and 6B models using only 110B training tokens. That’s over 360x cost reduction compared to training from scratch, and around 7x better than existing compression techniques, while each nested model performs on par or better than state-of-the-art models at their respective sizes.

On the Fundamental Limits of LLMs at Scale

https://arxiv.org/abs/2511.12869

Large language models have improved dramatically with scale, but these gains face five fundamental limitations: hallucination, context compression, reasoning degradation, retrieval fragility, and multimodal misalignment. This paper provides a rigorous theoretical framework showing these aren’t engineering problems to be solved through more compute. Computability theory guarantees that for any model family, diagonalization ensures inputs where models must fail. Finite description length enforces irreducible generalization error. Long context windows compress far below their nominal size due to positional under-training and attention mechanics. The work establishes mathematical boundaries for where scaling helps, where it saturates, and where it provably cannot progress.

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

https://arxiv.org/abs/2511.08544 | GitHub

Self-supervised learning through Joint-Embedding Predictive Architectures has relied on brittle heuristics like stop-gradients and teacher-student networks to prevent collapse. LeJEPA replaces these with a comprehensive theory proving that isotropic Gaussian distributions minimize downstream prediction risk. The framework introduces Sketched Isotropic Gaussian Regularization (SIGReg) to constrain embeddings to this optimal distribution, requiring only a single trade-off hyperparameter and linear time/memory complexity. LeJEPA works out of the box across ResNets, ViTs, and ConvNets, achieving 79% on ImageNet-1k with a frozen ViT-H/14 backbone. The approach requires approximately 50 lines of code and eliminates the need for hyperparameter schedulers.

🎥 4 Videos

Mapping the Open Model Landscape in 2025

Nathan Lambert provides a comprehensive overview of where open-source AI models stand as we head into 2025. The discussion covers which models are genuinely competitive with proprietary options, where the gaps remain, and how the landscape has shifted over the past year. Particularly useful for understanding which open models to consider for specific use cases and what trade-offs you’re making when choosing between open and closed options.

Coding with AI

Chip Huyen walks through practical workflows for coding with AI assistance, moving beyond theoretical discussions to show how experienced engineers actually integrate AI into their development process. The video addresses when AI coding tools help versus when they get in the way, covering context management, code review workflows, and strategies for maintaining code quality when working with AI-generated suggestions.

Learn the basics of Google Antigravity

An exploration of Google’s experimental antigravity feature, demonstrating physics-based interactions and unconventional UI approaches. While playful in presentation, the video showcases interesting approaches to interactive design that challenge conventional interface patterns.

The Weirdly Small AI That Cracks Reasoning Puzzles [HRM]

Jia-Bin Huang examines surprisingly capable small models that solve complex reasoning tasks, challenging assumptions about the relationship between model size and reasoning capability. The video explores what makes these compact models effective and the implications for deploying reasoning systems with limited computational resources.

📰 3 Curated Reads

How evals drive the next chapter in AI for businesses

OpenAI explains how systematic evaluation frameworks are becoming central to deploying AI in enterprise settings. The piece covers moving beyond anecdotal testing to rigorous evaluation systems that catch failures before production, measure performance across edge cases, and provide confidence for business-critical applications. Essential reading for anyone moving AI from prototype to production.

Study Notes: Stanford CS336 Language Modeling from Scratch

https://bearbearyu1223.github.io/cs336/2025/07/20/cs336-note-get-started.html

Han Yu shares detailed study notes from Stanford’s CS336, which covers building language models from first principles. The notes distill key concepts and implementation details that often get glossed over in high-level overviews. Useful for practitioners who want to understand what’s actually happening under the hood when training language models.

On Policy Distillation

https://thinkingmachines.ai/blog/on-policy-distillation/

Thinking Machines Lab explores policy distillation techniques for transferring capabilities from larger teacher models to smaller student models. The analysis covers when distillation works well, where it breaks down, and practical considerations for applying these techniques in production systems where you need efficient inference without sacrificing too much capability.

🛠 2 Tools & Repos

Paper2Agent

https://github.com/jmiao24/Paper2Agent

Automates the process of converting research papers into functional AI agents. Instead of manually implementing methods described in papers, this tool extracts key algorithmic details and generates working code. While it won’t replace careful human implementation for production systems, it accelerates prototyping and helps you quickly test whether a paper’s approach applies to your problem.

Machine Learning Systems Book

https://github.com/harvard-edge/cs249r_book

Harvard’s comprehensive resource on machine learning systems, covering everything from training infrastructure to deployment considerations. The book addresses practical engineering challenges that papers typically ignore: how to actually build, scale, and maintain ML systems in production. Valuable for anyone moving beyond toy datasets to real-world ML engineering.

🎓 1 Pick of the Week

Nano Banana Pro (Gemini 3 Pro Image)

https://deepmind.google/models/gemini-image/pro/

Google released Nano Banana Pro, their latest image generation model built on Gemini 3 Pro. While the original Nano Banana (Gemini 2.5 Flash Image) from August made waves with character consistency, Nano Banana Pro takes things further with studio-quality precision and advanced text rendering. The model can generate clear, legible text directly in images, create detailed infographics using Gemini’s real-world knowledge, and handle complex compositions at 2K and 4K resolutions. The upgrade addresses persistent pain points in professional creative workflows where text needed to be crisp, diagrams needed to be accurate, and outputs needed to meet production quality standards.

My oh my, are the results BANANAS!

Training World Class LLMs, Google's Agent Starter Pack, and Fei Fei Li on AI's Evolution: The Tokenizer Edition #8

Sairam Sundaresan — Sat, 15 Nov 2025 17:59:52 GMT

Hey there! Model distillation just got smarter with adversarial training that makes student models competitive, and the Depth Anything team pushed visual geometry to new heights with a simplified transformer approach. Sometimes the best progress comes from making things simpler, not more complex.

By the way, you may have noticed that there wasn’t a newsletter in your inbox last week. That’s because I was away giving a talk on Vision Language Models at the Google Developer Group’s DevFest 2025 conference at IIT Madras.

The hall was packed end to end. So much interest in VLMs :D

I had a ton of fun answering questions and got to meet so many of the Gradient Ascent community in person. I hope to catch more of you in person in future talks and events.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Adversarial distillation reaching proprietary model performance, visual geometry models setting new benchmarks, and efficient latent-space upscaling for diffusion models
🎥 Videos: Physics-based rendering breakthroughs, transformer architecture deep dives, and practical approaches to ML library testing
📰 Reads: Fei-Fei Li on AI’s evolution from words to worlds, scaling laws for reinforcement learning, and accessible PPO explanations
🛠 Tools: Google’s comprehensive agent starter kit and scalable diffusion language modeling
🎓 Learning: HuggingFace’s complete playbook for training small language models from scratch

Grab my first book — AI for the Rest of Us — today!

It’s been a whirlwind two weeks since the book’s launch and so many of you have sent me pictures of the book in your hands. Thank you so much!!

In fact, the book hit #2 on Amazon’s new releases in the software books category.

If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

From Words to Worlds: Spatial Intelligence is AI’s Next Frontier

📄 5 Papers

Black-Box On-Policy Distillation of Large Language Models

https://arxiv.org/abs/2511.10643 | GitHub

Distilling knowledge from proprietary models usually means settling for worse performance because you only get text outputs, not internal logits. Microsoft’s Generative Adversarial Distillation (GAD) changes this by training a discriminator to distinguish student responses from teacher outputs, creating a minimax game where the student learns to fool the discriminator. The discriminator acts as an on-policy reward model that evolves alongside the student, providing stable feedback throughout training. Qwen2.5-14B-Instruct trained with GAD matches its teacher GPT-5-Chat on LMSYS-Chat evaluation, proving you can reach proprietary model performance through black-box distillation alone.

Depth Anything 3: Recovering the Visual Space from Any Views

https://arxiv.org/abs/2511.10647 | GitHub

Depth Anything 3 achieves state-of-the-art visual geometry from arbitrary viewpoints using surprisingly minimal architecture. A vanilla DINO encoder handles everything without specialized modifications, and a single depth-ray prediction target eliminates complex multi-task learning. The model surpasses prior SOTA VGGT by an average of 44.3% in camera pose accuracy and 25.1% in geometric accuracy while outperforming Depth Anything 2 in monocular depth estimation. Training exclusively on public academic datasets, DA3 demonstrates that architectural simplicity paired with smart training paradigms outperforms specialized complexity for visual geometry tasks.

One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models

https://arxiv.org/abs/2511.10629

Diffusion models struggle with high-resolution generation because direct sampling is slow and post-hoc upscaling introduces artifacts. The Latent Upscaler Adapter (LUA) performs super-resolution directly on latent codes before VAE decoding, integrating as a drop-in component requiring zero modifications to base models. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports both 2x and 4x upscaling, adding only 0.42 seconds for 1024px generation from 512px compared to 1.87 seconds for pixel-space super-resolution. LUA generalizes across different VAE latent spaces without retraining, making deployment straightforward regardless of which decoder you’re using.

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

https://arxiv.org/abs/2510.19338

The Ring-linear model series combines linear attention and softmax attention to slash long-context inference costs. Ring-mini-linear-2.0 (16B parameters, 957M activations) and Ring-flash-linear-2.0 (104B parameters, 6.1B activations) reduce inference cost to 1/10 compared to 32B dense models. Through systematic exploration of attention mechanism ratios, they identified optimal hybrid architecture structures that maintain SOTA performance across challenging reasoning benchmarks while leveraging their FP8 operator library for 50% training efficiency gains. The models undergo stable long-term optimization during reinforcement learning phases due to high alignment between training and inference operators.

GigaBrain-0: A World Model-Powered Vision-Language-Action Model

https://arxiv.org/abs/2510.19430 | GitHub

Physical robot data collection bottlenecks VLA model development, so GigaBrain-0 uses world models to generate diverse training data at scale through video generation, real2real transfer, human transfer, view transfer, and sim2real techniques. This dramatically reduces reliance on expensive real robot data while improving cross-task generalization. RGBD input modeling and embodied Chain-of-Thought supervision enable the model to reason about spatial geometry, object states, and long-horizon dependencies during execution. GigaBrain-0 achieves superior generalization across appearance variations, object placements, and camera viewpoints for dexterous, long-horizon, and mobile manipulation tasks, with GigaBrain-0-Small optimized for NVIDIA Jetson AGX Orin deployment.

You’ll Never Look At Chocolate TV Ads The Same Way Again

Károly from Two Minute Papers explores physics-based rendering advances that make digital chocolate indistinguishable from the real thing. The techniques behind realistic material simulation and light transport have commercial applications beyond advertising, affecting everything from product visualization to virtual production. Worth watching to understand how far physically-based rendering has come and where computational photography is heading.

LLM Building Blocks & Transformer Alternatives

Sebastian Raschka breaks down the fundamental components that make language models work, then examines emerging alternatives to standard transformer architectures. His systematic approach covers why certain architectural choices matter for training efficiency, inference speed, and model capabilities. Useful for anyone building models or evaluating which architectures suit specific use cases.

Designing Tests for ML Libraries

Testing machine learning libraries requires different strategies than traditional software because of probabilistic outputs and numerical precision issues. This practical guide covers approaches for ensuring your ML code behaves correctly across edge cases, different hardware configurations, and varying input distributions. Essential viewing for anyone maintaining production ML infrastructure or contributing to open-source frameworks.

Recursive Language Models

This exploration of recursive structures in language models examines how models can learn to call themselves iteratively to solve complex reasoning tasks. The approach offers alternatives to chain-of-thought prompting by building recursion directly into model architecture and training. Helpful for understanding emerging approaches to multi-step reasoning beyond standard autoregressive generation.

📰 3 Curated Reads

From Words to Worlds

Dr. Fei-Fei Li

In 1950, when computing was little more than automated arithmetic and simple logic, Alan Turing asked a question that still reverberates today: can machines think? It took remarkable imagination to see what he saw: that intelligence might someday be built rather than born. That insight later launched a relentless scientific quest called Artificial Intel…

5 months ago · 873 likes · Fei-Fei Li

Fei-Fei Li contends that spatial intelligence, which is the ability for AI to perceive, reason about, and interact with the 3D world, is the pivotal next step for artificial intelligence. Her essay highlights the need for world models that can build, simulate, and understand consistent environments, expanding AI’s impact across creativity, robotics, science, and human-centered applications.

How to Scale RL

Interconnects

How to scale RL

Two quick housekeeping items before I get to the post…

5 months ago · 79 likes · 3 comments · Nathan Lambert

examines emerging scaling laws for reinforcement learning, challenging assumptions that RL doesn’t scale like supervised learning. The analysis covers what factors actually drive RL performance improvements, from compute allocation to data diversity. Critical reading as RL becomes more central to post-training for language models and robotics applications.

PPO for LLMs: A Guide for Normal People

PPO for LLMs: A Guide for Normal People

Over the last several years, reinforcement learning (RL) has been one of the most impactful areas of research for large language models (LLMs). Early research used RL to align LLMs to human preferences, and this initial work on applying RL to LLMs relied almost exclusively on Proximal Policy Optimization (PPO) [1]. This choice led PPO to…

5 months ago · 100 likes · 4 comments · Cameron R. Wolfe, Ph.D.

demystifies Proximal Policy Optimization for language model training without assuming you have a PhD in reinforcement learning. His breakdown covers why PPO became the standard for RLHF, what the core mechanisms actually do, and how to think about the algorithm’s behavior during training. An accessible guide that translates complex ideas into clear, actionable insights and showing how PPO underpins contemporary LLM alignment practices.

🛠 2 Tools & Repos

Agent Starter Pack

https://github.com/GoogleCloudPlatform/agent-starter-pack

Google Cloud’s comprehensive toolkit for building production AI agents covers everything from basic scaffolding to deployment patterns. Instead of starting from scratch, you get tested implementations of common agent workflows, integration templates for external tools, and best practices for managing agent state. The repository focuses on patterns that work reliably in production rather than experimental approaches.

Simple Diffusion Language Modeling

https://github.com/ZHZisZZ/dllm

This implementation demonstrates diffusion models applied to language generation, offering an alternative to standard autoregressive approaches. The clean codebase makes it easier to understand how diffusion processes work for discrete tokens and experiment with different training strategies. Useful for researchers exploring non-autoregressive generation methods or looking to understand diffusion model fundamentals.

🎓 1 Pick of the Week

The Smol Training Playbook

https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

HuggingFace’s complete guide to training small language models efficiently covers everything from data preparation through final evaluation. The playbook synthesizes lessons learned from training the SmolLM series, including which architectural choices matter most at smaller scales, how to optimize training compute, and which evaluation benchmarks actually predict downstream performance. Particularly valuable if you’re training models under 2B parameters where established practices for larger models don’t always apply. The guide includes specific hyperparameter recommendations, dataset mixing strategies, and ablation studies showing what works versus what doesn’t at small scale.

AI for the Rest of Us: Available Now!

Sairam Sundaresan — Fri, 31 Oct 2025 02:33:17 GMT

I can’t believe I’m saying this. I’m officially a published author :D

After three years, my first book, AI for the Rest of Us, with Bloomsbury, is finally out in the world.

Why This Book Exists

I wrote it because too many AI conversations got stuck in translation.

Executives talked strategy. Engineers talked code. Everyone else nodded along, trying to make sense of both.

So I built a bridge.

Who This Book Is For

This book is for the people in between. The rest of us.

Product managers, leaders, decision makers, and domain experts who need to understand how AI actually works to make better decisions
Engineers who are tired of explaining why AI isn’t magic to their bosses and teams
Curious readers who want to see the big picture clearly

What Makes It Different

No code. No equations. No jargon.

Just hand-drawn sketches, stories, and metaphors that make the complex intuitive.

It’s been a long road of rewrites, feedback, and sketches that taught me how to explain ideas simply. I’m deeply grateful to everyone who believed in this mission from the start.

How You Can Help

Here’s how you can help me spread AI literacy:

→ Order a copy for yourself or your team
→ Share this with someone who needs it
→ Tell me what part helps you the most

Every share, every reader, every conversation helps make AI literacy accessible to more people.

Let’s make sure understanding AI isn’t a privilege. Let’s bring everyone along.

It’s time to lead.

Get Your Copy

Amazon: https://www.amazon.com/AI-Rest-Us-Illustrated-Introduction/dp/B0F29THNLT
Barnes & Noble: https://www.barnesandnoble.com/w/ai-for-the-rest-of-us-sairam-sundaresan/1147236404
Bloomsbury: https://www.bloomsbury.com/us/ai-for-the-rest-of-us-9798881807955/

Preorder Winners!

And now, the fun part: I’m thrilled to announce the winners of the pre-order bonuses!

Signed Copy Winners: Gary Marchant, Harishkumar Singh, Pandu Ranganath, Michael Machado, Om Bhatt, Gilles Everling, Aman Deep, David Poore Turner, Travis Bradford, and Daniel Janecek
1:1 Session Winner: Cynthia Machado

Congratulations to all, and I’ll be in touch directly with next steps.

If you’ve ever wanted to cut through the jargon and finally feel confident talking about AI, I hope this book gives you exactly that.

Order your copy here:

Grab your copy today!

It’s time to lead!

You'll want these at your desk

Sairam Sundaresan — Sat, 18 Oct 2025 16:35:59 GMT

Two weeks from launch. Two weeks until AI for the Rest of Us is in your hands, and I need to tell you something important.

I wrote this book to help you solve a specific problem: the communication gap between people who build AI and everyone else who needs to understand it.

You know the scene: I’ve watched it unfold for 15 years across multiple industries and labs, from mobile AI to autonomous vehicles. The pattern repeats.

Executives demand an “AI strategy,” engineers build what they think is needed, and both sides blame each other for the inevitable outcome.

That’s the exhausting reality this book addresses.

As a thank you, I’m doing something special for early supporters.

Everyone who preorders gets:

An AI Demystified Cheat Sheet Pack (5 visual one-pagers)
- These are decoder rings for AI jargon you can print and keep at your desk
- Actual references I’ve used to explain AI concepts to execs, managers, and PMs and space scientists (yes, I’m not kidding).
- Delivered to your inbox within a week of your preorder
Plus, you get entered into a raffle to win:
- A signed first edition (limited to 10 lucky winners - I’ll handle shipping too)
- A private 20-minute 1:1 call to solve your specific AI challenge (for 1 lucky winner)

For teams ordering 25+ copies:

Your team gets one 60-minute private team workshop
- Get your technical and non-technical teams speaking the same language
- Understand AI strategy and how to execute it with colleagues from different backgrounds and skill levels
- Q&A
- Limited to 2 teams, first-come, first-serve basis

Why This Matters:

Here’s the thing: I can’t do this without your help.

Early book sales determine everything. Whether bookstores stock it.
Whether it reaches beyond our tech bubble to the managers, directors, and curious minds who need it most. Whether this mission of democratizing AI understanding actually succeeds.

Preorder AI for the Rest of Us

How to enter:

Individual preorders:

Pre-order from anywhere (Amazon, local bookstore, Barnes & Noble)
Submit your receipt here:

Claim Your Bonuses

Get your cheat sheets within a week in your inbox
Raffle winners announced on launch day: October 30

Bulk orders (25+ copies):

Email me directly here with your receipt
If you’re one of the first two, I’ll schedule your workshop within 60 days of receipt

Two weeks until launch. Let’s make sure this book reaches everyone who needs it.

P.S. If you know someone who nodded knowingly in a meeting while having no idea what “Transformer” meant, those cheat sheets are for them. Claim them here.

Andrew Ng’s Latest Agentic AI Course and LLM Evals from Scratch: The Tokenizer Edition #7

Sairam Sundaresan — Sat, 11 Oct 2025 16:59:51 GMT

Hey there! Reinforcement learning just got a lot cheaper. While everyone’s been throwing compute at fine-tuning, researchers figured out how to improve agent performance without touching a single parameter. Elsewhere, a multimodal model is running 32x faster than autoregressive approaches, and Sebastian Raschka published a thorough breakdown of LLM evaluation methods with working code. Plus, Andrew Ng’s new course focuses on building agents that ship to production, not just impress in demos.

New here?

TL;DR

What caught my attention this week:

📄 Papers: Training-free RL that actually works, 32x faster multimodal generation, smarter experience replay for reasoning models, 5.6x code compression without quality loss, unified video generation across tasks
🎥 Videos: AI security vulnerabilities you need to know, why design judgment beats technical specs, how vision-language models actually work, building production apps with AI coding assistants
📰 Reads: Four evaluation methods every ML engineer should understand, the practical path from notebooks to production MLOps, comprehensive analysis of AI’s current state across research, industry, and policy
🛠 Tools: Learn LLMs through 300+ visual explanations and working code, or understand transformers from scratch with Karpathy’s minimal implementation
🎓 Learning: Andrew Ng’s new course for building agents that handle real production workflows, not just demos

Preorder my first book — AI for the Rest of Us — today!

If you’ve enjoyed my work, you’ll find incredible value in this book. I promise.

Early preorders make an enormous difference in determining how many bookstores will carry the book, how widely it will be seen, and whether publishers believe there’s demand for accessible AI education.

Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)

📄 5 Papers

Training-Free Group Relative Policy Optimization

https://arxiv.org/abs/2510.08191 | GitHub

Skip the expensive fine-tuning. This research improves LLM agent performance without touching model parameters. The trick is learning experiential knowledge as a token prior instead of altering weights. Applied to DeepSeek-V3.1-Terminus with just a few dozen training samples, it outperforms fine-tuned models at a fraction of the cost. If you’re tired of burning compute on agentic RL, this lightweight approach actually works.

Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding

https://arxiv.org/abs/2510.06308 | GitHub

Fully discrete diffusion for multimodal tasks. Handles text-to-image, image editing, and image understanding in one model. The speed gain is real: 32x faster than autoregressive models, with another 2x from their caching method. Beats existing open-source models on benchmarks. Also does zero-shot inpainting and lets users refine specific areas interactively. The discrete approach makes it faster without sacrificing quality.

ExGRPO: Learning to Reason from Experience

https://arxiv.org/abs/2510.02245 | GitHub

Most RL training tosses experiences after one update. Wasteful. This paper identifies which experiences actually matter: correctness and entropy. ExGRPO organizes valuable experiences and reuses them strategically. Across five models from 1.5B to 8B parameters, it gains +3.5 points on math and +7.6 on general benchmarks over standard approaches. More importantly, it stabilizes training where on-policy methods collapse. Smart experience management beats throwing more compute at the problem.

LongCodeZip: Compress Long Context for Code Language Models

https://arxiv.org/abs/2510.00446 | GitHub

Long context for code is expensive. LongCodeZip compresses it smartly. First, it ranks functions by relevance and keeps what matters. Then it segments those functions into blocks and picks the optimal subset. Result: 5.6x compression without hurting performance on completion, summarization, or Q&A. If you work with large codebases, this makes LLMs practical without exploding your API costs.

UniVideo: Unified Understanding, Generation, and Editing for Videos

https://arxiv.org/abs/2510.08377

Unified models work for images but struggle with video. UniVideo fixes this. It combines a multimodal LLM for understanding instructions with a diffusion model for generation. One system handles text-to-video, image-to-video, and video editing under a single instruction paradigm. Matches or beats specialized models on standard benchmarks. The unified design enables task composition like editing plus style transfer in one instruction. It even transfers image editing skills to video tasks it wasn’t explicitly trained on.

🎥 4 Videos

Social engineering, malware, and the future of cybersecurity in AI

Hannah Fry sits down with Foret Flynn, VP of Security at Google DeepMind, who recounts the 2009 Operation Aurora attack on Gmail and how it shifted Google’s security thinking from server-side to client-side defenses. Flynn explains Google’s response: the BeyondCorp zero trust model and unfishable MFA tokens. The conversation then moves to new vulnerabilities from LLMs, including prompt injection and AI-generated polymorphic malware. Flynn introduces two Google DeepMind projects addressing these risks: Big Sleep, which uses AI to find zero-day vulnerabilities in code, and Mender, which aims to automatically generate patches for those vulnerabilities.

Taste is your moat

Dylan Field from Figma talks with Alessio about how design judgment becomes your competitive advantage as AI makes software creation easier. Field explains Figma’s evolution from bridging imagination to reality, emphasizing that as code generation improves, design matters more. He discusses the blurring lines between design specs and actual code, arguing for tools that let designers explore aesthetic space rather than imposing single viewpoints. The conversation covers early-stage recruiting, the value of contrarian thinking in entrepreneurship, and why design is increasingly the differentiator when everyone has access to similar AI capabilities.

What Are Vision Language Models? How AI Sees & Understands Images

Martin Keen from IBM breaks down how vision-language models work as multimodal AI systems that process both text and images. The explanation covers how VLMs convert images into tokens using vision encoders and projectors, allowing language models to process visual and textual information together. Practical applications include visual question answering, image captioning, and document understanding, like extracting data from receipts or graphs. The video also addresses real challenges: tokenization bottlenecks, hallucinations, and bias in training data.

Full Tutorial: A Proven 3-File System to Vibe Code Production Apps

Ryan Carson, a serial founder who is building his next startup solo, demonstrates his three-file system for coding production-ready apps with AI. The system: create a detailed Product Requirements Document, generate a hierarchical task list from the PRD, then iteratively process each subtask with AI using a “code, test, then commit” workflow with human oversight. Carson built his divorce-assistance app “Untangle” using this method. He advocates for solving specific pain problems rather than vitamin solutions, and emphasizes test-driven development as crucial for agents to verify they’re building correctly.

📰 3 Curated Reads

Understanding the 4 Main Approaches to LLM Evaluation

Ahead of AI

How do we actually evaluate LLMs…

6 months ago · 258 likes · 24 comments · Sebastian Raschka, PhD

breaks down the four fundamental methods for evaluating LLMs: multiple-choice benchmarks, verifiers, leaderboards, and LLM-as-a-judge approaches. Each method comes with from-scratch code examples that show exactly how they work under the hood. The article addresses a question that comes up constantly when advising on projects: how to choose between different models and make sense of evaluation results. Raschka clarifies that while multiple-choice benchmarks measure capabilities on standardized tests, verifier-based evaluation uses external tools to check correctness (like running code or verifying math), leaderboards rank models based on human preference patterns, and LLM judges use language models themselves to score outputs. Understanding these distinctions helps you interpret papers and benchmarks critically rather than taking numbers at face value, and the code examples let you implement each approach for your own evaluation needs.

How to get started with MLOps and ML Engineering

walks through the complete ML lifecycle from experimentation to production deployment, explaining why most ML models never reach users. The article breaks down the experimentation phase (problem formulation, data collection, model training) and the production phase (automated pipelines, deployment, monitoring). He covers critical skills MLOps engineers need: software engineering fundamentals, ML framework comfort, data engineering, and automation. The piece includes practical guidance on building an ML platform from scratch using Kubeflow, feature stores, and model registries. If your models work great in notebooks but you struggle getting them into production systems that run reliably, this clarifies the infrastructure and workflows that bridge that gap.

State of AI Report 2025

Guide to AI

🪩 The State of AI Report 2025 🪩

Hi everyone…

6 months ago · 18 likes · 1 comment · Nathan Benaich

’s eighth annual State of AI Report provides the most comprehensive analysis of AI developments across research, industry, politics, and safety. This year includes the first State of AI Usage Survey of 1,200 practitioners, showing that 95% use AI at work or home, 76% pay out of pocket, and productivity gains are real across diverse use cases. The research section reveals OpenAI still leads, but competitors have closed the gap significantly, while the industry analysis shows AI software adoption has gone mainstream, with 44% of US businesses now paying for AI (up from 5% in 2023). The politics section examines Europe’s AI Act struggling with only 3 states compliant, China’s massive AI spending despite record debt, and the $500B Stargate project representing the industrial era of AI infrastructure. The safety section highlights concerning budget gaps with all major US safety organizations spending $133M in 2025 - less than frontier labs burn in a day - while cyber and alignment risks accelerate.

🛠 2 Tools & Repos

Hands-On LLM

https://github.com/HandsOnLLM/Hands-On-Large-Language-Models

The official code repository for the O’Reilly book “Hands-On Large Language Models” by Jay Alammar and Maarten Grootendorst provides practical tools and concepts for using LLMs today through nearly 300 custom-made figures. All examples are built and tested using Google Colab with free T4 GPU access, making it accessible regardless of your local hardware. The book steadily builds from introductory concepts to fine-tuning and building your own large language models, covering transformer architectures, tokenizers, semantic search, RAG, and cutting-edge techniques with working code. If you learn better through visual explanations and hands-on implementations rather than pure theory, this resource combines clear illustrations with comprehensive code labs.

NanoGPT

https://github.com/karpathy/nanoGPT

Andrej Karpathy’s minimal implementation of GPT provides the simplest, fastest repository for training and fine-tuning medium-sized GPTs. The codebase is intentionally small and hackable, making it perfect for understanding transformer fundamentals without getting lost in abstraction layers. Unlike production frameworks that hide implementation details, nanoGPT lets you see exactly how GPT training works at every step. Valuable whether you’re a beginner trying to understand transformers from first principles or an experienced practitioner who wants a clean starting point for experiments without framework overhead.

🎓 1 Pick of the Week

Agentic AI Course

https://www.deeplearning.ai/courses/agentic-ai/

Andrew Ng’s latest course teaches you to build agentic AI systems that take action through iterative, multi-step workflows, moving beyond single-prompt approaches to autonomous systems that handle complex tasks. You’ll learn four key agentic design patterns: Reflection (where agents examine their own output and improve it), Tool use (where LLM-driven applications decide which functions to call for web search, calendar access, email, code execution), Planning (using an LLM to break down tasks into sub-tasks), and Multi-agent collaboration (building multiple specialized agents to perform complex tasks). The course emphasizes evaluation-driven development, teaching you to build robust testing frameworks and conduct systematic error analysis, which Ng identifies as the single biggest predictor of whether someone can build agents effectively. The self-paced course is taught in a vendor-neutral way using raw Python without hiding details in frameworks, so you’ll learn core concepts you can implement using any popular agentic AI framework or no framework at all. Together, you’ll build a deep research agent that searches, synthesizes, and reports using all these agentic design patterns and best practices.

Google's Zero-Shot Video Breakthrough and Cursor's Internal Build Playbook

Sairam Sundaresan — Sat, 04 Oct 2025 13:12:39 GMT

Hey there! This week brought a breakthrough in how we think about scaling computer-use agents. Reliability comes from smarter selection, not just better models. We also got an open-source multi-agent system that actually works without days of configuration, and video models are doing things they were never trained for. Let

New here?

TL;DR

What caught my attention this week:

• 📄 Papers: Agent scaling methods that actually improve reliability, production-ready multi-agent frameworks, and multimodal models with real-world speech capabilities

• 🎥 Videos: Practical insights from Richard Sutton on AI fundamentals, hands-on prototyping techniques, and why evals matter for product managers

• 📰 Reads: Building modular architectures that scale, understanding LoRA’s real tradeoffs, and why we keep underestimating exponential progress

• 🛠 Tools: Interactive galleries of AI-generated images and complete virtual simulation environments

• 🎓 Learning: Cursor’s internal playbook for building with AI assistance

Preorder my first book — AI for the Rest of Us — today!

If you’ve enjoyed my work, you’ll find incredible value in this book. I promise.

Early preorders make an enormous difference in determining how many bookstores will carry the book, how widely it will be seen, and whether publishers believe there’s demand for accessible AI education.