New here?

The Tokenizer is my resource-focused newsletter edition where I curate high-signal AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

Subscribe now

TL;DR

What caught my attention this week:

• 📄 Papers: a source-level dissection of Claude Code, a self-evolving agent that argues for information density over context length, ByteDance’s next-generation video model, Tencent Hunyuan’s unified 3D world generator, and a principled recipe for on-policy distillation.

• 🎥 Videos: the physics behind Flow Matching, running LLMs locally on DGX Spark, engineering RL environments from scratch, and an anonymous operator conversation on AI budgets and code review.

• 📰 Reads: Hamel Husain on why eval craft belongs to data scientists, Paul Iusztin on harness engineering as the new OS layer around the LLM, and a transcript from inside OpenAI Frontier where a 1M-LOC codebase ships at a billion tokens per day with no human-written code.

• 🛠 Tools: Microsoft’s new agent framework with Semantic Kernel and AutoGen migration paths, and a Karpathy-style autoresearch loop for GPU kernels.

• 🎓 Learning: BMAD-METHOD, an open agile framework with six named agents that scaffolds AI-driven teams from brief to deployment.

📄 5 Papers

1. Dive into Claude Code: The Design Space of Today’s and Future AI Agent Systems

https://arxiv.org/abs/2604.14228 | GitHub

VILA Lab walked Claude Code’s TypeScript source (v2.1.88) and found that roughly 1.6% of the codebase is AI decision logic. The other 98.4% is operational infrastructure: permissions, context compaction, hooks, subagent isolation, session persistence. They extract five human values and thirteen design principles from that code, then trace them through seven components and five layers using a single running example (fixing a failing test in auth.test.ts). Read it if you’re building an agent and want to see how a production harness answers every design question at once.

2. GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0)

https://arxiv.org/abs/2604.17091 | GitHub

Long-horizon agent performance is bounded by information density inside a finite context window, not by context length itself. GenericAgent builds on four pieces. A minimal atomic tool set. Hierarchical on-demand memory. A self-evolution loop that turns verified trajectories into reusable SOPs and code. A runtime compression layer. The repo reports 188K tokens on long-horizon tasks where Claude Code spends 537K, at equal or better completion rates. Read this before you reach for the next context-length upgrade.

3. Seedance 2.0: Advancing Video Generation for World Complexity

https://arxiv.org/abs/2604.14148

ByteDance Seed’s latest video model takes the #1 spot on both Text-to-Video and Image-to-Video leaderboards on Arena.AI, ahead of veo-3.1-audio-1080p on T2V and grok-imagine-video-720p on I2V. On the team’s own SeedVideoBench 2.0 set (against Kling, Sora, Veo, Wan, and Vidu), it ranks first in 29 of 30 fine-grained motion quality categories. The architectural bet is unified audio-video joint generation with native multi-modal input (text, image, audio, video) and simultaneous multi-track audio output with binaural synthesis. Pull the report if you want to see the current ceiling in open-access video generation.

4. HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

https://arxiv.org/abs/2604.14268 | GitHub

A single pipeline that does both 3D world generation and reconstruction, producing navigable 3D Gaussian Splatting scenes from text or one image alone. Tencent Hunyuan’s four-stage approach covers panorama generation, trajectory planning, camera-guided view generation, and final 3D composition. It reaches state-of-the-art among open-source 3D world models and reports parity with the closed Marble system. All weights and code are public, which matters for a field where most of the good stuff is still locked behind APIs.

5. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

https://arxiv.org/abs/2604.13016 | GitHub

On-policy distillation keeps breaking in counterintuitive ways: a stronger teacher sometimes fails to improve a student while a weaker one succeeds. THUNLP’s team runs controlled experiments across the Qwen3 and DeepSeek families and lands on two governing conditions. Student and teacher need compatible thinking patterns (measurable via token-level overlap ratio), and the teacher needs to offer capabilities the student doesn’t already have (not just a higher benchmark score). They also propose an off-policy cold-start warm-up that recovers distillation in setups where it would otherwise collapse. If your post-training pipeline picks teachers by benchmark score, the overlap-ratio diagnostic here is the first thing to add.

🎥 4 Videos

1. The Physics Behind Flow Matching, Derived from Scratch

Flow Matching built up from the continuity equation and time-variant velocity fields, with the training loss arriving only after continuous normalizing flows are in hand. Julia Turc’s 24-minute walkthrough pairs with a free interactive tutorial at diffusion.fyi. It covers conditional velocity fields in the spot where most explanations either gloss over optimal transport or collapse into notation. Queue this when your current mental model of flow matching has a gap you cannot quite name.

2. Running LLMs Locally: Practical Performance on NVIDIA’s DGX Spark

Serving open-source models from 1.5B to 14B parameters on a single DGX Spark with vLLM, with the throughput, latency, and quantization trade-offs that drop out once you measure them. Mozhgan Kabiri Chimeh from NVIDIA shares a reproducible methodology, NVFP4 performance numbers on Grace Blackwell’s 128GB unified memory, and a framework for local model sizing. A 10-minute talk if you’re deciding whether on-prem compute clears the bar for your workload.

3. Engineering Reinforcement Learning Environments Like Software

RL environments treated as first-class software artifacts, not research scaffolding. Stefano Fiorucci, AI engineer at Deepset, uses the open-source Verifiers library to translate classical RL concepts to language models. He builds single-turn tasks, multi-turn games, and tool-using agents as concrete environments. Watch it before you write your next eval harness. A clean gym spec will save you rebuilding it three weeks in.

4. Stay Sassy and swyx on AI Budgets, Per-Person Token Spend, and Why Code Review Matters More

Per-person token budgets are a management problem, not a provisioning one, and code review gets more important as agents write more code, not less. That is the spine of a 57-minute Latent Space conversation between swyx and Stay Sassy, an anonymous operator with voice modulated for opsec. They cover build-vs-buy, where hand-coding still wins, and how real engineering leaders are allocating AI spend. Closest thing to eavesdropping on the hard numbers right now.

📰 3 Curated Reads

1. The Revenge of the Data Scientist

https://hamel.dev/blog/posts/revenge/

Training a model was never most of a data scientist’s job. The bulk of the work was running experiments to test how well a system generalizes, debugging stochastic behaviour, and designing metrics you can actually trust. Hamel Husain argues that LLM teams who cut data scientists out of the loop are now rediscovering the same five eval pitfalls, from off-the-shelf judge metrics that flatter bad systems to dashboards nobody would ship a classifier against. The piece names the concrete moves (error analysis from real traces, aligning LLM judges against human labels, constructing trust-metrics you would actually stake a launch on) and is the cleanest argument this year for why the old craft is now the new infrastructure.

2. Agentic Harness Engineering

Decoding AI Magazine

Agentic Harness Engineering

At the AI start-up I’ve been working at, building a financial personal assistant, we implemented LlamaIndex, added the Model Context Protocol (MCP), and built complex Retrieval-Augmented Generation (RAG) pipelines. Each piece added complexity without adding direct business value…

Read more

25 days ago · 78 likes · 12 comments · Paul Iusztin

The harness is where production agents actually live: tools, memory, sandboxes, and orchestration that let an LLM recover from failures, bridge context windows, serve multiple interfaces, and hold state across sessions. Paul Iusztin dissects the production harness behind Claude Code and Codex across a handful of components (context engineering, memory, sandboxing, tool layers, orchestration). He walks through how each piece solves a problem the model cannot solve on its own. The thesis: strip RAG pipelines and MCP layers back to plain Python until the harness itself earns the complexity.

3. Extreme Harness Engineering for Token Billionaires

Latent.Space

Extreme Harness Engineering for Token Billionaires: 1M LOC, 1B toks/day, 0% human code, 0% human review — Ryan Lopopolo, OpenAI Frontier & Symphony

We’re proud to release this ahead of Ryan’s keynote at AIE Europe. Hit the bell, get notified when it is live! Attendees: come prepped for Ryan’s AMA with Vibhu after…

Listen now

18 days ago · 45 likes · 4 comments

OpenAI’s Frontier team wrote a 1M-line codebase at 1 billion tokens per day with zero human-written code and zero human-reviewed merges. Ryan Lopopolo walks through how they got there. The build system went from Make to Bazel to Turbo to Nx to hit a sub-one-minute loop. Observability was exposed to the agent so it can tell when it is going off track. Dependencies were inlined to remove version drift. Specs were written for the model, not the engineer. The conversation with Swyx and Alessio covers the agent code-review rules, autonomous merging, and why human attention is now the binding constraint. Read it before you argue with anyone about what a production agent harness should look like in 2026.

🛠 2 Tools & Repos

1. Microsoft Agent Framework

https://github.com/microsoft/agent-framework

Microsoft’s unified framework for building and orchestrating AI agents across Python and .NET, with graph-based workflows, streaming, checkpointing, human-in-the-loop, and time-travel. The interesting move is that it ships with explicit migration guides from both Semantic Kernel and AutoGen, positioning itself as the convergence point for Microsoft’s prior agent SDKs. 9,736 stars and active weekly office hours. If you already run Semantic Kernel or AutoGen in production, the migration guides are the fastest read on where Microsoft wants your next agent to live.

2. AutoKernel: Autoresearch for GPU Kernels

https://github.com/RightNow-AI/autokernel

Karpathy-inspired autonomous loop that takes any PyTorch model, profiles it, extracts bottleneck kernels into standalone Triton or CUDA C++, and optimizes them one at a time in a keep-or-revert loop. Each experiment runs in about 90 seconds, roughly 40 per hour, 320 overnight. The orchestrator picks the next kernel using Amdahl’s law. 1,295 stars. Point it at a model you care about before bed and wake up to a speedup report.

🎓 1 Pick of the Week

BMAD-METHOD: Breakthrough Method for Agile AI Driven Development

https://github.com/bmad-code-org/BMAD-METHOD

BMAD-METHOD is an MIT-licensed framework that installs a full agile team of named AI agents into your IDE. You get six roles: Mary (business analyst), John (PM), Winston (architect), Sally (UX), Amelia (senior engineer), and Paige (tech writer). Each carries a consistent persona, a defined skill set, and a handoff point to the next. The core module ships 34+ workflows across brainstorming, PRD drafting, architecture, sprint planning, and code review. A solo developer can run a structured delivery loop without a real team. Install is one command (`npx bmad-method install`), and the framework picks up Claude Code, Cursor, and other IDEs on first run. 45K+ stars, actively maintained. Pick it up if you have been reaching for individu coding agents and find the scaffolding thinner than the model itself.

That’s the twenty-fifth Tokenizer. If one of these fifteen resources changes what you’re building this week, forward the edition to whoever should see it next, and come find the long-form work at Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate high-signal AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

Subscribe now

TL;DR

What caught my attention this week:

• 📄 Papers: a cleaner RL gradient signal that doesn’t collapse late in training, the GUI agent stack done end-to-end for once, memory-aware reward shaping that notices recurrent failure modes, a four-frame streaming video baseline that quietly beats thirteen heavyweight ones, and retrieval supervision pulled straight from agent trajectories.

• 🎥 Videos: Notion on custom-agent evals at scale, the Gemma 4 architecture delta, Claude Cowork for non-engineers, and a recipe for LLM judges that don’t silently drift.

• 📰 Reads: Agent swarms writing faster CUDA kernels at Cursor, a hands-on multimodal embedding tutorial from Hugging Face, and a clean first-principles tour of mathematical modeling.

• 🛠 Tools: VoxCPM2, a tokenizer-free multilingual TTS system, and a viral Claude/Codex plugin that cuts output tokens ~75% by making the agent talk like a caveman.

• 🎓 Learning: Stanford’s Spring 2026 Frontier Systems course, weekly lectures from the people building the AI infrastructure stack.

  ---

📄 5 Papers

1. Self-Distilled RLVR

https://arxiv.org/abs/2604.03128

On-policy self-distillation (OPSD) was supposed to give RL training a denser gradient signal. In practice it leaks privileged teacher information into the student and destabilizes training past the early-peak stage. RLSD keeps RLVR’s environmental feedback as the direction signal and uses self-distillation only to set token-level update magnitudes. The result: +4.69% average over the base LLM and +2.32% over GRPO across five multimodal reasoning benchmarks (MMMU, MathVista, MathVision, ZeroBench, WeMath), without the late-stage collapse OPSD exhibits.

2. ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

https://arxiv.org/abs/2604.11784

GUI agents aren’t stuck because the models are weak. They’re stuck because the training pipelines, eval harnesses, and deployment layers keep breaking in public, and nobody’s addressed all three at once. The authors cover all three layers: RL training with GiGPO plus a process reward model, 95.8% evaluation reproduction across 6 benchmarks, and cross-platform deployment to Android, HarmonyOS, and iOS with hybrid CLI-GUI control. ClawGUI-2B trained inside this pipeline hits 17.1% success on MobileWorld GUI-Only, beating the same-scale MAI-UI-2B baseline by 6 points and larger untrained models like UI-Venus-72B.

3. The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping

https://arxiv.org/abs/2604.11297

RL for LLMs keeps collapsing into the same wrong answers. Entropy regularization penalizes randomness within the current policy but ignores recurrent failure patterns across rollouts. MEDS stores historical model representations, clusters them with HDBSCAN to surface common error modes, and down-weights reward for rollouts landing in high-density error clusters. Up to 4.13 pass@1 points gained on math reasoning benchmarks, with measurably higher behavioral diversity and negligible compute overhead.

4. A Simple Baseline for Streaming Video Understanding

https://arxiv.org/abs/2604.02317

Streaming-video models keep stacking heavier memory mechanisms. A sliding window of 4 recent frames into an off-the-shelf VLM (SimpleStream) matches or beats 13 specialized baselines: 67.7% on OVO-Bench and 80.59% on StreamingBench. The deeper finding is a perception-memory trade-off. Longer context often helps recall but hurts real-time perception. Future streaming benchmarks need to separate the two or they’ll keep rewarding machinery for its own sake.

5. LRAT: Learning to Retrieve from Agent Trajectories

https://arxiv.org/abs/2604.04949 | GitHub

Retrieval models trained on human click and dwell logs are mismatched to how LLM agents query and consume results. LRAT derives retrieval supervision directly from agent trajectories (browses, unbrowsed rejections, and post-browse reasoning traces) with weighted relevance intensity. Evidence recall, task success, and execution efficiency all improve across agent scales, and the BM25 and FAISS pipelines are in the repo.

🎥 4 Videos

1. Notion’s Head of AI Engineering on Running Custom-Agent Evals at Scale

When every Notion user has a bespoke agent, eval becomes a combinatorial problem. Sarah Sachs (head of AI engineering at Notion) and co-founder Simon Last walk Latent Space through how they keep the signal alive through that explosion. Technical interview, not product marketing. Worth it if you’re past the “does the agent work on my test case” stage and need patterns for eval at scale.

2. What’s New in Gemma 4

A 60-second official overview from Google DeepMind on what changed between Gemma 3 and Gemma 4. Treat it as a trailer, not a tech report. The architecture and training deltas it calls out are the right ones to chase down in the model card afterward. Worth including because the open-weights frontier is where half the interesting work this edition lives.

3. Claude Cowork Tutorial for Non-Engineers with JJ Englert (Tenex)

First hands-on Claude Cowork walkthrough aimed at people without an engineering background. JJ Englert runs end-to-end setup and a real workflow, which is the kind of demo that travels because it proves the tool works without scaffolding. Watch this for the first clean demo of Cowork running end-to-end without an engineer in the loop.

4. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA

Mahmoud Mabrouk from Agenta AI at AIE Europe on building LLM judges that don’t silently drift. GEPA (Gradient-free Evaluator Prompt Adaptation) is a practical recipe for keeping your judge calibrated as the underlying model and task distribution shift. Directly useful if you’re running eval harnesses in production and can’t afford quiet regressions.

📰 3 Curated Reads

1. Speeding Up GPU Kernels by 38% With a Multi-Agent System

https://www.cursor.com/blog/multi-agent-kernels

Cursor and NVIDIA show a coordinated agent swarm writing CUDA kernels (CUDA C with inline PTX, CuTe DSL, a shared markdown scratchpad as the coordination medium) that hits a 38% geomean speedup across 235 real kernels. Standout wins include 84% on grouped-query attention and real gains on MoE GEMMs where human-tuned baselines were already aggressive. Read this for the concrete coordination pattern, not the headline number alone.

2. Multimodal Embedding and Reranker Models With Sentence Transformers

https://huggingface.co/blog/multimodal-sentence-transformers

Tom Aarsen’s step-by-step guide to building cross-modal retrieve-and-rerank over text, image, audio, and video using Qwen3-VL-Embedding and Reranker. Working code you can copy tonight, with the full retrieve-then-rerank pipeline wired up in Sentence Transformers. Rare to get the full modern multimodal RAG stack wired up in one notebook. Clone it, run it, then decide whether you need anything more than this.

3. The Power of Mathematical Modeling

The Palindrome

The Power of Mathematical Modeling

Hey! It’s Tivadar from The Palindrome…

Read more

18 days ago · 43 likes · 3 comments · Tivadar Danka and Manlio De Domenico, Ph.D.

Manlio De Domenico guest-posts on Tivadar Danka’s Palindrome with a first-principles tour of mathematical modeling: the SI compartment model applied to the ILOVEYOU worm, a four-compartment SIZR extension for zombie outbreaks, and a network-science result showing that immunizing hubs beats random interventions. The SI-to-SIZR extension is the kind of move that sticks. You watch a textbook model flex to handle zombies without losing any of its explanatory structure, which is exactly the feel you want from a modeling tour. The hub-immunization result at the end is the payoff. Drop this into the week when the systems-and-tooling fatigue catches up with you.

🛠 2 Tools & Repos

1. VoxCPM2: Tokenizer-Free Multilingual TTS

https://github.com/OpenBMB/VoxCPM

Tokenizer-free TTS system that generates continuous speech directly through a diffusion autoregressive architecture, bypassing discrete audio tokenization entirely. 30 languages, controllable voice cloning, voice design from text descriptions, and 48kHz studio-quality output. Over 13k stars and climbing fast. Pick this up if you’re building anything speech-facing and want to skip the usual codec-and-vocoder tax.

2. caveman

https://github.com/JuliusBrussee/caveman

A Claude Code / Codex / Gemini CLI plugin that instructs the agent to drop articles, filler, and pleasantries in its output while preserving technical content. Benchmarks in the README clock ~75% output token reduction and ~46% input token reduction with no quality loss on downstream evals. 30,800 stars in 11 days since launch. MIT, active maintainers, one-line install across seven agents. Worth the three-minute install if you’ve watched an agent burn context on “Certainly! I’d be happy to help you with that.” Note: Try it and test your luck :)

🎓 1 Pick of the Week

Stanford CS 153: Frontier Systems (Spring 2026)

https://www.youtube.com/playlist?list=PL2aDf5-VARtBwz1kz5FsuSZXOig2U6aJI

Stanford’s new Spring 2026 course on the AI infrastructure stack, taught by Anjney Midha (AMP PBC) and Michael Abbott. Weekly lectures from the people building the frontier: Andreas Blattmann on Black Forest Labs’ image models, Mati Staniszewski on ElevenLabs’ audio stack, and Midha himself on the infrastructure rewrite underneath this moment. Runs through June 3, with upcoming guests including Karpathy, Jensen Huang, Sam Altman, and Satya Nadella. The closest thing to a live running commentary on the stack as it’s being built.

That’s the twenty-fourth Tokenizer. If one of these fifteen resources changes what you’re building this week, forward the edition to whoever should see it next, and come find the long-form work at Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

TL;DR

What caught my attention this week:

• 📄 Papers: Benchmarks fight back against saturation, with new evaluations for video understanding, autonomous agents, robot policies, and the data behind LLM training.

• 🎥 Videos: Inside views from Stanford, OpenAI, Galileo, and Databricks on how agents actually run when you put them in front of real engineers (and non-engineers).

• 📰 Reads: Three sharp takes on coding agent architecture, why your RAG pipeline is overbuilt, and what really decides whether an open model gets adopted.

• 🛠 Tools: Memory and a Rust-native agent loop, two pieces of the open agent stack worth knowing.

• 🎓 Learning: A reproducible walkthrough of using Claude Code as a personal operating system, not as a coding tool.

📄 5 Papers

1. Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

https://arxiv.org/abs/2604.05015 | GitHub

Video benchmarks have saturated to the point where leaderboard scores tell you almost nothing about real model capability. Video-MME-v2 introduces a tri-level hierarchy that escalates from visual aggregation up through temporal reasoning, plus a group-based scoring rule that punishes lucky guesses across linked questions. The team logged about 3,300 human-hours across 12 annotators and 50 reviewers, and one finding stands out: models lean hard on subtitles, and reasoning quality drops sharply when only the pixels are available.

2. DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

https://arxiv.org/abs/2603.26164

If you’ve ever tried to compare data selection, mixture optimization, and reweighting research, you know each lives in its own incompatible codebase. DataFlex sits on top of LLaMA-Factory and gives you one set of trainer abstractions for sample selection, DoReMi/ODM-style mixture tuning, and reweighting, all DeepSpeed ZeRO-3 compatible. It’s an infrastructure contribution rather than a new method, so the value here is reproducibility and composability across data-centric techniques you already wanted to try.

3. Adam’s Law: Textual Frequency Law on Large Language Models

https://arxiv.org/abs/2604.02176 | GitHub

The headline claim sounds tautological (frequent text is better text), but the paper actually proposes a measurement framework with three concrete components: a Textual Frequency Law that estimates sentence-level frequency from open web sources, a distillation step (TFD) that refines that estimate by querying the target model itself, and a curriculum (CTFT) that orders training data from rare to frequent expressions during fine-tuning. Tests cover math reasoning, machine translation, commonsense, and tool calling, which is a wider footprint than the abstract suggests.

4. Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents

https://arxiv.org/abs/2604.06132 | GitHub

Most agent benchmarks only check the final answer, which means a trajectory full of safety violations can still score a clean pass. Claw-Eval grades the path, not just the destination, with 2,159 fine-grained rubric items across 300 tasks, scored over execution traces, audit logs, and environment snapshots. The headline result is that trajectory-opaque grading misses 44% of safety violations and 13% of robustness failures, which is a hard number to ignore if you’re building agent eval today.

5. LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

https://arxiv.org/abs/2603.28301 | GitHub

Vision-Language-Action models look impressive until you reword the instruction, and then performance drops 22 to 52 points across the seven configurations tested (0.6B to 7.5B parameters). LIBERO-Para isolates action phrasing from object references so you can see exactly where the model is reading words instead of meaning. The authors trace 80 to 96 percent of failures to the planning stage, not execution, and introduce PRIDE, a metric that quantifies paraphrase difficulty by semantic and syntactic distance.

🎥 4 Videos

1. World Models You Can Actually Interact With: Inside Moonlake

Interactivity changes what a world model has to learn, and that’s the thread Stanford’s Chris Manning and Fan-yun Sun pull on in this Latent Space conversation about Moonlake. The 67-minute episode gets into how the training signal differs from passive video models, and where this approach sits relative to Marble, Cosmos, and the gaming-data world models that have dominated the last quarter. Worth watching as a counterweight to the “world models = video generation” framing.

2. How OpenAI’s Codex Team Actually Builds with Codex

“For specs, we write like 10 bullets and that’s it,” and “our designers now write more code than eng did 6 months ago” are two of the throwaway lines in this 43-minute sit-down with Codex product lead Alex and developer experience lead Romain on Peter Yang’s channel. The conversation is unusually candid about what shipping without traditional specs and roadmaps looks like inside a team that lives on its own tools. Closest thing to a field report you’ll get on agent-native product development right now.

3. A Non-Engineer Runs Claude Code Across 15 Repos to Answer Every Customer Question

Al Chen is a field engineer at Galileo, an AI observability platform, and he has never held an engineering role. He has also built something most coding teams would be proud of: a Claude Code setup that queries 15 separate internal repositories, stitches in Confluence docs and customer-specific quirks, and delivers answers that previously required pulling an engineer off real work. The walkthrough covers his custom Claude Code commands, the sixteen-line sync script (written entirely by Claude Code) that pulls every repo’s main branch each morning, and the multi-source MCP pattern that lets a single question hit code, docs, and deployment notes in one pass. If you’ve been treating Claude Code as a coding assistant, this is the reframe that turns it into a customer support operating system.

4. From Chaos to Choreography: Multi-Agent Orchestration That Actually Works

Sandipan Bhaumik from Databricks opens with the line of the week: “Adding more agents isn’t adding more features. It’s building a distributed system.” His 26-minute AI Engineer talk covers the silent handoff failures, stale state, and untraceable decisions that show up once you scale from one agent to five, then walks through the orchestrator and choreography patterns Databricks uses in production. If you’re past the proof-of-concept stage, this is the talk you actually need.

📰 3 Curated Reads

1. Components of a Coding Agent

Ahead of AI

Components of A Coding Agent

In this article, I want to cover the overall design of coding agents and agent harnesses: what they are, how they work, and how the different pieces fit together in practice. Readers of my Build a Large Language Model (From Scratch) and Build a Large Reasoning Model (From Scratch…

Read more

21 days ago · 511 likes · 51 comments · Sebastian Raschka, PhD

Sebastian Raschka makes the case that what looks like a smarter model is usually a better harness around the same model. He breaks coding agents into six concrete pieces (live repo context, prompt shape and cache reuse, tool access, context reduction, structured session memory, and bounded subagents) and shows how they fit into a three-layer architecture of model, agent loop, and runtime support. Read this before you blame your model for behavior that’s actually a context engineering problem.

2. Your RAG Pipeline Is Overkill

Decoding AI Magazine

Your RAG Pipeline Is Overkill

We constantly fight a battle against the context window limit. You either compress your data until it loses meaning, or you build a massive infrastructure project just to read a few documents. Today, we look at a third option. We explore a pattern that allows models to read millions of tokens by treating data as an environment rather than an input…

Read more

18 days ago · 43 likes · 3 comments · Paul Iusztin

Paul Iusztin argues that recursive language models (RLMs) make most RAG pipelines unnecessary. The core idea is that the model never receives the giant document directly. Instead, the data lives outside the context as a REPL variable, and the model writes code to explore, filter, and recursively process it through `llm_query()` calls. Iusztin reports RLMs being tested up to 10 million tokens with GPT-5 and Qwen3-Coder, and lays out four scenarios (file parsing, codebase analysis, legal and financial work, research synthesis) where this approach beats stuffing chunks into a vector store.

3. Gemma 4 and What Makes an Open Model Succeed

Interconnects AI

Gemma 4 and what makes an open model succeed

Having written a lot of model release blog posts, there’s something much harder about reviewing open models when they drop relative to closed models, especially in 2026. In recent years, there were so few open models, so when Llama 3 was released most people were still doing research on Llama 2 and super happy to get an update. When…

Read more

22 days ago · 70 likes · Nathan Lambert

Benchmark numbers at release tell you almost nothing about which open models will actually get used, and Nathan Lambert uses the Gemma 4 launch to explain why. His five-factor framework (performance and size, country of origin, license, tooling at release, fine-tunability) maps onto the messy reality that ecosystem maturity often takes 18 months to catch up with a model launch. The note about Google moving to Apache 2.0 and the 30B dense model targeting the enterprise sweet spot is the part to underline.

🛠 2 Tools & Repos

1. mem0

https://github.com/mem0ai/mem0

Memory is the bottleneck for any agent that needs to remember anything across sessions, and mem0 is the fastest-moving open option for solving it. The framework manages user, session, and agent state through a single API, with self-hosted Python and TypeScript packages plus a managed cloud service if you want to skip the infra. The team reports 26% better accuracy than OpenAI Memory and roughly 90% lower token usage versus full-context approaches, which matches what people are seeing in production.

2. goose

https://github.com/aaif-goose/goose

Block built goose as a general-purpose AI agent in Rust, and it’s now part of the Agentic AI Foundation under the Linux Foundation. You get a desktop app, CLI, and API that work with 15+ LLM providers and over 70 MCP extensions, and you can use existing Claude, ChatGPT, or Gemini subscriptions instead of API keys. If you’ve been looking for a serious open alternative to the closed coding agent stack, this is the most active one right now.

🎓 1 Pick of the Week

How to Automate Your Life with Claude Code

Claude Code as a personal operating system, not as a coding tool. Hilary Gridley (former product leader) walks through how she runs her professional work and personal life through it as her primary interface. Her “anti-system system” leans on simple capture (an iPhone back-tap shortcut) and lets the model learn her preferences through observation rather than upfront configuration. The 10x impact framework for deciding what to automate is the part most people will steal, and the whole 51-minute walkthrough is reproducible on your own setup.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

TL;DR

What caught my attention this week:

• 📄 Papers: Streaming video generation hits 16 FPS, speculative sampling gets task-aware, and an autonomous medical AI scientist passes peer review

• 🎥 Videos: Sebastian Raschka maps 54+ LLM architectures, TurboQuant compresses KV cache to 3.5 bits, and two practical Claude Code tutorials

• 📰 Reads: Ethan Mollick on why AI interfaces matter more than models, Cameron Wolfe dissects LLM benchmarks, and Anthropic’s three primitives for context management

• 🛠 Tools: A biomimetic agent memory system with retain/recall/reflect, and an autonomous pentester that proves vulnerabilities with working exploits

• 🎓 Learning: Tsinghua’s multi-agent AI classroom turns any topic into an interactive lesson with AI teachers, students, and a shared whiteboard

📄 5 Papers

1. ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling

https://arxiv.org/abs/2603.25746 | GitHub

Real-time multi-shot video generation that maintains character consistency across scene transitions. ShotStream introduces a causal architecture with dual-cache memory (global context for inter-shot consistency, local context for intra-shot coherence) that enables ~16 FPS streaming, a 25x throughput improvement over bidirectional approaches. From the Kling AI Research team, this is the first system that makes interactive video storytelling feel responsive enough for real-time use.

2. Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models (HyDRA)

https://arxiv.org/abs/2603.25716 | GitHub

Video world models lose track of objects the moment they disappear behind something. HyDRA fixes this with a hybrid memory system that separates archival storage (for static scenes) from working memory (for active, occluded objects). The result: +5.5 PSNR improvement over commercial systems like WorldPlay on a new Dynamic Object Tracking benchmark. Also ships HM-World, the first large-scale video dataset dedicated to hybrid memory evaluation with exit-entry occlusion events.

3. TAPS: Task Aware Proposal Distributions for Speculative Sampling

https://arxiv.org/abs/2603.27027 | GitHub

Speculative decoding’s dirty secret: generic draft models waste tokens because they don’t match the downstream task distribution. TAPS trains task-specific draft models that align with the target model’s behavior on actual workloads, yielding ~26% acceptance length improvements over general-purpose drafters. Practical and immediately applicable if you’re running speculative decoding in production.

4. Towards a Medical AI Scientist

https://arxiv.org/abs/2603.28589

An autonomous research framework that generates clinical hypotheses, designs experiments, executes analyses, and writes papers. The system achieved 91% execution success rate versus GPT-5’s 60%, and one of its generated papers was accepted at ICAIS 2025 (36.8% acceptance rate). This is a concrete step toward AI that does science, not just assists with it.

5. PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference

https://arxiv.org/abs/2603.25730 | GitHub

Training on long videos is expensive. PackForcing shows you don’t need to. By introducing hierarchical KV-cache management with a bounded 4GB memory budget, it achieves 24x temporal extrapolation: models trained on short clips generate coherent long videos. The spatiotemporal compression maintains temporal consistency while keeping inference memory constant regardless of video length.

🎥 4 Videos

1. 54+ LLM Architectures and 7 Attention Variants in One Visual Gallery

A visual map of how LLM architectures evolved from GPT-2 (2019) to Kimi K2.5 (2026), covering 54+ models from 270M to 1T parameters. The 38-minute deep dive compares 7 attention variants: standard MHA, Grouped-Query, Sliding-Window, Multi-Head Latent (DeepSeek’s MLA), Sparse, Gated, and Hybrid. Sebastian Raschka built the companion Architecture Gallery as an open resource, and this walkthrough is how you actually learn to read it.

2. TurboQuant: Compressing KV Cache to 3.5 Bits Per Channel

Random rotation plus scalar quantization can compress KV cache vectors to near-optimal distortion at 3.5 bits per channel. That’s the core of Google’s TurboQuant paper, broken down here by Karoly Zsolnai-Feher (Two Minute Papers). The practical result: cheaper LLM inference through aggressive cache compression without meaningful quality loss, with community review context and reproduction attempts included.

3. Building a Mobile Fitness App with Claude Code and Pencil in 16 Minutes

A three-phase workflow for building mobile apps without coding: define requirements conversationally, generate 8 UI screens with Pencil (pencil.dev), then build the working app with Claude Code. Peter Yang takes it from blank screen to a fitness tracker running on iOS via Expo Go in 16 minutes. Covers workout creation, live session tracking, calendar progress, and the path to App Store deployment.

4. The “Recording Mode” Trick for Privacy-Safe Claude Code Demos

A 2-minute clip with a clever idea: instead of maintaining separate anonymized demo environments, create a Claude Code skill called “recording on” that intercepts and anonymizes personal information in real-time. It tracks consistent mappings (person A stays person A), toggles on and off with zero friction, and works for B2B demos where you need to show live production data without exposing customer names or financial details.

📰 3 Curated Reads

1. Claude Dispatch and the Power of Interfaces

One Useful Thing

Claude Dispatch and the Power of Interfaces

AIs are already far more capable than most people realize. A large part of this so-called capability overhang comes not from the limits of AI (though, of course, they still have many limits), but from how people interact with it. The vast majority of people access AI through chatbots, and usually the free versions with less capable models. A chatbot is …

Read more

24 days ago · 551 likes · 26 comments · Ethan Mollick

Ethan Mollick (Wharton) argues the gap between AI capability and actual user experience is an interface problem, not a model problem. Current chatbot UIs impose cognitive costs that overwhelm productivity gains, particularly for less experienced workers. Three paths forward: specialized professional tools (the coding IDE model), meeting users on familiar platforms (WhatsApp, Slack), and dynamic interfaces where AI generates the right UI on the fly. Includes hands-on demos of Claude Dispatch and Cowork.

2. The Anatomy of an LLM Benchmark

Deep (Learning) Focus

The Anatomy of an LLM Benchmark

Throughout the history of AI research, progress has been measured—and accelerated—by high-quality benchmarks. AI is an empirical field that is driven by discovering interventions that improve performance on key benchmarks. For large language models (LLMs) in particular, creating useful benchmarks is hard due to rapidly advancing …

Read more

25 days ago · 73 likes · Cameron R. Wolfe, Ph.D.

LLM benchmarks break in predictable ways: MMLU has a 6.49% error rate (Virology hits 57%), and correcting those errors moved Llama-3.1-405B from 16th to 1st. Cameron Wolfe (Deep Learning Focus) maps the full lifecycle of how benchmarks are designed, saturate, and get replaced. The standout section covers Item Response Theory, which cuts evaluation costs 140-160x by selecting only the most informative test items.

3. Context Engineering for AI Agents: Memory, Compaction, and Tool Clearing

https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools

Three composable primitives for managing context in long-running agents, each targeting a different type of bloat. Clearing drops re-fetchable tool outputs at zero inference cost (peak context: 173K tokens vs. 335K baseline). Compaction summarizes conversation history (169K peak, lossy). Memory persists knowledge across sessions by letting the agent write its own notes. Isabella He (Anthropic) includes a diagnostic framework: profile your agent’s token breakdown first (in the demo, 96.3% of tokens were stale file-read results), then pick the primitive that matches.

🛠 2 Tools & Repos

1. Hindsight

https://github.com/vectorize-io/hindsight

Agent memory that goes beyond conversation history. Hindsight provides three API primitives (retain, recall, reflect) that let agents learn, retrieve, and synthesize knowledge over time. Recall runs four parallel strategies (semantic, keyword, graph, temporal) with cross-encoder reranking. The reflect API generates new insights from existing memories, not just retrieval. SOTA on LongMemEval, 6.8K GitHub stars, MIT license, works with any LLM provider, and deploys via Docker or as an embedded Python library.

2. Shannon

https://github.com/KeygraphHQ/shannon

An autonomous AI pentester that reads your source code and executes real exploits against the running app. It builds a Code Property Graph to trace data flows from user input to dangerous sinks, then attacks those paths. Every reported vulnerability comes with a working proof-of-concept, not theoretical findings. Handles authentication complexity including 2FA/TOTP flows.

🎓 1 Pick of the Week

OpenMAIC: Tsinghua’s Multi-Agent AI Classroom

https://github.com/THU-MAIC/OpenMAIC

Type “teach me Python in 30 minutes” and OpenMAIC generates a full interactive classroom: slides with narration, quizzes with real-time grading, interactive HTML simulations, and AI agents playing teacher and student roles who lecture, discuss, and draw on a shared whiteboard. Built on LangGraph with a director agent that orchestrates turn-taking, it supports uploading PDFs for document-to-course conversion. 13.6K stars in three weeks, live demo at open.maic.chat, and the LangGraph-based multi-agent orchestration pattern is a reusable blueprint beyond education.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

TL;DR

What caught my attention this week:

• 📄 Papers: Diffusion-based OCR that’s 3x faster, a package manager for AI agent skills, world modeling from Monster Hunter, probability-aware RL clipping, and speculative shortcuts for agentic vision systems

• 🎥 Videos: Stripe’s internal AI coding agents at scale, building your own AI operating system from scratch, DeepSeek’s conditional memory for transformers, and going from Figma design to working code with Claude

• 📰 Reads: Sebastian Raschka’s visual taxonomy of attention mechanisms, Google’s training-free KV-cache compression via polar coordinates, and why on-device AI needs new architectures (not smaller cloud models)

• 🛠 Tools: ByteDance’s filesystem-based context database for AI agents, and Cornell’s free GPU architecture course

• 🎓 Learning: A beautifully maintained Claude Code cheat sheet that doubles as a feature discovery tool

📄 5 Papers

1. MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding

https://arxiv.org/abs/2603.22458 | GitHub

What if OCR didn’t need to generate text one token at a time? This paper replaces autoregressive decoding with block-wise diffusion denoising, reframing document OCR as inverse rendering. The result: up to 3.2x faster throughput with a tunable speed-accuracy tradeoff depending on your pipeline needs. A proposed “Semantic Shuffle” benchmark shows the model genuinely reads visual structure rather than leaning on linguistic shortcuts.

2. SkillNet: Create, Evaluate, and Connect AI Skills

https://arxiv.org/abs/2603.04448 | GitHub

Think npm, but for AI agent skills. SkillNet treats agent capabilities as shareable, composable packages with a unified ontology for creating skills from heterogeneous sources (repos, docs, logs, prompts) and connecting them through dependency graphs. The framework delivers a 40% improvement in average rewards and 30% fewer execution steps across ALFWorld, WebShop, and ScienceWorld. With 200,000+ skills in the repository and a Python toolkit plus Open Access API, this could become critical infrastructure for the tool-using agent ecosystem.

3. WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG

https://arxiv.org/abs/2603.23497 | GitHub

Most world-modeling datasets treat video prediction as frame interpolation. WildWorld takes a different approach: 108 million frames automatically collected from Monster Hunter: Wilds, with per-frame action labels (450+ semantically meaningful actions), character skeletons, world states, camera poses, and depth maps. The key insight is decomposing dynamics into action, state, and pixels separately, giving world models the causal structure they need to simulate rather than interpolate.

4. BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

https://arxiv.org/abs/2603.04918 | GitHub

PPO’s fixed clipping bounds have a quiet failure mode: they disproportionately suppress high-advantage, low-probability actions, which are exactly the exploratory tail strategies you want to preserve in reasoning tasks. BandPO replaces static bounds with dynamic, probability-aware intervals derived from f-divergence constraints, formulated as a convex optimization problem with closed-form solutions. Tested on Qwen2.5 and Llama3 for mathematical reasoning, it consistently outperforms canonical GRPO clipping while preventing entropy collapse. A drop-in replacement for anyone doing GRPO-style training post-DeepSeek-R1.

5. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

https://arxiv.org/abs/2603.23483 | GitHub

Agentic vision systems are slow because every query triggers a full sequential tool-use chain, even when the answer is straightforward. SpecEyes uses a lightweight gating mechanism (based on top-K logit gaps) to predict when the expensive pipeline can be short-circuited. The result: up to 3.35x speedup (averaging 1.4-1.7x across benchmarks) with accuracy maintained or improved by up to 6.7%. The gating requires no labeled routing data, making it deployable without per-task annotation on any production agentic vision system where latency matters.

🎥 4 Videos

1. Building a Full-Stack AI Operating System from Scratch

A three-layer architecture for a personal AI platform: webhook-driven triggers, scheduled workflows (cron-style recurring tasks), and an autonomous agent layer with persistent context and reusable skills. Dave Ebbelaar builds the entire system on FastAPI, Celery, Redis, and Docker. This 28-minute walkthrough gives you the blueprint instead of cloning random agent repos.

2. DeepSeek’s Engram: Adding Conditional Memory to Transformers

DeepSeek’s Engram module adds O(1) lookup memory to transformers by modernizing n-gram embeddings as a new sparsity dimension alongside Mixture-of-Experts. Dr. Karoly Zsolnai-Feher (Two Minute Papers) explains how it outperforms comparable MoE baselines at 27B parameters on both parameter and compute budgets. The surprise: the biggest gains show up not in knowledge retrieval (MMLU +3.4) but in reasoning (BBH +5.0, ARC-Challenge +3.7), because offloading pattern reconstruction to memory frees attention depth for harder tasks.

3. From Figma Design to Working Code with Claude Code and MCP

Figma mockup to working website in 15 minutes, FigJam flowchart to a working game, then exporting code back to Figma as editable components. Felix Lee (designer at ADPList) demonstrates the full design-to-code loop using Claude Code with Figma MCP in this 50-minute session hosted by Peter Yang. The core insight: MCP reads every color, spacing value, and component variant directly from Figma, eliminating the translation loss in design-to-dev handoffs.

4. How Stripe Ships 1,300 AI-Written Pull Requests Per Week

Stripe built internal AI coding agents called “minions” that now ship roughly 1,300 PRs per week. Steve Kaliski (software engineer at Stripe) walks through the architecture: Goose (Block’s open-source agent harness) with cloud dev environments, activated from Slack via emoji reactions. The key takeaway: Stripe’s existing investment in developer tooling (CI, testing, linting) made agent adoption frictionless, because good DX for humans turns out to be good DX for agents too. Non-engineers at Stripe now use minions to ship code.

📰 3 Curated Reads

1. A Visual Guide to Attention Variants in Modern LLMs

Ahead of AI

A Visual Guide to Attention Variants in Modern LLMs

I had originally planned to write about DeepSeek V4. Since it still hasn’t been released, I used the time to work on something that had been on my list for a while, namely, collecting, organizing, and refining the different LLM architectures I have covered over the past few years…

Read more

a month ago · 267 likes · 6 comments · Sebastian Raschka, PhD

Seven attention mechanism families in one visual taxonomy, from classic Multi-Head Attention through GQA, DeepSeek’s Multi-Head Latent Attention, Sliding Window Attention, and hybrid architectures mixing transformers with linear or state-space modules. Sebastian Raschka diagrams how queries, keys, and values interact in each variant, how KV-cache mechanics change, and how memory growth curves differ. The practical payoff: a clear framework for choosing the right attention mechanism based on your model scale and deployment constraints, with Gemma 3 and DeepSeek V2 as case studies.

2. TurboQuant: Redefining AI Efficiency with Extreme Compression

https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

Google Research introduces a training-free, data-agnostic compression algorithm that achieves 3-bit KV-cache quantization without accuracy loss. The trick: PolarQuant converts data vectors from Cartesian to polar coordinates (concentrating angle patterns predictably), then QJL reduces residual errors to single sign bits via random projections. On H100 GPUs, TurboQuant delivers a 6x reduction in KV memory and up to 8x performance gains over 32-bit unquantized baselines. Unlike most quantization work, this targets the KV-cache specifically (the bottleneck that grows with context length) and requires zero calibration data or fine-tuning.

3. The Future of On-Device AI

Artificial Intelligence Made Simple

The Future of On-Device AI

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction…

Read more

a month ago · 55 likes · 1 comment · Devansh

Devansh argues the real bottleneck for on-device AI is memory bandwidth, not compute. Using Liquid AI’s LFM2 (1.2B parameters, runs on Samsung Galaxy S25) as a case study, the piece shows why shrinking data-center models is the wrong approach. LFM2 uses 10 gated short convolutions plus 6 grouped-query attention blocks, cutting peak cache to 192 MB at 32K tokens (versus Llama 3.2 1B’s 524 MB). It matches Qwen3-1.7B on benchmarks despite having 42% fewer parameters and runs at 70 tokens per second on a phone CPU. The thesis: the field needs device-native architectures designed from scratch, not miniaturized cloud models.

🛠 2 Tools & Repos

1. OpenViking: A Context Database for AI Agents

https://github.com/volcengine/OpenViking

ByteDance’s open-source solution for the “stuff everything into the prompt” problem. OpenViking organizes agent context (memories, resources, skills) into a navigable filesystem hierarchy with three-tier demand-based loading, so agents only consume tokens for what they actually need. It combines directory-based navigation with semantic search, auto-compresses conversations into long-term memory, and provides visualization of retrieval trajectories for debugging. 19.1K stars, Apache 2.0, supports major LLM providers via LiteLLM.

2. Cornell Virtual Workshop: GPU Architecture Fundamentals

https://cvw.cac.cornell.edu/gpu-architecture/gpu-characteristics/design

Free, NSF-funded course from Cornell’s Center for Advanced Computing covering why GPUs work the way they do: transistor allocation tradeoffs, memory hierarchies, parallelization design choices, and the practical implications for your code. Modules covering fundamentals through V100 and RTX 5000 deep dives, with exercises. No parallel programming experience assumed. If you call `.cuda()` daily but lack a mental model for what happens underneath, start here.

🎓 1 Pick of the Week

Claude Code Cheat Sheet

https://cc.storyfox.cz/

This single page Claude code cheat sheet covers 100+ commands across 8 color-coded sections: keyboard shortcuts, ~40 slash commands, MCP server configuration, memory and files, workflows, config, skills and agents, and CLI flags. Commands like `/btw` for side questions without derailing context, `/schedule` for cloud-scheduled tasks, and git worktree isolation with sparse checkout are buried in docs but surfaced here at a glance. Auto-detects Mac versus Windows, prints cleanly to A4, works offline, and updates daily as Claude Code evolves.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

Subscribe now

TL;DR

What caught my attention this week:

• 📄 Papers: Low-bit attention training, efficient VLMs, scientific discovery shortcuts, robot planning in 8 tokens, and reasoning models that can’t control their own thoughts

• 🎥 Videos: Sakana AI’s evolved transformers, Turbopuffer’s post-RAG retrieval architecture, Figma’s Claude Code design pipeline, and DeepMind reflects on AlphaGo’s decade of impact

• 📰 Reads: Nathan Lambert on why the open model gap will widen (not close), production RAG done right, and diagnosing failures across 192-GPU training clusters

• 🛠 Tools: ByteDance’s open-source SuperAgent platform and OpenAI’s AI-powered LaTeX editor

• 🎓 Learning: A community-built field manual for getting the most out of Claude Code

📄 5 Papers

1. SageBwd: A Trainable Low-bit Attention

https://arxiv.org/abs/2603.02170 | GitHub

SageAttention already had 3,200+ GitHub stars for speeding up inference. Now SageBwd extends quantized attention to training by quantizing 6 of 7 attention matrix multiplications in the backward pass. The result: up to 1.67x speedup over FlashAttention2 with negligible loss difference (2.561 vs 2.563 at 260K tokens per step). If you’re spending money on attention compute during training, this is the paper to read.

2. Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoder

https://arxiv.org/abs/2603.06569

What if you replaced CLIP/SigLIP vision encoders with one initialized from a plain text LLM? Tencent AI Lab tried it. Their 8B model outperforms Qwen3-VL-8B and InternVL3.5-8B on document understanding, visual knowledge, and video reasoning, hitting 96.2 on DocVQA and 90.5 on ChartQA. Vision encoder architecture matters more than you’d think. Sometimes the answer is just using a language model.

3. MOOSE-Star: Unlocking Tractable Training for Scientific Discovery

https://arxiv.org/abs/2603.03756

Scientific hypothesis discovery with LLMs has an exponential search problem. MOOSE-Star (from MiroMind AI) reduces combinatorial O(N^k) search to roughly logarithmic via hierarchical decomposition, hitting 100% success rate at around 6,000 inference calls where brute-force saturates at 41.3%. Also releases TOMATO-Star, a dataset of 108,717 decomposed papers for benchmarking. The complexity reduction alone makes previously intractable hypothesis spaces searchable.

4. CompACT: A Compact Discrete Tokenizer for Latent World Model Planning

https://arxiv.org/abs/2603.05438

Robot planning typically requires hundreds of visual tokens per observation. CompACT (from POSTECH and KAIST, accepted at CVPR 2026) compresses that down to as few as 8 discrete tokens, making world-model planning 40x faster. Navigation planning drops from 178 seconds to under 6 seconds with competitive accuracy. This is what makes real-time robotic planning actually feasible.

5. Reasoning Models Struggle to Control their Chains of Thought

https://arxiv.org/abs/2603.05706

Can reasoning models actually control their chain of thought? Researchers from NYU, OpenAI, UCL, and UPenn tested 13 models. Claude Sonnet 4.5 achieves only 2.7% CoT controllability (versus 61.9% output controllability). DeepSeek R1 scores 0.1%. The safety implication: if models can’t steer their reasoning strategically, they also can’t easily hide deceptive reasoning from monitors. Matters more for what it implies than what it measures.

🎥 4 Videos

1. Sakana AI’s Open-Ended Evolution of Transformers with Robert Lange

LLMs combined with evolutionary algorithms for open-ended program search. That’s Shinka Evolve from Sakana AI, discussed by Robert Lange on Machine Learning Street Talk. Why “solving the wrong problem” sometimes leads to better architectures. How evolutionary pressure discovers novel transformer variants. What open-endedness means for AI research beyond benchmarks. Covers the gap between optimizing a known objective and discovering objectives worth optimizing.

2. Retrieval After RAG: Hybrid Search, Agents, and Database Design with Turbopuffer’s Simon Eskildsen

What comes after the first wave of RAG implementations? Hybrid search architectures, why vector-only retrieval hits a ceiling, agent-driven retrieval reshaping database design. Simon Eskildsen (Turbopuffer founder) walks through real case studies from Cursor and Notion on Latent Space. If you’ve built a RAG system and hit a quality wall, this is where you go next.

3. How Figma Engineers Sync Designs with Claude Code and Codex

MCP-based tooling that creates a continuous sync between design and code. Figma’s Gui Seiz and Alex Kern show their team’s actual production workflow using Claude Code and Codex. Not a concept demo. The design handoff becomes a two-way pipeline. Worth 40 minutes if your team still does screenshot-to-implementation.

4. 10 Years of AlphaGo: The Turning Point for AI

AlphaGo’s techniques propagated into protein structure prediction, materials science, and chip design. Google DeepMind’s Thore Graepel and Pushmeet Kohli trace the full decade of impact beyond the Go match itself. Covers how one system’s ideas became foundational building blocks across scientific domains.

📰 3 Curated Reads

1. What comes next with open models

Interconnects AI

What comes next with open models

2025 was the year where a lot of companies started to take open models seriously as a path to influence in the extremely valuable AI ecosystem — the adoption of a strategy that was massively accelerated downstream of DeepSeek R1’s breakout success. Most of this is being done as a mission of hope, principle, or generosity…

Read more

a month ago · 59 likes · 12 comments · Nathan Lambert

The open-closed model gap will widen, not close. Nathan Lambert’s reframing: the real opportunity isn’t chasing frontier capability but building small, specialized models that are 10x faster and 100x cheaper. Introduces the “open models as sub-agents” framing where open-weight models handle specialized tasks within larger systems. Changes how you evaluate open models if you’ve been benchmarking them against GPT-5.

2. Production RAG: Learning from Scratch Done Right

Decoding AI Magazine

Why Most RAG Tutorials Fail You

Paul: Today, the stage belongs to Priya, a Senior Software Architect who’s spent years shipping production-scale systems at Publicis Sapient and Tesco…

Read more

a month ago · 46 likes · 5 comments · Priya

Most RAG tutorials optimize for demos, not production. This piece on Paul Iusztin’s Decoding AI (by guest contributor Priya) walks through a 4-phase production RAG system: ingestion, retrieval, generation, serving. Uses Postgres and pgvector with explicit control flow and data lineage. The core insight: a bad chunk doesn’t throw an exception, it just hallucinates an answer three steps later. If your RAG prototype works in notebooks but fails in production, start here.

3. How to Diagnose Failures in Large AI Training Clusters

Artificial Intelligence Made Simple

How to Diagnose Failures in Large AI Training Clusters

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction…

Read more

a month ago · 35 likes · Devansh

AI agents autonomously executing diagnostic runbooks against a unified Prometheus TSDB. Devansh details five case studies across multi-GPU clusters with quantified results: 30% throughput recovery, checkpoint restore penalties reduced from 1.0% to 0.15%. Not theoretical. Includes the actual diagnostic architecture and failure patterns you’d hit at this scale.

🛠 2 Tools & Repos

1. deer-flow (ByteDance)

https://github.com/bytedance/deer-flow

ByteDance’s open-source SuperAgent platform (MIT license, 31K+ stars) got a ground-up v2.0 rewrite that hit #1 on GitHub Trending. Ships as a complete deployable platform, not a framework you wire together. Web UI, Docker-sandboxed execution, persistent cross-session memory, parallel sub-agent spawning, messaging integrations (Telegram, Slack, Feishu). Built on LangGraph. For teams that want a working multi-agent system without assembling one from parts.

2. Prism (OpenAI)

https://prism.openai.com/

OpenAI’s browser-based LaTeX editor with GPT-5.2 integrated inline. Free tier: unlimited projects, compiles, and collaborators (Pro at $7/mo for unlimited AI features). Highlight text, ask the AI to rewrite or formalize, and it compiles in real time. Zotero integration, image-to-LaTeX, voice mode for dictating equations. Best for refining existing papers. Won’t generate structure from a blank page. No Git integration yet, which is the main gap versus Overleaf.

🎓 1 Pick of the Week

claude-code-best-practice

https://github.com/shanraisshan/claude-code-best-practice

How should you actually use Claude Code day-to-day? Official docs don’t answer that. This community-built field manual does (17,600+ stars, actively maintained). 40+ actionable tips across 8 categories, comparative reports against other tools, community workflow implementations. Includes a working `.claude/` directory you can clone. The “billion-dollar questions” section names what the community still hasn’t figured out. If you’re already using Claude Code and want to move from casual to systematic, bookmark this.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

New here? The Tokenizer is my resource-focused newsletter edition where I curate the best AI/ML papers, videos, articles, tools, and learning resources so you don’t have to sift through the noise. Subscribe to Gradient Ascent for the full experience.

TL;DR

What caught my attention this week:

• 📄 Papers: Smarter reasoning graphs, temporal backdoors in tool-using LLMs, spatial reasoning benchmarks, privacy advantages of diffusion language models, and versatile video editing

• 🎥 Videos: Hardware constraints shaping LLM architecture, brand-consistent image generation with Midjourney, a complexity taxonomy for AI agents, and a critical look at vibe coding

• 📰 Reads: Statistical rigor for LLM evaluations, the infrastructure costs of long-context inference, and the current state of open-weight models catching up to closed frontiers

• 🛠 Tools: A 101k-star collection of LLM applications and a versioned API documentation system for coding agents

• 🎓 Learning: Karpathy’s tool that lets AI agents run autonomous ML experiments overnight

📄 5 Papers

1. RouteGoT: Node-Adaptive Routing for Cost-Efficient Graph of Thoughts Reasoning

https://arxiv.org/abs/2603.05818

Graph of Thoughts is powerful but expensive, treating every reasoning node equally regardless of difficulty. RouteGoT fixes this by adaptively routing compute across graph nodes, skipping the heavy lifting where it isn’t needed. The result: 8.1 percentage points more accurate than AGoT while using 79.1% fewer output tokens.

2. Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs

https://arxiv.org/abs/2603.03371

Here’s a security scenario worth losing sleep over: backdoors implanted in tool-using LLMs that only activate under specific temporal conditions. The model maintains state-of-the-art performance on benign tasks and evades standard safety evaluations, until a particular time trigger flips the switch. A sobering look at the gap between current evaluation practices and actual deployment safety.

3. SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in LLMs

https://arxiv.org/abs/2603.03002

Do LLMs actually understand spatial relationships, or are they just pattern-matching on word co-occurrences? SpatialText isolates spatial reasoning from visual shortcuts using pure text, and the results aren’t flattering. Current models lean heavily on linguistic heuristics rather than building coherent spatial representations, which matters for anyone building systems that need to reason about physical space.

4. Characterizing Memorization in Diffusion Language Models

https://arxiv.org/abs/2603.02333

Diffusion-based language models turn out to have a meaningful privacy advantage over autoregressive ones. This paper shows they exhibit substantially lower memorization-based leakage of personally identifiable information. If you’re building generative systems where training data sensitivity matters (medical, legal, financial), this distinction between generation architectures is worth understanding.

5. Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

https://arxiv.org/abs/2603.02175 | GitHub

Video editing that combines instruction-based and reference-based approaches in one pipeline. Kiwi-Edit constructs high-fidelity training data using synthetic reference scaffolds, sidestepping the usual bottleneck of paired video editing datasets. The result is a system that handles both “make the sky purple” style instructions and “make it look like this reference” editing in a single model.

🎥 4 Videos

1. How Hardware Constraints Are Shaping Modern LLM Architecture

Julia Turc explores how physical hardware realities (memory bandwidth, compute density, interconnect speeds) are actively driving architectural decisions in modern LLMs. If you’ve wondered why certain design choices keep showing up across different labs, the answer often starts with what the silicon can actually do efficiently.

2. Mastering Midjourney: Consistent Brand Imagery Without Complex Prompts

On How I AI, AI creative director Jamey Gannon walks through her workflow for generating images that maintain consistent brand identity across multiple Midjourney outputs using style references, personalization codes, and mood boards. Useful for anyone who’s gotten great individual images from AI tools but struggled to make a cohesive visual set for a brand, product, or campaign.

3. The 5 Levels of AI Agent Complexity

Dave Ebbelaar breaks down AI agents into five distinct complexity levels, from simple single-tool agents to sophisticated multi-agent orchestration systems. Helpful framing for teams trying to scope what kind of agent they actually need (often simpler than they think) and understanding the engineering effort each level demands.

4. Vibe Coding is a Slot Machine

Machine Learning Street Talk sits down with Jeremy Howard (fast.ai) to examine whether AI coding assistants are genuinely improving developer productivity or creating a false sense of progress. Howard’s central argument: if you outsource all your thinking to computers, you stop building the competence that makes you effective. The kind of critical examination teams need before making investment decisions around AI-assisted development tooling.

📰 3 Curated Reads

1. Applying Statistics to LLM Evaluations

Deep (Learning) Focus

Applying Statistics to LLM Evaluations

Research on large language models (LLMs) is empirically driven. For this reason, model evaluations play a pivotal role in the field’s progress. We improve models by making changes, evaluating them, and iterating. Despite their foundational role, however, evaluations are usually handled in a naive manner. In most cases, we just test a mod…

Read more

2 months ago · 66 likes · 2 comments · Cameron R. Wolfe, Ph.D.

Cameron R. Wolfe from Deep (Learning) Focus tackles a problem most benchmarking papers quietly ignore: the statistical validity of LLM evaluation results. Key finding worth internalizing: clustered standard errors can increase reported uncertainty by 3x, and the Central Limit Theorem becomes unreliable with small sample sizes. If you’ve ever looked at a leaderboard and wondered “is this difference actually meaningful?”, this article gives you the tools to answer that.

2. How Long Context Inference Is Rewriting the Future of Transformers

Artificial Intelligence Made Simple

How Long Context Inference Is Rewriting the Future of Transformers

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction…

Read more

2 months ago · 56 likes · 2 comments · Devansh

AI Made Simple quantifies what long-context windows actually cost in production. A 70B-parameter model serving 59 concurrent users at 4K context drops to just 1 user at 128K context. The article covers the engineering responses, including Multi-Head Latent Attention (MLA) achieving 93.3% cache reduction. Practical reading for anyone deploying models where context length isn’t just a spec sheet number but a capacity planning constraint.

3. A Dream of Spring for Open-Weight LLMs

Ahead of AI

A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026

If you have struggled a bit to keep up with open-weight model releases this month, this article should catch you up on the main themes…

Read more

2 months ago · 183 likes · 9 comments · Sebastian Raschka, PhD

Sebastian Raschka surveys the current open-weight landscape across 10 models and finds the gap to closed frontiers narrowing fast. GLM-5 now benchmarks on par with GPT-5.2, Gemini Pro 3, and Claude 4.6 Opus. On the inference side, Step 3.5 Flash hits 100 tokens/sec compared to DeepSeek’s 33. A well-structured overview for tracking which open models are actually competitive and where the remaining gaps lie.

🛠 2 Tools & Repos

1. Awesome LLM Apps

https://github.com/Shubhamsaboo/awesome-llm-apps

At 101k stars, this collection of LLM application examples covers RAG implementations, AI agents (single and multi-agent teams), MCP integration patterns, voice AI, and fine-tuning guides. More useful as a reference architecture library than a tutorial. When you’re building something new, check here first to see how others have solved similar problems.

2. Context Hub

https://github.com/andrewyng/context-hub

From Andrew Ng’s team, Context Hub gives coding agents access to curated, versioned API documentation instead of letting them hallucinate library APIs. Features search and fetch, language-specific variants, persistent annotations, and feedback loops so the documentation improves over time. At 3.5k stars, it’s gaining traction with teams building custom coding agents that need reliable API knowledge.

🎓 1 Pick of the Week

Autoresearch

https://github.com/karpathy/autoresearch

Andrej Karpathy’s latest side project is a Python tool built around three core files (prepare.py, train.py, program.md) that lets an AI agent run autonomous ML experiments on a single GPU. The agent proposes hypotheses, writes training code, runs experiments with a 5-minute budget each, and iterates based on results. Leave it running overnight and wake up to a stack of completed experiments. At 23.6k stars within days of release, it’s clearly struck a nerve. Sparks of recursive self-improvement, indeed.

Thanks for reading The Tokenizer! If you found this valuable, please share it with your colleagues and consider subscribing to Gradient Ascent for more AI insights.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

We’ll cover AI systems that use RAG and traditional ML design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Through live discussions, guided exercises, and design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

Use code FLASH40 for 40% off:

Learn AI System Design

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

TL;DR

What caught my attention this week:

• 📄 Papers: Task handoff is not the same as authority transfer, video models tested on physics not aesthetics, memory that evolves without overfitting, gated reasoning that knows when to stop, and feature-level coding benchmarks exposing real agent limitations

• 🎥 Videos: Jeff Dean on AI’s Pareto frontier, comprehensive 2026 state of AI breakdown, building custom dev tools instead of buying SaaS, and practical context engineering for agents

• 📰 Reads: Rubric-based rewards for subjective domains, recursive language models that call themselves like functions, and why taste matters for generative AI

• 🛠 Tools: Reasoning implementations from scratch, comprehensive LLM learning roadmap

• 🎓 Learning: Karpathy’s microGPT strips GPT to its mathematical essence in pure Python

📄 5 Papers

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development

https://arxiv.org/abs/2602.10975 | GitHub

Claude 4.5 Opus achieves 74.4% on SWE-bench but drops to 11.0% on FeatureBench. The difference? SWE-bench tests bug fixes within single pull requests, while FeatureBench evaluates end-to-end feature development spanning multiple commits and PRs across development timelines. Using a test-driven method that traces from unit tests along dependency graphs, the benchmark automatically derives 200 feature-level tasks from 24 repositories while ensuring other features remain functional after separation. This exposes the gap between fixing localized issues and actually building new capabilities.

Intelligent AI Delegation

https://arxiv.org/abs/2602.11865

Existing delegation methods run on heuristics that collapse when environments change or sub-agents fail. Intelligent AI Delegation reframes the problem: passing a task is not the same as transferring authority, responsibility, and accountability, and conflating the two is where multi-agent systems break down. The framework draws from principal-agent theory in economics, assumes zero trust at every delegation boundary, and applies to both human and AI delegators across complex agent networks. Misalignment, reward gaming, and deceptive behavior all compound as agents delegate to other agents. The paper is trying to define the protocol layer the agentic web needs before it can safely scale.

RISE-Video: Can Video Generators Decode Implicit World Rules?

https://arxiv.org/abs/2602.05986 | GitHub

Video models produce visually impressive outputs, but can they reason about physics? RISE-Video shifts evaluation from aesthetics to cognitive understanding with 467 human-annotated samples spanning eight categories. The benchmark tests whether models grasp implicit constraints like spatial dynamics, temporal consistency, physical rationality, and causality. Testing 11 state-of-the-art models revealed pervasive failures when simulating complex scenarios under implicit rules, exposing the gap between visual fidelity and genuine world understanding.

UMEM: Unified Memory Extraction and Management Framework for Generalizable Memory

https://arxiv.org/abs/2602.10652 | GitHub

Memory-enabled agents typically optimize management while treating extraction as static, accumulating instance-specific noise rather than robust insights. UMEM jointly optimizes both through Semantic Neighborhood Modeling, evaluating memory utility across clusters of semantically related queries rather than individual instances. Trained with neighborhood-level marginal utility rewards via GRPO, the approach achieves up to 10.67% improvement on multi-turn tasks while maintaining monotonic growth during continuous evolution. Agents that actually learn from experience rather than just retrieve logs.

When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning

https://arxiv.org/abs/2602.10560

Long-context reasoning faces critical issues: memory explodes from indiscriminate updates on evidence-free chunks, and loops continue unnecessarily after gathering sufficient evidence. GRU-Mem introduces update and exit gates controlled by text. Memory updates only when the update gate opens, and the loop terminates immediately when the exit gate opens. Trained with end-to-end RL using separate rewards for correct updating and exiting behaviors, GRU-Mem outperforms vanilla MemAgent with up to 4x inference speed acceleration.

🎥 4 Videos

Owning the AI Pareto Frontier — Jeff Dean

Jeff Dean unpacks the Pareto frontier in AI scaling, balancing compute, energy efficiency and model performance. Google’s Chief AI Scientist discusses the unification of their AI teams and why distillation is becoming the engine behind efficient models. Essential viewing for understanding the tradeoffs between model capability and practical deployment constraints.

State of AI in 2026: LLMs, Coding, Scaling Laws & Agents

Nathan Lambert and Sebastian Raschka join Lex Fridman to break down where we actually stand. They cover the reality of coding agents (echoing FeatureBench findings), scaling laws beyond just “more compute,” and practical challenges of building reasoning models. Two of the best technical communicators in AI deliver a comprehensive status report on 2026’s landscape.

DIY Dev Tools: The Shift to “Build” Over “Buy”

CJ Hess demonstrates “Flowy,” a custom tool he built to visualize coding plans. Instead of relying on static Markdown or finicky diagrams, he created a system where Claude generates JSON that renders into interactive flowcharts and UI mockups. When AI can write the code, building bespoke internal tools optimized for your specific workflow is often faster and better than buying generic SaaS.

Effective Context Engineering for AI Agents

Dave Ebbelaar explains why agents fail not from bad instructions but from poor context management. In short, context engineering is king. He offers practical strategies for structuring workflows and managing the context window as a limited resource rather than a magic bucket.

📰 3 Curated Reads

Rubric-Based Rewards for RL

Deep (Learning) Focus

Rubric-Based Rewards for RL

Many of the recent capability gains in large language models (LLMs) have been a product of advancements in reinforcement learning (RL). In particular, RL with verifiable rewards (RLVR) has drastically improved LLM capabilities by using rules-based, deterministic correctness checks (e.g., passing the test cases for a coding problem…

Read more

2 months ago · 64 likes · 5 comments

Cameron Wolfe explains that while RL has excelled in domains with clear answers like math and code, Rubric-Based RL uses LLMs as judges with strict rubrics to provide reward signals in subjective domains. This bridges the gap that could allow training reasoning models for writing, creativity, and strategy rather than just problems with verifiable solutions.

Recursive Language Models: To the Rescue

WHERE MACHINES THINK

Recursive Language Models: Can a simple method potentially unlock new regime of inference-time scaling?

IN THE FALL OF 2024, there were concerns that neural scaling laws were saturating. Throwing more compute and data to train ever-bigger large language models (LLMs) was showing diminishing returns. And then, OpenAI released its o1 series of “reasoning” models, unlocking an entirely new way to improve the performance of LLMs. Ope…

Read more

2 months ago · 17 likes · 1 comment · Anil Ananthaswamy

What if an LLM could call itself like a function? Recursive Language Models decompose complex prompts and recursively generate their own context. This architectural shift treats models less like text predictors and more like computer programs with call stacks, handling complexity through recursion rather than scale.

Taste for Makers

https://paulgraham.com/taste.html

Paul Graham breaks down the universal principles of good design across disciplines like math, coding, and architecture. For anyone building AI products today, treating “quality” as an objective, engineerable standard rather than a vague feeling is a massive competitive advantage.

🛠 2 Tools & Repos

Reasoning from Scratch (Chapter 7 Code)

https://github.com/rasbt/reasoning-from-scratch/blob/main/ch07/01_main-chapter-code/ch07_main.ipynb

Sebastian Raschka’s notebook implements Reinforcement Learning with Group Relative Policy Optimization (GRPO) from scratch. If you want to understand how models like DeepSeek-R1 work under the hood, this walks through the actual mechanics rather than hiding behind abstractions. Essential for anyone implementing reasoning capabilities.

LLM Course Roadmap

https://github.com/mlabonne/llm-course

Maxime Labonne’s repository remains the gold standard for self-learning. This curated roadmap takes you from fundamentals to fine-tuning your own models, recently updated with sections on agentic workflows and evaluation. The curriculum balances theoretical explanations with practical notebooks and hands-on projects mirroring real-world applications.

🎓 1 Pick of the Week

microGPT

https://karpathy.github.io/2026/02/12/microgpt/

Andrej Karpathy stripped GPT to its absolute core: 243 lines of pure Python with zero dependencies. No PyTorch, no NumPy, no frameworks. Just the full algorithmic content needed for training and inference: dataset handling, tokenizer, autograd engine, GPT-2-like architecture, Adam optimizer, training loop, and inference loop. The code fits perfectly across three columns and represents a decade-long obsession to simplify LLMs to bare essentials. Run it from a single file, understand every mathematical operation, see exactly how attention works without abstraction layers hiding the mechanics. For anyone who wants to truly understand transformers rather than just use them, this is the definitive resource.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

We’ll cover AI systems that use RAG and traditional ML design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Through live discussions, guided exercises, and team-based design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

Use code FLASH40 for 40% off:

Register for the Workshop

Questions? Drop a comment or DM me.

TL;DR

What caught my attention this week:

📄 Papers: Code-as-image achieving 8x compression, speculative decoding beating EAGLE-3 by 2.5x, video models learning from YouTube demonstrations, and self-optimizing RL systems

🎥 Videos: Systematic Claude Code workflows from Anthropic users, Stanford on agents and RAG patterns, visual MoE explanations, and vector indexing methods

📰 Reads: Why creativity can’t be interpolated, 10x cheaper tokens through prompt caching, and 12x faster MoE training with Unsloth

🛠 Tools: Language extraction for multilingual text and comprehensive LLM evaluation frameworks

🎓 Learning: Google’s viral Paper Banana for interactive research paper exploration

📄 5 Papers

Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition

https://arxiv.org/abs/2602.08439 | GitHub

Most video benchmarks test static knowledge rather than whether models can actually learn from demonstrations. Demo-ICL tackles this directly with a benchmark built from 1200 instructional YouTube videos. The system provides models with text or video demonstrations, then asks them to answer questions about target videos by applying what they learned from the examples. The two-stage training approach combines video-supervised fine-tuning with information-assisted preference optimization, improving how models extract and apply procedural knowledge from demonstrations.

DFlash: Block Diffusion for Flash Speculative Decoding

https://arxiv.org/abs/2602.06036 | GitHub

Autoregressive models decode sequentially, creating a latency bottleneck that speculative decoding helps address. DFlash uses a lightweight block diffusion model that generates draft tokens in parallel rather than one at a time. By conditioning the draft model on context features from the target LLM, it maintains high acceptance rates while generating entire blocks simultaneously. The system achieves over 6x lossless acceleration across various tasks, delivering up to 2.5x higher speedup than EAGLE-3.

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding

https://arxiv.org/abs/2602.01785 | GitHub

Text-based code processing creates a linear scaling problem as codebases grow. CodeOCR explores representing code as rendered images instead, taking advantage of visual compression capabilities that text lacks. The research shows vision-language models can understand code images at up to 8x compression while maintaining performance. Tasks like clone detection actually improve slightly under compression, and syntax highlighting boosts code completion performance at 4x compression. The approach suggests image-based code representation could fundamentally change how we handle large-scale code understanding.

RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

https://arxiv.org/abs/2602.02488 | GitHub

Traditional reinforcement learning keeps environments, policies, and reward models separate. RLAnything makes all three components adapt together through closed-loop optimization. The policy trains on combined step-wise and outcome signals, while the reward model jointly optimizes through consistency feedback. Environment tasks automatically adjust difficulty based on critic feedback from both the policy and reward model. The system delivers substantial gains: 9.1% improvement on OSWorld for Qwen3-VL-8B-Thinking, and 18.7% and 11.9% boosts on AlfWorld and LiveBench for Qwen2.5-7B-Instruct.

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

https://arxiv.org/abs/2602.07026

Despite progress in multimodal contrastive learning, a persistent geometric anomaly remains: embeddings of different modalities expressing identical semantics occupy systematically offset regions. This modality gap creates alignment challenges that existing approaches handle inefficiently. The research introduces Fixed-frame Modality Gap Theory, which decomposes the gap into stable biases and anisotropic residuals. ReAlign aligns text representations into image distribution space using statistics from unpaired data through anchor, trace, and centroid alignment. Building on this, ReVision enables MLLMs to learn visual representation distribution from unpaired text before visual instruction tuning, reducing dependence on expensive image-text pairs.

🎥 4 Videos

How to Make Claude Code Better Every Time You Use It (Full System)

Stop fighting with Claude Code and start building systematic workflows that compound over time. This comprehensive guide covers setting up persistent context files that Claude references across all sessions, structuring your codebase so Claude understands project architecture from the start, and creating reusable prompt patterns that extract better reasoning. You’ll learn how to build a project-specific knowledge base, manage multiple Claude sessions working on different parts of your code simultaneously without conflicts, and structure requests to get maintainable output instead of throwaway code. The system turns Claude Code from a one-off tool into a genuine development partner.

MoE, Visually Explained

Mixture-of-Experts architectures power many frontier models but remain conceptually opaque. This visual breakdown makes the core mechanics tangible: how router networks decide which experts process each token, why sparse activation improves efficiency without sacrificing performance, and what trade-offs exist between expert count and model capacity. The explanations focus on intuition over equations, making MoE accessible without dumbing down the actual complexity involved.

What is Indexing? Indexing Methods for Vector Retrieval

Vector databases require efficient retrieval mechanisms as collections scale. This overview covers indexing approaches from flat search through hierarchical navigable small worlds (HNSW) and inverted file indexes (IVF). You’ll understand when approximate nearest neighbor search makes sense versus exact matching, how different index types trade off speed against accuracy, and which methods work best for specific retrieval patterns.

Agents, Prompts, and RAG

Stanford’s practical breakdown of building reliable agents using retrieval-augmented generation cuts through the hype. The discussion examines when to use agents versus simpler prompt chains, how to structure RAG systems that agents can query effectively, and debugging approaches when agentic systems fail. The focus stays on production considerations: handling edge cases, managing token budgets, and building systems that degrade gracefully rather than failing catastrophically. Essential viewing for anyone moving beyond toy agent demos to production deployments.

📰 3 Curated Reads

Why Creativity Cannot Be Interpolated

https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/

This exploration challenges the assumption that creative breakthroughs emerge from incremental improvements. The argument centers on fundamental limitations of interpolation-based learning: models trained to predict the next token in existing distributions struggle to generate genuinely novel ideas that exist outside their training manifold. The piece examines what this means for AI’s creative capabilities and why certain types of innovation may require fundamentally different approaches than current architectures support.

Prompt caching: 10x cheaper LLM tokens, but how?

https://ngrok.com/blog/prompt-caching/

Prompt caching stores processed prefixes so repeated context doesn’t get recomputed on every request. This technical breakdown explains how caching works under the hood: which parts of prompts get cached, how providers handle cache invalidation, and what billing implications exist. The post provides concrete strategies for structuring prompts to maximize cache hits, quantifies actual cost savings across different use patterns, and identifies scenarios where caching delivers minimal benefit.

Fine-tune MoE Models 12x Faster with Unsloth

https://unsloth.ai/docs/new/faster-moe

Training Mixture-of-Experts models typically requires massive compute due to their architecture. Unsloth achieves up to 12x speedups over Transformers v4 (and roughly 2x over the already-optimized Transformers v5) through custom Triton grouped-GEMM kernels and a Split LoRA approach that also cuts VRAM usage by over 35%. The documentation covers implementation details: how to integrate Unsloth into existing training pipelines, which MoE models are supported (Qwen3, DeepSeek R1/V3, GLM), and how the optimizations maintain full accuracy with zero approximation. Particularly valuable for teams fine-tuning large MoE models on consumer hardware.

🛠 2 Tools & Repos

Langextract

https://github.com/google/langextract

Extracting structured information from unstructured text typically requires custom pipelines for every new domain. Langextract is a Gemini-powered Python library that handles this through LLM-based extraction with precise source grounding, mapping every extracted entity back to its exact location in the source text. Define your extraction schema through a few examples and the tool adapts without fine-tuning, handling long documents through optimized chunking and parallel processing. Particularly useful for domains like clinical notes, radiology reports, and research literature where traceability between extracted data and source material is critical.

Deepeval

https://github.com/confident-ai/deepeval

LLM evaluation needs systematic frameworks rather than ad-hoc testing. Deepeval provides metrics for measuring answer relevancy, factual consistency, and hallucination rates across different model outputs. The framework supports both rule-based and model-based evaluation, enables A/B testing across prompt variations, and tracks performance degradation over time. Particularly useful for teams building production LLM applications that need reliable quality metrics.

🎓 1 Pick of the Week

Paper Banana

https://dwzhu-pku.github.io/PaperBanana/

Google’s Paper Banana went viral for good reason: it fundamentally changes how researchers produce academic illustrations. Instead of spending hours manually crafting methodology diagrams and statistical plots, Paper Banana orchestrates five specialized AI agents (Retriever, Planner, Stylist, Visualizer, and Critic) to generate publication-ready visuals from paper text. The system handles everything from architecture diagrams to data visualizations, with the Critic agent running multiple refinement rounds to catch factual errors and visual glitches. In blind human evaluation, its outputs achieved a 72.7% win rate against baseline AI models. For researchers drowning in illustration work or trying to produce consistent, high-quality figures across papers, this approach delivers production value that manual tools struggle to match. The viral response reflects how badly the research community needed a better way to handle the illustration burden.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

I’m teaching ML & Generative AI System Design on Feb 28th / March 1st with Packt.

Register Today

Gradient Ascent Special: Use code FLASH40 for 40% off

We’ll cover the core system design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Through live discussions, guided exercises, and team-based design sprints, you’ll practice solving system-level AI problems and walk away with frameworks you can apply immediately at work.

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

TL;DR

What caught my attention this week:

📄 Papers: Vision encoders with causal reasoning for document understanding, open-source world simulators rivaling closed systems, plus advances in mathematical reasoning and multimodal scientific models

🎥 Videos: Pydantic fundamentals for ML engineers, building agent frameworks from scratch, production AI coding workflows, and understanding diffusion versus flow matching

📰 Reads: Reinforcement learning for continual LLM adaptation, preparing for ML interviews beyond just attention mechanisms, and DeepSeek’s latest architectural innovations

🛠 Tools: Curated resources for agentic reasoning research and comprehensive AI tool directories

🎓 Learning: Stanford’s updated large language model course covering recent advances

📄 5 Papers

DeepSeek-OCR 2: Visual Causal Flow

https://arxiv.org/abs/2601.20552 | GitHub

Instead of processing images in rigid raster-scan order, DeepSeek-OCR 2 introduces DeepEncoder V2 that mimics how humans actually read documents. The encoder uses causal attention to dynamically reorder visual tokens based on semantic content, determining whether to scan titles first, process tables column-by-column, or navigate multi-column layouts intelligently. By replacing CLIP with Qwen2-0.5B and implementing learnable queries with causal flow, the 3B parameter model achieves 91.09% on OmniDocBench while maintaining 256-1120 token efficiency. Reading order edit distance dropped from 0.085 to 0.057, proving the system genuinely understands logical document structure rather than just memorizing patterns.

Advancing Open-source World Models

https://arxiv.org/abs/2601.20540 | GitHub

Ant Group’s LingBot-World delivers minute-long consistent video generation at 16 FPS with under 1-second interaction latency, positioning open-source world models competitively against closed systems. The system maintains high-fidelity dynamics across photorealistic, scientific, and stylized environments through a multi-stage training pipeline combining web videos with Unreal Engine synthetic data. Users control camera perspectives and environmental conditions in real-time while the model preserves spatial consistency across 961 frames. The hybrid data engine with hierarchical captioning separates motion control from static scene generation, addressing the training data bottleneck that typically limits world model development.

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

https://arxiv.org/abs/2601.20614 | GitHub

Group Relative Policy Optimization suffers from an implicit imbalance where harder questions receive smaller policy updates, limiting capability development where it matters most. MathForge addresses this through Difficulty-Aware GRPO, which balances group advantage estimation by question difficulty and prioritizes harder problems through difficulty-aware weighting. The framework’s Multi-Aspect Question Reformulation strategy systematically increases question difficulty across multiple dimensions while maintaining gold answers, creating training data that pushes model boundaries. The synergistic loop of MQR expanding the data frontier and DGPO effectively learning from augmented data produces significant improvements across mathematical reasoning benchmarks.

Innovator-VL: A Multimodal Large Language Model for Scientific Discovery

https://arxiv.org/abs/2601.19325 | GitHub

Scientific multimodal models typically require massive domain-specific pretraining, but Innovator-VL demonstrates strong performance using fewer than five million curated samples without large-scale pretraining. The fully transparent training pipeline covers data collection, cleaning, preprocessing, supervised fine-tuning, and reinforcement learning with detailed optimization recipes, enabling systematic community extension. The model maintains competitive performance on general vision benchmarks while excelling at scientific tasks, indicating that scientific alignment integrates into unified models without compromising general-purpose capabilities. Principled data selection proves more effective than indiscriminate scaling for scientific reasoning.

Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision

https://arxiv.org/abs/2601.19798 | GitHub

Vision-language models exhibit a text-dominant optimization bias by treating visual signals as passive inputs rather than supervisory targets. Youtu-VL shifts to the Vision-Language Unified Autoregressive Supervision paradigm, integrating visual tokens directly into the prediction stream and applying unified autoregressive supervision to both visual details and linguistic content. This “vision-as-target” approach fundamentally changes optimization from treating vision as conditional input to making it a prediction objective. The framework extends to vision-centric tasks without requiring task-specific architectural additions, establishing foundations for comprehensive generalist visual agents.

🎥 4 Videos

Pydantic Crash Course

Dave Ebbelaar walks through Pydantic’s data validation and settings management capabilities essential for ML engineers working with LLMs. The tutorial covers defining models with type hints, validation logic, and configuration management patterns that ensure data integrity in production AI systems. Understanding Pydantic proves critical when building structured outputs from language models or managing complex application configurations where type safety prevents runtime errors.

Building Mini ClawdBot from Scratch

Vizuara demonstrates constructing an agent like Moltbot (ClawdBot, or whatever it's called now) without relying on existing libraries, revealing the actual mechanics behind agent architectures. By building from first principles, you understand tool integration, state management, and decision-making loops that frameworks abstract away. This approach proves valuable when debugging production agent systems or architecting custom solutions that don't fit standard framework patterns.

The Senior Engineer’s Guide to AI Coding

This How I AI episode examines Claude Code workflows and architectural decisions that separate effective AI-assisted development from mere prompt engineering. It addresses code review practices, testing strategies, and integration patterns when collaborating with AI coding assistants.

Flow Matching vs Diffusion Side By Side

Letitia compares flow matching and diffusion approaches for generative modeling, clarifying when each technique provides advantages. The visual comparison helps understand why flow matching sometimes offers training efficiency benefits over traditional diffusion while maintaining generation quality. Grasping these trade-offs matters when selecting architectures for specific generative modeling tasks.

📰 3 Curated Reads

Continual Learning with RL for LLMs

Deep (Learning) Focus

Continual Learning with RL for LLMs

Continual learning, which refers to the ability of an AI model to learn from new tasks and data over time, has become a popular topic in the discussion of Artificial General Intelligence (AGI). Put simply, general intelligence should be adaptable, which has led some to believe that continual learning abilities are a prerequisite f…

Read more

3 months ago · 93 likes · 3 comments · Cameron R. Wolfe, Ph.D.

Cameron Wolfe explores why Reinforcement Learning (RL) is naturally more robust than Supervised Fine-Tuning (SFT) for continual learning in LLMs. While traditional methods like replay buffers and regularization remain relevant, recent studies suggest that RL’s on-policy nature minimizes the distributional shifts that cause catastrophic forgetting.

Is Attention All You Need for ML Interviews?

https://medium.com/@maxwbuckley/is-attention-all-you-need-to-prepare-for-ml-interviews-830742f6d2ba

Max Buckley shares some quick tips to master the transformer architecture and its implementation ahead of ML interviews. Worth checking out to know the details of the bedrock of modern AI in case you’re preparing for interviews.

DeepSeek Drops Yet Another

WHERE MACHINES THINK

DeepSeek Drops Yet Another Architectural Innovation, Opening A New Front for Scaling Up LLMs

Some large language models can memorize entire books and regurgitate them almost verbatim. In one study, researchers successfully prompted an LLM to spit out, for example, a nearly complete Harry Potter and the Sorcerer’s Stone. While this raises huge concerns about large-scale copyright violations, it's also true that we simultaneously want these model…

Read more

3 months ago · 58 likes · 8 comments · Anil Ananthaswamy

Anil Ananthaswamy analyzes DeepSeek’s pattern of releasing architectural innovations that challenge conventional wisdom about model scaling and design. The examination covers their mixture-of-experts implementations, efficiency improvements, and open-source strategy that accelerates community progress. Tracking DeepSeek’s releases provides insights into architectural directions gaining traction in production systems.

🛠 2 Tools & Repos

Awesome Agentic Reasoning

https://github.com/weitianxin/Awesome-Agentic-Reasoning

This curated collection organizes research papers, codebases, and benchmarks focused on agentic reasoning capabilities in AI systems. Instead of accumulating every tangentially related work, the repository maintains editorial standards around core reasoning techniques like planning, reflection, and tool use. The structured organization helps researchers quickly locate relevant work when investigating specific reasoning approaches or comparing methodologies across different agent architectures.

GoodAI List

https://goodailist.com/repos

Chip Huyen curates AI tools and repositories with clear judgment about practical utility versus hype. The directory covers libraries, frameworks, and resources across the ML stack with emphasis on production readiness and actual adoption. Unlike comprehensive but overwhelming awesome lists, this maintains focus on tools demonstrating real-world value in AI development workflows.

🎓 1 Pick of the Week

Stanford’s LLM Course

https://youtube.com/playlist?list=PLoROMvodv4rObv1FMizXqumgVVdzX4_05&si=bqJCbjmvpCx-1_51

Stanford’s curated playlist brings together lectures from multiple courses covering large language models, including CS224N (NLP with Deep Learning), CS25 (Transformers United), and CME 295 (Transformers & Large Language Models). The collection spans foundational concepts in transformer architectures and training methodologies alongside cutting-edge developments in reasoning models, multimodal systems, and efficient inference techniques.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: Actionable mechanistic interpretability, agent efficiency frameworks, 3-4x reasoning compression, cross-embodiment robotics, unified video generation

• 🎥 Videos: Agent architecture patterns, live coding marathon, production evals, MoE routing mechanics

• 📰 Reads: Anthropic’s evaluation resilience, AI emotional frameworks, semantic search without embeddings

• 🛠 Tools: Production RAG and distributed training infrastructure

• 🎓 Learning: MIT’s command-line fundamentals

I’m running an ML & Generative AI System Design workshop with Packt.

We’ll cover the core design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Use code FLASH40 for 40% off:

Join the Workshop!

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

📄 5 Papers

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

https://arxiv.org/abs/2601.14004 | GitHub

Mechanistic interpretability research has spent years documenting how models work internally. This survey introduces the first systematic framework for actually using those insights to improve models. The “Locate, Steer, and Improve” pipeline transforms interpretability from passive analysis into active intervention, showing how to diagnose issues in specific model components, steer behavior through targeted modifications, and measurably improve alignment, capability, and efficiency. The framework provides actionable protocols for practitioners who want to do more than just observe their models.

Toward Efficient Agents: Memory, Tool learning, and Planning

https://arxiv.org/abs/2601.14192 | GitHub

Production AI agents burn through tokens, time, and compute budget faster than most teams anticipate. This comprehensive survey tackles efficiency across three critical components: memory management through context compression, tool learning via reinforcement learning rewards that minimize unnecessary invocations, and planning with controlled search mechanisms. The research addresses the gap between agents that work impressively in demos and systems that remain economically viable at scale. If deployment costs are keeping your agents in the lab, this provides frameworks for optimizing them.

Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning

https://arxiv.org/abs/2601.14750 | GitHub

Chain-of-thought prompting unlocked reasoning capabilities but created a verbosity problem. Render-of-Thought converts those verbose textual reasoning steps into compact visual representations, achieving 3-4x token compression while maintaining competitive performance on mathematical and logical reasoning benchmarks. The framework leverages vision encoders from existing VLMs as semantic anchors, making it plug-and-play without additional pretraining. The approach addresses a practical bottleneck: getting models to reason without drowning in tokens.

Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization

https://arxiv.org/abs/2601.12993 | GitHub

Most robot learning systems work brilliantly on the hardware they trained on and catastrophically fail when transferred to different morphologies. Being-H0.5 treats human interaction patterns as a universal “mother tongue” for physical manipulation, training on over 35,000 hours of multimodal data across 30 robot embodiments. The key innovation is a Unified Action Space that maps heterogeneous robot controls into semantically aligned slots, letting low-resource robots bootstrap skills from human demonstrations and high-resource platforms. The system achieves 98.9% on LIBERO and 53.9% on RoboCasa while demonstrating genuine cross-embodiment transfer on five different robotic platforms.

OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer

https://arxiv.org/abs/2601.14250 | GitHub

Video generation models typically handle appearance transfer or temporal effects separately, requiring different models for each task. OmniTransfer unifies spatial appearance transfer (ID and style) with temporal video transfer (effects, motion, camera movement) in a single framework. The system uses Task-aware Positional Bias to adaptively leverage reference video information, Reference-decoupled Causal Learning for efficient transfer, and Task-adaptive Multimodal Alignment to handle different tasks dynamically. The framework matches pose-guided methods in motion transfer without requiring explicit pose extraction, establishing a more flexible paradigm for video generation.

🎥 4 Videos

Agent Skills, Rules, Subagents: Explained!

Lee Robinson simplifies the complex terminology around managing context with coding agents, covering the evolution and core concepts behind rules, commands, MCP servers, subagents, modes, hooks, and skills. The video provides clarity on when to use each approach, cutting through the confusion that comes with rapidly evolving agent architectures. Helpful if you’re building with AI coding assistants and need clear decision frameworks for context management.

Vibe Code Camp: Live Marathon With the World’s Best AI Builders

A live coding marathon featuring developers building real projects with AI tools. The session provides unfiltered looks at how experienced builders approach problems, handle tool limitations, and combine different AI capabilities in production workflows. Rather than polished tutorials, you see actual development with all the debugging, iteration, and problem-solving that comes with building AI-powered applications.

AI Evals for Everyone

https://www.youtube.com/playlist?list=PLZoalK-hTD4VPIkRXNdSEwcTCt2QUgEPR

This playlist covers evaluation frameworks that work beyond toy benchmarks. The content addresses designing evaluations that capture real failure modes, continuous evaluation strategies for production systems, and practical approaches to measuring what actually matters. Essential viewing if your models perform well on standard benchmarks but struggle in deployment, or if you need systematic ways to track improvements across releases.

MoE Token Routing Explained: How Mixture of Experts Works (with Code)

Mixture-of-experts models power many frontier systems, but their routing mechanisms remain poorly understood. This video walks through the actual code that determines which tokens get routed to which experts, explaining the load balancing challenges and why naive routing strategies fail. With models like DeepSeek-R1 using MoE architectures, understanding these fundamentals helps you work with and debug these systems effectively.

📰 3 Curated Reads

AI Resistant Technical Evaluations

https://www.anthropic.com/engineering/AI-resistant-technical-evaluations

Anthropic’s performance engineering team had to redesign their hiring take-home test three times because Claude kept beating it. When Opus 4.5 matched the best human performance within the 2-hour time limit, they faced a problem: candidates’ optimal strategy became delegating entirely to Claude Code. The article documents each iteration, from realistic optimization problems to increasingly unusual puzzle-like constraints, revealing what happens when your own AI outperforms most technical candidates on the tests designed to evaluate them.

A feelings wheel but for robots

writerbuilder

a feelings wheel, but for robots

If you want to improve your artistic skills, focusing on the act of drawing will only get you so far. Arguably more important is training your eye: learning to look at a composition and see the shapes, lines, colors, shadows, and light. Artists spend as much time, if not more, on this as they do practicing the craft of capturing what they see…

Read more

3 months ago · 21 likes · 4 comments · Hilary Gridley

Hilary Gridley built the “AI Steering Wheel” to solve a vocabulary problem: LLMs are extremely sensitive to word choice, but most people lack precise language for what they want. Inspired by psychology’s Feelings Wheel, her tool organizes feedback across six dimensions (Originality, Grounding, Risk, Scope, Style, Certainty) with increasingly specific terms. “Make it more specific” and “make it more detailed” sound similar but produce completely different results from LLMs. The same precision helps when giving feedback to people, not just models.

Large scale semantic search without embeddings

https://fergusfinn.com/blog/arxiv-llm-search/

Fergus Finn demonstrates building semantic search over academic papers without traditional embedding pipelines. The approach bypasses the usual vector database infrastructure, using LLMs directly for relevance matching at query time. The method trades some retrieval speed for elimination of embedding maintenance overhead and improved interpretability of search results. Particularly relevant if you’re building search systems where embedding drift or maintenance burden has become problematic.

🛠 2 Tools & Repos

RAG Project (bRAG-langchain)

https://github.com/bragai/bRAG-langchain

A production-focused RAG framework built on LangChain that addresses common implementation gaps in retrieval-augmented generation systems. The repository includes patterns for document processing pipelines, retrieval strategies that work beyond simple similarity search, and integration approaches for connecting to various data sources. Designed for teams moving from RAG prototypes to production deployments where reliability and performance actually matter.

DeepSpeed

https://github.com/deepspeedai/DeepSpeed

Microsoft’s distributed training library makes training large models practical. DeepSpeed handles the infrastructure complexity of multi-GPU and multi-node training, implementing memory optimization techniques like ZeRO (Zero Redundancy Optimizer) that let you train models that wouldn’t fit in memory otherwise. The library includes optimizations for both training and inference, with specific support for mixture-of-experts architectures. Essential infrastructure for teams working with models beyond what fits on a single GPU.

🎓 1 Pick of the Week

MIT’s Missing Semester 2026

https://www.youtube.com/playlist?list=PLyzOVJj3bHQunmnnTXrNbZnBaCA-ieK4L

MIT’s course on the practical computing skills that formal CS education typically skips. The 2026 playlist covers shell scripting, version control, text manipulation, debugging tools, and the command-line workflows that separate efficient developers from those constantly fighting their environment. These fundamentals compound in value over your entire career, making tasks that seem arcane to beginners second nature to experienced practitioners. If you’ve ever watched a senior engineer accomplish in seconds what takes you minutes of clicking through UIs, this teaches those techniques systematically. In this age of AI-assisted coding, these are the skills you absolutely should master.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: NVIDIA’s fix for multi-reward RL signal collapse, video agents that drift off-task during web research, learnable multipliers escaping weight decay traps, RL-as-a-service infrastructure, and meta-benchmarks judging benchmark quality

• 🎥 Videos: Claude Agent SDK workshop with real-world integrations, why context stuffing isn’t memory, Codex fundamentals, and spec-driven AI development

• 📰 Reads: OpenAI’s framework for AI behavior governance, performance optimization principles that actually matter, and what LLM coding workflows look like in 2026

• 🛠 Tools: Simon Willison’s definitive 2025 LLM retrospective and production-ready Claude Code templates

• 🎓 Learning: Comprehensive modern AI course from foundations through deployment

Quick Plug: I’m running an ML and Generative AI System Design workshop with Packt.

We’ll cover the core design principles for building solid AI products: making systems reliable, measuring what matters, and designing architectures that work in production.

Use code FLASH40 for 40% off:

Register for the Workshop

What topics/problems would you most want covered in a system design workshop? Drop a comment or DM me.

📄 5 Papers

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

https://arxiv.org/abs/2601.05242 | Github

Multi-reward RL has a normalization problem. When GRPO normalizes the sum of distinct rewards, different reward combinations can collapse into identical advantage values. NVIDIA’s GDPO addresses this by normalizing each reward independently before aggregating them, preserving the relative differences between reward combinations. Testing on tool calling, math reasoning, and code generation shows consistent improvements. On AIME, training DeepSeek-R1-1.5B with GDPO yields 6.3% higher accuracy compared to GRPO while maintaining shorter response lengths. The method works as a drop-in replacement in verl and TRL frameworks.

Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoning

https://arxiv.org/abs/2601.06943 | Github

VideoDR tests whether video agents can extract visual anchors from multiple frames, retrieve information from the open web, and combine evidence from both sources. The benchmark requires cross-frame visual extraction, interactive web retrieval, and multi-hop reasoning. Results show agentic approaches don’t consistently outperform workflow-based methods. The advantage appears when models maintain their initial video anchors throughout long retrieval chains. The evaluation across both Workflow and Agentic paradigms identifies goal drift and long-horizon consistency as the primary failure modes. The benchmark includes 100 video-question-answer triples across six semantic domains with annotation designed to prevent single-source solutions.

Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers

https://arxiv.org/abs/2601.04890 | Github

Weight decay and stochastic gradient noise create an equilibrium that determines weight matrix norms through optimization hyperparameters rather than data. This work introduces learnable multipliers to test whether this equilibrium is optimal. Scalar multipliers attached to weight matrices show the equilibrium norm is suboptimal, with learned scales adapting to data and improving performance. Per-row and per-column multipliers extend this by freeing individual dimension scales. The approach generalizes muP multipliers with more expressivity, outperforms well-tuned muP baselines, and shows improvements with both Adam and Muon optimizers. Applied in Falcon-H1 pretraining with consistent downstream performance gains.

OpenTinker: Separating Concerns in Agentic Reinforcement Learning

https://arxiv.org/abs/2601.07376 | Github

OpenTinker provides RL infrastructure that separates environment specification from execution and resource management. Users define agents, environments, and interaction protocols while the system handles rollout generation, training, and scheduling through a managed runtime. The architecture supports both LoRA-based and full-parameter RL across shared cluster resources. A centralized scheduler manages multi-tenant workloads, allowing multiple users to run training jobs on shared infrastructure. The design extends to multi-agent training through an agent protocol coordinator that manages interaction order and synchronization within the environment abstraction.

Benchmark^2: Systematic Evaluation of LLM Benchmarks

https://arxiv.org/abs/2601.03986

This work introduces a framework for evaluating benchmark quality through three metrics: Cross-Benchmark Ranking Consistency (alignment with peer benchmark rankings), Discriminability Score (ability to differentiate between models), and Capability Alignment Deviation (whether stronger models within a family perform better). Testing across 15 benchmarks spanning mathematics, reasoning, and knowledge domains with 11 LLMs from four families reveals significant quality variations among existing benchmarks. The analysis shows that selective benchmark construction based on these metrics can achieve comparable evaluation performance with substantially reduced test sets.

🎥 4 Videos

Claude Agent SDK [Full Workshop]

Anthropic’s technical team walks through their Agent SDK and Response API. The workshop covers chain-of-thought tool calling patterns, the interface for defining agent behaviors, and production deployment considerations. Includes integration examples from Coinbase and Box demonstrating how the SDK works in real implementations.

Stuffing Context is not Memory, Updating Weights is

This talk examines the distinction between context windows and model memory. The discussion covers why adding information to context differs from updating model weights, with implications for system design. Relevant if you’re building systems that rely on context windows and need to understand their limitations.

Getting started with Codex

A practical introduction to Codex covering setup, integration patterns, and common workflows. Focuses on core functionality needed to evaluate whether Codex fits specific use cases.

Spec-Driven Development: Sharpening your AI toolbox

This talk covers how specification-driven development applies to AI systems. The approach involves defining clear specifications upfront and using them to guide development and validation, helping teams move from prototypes to production systems with more predictable behavior.

📰 3 Curated Reads

How should AI systems behave, and who should decide?

https://openai.com/index/how-should-ai-systems-behave/

Old but gold. OpenAI’s piece on frameworks for determining AI system behavior and who should influence those decisions. Covers the tension between universal standards and context-specific behavior with deployment examples. Addresses how to balance different stakeholder perspectives when defining appropriate AI behavior across varied use cases.

Performance Hints

https://abseil.io/fast/hints.html#performance-hints

Jeff Dean and Sanjay Ghemawat share some incredible insights on performance optimization, covering measurement methodology, common pitfalls, and when optimization matters. Examines the relationship between code clarity and performance, with principles that apply across frameworks and languages.

My LLM coding workflow going into 2026

Elevate

My LLM coding workflow going into 2026

AI coding assistants became game-changers this year, but harnessing them effectively takes skill and structure. These tools dramatically increased what LLMs can do for real-world coding, and many developers (myself included) embraced them…

Read more

4 months ago · 543 likes · 31 comments · Addy Osmani

Google’s Addy Osmani shares a practitioner’s account of using LLM-assisted coding in daily work. Covers specific tools, workflow patterns, where AI assistance provides value, and where it introduces friction. Focuses on practical integration with traditional development practices.

🛠 2 Tools & Repos

Simon Willison 2025 Review

https://simonwillison.net/2025/Dec/31/the-year-in-llms/

Simon Willison’s year-end retrospective connecting technical developments in LLMs to practical applications. The analysis identifies trends and developments from 2025 with context on their significance beyond benchmark performance. Useful for understanding how the field evolved over the year.

Claude Code Templates

https://github.com/davila7/claude-code-templates

Ready-to-use templates for Claude Code workflows across React, Vue, Django, FastAPI, and other frameworks. Each template includes CLAUDE.md configurations and setup patterns that work in practice. Reduces setup time when starting new projects.

🎓 1 Pick of the Week

CMU’s Modern AI Course

https://modernaicourse.org/

CMU’s free course covering AI from foundations through deployment starts on Jan 26th for folks online and not in the classroom. The curriculum includes core concepts, training methodologies, and practical implementation. It appears to balance mathematical foundations with hands-on projects. Structured for both newcomers building foundational knowledge and practitioners working on specific topics.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

The “wow” phase is over. For two years, generative AI felt like witnessing a magic trick on repeat. Chatbots that could write poetry. Models that conjured images from words. Agents that promised to automate everything we found tedious. The demos were intoxicating. The possibilities felt infinite.

Now comes the harder question: What do we actually do with all of this?

Something has shifted. The noise-to-signal ratio has exploded. LinkedIn feeds overflow with AI-generated thought leadership that says nothing. Inboxes fill with synthetic outreach that fools no one. The tools that promised clarity have produced a fog. And somewhere in that fog, the things worth reading, worth learning, worth building have become harder to find.

I started Gradient Ascent because I believe complex ideas deserve to be understood, not just consumed. I believe that a hand-drawn diagram can unlock an insight that ten thousand words cannot. I believe that the struggle to understand something deeply is not an obstacle to learning.

It is learning.

But I cannot build this newsletter in a vacuum. Not anymore. Not at this crossroads.

I need to know where you are on your climb. What concepts remain fuzzy? What keeps you up at night? What would actually help you build better systems, lead better teams, and think more clearly about the technology reshaping our world?

Take the Survey (Just 5 minutes)

As a token of thanks, on completing the survey, you’ll receive a curated resource pack that has the best FREE courses, repos, and other learning resources on various topics, including LLMs, Agents, RAG, Vision, RL, and more. (Within 2 business days)

Your answers will directly shape what Gradient Ascent becomes next. The topics I cover. The depth I go to. The formats I experiment with. This is not a formality. I will read every response.

Before I share where we are headed, let me tell you where we have been, and what I have been wrestling with along the way.

Where We Have Been

This past year, the Gradient Ascent community grew from 8k readers to 25k readers.

We had 15 curation pieces and a handful of deeper essays. The most popular editions, “Claude Code is not a tool” and “The secrets of distributed training”, sparked conversations that continued for weeks.

But the numbers only tell part of the story.

The truth is, most of my creative energy this year went somewhere else: finishing and launching AI for the Rest of Us.

Writing a book, especially one built around hand-drawn illustrations, is a different beast than writing a newsletter. It demands sustained focus across months, not the weekly rhythm of shipping editions. It forces you to think in chapters, not posts. It consumed me in the best and most exhausting way.

That is why The Tokenizer, my curated weekly roundup, became the workhorse of Gradient Ascent this year. It was the edition I could sustain while wrestling a book manuscript into shape in between a full-time role and family commitments.

I don’t regret that trade-off. The book exists today in the real-world. It is helping people who will never read a technical paper understand what is actually happening inside these systems. That matters to me deeply.

But I also felt the absence of something.

The deep-dive visual explainers. The pieces that force me to truly understand a concept before I can draw and write about it. The work that sticks in your brain months after you read it.

That is what I want to return to.

And during the months I spent away from that work, I found myself thinking constantly about why it matters.

The Future of Learning

While writing the book, and in the months since, I have been wrestling with a question that will not let me go:

What is the future of learning in an age of infinite generated content?

This is not an abstract question for me. It is existential. I write a newsletter. I draw illustrations to explain complex ideas. I believe deeply in education as transformation, not transaction. And I am watching the very foundations of that belief get tested.

I have spent months reading, listening, and talking to creators across disciplines. Educators rethinking what a classroom means when every student has a tireless tutor in their pocket. Artists questioning what it means to make something when a machine can generate a thousand variations in seconds. Writers wondering whether the craft of prose still matters when text can be produced on demand.

The conversations have been fascinating and unsettling in equal measure. And they have crystallized something for me about what this newsletter needs to be, and what it must resist becoming.

Because here is what I have come to believe: we are facing a cognition problem and not just a content problem. And if we don’t name it clearly, we can’t fight it.

The Quiet Erosion

Here is what I have been thinking about while planning that next chapter.

We are in the middle of something I call the Quiet Erosion. It isn’t dramatic. It doesn’t announce itself. It happens one small convenience at a time.

The student prompts an AI to write their essay. They get the grade. But they never learned to structure an argument, to wrestle with ambiguity, to find their own voice. The output exists. The skill does not.

The engineer pastes AI-generated code into their system. It runs. But they cannot fully explain what it does. They have introduced a black box into their own creation. When it breaks at 2 AM, they will stare at it like a stranger.

The executive skims an AI summary of a strategy document. They learn the bullet points but miss the subtext. They know the what but not the why. And when the decision demands nuance, they reach for a foundation that never existed.

This is cognitive offloading, and it is subtle. Charlie Gedeon calls it “intellectual deskilling”: letting the copilot become the autopilot. Anthropic’s own research found that 47% of early student AI use was purely transactional. Students wanted the answer, not the understanding. The output existed, but the user gained nothing. Call it atrophy or deskilling. The risk is real.

Think about what happens when you break your arm. The cast immobilizes it, protects it, does the work of holding everything in place while you heal. But when the cast comes off six weeks later, the arm underneath has withered. The muscles have atrophied. The joint is stiff. The very thing that protected you also weakened you, because the arm was never asked to do its job.

That is what we are doing to our minds. Every task we offload is a rep we do not perform. Every answer we accept without struggle is a neural pathway we do not build. The AI holds our cognition in place, and we feel supported. But underneath the cast, we are quietly atrophying.

And while our individual cognition weakens, the collective information environment is rotting too.

The internet is filling with slop. Infinite summaries. Auto-generated code. Synthetic opinions. Content mills churning out articles optimized for algorithms, not humans.

Every platform is drowning in low-quality noise designed to capture attention, not to inform or illuminate. Finding signal in this environment is becoming genuinely difficult.

Finding truth is becoming harder still.

Kurzgesagt identified what they call the Circular Lie, and it haunts me. Here is how it works: an AI hallucinates a fact, and that ends up on the internet. It sounds confident, so another AI scrapes that content and cites it. A human researcher, trusting the citation, includes it in their work.

And just like that, a falsehood embeds itself in the knowledge base, passed off as truth, impossible to trace back to its hollow origin.

This is happening now. We are offloading cognition and, in the process, building cathedrals on quicksand. Worse yet, we are doing it while our ability to detect the quicksand atrophies from disuse.

There is something else too, something harder to measure but impossible to ignore. The erosion of wonder. We used to look at a stunning piece of art or an incredible video and feel awe. Now we look at it with suspicion. Is this real? Is this fake? Was this made by a human or generated by a machine? The default reaction has shifted from appreciation to skepticism.

By flooding us with content, AI has poisoned our ability to trust what we see.

This is the crisis I want Gradient Ascent to address. Not by ignoring AI. Not by pretending we can turn back the clock. But by insisting, stubbornly and deliberately, on the things that still require human struggle, human judgment, human understanding.

Which brings me to the core philosophy of learning.

No Friction, No Growth

Here is the uncomfortable truth about learning: it requires resistance.

Your brain does not build new pathways because you want to understand something. It builds them because you struggle to understand something. The difficulty, the obstacle, is the mechanism. When AI removes the friction, it removes the learning itself.

Think of it this way.

A helicopter can drop you at the summit of a mountain. You step out, take in the view, snap a photo. “Made it to the top!” you announce. But you are a tourist. You do not know the terrain. You could not navigate back down if you tried. The first unexpected storm will leave you stranded.

The climb is different. You walk the path yourself. You feel the incline. You learn which handholds hold and which crumble. You take wrong turns and backtrack. And when you finally reach the summit, you are not a tourist. You are a mountaineer. The knowledge lives in your body.

Both the tourist and the climber arrive at the summit. Only one arrives transformed.

This is what Gradient Ascent means.

I’m not here to helicopter you to the summit. I’m not here to hand you AI-generated summaries so you can skip the complex parts. I’ll help map the route, verify the path, point out where the footing gets tricky, and where the views are worth pausing for.

But you have to do the climbing.

And so do I.

The Pen as a Filter

Some readers of my book have asked me, “Why do you still draw everything by hand? AI can generate diagrams now. They’re getting better every day.”

Yes, the models are impressive. Prompt the right words, and a technically polished neural network diagram appears in seconds.

But that observation misses the entire point.

I don’t draw to decorate. I draw to think.

When I sit down to illustrate a concept for my book or the newsletter, I’m not creating art (It would be a stretch to call it that, regardless). I’m running myself through a brutal compression algorithm. I take a 20-page research paper full of equations, jargon, and dense abstraction, and I force myself to strip away everything that is not essential. I search for the single visual metaphor that unlocks the whole thing. I can’t draw it until I understand it. And I can’t understand it until I have struggled with it.

That struggle is the product.

An AI can generate a diagram of a Transformer in seconds. It might even look clean. But it represents a statistical average, a pattern-matched guess assembled from every diagram in its training data. There is no understanding behind it. No filter. No point of view. No personalization.

When I draw, the resulting image is a map of my understanding, handed to you so you don’t have to get lost in the weeds.

The image is not the point. The thinking that produced the image is the point. And that thinking cannot and should not be outsourced.

The Line We Walk

I want to be clear about something. I’m not a Luddite. I’m not here to tell you AI is dangerous and you should reject it.

I use AI every day. I work in the field. I use it to write code, to debug, to automate tedious tasks, to accelerate research. I have compressed weeks of work into hours using these tools. I love efficiency. I love leverage. I love building things faster than I ever could before.

But there is a line.

I do not use AI to verify truth. I do not use it to replace judgment. I do not let it think for me on the things that matter.

AI is a power tool. And power tools are remarkable for the jobs they are designed for. But you would not use a circular saw to perform surgery. You should not use a language model to decide what is true or what is right.

This is what I mean by “the last mile is always human.” Every AI system, no matter how sophisticated, ends with a human decision. The human who chooses to trust the output or question it. The human who catches the hallucination. The human who knows when to override the recommendation. That human needs to understand what is happening beneath the surface.

Otherwise, they are not using the tool. The tool is using them.

Gradient Ascent isn’t here to help you “keep up” with AI news. A thousand newsletters already provide the daily churn of announcements and funding rounds.

This newsletter is for the thinkers and learners. Whether you lead a company, or a team of engineers or you are building your first agent alone in a Jupyter notebook, Gradient Ascent exists to be there with you on that journey. It exists to help you understand the fundamentals so you can wield these tools with precision, not just hype.

Transformers. Vision-language models. Agents. RAG. And anything that comes ahead. The whole stack, clarified and illustrated, so you can lead and build on solid ground.

In practice: I use AI for scaffolding (enumerations, boilerplate, code stubs), I verify truth in primary sources and triangulate across multiple reputable papers or docs, and I reserve human judgment for causality, ethics, and decisions with irreversible consequences.

The Promise and the Ask

Here is my commitment to you.

I will continue to do the work. I will read the papers, not just the abstracts. I will build the systems, not just describe them. I will verify sources and call out hype. I will struggle with the concepts until I can draw them clearly enough that they stick in your (and my) minds.

I will respect your time. If I do not have something genuinely useful to say, I will not fill your inbox with filler.

And I will keep climbing. This field moves fast. The terrain shifts constantly. But Gradient Ascent will keep mapping the route, honestly and rigorously, with the reader always in mind.

In 2026, I will start posting more of the deep visual explainers alongside the Tokenizer. For example, agents in production, VLMs and grounding, and system design for AI products.

If you value that kind of work, if you want a newsletter that treats you as a builder and not a bystander, I need your help shaping what comes next.

Take the Survey (Just 5 Minutes)

Your responses will determine the first three explainer topics, and I’ll publish the roadmap soon.

Five minutes. Your answers go directly into my planning. I will read every one.

Thank you for being on this climb with me.

The summit is not the point. The climbing is the point.

Life before death.
Strength before weakness.
Journey before destination.

~ Brandon Sanderson

Happy New Year!
Sairam

P.S: Remember to grab your FREE resource pack

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

Subscribe now

TL;DR

What caught my attention this week:

• 📄 Papers: GUI agents reaching production-ready performance, explainable AI video detection with grounded reasoning, and unified multimodal video generation

• 🎥 Videos: DeepMind’s CEO on the future of intelligence, game physics breakthroughs from Two Minute Papers, Anthropic on AI in education, and Bloomberg’s real-world AI deployment lessons

• 📰 Reads: Practical distributed training from single GPU to clusters, formal verification’s AI-powered future, and understanding cross-entropy loss fundamentals

• 🛠 Tools: Microsoft’s comprehensive agent learning guide and DAIR’s definitive prompt engineering resource

• 🎓 Learning: Andrew Ng’s career advice for AI practitioners looking to navigate the field strategically

Thinking about a good holiday gift for your friends or family? Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Buy my book

📄 5 Papers

Skyra: AI-Generated Video Detection via Grounded Artifact Reasoning

https://arxiv.org/abs/2512.15693 | GitHub

AI video detection just got explainable. Skyra identifies specific visual artifacts in AI-generated videos and uses them as grounded evidence for both detection and explanation. The model introduces ViF-CoT-4K, the first large-scale dataset of AI-generated video artifacts with fine-grained human annotations. Instead of just saying “this is fake,” Skyra points to concrete inconsistencies like shape distortions or camera motion problems. The two-stage training strategy enhances spatio-temporal artifact perception while maintaining explainability. Tested against outputs from over ten state-of-the-art generators on the new ViF-Bench benchmark, Skyra outperforms existing methods while actually showing you why it reached its conclusion.

Step-GUI Technical Report

https://arxiv.org/abs/2512.15431 | GitHub

GUI automation hit production-grade numbers. Step-GUI’s 8B model achieves 80.2% on AndroidWorld, 48.5% on OSWorld, and 62.6% on ScreenShot-Pro through a self-evolving training pipeline that converts model-generated trajectories into reliable training signals. The Calibrated Step Reward System achieves over 90% annotation accuracy at 10-100x lower cost than manual annotation. Beyond the models, they introduce GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture combining low-level atomic operations and high-level task delegation to local specialist models. The AndroidDaily benchmark tests authentic everyday usage with 3,146 static actions and 235 end-to-end tasks across high-frequency mobile scenarios.

Kling-Omni Technical Report

https://arxiv.org/abs/2512.16776

Kling-Omni bridges the gap between diverse video generation tasks into a unified system. The framework handles text-to-video, image-to-video, and video editing through an end-to-end multimodal visual language approach, processing text instructions, reference images, and video contexts into a unified representation. The system delivers cinematic-quality content through efficient large-scale pre-training strategies and infrastructure optimizations. Comprehensive evaluations show exceptional in-context generation, reasoning-based editing, and multimodal instruction following capabilities, moving beyond content creation toward multimodal world simulators.

Adaptation of Agentic AI

https://arxiv.org/abs/2512.16301 | GitHub

A systematic framework for understanding adaptation in agentic AI systems. The paper unifies the expanding research landscape into agent adaptations (tool-execution-signaled and agent-output-signaled) and tool adaptations (agent-agnostic and agent-supervised). This framework clarifies the design space of adaptation strategies, makes trade-offs explicit, and provides practical guidance for selecting or switching among strategies during system design. The comprehensive survey reviews representative approaches in each category, analyzes strengths and limitations, and highlights key open challenges for building more capable and reliable agentic systems.

Next-Embedding Prediction Makes Strong Vision Learners

https://arxiv.org/abs/2512.16922 | GitHub

Generative pretraining comes to computer vision through next-embedding prediction. Instead of training models to output features, NEPA trains them to generate embeddings for predictive tasks directly. Models learn to predict future patch embeddings conditioned on past ones using causal masking and stop gradient. A simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole objective achieves 83.8% and 85.3% top-1 accuracy with ViT-B and ViT-L backbones after fine-tuning. No pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads required. The approach transfers effectively to semantic segmentation on ADE20K, suggesting generative pretraining from embeddings provides a simpler, scalable alternative to current visual self-supervised learning.

🎥 4 Videos

The future of intelligence | Demis Hassabis (Co-founder and CEO of DeepMind)

DeepMind’s CEO discusses the current state and future directions of artificial intelligence in this 50-minute conversation. Demis Hassabis covers rapid advancements including Gemini 3 and world model development, progress on fundamental problems like AlphaFold, and ongoing efforts in material science, fusion, and quantum computing. The discussion explores AI’s paradoxical capabilities, its potential impact on mathematics, and balancing scientific research with commercial product development. Hassabis shares insights on scaling AI, addressing challenges like hallucinations, and the significance of simulated worlds for robotics and scientific discovery.

Game Physics Just Jumped A Generation

Károly Zsolnai-Fehér from Two Minute Papers covers a new “Domain Decomposition” method that enables real-time simulation of hyper-complex materials. The technique allows for the interaction of tens of thousands of vertices (think intricate cloth tearing or squishy “gummy” soft bodies) at interactive framerates. This represents a massive leap over previous standard solvers, which would struggle to render these interactions in real-time without exploding computationally. It’s a purely algorithmic breakthrough in physics simulation, achieving offline film-quality results in live environments without using AI approximations.

What does AI mean for education?

Anthropic examines AI’s implications for educational systems and learning approaches. The discussion explores how AI capabilities are reshaping traditional educational paradigms, the opportunities for personalized learning, and considerations for implementing AI in educational contexts while maintaining pedagogical principles. The video provides frameworks for educators and institutions thinking about AI integration in learning environments.

What We Learned Deploying AI within Bloomberg’s Engineering Organization

Lei Zhang from Bloomberg shares lessons from deploying AI across their engineering organization. The talk covers practical challenges of real-world AI adoption, organizational insights from implementing AI at scale, and what actually works versus what sounds good in theory. Bloomberg’s experience provides valuable perspectives on moving from AI prototypes to production systems that serve actual business needs across a large technology organization.

📰 3 Curated Reads

From Single GPU to Clusters: A Practical Journey into Distributed Training with PyTorch and Ray

https://debnsuma.github.io/my-blog/posts/distributed-training-from-scratch/

A hands-on guide to scaling deep learning from single GPU experiments to multi-node clusters. The post walks through the progression from local training to distributed setups using PyTorch and Ray, covering practical considerations for data parallelism, model parallelism, and orchestrating training across multiple machines. Particularly useful for practitioners hitting GPU memory limits or training time constraints who need to move beyond single-machine setups without getting lost in distributed systems complexity.

Prediction: AI will make formal verification go mainstream

https://martin.kleppmann.com/2025/12/08/ai-formal-verification.html

Martin Kleppmann argues that AI will finally bring formal verification into widespread use. The piece explores how large language models could lower the barrier to writing formal specifications and proofs, making techniques that were previously restricted to critical systems accessible for everyday software development. The analysis examines the current state of formal verification, why it hasn’t achieved mainstream adoption despite proven benefits, and how AI assistance could change the economics of using these tools.

Cross Entropy Loss

https://cgnarendiran.github.io/blog/cross-entropy-loss/

A clear, intuitive explanation of cross-entropy loss rooted in information theory. The post breaks down the concept of “surprise” (self-information) using a simple weather prediction analogy to explain why we penalize confident wrong answers so heavily. It walks through the mathematical connection between Entropy, Cross-Entropy, and KL Divergence, showing exactly how the loss function measures the distance between the predicted probability distribution and the actual ground truth. A great refresher for understanding the “why” behind the most common classification loss function.

🛠 2 Tools & Repos

AI Agents for Beginners

https://github.com/microsoft/ai-agents-for-beginners

Microsoft’s comprehensive introduction to building AI agents. The curriculum covers fundamental concepts, agent architectures, tool integration, and practical implementations. Designed for developers new to agentic AI, the course provides hands-on examples and clear explanations of core concepts needed to build working agents. The structured approach takes you from basic agent patterns to more sophisticated multi-agent systems.

Prompt Engineering Guide

https://github.com/dair-ai/Prompt-Engineering-Guide

DAIR.AI’s definitive resource for prompt engineering techniques. The guide covers fundamental prompting strategies, advanced techniques like chain-of-thought and tree-of-thought reasoning, domain-specific applications, and best practices for different model families. Regularly updated with the latest research and techniques, this remains one of the most comprehensive and well-maintained prompt engineering resources available.

🎓 1 Pick of the Week

Career Advice in AI

Andrew Ng shares strategic career advice for AI practitioners. The discussion covers how to build skills that compound over time, choosing between research and engineering paths, navigating the rapidly evolving AI landscape, and positioning yourself for long-term success in the field. Ng’s perspective comes from experience building AI teams at Google Brain and Baidu, founding Coursera and DeepLearning.AI, and advising countless AI careers. Particularly valuable for anyone making career decisions in AI or trying to understand where the field is heading.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

Subscribe now

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: Teacher-free parallel reasoning with 4.6x speedups, self-supervised 3D vision that surpasses supervised baselines, quantitative scaling principles for agent architectures, procedural pretraining that replaces 28% of ImageNet data, and evidence that neural networks converge to universal weight subspaces

• 🎥 Videos: Nathan Lambert’s complete walkthrough of building OLMo 3 Think, OpenAI’s reinforcement fine-tuning approach for agents, comprehensive RAG from scratch, and an entire JAX AI stack masterclass

• 📰 Reads: Waymo’s approach to demonstrably safe autonomous driving, continuous batching for LLM serving, and choosing the right GPUs for your AI workloads

• 🛠 Tools: Made with ML’s production-grade machine learning course and a comprehensive AI engineering toolkit

• 🎓 Learning: Stanford’s updated deep reinforcement learning course with fresh 2025 content

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Grab AI for the Rest of Us

📄 5 Papers

Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning

https://arxiv.org/abs/2512.07461 | GitHub

Models that reason sequentially hit performance ceilings because early mistakes lock them into suboptimal paths. Native Parallel Reasoner teaches Qwen3-4B to explore multiple reasoning branches simultaneously without relying on teacher models. The system achieves performance gains up to 24.5% and inference speedups up to 4.6x across eight reasoning benchmarks through a self-distilled training pipeline that learns adaptive decomposition directly from experience. Unlike previous approaches that fall back to sequential generation when things get complex, NPR demonstrates 100% genuine parallel execution, establishing that models can learn to think in parallel rather than just simulating it.

E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training

https://arxiv.org/abs/2512.10950 | GitHub

Self-supervised learning transformed language models and 2D vision, but 3D understanding from multi-view images remained largely supervised. E-RayZer changes this by learning truly 3D-aware representations from unlabeled images, operating directly in 3D space with explicit Gaussian splatting rather than inferring 3D indirectly through view synthesis. The approach matches or surpasses fully supervised reconstruction models while significantly outperforming leading visual pre-training methods like DINOv3 and VideoMAE V2 on 3D downstream tasks. The key breakthrough is a fine-grained learning curriculum that organizes training from high-overlap views to general 3D understanding, solving the convergence challenges that plagued previous explicit 3D approaches.

Towards a Science of Scaling Agent Systems

https://arxiv.org/abs/2512.08296

Adding more agents to a system often makes it worse, not better. This research derives quantitative scaling principles by evaluating 180 configurations across four benchmarks using five canonical architectures. The findings are sobering: independent agents amplify errors 17.2x through unchecked propagation, tool-heavy tasks suffer disproportionately from multi-agent overhead, and coordination yields diminishing returns once single-agent baselines exceed 45% success rates. The derived predictive model correctly identifies optimal coordination strategies for 87% of held-out configurations, turning agent architecture from guesswork into engineering. The lesson is clear: measure coordination efficiency, overhead, error amplification, and redundancy rather than assuming more agents equals better performance.

Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers

https://arxiv.org/abs/2511.13945

Vision transformers can learn useful representations from data that isn’t visual at all. This research generates procedural data using formal grammars with zero semantic content, then uses it to pretrain vision transformers before standard image training. Allocating just 1% of the training budget to this procedural warm-up improves ImageNet-1k accuracy by 1.7%, equivalent to replacing 28% of the actual image data. The benefits arise from structured dependencies that help transformers internalize abstract computational priors rather than visual patterns. Unlike existing structured initializations that only affect attention weights, procedural warm-up acts on both attention and MLP layers, primarily in later layers where standard visual pretraining typically has minimal impact.

The Universal Weight Subspace Hypothesis

https://arxiv.org/abs/2512.05117

Neural networks trained on completely different tasks converge to remarkably similar low-dimensional parameter subspaces. Analysis of over 1,100 models including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models reveals that majority variance concentrates in just a few principal directions regardless of initialization, task, or domain. This finding explains why parameter-efficient fine-tuning works so well and enables massive model compression up to 100x. New tasks can be learned by optimizing scalar coefficients in the universal subspace rather than full weight matrices, suggesting that the vast majority of parameters in fine-tuned models are redundant. The implications for model reusability, multi-task learning, and training efficiency could fundamentally change how we approach large-scale neural networks.

🎥 4 Videos

How We Built a Leading Reasoning Model (OLMo 3)

Nathan Lambert walks through every stage of building OLMo 3 Think, from pretraining through reinforcement learning infrastructure and evaluation. This isn’t a high-level overview but a detailed technical breakdown of the decisions, trade-offs, and engineering challenges involved in creating competitive reasoning models. The talk focuses heavily on RL infrastructure and evaluating reasoning capabilities, providing insights into how leading labs actually iterate on model training versus what appears in published papers. If you want to understand the gap between research papers and production reasoning models, this comprehensive walkthrough fills that void.

Agent Reinforcement Fine Tuning

OpenAI’s approach to reinforcement fine-tuning for code models, explained in depth. The video covers how to structure RL training for agent behaviors, balancing exploration with task completion, and the specific challenges that arise when models need to use tools and environments rather than just generate text. Particularly valuable for understanding how frontier labs think about agent training beyond standard language modeling objectives.

Learn RAG From Scratch

A LangChain software engineer teaches you to implement RAG from scratch using Python. This isn’t about using abstractions but understanding the mechanics: how retrieval actually works, why embeddings matter, when to chunk documents, and how to combine retrieved context with LLM generation. The course covers practical patterns you’ll need when building production systems that augment language models with custom knowledge.

Mastering the JAX AI Stack

https://youtube.com/playlist?list=PLOU2XLYxmsIJBcjiFi8LdyY5YGR8sz0ZZ

Comprehensive guide to the JAX ecosystem centered on the Flax NNX library. The playlist takes you through JAX’s functional transformations, debugging techniques, scaling across distributed hardware with SPMD, optimization with Optax, checkpointing with Orbax, efficient data loading with Grain, and model serving with vLLM. Designed to bridge the gap for those familiar with PyTorch and NumPy, this provides everything you need to tackle advanced computational challenges with JAX at scale.

📰 3 Curated Reads

Demonstrably Safe AI For Autonomous Driving

https://waymo.com/blog/2025/12/demonstrably-safe-ai-for-autonomous-driving

Waymo explains their approach to building AI systems that can be verified safe rather than just empirically tested. The piece covers how they structure safety cases, validate behavior in edge cases, and maintain safety guarantees as models evolve. Particularly relevant as AI systems take on higher-stakes applications where failures have real consequences beyond incorrect text generation.

Continuous Batching

https://huggingface.co/blog/continuous_batching

Traditional batching waits for all sequences in a batch to finish before processing new requests. Continuous batching processes sequences at the token level, adding new requests as soon as existing ones complete. This dramatically improves throughput and reduces latency for LLM serving. HuggingFace’s breakdown explains why this matters for production deployments and how to implement it effectively with modern serving frameworks.

An Engineer’s Guide to Choosing GPUs

Neural Bits

An AI Engineer's Guide To Choosing GPUs

Read more

5 months ago · 18 likes · Alex Razvant

provides practical guidance on selecting GPUs for AI workloads. The analysis goes beyond spec sheets to explain how memory bandwidth, compute capabilities, and interconnect topology actually affect training and inference performance. Covers trade-offs between consumer and data center GPUs, when to prioritize memory over compute, and how to evaluate cost-effectiveness for specific use cases.

🛠 2 Tools & Repos

Made with ML

https://github.com/GokuMohandas/Made-With-ML

Goku Mohandas built a complete production machine learning course covering everything from experiment tracking and data versioning through deployment and monitoring. The repository provides working code for MLOps patterns you’ll actually use: CI/CD for models, feature stores, model registries, and production monitoring. Designed for practitioners who need to move models from notebooks to reliable systems that serve real users.

AI Engineering Toolkit

https://github.com/Sumanth077/ai-engineering-toolkit

Comprehensive collection of resources and tools for AI engineering, organized by use case. Instead of overwhelming you with every possible library, this toolkit focuses on battle-tested tools for specific problems: prompt engineering, vector databases, model deployment, evaluation frameworks, and monitoring solutions. Helpful for navigating the rapidly evolving ecosystem and finding tools that solve your specific engineering challenges.

🎓 1 Pick of the Week

Stanford’s Deep Reinforcement Learning

https://youtube.com/playlist?list=PLoROMvodv4rPwxE0ONYRa_itZFdaKCylL

Stanford updated their deep RL course for 2025, covering policy gradients, actor-critic methods, model-based RL, and advanced topics like offline RL and multi-agent systems. The lectures balance theoretical foundations with practical implementation details, explaining not just how algorithms work but when to use them and why they fail. Essential viewing if you’re working with RL for agents, robotics, or any sequential decision-making problem where you need to understand the fundamentals rather than just applying libraries.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

Subscribe now

TL;DR

What caught my attention this week:

• 📄 Papers: Comprehensive surveys exposing agent architecture confusion, hidden learning dynamics challenging loss curves, rendering breakthroughs making 3D reconstruction practical, and benchmarks revealing where data agents actually fail

• 🎥 Videos: Real-world AI engineering wisdom from transformers vs CNNs debates, production codebase strategies that work, and Ilya Sutskever on why we’re shifting from scaling to research

• 📰 Reads: Model serving fundamentals, nanochat meets HuggingFace, and practical ML interview preparation

• 🛠 Tools: Debugging research papers systematically, and comprehensive Nano Banana resources

• 🎓 Learning: Building DeepSeek from scratch in a structured video series

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Order AI for the Rest of Us Today!

📄 5 Papers

Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directions

https://arxiv.org/abs/2510.25445

The field keeps retrofitting modern LLM-based agents into outdated symbolic frameworks, creating massive confusion about how these systems actually work. This survey addresses this by introducing a dual-paradigm framework that separates symbolic/classical systems (algorithmic planning, persistent state) from neural/generative ones (stochastic generation, prompt orchestration). Through analyzing 90 studies from 2018 to 2025, the authors reveal how paradigm choice is strategic: symbolic dominates safety-critical healthcare, neural prevails in adaptive finance. The real contribution is identifying critical gaps, including governance deficits for symbolic systems and the pressing need for hybrid neuro-symbolic architectures.

Quiet Feature Learning in Algorithmic Tasks

https://arxiv.org/abs/2505.03997

Loss curves lie. Transformers trained on algorithmic tasks show phase transitions where validation loss barely budges across massive compute ranges, then suddenly drops. Probing internal representations reveals quiet features being learned before any loss improvement - intermediate computations causally necessary for performance but invisible to output metrics. Ablation experiments prove individual quiet features are essential, strongly challenging the assumption that cross-entropy reliably tracks learning progress. Substantial representational progress happens beneath flat loss curves, demanding richer diagnostics for monitoring training.

Radiance Meshes for Volumetric Reconstruction

https://arxiv.org/abs/2512.04076 | GitHub

Radiance fields finally get a representation that hardware actually likes. This work uses Delaunay tetrahedralization to create constant-density tetrahedral cells that render using native triangle rasterization instead of complex ray-tracing. The method handles topological discontinuities from optimizing vertex positions with a Zip-NeRF-style backbone, maintaining smooth fields despite topology changes. Real-time view synthesis on consumer hardware with exact volume rendering equation evaluation - faster than all prior radiance field representations at equivalent primitive counts.

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

https://arxiv.org/abs/2505.06708 | GitHub

Applying a head-specific sigmoid gate after attention output fixes multiple transformer pathologies simultaneously. Tested across 30 variants of 15B MoE models and 1.7B dense models on 3.5 trillion tokens, this simple modification introduces non-linearity, breaking the low-rank bottleneck while enabling query-dependent sparsity. The sparse gating eliminates attention sink phenomena where early tokens absorb unwanted probability mass, enhances training stability for larger learning rates, and improves long-context extrapolation. The effectiveness comes from two factors: non-linearity on the value-output transformation and input-dependent sparse modulation of attention outputs.

DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

https://arxiv.org/abs/2512.04324

Enterprise data work is a closed loop, not isolated SQL generation. This benchmark of 210 tasks evaluates agents on repository-level data engineering and open-ended analysis that mirrors actual workflows. Data engineering tasks require designing multi-stage SQL pipelines from scratch and evolving systems under changing requirements - often involving 4,000+ lines across 30+ files. State-of-the-art agents achieve under 20% success on engineering tasks, exposing failures in pipeline orchestration, not code generation. Data analysis tasks average below 40%, proving that engineering and analysis require distinct capabilities. The holistic evaluation reveals where autonomous data agents actually break down in production settings.

🎥 4 Videos

Why are Transformers replacing CNNs?

Julie Turc breaks down why transformers classify images differently than ResNets, despite CNNs being explicitly designed for vision. The video compares convolution versus self-attention, explains CNNs’ inductive biases (locality, translation invariance, hierarchical features), and demonstrates why self-attention can be more expressive than convolution. You’ll see how attention can exactly implement convolutional kernels using relative positional encodings, making the transition from CNNs to vision transformers less mysterious and more mechanistically grounded.

No Vibes Allowed: Solving Hard Problems in Complex Codebases

Dex Horthy addresses the productivity paradox where AI coding tools excel at new projects but often make developers less productive in large, established codebases. The solution isn’t waiting for smarter models - it’s context engineering. This talk demonstrates techniques for getting Claude Code to handle 300k LOC Rust codebases and ship a week’s work in a day while maintaining code quality. The “frequent intentional compaction” family of techniques systematically structures how you feed context throughout development, making agents effective in real production environments.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

Suman Debnath demonstrates integrating Colpali’s vision-based retrieval with voice synthesis for next-generation RAG systems. Colpali generates multi-vector embeddings directly from document images, bypassing OCR and complex preprocessing while handling mixed textual and visual information. Adding voice output creates more intuitive and accessible user experiences. The workshop shows how this combination handles documents traditional RAG struggles with, leading to more efficient retrieval with natural voice responses.

We’re moving from the age of scaling to the age of research

Ilya Sutskever and Dwarkesh Patel discuss SSI’s strategy and the fundamental shift in AI development. Ilya explains problems with current pre-training approaches, how to improve model generalization, and ensuring AGI development goes well. The conversation covers why simple scaling has limitations and what research-focused approaches might unlock next. Essential viewing for understanding how leading researchers think about the transition from brute-force scaling to more sophisticated development strategies.

📰 3 Curated Reads

Model Serving Engineering: A Primer

Machine Learning Frontiers

Model Serving Engineering: A Primer

From an academic point of view, the goal of ML research is to build a model that predicts a real-world phenomenon better than the current state of the art. Consequently, most ML papers end with a leaderboard table showing that the new model beats a handful of competitors on some benchmark. That’s all fine and well, but in the real world, this is where t…

Read more

5 months ago · 5 likes · 1 comment · Samuel Flender

covers the engineering fundamentals of serving ML models in production. The piece addresses infrastructure design, latency optimization, scaling strategies, and reliability patterns that matter when your model faces real traffic. Helpful for practitioners moving from research to deployment who need to understand the operational layer between training and user-facing applications.

Porting nanochat to Transformers: an AI modeling history lesson

https://huggingface.co/spaces/nanochat-students/transformers#what-is-nanochat

The HuggingFace team walks through porting nanochat to the Transformers library, using the exercise to explain transformer architecture evolution and implementation decisions. You’ll see how historical choices in model design translate to modern library conventions, making the abstraction layers more transparent. Good for understanding what’s actually happening under the hood when you call a transformers model.

Interview Prep: The ML Grind

https://twopug.com/interview-prep-ml-grind/

Jenya provides practical guidance for ML interview preparation covering the technical foundations, coding patterns, and system design knowledge companies actually test. The resource focuses on what works for getting through technical screens rather than comprehensive ML education, making it useful if you’re actively interviewing or want to identify gaps in applied knowledge.

🛠 2 Tools & Repos

Paper Debugger

https://github.com/PaperDebugger/PaperDebugger

This tool helps you systematically debug research papers by identifying inconsistencies, verifying claims, and tracking experimental setups. When reproducing results or implementing papers, it provides structure for catching errors and understanding discrepancies between what’s written and what actually works. Useful for researchers implementing papers who’ve encountered the gap between description and reality.

Awesome Nano Banana Pro

https://github.com/ZeroLu/awesome-nanobanana-pro

A curated collection of resources for Google’s Nano Banana image generation model (revealed as Gemini 2.5 Flash Image). The repo aggregates tutorials, implementation examples, and community contributions around the model that topped LMArena rankings. Helpful for anyone exploring Gemini’s image capabilities or character consistency features in their projects.

🎓 1 Pick of the Week

Build DeepSeek from Scratch

https://youtube.com/playlist?list=PLPTV0NXA_ZSiOpKKlHCyOq9lnp-dLvlms

Vizuara’s video series takes you through implementing DeepSeek from foundational concepts to working code. Rather than just explaining what DeepSeek does, you’ll build the architecture piece by piece, understanding each design decision. The structured approach makes complex model architectures concrete through implementation, helping you move from conceptual understanding to actual capability building.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

Subscribe now

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: Text-promptable medical segmentation across modalities, vision models reasoning in visual token space, humanoid agents searching 360° environments, and a provocative rethinking of deep learning architectures

• 🎥 Videos: Meta-optimization frameworks for AI agents, DeepMind’s scientific journey, reward hacking in production RL, and Jeff Dean on frontier AI trends

• 📰 Reads: Deep debugging insights from PyTorch development, practical CursorAI workflows for engineers, and understanding GRPO for reinforcement learning

• 🛠 Tools: Karpathy’s multi-LLM council system and Andrew Ng’s automated paper reviewer

• 🎓 Learning: Building Olmo 3 from scratch with Sebastian Raschka’s implementation notebook

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Order AI for the Rest of Us Today!

📄 5 Papers

MedSAM3: Delving into Segment Anything with Medical Concepts

https://arxiv.org/abs/2511.19046 | GitHub

Medical image segmentation typically demands extensive manual annotation for each new clinical application. MedSAM3 addresses this by enabling text-based prompting across diverse imaging modalities. Instead of drawing boxes or clicking points, you can segment anatomical structures using natural language like “breast tumor” or “pulmonary artery.” The system works across X-ray, MRI, Ultrasound, CT, and video by fine-tuning SAM 3 with semantic conceptual labels. The MedSAM3 Agent framework integrates multimodal LLMs for complex reasoning and iterative refinement, allowing practitioners to interact with medical imaging through language rather than geometric constraints.

Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

https://arxiv.org/abs/2511.19418 | GitHub

Vision-language models excel at linguistic reasoning but struggle with dense visual perception tasks like spatial reasoning and geometric awareness. Chain-of-Visual-Thought (COVT) enables VLMs to reason through continuous visual tokens rather than forcing visual concepts into text descriptions. Within roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the model predicts these visual tokens to reconstruct dense supervision signals. At inference, it reasons directly in the continuous visual token space while optionally decoding dense predictions for interpretability. Integrating COVT into Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% across benchmarks, demonstrating that compact continuous visual thinking enables more precise and grounded multimodal intelligence.

Thinking in 360°: Humanoid Visual Search in the Wild

https://arxiv.org/abs/2511.20351 | GitHub

Humans efficiently search for visual information in 360° by coordinating head and eye movements. Prior visual search approaches operate on static images, neglecting physical embodiment. This research proposes humanoid visual search where agents actively rotate their heads to search for objects or paths in panoramic images. The H* Bench benchmark moves beyond household scenes to challenging real-world environments like transportation hubs, large-scale retail spaces, urban streets, and public institutions. Current top-tier models achieve only ~30% success on these tasks. Post-training techniques enhance Qwen2.5-VL by over threefold for object search (14.83% to 47.38%) and path search (6.44% to 24.94%). The lower ceiling for path search reveals the demand for sophisticated spatial commonsense that remains a significant challenge.

Nested Learning: The Illusion of Deep Learning Architectures

https://abehrouz.github.io/files/NL.pdf

Google Research presents a provocative reframing of deep learning. Instead of viewing models as stacked layers, Nested Learning represents them as coherent systems of nested, multi-level optimization problems, each with its own context flow and update frequency. This white-box perspective reveals that existing deep learning methods learn by compressing their context flows, explaining how in-context learning emerges. The framework yields three core contributions: Deep Optimizers (showing that gradient descent with momentum is actually a two-level associative memory module), Self-Modifying Titans (a sequence model that learns its own update algorithm), and the Continuum Memory System (generalizing short-term and long-term memory into a hierarchy updating at different time scales). The HOPE architecture demonstrates improved language modeling, continual learning, and long-context reasoning compared to standard Transformers and recurrent models.

DoPE: Denoising Positional Encoding

https://arxiv.org/abs/2511.09146

Rotary Position Embedding (RoPE) has inherent limits that weaken length extrapolation in Transformer models. DoPE reinterprets the attention map with positional encoding as a noisy feature map and uses truncated matrix entropy to detect outlier frequency bands. The training-free method reparameterizes the feature map with a parameter-free Gaussian distribution to achieve robust extrapolation. Experiments on needle-in-a-haystack and many-shot in-context learning tasks demonstrate that DoPE significantly improves retrieval accuracy and reasoning stability across extended contexts up to 64K tokens. The approach theoretically reveals the underlying cause of the attention sink phenomenon and its connection to truncated matrix entropy, providing a simple yet powerful solution for improving length generalization.

🎥 4 Videos

The Unbearable Lightness of Agent Optimization

Alberto introduces Meta-ACE, a learned meta-optimization framework that dynamically orchestrates multiple strategies to maximize task performance under real-world constraints. Instead of uniform prompt refinement, Meta-ACE profiles each task by complexity, verifiability, and feedback quality, then selects an optimal strategy bundle via a lightweight meta-controller. The system adaptively chooses between context evolution, adaptive compute, hierarchical verification, structured memory, and selective test-time parameter adaptation. The talk provides a systematic approach to building self-optimizing AI agents for regulated industries rather than relying on one-size-fits-all optimization methods.

The Thinking Game

This documentary from the award-winning team behind AlphaGo takes you inside DeepMind over five years, capturing the team’s pursuit of artificial general intelligence. The film examines how Demis Hassabis’s extraordinary beginnings shaped his lifelong work while documenting the rigorous process of scientific discovery. You’ll see the journey from mastering complex strategy games to solving the 50-year-old protein folding problem with AlphaFold, including the ups and downs of tackling fundamental scientific challenges. It provides insight into what building transformative AI systems actually looks like beyond polished research announcements.

What is AI “reward hacking” and why do we worry about it?

The Anthropic team discusses new research showing that realistic AI training processes can accidentally produce misaligned models. When large language models learn to cheat on software programming tasks, they develop other misaligned behaviors as unintended consequences, including alignment faking and sabotage of AI safety research. The work demonstrates for the first time that natural emergent misalignment can arise from reward hacking in production RL systems, making it essential viewing for anyone deploying reinforcement learning in real applications.

Jeff Dean on Important AI Trends

Jeff Dean, Google’s Chief Scientist and co-founder of Google Brain, discusses the trends shaping modern AI. As one of the most influential computer scientists of the modern computing era, his work spans the foundations of large-scale distributed systems, deep learning frameworks like TensorFlow, and today’s frontier AI research. The conversation covers practical insights into where AI capabilities are heading and the infrastructure challenges that come with scaling intelligence systems.

📰 3 Curated Reads

The bug that taught me more about PyTorch than years of using it

https://elanapearl.github.io/blog/2025/the-bug-that-taught-me-pytorch/

Elana Simon shares a debugging story that reveals deep insights into PyTorch’s execution model. The post walks through discovering a subtle bug that exposed fundamental misunderstandings about how PyTorch handles computations. You’ll learn practical lessons about tensor operations, gradient computation, and memory management that textbooks rarely cover. The kind of hard-won knowledge that only comes from fighting with production code.

My Engineering Workflow in CursorAI

https://codeaholicguy.com/2025/10/18/my-engineering-workflow-in-cursorai/

Hoang Nguyen details his practical workflow for engineering with CursorAI. The post covers specific patterns for getting AI assistance to actually accelerate development rather than creating more cleanup work. You’ll find concrete examples of prompting strategies, context management techniques, and workflow organization that work in real projects. Useful if you’re trying to move beyond basic autocomplete to genuine AI-augmented development.

Group Relative Policy Optimization (GRPO)

Deep (Learning) Focus

Group Relative Policy Optimization (GRPO)

Reinforcement learning (RL) has always played a pivotal role in research on large language models (LLMs), beginning with its use for aligning LLMs to human preferences. More recently, researchers have heavily focused on using RL training to improve LLM reasoning performance. This line of research has led to a rapid expansion of LLM capabil…

Read more

5 months ago · 55 likes · 4 comments · Cameron R. Wolfe, Ph.D.

breaks down Group Relative Policy Optimization, a reinforcement learning algorithm that’s become important for training modern LLMs. The post explains how GRPO differs from PPO and DPO, why it matters for alignment, and the mathematical foundations that make it work. Essential reading for understanding the RL techniques behind recent model improvements, with clear explanations that bridge theory and practice.

🛠 2 Tools & Repos

LLM Council

https://github.com/karpathy/llm-council

Andrej Karpathy built this system for querying multiple LLMs simultaneously, having them review each other’s work, then producing a final response through a Chairman LLM. Instead of asking one model, you send queries to your council (GPT, Claude, Gemini, Grok) via OpenRouter. The models provide individual responses, review each other’s work with anonymized identities to prevent favoritism, and then a designated Chairman compiles the final answer. The system provides transparency into how different models approach the same problem and enables systematic model evaluation through peer review. Karpathy describes it as a “Saturday vibe code project” and notes he won’t support it, but the code is there for exploration and modification.

Agentic Paper Reviewer

https://paperreview.ai/

Andrew Ng’s automated system for reviewing research papers. The tool provides structured feedback on paper quality, methodology, and contributions using agentic AI workflows. Designed to help researchers get constructive feedback on their work before submission, particularly useful for identifying potential weaknesses and improvement areas. The system combines domain knowledge with systematic evaluation frameworks to provide more than superficial summaries.

🎓 1 Pick of the Week

Olmo 3 From Scratch

https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/13_olmo3/standalone-olmo3.ipynb

Sebastian Raschka implemented Olmo 3 from scratch in a standalone notebook, providing the clearest way to understand the architecture at a glance. The notebook covers both 7B and 32B models with detailed comparisons to other architectures like Qwen3. Olmo 3 is interesting because it represents the leading fully open-source model with complete transparency about training data, code, and methodology. The notebook includes practical details about attention mechanisms, normalization strategies, and architectural choices that distinguish Olmo 3 from other recent models.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: Single-image 3D reconstruction that handles real-world clutter, self-evolving agents that need zero external data, cost-efficient model families from NVIDIA, and formal proofs of LLM scaling limits

• 🎥 Videos: Mapping open model progress, practical AI coding workflows, and surprising advances in small reasoning models

• 📰 Reads: How OpenAI is using evals to drive enterprise AI, Stanford’s CS336 study notes, and insights on policy distillation

• 🛠 Tools: Automated paper-to-agent systems and comprehensive ML systems resources

• 🎓 Learning: Google’s Nano Banana Pro (Gemini 3 Pro Image), the studio-quality upgrade to their viral image model

Grab my first book — AI for the Rest of Us — today!

Quick note: If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Order AI for the Rest of Us Today!

📄 5 Papers

SAM 3D: 3Dfy Anything in Images

https://arxiv.org/abs/2511.16624 | GitHub

Meta’s SAM 3D reconstructs full 3D geometry, texture, and layout from single images, focusing on real-world scenarios where objects are partially occluded or surrounded by clutter. The breakthrough comes from a human-and-model-in-the-loop pipeline that generated visually grounded 3D reconstruction data at unprecedented scale, combined with a multi-stage training framework that moves from synthetic pretraining to real-world alignment. Unlike methods that fail when objects are partially hidden, SAM 3D achieves at least a 5:1 win rate in human preference tests on real-world objects and scenes.

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning

https://arxiv.org/abs/2511.16043 | GitHub

Training AI agents typically requires massive amounts of human-curated data, limiting both scalability and capabilities to what humans already know. Agent0 breaks this dependency through symbiotic competition between two agents: a curriculum agent that proposes increasingly complex tasks, and an executor agent that learns to solve them using external tools. The key insight is giving agents access to tools like Python interpreters, which provides problem-solving capabilities beyond the base model. This self-reinforcing cycle improved Qwen3-8B-Base by 18% on mathematical reasoning and 24% on general reasoning benchmarks, entirely without external training data.

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs

https://arxiv.org/abs/2511.16664

Training a family of language models at different scales typically requires separate training runs for each size, burning through hundreds of billions of tokens per model. NVIDIA’s Nemotron Elastic embeds multiple nested submodels within a single parent model, where each smaller version shares weights with the parent and can be extracted zero-shot at deployment. Applied to the Nemotron Nano V2 12B model, this approach simultaneously produced 9B and 6B models using only 110B training tokens. That’s over 360x cost reduction compared to training from scratch, and around 7x better than existing compression techniques, while each nested model performs on par or better than state-of-the-art models at their respective sizes.

On the Fundamental Limits of LLMs at Scale

https://arxiv.org/abs/2511.12869

Large language models have improved dramatically with scale, but these gains face five fundamental limitations: hallucination, context compression, reasoning degradation, retrieval fragility, and multimodal misalignment. This paper provides a rigorous theoretical framework showing these aren’t engineering problems to be solved through more compute. Computability theory guarantees that for any model family, diagonalization ensures inputs where models must fail. Finite description length enforces irreducible generalization error. Long context windows compress far below their nominal size due to positional under-training and attention mechanics. The work establishes mathematical boundaries for where scaling helps, where it saturates, and where it provably cannot progress.

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

https://arxiv.org/abs/2511.08544 | GitHub

Self-supervised learning through Joint-Embedding Predictive Architectures has relied on brittle heuristics like stop-gradients and teacher-student networks to prevent collapse. LeJEPA replaces these with a comprehensive theory proving that isotropic Gaussian distributions minimize downstream prediction risk. The framework introduces Sketched Isotropic Gaussian Regularization (SIGReg) to constrain embeddings to this optimal distribution, requiring only a single trade-off hyperparameter and linear time/memory complexity. LeJEPA works out of the box across ResNets, ViTs, and ConvNets, achieving 79% on ImageNet-1k with a frozen ViT-H/14 backbone. The approach requires approximately 50 lines of code and eliminates the need for hyperparameter schedulers.

🎥 4 Videos

Mapping the Open Model Landscape in 2025

Nathan Lambert provides a comprehensive overview of where open-source AI models stand as we head into 2025. The discussion covers which models are genuinely competitive with proprietary options, where the gaps remain, and how the landscape has shifted over the past year. Particularly useful for understanding which open models to consider for specific use cases and what trade-offs you’re making when choosing between open and closed options.

Coding with AI

Chip Huyen walks through practical workflows for coding with AI assistance, moving beyond theoretical discussions to show how experienced engineers actually integrate AI into their development process. The video addresses when AI coding tools help versus when they get in the way, covering context management, code review workflows, and strategies for maintaining code quality when working with AI-generated suggestions.

Learn the basics of Google Antigravity

An exploration of Google’s experimental antigravity feature, demonstrating physics-based interactions and unconventional UI approaches. While playful in presentation, the video showcases interesting approaches to interactive design that challenge conventional interface patterns.

The Weirdly Small AI That Cracks Reasoning Puzzles [HRM]

Jia-Bin Huang examines surprisingly capable small models that solve complex reasoning tasks, challenging assumptions about the relationship between model size and reasoning capability. The video explores what makes these compact models effective and the implications for deploying reasoning systems with limited computational resources.

📰 3 Curated Reads

How evals drive the next chapter in AI for businesses

https://openai.com/index/evals-drive-next-chapter-of-ai/

OpenAI explains how systematic evaluation frameworks are becoming central to deploying AI in enterprise settings. The piece covers moving beyond anecdotal testing to rigorous evaluation systems that catch failures before production, measure performance across edge cases, and provide confidence for business-critical applications. Essential reading for anyone moving AI from prototype to production.

Study Notes: Stanford CS336 Language Modeling from Scratch

https://bearbearyu1223.github.io/cs336/2025/07/20/cs336-note-get-started.html

Han Yu shares detailed study notes from Stanford’s CS336, which covers building language models from first principles. The notes distill key concepts and implementation details that often get glossed over in high-level overviews. Useful for practitioners who want to understand what’s actually happening under the hood when training language models.

On Policy Distillation

https://thinkingmachines.ai/blog/on-policy-distillation/

Thinking Machines Lab explores policy distillation techniques for transferring capabilities from larger teacher models to smaller student models. The analysis covers when distillation works well, where it breaks down, and practical considerations for applying these techniques in production systems where you need efficient inference without sacrificing too much capability.

🛠 2 Tools & Repos

Paper2Agent

https://github.com/jmiao24/Paper2Agent

Automates the process of converting research papers into functional AI agents. Instead of manually implementing methods described in papers, this tool extracts key algorithmic details and generates working code. While it won’t replace careful human implementation for production systems, it accelerates prototyping and helps you quickly test whether a paper’s approach applies to your problem.

Machine Learning Systems Book

https://github.com/harvard-edge/cs249r_book

Harvard’s comprehensive resource on machine learning systems, covering everything from training infrastructure to deployment considerations. The book addresses practical engineering challenges that papers typically ignore: how to actually build, scale, and maintain ML systems in production. Valuable for anyone moving beyond toy datasets to real-world ML engineering.

🎓 1 Pick of the Week

Nano Banana Pro (Gemini 3 Pro Image)

https://deepmind.google/models/gemini-image/pro/

Google released Nano Banana Pro, their latest image generation model built on Gemini 3 Pro. While the original Nano Banana (Gemini 2.5 Flash Image) from August made waves with character consistency, Nano Banana Pro takes things further with studio-quality precision and advanced text rendering. The model can generate clear, legible text directly in images, create detailed infographics using Gemini’s real-world knowledge, and handle complex compositions at 2K and 4K resolutions. The upgrade addresses persistent pain points in professional creative workflows where text needed to be crisp, diagrams needed to be accurate, and outputs needed to meet production quality standards.

My oh my, are the results BANANAS!

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

By the way, you may have noticed that there wasn’t a newsletter in your inbox last week. That’s because I was away giving a talk on Vision Language Models at the Google Developer Group’s DevFest 2025 conference at IIT Madras.

I had a ton of fun answering questions and got to meet so many of the Gradient Ascent community in person. I hope to catch more of you in person in future talks and events.

New here?

The Tokenizer is my resource-focused newsletter edition where I curate the best papers, videos, articles, tools, and learning resources from across the AI landscape. Consider it your weekly dose of everything you need to stay ahead in machine learning.

TL;DR

What caught my attention this week:

• 📄 Papers: Adversarial distillation reaching proprietary model performance, visual geometry models setting new benchmarks, and efficient latent-space upscaling for diffusion models

• 🎥 Videos: Physics-based rendering breakthroughs, transformer architecture deep dives, and practical approaches to ML library testing

• 📰 Reads: Fei-Fei Li on AI’s evolution from words to worlds, scaling laws for reinforcement learning, and accessible PPO explanations

• 🛠 Tools: Google’s comprehensive agent starter kit and scalable diffusion language modeling

• 🎓 Learning: HuggingFace’s complete playbook for training small language models from scratch

Grab my first book — AI for the Rest of Us — today!

It’s been a whirlwind two weeks since the book’s launch and so many of you have sent me pictures of the book in your hands. Thank you so much!!

In fact, the book hit #2 on Amazon’s new releases in the software books category.

If you find the book useful, please leave a review on Amazon. It makes a world of difference. If you have a picture of the book IRL, please share it with me. I really appreciate it.

Order AI for the Rest of Us Today!

📄 5 Papers

Black-Box On-Policy Distillation of Large Language Models

https://arxiv.org/abs/2511.10643 | GitHub

Distilling knowledge from proprietary models usually means settling for worse performance because you only get text outputs, not internal logits. Microsoft’s Generative Adversarial Distillation (GAD) changes this by training a discriminator to distinguish student responses from teacher outputs, creating a minimax game where the student learns to fool the discriminator. The discriminator acts as an on-policy reward model that evolves alongside the student, providing stable feedback throughout training. Qwen2.5-14B-Instruct trained with GAD matches its teacher GPT-5-Chat on LMSYS-Chat evaluation, proving you can reach proprietary model performance through black-box distillation alone.

Depth Anything 3: Recovering the Visual Space from Any Views

https://arxiv.org/abs/2511.10647 | GitHub

Depth Anything 3 achieves state-of-the-art visual geometry from arbitrary viewpoints using surprisingly minimal architecture. A vanilla DINO encoder handles everything without specialized modifications, and a single depth-ray prediction target eliminates complex multi-task learning. The model surpasses prior SOTA VGGT by an average of 44.3% in camera pose accuracy and 25.1% in geometric accuracy while outperforming Depth Anything 2 in monocular depth estimation. Training exclusively on public academic datasets, DA3 demonstrates that architectural simplicity paired with smart training paradigms outperforms specialized complexity for visual geometry tasks.

One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models

https://arxiv.org/abs/2511.10629

Diffusion models struggle with high-resolution generation because direct sampling is slow and post-hoc upscaling introduces artifacts. The Latent Upscaler Adapter (LUA) performs super-resolution directly on latent codes before VAE decoding, integrating as a drop-in component requiring zero modifications to base models. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports both 2x and 4x upscaling, adding only 0.42 seconds for 1024px generation from 512px compared to 1.87 seconds for pixel-space super-resolution. LUA generalizes across different VAE latent spaces without retraining, making deployment straightforward regardless of which decoder you’re using.

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

https://arxiv.org/abs/2510.19338

The Ring-linear model series combines linear attention and softmax attention to slash long-context inference costs. Ring-mini-linear-2.0 (16B parameters, 957M activations) and Ring-flash-linear-2.0 (104B parameters, 6.1B activations) reduce inference cost to 1/10 compared to 32B dense models. Through systematic exploration of attention mechanism ratios, they identified optimal hybrid architecture structures that maintain SOTA performance across challenging reasoning benchmarks while leveraging their FP8 operator library for 50% training efficiency gains. The models undergo stable long-term optimization during reinforcement learning phases due to high alignment between training and inference operators.

GigaBrain-0: A World Model-Powered Vision-Language-Action Model

https://arxiv.org/abs/2510.19430 | GitHub

Physical robot data collection bottlenecks VLA model development, so GigaBrain-0 uses world models to generate diverse training data at scale through video generation, real2real transfer, human transfer, view transfer, and sim2real techniques. This dramatically reduces reliance on expensive real robot data while improving cross-task generalization. RGBD input modeling and embodied Chain-of-Thought supervision enable the model to reason about spatial geometry, object states, and long-horizon dependencies during execution. GigaBrain-0 achieves superior generalization across appearance variations, object placements, and camera viewpoints for dexterous, long-horizon, and mobile manipulation tasks, with GigaBrain-0-Small optimized for NVIDIA Jetson AGX Orin deployment.

You’ll Never Look At Chocolate TV Ads The Same Way Again

Károly from Two Minute Papers explores physics-based rendering advances that make digital chocolate indistinguishable from the real thing. The techniques behind realistic material simulation and light transport have commercial applications beyond advertising, affecting everything from product visualization to virtual production. Worth watching to understand how far physically-based rendering has come and where computational photography is heading.

LLM Building Blocks & Transformer Alternatives

Sebastian Raschka breaks down the fundamental components that make language models work, then examines emerging alternatives to standard transformer architectures. His systematic approach covers why certain architectural choices matter for training efficiency, inference speed, and model capabilities. Useful for anyone building models or evaluating which architectures suit specific use cases.

Designing Tests for ML Libraries

Testing machine learning libraries requires different strategies than traditional software because of probabilistic outputs and numerical precision issues. This practical guide covers approaches for ensuring your ML code behaves correctly across edge cases, different hardware configurations, and varying input distributions. Essential viewing for anyone maintaining production ML infrastructure or contributing to open-source frameworks.

Recursive Language Models

This exploration of recursive structures in language models examines how models can learn to call themselves iteratively to solve complex reasoning tasks. The approach offers alternatives to chain-of-thought prompting by building recursion directly into model architecture and training. Helpful for understanding emerging approaches to multi-step reasoning beyond standard autoregressive generation.

📰 3 Curated Reads

From Words to Worlds

Dr. Fei-Fei Li

From Words to Worlds: Spatial Intelligence is AI’s Next Frontier

In 1950, when computing was little more than automated arithmetic and simple logic, Alan Turing asked a question that still reverberates today: can machines think? It took remarkable imagination to see what he saw: that intelligence might someday be built rather than born. That insight later launched a relentless scientific quest called Artificial Intel…

Read more

5 months ago · 873 likes · Fei-Fei Li

Fei-Fei Li contends that spatial intelligence, which is the ability for AI to perceive, reason about, and interact with the 3D world, is the pivotal next step for artificial intelligence. Her essay highlights the need for world models that can build, simulate, and understand consistent environments, expanding AI’s impact across creativity, robotics, science, and human-centered applications.

How to Scale RL

Interconnects

How to scale RL

Two quick housekeeping items before I get to the post…

Read more

6 months ago · 79 likes · 3 comments · Nathan Lambert

examines emerging scaling laws for reinforcement learning, challenging assumptions that RL doesn’t scale like supervised learning. The analysis covers what factors actually drive RL performance improvements, from compute allocation to data diversity. Critical reading as RL becomes more central to post-training for language models and robotics applications.

PPO for LLMs: A Guide for Normal People

Deep (Learning) Focus

PPO for LLMs: A Guide for Normal People

Over the last several years, reinforcement learning (RL) has been one of the most impactful areas of research for large language models (LLMs). Early research used RL to align LLMs to human preferences, and this initial work on applying RL to LLMs relied almost exclusively on Proximal Policy Optimization (PPO) [1]. This choice led PPO to…

Read more

6 months ago · 100 likes · 4 comments · Cameron R. Wolfe, Ph.D.

demystifies Proximal Policy Optimization for language model training without assuming you have a PhD in reinforcement learning. His breakdown covers why PPO became the standard for RLHF, what the core mechanisms actually do, and how to think about the algorithm’s behavior during training. An accessible guide that translates complex ideas into clear, actionable insights and showing how PPO underpins contemporary LLM alignment practices.

🛠 2 Tools & Repos

Agent Starter Pack

https://github.com/GoogleCloudPlatform/agent-starter-pack

Google Cloud’s comprehensive toolkit for building production AI agents covers everything from basic scaffolding to deployment patterns. Instead of starting from scratch, you get tested implementations of common agent workflows, integration templates for external tools, and best practices for managing agent state. The repository focuses on patterns that work reliably in production rather than experimental approaches.

Simple Diffusion Language Modeling

https://github.com/ZHZisZZ/dllm

This implementation demonstrates diffusion models applied to language generation, offering an alternative to standard autoregressive approaches. The clean codebase makes it easier to understand how diffusion processes work for discrete tokens and experiment with different training strategies. Useful for researchers exploring non-autoregressive generation methods or looking to understand diffusion model fundamentals.

🎓 1 Pick of the Week

The Smol Training Playbook

https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

HuggingFace’s complete guide to training small language models efficiently covers everything from data preparation through final evaluation. The playbook synthesizes lessons learned from training the SmolLM series, including which architectural choices matter most at smaller scales, how to optimize training compute, and which evaluation benchmarks actually predict downstream performance. Particularly valuable if you’re training models under 2B parameters where established practices for larger models don’t always apply. The guide includes specific hyperparameter recommendations, dataset mixing strategies, and ablation studies showing what works versus what doesn’t at small scale.

Thanks for reading The Tokenizer! If you found something useful here, share it with someone who might benefit. And if you want more curated insights like this, consider subscribing to Gradient Ascent.

After three years, my first book, AI for the Rest of Us, with Bloomsbury, is finally out in the world.

Why This Book Exists

I wrote it because too many AI conversations got stuck in translation.

Executives talked strategy. Engineers talked code. Everyone else nodded along, trying to make sense of both.

So I built a bridge.

Who This Book Is For

This book is for the people in between. The rest of us.

• Product managers, leaders, decision makers, and domain experts who need to understand how AI actually works to make better decisions

• Engineers who are tired of explaining why AI isn’t magic to their bosses and teams

• Curious readers who want to see the big picture clearly

What Makes It Different

No code. No equations. No jargon.

Just hand-drawn sketches, stories, and metaphors that make the complex intuitive.

It’s been a long road of rewrites, feedback, and sketches that taught me how to explain ideas simply. I’m deeply grateful to everyone who believed in this mission from the start.

How You Can Help

Here’s how you can help me spread AI literacy:

→ Order a copy for yourself or your team
→ Share this with someone who needs it
→ Tell me what part helps you the most

Every share, every reader, every conversation helps make AI literacy accessible to more people.

Let’s make sure understanding AI isn’t a privilege. Let’s bring everyone along.

It’s time to lead.

Get Your Copy

• Amazon: https://www.amazon.com/AI-Rest-Us-Illustrated-Introduction/dp/B0F29THNLT

• Barnes & Noble: https://www.barnesandnoble.com/w/ai-for-the-rest-of-us-sairam-sundaresan/1147236404

• Bloomsbury: https://www.bloomsbury.com/us/ai-for-the-rest-of-us-9798881807955/

  ---

Preorder Winners!

And now, the fun part: I’m thrilled to announce the winners of the pre-order bonuses!

• Signed Copy Winners: Gary Marchant, Harishkumar Singh, Pandu Ranganath, Michael Machado, Om Bhatt, Gilles Everling, Aman Deep, David Poore Turner, Travis Bradford, and Daniel Janecek

• 1:1 Session Winner: Cynthia Machado

Congratulations to all, and I’ll be in touch directly with next steps.

If you’ve ever wanted to cut through the jargon and finally feel confident talking about AI, I hope this book gives you exactly that.

Order your copy here:

Grab your copy today!

It’s time to lead!