Phi-2, LLM Distillation Playbook, ML Interview Guide, and more...

Resources for the final week of 2023

It's a short update this week. Happy Holidays and New Year to you and yours!

Resources To Consider:

ML Systems Design Interview Guide

Link: http://patrickhalina.com/posts/ml-systems-design-interview-guide/

Staff ML Engineer Patrick Halina has put together a wonderful blog post on preparing for ML system design interviews. It's packed with useful tips on how to prepare and navigate these interview rounds. If you're in the market for an ML role, this will be a handy resource for you as you prepare. 

The Surprising Power of Small Models

This is a great talk on Phi-2 from one of the authors of the paper, a small model that matches or outperforms models 25x larger. Sébastian Bubeck discusses the work on Phi-2 and the idea of training smaller models with textbook-quality data. 

Visualizing Matrix Multiplication, Attention & Beyond

Link: https://pytorch.org/blog/inside-the-matrix/

"mm" is a fully interactive 3D visualization tool that runs in the browser. It keeps the complete state in the URL, so links are shareable sessions. Use it to visualize matrix multiplications, attention mechanisms, and more.

LLM Distillation Playbook

Link: https://github.com/predibase/llm_distillation_playbook

The folks from Predibase have created a neat repository that covers LLM distillation best practices. The authors' advice in this repository comes from first-hand experience of distilling models at Google and Predibase, so I'd highly recommend checking this out for your next LLM project.

Scaling Laws for AI

Link: https://txt.cohere.com/scaling-laws-of-ai/

Neil Thompson, the director of MIT's FutureTech Lab, discusses the impact of scaling computing power on AI development. In this interview, he highlights the trend towards specialized chips and the challenges it poses for software portability and discusses the implications of specialization for chip makers and AI model development.