SF Systems: Research to Practice, from Inference Engines to Multimodal Coding

the vibe

The SF Systems Club is back with a pair of exciting talks from researchers who are bringing cutting-edge science to the real world! Whether you’re a seasoned inference expert or programming languages enthusiast, you will learn a lot at this one. This meetup is generously hosted by LatchBio at their San Francisco office. Food and drinks will be provided! Agenda 5:30 PM: Doors Open, Food and Socializing 6:30 - 7:30: Introductions and Talks 7:30-: Community Socializing! Talks Shreya Shankar (CMU): Domain Specific LLM Inference for AI-SQL Database users have long wanted to query unstructured text, and LLMs have finally made this feasible. Major vendors now support AI-SQL, which adds AI-powered filters, joins, maps, and classifiers to SQL. To execute AI-SQL queries, databases lower these queries into plans with one LLM call per row, or, for joins, per tuple in a cross product. Even after query optimization, hundreds of thousands, or millions, of calls may remain! In theory, inference engines should be fantastic at this workload, because each call is almost entirely prefill and returns only a few tokens. Surprisingly, general-purpose inference engines take far longer than expected, because their KV caches do not know what the query will reuse, and processing so many requests adds high host overhead. We use these limitations to motivate Quail, an open source query engine for AI-SQL that our lab is building. We describe Quail's planner and execution engine, show preliminary exciting…