Making Your Database Perform at Scale: Optimizing for Billions of Rows

Scaling a database to handle billions of rows is a challenge that many developers only recognize once they're already deep into a production crisis — but Lukas Vileikis argues the warning signs appear much earlier. In this session, Lukas explains that performance problems typically don't begin at a billion rows; they begin around 2–3 million, in the early stages of most applications. Understanding this early means you can architect for scale before it becomes an emergency. Lukas walks through the fundamental building blocks of database performance: indexes, partitions, schema design, and query optimization. He emphasizes that optimizing database parameters can yield around 30% performance improvement, but the real gains come from modeling your data first and choosing hardware second — a discipline most teams get backwards. He also debunks the assumption that SSDs or NVMe drives are always necessary, noting that for data that doesn't change frequently, traditional hard drives can still do the job at a fraction of the cost. A large portion of the talk is dedicated to practical patterns for working with massive datasets. Lukas shares how importing raw parsed data instead of inserting row by row dramatically reduces overhead. He discusses when to use LOAD DATA instead of INSERT, how to think about SELECT queries at scale with proper indexing strategies like covering indexes, and how partitioning can make queries faster by limiting the data scanned. Lukas closes with real-world lessons from running BreachDirectory.com — a search engine operating on billions of records — covering backup strategies for large datasets, stress testing before deployment, and maintaining stability under heavy read loads. His session is a practical playbook for any engineer or DBA who needs their database to stay fast as data volume grows.