LLM System Design Interview, issue 68, Sep 12, 2026
The Web Crawler Throughput Trap
Senior Data Infrastructure Engineer interview at Anthropic, and the interviewer asks:
“Your crawler is hitting record pages-per-second. A week later a site owner is publicly complaining you hammered their servers a million times in 24 hours, and your corpus is 40% near-duplicates. Which crawl-policy decisions did you get wrong?”
Don’t say: “We should add rate limiting and a dedup step.”
Why parallelizing by URL quietly hammers third-party hosts while bloating your pre-training corpus, and the two queue-level metrics staff infra engineers optimize instead.
The full answer, with the mechanism and the arithmetic, is free on Substack.