Revised designs from measurements while running an LLM service and an overnight batch platform
PurplePepper — B2B2C reward-advertising startup (current employer)
I own an LLM service that drafts replies to small-business reviews and an unattended overnight batch platform that processes hundreds of thousands of jobs a night, validating each design decision with measurements.
Product and technical challenge
Problem: Across 8 boxes and 64 lanes, the 16GB boxes produced about 40% less than the 24GB boxes, and three lanes repeated the same failure 181 times while every monitor still reported them as healthy.
Approach: Re-tested my own CPU-only lane rule with controlled experiments, replaced it with a formula that also accounts for memory (swap), cleaned up orphaned browsers, and added automatic restarts for lanes that stop reporting.
Result: Cutting lanes 14% raised throughput 13% (reproduced by swapping roles); one night processed 494,126 jobs at a 98.4% success rate.
AI leverage
- Review-reply LLM: dropped an ontology approach after a 44% miss rate on 765 reviews, cut LLM calls from 3 to 1–2, and replaced LLM self-grading with three drafts the owner picks from
- LoRA-tuned Qwen3-4B on 1,883 owner replies for a zero-API-cost option at 92.5% of API quality
- Switched business-license OCR from CLOVA to AWS Bedrock Claude, cutting cost 96%
- Monitoring for 408 crawling servers: automated collection plus LLM summaries, with no tool access for the LLM to contain prompt injection