Milliseconds Equals Margin
In logistics, a 100ms delay in package classification is not a slower interface — it is a throughput ceiling. At tens of millions of items a day across distributed fulfilment centres, latency compounds into hours of downstream delay, and it does so against some of the thinnest margins in any industry.
The structural fix is to move inference from a central cloud region to the edge, close to where requests originate. Removing a round trip to a distant region is usually worth more than any amount of model optimisation, because the speed of light is the one constraint no amount of tuning negotiates with.
Rust vs. Python: The Inference War
Python is the lingua franca of AI training. It is rich, flexible, and has an ecosystem that is second to none. However, for high-frequency inference, it is a bottleneck. The Global Interpreter Lock (GIL) and dynamic typing overhead make it unsuitable for sub-millisecond SLOs.
Rewriting the tokeniser and embedding lookup layers in a compiled language moves the critical path off the interpreter entirely. The gain comes from rebuilding that path rather than wrapping it — memory safety and zero-cost abstractions let you push the hardware hard without trading away crash safety to do it.
Bare Metal K8s
Virtualisation adds overhead. Where a latency budget is tight enough to notice it, bypassing the hypervisor and deploying directly on bare metal removes the “noisy neighbour” problem and exposes the CPU’s NUMA topology, so memory access patterns can be tuned to the workload. This is a real cost trade, not a free win: bare metal scales in weeks, not seconds.