What actually breaks when you put an LLM in a product

The demo always works. You paste a prompt, the model returns something impressive, and the feature feels finished. The gap between that moment and a shipped product is larger than it looks, and almost none of it is about model quality.

Latency is the first wall. A response that takes six seconds is fine in a terminal and unacceptable in a form. That constraint reshapes the interface: you stream partial output, you show intermediate state, and you design something for the user to look at while they wait.

The second wall is non-determinism. UI code assumes shapes. Models return prose. Everything between them — schema validation, retries, fallbacks when the output does not parse — is unglamorous work that ends up being most of the feature.

The third is error states. What does the screen show when the model refuses, times out, or returns something confidently wrong? Those paths need designs, not just try/catch blocks, and they are the difference between a feature that feels reliable and one that feels like a toy.