A normal database query takes milliseconds. A model call can take seconds, or a minute for images and video. If you treat it like any other request, your product will feel slow and may time out.
Stream what you can
For text, stream the answer as it is written. People start reading after the first words instead of waiting for the last.
Move long jobs off the request
For anything that takes longer than a few seconds, such as video generation, accept the request, return a job id, and do the work in the background. The page then shows progress and picks up the result when it is ready.
Plan for failure
- Set timeouts, and retry only the calls that are safe to repeat.
- If a paid job fails, give the credit back automatically.
- Show a clear message, and keep the user’s input so they can try again.
Watch the cost per request
Log the model, the size of the input and output, and the time taken for every call. A dashboard of those few numbers shows you where money and seconds go.
By Foldox. All posts