LLM Inference
Also known as: Inference, Model Serving
LLM inference is the moment a trained language model actually works: it receives a request and generates a response, token by token. Every chat answer is inference.
Training vs. inference
Training is the degree, inference is the day job. Training happens once and costs millions. Inference happens with every single request, millions of times a day. That is why inference drives the running costs of AI operations.
How costs are measured
Providers bill per token, split into input (the request) and output (the response). Output tokens usually cost significantly more. Planning a budget means estimating both: long answers weigh heavier than long prompts.
What determines speed
Three factors decide. First, model size: more parameters mean more computation per token. Second, hardware: GPUs are the standard, while specialised chips such as LPUs accelerate things considerably. Third, batching: processing many requests in parallel raises throughput but can delay individual responses.
Cloud or self-hosted
Hyperscalers such as AWS Bedrock, Azure AI Foundry, and Google Vertex AI host a wide range of models. Specialists such as Groq, Cerebras, Together AI, and Fireworks AI compete on speed and price. The alternative: self-hosted AI on company hardware: no per-token billing, but full operational responsibility.
We can work out the next step together in a free introductory call. Contact →
As of: June 2026
Claude Cheat Sheet
An overview of Claude's features and models, with tips from our daily work. It is available in German and English, and we keep it up to date.
Open the cheat sheetDigital Impact in your inbox
With one sign-up you get three newsletters: the AI Insights Newsletter with our new articles every week, plus the Digital Impact Longread and the Digital Impact Update once a month. Sign-up uses double opt-in, and you can unsubscribe at any time.
AI Operations as a podcast
In the Digital Impact Podcast, André Hellmann, Christina D'Ilio, Christian Sattel and Sarah Stock talk with guests from practice about AI Operations. The episodes are easy to listen to on the go.
How this article was produced
- Topic selection
- Source selection
- Fact-checking
- Approval
- Research
- Drafting
- Diagrams
- Publishing
This article was produced with AI support. Ideation, editorial planning, substantive review and approval rest with a human; copy-editing sits with the AI. Editorial responsibility is held by André Hellmann.