Self-hosted AI
Also known as: Self-hosting, local LLMs, on-premise AI
Self-hosted AI means language models run on a company’s own infrastructure instead of a vendor’s cloud. Requests and data never leave the organisation. The foundation is usually an open source LLM.
Technical requirements
Graphics memory (VRAM) is the deciding factor. Small models with 7–8 billion parameters run compressed from around 8 GB of VRAM, a decent consumer GPU. The 70B class needs roughly 40–48 GB compressed, for example two 24 GB cards. Uncompressed, it takes around 140 GB, which means data centre hardware.
Common tools
Three open-source tools dominate: Ollama for an easy start on single machines, vLLM for high-performance production serving, and Open WebUI as a browser-based chat interface. All three are actively maintained.
Benefits and drawbacks
The benefits: full data sovereignty, no per-token API costs (LLM inference), and adaptability up to fine-tuning. The drawbacks: hardware investment, operational responsibility for updates, security and monitoring, and model upgrades handled in-house.
Who it makes sense for
Self-hosting pays off with sensitive data, high request volumes, or strict compliance requirements, such as data residency in the EU. At low volumes, cloud inference is usually cheaper and faster to launch.
We can work out the next step together in a free introductory call. Contact →
As of: June 2026
Claude Cheat Sheet
An overview of Claude's features and models, with tips from our daily work. It is available in German and English, and we keep it up to date.
Open the cheat sheetDigital Impact in your inbox
With one sign-up you get three newsletters: the AI Insights Newsletter with our new articles every week, plus the Digital Impact Longread and the Digital Impact Update once a month. Sign-up uses double opt-in, and you can unsubscribe at any time.
AI Operations as a podcast
In the Digital Impact Podcast, André Hellmann, Christina D'Ilio, Christian Sattel and Sarah Stock talk with guests from practice about AI Operations. The episodes are easy to listen to on the go.
How this article was produced
- Topic selection
- Source selection
- Fact-checking
- Approval
- Research
- Drafting
- Diagrams
- Publishing
This article was produced with AI support. Ideation, editorial planning, substantive review and approval rest with a human; copy-editing sits with the AI. Editorial responsibility is held by André Hellmann.