More and more Polish companies are asking about „private LLM”: a language model built in their own infrastructure, to which data does not leave your server room. There are several reasons: GDPR in regulated industries, the risk of sensitive data leaking to the public cloud, rising API bills with scale and the desire to become independent from one supplier. But a private LLM is not always a good decision and not always cheaper.
In this article, we show how a private LLM differs from a public one, when it is worth putting it on-premise, what the real costs are (calculator below), which open source models are at the GPT-4o level today and how to design a secure architecture. This is an extension of the topic we started in the article AI Automation with ChatGPT? 6 alternatives for companies in 2026.
What is Private LLM
Private LLM (self-hosted LLM) is a language model that runs on infrastructure controlled by your company, not on the servers of providers such as OpenAI, Anthropic or Google. Key difference: The data sent to the model and the responses it generates never leave the boundaries of your network.
In practice, „private LLM” appears in four variants:
- On-premise, i.e. your own server with GPU located in your server room or colocation. Full control, no transfer outside the building.
- VPC in the public cloud (AWS, Azure, GCP), where the model runs in a separate virtual network to which you have exclusive access. Physically in the cloud, logically „at your place”.
- Hosting in the Polish cloud (OVH, Beyond.pl, Netia, Atman), where the GPU server is located in a Polish data center and is subject to Polish jurisdiction.
- Hybrid, where the private LLM handles sensitive data and the public API (GPT-4o, Claude) handles everything else. Two models, one orchestration layer.
In each of these variants, the engine is usually an open source model whose scales you can download and run. Most common: Llama 3.3 (Meta), Mistral Large 2 (Mistral AI), Qwen 2.5 (Alibaba) or DeepSeek-V3.
Five signs that a public LLM is not enough for you
A private LLM is a big investment, so before we propose it to a client, we check whether the public model really can't handle it. Here are five signals that in practice tip the decision towards on-premise:
- Compliance prohibits. The medical sector (personal data of patients), legal (professional secrecy), defense (secrecy clauses), financial (GDPR + banking secrecy), public (KRI, sectoral laws) often exclude the external cloud in internal regulations. We have developed this topic in the study AI for lawyers: 5 pitfalls.
- The client requires in the contract. In public tenders and large B2B contracts, the clause „data cannot leave Poland/EU borders” or „processing only on the contractor's infrastructure” is increasingly included.
- The API bill increases linearly with usage. When the monthly cost of GPT-4o exceeds PLN 20-30 thousand, the mathematics begins to favor your own infrastructure.
- You need fine-tuning on sensitive data. Training the model on company documents in the public API means uploading these documents to the supplier. The private model solves this problem.
- Critical processes cannot depend on an external API. Public APIs have downtime (OpenAI several times in 2024 and 2025), change prices (GPT-4 became 10x cheaper, then became more expensive in Enterprise), change models (deprecation). On-premise = full control over the version.
Which open source models can be built locally
In 2026, open source LLMs have caught up with commercial models in most tasks. Below are models that are actually worth considering for production implementation.
| Model | Parameters | Context | Polish | When to choose |
|---|---|---|---|---|
| Llama 3.3 | 70B | 128k | very good | universal choice, largest community, tons of ecosystem tools |
| Mistral Large 2 Mistral AI | 123B | 128k | Good | European supplier, good quality reasoning, commercial license required |
| Qwen 2.5 Alibaba | 72B, 32B | 128k | mediocre | best in class in terms of quality, but geopolitical risk (China) |
| DeepSeek-V3 DeepSeek | 671B (MoE, 37B active) | 128k | mediocre | the most cost-efficient in its price class, needs more memory than a standard 70B |
| Phi-4 Microsoft | 14B | 16k | weak | when you need a small model for specific tasks, runs on a single A100 |
| Bielik SpeakLeash + Cyfronet | 11B | 32k | wybitny (polski) | a Polish adaptation of Mistral, the best open-source model for Polish-language quality |
Model choice directly affects hardware requirements. You can't run Llama 3.3 70B on a laptop, just as it makes no sense to set up a rack server with 8x H100 for a 14B model. The calculator below shows what you actually need.
GPU picker: which hardware for which model
Dobierak GPU pod prywatny LLM
Choose a model and operating mode. See the required VRAM, the suggested GPU setup and the estimated monthly cost in a Polish cloud.
A few observations from practice. A batch size greater than 1 is a must if the model has to serve many users at once, but every additional parallel session eats VRAM for the KV cache. INT8 quantization is the production standard (quality practically indistinguishable from FP16 for most tasks); we use INT4 only in very limited quality scenarios. MoE models (DeepSeek) have only ~10% of their parameters active at any given moment, but all the weights still have to be kept in VRAM.
Kalkulator TCO: on-premise vs API publiczne
The basic question: when does on-premise pay off? The answer depends on volume. The calculator below shows the break-even point over 3 years (a typical hardware depreciation horizon). The calculator assumes Llama 3.3 70B on 1x A100 80GB in a Polish cloud (PLN 7,500/month) vs the GPT-4o API at September 2026 prices.
TCO 3 lata: prywatny LLM vs publiczne API
The calculator leaves out one important thing: the cost of migration. Setting up a private LLM is typically a one-time project of PLN 60,000-120,000 (infrastructure engineering, RAG layer, integrations, testing, monitoring). Only after that does the TCO above kick in.
Fine-tuning or RAG: what to choose for a private LLM
Once you decide on a private LLM, a second question comes up: do you tune the model on your data (fine-tuning), or feed it the data at query time (RAG)? In practice we almost always start with RAG, which we wrote about in What RAG is and why AI makes things up without it.
- RAG (Retrieval Augmented Generation): the model receives the data in the prompt at query time. Pros: data is always up to date (edit the source and the change is immediate), easy to add documents, no training. Cost: a few thousand PLN for vector infrastructure.
- Fine-tuning : you train the model on your documents and it memorizes them “permanently”. Pros: shorter prompts (no need to append context), better performance in a narrow domain. Cost: PLN 20,000-80,000 one-time per project, plus the need to retrain when the data changes.
Rule of thumb: fine-tuning makes sense when the data is stable (doesn't change every week) and narrow in scope (technical documentation, law in a single field). Everything else is RAG.
Private LLM security: 4 layers
A private LLM doesn't guarantee security on its own; it only guarantees that data doesn't go out to the model vendor. All the other safeguards have to be designed:
- Network. The model is accessible only from the corporate network (VPN, private subnet). Never a public API without authorization.
- Autoryzacja. Every request is tokenized and tied to a user, with access levels for different data sources (not everyone sees all documents).
- Logs and audit. Every request and response is logged with a timestamp and user. A compliance requirement in most regulated industries.
- Retention and encryption. Logs encrypted at rest, a retention policy (typically 90-365 days), and automatic deletion of personal data from logs after a defined period.
When a private LLM does NOT make sense
A private LLM is a buzzword, but not always the right call. Signs it's worth sticking with a public API:
- Volume is low (under 1,000 requests per day). The TCO calculator shows that in that case a public API is dozens of times cheaper.
- You don't have a DevOps/MLOps team. A private LLM needs ongoing care: model updates, monitoring, incident response. Without a team to maintain it, the project falls apart within 6 months.
- The data isn't sensitive. If compliance allows public APIs and volume is moderate, a private LLM is overengineering.
- You need the absolute highest quality. In top benchmarks, GPT-4o, Claude 4.7 and Gemini 2.5 Pro still lead open source by a few percent. For some use cases that matters.
At many companies the most sensible option is the hybrid scenario we described in the article AI deployments: why they fail. A public model for 95% of tasks, a private LLM only for those that actually require it.
Three typical deployment architectures we build
From the deployments we've delivered as part of AI implementations for clients in Poland, three recurring architectures emerge:
- Single node in a Polish data center. A single server with 1-2x A100 at OVH or Beyond.pl. Llama 3.3 70B or Bielik model. RAG on PostgreSQL + pgvector. Handles 5,000-20,000 requests per day. Good for companies of 50-500 people with one or two AI processes.
- Cluster on-premise. 4-8x A100/H100 in your own server room. Load balancer, two model replicas, high availability. Handles 100k+ queries per day. For large organizations with mission-critical AI processes (finance, healthcare).
- Hybryda API + prywatny LLM. An orchestration layer (n8n or a custom backend) routes the queries. Data flagged as “sensitive” goes to the private LLM, everything else to GPT-4o or Claude. This architecture is increasingly common at companies that previously chose API-only.
Want to check whether a private LLM pays off for you?
We audit your volume, compliance requirements and budget. The result is a one-page recommendation: private LLM, public API or hybrid, with concrete 3-year TCO figures.
Schedule an AI deployment auditFrequently Asked Questions
Is a private LLM GDPR compliant?
The fact that the model runs on your side is one of the strongest arguments for GDPR compliance, because the data physically never leaves your infrastructure. But GDPR is a broader topic: you need a legal basis for processing, a data protection impact assessment (DPIA), log retention policies and a mechanism for honoring data subject rights. A private LLM makes compliance easier, but it does not replace it.
Which open source LLM is best for Polish?
The best open source model for Polish in terms of quality is Bielik from SpeakLeash and Cyfronet AGH, despite its small size (11B). Among 70B-class models, Llama 3.3 performs best, but the gap versus GPT-4o in Polish is still noticeable (a 5-10% lower score in our tests). Mistral Large 2 is right behind Llama, Qwen 2.5 further back, and DeepSeek last for Polish (good at code and English).
How many GPUs do I need to run Llama 3.3 70B?
With INT8 quantization, a single A100 80GB (or H100) is enough. The model then weighs about 70 GB, plus KV cache for several parallel queries and a system buffer. For a larger batch or full FP16 precision you need 2x A100 80GB. The calculator above shows this for your parameters.
How long does it take to deploy a private LLM in a company?
Simply installing a model on a GPU server takes 2-3 days. But a production deployment with a RAG layer, integrations with existing tools, authorization, logging and monitoring typically takes 6-10 weeks. The key cost is not the model itself, but everything around it. Our typical project ends with deployment 8 weeks after kick-off.
Is it better to run a private LLM in a Polish cloud or in your own server room?
For most companies, renting a GPU server in a Polish data center (OVH, Beyond.pl, Netia) makes more sense. Downsides of your own server room: upfront capital cost (PLN 100,000-500,000 for hardware), risk of failure, the need for cooling and backup power, and the hardware depreciation cycle (GPUs age every 2 years). Advantages of a Polish cloud: no CapEx, flexibility to change the configuration, vendor support. Your own server room makes sense for organizations that already have server infrastructure in place (banks, telcos, public administration).
Is a private LLM slower than ChatGPT?
It depends on the hardware. On a well-chosen GPU (A100 or H100), a private Llama 3.3 70B generates responses at 30-50 tokens per second, which is comparable to ChatGPT. On weaker hardware (RTX 4090 with INT4) it is more like 15-25 tokens/s, still acceptable for most use cases, but slower than public APIs. Batch processing on good hardware can handle 500+ tokens/s of total throughput.
Can I change the model in a private LLM after deployment?
Yes, and that is one of the main advantages of your own infrastructure. The application layer (RAG, API, integrations) is independent of the specific model, so swapping Llama 3.3 for Mistral Large 2 or Qwen 2.5 usually takes 1-2 days of work. For comparison: migrating from ChatGPT to Claude requires rewriting most of the prompts and call logic.
