At ArcFusion, we believe in learning out loud — sharing what we build, how we build it, and what we learn along the way. This post is adapted from one of our internal engineering knowledge-sharing sessions.
The Growing Demand for Local LLMs
Working with enterprise clients across Southeast Asia, one question keeps coming up in our conversations: "Can we host the LLM on our own servers?"
It's a fair question. Organizations sitting on sensitive intellectual property, handling PII under strict compliance regimes, or simply wanting full control over their AI infrastructure naturally gravitate toward on-premise solutions. The motivations generally fall into three buckets:
- Data sovereignty — clients with trade secrets, proprietary data, or IP they want to keep entirely within their walls.
- Compliance — organizations handling PII that must ensure data never leaves their environment, particularly in regulated industries.
- Control — wanting to own the infrastructure end-to-end, including the freedom from content restrictions or filters that come with hosted APIs.
These are legitimate concerns. But before committing to a seven-figure infrastructure build, it's worth pressure-testing whether on-premise is actually the right answer.
But First: Do You Actually Need On-Premise?
All three major providers — Google Gemini, OpenAI, and Anthropic — offer enterprise tiers that explicitly guarantee no data retention and no training on customer data. They comply with SOC 2, HIPAA, and BAA requirements.
In many client conversations, sharing these enterprise guarantees resolves the concern entirely. But not always. Some industries — particularly banking and fintech in Indonesia — operate under regulatory frameworks that demand more certainty. And some clients, regardless of documentation, simply don't trust that their data won't be used. That's a legitimate business concern, even when the technical guarantees exist.
So the question becomes: if enterprise APIs don't satisfy your requirements, what does on-premise actually look like in practice?
The Open Model Landscape: Closing the Gap, But Not There Yet
The open-source model ecosystem has made remarkable progress. As of early 2026, leaderboards from Artificial Analysis show the gap between proprietary and open models narrowing significantly.
The current top open models — GLM5 (744 billion parameters) and Kimi 2.5 (1 trillion parameters) — score competitively on text-based intelligence benchmarks. GLM5 sits just a few points behind the leading proprietary models.
However, there's a critical gap that benchmarks don't always capture: visual and multimodal capabilities. Proprietary models like Gemini 3.1, Claude 4.6, and GPT-5.4 handle screenshot-to-code, OCR, and image understanding with high accuracy. Many open models either lack these capabilities entirely or perform them poorly. For workflows where you might screenshot a UI design and ask the model to generate code, that's a significant limitation — you'd need to describe layouts in ASCII or structured text instead.
Quantization: Making the Impossible Merely Expensive
Running a 744-billion-parameter model at full precision is impractical for nearly any organization. This is where quantization becomes essential.
Think of it like audio compression: WAV files preserve every detail, but MP3s sound nearly identical at a fraction of the size. Similarly, reducing model weights from 16-bit to 4-bit precision cuts VRAM requirements by roughly 75% while degrading accuracy by less than 2% — a difference rarely noticeable in real-world usage.
Even with quantization, though, running top-tier open models like Kimi 2.5 still requires approximately 500 GB of VRAM. That means a multi-GPU setup — typically an 8-node cluster of 80 GB Nvidia H100s — making it a serious investment even at the entry point.
The Real Cost: It's Not Just GPUs
Here's where the conversation gets sobering, particularly for the Indonesian market.
A setup capable of running quantized versions of top-tier open models (8× H100 GPUs) costs $350,000–$450,000 USD in Indonesia — roughly 5.5 to 7 billion IDR. That includes a ~25% tax premium over US pricing.
But that number should realistically be doubled. Why? Model upgrades. When a new model version releases, you can't simply swap it in. You need to run both the old and new versions simultaneously during migration — users depend on the existing model, and you need to validate the new one before cutting over.
That pushes realistic GPU investment alone toward $700,000–$900,000 USD — and that's before counting memory, storage, networking, cooling infrastructure, or the engineering team to maintain it all. Breaking a million dollars total is essentially guaranteed.
The Rental Alternative (and Its Trade-Off)
GPU rental platforms like Vast AI and Runpod offer a compelling alternative on paper. An RTX Pro 6000 with 96 GB VRAM can be rented for $1–2 per hour — transforming a massive capital expenditure into manageable operating costs.
But there's a catch: if your motivation for on-premise was data sovereignty or compliance, renting GPUs in Poland or the US defeats the purpose entirely. Your data is now running on someone else's hardware in another country. At that point, you might as well use enterprise APIs from Google, OpenAI, or Anthropic — you'd get better model performance, no maintenance burden, and equivalent (or better) data handling guarantees.
The Jakarta Factor: Infrastructure Reality
Indonesia presents its own infrastructure challenges. The Cyber Building in Jakarta serves as a data center hub for many enterprises, but it's a traditional facility designed for conventional servers — not the intense cooling demands of AI GPU clusters.
New AI-capable data centers are under construction outside Jakarta, but they're still in the building phase. Meanwhile, both Google Cloud and AWS operate regional cloud centers in Jakarta, and Google is expanding Gemini's presence in the region. For most compliance requirements, hosting on GCP Jakarta already satisfies Indonesian data residency regulations.
When Does On-Premise Actually Make Sense?
After examining the full picture, the honest answer is: rarely, and only at massive scale.
On-premise LLM hosting is justifiable when you're building a consumer-facing application with enormous adoption — think millions of daily users generating constant inference load. At that scale, the fixed cost of infrastructure amortizes across enough usage to become competitive with per-token API pricing over a 3–5 year horizon.
For enterprise internal use cases — document processing, internal chatbots, knowledge management — the math almost never works out. The combination of enterprise API guarantees, regional cloud availability, and the operational complexity of maintaining GPU infrastructure makes hosted solutions the pragmatic choice.
Key Takeaways
The decision framework comes down to two questions:
- Is your data concern addressed by enterprise API guarantees?
- If not, does a regional cloud hyperscaler (like GCP Jakarta) satisfy your regulatory requirements?
Only if both answers are "no" — and you have the budget, the engineering team, and the scale to justify it — should on-premise hosting enter serious consideration.
For most organizations, the answer is clear: leverage enterprise APIs or regional cloud infrastructure. Save the million-dollar GPU investment for when your scale truly demands it.
Have questions about LLM deployment strategies for your organization? The ArcFusion team works with enterprises across Southeast Asia to find the right AI infrastructure approach. Reach out to us at arcfusion.ai.