Building a Private Local LLM Server on Windows with llama.cpp, Vulkan and Qwen
The hardware was not designed as an AI workstation:
CPU: AMD Ryzen 5 7430U
Cores: 6 cores / 12 threads
RAM: 32 GB
GPU: AMD Radeon integrated graphics
GPU backend: Vulkan
OS: Windows
Dedicated VRAM: only around 512 MB
Shared GPU memory: approximately 16 GB
Despite those limitations, the final result was:
Windows Server
│
▼
llama.cpp + Vulkan
│
▼
Multi-model llama-server router
│
├── Qwen3-14B Q5_K_M
│ "fast"
│
└── Qwen3-Coder-30B-A3B Q4_K_M
"coder"
│
▼
OpenAI-compatible API
│
├── Browser UI
├── Python
├── C#
├── agents
└── other machines on the LAN
The server now starts automatically after a reboot and exposes both models through one endpoint.
This post documents the complete process, including the failures along the way.
1. Starting with llmfit
I first used llmfit to work out what models were theoretically capable of running on the machine.
It detected roughly:
CPU: AMD Ryzen 5 7430U
RAM: ~31 GB
GPU: AMD Radeon Graphics
Backend: Vulkan
The important discovery was that although Windows initially reported only around:
0.5 GB VRAM
this was misleading.
The Radeon is an integrated GPU using Unified Memory Architecture (UMA), meaning it can borrow system memory.
After installing llama.cpp, this became much clearer:
llama-cli.exe --list-devices
returned:
Available devices:
Vulkan0: AMD Radeon (TM) Graphics (16345 MiB, 15527 MiB free)
So Vulkan could address approximately 16 GB of shared GPU memory.
That dramatically changed what models were practical.
2. Why llama.cpp instead of Ollama
Ollama is excellent for convenience, but I wanted lower-level control over:
GPU layer offloading
Vulkan
context size
batch size
micro-batch size
KV-cache placement
memory allocation
model quantization
multi-model routing
llama.cpp also provides an OpenAI-compatible API, so applications do not need to be coupled to a llama.cpp-specific protocol.
The resulting architecture is:
GGUF
↓
llama.cpp
↓
llama-server
↓
/v1/chat/completions
This means the same client abstractions can later work with:
OpenAI
llama.cpp
Ollama's OpenAI compatibility endpoint
vLLM
LM Studio
LocalAI
other OpenAI-compatible servers
3. Installing llama.cpp on Windows
There was no need to compile llama.cpp.
I downloaded the official Windows x64 Vulkan build from the llama.cpp releases.
The relevant package is named approximately:
llama-bXXXXX-bin-win-vulkan-x64.zip
I extracted it to:
C:\llama.cpp
The directory contained binaries such as:
C:\llama.cpp\
llama-cli.exe
llama-server.exe
llama-bench.exe
...
Testing the executable:
C:\llama.cpp\llama-cli.exe --version
gave:
version: 0.1.2-dev
build: 10517
commit: dc72703fc
built with Clang 20.1.8 for Windows x86_64
4. The first problem: Windows "Bad Image"
Initially Windows refused to start one of the llama.cpp DLLs with a message similar to:
llama-cli.exe - Bad Image
Error status 0xC0E90002
The failing DLL was:
llama-cli-mtmd.dll
This turned out not to be a Vulkan problem.
Windows security controls were blocking one of the unsigned/open-source binaries.
After allowing the application appropriately, this worked:
C:\llama.cpp\llama-cli.exe --version
This was an important distinction.
A true Vulkan failure would more likely produce something such as:
VK_ERROR_...
failed to load Vulkan backend
no Vulkan devices found
Instead, Windows had been stopping the process before Vulkan initialization even happened.
5. Confirming Vulkan actually worked
The definitive test was:
C:\llama.cpp\llama-cli.exe --list-devices
Output:
Available devices:
Vulkan0: AMD Radeon (TM) Graphics (16345 MiB, 15527 MiB free)
At that point I knew:
the Vulkan llama.cpp build worked;
the AMD driver worked;
llama.cpp could see the Radeon;
approximately 15.5 GB could potentially be allocated by Vulkan.
6. Choosing the first model
The first model I tested was:
Qwen3-Coder-30B-A3B-Instruct
using:
Q4_K_M
quantization.
The file was approximately:
18.6 GB
and was stored as:
C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
The model uses a Mixture-of-Experts architecture, which made it particularly interesting for relatively modest hardware.
7. Hugging Face SSL problems
Initially I tried letting llama.cpp download the model directly:
llama-cli.exe -hf lmstudio-community/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M
Instead I received:
HTTPLIB failed:
SSL server verification failed
error:
--model is required
The second error was just a consequence of the first: llama.cpp failed to download anything and therefore had no model to load.
Rather than waste time debugging Windows certificate handling, I downloaded the GGUF manually.
The resulting structure was:
C:\
├── llama.cpp\
│ ├── llama-cli.exe
│ ├── llama-server.exe
│ └── ...
│
└── models\
└── Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
I actually prefer this arrangement for a server.
Once downloaded, inference no longer depends on:
Internet connectivity
Hugging Face
SSL certificates
authentication
external availability
8. The first GPU memory failure
I initially tried maximum GPU offloading:
C:\llama.cpp\llama-cli.exe ^
-m C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf ^
-c 8192 ^
-ngl 99 ^
-p "Write a high performance C# lock-free ring buffer"
This failed with:
ggml_vulkan:
Device memory allocation failed
vk::Device::allocateMemory:
ErrorOutOfDeviceMemory
This made sense.
The model itself was approximately:
18.6 GB
while Vulkan could address around:
15.5 GB
and Vulkan also needs memory for:
compute buffers
KV cache
staging buffers
runtime allocations
graph execution
Windows/WDDM allocations
So -ngl 99 was too aggressive.
9. Partial GPU offloading
I next tried:
-ngl 20
with a smaller 4K context:
C:\llama.cpp\llama-cli.exe ^
-m C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf ^
-c 4096 ^
-ngl 20 ^
-p "Write a high performance C# lock-free ring buffer"
This got further, but failed while creating the compute buffers:
failed to allocate Vulkan0 buffer
failed to allocate compute pp buffers
The allocation that failed was only around:
228 MB
which showed how close the machine was to its Vulkan memory ceiling.
10. Reducing batch and micro-batch sizes
The eventual working configuration was:
C:\llama.cpp\llama-cli.exe ^
-m C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf ^
-c 4096 ^
-ngl 10 ^
-b 128 ^
-ub 128 ^
--no-kv-offload ^
-p "Write a high performance C# lock-free ring buffer"
Or on one line:
C:\llama.cpp\llama-cli.exe -m C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf -c 4096 -ngl 10 -b 128 -ub 128 --no-kv-offload -p "Write a high performance C# lock-free ring buffer"
This worked.
The important settings were:
-c 4096
4,096-token context.
-ngl 10
Only ten model layers were explicitly offloaded to Vulkan.
-b 128
Small prompt-processing batch.
-ub 128
Small micro-batch.
--no-kv-offload
Keep the KV cache out of the already-constrained Vulkan allocation.
11. First benchmark
The model produced approximately:
Prompt: 14.6 tokens/s
Generation: 4.6 tokens/s
The UI later showed approximately:
4.8 tokens/s
for generation.
That is not particularly fast, but it is entirely usable for a 30B-class coding model running on a low-power Ryzen APU.
The main limitation is memory bandwidth.
The architecture looks roughly like this:
32 GB DDR RAM
│
┌────────┴────────┐
│ │
Ryzen CPU Radeon iGPU
│ │
└────────┬────────┘
│
Same memory bus
Unlike a discrete GPU, the Radeon does not have hundreds of GB/s or ~1 TB/s of dedicated VRAM bandwidth.
Both CPU and GPU are competing for system-memory bandwidth.
12. Starting the browser UI
Once command-line inference worked, the next step was llama-server.exe.
I used:
C:\llama.cpp\llama-server.exe ^
-m C:\models\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf ^
-c 4096 ^
-ngl 10 ^
-b 128 ^
-ub 128 ^
--no-kv-offload ^
--host 127.0.0.1 ^
--port 8080
Then I opened:
http://127.0.0.1:8080
and the built-in llama.cpp chat UI appeared.
At the same time, the OpenAI-compatible API became available under:
http://127.0.0.1:8080/v1
13. Calling the local model from Python
Because llama.cpp exposes OpenAI-compatible endpoints, the normal OpenAI Python client can be used.
For example:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="local"
)
response = client.chat.completions.create(
model="local",
messages=[
{
"role": "user",
"content": "Write a C# bounded concurrent queue."
}
]
)
print(response.choices[0].message.content)
The application does not need to know much about the underlying inference engine.
Conceptually:
Application
│
│ OpenAI API
▼
localhost:8080/v1
│
▼
llama-server
│
▼
Qwen GGUF
14. The context-size problem
Eventually the UI produced:
request (7166 tokens) exceeds the available context size (4096 tokens)
This happened because the server was started with:
-c 4096
and the conversation had grown beyond that.
A context contains more than just the latest user message:
system prompt
+
previous user messages
+
previous assistant responses
+
new request
+
generated response
A larger context can be configured, for example:
-c 8192
or:
-c 12288
or:
-c 16384
but larger context windows have costs:
larger KV cache
more RAM
slower prompt processing
longer cold-context ingestion
On this machine, a roughly 8K–16K context is far more sensible than chasing enormous theoretical context windows.
15. Adding a faster model
The 30B coding model worked, but approximately:
4.8 tokens/s
is slow for everyday interactive use.
So I added:
Qwen3-14B
using:
Q5_K_M
quantization.
The second model became:
C:\models\Qwen3-14B-Q5_K_M.gguf
The final model directory was therefore:
C:\models\
├── Qwen3-14B-Q5_K_M.gguf
└── Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
The idea was:
Qwen3-14B
↓
fast/default work
Qwen3-Coder-30B
↓
harder coding work
16. Multi-model routing
I did not want two permanent llama-server.exe processes consuming RAM simultaneously.
Instead, llama.cpp supports model presets and routing.
The goal was:
llama-server
│
--models-max 1
│
┌────────────┴────────────┐
│ │
"fast" "coder"
Qwen3-14B Qwen3-Coder-30B
│ │
└────────────┬────────────┘
│
only one loaded
at a time
That is exactly what a 32 GB machine needs.
17. Creating models.ini
I created:
C:\llama.cpp\models.ini
with:
version = 1
[*]
parallel = 1
jinja = true
cache-prompt = true
[fast]
model = C:/models/Qwen3-14B-Q5_K_M.gguf
ctx-size = 8192
n-gpu-layers = 99
batch-size = 128
ubatch-size = 128
no-kv-offload = true
load-on-startup = true
[coder]
model = C:/models/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
ctx-size = 8192
n-gpu-layers = 10
batch-size = 128
ubatch-size = 128
no-kv-offload = true
load-on-startup = false
The names:
[fast]
[coder]
become model aliases.
18. Another mistake: unsupported default-model
Initially the configuration contained:
default-model = true
under the fast model.
The server failed with:
failed to initialize router models:
option 'default-model' not recognized in preset 'fast'
That option was not supported by the llama.cpp build I was using.
Removing it solved the issue.
The important startup option was instead:
load-on-startup = true
for the fast model.
19. Starting the multi-model router
The router was started using:
C:\llama.cpp\llama-server.exe ^
--models-preset C:\llama.cpp\models.ini ^
--models-max 1 ^
--host 127.0.0.1 ^
--port 8080
The critical option is:
--models-max 1
Without it, multiple models might remain loaded simultaneously.
That would be a bad idea with:
~10 GB 14B model
+
~19 GB 30B model
+
Windows
+
KV caches
+
compute buffers
With a one-model limit, llama.cpp can effectively behave like:
request model="fast"
↓
load fast
request model="coder"
↓
unload fast
↓
load coder
The load transition takes time, but RAM remains manageable.
20. Calling different models through the same API
Now one server can serve both models.
The base endpoint remains:
http://127.0.0.1:8080/v1
For normal work:
response = client.chat.completions.create(
model="fast",
messages=[
{
"role": "user",
"content": "Explain async iterators in C#."
}
]
)
For harder coding tasks:
response = client.chat.completions.create(
model="coder",
messages=[
{
"role": "user",
"content": """
Review this MPMC queue implementation for
memory-ordering and publication bugs.
"""
}
]
)
Same API.
Different backend model.
21. Making the server accessible over the LAN
For access from other machines, I changed:
--host 127.0.0.1
to:
--host 0.0.0.0
The resulting server command became approximately:
C:\llama.cpp\llama-server.exe ^
--models-preset C:\llama.cpp\models.ini ^
--models-max 1 ^
--sleep-idle-seconds 600 ^
--host 0.0.0.0 ^
--port 8080
Now another machine could use:
http://SERVER-NAME:8080
For example, the final browser UI was reachable using the Windows hostname directly:
http://win-2nsdp31hmq3:8080
22. Windows Firewall got in the way
During unattended startup, I eventually discovered another issue.
Windows displayed:
Do you want to allow public and private networks
to access this app?
llama-server.exe
This explained why startup appeared to fail.
The process was waiting for an interactive firewall decision.
When no user was logged in, nobody could click the dialog.
For a LAN-accessible server, the cleaner solution is a narrow Windows Firewall rule.
From elevated PowerShell:
New-NetFirewallRule `
-DisplayName "llama.cpp API - Private LAN" `
-Direction Inbound `
-Program "C:\llama.cpp\llama-server.exe" `
-Protocol TCP `
-LocalPort 8080 `
-RemoteAddress LocalSubnet `
-Profile Private `
-Action Allow
This permits:
llama-server.exe
TCP 8080
Private profile
Local subnet
rather than allowing arbitrary public-network access.
23. Making llama.cpp start after reboot
Initially I tried a Task Scheduler trigger:
At startup
but this was unreliable.
The machine also runs Docker Desktop, which normally expects an actual Windows user session.
There is also a practical issue with running GPU/Vulkan applications in a non-interactive Windows session.
The eventual solution was:
reboot
↓
automatic Windows login
↓
interactive desktop session created
↓
Docker Desktop starts
↓
llama.cpp starts
↓
workstation locks
This has proved much more reliable.
24. Windows automatic logon
For automatic logon I used Microsoft's Sysinternals Autologon utility rather than manually storing a plaintext password in Winlogon registry keys.
After enabling automatic logon for the server account:
Windows boots
↓
user automatically logs in
This creates the normal user session required by the software stack.
25. Starting llama.cpp at logon
Instead of:
At startup
I changed the llama.cpp Task Scheduler trigger to:
At log on
for the server user.
I also added a short delay so Windows, networking and the graphics stack could initialize first.
Approximately:
At log on
Delay: 30 seconds
The task runs:
C:\llama.cpp\start-router.cmd
26. The final startup script
The launcher is approximately:
@echo off
cd /d C:\llama.cpp
echo ============================================== >> C:\llama.cpp\llama-server.log
echo Starting llama.cpp at %date% %time% >> C:\llama.cpp\llama-server.log
echo ============================================== >> C:\llama.cpp\llama-server.log
C:\llama.cpp\llama-server.exe ^
--models-preset C:\llama.cpp\models.ini ^
--models-max 1 ^
--sleep-idle-seconds 600 ^
--host 0.0.0.0 ^
--port 8080 ^
>> C:\llama.cpp\llama-server.log 2>&1
This gives me a persistent log:
C:\llama.cpp\llama-server.log
which is very useful for diagnosing unattended-startup problems.
27. Automatically locking the workstation
Automatic login would otherwise leave the machine unlocked.
So after login and startup I schedule:
rundll32.exe
with:
user32.dll,LockWorkStation
after a short delay.
The resulting sequence is:
Machine reboots
↓
Windows boots
↓
Automatic login
↓
AMD/Vulkan session initialized
↓
Docker Desktop starts
↓
llama.cpp router starts
↓
Firewall already configured
↓
Windows locks
↓
Services remain running
Locking is important.
It preserves the logged-in session; logging out would terminate processes belonging to that user.
28. The final architecture
The finished environment looks like this:
WINDOWS MACHINE
│
▼
automatic login
│
┌────────────┴────────────┐
│ │
Docker Desktop llama.cpp
│
▼
llama-server router
port 8080
│
--models-max 1
│
┌────────────────────┴────────────────────┐
│ │
▼ ▼
fast coder
Qwen3-14B Q5_K_M Qwen3-Coder-30B-A3B
~10 GB ~18.6 GB
full/large GPU partial Vulkan
offload offload
│ │
└────────────────────┬────────────────────┘
│
ONE MODEL AT A TIME
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
Web UI Python C#
│ │ │
└──────────────────┼──────────────────┘
│
OpenAI-compatible API
│
http://SERVER:8080/v1
29. What I learned
Integrated GPUs are more useful than the VRAM number suggests
Windows initially showed roughly:
512 MB dedicated VRAM
which sounded hopeless.
But Vulkan reported:
~16 GB accessible GPU memory
because the Radeon uses shared system RAM.
That made reasonably large models practical.
Shared memory is not extra memory
The machine does not have:
32 GB system RAM
+
16 GB VRAM
=
48 GB
It has:
32 GB physical RAM
shared between:
Windows
CPU inference
GPU inference
KV cache
compute buffers
other applications
The 16 GB figure is effectively a GPU allocation budget over the same physical memory.
Maximum GPU offload is not necessarily optimal
For the 30B model:
-ngl 99
failed.
Even:
-ngl 20
eventually exhausted Vulkan compute-buffer memory.
The stable configuration became:
-ngl 10
There is little value in maximizing GPU layers if doing so makes the system unstable.
Batch size matters enormously
Reducing:
-b
and:
-ub
made the difference between failure and successful inference.
For interactive use, giant prompt-processing batches are unnecessary.
Keep enough memory for runtime buffers
A model fitting numerically into GPU memory does not imply it will run.
You also need room for:
model tensors
+
KV cache
+
compute buffers
+
graph buffers
+
Vulkan overhead
+
Windows/WDDM
Always leave headroom.
Large context windows are expensive
A model may advertise huge context sizes, but hardware reality matters more.
On this machine:
8K–16K
is far more useful than trying to force:
32K
64K
128K
contexts into memory.
Prompt processing also becomes increasingly expensive as conversations grow.
A smaller model can be a better default
The 30B model gives better quality on harder tasks, but around:
4.8 tokens/s
is not ideal for every interaction.
Using a smaller model for everyday work and escalating difficult requests to a larger model is a better architecture.
Hence:
fast
↓
Qwen3-14B
coder
↓
Qwen3-Coder-30B
30. Current model configuration
The final models.ini is:
version = 1
[*]
parallel = 1
jinja = true
cache-prompt = true
[fast]
model = C:/models/Qwen3-14B-Q5_K_M.gguf
ctx-size = 8192
n-gpu-layers = 99
batch-size = 128
ubatch-size = 128
no-kv-offload = true
load-on-startup = true
[coder]
model = C:/models/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
ctx-size = 8192
n-gpu-layers = 10
batch-size = 128
ubatch-size = 128
no-kv-offload = true
load-on-startup = false
And the router starts with:
C:\llama.cpp\llama-server.exe ^
--models-preset C:\llama.cpp\models.ini ^
--models-max 1 ^
--sleep-idle-seconds 600 ^
--host 0.0.0.0 ^
--port 8080
31. Testing the server
Local health check:
curl http://127.0.0.1:8080/health
Model list:
curl http://127.0.0.1:8080/v1/models
From another machine:
curl http://SERVER-NAME:8080/health
Browser:
http://SERVER-NAME:8080
API:
http://SERVER-NAME:8080/v1
32. OpenAI-compatible Python example
from openai import OpenAI
client = OpenAI(
base_url="http://SERVER-NAME:8080/v1",
api_key="local"
)
response = client.chat.completions.create(
model="fast",
messages=[
{
"role": "system",
"content": "You are an experienced software engineer."
},
{
"role": "user",
"content": "Explain the LMAX Disruptor architecture."
}
]
)
print(response.choices[0].message.content)
For the larger model:
response = client.chat.completions.create(
model="coder",
messages=[
{
"role": "user",
"content": """
Design a bounded lock-free MPMC queue in C#.
Discuss memory ordering, ABA issues and false sharing.
"""
}
]
)
33. What I would improve next
There are several obvious next steps.
API authentication
For anything beyond a trusted local machine, add:
--api-key
and never expose port 8080 directly to the public Internet.
Tailscale
For remote access away from home, I would use:
Laptop
│
Tailscale
│
Home LLM server
rather than router port forwarding.
Open WebUI
llama.cpp's native UI is already surprisingly usable.
A future enhancement could be:
Open WebUI
↓
llama.cpp /v1
for richer:
conversation management
file handling
RAG
authentication
multiple users
persistent histories
A genuinely fast small model
I may also add a third model in the 7B–8B range:
fastest
↓
7B model
↓
quick tasks / summarization / simple coding
leaving:
14B
for normal work and:
30B
for difficult coding/reasoning.
That would effectively create three local inference tiers:
FASTEST
7B
↓
cheap/simple work
FAST
14B
↓
default
QUALITY
30B
↓
hard problems
34. Final result
What started as a modest Windows machine with an integrated Radeon GPU has become a completely private LLM inference server.
It now provides:
local inference;
no per-token fees;
no external inference API;
no model prompts leaving the machine;
AMD GPU acceleration through Vulkan;
multiple GGUF models;
automatic model routing;
an OpenAI-compatible REST API;
a browser-based chat interface;
LAN access;
automatic startup after reboot;
Docker running alongside it;
and unattended operation.
The most surprising part was that the hardware was substantially more capable than its specification initially suggested.
The key was not trying to force desktop-GPU assumptions onto an integrated Radeon.
Instead, the working solution used:
Vulkan
+
shared GPU memory
+
partial model offload
+
small batches
+
CPU-side KV cache
+
quantized GGUF models
+
dynamic model routing
The end result is not going to compete with a high-end NVIDIA inference server.
It does not need to.
For private coding, experimentation, agents, document processing and local application integration, it is already a remarkably capable little LLM server sitting on hardware that was available anyway.
And, perhaps most importantly:
The API is mine.
The models are mine.
The data stays local.
%20applied%20to%20Transformer%20models%20in%20machine%20learning.%20The%20image%20shows%20a%20neural%20networ.webp)
Comments
Post a Comment