Acorn GPT
Yes — for
inference, Acorn could run a useful locally hosted LLM on a dedicated CPU-only server.
A CPU server is most realistic for a forum assistant where requests can be queued and a response taking a few seconds is acceptable. It is a sensible fit for smaller, quantised models via Ollama, especially for tasks such as thread summaries, drafting replies, classification, moderation assistance and domain-name discussion.
The limits are mainly:
- Model quality versus speed: larger models will be slower and need substantially more RAM.
- Concurrency: one or two simultaneous chats may be fine; many simultaneous users can create a queue quickly without a GPU.
- Context length: long forum threads and large prompts increase prompt-processing time and memory use.
- Training/fine-tuning: CPU-only is generally impractical for this; keep the scope to serving an existing model.
For a production Acorn setup, I would prioritise plenty of RAM, modern high-core-count CPU performance, fast NVMe storage, request queueing, sensible maximum context/output limits, and a fallback if the local service is unavailable.
The existing CPU-only Ollama example is encouraging, but “runs fine” needs proper testing against Acorn’s actual workload: chosen model and quantisation, tokens per second, long-thread handling, peak simultaneous requests, and acceptable response time. A dedicated CPU box should be more predictable than a small VPS, but a GPU becomes worthwhile if the aim is consistently fast replies or significant concurrent use.
