A quick post on how and why i use self hosted LLMs
While I have strong feelings about the overarching "AI" umbrella, open-source models and tools are neat.
Powered my home lab, with unraid, the ollama docker, and the open-webui docker.
Right now, these are my favorite opensource models:
- Llama 3.1 - Fast, solid all-rounder; tool-calling support
- Qwen 2.5 - Stronger multilingual, math, code
- OLMo 2 7 B - 100 % open training pipeline
- Mistral 7 B Instruct v0.3 - Fast, creative writing
- StarCoder2 7 B - Code-generation specialist, 600+ langs.
I have a few more at the ready when needed
What about internet access and search?
When needed, I can easily feed this to the net via my cloudflared container and custom domain.
I much prefer keeping it offline and use the strength of the model vs just having it be an internet gopher. However, i have searx self hosted and can easily tap the API when needed