Building a local AI server

ai
Published

June 3, 2025

I have been a avid tinkerer with home pc’s. I enjoy building pc. Now that LLM’s are a necessity for a developer, I wanted to build a great performant system for myself.

Requirements

Hardware

My requirements were quite basic:

  • A minimum of 16GB VRAM (preferably Nvidia)
  • A minimum of 16GB RAM
  • A minimum of 6 cores / 12 threads Ryzen CPU
  • A motherboard with 2 PCI GPU slots (does not need to be 16 lanes PCI)
  • A minimum of 600 watt power supply
  • A wi-fi smart-plug with scheduling capabilities
  • A motherboard which support power-on after power restoration

Software

  • Linux OS with a long-term support and used by a lot of users
  • Linux OS with Tailscale native support (systemd)
  • Linux OS with easy installation of Docker, Nvidia Container Toolkit and Cuda Toolkit
  • Linux OS with OpenSSH server to manage server remotely
  • Linux OS with crontab to schedule running of scripts

Server

I ended up purchasing a second-hand PC with all the above requirements and then purchased two RTX 3060 GPUs with 12 GB Vram each. The RAM capacity is somewhat low for this use-case and I plan to upgrade it in the future.

Scheduling

Updates

As everything runs on an Ubuntu LTS 24.04.

Docker

Ollama and OpenWebUi Bundle

Docker exec

Power automation