Skip to main content
← All projects

Project

Run a Local LLM

Get a capable open-weight model running on your laptop in 20 minutes. Chat with it, hit it from a script, understand the trade-offs.

LocalOllamaOpen Weights~30 min

By the end of this project you'll have a real, capable model running on your machine that you can talk to in the terminal and call from a script — no API keys, no per-token cost. Local models aren't always the right tool, but knowing how to reach for them is part of being fluent.

Tip

Plan for storage. Models are 4–40 GB on disk. Pick one to start.

Step-by-step

The build

  1. 1. Install Ollama

    brew install ollama on macOS, or grab the installer for your OS. Run ollama serve in one terminal — that's the daemon.

  2. 2. Pull a model

    ollama pull llama3.1:8b is a good starter — capable enough for most tasks, fits in 8GB of RAM. Bigger models exist; bigger models are slower.

  3. 3. Chat with it

    ollama run llama3.1:8b. Compare its replies to whichever cloud model you usually use. Notice where it's surprisingly good, and where it falls over. Both matter.

  4. 4. Call it from a script

    Ollama exposes an OpenAI-compatible API at http://localhost:11434. Point any OpenAI SDK at that URL — your existing code works unchanged.

  5. 5. Note the trade-offs

    Latency: usually faster than cloud. Quality: lower at the same size. Privacy: full. Cost: zero per call, expensive in machine time. Pick local when privacy or offline use matters more than peak quality.

What you learned

  • A capable model fits on a laptop. The future where every app can ship with one is closer than it looks.
  • The OpenAI API shape has won; everything emulates it. That portability matters.
  • Local-vs-cloud is a workload question, not a moral one.