Exploring Local AI: Why Running Large Models On-Device Is Exciting, Overwhelming, and Often Frustrating
As privacy concerns make users hesitant to share personal data with cloud providers like OpenAI, Google, Microsoft, and Anthropic, running AI models locally has emerged as a compelling alternative. In a hands-on testing diary, The Verge's Antonio G. Di Benedetto explores deploying local AI using the open-source Hermes Agent desktop application and a 125-billion-parameter Qwen 3.8 Flash Next model on an M5 Ultra Mac Studio equipped with 256GB of unified memory. While self-hosting eliminates per-token costs and guarantees data privacy, the experience proves overwhelming and occasionally frustrating. Users face decision fatigue across countless models, struggle with prompt initiation, and deal with fragile automations, such as recurring morning briefings that break. Ultimately, the diary evaluates whether substantial hardware investments are truly justified for everyday local AI utility.
