I'm Building My Own Daily AI Companion
This is the anchor project. Not a weekend build. A month-long build, in public, with everything logged here.
Two years ago, I tried to build a personal AI companion that could talk to me, remember what I was working on, and help me stay productive. The idea was good, but the tools were not ready. Now I'm trying again – this time as a serious month-long VibeCodeTools build.
The project is called DeskSpirit. It sounds personal, not corporate. That's intentional. This isn't a SaaS product. It's a tool built for one person – me – and documented so anyone building something similar can follow the real process, not a polished retrospective.
Why I'm building this
Every productivity tool I've tried assumes I work the same way every day. I don't. Some days I'm deep in a build. Some days I'm reading and thinking. Some days I can't remember what I was doing 20 minutes ago.
I want something that sits alongside me, knows what mode I'm in, and adjusts. Not a calendar app. Not another task manager. Something that can ask me what I'm working on and actually remember the answer next time.
That's DeskSpirit. The goal is a lightweight desktop companion that holds context across sessions, surfaces what I was doing, and gets out of the way when I don't need it.
This is also the build this site needed. Everything else I've published here was a weekend project. DeskSpirit is different – it's longer, harder, and the kind of proof that a solo builder with AI tools can create serious personal software.
What I tried two years ago
The first attempt was a mess in the best possible way. I had a rough idea, a chatbot API, and a lot of enthusiasm. What I didn't have was a model with real memory, reliable local storage that didn't corrupt on restart, or a UI that didn't feel like a demo project.
The core problem: I could get it to respond, but I couldn't get it to remember. Every session started blank. Every context window was too small to hold anything meaningful. I built a workaround with manual notes that got pasted in – which is exactly the kind of friction I was trying to remove.
I stopped after about three weeks. Not because the idea was wrong, but because the tools weren't good enough to make it feel real.
What changed now
The models are better. Context windows are longer. Local-first tooling has caught up. And the AI coding tools I use every day – the same ones documented across this site – are capable of building something like this without a full engineering team.
The thing that actually unlocked this: I stopped trying to build a general-purpose tool and got specific. DeskSpirit isn't for everyone. It's for me. That constraint makes every decision easier.
I'm also building it differently. Two years ago I started with the UI. This time I'm starting with the memory layer – how context is stored, retrieved, and kept relevant. Everything else builds on top of that.
Current build status
Early stage. The foundation is being laid. No UI yet – deliberately. I'm building the context engine first and I won't touch the interface until it works the way I want.
Current focus: how sessions are stored, how context is retrieved on startup, and how the companion decides what's relevant to surface.
Everything I figure out – including what breaks and what I have to rebuild – gets logged in the weekly progress section below.
Tech stack
Still being finalised, but the constraints are set:
- Local-first. No cloud sync. No account required.
- Persistent memory across sessions – not just a long chat history, but structured context that can be queried.
- Lightweight UI. Nothing that feels like an app. Something that feels like it's always been there.
- Built with AI tools throughout – every decision, every prompt, every rebuild logged here.
I'll update this section as the stack locks in. Right now I'm testing approaches, not committing to one.
Weekly progress log
This section gets updated every week. Everything that happened – good, bad, and scrapped.
Week 1 – April 2026
The voice loop works. Microphone recording, speech-to-text via Ollama, local response, text-to-speech, conversation saved to JSONL. End-to-end in one session.
What got cut: UI, memory system, multi-step tasks. All removed to make the core loop work first. That's the only constraint that mattered this week.
First real blocker: Python not on PATH. Not an AI problem – a setup problem. Fixed it, then hit microphone timing issues. Working through them.
Week 2 – May 2026
The memory layer is now working in a basic but usable form. The assistant can add tasks, list tasks, mark tasks done, save project memory, generate progress summaries, and recall recent work context.
What got cut: vector databases, embeddings, Notion, web search, external integrations. The local JSON-first setup was enough to expose the real problems without adding complexity.
Key lesson: the hardest part wasn't AI. It was structure. The folder + markdown workflow from Week 1 is keeping the project from drifting.
Week 3 – May 2026
The focus is now the productivity brain: using local memory to review unfinished tasks, summarize progress, and suggest what deserves attention next.
What is still cut: Notion, Gmail, calendar, web search, vector memory, and UI work. Week 3 has to prove the local memory can guide decisions before more inputs get added.
Key question: can DeskSpirit answer “What should I focus on today?” without giving generic productivity advice?
Problems I hit
Running list. Updated as they happen.
- Defining context structure – what gets stored, how it's retrieved, how stale context gets aged out
- Session boundaries – when does one session end and another begin, and does it matter
More to follow. This section will grow.
What worked
Running list. Updated as I find things that actually hold up.
- Starting with constraints before writing any code – it forced clarity on what this actually needs to do
- Naming the project early – DeskSpirit gave it an identity that makes decisions easier
What failed
Running list. No filtering – if it got scrapped, it goes here.
- First schema attempt was too flat – treated all context equally, which doesn't reflect how I actually think about what I'm working on
Next milestone
A working context engine that can store a session, survive a restart, and surface what I was doing without me having to re-explain it.
No UI. Just the foundation. If that works, everything else is buildable.
This is the build this site is built around. Follow it here – every week, every problem, every decision.