A Local AI Productivity Companion With Python and Ollama — Week 2: Memory

What Week 2 was supposed to do

The goal was simple: take saved conversations and turn them into something the assistant could actually use later.

That meant building:

• project memory
• task memory
• daily notes
• recall commands
• progress summaries

The assistant should eventually be able to answer questions like “What was I working on yesterday?” using saved local memory instead of pretending to remember.

The important part was keeping the system simple. No vector database. No embeddings. No Notion. No web search. No email. No calendar. Just local JSON-first memory.

That constraint mattered a lot more than I expected.

What actually got built

The memory layer is now working in a basic but usable form.

The assistant can now:

• add tasks
• list tasks
• mark tasks as done
• save project memory
• generate simple progress summaries
• recall recent work context

The project now has:

• conversation indexing
• daily notes
• task memory
• project memory
• CLI recall commands
• memory commands wired directly into the voice loop

That last part matters. The conversations are no longer just saved as logs. Parts of them can actually be reused later.

Every conversation is still stored locally in simple text-based memory files (JSON-first for now). That makes the system easier to debug and easier to understand while the structure is still changing.

What went wrong

The memory is still shallow. That became obvious very quickly.

Right now, most of the summaries are still rule-based rather than genuinely intelligent.

The “yesterday” recall also exposed a practical problem: the system depends on having saved memory for that exact date. During one test, the recall technically worked correctly, but there was no May 10 memory available to retrieve. I left that failure in the log instead of hiding it — that is part of building systems like this.

There was also a permission error while saving task memory that still needs cleanup.

The useful lesson from Week 2 was not adding more AI features. It was learning what not to build yet.

The project became more stable once I stopped trying to add vector memory, embeddings, external integrations, and “smart agent” systems. The local JSON-first setup was already enough to expose the real problems.

Why this matters

Week 1 proved the assistant could talk. Week 2 proved it could start carrying information forward.

That changes the project completely.

Without memory, the assistant is basically a voice demo. With memory, it starts becoming a real productivity companion.

Even though the memory system is still basic, the assistant now has continuity between sessions. That is the first real step toward something genuinely useful.

What surprised me most

The hardest part was not AI. It was structure.

Once the memory layer appeared, the project could have become messy very quickly. So I kept the same folder + markdown workflow from Week 1. Each workspace still has clear scope, context files, constraints, and “what not to build” rules.

I originally got the idea for this workflow from a Jake van Clief video about structuring projects for tools like Codex and Claude Code. That structure is probably saving the project right now.

Without it, I would already be trying to add Notion, email summaries, web search, animated interfaces, and autonomous agents — before the foundation is stable.

Setup used for Week 2

The stack is still intentionally small.

Requirements

• Python 3.12 or similar
• Ollama installed locally
• A working microphone

Install Python libraries

These libraries handle microphone input, local speech-to-text, and offline voice output.

pip install sounddevice faster-whisper pyttsx3

Ollama model

ollama pull gemma3:4b

Main tools

• sounddevice – microphone recording
• faster-whisper – local speech-to-text
• pyttsx3 – offline text-to-speech
• Ollama – local AI model runtime

What I still deliberately avoided

Week 2 was still intentionally narrow. I avoided:

• Notion
• Gmail
• Google Calendar
• web search
• morning digest systems
• vector databases
• embeddings

That was deliberate. The goal was not to build a giant AI system. The goal was to make local memory useful first. A small working system teaches more than a giant unfinished one.

Next step — Week 3

Week 3 is the Productivity Brain.

The assistant can now remember things in a basic way. The next question is: can it help me decide what actually matters?

That means:

• unfinished task review
• progress summaries
• focus suggestions
• morning briefings built from local memory

The important part is that all of those systems should build on the existing memory layer instead of bypassing it.