GuidesOctober 25, 2026
Step-by-step guide · Claude Code
Build your own Jarvis
A voice assistant that lives in your home: you say “hey Jarvis”, it understands you in English or Spanish, thinks with Claude and answers in a natural voice. It turns on the lights (if they’re smart), plays music, reads your calendar, wakes you up with an alarm that slowly gets louder, knows when you leave and come home, and only listens to your voice.
Mac · Windows · LinuxEnglish and SpanishNo coding needed
The good part: almost everything runs on your own computer. Your audio never leaves your home: the only thing that goes online is the text of what you ask, so Claude can think about it. A full day of talking to it costs cents.
You don't need to know how to code. Claude Code writes the code: you paste the prompts from this guide, one by one, and test them. You do need patience: each piece gets tuned by testing it with your voice, in your home.
1. What you need
A computer that stays on
RequiredIdeal: a Mac with an M chip (M1 or newer) understands your voice very fast. A Windows or Linux PC works too; with an NVIDIA card it's just as fast, without one a bit slower.
A Claude API account
RequiredAt console.anthropic.com. Pay as you go (prepaid); a normal day costs cents.
Claude Code
RequiredIt writes the code for you (claude.com/claude-code). Needs a paid Claude plan (Pro or higher) or API credits.
A microphone in the room
RequiredThe computer's, a USB one, or an old Android phone (see path B).
A speaker
RequiredThe computer's, a Bluetooth one or an Echo (Alexa) you already have, paired over Bluetooth.
Smart devices (Smart Life / Tuya bulbs, switches or plugs)
OptionalOnly to control your home. Jarvis can't turn on a regular light: see below.
Spotify Premium
OptionalTo play music by voice. Plus a free app at developer.spotify.com (Claude Code walks you through it).
Google Calendar
OptionalFor your schedule and meeting reminders. It connects through a free Google Cloud project (Claude Code walks you through it).
To control the lights: smart devices
- Jarvis has no hands: to turn something on or off, that something has to be controllable over Wi-Fi. It doesn't work with a regular bulb or switch.
- What works: smart bulbs, wall switches or plugs that you control with the Smart Life or Tuya app. Many cheap brands use Tuya inside: if they set up with one of those two apps, they work.
- A fan, a lamp or any appliance: use a smart plug. Jarvis turns the plug on and off.
- Once, you create a free developer account on the Tuya platform to get each device's local key. After that Jarvis talks to them directly over your network, with no cloud, and they work even if the internet goes down.
- Got another brand (Philips Hue, TP-Link Kasa…)? Ask Claude Code to connect it instead of Tuya; the rest of the guide stays the same.
- No smart devices at all? Jarvis still talks with you, reads your calendar, tells you the weather, searches the internet, wakes you up and runs your routines. Lights and plugs are the only part that needs devices.
For music: Spotify Premium and a speaker
Jarvis plays music with your Spotify Premium account (without Premium, Spotify won't let another app control it) and it plays through the computer's speaker or any Spotify Connect device: an Echo, a TV, a smart speaker. If you already have an Alexa, don't throw it away: pair it with the computer over Bluetooth and Jarvis talks and plays music through it.
What I use
Computer
Mac mini M4 with 16 GB, always on.
Ear
An old Android phone (Redmi Note 10), fixed and plugged in in the room (path B).
Speaker
An Echo Dot I already had, paired with the Mac over Bluetooth (and Spotify Connect if Bluetooth drops).
Lights
A three-button Smart Life wall switch (bedroom and terrace lights) and a Smart Life plug for the fan.
Music and calendar
Spotify Premium and Google Calendar.
Brain
Claude Haiku 5.5 all day and Claude Opus 5.5 when I say “think hard” (Claude API).
Speech to text
Whisper large-v3-turbo (mlx-whisper) on the Mac.
Voice
Kokoro-82M: em_alex in Spanish and bm_george (British) in English, at 1.15 speed.
Who's talking
SpeechBrain (ECAPA): it only listens to me.
Spend
A full day of testing, 79 Claude answers: USD 0.02.
2. Pick your path
Path A
Just your computer (the simplest)
The computer listens all the time with its microphone, detects “hey Jarvis” by itself and answers through its speaker.
- Pros
- Pros: One machine, fewer pieces, faster to build.
- Cons
- Cons: The computer has to be in the room where you talk, and listening all the time uses a little of it (not much: the wake word is a small model). If your computer has no microphone (a Mac mini, for example), plug in a USB one or a headset.
Path B
Computer + an old phone as the ear
An Android phone you no longer use stays plugged in, in the room. It listens for “hey Jarvis” and only then sends your voice to the computer over your home Wi-Fi. The computer does the rest.
- Pros
- Pros: The microphone is where you talk (the computer can be somewhere else), the computer spends nothing on listening, and audio only travels after the wake word. The phone's screen becomes Jarvis's “face”.
- Cons
- Cons: You need a small Android app (Claude Code writes it) and to install it with a cable once.
Not sure? Start with A. If you want a separate ear later, the brain is already built: you just add the phone.
Path A
Just your computer
Computer
earspeech to textvoicehands
Claude thinks of the answer
Path B
Computer + phone as the ear
Old phone
ear
Computer
speech to textvoicehands
Claude thinks of the answer
3. How it works (the 5 pieces)
- 01
Ear
Detects “hey Jarvis” offline (openWakeWord). Then it listens until you stop talking (WebRTC VAD).
- 02
Speech to text
Whisper (large-v3-turbo) turns your voice into text on your computer (mlx-whisper on a Mac, faster-whisper on a PC).
- 03
Brain
Claude thinks of the answer, through the Claude API. Claude Haiku 5.5 (claude-haiku-5-5, low effort) for everyday things: it’s fast and the cheapest. If you say “think hard”, that request is answered by Claude Opus 5.5 (claude-opus-5-5, high effort). The summaries (the morning one, one per conversation and what it learns about you) are also done by Haiku 5.5. Claude uses tools: turn on lights, play music, read the calendar.
- 04
Voice
Kokoro, a natural, free voice that runs on your computer, speaks the answer sentence by sentence.
- 05
Hands
The connections to your home and accounts (Tuya, Spotify, Google), each with its own permission.
4. Step by step with Claude Code
Create an empty folder, open it with Claude Code and paste the prompts in order. Test each step before moving on. If something fails, tell Claude Code what happened in your own words (“it hears me but answers too late”): that's how it gets tuned.
Prompt 0The project
Let's build a personal voice assistant called Jarvis, in Python, running on this computer.
Rules from the start:
- Keys (Claude API, tokens) go in the system keychain (keyring library), never in files.
- Everything local except the text sent to Claude. Audio is never stored.
- It can be turned off at any moment, and it says so out loud.
- Clean modules (ear, speech to text, brain, voice, tools), with tests from day one.
Create the structure with uv, a `jarvis` command and a `jarvis key` command to store my Claude key.Prompt 1AThe ear on the computer (path A)
Make Jarvis listen with this computer's microphone:
- openWakeWord with the "hey_jarvis" model to detect the wake word, offline. Configurable threshold.
- When it hears it: a short sound ("I'm listening"), then record until I stop talking with webrtcvad (plus an energy threshold, because background noise sometimes passes as speech). Keep the second before the wake word too, so nothing I say right after "hey Jarvis" gets lost.
- Turn the audio into text with Whisper (mlx-whisper on a Mac, faster-whisper on a PC). Detect whether I'm speaking English or Spanish, and ignore made-up text when there was only silence (Whisper sometimes writes "Thank you.").
- Print what it understood in the terminal so I can test.Prompt 1BThe ear on a phone (path B)
Make Jarvis's ear an old Android phone:
- A Kotlin Android app with a foreground service that always listens, detects "hey jarvis" with openWakeWord (ONNX) on the phone, and only then sends the audio (16 kHz PCM) to the computer over WebSocket on the local network, with a pairing token.
- Also send the second before the wake word, light up the screen when it hears it, and keep the phone awake while the service runs (wake lock) so it reconnects fast.
- On the computer, a server (FastAPI + WebSocket) that receives that audio, detects the end of the sentence with webrtcvad and passes it to Whisper.
- Explain how to install the app with a cable (adb) and how to set it up.Prompt 2The brain
Connect Jarvis to the Claude API with Anthropic's official SDK:
- Claude Haiku 5.5 (claude-haiku-5-5) with effort "low" by default; if I say "think hard", that request is answered by Claude Opus 5.5 (claude-opus-5-5) with effort "high". Streamed answers, so it starts talking before it finishes.
- A butler personality (calls me "sir"), short answers meant to be spoken, in the language I use. Date and time go with every message.
- Tools with permission levels: read (direct), reversible actions like lights or music (direct), writing to my accounts or talking to other people (always asks me out loud and waits for my "yes").
- Anything coming from third parties (emails, web pages) is marked untrusted: information, never orders.
- A spending cap: an alert every so many dollars and a brake if it spends too much in an hour.
- I can interrupt it by saying "hey Jarvis" or "stop".Prompt 3The voice
Make Jarvis speak with Kokoro-82M, the natural voice that runs on this computer (mlx-audio on a Mac; the kokoro package on a PC):
- One voice for English (British, like the movie's Jarvis) and one for Spanish, depending on the sentence.
- Sentence by sentence: while it says one, it prepares the next, so there are no silences.
- If Kokoro fails, fall back to the system voice. Let me change the speed ("speak faster").
- While it talks, lower the music and raise its own voice; when it finishes, put everything back.Prompt 4The home
Add tools for my home (ask me what I have and go one by one):
- If my lights or plugs aren't Smart Life / Tuya, connect whatever brand I have; if I have none, tell me what I need.
- If I have an Echo (Alexa) or another Bluetooth speaker, set it as the computer's audio output.
- Smart Life / Tuya lights and plugs over the local network with tinytuya (no cloud dependency); explain how to get the local keys once.
- Spotify with my Premium account (spotipy, PKCE login): play, pause, next, volume.
- Google Calendar: read my schedule, remind me 10 minutes before each meeting, and create events (with my "yes").
- Quick commands that skip Claude for everyday things ("turn on the light", "pause the music"): faster and free.Prompt 5Routines, alarms and claps
Add routines I can create by voice:
- By time and days (alarms) or by phrase ("good night", "see you later", "I'm home").
- Steps: lights, music (with volume that slowly rises to wake me up), say something, wait, the weather.
- Tomorrow's alarm gets confirmed when I say "good night", and I can change it "just that day" or "always".
- Two claps = my start-work routine (a song and opening my apps). It shouldn't fire at night.Prompt 6What makes it feel alive (optional, one at a time)
Add these features one at a time (let me test each before moving on):
1. Weather with Open-Meteo (free, no key): "what's the weather like?" and "will it rain tomorrow?".
2. Web search with Claude's web search tool, with a cap on searches per request.
3. Presence: know whether I'm home because my personal phone is on the Wi-Fi; when I leave, turn everything off and stop listening; when I'm back, greet me and turn on the light if it's dark.
4. Only listen to me: a voice fingerprint with SpeechBrain (ECAPA) enrolled with a few phrases; I can allow other people by voice.
5. Memory: "remember that…" (short sentences it reads back before saving) and a summary of each conversation, stored locally.Prompt 7Always on
Run Jarvis as a system service that starts at login and restarts if it crashes (launchd on Mac, Task Scheduler on Windows, systemd on Linux), with a menu bar icon to see if it's listening, close the microphone and turn it off.
Keep the computer from sleeping while it runs, and make restarting it safe.5. Mark II: what I added later
Once the first version worked, I put it to work with me. These are four more pieces, each one on its own: paste its prompt in the same Jarvis folder, with everything before it working, and test it before moving on.
Claude Code by voice
- What it does
- You say “Jarvis, tell Claude to check why the login fails in my project”, it asks for your “yes” reading back the project, and Claude Code gets to work. You go do something else and, when Claude is done, Jarvis tells you out loud with the first sentence of the result. Same with your regular Claude Code chats: if a job took more than 2 minutes and you haven’t touched the computer for 1 minute, it tells you it finished or that it needs your permission. For safety, what you ask by voice can only read the project: it runs no commands, doesn’t load your permissions and doesn’t touch your keys.
- What you need
- Claude Code installed on the same computer, with your account. The voice fingerprint from prompt 6, so only you can give it work.
I want to give Claude Code work by voice and have Jarvis tell me out loud when it's done. Claude Code is already installed on this computer with my account.
1. Jobs by voice ("Jarvis, tell Claude to check why the login fails in <project>"):
- Find my projects (git folders inside ~/dev, up to two levels deep) and pick the one that has all the words I said; if there are several, ask me which.
- Only if it's my voice (the voice fingerprint), and always with my spoken "yes", reading back the project and the request.
- Run `claude -p` in the project folder, read-only and isolated: only the Read, Grep and Glob tools (no terminal: no git, no npm), without loading my permissions or the project's (--setting-sources ""), no MCP servers (--strict-mcp-config), no hooks (disableAllHooks), --permission-mode dontAsk, and block .env*, ~/.ssh and ~/.claude. One at a time, 20 minutes max; if it runs over, kill all of it.
- When it finishes, save the full result to a local file and tell me the first sentence out loud.
- "How's Claude doing?": tell me where it's at, without going through Claude.
2. Make Claude Code talk to me through Jarvis's voice:
- A local endpoint in Jarvis (127.0.0.1 only) so other programs can ask it to say something, at low priority: at night, in a meeting or if I talked to it less than 90 seconds ago, it waits.
- Claude Code Stop and Notification hooks in my ~/.claude/settings.json (async) that call a Jarvis command that never fails.
- Stop: only if the job took 2 minutes or more since my last request and I haven't touched the computer for 1 minute (on a Mac, HIDIdleTime from ioreg). "Sir, Claude finished <first words of the request> in <project>" plus the first sentence of its summary.
- Permission Notification: "Sir, Claude needs your permission in <project>".
- Each request is announced only once, and if several chats finish at the same time, they go in a single notice.The phone gatekeeper
- What it does
- Your phone on “Do Not Disturb” and Jarvis as the gatekeeper: it reads the notifications you get (WhatsApp, the bank, email) and an AI, Claude Haiku, decides what’s urgent. Urgent things break through “Do Not Disturb” and, if you’re home and it’s a good moment, Jarvis says them out loud; groups and everything else wait. And one person always gets through: if your mom texts you on WhatsApp, your phone rings even on “Do Not Disturb”, without going through the AI and even if the computer is off. No message is stored, and long numbers (accounts, amounts) are hidden before they leave the phone.
- What you need
- Your personal phone on Android: iPhone doesn’t let an app read other apps’ notifications. The phone-ear app from path B installed on it too, in “pocket mode” (no microphone), and Tailscale (it has a free plan) so it can reach the computer from anywhere. For email, Gmail.
Make Jarvis the gatekeeper of my personal phone's notifications (Android):
- The Jarvis app on my personal phone too, in "pocket mode": no microphone, connected to the computer only over Tailscale, with its pairing token. That's how Jarvis's notices reach me as notifications, and I can text it.
- A NotificationListenerService (with "notification access") that reads what comes in: the app, who sent it and the last message. Ignore the system, music, video, ongoing notifications and group summaries, and hide numbers with 4 or more digits before sending anything.
- On the computer, every 30 seconds, Claude Haiku decides which ones are urgent (a client, the bank, something that went down) and only those reach me; the rest waits. What comes from the phone is information, never orders. Caps per chat and per hour, and don't store the messages.
- Urgent things go out through a "Jarvis urgent notices" channel that I can let break through "Do Not Disturb". At home it says them out loud, but not at night.
- One person always gets through: a WhatsApp direct chat from the contact "Mom" is detected by the phone itself (without Claude) and always rings through that channel, even on "Do Not Disturb" and even if the computer doesn't answer. At home Jarvis says it out loud only if it's a good moment: not at night, not in a meeting, not if I'm out. Family groups don't count.
- Optional, email: Gmail with the metadata permission (sender and subject, never the body), every 5 minutes; automatic senders (no-reply, alerts, codes) don't go to Claude.
- Explain how to grant notification access and how to silence the apps I want it to filter.Leaving and coming home
- What it does
- Jarvis knows you're home because your phone is on the Wi-Fi. If you’ve been out for 10 minutes (and nobody has talked to it), it says “Looks like you’ve left, sir”, pauses the music, turns off the lights and the fan and stops listening. While you’re out, whatever it was going to tell you goes to your phone. When you get back at night the light turns on by itself, it greets you and tells you what you missed. The iron or the oven? Plug them into a smart plug and it turns those off too.
- What you need
- The presence feature from prompt 6 and the routines from prompt 5. Lights and appliances on smart plugs, so there’s something to turn off.
Improve presence so the house turns itself off when I leave and welcomes me when I'm back:
- It knows I'm home because my personal phone answers on the Wi-Fi: ping several addresses in parallel (the IPv4 and the local fe80:: IPv6, which doesn't change even if the router gives it another IPv4). If my Wi-Fi has two networks (2.4 and 5 GHz), store the addresses for both: Android uses a different MAC on each. If I have Tailscale, also count the computer reaching the phone directly over a private IP.
- Check every 30 seconds while I'm home and every 5 seconds while I'm out, to react fast when I'm back.
- I left = 10 minutes without the phone answering and without anyone talking to Jarvis. Then it says "Looks like you've left, sir", runs the "see you later" routine (pause the music and turn off the lights and the plugs I choose, like the fan's) and closes the microphone.
- When I'm back: open the microphone (only if it closed it), run "I'm home" (the light only if it's already dark), greet me and tell me what I missed.
- While I'm out, whatever it was going to tell me goes to my phone instead of waiting (not at night).
- A test command, "jarvis presencia-probar salir" and "volver", to try it without actually leaving.Meetings and “what did I miss?”
- What it does
- If someone books a meeting with you (through Calendly or an invite), moves it or cancels it, you get a notice on your phone: “New meeting: «…», tomorrow at 3:00 pm”. And if it overlaps with another one, it says so: “Clashes with «…»”. If it starts in less than 2 hours, the notice breaks through “Do Not Disturb”. While you’re in a meeting Jarvis stays quiet and keeps what it was going to say. When the meeting ends, when you get home or when you ask “what did I miss?”, it tells you in one sentence. Only to your voice or from your phone: to anyone else it says it only tells you.
- What you need
- Google Calendar connected (prompt 4). Calendly writes its bookings to Google Calendar, so it needs nothing extra. The phone in pocket mode from prompt 9 and the voice fingerprint from prompt 6.
Add notices for new meetings and "what did I miss?":
- Every 2 minutes, across all my Google Calendar accounts, fetch the events in the next 60 days that changed since the last check (updatedMin, including deleted ones). If I use Calendly, its bookings already land in Google Calendar: no need for its API.
- A meeting is an event with a time, marked busy, with someone else. If it's new, moved or cancelled, send a notification to my phone (not out loud): "Sir, new meeting: «<title>», tomorrow at 3:00 pm". The same meeting in two accounts is one notice, and the first check only takes note of the ones already there.
- Clashes: if it overlaps with anything else busy on the calendar, the notice ends with "Clashes with «<title>»".
- If it starts in less than 2 hours, the notice is urgent and breaks through "Do Not Disturb"; the others respect it. Anything that couldn't be sent is retried, even after a restart.
- In a meeting it doesn't talk: what it was going to say goes as a silent notification on the computer's screen.
- "What did I miss?": keep in memory (12 hours, the last 20) what I didn't hear: what it kept quiet in a meeting, what it sent to my phone because I was out and what came in during silent mode. Tell me in one sentence when the meeting ends, when I get home or when I ask "what did I miss?", then clear it. Only to my voice or from my phone; to another voice it says "I only tell that to <my name>" and keeps it.6. What it costs
Hearing, understanding your voice, the natural voice, recognizing who's talking, controlling the lights, weather
Free: runs on your computer or uses free services.
Claude Haiku 5.5 (all day, plus the phone gatekeeper)
USD 0.10 per million input tokens and 0.50 per million output tokens. A normal day: cents.
Claude Opus 5.5 (“think hard”)
USD 4 per million input and 20 per million output: about 40 times more, so only when you ask.
Web searches
USD 10 per thousand searches, plus the tokens of what Claude reads.
Claude Code
A paid Claude plan or API credits, while you build it. The Mark II voice jobs use that same account.
Smart devices
Bought separately, once (only if you want to control lights or plugs).
Spotify
Premium, to control it by voice.
Tailscale (the phone in pocket mode)
The personal plan is free.
7. Privacy and safety
Audio never leaves your home and is never stored. Only the text of what you ask goes to Claude.
Keys go in the system keychain, never in a project file.
Nothing gets written to your accounts without your “yes” (creating events, sending messages), and nothing it reads in an email or a web page can give it orders.
It can always be silenced and turned off: “stop”, “stop listening”, “shut down”.
The gatekeeper doesn’t store your messages: long numbers are hidden on the phone, from email it only reads sender and subject, and nothing it reads is ever an order.
What you ask Claude Code by voice can only read: no terminal, no permissions of yours and no keys.
Licenses, in case you ever want to sell it: openWakeWord's “hey Jarvis” models and Open-Meteo's free plan are for personal use only, and Kokoro’s Spanish pronunciation uses espeak, which is GPL. A product needs its own trained wake word, a paid weather plan and a look at the espeak license. Claude API prices as of October 2026.
8. Common problems (and what worked)
It answers noises or the TV
Ask Claude Code for
An energy threshold in the voice detector and a filter for made-up phrases.
It hears itself and answers itself
Ask Claude Code for
Wait a moment after speaking and discard anything that sounds like what it just said.
It misses what you say right after “hey Jarvis”
Ask Claude Code for
Keep the second before the wake word.
Music keeps it from understanding you
Ask Claude Code for
Lower the music to 10 % while it listens and talks.
Spotify won't start the desktop app
Ask Claude Code for
Control the local app directly (AppleScript on a Mac) instead of the API.
The first command of the morning fails
Ask Claude Code for
Retry once if the connection to the service dropped.
The phone ear takes long to reconnect
Ask Claude Code for
Keep it awake while the service runs (wake lock).
It talks too quietly or too loudly
Ask Claude Code for
Give the voice its own volume, separate from the music.
It thinks you left when you just went downstairs
Ask Claude Code for
Store the phone’s addresses on both Wi-Fi networks (2.4 and 5 GHz).
Want one for your business?
The same idea, an assistant that understands, answers and acts, works for serving customers on WhatsApp, Instagram or your website, booking appointments or answering questions at any hour. If you want one, let's talk.
More guides
See all guidesClaude Code + Android
Remove the ads from your Android
The carrier and brand apps that show you ads: how to remove them with Claude Code and a cable, what you can remove yourself from Settings, and a plan B by hand. Every step can be undone.
October 29, 2026 · 9 min
Claude + video
Your video editor is Claude
Record in one take and Claude does the cuts, captions, zooms and music in your style. The free tools, the prompts for Mac and Windows and how to teach it your brand.
October 27, 2026 · 9 min