Voice, dictation, and memory

Spoken replies

Pick a voice engine in App Settings, choose a voice, and use the speaker control on a response. Auto-speak can read new replies as they arrive. Four engines are available:

• ElevenLabs — cloud voices, needs an API key.
• Fish Audio — cloud voices from the Fish Audio library, needs its own API key.
• Built-in Mac voices — the voices already installed on macOS, no key.
• Chatterbox (local) — an open-source, MIT-licensed engine reached through an OpenAI-compatible server you configure. With a server on your own machine there is no cloud call; the address can also point at any HTTP(S) server you choose.

The harness rewrites code-heavy Markdown into speech-friendly text before synthesis. Keys remain on the harness and never reach the renderer.

Fish Audio

1. Create a key in Fish Audio API Keys.
2. In an agent profile, choose Fish Audio, paste the key, and save it.
3. Pick one of the voices returned by your Fish Audio account, then use Try before enabling spoken replies.

Fish Audio and ElevenLabs keys are stored separately, so switching engines does not replace the other provider's credential. The picker combines popular public voices with voices owned by your Fish Audio account. Only use a voice when you have permission to do so.

Chatterbox (local)

MengBot speaks through any OpenAI-compatible server that runs Chatterbox behind POST /v1/audio/speech — for example LocalAI with a Chatterbox model installed, or a small wrapper service around pip install chatterbox-tts.

1. Start your Chatterbox server locally and note its address, for example http://127.0.0.1:4123.
2. In App Settings, pick Chatterbox (local) as the voice engine, paste the server address, and set the model id your server expects (the default is chatterbox-turbo).
3. Pick a voice. Voice ids come from the server's /v1/models listing when it has one, or /v1/audio/voices when it doesn't; otherwise a Default entry uses the server's built-in voice.

There is no API key. MengBot sends the text to the configured server address, so synthesis stays on your machine only when that server is local. Per-agent voice overrides work the same as with the other engines, and a dead or unreachable server is reported as a clear error instead of a hang.

Voice notes

An agent can send a voice note: a short spoken message it writes itself, synthesized with the configured voice engine and attached to its reply when the turn ends. The same text becomes the note's visible caption and transcript, so the conversation stays readable — and searchable — without playing anything. Notes are limited to 1000 characters, and the same limit is enforced again at synthesis.

A note posted to a shared room works the same way: the agent records the note, then attaches it to its room post, where the room members can play it inline.

Voice notes can be turned off per agent. The switch sits with the voice settings in the agent profile; while it is off, the agent is not offered the voice-note tool at all, even with a voice configured.

The audio itself is an mp3 file in the attachments directory on your computer, with a generated name and owner-only permissions. It counts against the same 512 MiB attachment quota as images and documents.

Call mode

On macOS, call mode combines native on-device speech recognition with the configured voice. It is deliberately half-duplex: the microphone pauses while the bot speaks to prevent the app from transcribing its own audio.

Tool progress and approval requests can be narrated during a call so long-running work does not sound disconnected.

Take turns or Live

On the desktop, the call button starts the kind of call you picked last; the arrow next to it picks Take turns or Live and starts that call.

• Take turns is the call described above: on-device listening, the bot's own voice, one side talks at a time. It needs the macOS app on This computer.
• Live runs the conversation through OpenAI GPT-Live. You can talk over it and interrupt at any time. It works on the desktop, iPhone and Android.

A Live call keeps the chat open: a call bar above the composer shows the call time and a caption of what the voice says, with a settings gear, Mute and Hang up. On the desktop, Cmd+Shift+M mutes or unmutes and Cmd+Shift+H hangs up (Ctrl+Shift+M and Ctrl+Shift+H on Windows and Linux).

The voice speaks as the bot, in the first person, but the bot does the work: each request goes to it as a normal message, marked "via call", with the bot's own engine, tools, memory and approvals, and the voice reads the answer back. The voice's own words appear only as captions and are not saved. While the bot works, the voice gets a quiet status note every 30 seconds, so it can say the bot is still busy instead of guessing; if a turn ends without an answer, it says the result is in the chat.

When a call that connected ends, the chat keeps one record of it: a small row with the call's length and what the bot did for the requests you spoke, its steps by name and the permission requests it raised, such as running a command, with how each was decided. The record reads "Call with" and the bot's name. With generated thread titles turned on ("features": { "llmThreadTitles": true } in ~/.mengbot/config.json), the bot's own engine, which already saw those requests, gives it a short title from what you asked on the call; the call sends nothing more to OpenAI. The requests you spoke and the bot's answers stay in the chat as before. The record holds no speech: the voice's own words are still not saved. A call that never connected leaves no record, and a phone app that does not know the record yet shows it as one short line, such as "Call with Ada · 1:42".

A permission request, such as running a command, can be answered with a clear spoken yes or no, decided by MengBot, not by the voice model; the card stays on screen too, and a card decided on the call is marked "by voice" there and in the decision log. On a Live call the microphone stays open while the voice speaks, so "okay", "sure" or "fine" on their own do not count as a yes, and words heard while the voice itself was talking are left out; say "yes" or "no". A bot's changes to itself, such as a new skill or a routine, apply without a card and can be undone on screen; proposals that change another bot or the team, such as a team setup, are decided on screen only, and when the bot waits for you to connect an app or enter a credential, the voice tells you to do it in the chat (never to say a credential aloud).

A message typed during a call is answered by the bot, and the voice reads that answer once. With Read replies to typed messages off (under the gear), messages you type during a call and the bot's answers to them are not sent to OpenAI, with two exceptions. The voice still hears what the bot asks of you while it works on a typed message: an approval or a question (you can answer those on the call), a proposal, and a request to connect an app or enter a credential. And an answer that also covers something you said aloud is read out: lines that waited in the queue together get one answer.

A Live call hangs up after 5 minutes in which nobody speaks and the bot is not working (the gear changes the minutes). It drops if its audio does not connect within 20 seconds, and on a phone it ends when you switch away from the app or lock the phone. It also ends at once when the phone that started it is unpaired, or the sign-in that started it is signed out or revoked. Voice notes do not play while you are on a Live call.

Live calls are free for everyone, from the desktop app, a web browser or your phone; no plan is needed. They use your own OpenAI API key, from a project with GPT-Live access: the OpenAI key in Settings → API keys, or a separate key for Live calls set from the call's settings, which wins when both are saved. Without a key, the Live button says "Live calls use your own OpenAI key. Add it in Settings → API keys." and offers Open API keys; a phone shows the same message, since keys are added on your computer. The key stays where you saved it, on This computer or on My Cloud; phones never receive it. OpenAI bills Live audio at about $0.05 per minute; the bot's own work is billed as usual. Live calls work in any desktop build with a microphone (no on-device dictation needed), and are for one bot at a time. Room calls always take turns.

Live calls through Azure OpenAI. If your organisation runs OpenAI models on Azure, a Live call can use a GPT-Live deployment on your own Azure OpenAI or Microsoft Foundry resource instead of OpenAI. Deploy gpt-live-1 on the resource, then in the call's settings (or the prompt the Live button shows when a call is not set up) choose Azure OpenAI as the Service, paste the resource endpoint (for example https://my-resource.openai.azure.com) and the deployment name (empty means gpt-live-1), and add a key of that resource. Azure bills the call to that resource. The OpenAI API key is never used for Azure. A saved Live key goes only to the service it was saved for and, on Azure, to that endpoint's host: changing the Service or the endpoint's host asks for the key again, even if you change it back. Calls go only to an https address on *.openai.azure.com, *.services.ai.azure.com, *.cognitiveservices.azure.com, *.openai.azure.us, *.openai.azure.cn or *.azure-api.net (API Management); any other address is refused, and so is a redirect. On a server you can set the same thing with OMB_LIVE_PROVIDER=azure, OMB_AZURE_OPENAI_LIVE_ENDPOINT, OMB_AZURE_OPENAI_LIVE_DEPLOYMENT and OMB_OPENAI_LIVE_KEY (the resource key); set together there, the key belongs to that endpoint. With Azure, everything below that goes to OpenAI goes to your Azure resource instead.

Who can make a call through Azure. Only an admin, or the owner of a server one person uses. Azure does not accept the setting that limits the caller's app to hanging up, so on an Azure call that app's connection to the voice is wider, and could carry the bot's instructions. Everyone else, such as a User on a Team Cloud or someone who is not an admin on a shared company workspace, reads "Live calls here go through Azure OpenAI, so only an admin or the owner of this server can start one. Ask an admin if you need a call." No call starts, and it never goes through OpenAI instead.

What a Live call sends to OpenAI. A Live call sends your voice to OpenAI, along with the chat's recent messages, the bot's answers and the details of any approval it asks for. The OpenAI key stays where you saved it: on your computer, or on My Cloud. The desktop says this where you choose Live and where you set it up; a phone says it once, before its first Live call, with Start call and Cancel. In detail: the chat's last eight text messages when the call starts (so "and the other one?" makes sense), every answer the bot gives on the call's chat (an answer to typed messages only while Read replies to typed messages is on, or when it also answers something you said), what a permission card asks (for example the command it wants to run), question options and proposal titles, the name of the app or credential a card waits for, the names of the bot's steps (never their arguments), and the bot's name, title and description.

Dictation

The composer microphone uses Apple's local speech recognition in the macOS desktop build. Linux and Windows dictation are not currently shipped.

Memory

Each bot can keep plain-text memory that you can inspect, correct, or delete. Memory is not a hidden remote profile; it lives with the local MengBot data.