Manual:Speech to Text

DRAFT — This page is under discussion and subject to change. Do not implement against this text.

Mudlet can turn your speech into game commands: press a microphone button, say get torch, and the words land on your command line, ready to send. Recognition runs on your own computer, and nothing is switched on until you set it up. Speech to text was built for accessibility first — playing when typing is difficult — but it is there for anyone who would rather talk than type.

What is Speech to Text?

Speech recognition in Mudlet is made of up to three parts, so your client carries nothing extra until you ask for it:

• A speech engine — the software that decodes audio. Mudlet supports three: Vosk and sherpa-onnx, which you install yourself, and the recognizer built into macOS, which is already there.
• A language model — the data files for your spoken language. Models belong to a particular engine and install into Mudlet's own folders. The macOS recognizer needs none.
• The MudletSTT package — the player-facing controls: the microphone button, the stt alias, and the settings below. Without it the feature is available to scripts only.

Which engine you get.

Note: On a Mac you can skip the engine and the model entirely. If you install neither Vosk nor sherpa-onnx, Mudlet uses the system recognizer: no download, no model, no disk space. If you do install one of the others, Mudlet uses that instead.

Privacy

• Recognition is on-device for every engine Mudlet ships: audio and transcripts stay on your computer. The macOS recognizer is explicitly held to this — Mudlet requires its on-device mode rather than letting it fall back to Apple's servers. A script can check for itself with stt.getInfo().capabilities.onDevice, which any engine must claim deliberately; anything that has not claimed it reports false.
• Your operating system will ask for microphone permission, and on macOS for speech recognition as well. You can change your answers later in System Settings → Privacy & Security.
• The microphone closes when you stop listening, and while it is open Mudlet shows it in two places: the microphone button pulses, and the window title gains (listening) — so an open microphone stays visible even when the window is minimised or behind something else.
• You can check what you are installing. Both packages are published straight from their source repositories by CI, so their package page shows a Published from source by CI panel naming the repository, the workflow and the exact commit a release was built from — worth a look for anything you are granting microphone access to.

Setting it up

Speech to text needs Mudlet [CONFIRM: the release that introduces the speech API — see the talk page] or later, with MudletSTT 1.5.0 and MudletMCVP 2.1.0. Earlier versions, 5.0.1 included, have no speech support at all: neither the microphone button nor the stt alias exists there, so there is nothing to fall back to.

1. Install the package

Install STT with mpkg:

mpkg install STT

That is the whole installation on macOS. On Windows and Linux, carry on to step 2.

2. Install an engine and a model

On macOS, skip this step unless you want a different engine — see the note above.

There is no one-click engine pack yet, so this is a manual step: place the engine library in the folder stt.getLibraryPath() names, and unpack models into stt.getModelPath().

Run lua stt.getPlatformKey() to see which build your machine needs: macos, windows-x64, windows-x86, linux-x86_64 or linux-aarch64. It returns nothing on platforms with no published build — Windows on ARM, and 32-bit Linux — where Vosk and sherpa-onnx are not an option.

The folder a model is unpacked into decides which engine loads it: Mudlet recognizes each engine's layout, so stt model switching between a Vosk model and a sherpa one just works.

3. Check

stt status

This reports the engine, the loaded model, and every setting. stt models lists the models Mudlet found.

[PASTE: real stt status output from a working setup, as text in a <pre> block rather than a screenshot — see the talk page]

Using it

Toggle listening any of three ways — they are all the same control:

• type stt (or stt on / stt off),
• click the Speech microphone button on the main toolbar,
• or use Options → Extensions → Speech in the menu — on macOS that is the menu bar at the top of the screen.

Then speak. While you talk, the words appear live in your command line; when you pause, the finished phrase replaces them, ready to send with Enter. If you would rather Mudlet sends each phrase straight to the game, turn on stt autosend on.

you say:            get torch
command line:       get torch          ← press Enter to send

One microphone, several profiles

Mudlet has a single recognizer, so only one profile can be listening at a time. Starting speech in a second profile takes the microphone: the first profile's session ends there and then, and that game is told it has lost it.

What that means in practice:

• Switching games mid-phrase does not move the session with you. Results belong to the game the words were spoken to, not to whichever game is in front when you finish.
• From the profile that is listening, the control stops the session as you would expect. From any other profile it takes the microphone instead, because from there the control reads as off however busy the recognizer is.
• A profile cannot switch off a session another profile started. stt off there simply does nothing: it has no session of its own to stop, and it will not reach across and stop the other game's.

Settings

All settings are saved per profile and survive restarts.

Command	What it does
stt autosend on|off	Send finished phrases to the game instead of the command line.
stt preview on|off	Show words live in the command line while you speak.
stt correct on|off	Correct finished phrases against the game's vocabulary (needs MCVP, below). On by default.
stt focus stop|keep	Whether listening stops when Mudlet is not the active application. stop by default: a microphone left open to a game nobody is playing is the more expensive mistake. keep suits dictating while reading something in another window.
stt lowercase on|off	Lowercase the first letter, the way commands are typed.
stt sensitivity short|default|long	How soon a phrase counts as finished. default is the default even for commands: short can cut a one-word command off before the decoder has emitted it, and a wrong command costs more than a slow one. long suits dictation. Not every engine can be tuned this way: the macOS recognizer decides its own phrase endings, and Vosk needs a library carrying the endpointer. A script can ask with stt.getInfo().capabilities.sensitivityTuning.
stt timeout <ms>	Stop listening after this much silence; 0 keeps listening open-ended.
stt bias on|off	Bias the decoder toward the game's vocabulary. On by default where the engine and model support it — stt vocab shows what the biasing budget is being spent on, and a script can ask with stt.getInfo().capabilities.biasing. It is not free: a few phrases come out worse under biasing, which is what off is for, and stt test measures the difference on your own setup.
stt model <name>	Load a different installed model, switching engine if that model belongs to another one. Your choice is remembered.
stt models	List installed speech models.
stt vocab	Which of the game's published words can be spoken at all, and what the biasing budget is being spent on. Aimed at game authors working on a catalog.
stt status	Engine, state and settings at a glance.

What each engine can do

Engines differ, and Mudlet reports the differences rather than pretending they are the same. stt status shows what the one you are running can do.

	Vosk	sherpa-onnx	macOS built-in
Anything to install	library + model	library + model	nothing
Bias toward your game's words	no	with a model that carries the right vocabulary file	yes
Per-word timing and confidence (sysSTTWords)	yes	not yet	yes
Tune how soon a phrase ends (stt sensitivity)	with a library that carries the endpointer	yes	no — it decides for itself
Runs on	Windows, Linux, macOS	Windows, Linux, macOS	macOS only

Note: These are the general shape of things; some of them are properties of the model rather than the engine, so they can change when you load a different one. stt status always answers for what you are actually running.

Hearing your game's words

Speech engines know everyday English; they do not know your game's verbs, socials and monster names. Two things close that gap:

• Correction: finished phrases are corrected against the vocabulary your game publishes, so a near-miss becomes the word the game actually accepts. This needs a game that supports the MUD Client Vocabulary Protocol. The MCVP package that reads it is a dependency of STT, so mpkg install STT brings it in — there is nothing separate to install. It works with every engine, because the correction happens after recognition.
• What is in reach: the names of things in your room and inventory join in automatically on supporting games, so get talisman works on the talisman actually in front of you.

Where biasing and correction each act.

Biasing is the stronger version of the same idea — steering the decoder itself toward those words rather than fixing the result afterward — but only some engines and models support it. See the table above.

On games without vocabulary support, speech to text still works — you simply get the engine's plain recognition.

Checking how well it hears you

stt test              ← a fixed phrase list; nothing is sent to the game
stt test 3            ← the same list three times, so one odd run cannot carry the result
stt test scope        ← phrases naming what is in this room and your inventory
stt test game         ← phrases from this game's published catalog
stt test phrases score guild; guild score
                      ← your own phrases, separated by semicolons
stt test repeat       ← the last set again, so two runs compare rather than ask different questions
stt test stop         ← abandon a run in progress

Each of the run forms takes a trailing number for the pass count; stop does not. Each run scores what was heard against what was asked — exact matches, word errors, and which phrases failed — so you can compare engines, models, sensitivities and biasing on numbers rather than impressions.

Troubleshooting

Symptom	Likely cause
No microphone button	MudletSTT not installed, or your Mudlet predates speech support — see Setting it up for the version you need.
"This Mudlet build has no speech-to-text support"	Your Mudlet predates the speech API — update Mudlet.
"the speech engine library is not installed … looked in: …"	No engine is installed and there is no built-in one on this platform. The message lists the exact folders searched.
"no model path provided and no language model is installed — install one into …"	The engine is there but no model is. The message names the folder to unpack into.
"another profile is listening, and only the profile that started a session can stop it"	Exactly what it says: switch to the profile that is listening and stop it there, or start speech here to take the microphone. See One microphone, several profiles.
Speech stopped on its own	Another profile took the microphone. Only one profile can listen at a time.
Microphone permission denied	Grant it in your system's privacy settings, then start listening again. On macOS check both Microphone and Speech Recognition.
Listening, but nothing is heard	Check the input level in stt status — near zero means the wrong input device or a muted microphone.
stt bias on has no effect	This engine or model cannot be biased. Check with lua stt.getInfo().capabilities.biasing, or switch model with stt model. See the engine table above.
stt sensitivity has no effect	This engine decides its own phrase endings. Check with lua stt.getInfo().capabilities.sensitivityTuning.
A Mac ignores the built-in recognizer	You have Vosk or sherpa-onnx installed, and Mudlet prefers those. Remove the engine library, or name the built-in one explicitly.

For scripters

Everything above is built on the stt.* Lua API, which any script or package can use directly. Return values follow one convention: true means done, and nil plus a message is a refusal or a fault — the message says which, and the same text arrives as sysSTTError. The queries return plain booleans, and stt.setVocabulary() returns a boolean that answers can this engine use these words rather than reporting success or failure.

The full contract — written so other clients can implement the same surface — is docs/stt-api.md in Mudlet's own source.

Group	Functions
Lifecycle	stt.init([modelPath]), stt.start(), stt.stop(), stt.toggle(), stt.close()
Queries	stt.available(), stt.initialized(), stt.listening(), stt.getInfo()
Tuning	stt.setSilenceTimeout(ms), stt.setSensitivity(mode), stt.setVocabulary(words)
Installation	stt.getModelPath(), stt.getLibraryPath(), stt.listModels(), stt.getPlatformKey(), stt.reloadLibrary(), stt.unloadLibrary()

stt.init(modelPath) picks the engine from that folder's layout, so a script never names an engine. stt.init() with no argument loads the default installed model, or the built-in macOS recognizer when no other engine is installed.

stt.getInfo() reports backend, state, available, initialized, listening, modelPath, language, version, audioLevel, sensitivity, silenceTimeout, searchPaths and a capabilities table of biasing, grammar, words, sensitivityTuning and onDevice. Read the capabilities rather than assuming: they differ per engine, and on some engines per model — so re-read them after stt.init(), or follow sysSTTCapabilitiesChanged, rather than caching them at startup.

Note: stt.reloadLibrary() and stt.unloadLibrary() currently act on Vosk only. They do nothing for sherpa-onnx, and are meaningless for the macOS recognizer, which loads no library.

Which profile an event reaches

There is one recognizer per client, so at most one profile can be listening. Events are raised on the profile holding the microphone — the one whose stt.start() began the session — and on the active profile when no session is running. Those are the same profile in the ordinary case, and differ when the player moves to another game mid-session.

A second profile calling stt.start() or stt.toggle() takes the microphone: the running session stops and the profile that lost it is told through sysSTTHandover. stt.toggle() therefore only ever stops a session the calling profile owns; from anywhere else it is a request for the microphone. stt.stop() refuses outright from a profile that does not hold the session.

sysSTTCapabilitiesChanged is the one exception to all of this and reaches every open profile, because it does not describe a session: what a backend can do is a property of the engine, and every profile reads the same answer back from stt.getInfo().

Every handler receives two string arguments: the event name, then the payload below.

Event	Carries	When
sysSTTResult	text	A phrase was finished.
sysSTTPartialResult	text	Words recognized so far, while speaking.
sysSTTWords	JSON	Per-word timing and confidence, alongside each result, on engines whose capabilities.words is true. Timings are seconds within the utterance on sherpa-onnx and the macOS recognizer, and run from the start of the listening session on Vosk.
sysSTTStateChanged	state	uninitialized, ready, starting, listening, processing or error.
sysSTTHandover	profile name	Another profile took the microphone and this session is over. Raised on the profile that lost it, naming the one that now holds it. Nothing else says why a session stopped — the state change that follows looks like any ordinary stop.
sysSTTCapabilitiesChanged	JSON	The capabilities table changed: a model loaded or was released, an engine was created or swapped, or the engine library was unloaded or reloaded. Raised on every open profile, unlike every other event here. Same keys as getInfo().capabilities.
sysSTTError	message	Something went wrong, in words a player can act on.

The states a recognizer moves through.

starting means the request was accepted but audio is not flowing yet — a permission prompt is usually the reason. Wait for listening rather than treating a successful stt.start() as proof the microphone is open.

registerAnonymousEventHandler("sysSTTResult", function(_, text)
  echo("Heard: " .. text .. "\n")
end)

registerAnonymousEventHandler("sysSTTHandover", function(_, profile)
  echo("Speech moved to " .. profile .. "\n")
end)

local ok, err = stt.init()   -- the default installed model, or the built-in one on macOS
if ok then stt.start() else echo(err .. "\n") end

See also

• Manual:Mudlet Packages — installing packages, mpkg, and the Package Repository
• Standards:MUD Client Vocabulary Protocol — how games publish their vocabulary
• MudletSTT and MudletMCVP on the package repository