Manual:Speech to Text

DRAFT — This page documents functionality that is not yet released and is subject to change. Do not follow this text until release.

Mudlet can turn your speech into game commands: press a microphone button, say get torch, and the words land on your command line, ready to send. Recognition runs on your own computer, and nothing is switched on until you set it up. Speech to text was built for accessibility first, for playing when typing is difficult, but it is there for anyone who would rather talk than type.

What is Speech to Text?

Speech recognition in Mudlet is made of up to three parts, so your client carries nothing extra until you ask for it:

• A speech engine — the software that decodes audio. Mudlet supports three: Vosk and sherpa-onnx, which you install yourself, and the recognizer built into macOS, which is already there.
• A language model — the data files for your spoken language. Models belong to a particular engine and install into Mudlet's own folders. The macOS recognizer needs none.
• The MudletSTT package — the player-facing controls: the microphone button, the stt alias, and the settings below. Without it the feature is available to scripts only.

Which engine you get.

Note: On a Mac you can skip the engine and the model entirely. If you install neither Vosk nor sherpa-onnx, Mudlet uses the system recognizer, which needs no download and no disk space. If you do install one of the others, Mudlet uses that instead.

Privacy

• Recognition is on-device for every engine Mudlet ships: audio and transcripts stay on your computer. Mudlet requires the macOS recognizer's on-device mode, so it cannot fall back to Apple's servers. A script can check for itself with stt.getInfo().capabilities.onDevice, which any engine must claim deliberately; anything that has not claimed it reports false.
• Your operating system will ask for microphone permission, and on macOS for speech recognition as well. You can change your answers later in System Settings → Privacy & Security.
• The microphone closes when you stop listening, and while it is open Mudlet shows it in two places: the microphone button pulses, and the window title gains (listening). An open microphone stays visible even when the window is minimized or behind something else.
• You can check what you are installing. Both packages are published straight from their source repositories by CI, so their package page shows a Published from source by CI panel naming the repository, the workflow and the exact commit a release was built from.

Setting it up

Speech to text needs Mudlet [CONFIRM: the release that introduces the speech API; see the talk page] or later, with MudletSTT 1.5.1 and MudletMCVP 2.1.0. Earlier versions, 5.0.1 included, have no speech support at all: neither the microphone button nor the stt alias exists there, so there is nothing to fall back to.

1. Install the package

Install STT with mpkg:

mpkg install STT

That is the whole installation on macOS. On Windows and Linux, carry on to step 2.

2. Install an engine and a model

On macOS, skip this step unless you want a different engine (see the note above).

There is no one-click engine pack yet, so this is a manual step: place the engine library in the folder stt.getLibraryPath() names, and unpack models into stt.getModelPath().

Run lua stt.getPlatformKey() to see which build your machine needs: macos, windows-x64, windows-x86, linux-x86_64 or linux-aarch64. It returns nothing on platforms with no published build (Windows on ARM, and 32-bit Linux), where Vosk and sherpa-onnx are not an option.

The folder a model is unpacked into decides which engine loads it: Mudlet recognizes each engine's layout, so stt model can switch between a Vosk model and a sherpa one.

3. Check

stt status

This reports the engine, the loaded model, and every setting. stt models lists the models Mudlet found.

[PASTE: real stt status output from a working setup, as text in a <pre> block rather than a screenshot; see the talk page]

Using it

Toggle listening any of three ways; they are all the same control:

• type stt (or stt on / stt off),
• click the Speech microphone button on the main toolbar,
• or use Options → Extensions → Speech in the menu (on macOS, the menu bar at the top of the screen).

Then speak. While you talk, the words appear live in your command line; when you pause, the finished phrase replaces them, ready to send with Enter. If you would rather Mudlet sends each phrase straight to the game, turn on stt autosend on.

you say:            get torch
command line:       get torch          ← press Enter to send

One microphone, several profiles

Mudlet has a single recognizer, so only one profile can be listening at a time. Starting speech in a second profile takes the microphone: the first profile's session ends there and then, and that game is told it has lost it.

What that means in practice:

• Switching games mid-phrase does not move the session with you. Results belong to the game the words were spoken to, even if another game is in front when you finish.
• From the profile that is listening, the control stops the session as you would expect. From any other profile it takes the microphone instead, because from there the control reads as off however busy the recognizer is.
• A profile cannot switch off a session another profile started. stt off there does nothing: it has no session of its own to stop, and it will not reach across and stop the other game's.

Settings

All settings are saved per profile and survive restarts.

Command	What it does
stt autosend on|off	Send finished phrases to the game instead of the command line.
stt preview on|off	Show words live in the command line while you speak.
stt correct on|off	Correct finished phrases against the game's vocabulary (needs MCVP, below). On by default.
stt focus stop|keep	Whether listening stops when Mudlet is not the active application. stop by default: a microphone left open to a game nobody is playing is the more expensive mistake. keep suits dictating while reading something in another window.
stt lowercase on|off	Lowercase the first letter, the way commands are typed.
stt sensitivity short|default|long	How soon a phrase counts as finished. default is the default even for commands, because short can cut a one-word command off before the decoder has emitted it. long suits dictation. Not every engine can be tuned this way: the macOS recognizer decides its own phrase endings, and Vosk needs a library carrying the endpointer. stt status says so when yours cannot.
stt timeout <ms>	Stop listening after this much silence; 0 keeps listening open-ended.
stt bias on|off	Bias the decoder toward the game's vocabulary. On by default where the engine and model support it. stt status shows the setting and says so when the loaded model cannot honor it, and stt vocab shows what the biasing budget is being spent on. A few phrases come out worse under biasing, which is what off is for, and stt test measures the difference on your own setup.
stt model <name>	Load a different installed model, switching engine if that model belongs to another one. Your choice is remembered.
stt models	List installed speech models.
stt vocab	Which of the game's published words can be spoken at all, and what the biasing budget is being spent on. Aimed at game authors working on a catalog.
stt status	Engine, state and settings at a glance.

What each engine can do

Engines differ. stt status shows what the one you are running can do.

	Vosk	sherpa-onnx	macOS built-in
Anything to install	library + model	library + model	nothing
Bias toward your game's words	no	with a model that carries the right vocabulary file	yes
Per-word timing and confidence (sysSTTWords)	yes	not yet	yes
Tune how soon a phrase ends (stt sensitivity)	with a library that carries the endpointer	yes	no, it decides for itself
Runs on	Windows, Linux, macOS	Windows, Linux, macOS	macOS only

Note: These are the general shape of things; some of them are properties of the model rather than the engine, so they can change when you load a different one. stt status always answers for what you are actually running.

Hearing your game's words

Speech engines know everyday English; they do not know your game's verbs, socials and monster names. Two things close that gap:

• Correction: finished phrases are corrected against the vocabulary your game publishes, so a near-miss becomes the word the game actually accepts. This needs a game that supports the MUD Client Vocabulary Protocol. The MCVP package that reads it is a dependency of STT, so mpkg install STT brings it in, and there is nothing separate to install. It works with every engine, because the correction happens after recognition.
• What is in reach: the names of things in your room and inventory join in automatically on supporting games, so get talisman works on the talisman actually in front of you.

Where biasing and correction each act.

Biasing is the stronger version of the same idea: it steers the decoder itself toward those words instead of fixing the result afterward. Only some engines and models support it. See the table above.

On games without vocabulary support, speech to text still works; you get the engine's plain recognition.

Checking how well it hears you

stt test              ← a fixed phrase list; nothing is sent to the game
stt test 3            ← the same list three times, so one odd run cannot carry the result
stt test scope        ← phrases naming what is in this room and your inventory
stt test game         ← phrases from this game's published catalog
stt test phrases score guild; guild score
                      ← your own phrases, separated by semicolons
stt test repeat       ← the last set again, so two runs compare like for like
stt test stop         ← abandon a run in progress

Each of the run forms takes a trailing number for the pass count; stop does not. Each run scores what was heard against what was asked (exact matches, word errors, and which phrases failed), so you can compare engines, models, sensitivities and biasing on numbers rather than impressions.

Troubleshooting

Symptom	Likely cause
No microphone button	MudletSTT not installed, or your Mudlet predates speech support; see Setting it up for the version you need.
"This Mudlet build has no speech-to-text support"	Your Mudlet predates the speech API. Update Mudlet.
"the speech engine library is not installed … looked in: …"	No engine is installed and there is no built-in one on this platform. The message lists the exact folders searched.
"no model path provided and no language model is installed - install one into …"	The engine is there but no model is. The message names the folder to unpack into.
"another profile is listening, and only the profile that started a session can stop it"	Switch to the profile that is listening and stop it there, or start speech here to take the microphone. See One microphone, several profiles.
Speech stopped on its own	Another profile took the microphone. Only one profile can listen at a time.
Microphone permission denied	Grant it in your system's privacy settings, then start listening again. On macOS check both Microphone and Speech Recognition.
Listening, but nothing is heard	Run stt test: it reports the mean input level, and flags it when that is near silence. Near zero means the wrong input device or a muted microphone.
stt bias on has no effect	Read what the command answered. no model is loaded to bias means the engine has not started yet, so there is no decoder to steer: stt on loads one and applies the biasing as it does. this model cannot bias its decoding is the model itself, and stt status repeats it next to the bias setting; switch model with stt model, or see the engine table above. no game vocabulary has arrived to bias toward means this game has published no catalog. takes effect at the next model load means the request was kept but the decoder is still running with the old word list; stop listening and ask again.
stt sensitivity has no effect	This engine decides its own phrase endings, and stt status says so.
A Mac ignores the built-in recognizer	You have Vosk or sherpa-onnx installed, and Mudlet prefers those. Remove the engine library, or name the built-in one explicitly.

For scripters

Everything above is built on the stt.* Lua API, which any script or package can use directly. Return values follow one convention: true means done, and nil plus a message is a refusal or a fault. The message says which, and the same text arrives as sysSTTError.

Two kinds of refusal answer in the return value alone. The first is one your own call caused: a malformed argument, or a model path handed to a backend that loads no models, which is your package's mistake and not news for every other package on the profile. The second is stt.setSensitivity() on an engine whose capabilities.sensitivityTuning is false, because that is an answer about the engine rather than a fault, and announcing it would fire on every start for a package that reapplies its saved settings. Everything the engine itself refused speaks, stt.stop() in the error state included. The queries return plain booleans, and stt.setVocabulary() returns a boolean that answers can this engine use these words rather than reporting success or failure.

The full contract is docs/stt-api.md in Mudlet's own source, written so other clients can implement the same surface.

Group	Functions
Lifecycle	stt.init([modelPath]), stt.start(), stt.stop(), stt.toggle(), stt.close()
Queries	stt.available(), stt.initialized(), stt.listening(), stt.getInfo()
Tuning	stt.setSilenceTimeout(ms), stt.setSensitivity(mode), stt.setVocabulary(words)
Installation	stt.getModelPath(), stt.getLibraryPath(), stt.listModels(), stt.getPlatformKey(), stt.reloadLibrary(), stt.unloadLibrary()

stt.init(modelPath) picks the engine from that folder's layout, so a script never names an engine. stt.init() with no argument loads the default installed model, or the built-in macOS recognizer when no other engine is installed.

stt.getInfo() reports backend, state, available, initialized, listening, modelPath, language, version, audioLevel, sensitivity, silenceTimeout, searchPaths and a capabilities table of biasing, grammar, words, sensitivityTuning and onDevice. Capabilities differ per engine, and on some engines per model, so do not cache them at startup: re-read them after stt.init(), or follow sysSTTCapabilitiesChanged.

stt.reloadLibrary() and stt.unloadLibrary() are the exception to that convention: they answer true, or false plus a message, and never nil. A refusal while the engine is in use says so, and arrives as sysSTTError as well. A reload that simply finds nothing installed is not a refusal and carries no message: it answers false on its own, and what that answers is whether speech can be had at all afterwards rather than whether a library in particular turned up.

Note: stt.reloadLibrary() re-runs detection for both Vosk and sherpa-onnx, so a library installed while Mudlet is running is found without a restart. stt.unloadLibrary() still acts on Vosk alone: with sherpa-onnx loaded it refuses and names that engine rather than reporting an unload it cannot perform, so replacing a sherpa-onnx library on Windows means quitting Mudlet. The macOS recognizer loads no library of its own, but it is still an engine: both calls refuse while any recognizer is loaded, so stt.close() comes first.

Which profile an event reaches

There is one recognizer per client, so at most one profile can be listening. Events are raised on the profile holding the microphone, the one whose stt.start() began the session, and on the active profile when no session is running. Those are the same profile in the ordinary case, and differ when the player moves to another game mid-session.

A second profile calling stt.start() or stt.toggle() takes the microphone: the running session stops and the profile that lost it is told through sysSTTHandover. stt.toggle() therefore only ever stops a session the calling profile owns; from anywhere else it is a request for the microphone. stt.stop() refuses outright from a profile that does not hold the session.

sysSTTCapabilitiesChanged follows the same rule, which is worth knowing because it does not describe a session: a model loaded or a library reloaded while another profile holds the microphone is announced there, not to every profile that could read the same answer back from stt.getInfo().

Every handler receives two string arguments: the event name, then the payload below.

Event	Carries	When
sysSTTResult	text	A phrase was finished.
sysSTTPartialResult	text	Words recognized so far, while speaking.
sysSTTWords	JSON	Per-word timing and confidence, alongside each result, on engines whose capabilities.words is true. Timings are in seconds, but which clock they are on is the engine's own: the macOS recognizer restarts them with each phrase, while Vosk accumulates them across the whole listening session, so treat them as relative within one payload. sherpa-onnx does not raise this event at all.
sysSTTStateChanged	state	uninitialized, ready, starting, listening, processing or error.
sysSTTHandover	profile name	Another profile took the microphone and this session is over. Raised on the profile that lost it, naming the one that now holds it. Nothing else says why a session stopped; the state change that follows looks like any ordinary stop.
sysSTTCapabilitiesChanged	JSON	The capabilities table changed: a model loaded or was released, an engine was created or swapped, or the engine library was unloaded or reloaded. Same keys as getInfo().capabilities.
sysSTTError	message	Something went wrong, in words a player can act on.

The states a recognizer moves through.

starting means the request was accepted but audio is not flowing yet, usually because of a permission prompt. A successful stt.start() is not proof that the microphone is open; wait for listening.

registerAnonymousEventHandler("sysSTTResult", function(_, text)
  echo("Heard: " .. text .. "\n")
end)

registerAnonymousEventHandler("sysSTTHandover", function(_, profile)
  echo("Speech moved to " .. profile .. "\n")
end)

local ok, err = stt.init()   -- the default installed model, or the built-in one on macOS
if ok then stt.start() else echo(err .. "\n") end

See also

• Manual:Mudlet Packages — installing packages, mpkg, and the Package Repository
• Standards:MUD Client Vocabulary Protocol — how games publish their vocabulary
• MudletSTT and MudletMCVP on the package repository