Project · Open source · Speech recognition · Local AI · October 2026

mlsubgen — subtitles for every video in the house, made at home and now open source

Our library has films and series in Japanese that never came with subtitles, and a household that watches in English and in Thai. This is the tool we built on the AI appliance to fix that: it listens to a video, works out what language is being spoken and where, transcribes it with two speech engines that check each other, translates the result into whichever of 45 languages you tick, and drops a subtitle file next to the video — overnight, unattended, and without a single byte leaving the building. As of October 2026 it is on GitHub for anyone with an NVIDIA card to run.

github.com/dodgypast/mlsubgen Apache-2.0 · Python · Docker or host install · 8 GB GPU and up

What it does

Hears the language before it transcribes. Every ten seconds of speech is judged twice — by a probability from one engine and a trial decode from the other — and only agreement on real words counts. A film that switches language mid-scene gets each stretch transcribed in its own language; one that is all Japanese is never mistaken for Chinese because someone hummed.
Transcribes with two engines and a referee. Qwen3-ASR has the lowest Japanese character error rate of the open models; whisper large-v3 is the more robust listener under music and overlapping voices and the better speller of names. Both decode every stretch of audio. Where they agree, done. Where they differ, the appliance’s own language model reads both versions and writes the one transcript — keeping every line either engine heard, taking the sounds from one and the kanji from the other.
Times every line from the engines themselves. Each character of the final transcript inherits the timestamp of the engine it came from; a forced aligner gives frame-accurate timing where it holds, and whisper’s attention-based timing takes over where it doesn’t. Nothing is timed by guesswork.
Translates into 45 languages, one file each. Tick the languages in the browser front end or pass --target en,th,de. The translator is chosen per language pair, works in windows of twenty lines with the surrounding dialogue as context, and is told the genre — a cheeky five-year-old in a comedy stays cheeky in Thai, with the particles a child would use, not the ones a bank clerk would. Chinese comes out Simplified, Cantonese Traditional; Japanese and Chinese keep their own punctuation and get a slower reading speed; Thai, Lao, Khmer and Burmese never break a line between a consonant and its vowel.
Uses what is already there before it listens. A video with an embedded English track is left alone for English. Any other full subtitle track — in any of the 45 languages, as long as it is not a forced or signs-and-songs track — becomes the source for the missing languages, the spoken language’s track first, because a human translation beats any transcription and the original beats a translation of a translation. A subtitle file already beside the video is reused the same way, so adding a language to a finished folder is a translation pass, not a rerun.
Runs as a service. Jobs go into a queue; a worker takes them one at a time, waits for the network shares and the GPU after a reboot, checkpoints every stage and every decoded chunk, and reports to a browser front end where you queue a folder or a file, preview exactly which files a job would touch, watch the live log, pause a job across reboots and resume it, and pull the models with a click.
Knows who is speaking, if you ask it to. An optional pass finds the speaker turns on the CPU, from pyannote’s segmentation model and a speaker-embedding model that sherpa-onnx distributes from GitHub — no account, no token. A change of voice closes a subtitle, the language detector works one voice at a time, and the translator is told which lines share a voice, with the rule that the tags mean only “same voice” and never appear in the output. Where the evidence is thin or muddled the file is processed as if the option were off.

Run it yourself

Linux with an NVIDIA GPU, ffmpeg, Docker with the NVIDIA Container Toolkit, and Ollama on the same machine for the translator. The only thing to fill in is where your videos are.

git clone https://github.com/dodgypast/mlsubgen.git && cd mlsubgen
cp .env.example .env                  # MEDIA_DIR = your videos; your uid/gid; default languages
mkdir -p data models                  # state and the model cache, created by you so the containers can write them
docker compose up -d --build          # ~14 GB image (CUDA torch and the speech stacks), 10-20 minutes the first time
docker compose run --rm worker pull   # the speech models (~8 GB) and the translators your card can hold
docker compose run --rm worker pull speakers   # optional: the two speaker models (33 MB) for --speakers

Then open http://<host>:8790, queue a folder, and watch the log. There is a host install too (setup.sh, user units for the worker and the web UI), a command line for everything the browser does, and mlsubgen --help that explains itself. The card decides the rest:

GPU	profile	what changes
24 GB	`full`	both speech engines resident; the 27–31B translators we use here
12–20 GB	`12gb`	whisper in int8; a 12B translator (7 GB) for every language pair
8 GB	`8gb`	one speech engine on the card at a time; a 4B translator for every pair

Picked from the GPU’s memory at start, or forced with MLSUBGEN_PROFILE. The smaller translators are noticeably weaker, most of all for Japanese to English; mlsubgen bench runs the speech stage once and every translator on the same lines, side by side, so you can see by how much on your own material before a long run.

About the code. It was written with the help of Claude (Anthropic’s model), reviewed line by line and run in daily use on our own library — a hobby project maintained by one person, not a product. Issues on GitHub get answered; the pinned dependencies in the repository are the reference environment, and “works for me” means works there.

The finding that made it work

The first versions transcribed beautifully and still missed whole conversations. It took a measuring tool rather than a better model to find out why: a voice-activity detector was deciding which parts of the audio the speech engines were allowed to hear, and on animation with a music bed it simply does not register a lot of dialogue. Thirty seconds of an argument, gone, because the detector called it music. Every downstream improvement — better engines, better merging — could only work on what it was given.

So the detector was demoted. The engines now hear the whole timeline, cut into thirty-second pieces at the quietest moments, skipping only stretches that are silent by energy. The detector keeps two advisory jobs — choosing speech-dense windows for language identification, and suggesting cut points — and never again decides what gets transcribed. Anything the engines invent over music is dealt with where it should be: by two engines disagreeing, by a referee that knows what a looping artefact looks like, and by a gate that checks a doubtful line has sound under it before keeping it.

On the episode that exposed the problem, the change recovered eight minutes of dialogue the engines had never been allowed to hear.

Thirty seconds is a rule, not a preference. Every published integration of the Japanese engine feeds it thirty-second pieces, and its own benchmarks were run that way. Longer pieces make the decoder skim or fall silent over music, and make its forced aligner collapse whole runs of words onto a single timestamp — on a sparse film, half the words. Short, speech-dense pieces keep both on track, and decode faster into the bargain.

How a run works

Point it at a folder and it works in rounds of ten files, so subtitles appear every round rather than at the end of the night. For each file:

Embedded tracks first. A text track in a target language means that target is done. Any other full text track — the spoken language’s first — becomes the transcript, and the audio stages are skipped entirely.
Audio, energy, language. The audio is extracted to a 16 kHz wav, the energy profile decides what is sound and what is silence, and the language detector labels every stretch of speech.
Two transcriptions. Both engines decode every piece with the detected language forced — the accurate mode — and the script of each result is checked against the label; a mismatch is decoded again with the language free rather than accepted. Every decoded piece is checkpointed, so a pause or a power cut costs at most one.
The referee. Once the engines have left the GPU, the language model reconciles the pieces where they disagreed — on a lively anime episode that is most of them — and the lines are timed from the engines’ own timelines.
Cues and the gate. Words become cues at sentence ends and pauses. A cue the detector never saw is kept only if it looks like speech and has sound under it; filler, loops and stock phrases over silence go.
Translation, one model at a time. The round is translated target by target so the card holds one translator at a time; lines already in the target language are copied through, cleaned. Line width and reading speed follow the language.
One file per language. film.en.srt, film.th.srt, written atomically, beside the video, where every player finds them.

Rules that came from measurement, not opinion

Each of these was a change made after a tool showed the size of the problem, and each one is checkable again on any file by re-running the tool.

Agreement without words is worth nothing. Two detectors agreeing that a stretch of humming is Chinese is how twenty-one Japanese films were once skipped. A window only counts when there are words in it.
Judge a doubtful line by what it is. “No detector activity” used to mean “drop”. It now means “look closer”: is it made of words, is it a known stock phrase, is it physically speakable in its duration, is there sound under it.
A model’s working conditions are part of its accuracy. The same engine that misses half of a four-minute piece is the best there is on a thirty-second one.
Register is a translation decision. Thai carries politeness in its particles. Without a rule, every character was translated as a well-brought-up adult. With the genre and one standing rule, a rude child is rude. The same goes for du and Sie, tu and vous, and Japanese and Korean speech levels.
Keep the singing. The referee was once told that lyrics are artefacts and quietly dropped the opening song. Sung lines are content.
A checkpoint is only as fine as the work you can afford to lose. Per-file checkpoints meant a pause at chunk 144 of 247 threw away 144 chunks. The ASR now checkpoints after every chunk, and a pause survives reboots until you resume it.
A forced subtitle track is a free ground truth — and only a lower bound. On a multilingual film, the track that subtitles only the foreign-language lines says where foreign speech is. Scoring the language detector against it on Babel, Inglourious Basterds and Only God Forgives showed it found most foreign speech but reproduced only half the language changes within three seconds: the ten-second detection window was taking one label across a switch. Moving the switch to the exact span, letting the words tell English from Italian, and letting one strongly evidenced window survive the smoothing lifted that to two thirds, and Italian appeared where it had been missed entirely. The same scoring also showed that every “false alarm” we checked by ear was the detector being right where the subtitler stayed silent — singing, a “[speaking German]” card — so verified stretches are now excluded from the score rather than counted against it.
Measure the piece you can change, not the one you hope moved. Speaker turns helped the detector where voices map to languages and were neutral where they don’t; five speaker-embedding models made no difference to detection at all, because the useful part — the turn boundaries — comes from the segmentation model, while what the embedding changes is how well the turns cluster into characters. The model that clustered best went in as the default; the feature stays off until the bench says otherwise for a given library.

What it runs on here

One GPU with 24 GB. Both speech engines sit on the card together during transcription; the translator takes it over afterwards. Everything here runs on the GPU box in our setup — the same card that watches the cameras.
Two language models, served locally. One for Japanese-to-English and for refereeing the engines, a larger multilingual one for everything into Thai and for every other language pair. Smaller cards get smaller ones automatically; see the profiles above.
ffmpeg and a network share. Videos stay on the NAS; only the 16 kHz audio is pulled across, and the next file’s audio is extracted while the current one transcribes.
Nothing else. No account, no upload, no per-minute bill. A twenty-four-minute episode takes about twenty minutes for both languages; a full season runs itself overnight.

Where it fits

The GPU box in the rack does the listening; its sibling, the N5 AI MAX, runs the house’s voice assistant on its own hardware, so an overnight subtitling job never competes with someone asking for the lights. Subtitling is simply the best use of a 24 GB card while everyone is asleep, and the same rules apply as to everything else we run — measure before changing, fix at the layer that can hold the fix, and never let a component that cannot be trusted make a decision for one that can. If your library has the same problem, the code is on GitHub; if your house does, write to us.