Subtitles for every video in the house, made at home

Our library has films and series in Japanese that never came with subtitles, and a household that watches in English and in Thai. This is the pipeline we built on the AI appliance to fix that: it listens to a video, works out what language is being spoken and where, transcribes it with two speech engines that check each other, translates the result into every language the house reads, and drops a subtitle file next to the video — overnight, unattended, and without a single byte leaving the building.

What it does

Hears the language before it transcribes. Every ten seconds of speech is judged twice — by a probability from one engine and a trial decode from the other — and only agreement on real words counts. A film that switches language mid-scene gets each stretch transcribed in its own language; one that is all Japanese is never mistaken for Chinese because someone hummed.

Transcribes with two engines and a referee. Qwen3-ASR has the lowest Japanese character error rate of the open models; whisper large-v3 is the more robust listener under music and overlapping voices and the better speller of names. Both decode every stretch of audio. Where they agree, done. Where they differ, the appliance’s own language model reads both versions and writes the one transcript — keeping every line either engine heard, taking the sounds from one and the kanji from the other.

Times every line from the engines themselves. Each character of the final transcript inherits the timestamp of the engine it came from; a forced aligner gives frame-accurate timing where it holds, and whisper’s attention-based timing takes over where it doesn’t. Nothing is timed by guesswork.

Translates into every language the house reads. English and Thai by default, each with its own subtitle file. The translator is chosen per language pair, works in windows of twenty lines with the surrounding dialogue as context, and is told the genre — a cheeky five-year-old in a comedy stays cheeky in Thai, with the particles a child would use, not the ones a bank clerk would.

Respects what is already there. A video with an embedded English track is left alone for English; the same track becomes the source for the Thai translation, because a human translation beats any transcription. A subtitle file already beside the video is reused the same way, so adding a new language to a finished folder is a translation pass, not a rerun.

Runs as a service. Jobs go into a queue; a worker takes them one at a time, waits for the network shares and the GPU after a reboot, resumes from its checkpoints, and reports to a browser front end where you can queue a folder or a file, watch the live log, and see which files have which languages.

The finding that made it work

The first versions transcribed beautifully and still missed whole conversations. It took a measuring tool rather than a better model to find out why: a voice-activity detector was deciding which parts of the audio the speech engines were allowed to hear, and on animation with a music bed it simply does not register a lot of dialogue. Thirty seconds of an argument, gone, because the detector called it music. Every downstream improvement — better engines, better merging — could only work on what it was given.

So the detector was demoted. The engines now hear the whole timeline, cut into thirty-second pieces at the quietest moments, skipping only stretches that are silent by energy. The detector keeps two advisory jobs — choosing speech-dense windows for language identification, and suggesting cut points — and never again decides what gets transcribed. Anything the engines invent over music is dealt with where it should be: by two engines disagreeing, by a referee that knows what a looping artefact looks like, and by a gate that checks a doubtful line has sound under it before keeping it.

On the episode that exposed the problem, the change recovered eight minutes of dialogue the engines had never been allowed to hear.

Thirty seconds is a rule, not a preference. Every published integration of the Japanese engine feeds it thirty-second pieces, and its own benchmarks were run that way. Longer pieces make the decoder skim or fall silent over music, and make its forced aligner collapse whole runs of words onto a single timestamp — on a sparse film, half the words. Short, speech-dense pieces keep both on track, and decode faster into the bargain.

How a run works

Point it at a folder and it works in rounds of ten files, so subtitles appear every round rather than at the end of the night. For each file:

Embedded tracks first. A text track in a target language means that target is done. A text track in the spoken language becomes the transcript and the audio stages are skipped entirely.

Audio, energy, language. The audio is extracted to a 16 kHz wav, the energy profile decides what is sound and what is silence, and the language detector labels every stretch of speech.

Two transcriptions. Both engines decode every piece with the detected language forced — the accurate mode — and the script of each result is checked against the label; a mismatch is decoded again with the language free rather than accepted.

The referee. Once the engines have left the GPU, the language model reconciles the pieces where they disagreed — on a lively anime episode that is most of them — and the lines are timed from the engines’ own timelines.

Cues and the gate. Words become cues at sentence ends and pauses. A cue the detector never saw is kept only if it looks like speech and has sound under it; filler, loops and stock phrases over silence go.

Translation, one model at a time. The round is translated target by target so the card holds one translator at a time; lines already in the target language are copied through, cleaned. Thai lines wrap without breaking a vowel from its consonant.

One file per language. film.en.srt, film.th.srt, written atomically, beside the video, where every player finds them.

Rules that came from measurement, not opinion

Each of these was a change made after a tool showed the size of the problem, and each one is checkable again on any file by re-running the tool.

Agreement without words is worth nothing. Two detectors agreeing that a stretch of humming is Chinese is how twenty-one Japanese films were once skipped. A window only counts when there are words in it.

Judge a doubtful line by what it is. “No detector activity” used to mean “drop”. It now means “look closer”: is it made of words, is it a known stock phrase, is it physically speakable in its duration, is there sound under it.

A model’s working conditions are part of its accuracy. The same engine that misses half of a four-minute piece is the best there is on a thirty-second one.

Register is a translation decision. Thai carries politeness in its particles. Without a rule, every character was translated as a well-brought-up adult. With the genre and one standing rule, a rude child is rude.

Keep the singing. The referee was once told that lyrics are artefacts and quietly dropped the opening song. Sung lines are content.

What it runs on

One GPU with 24 GB. Both speech engines sit on the card together during transcription; the translator takes it over afterwards. Everything here runs on the GPU box in our setup — the same card that watches the cameras.

Two language models, served locally. One for Japanese-to-English and for refereeing the engines, a larger multilingual one for everything into Thai and for every other language pair — 28 source languages are covered.

ffmpeg and a network share. Videos stay on the NAS; only the 16 kHz audio is pulled across, and the next file’s audio is extracted while the current one transcribes.

Nothing else. No account, no upload, no per-minute bill. A twenty-four-minute episode takes about twenty minutes for both languages; a full season runs itself overnight.

Where it fits

The GPU box in the rack does the listening; its sibling, the N5 AI MAX, runs the house’s voice assistant on its own hardware, so an overnight subtitling job never competes with someone asking for the lights. Subtitling is simply the best use of a 24 GB card while everyone is asleep, and the same rules apply as to everything else we run — measure before changing, fix at the layer that can hold the fix, and never let a component that cannot be trusted make a decision for one that can. If your library has the same problem, write to us.