hpr-knowledge-base/hpr_transcripts/hpr4596.txt

Episode: 4596
Title: Adding voice-over audio track created using text to speech on the movie subtitles
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4596/hpr4596.mp3
Transcribed: 2026-07-31 16:14:44 (official HPR transcript)

---

This is Hacker Public Radio Episode 4596, for 2026-03-16
Today's show is entitled, "Adding voice-over audio track created using text to speech on the movie subtitles"
The host is Ken Fallon and the duration is 00:28:09
The flag is Clean, and the license is CC-BY-SA
The summary is "Ken uses piper to convert srt subtitle files into a new audio track to add to foreign language movie"
Hi everybody, my name is Ken Fallon and you're listening to another episode of Hacker Public Radio.
Today I'm going to use the Piper text-to-speech engine to convert subtitle files into a new audio track
that you can listen to alongside a movie.
So a lot of my colleagues have been recommending movies to me and I've been asking them for their recommendations
and they're quite often foreign language films by definition, languages that I don't speak
which makes every movie I guess a foreign language movie to somebody.
The subtitles can often be distracting and particularly in a fast-paced scene
you're reading the subtitles and you're missing the action that's happening on the screen
or the acting that's happening on the screen.
So I've thought of doing this, it's called voiceover or lecturing and if you look it up in Wikipedia
it was first used in Soviet countries to read newspaper propaganda from lectors
and in Polish it says
in Polish lector is often used to mean off-screen reader or voiceover artist.
A lector is usually a male reader who provides the Polish voiceover on foreign language programs and films
where voiceover translation technique is used.
This is the standard localization technique on Polish television and as an option on many DVDs
full dubbing is generally reserved for children's material, links in the show notes.
So to give you an idea of what I mean by that
what I'll do is I'll first play you
a sample from a non-copyright movie
Night of the Living Dead
which was released mistakenly into the public domain.
So first I'll play you a sample of the movie
with the original soundtrack.
They ought to make the day the time changes
the first day of summer.
What?
Well it's eight o'clock and it's still light.
A lot of good the extra daylight does us.
Now we've still got a three hour drive back.
We're not going to be home until after midnight.
Well if it really bugged you Johnny you wouldn't do it.
You think I want to blow Sunday on a scene like this?
You know, I figure we're either going to have to move mother out here or move the grave in the Pittsburgh.
She can't make a trip like this.
Oh, don't that she can.
And now the same segment with the Polish voiceover.
So I don't know about you.
But as a native English speaker I find that difficult to follow
because the voiceover track I'm trying to focus on the underlying audio
but talking to my Polish friends
they said they didn't have a problem with that.
So if you're Polish as your native language you're listening to the Polish.
So to put this to the test I wanted to try out a movie to see if that indeed was the case.
I asked a Mastodon for a non-English movie that would have English subtitles available
and Windigo was able to point me to Nasty Old People
which is a 2009 Swedish movie released under the CC BY non-commercial share alike license.
So my idea was to take each bit of the subtitled text, convert it to audio
then have the generated audio play at the same time that the subtitles appear on the screen.
So we can use, we use Piper text-to-speech here in HPR
all the intros are done with that
and we also generate SRT sub-title files for each show
so I'm also familiar with those.
Links to all of this stuff is in the show notes.
If you don't know the format of a sub-RIP file or an SRT file
it's described as part of the Matroska multimedia container format
as perhaps the most basic of all subtitle formats
but that doesn't make it bad
makes it actually quite good.
So what you've got, there are four lines to each
well four different parts to each segment
the first is, and each segment is split up by a blank line.
So you've got numbers generally starting at
one, two, three, four, five, six
so subtitle 56 is subtitle 56
and then there's a new line
and then there's going to be a time code
so if you haven't seen those before
they're hour, minutes, seconds, comma, milliseconds
and then after that is the subtitle text itself
which can be one or more lines
and then there's a blank line
indicating that's the end of that subtitle segment.
So I downloaded the movie from the Internet Archive
used Piper Voice to convert it
each of the subtitle lines to text
Piper Voice is a fast local
neural text-to-speech engine
that embeds eSpeakNG for feminisations
and are released under the GPL V3.
So once I had the audio
I prepared a sample for the subtitles
and I brought it over to Audacity
to create a new subtitle audio track
and Audacity, if you don't know
is the world's most popular audio editing
and recording app.
It's a GPL V2 or laser.
So the timing of the segments would be a problem
were it not for the fact
that Audacity supports SRT files as labels.
I've already done a show on labels in Audacity
which I need to update
but if you go to File, Import and Labels
then select the SRT file
the subtitle track with the text
of the audio will be displayed
along the lines
so little lines telling you exactly where
all the subtitle segments needed to be lined up.
So I was able to do that, create an audio track
and then go into KDN Live
which is KD's non-linear video editor
which works on Windows
Mac OS, BSD and Linux of course
GPL V3 or later
and Jean-Marc had an excellent article
on how to add subtitles easily
to KDN Live
Arcadian Live
and you go to Project, Subtitles
Add Subtitle Track
it'll import the subtitle file
and then you can select the subtitle file
and you can align the subtitle track
with the audio file
and there's screenshots of all of this
in the show notes
and here also you can see the text
that is in the subtitle
and make sure that it's aligned
and then I was able to render it
and I had myself a sample movie
and I was able to convince myself
that this in fact was something worth doing.
So the script itself will be linked
in the show notes of this episode
but rather than going through it line by line
I'll give you the basic ideas
and the script is only intended
as a proof of concept.
I firmly hope
and probably believe
that FFmpeg will shortly have an option
to be able to do this
and I really hope it does
because there are intricacies
which I will get to at the end.
So what my script does
it first of all
creates a new audio track
for the subtitles
and it merges this
with the original soundtrack
to create a new selectable audio track.
So basically as you're watching the movie
you've got English
you've got the native language
say it's Swedish in this case
and then you'll have another subtitle track
which might be English
and that's what
you can switch between the two.
So you can turn this off
in the same way
you can turn on and off subtitles
you can turn this track on and off.
So the script begins
by creating a length of audio silence
up to where the first subtitle timestamp begins.
And the subtitles from the SRT file
have a timestamp that it has
the beginning when the subtitle should be shown
and the end when the subtitle should be shown.
It converts the first segment
using text-to-speech
into a WAV file I think
and then the segment is added
to the initial silence track.
So fair enough.
So somebody comes on
and says hello world
and somebody else says
hello back.
Yep.
So the first subtitle is
hello world
and then silence
hello world.
So then what I do is
I check the total length so far
to see if the audio
that we've recorded
is longer
or shorter
than the next subtitle.
So somebody's response
to the question.
If it's shorter
then I add a filler piece of silence
to bring it up to the start
of the second subtitle.
If it's not
I just play the second subtitle
immediately after
which means there can be a pileup.
And the pileup
I was very worried about
but actually it turns out
to be not a problem
in practice
because you know
it's
there's action
or there's movement
or somebody picks up
a cup of tea or something
and there's always
a slight delay
in the narrative.
So that's basically
how it works
and we do that
over and over again
right up to the very end
of the movie
and then at the end
the new subtitle
speech audio track
is merged back
into the media
as a new audio track.
Now I included some links
in the show notes
so you can see
what the subtitle looks like
on the screen
and
with screenshots
from the movie
as well
and the time code
paused
so that you can see
that it lines up
at a particular time.
So let's listen
to a segment
of this Swedish movie.
First
the original.
and now
the same segment
but with the
voiceover track
as produced
by my script.
It will be two years
before it's this big.
But don't you bother
by then I'll be long gone.
It was just a question.
Good morning.
So that's the script
and it works
pretty well.
I've done quite a lot
of movies now
and
I've learned
a lot of lessons
over time.
The script itself
sometimes bails
at the creation
of the audio track
and merging it back
into the main file
but I'll go through
workarounds
for that
as well.
The first thing though
that you should do
is the preparation
of the subtitle file
because that's going
to be the basis
of the voice
that you're listening to
and it should be
good quality
so that it matches
when the audio
actors speak.
And some subtitle files
are a lot better
than others.
Some go to
for example
if somebody's
rhyming
in the native language
they will try
and rhyme in English
as well.
So sometimes
there will be a song
and the subtitles
will include
the subtitles
of the song.
So stuff like that
it's good to know.
Sometimes they will
have
if there is a banner
up and that's
important to the plot
and they're referring
to the banner
throughout the movie
but they don't tell you
what's on the banner
then that can be an issue.
So
you know
basically
the standard stuff
when getting a subtitle file
but
you also need to make sure
it's aligned correctly.
you should also
make sure that
the spelling
is correct
because the source
of the subtitle
files
the best
place that you can
get them
is from the DVDs
themselves.
There are several
methods of
putting subtitles on.
The first is
like with a text
based
stream
in the MPEG
and that's perfect
because this is
exactly what they
wanted you
to experience
and sometimes
they'll even have
the
you know
floor creaks
that sort of stuff
track
so you can choose
which one
you would prefer
to hear
and that's
easy peasy
lemon squeezy
to get out.
The other
time
the other type
is
they
where
the video
the subtitles
themselves
are bitmap images
so instead of
hello world
it's a picture
of the words
hello world
and then that's run
through optical character
recognition
and sometimes
that can be very bad.
So for example
one of the movies
had
for every
letters
O
capital letter O
it had the letter
the number zero
so
you would hear
I
L0
you
very
and I also
L0
you
you know
that sort of stuff
then
there can also be
HTML characters
in there
so
you will be listening
to
you can put
emphasis
onto subtitles
or perhaps
they
want to show
that the person's
in another room
talking
and they'll put it
in italics
so you're listening
to a few hours
of less than
forward slash
I
greater than
forward slash
I
another one was
mixing up the
capital letter
I
with the
letter L
so
L
I'm from
Lindia
so a lot of that
can get very tedious
so spell checking
so spell checking
your files
is also a good
thing to do
but if they
if you can get
the subtitle files
from the DVD
directly
you can use
ffmpeg
dash y
to answer
yes or overwrite
dash height banner
space
dash log level
space er
so it's a lot less
for both
and then
dash text
underscore format
space text
dash I
and that's the
input movie
and then
the output
can be the
output file
with the
SRT extension
one top tip
is if you can
keep the
subtitle names
the same as
the movie name
so if it's
movie dot
mp4
then keep
your subtitles
movie dot
SRT
and that
by default
a lot of
players will
try and play
that same
subtitle
they will look
and see if
there's a
subtitle
and automatically
use that
just in general
and by the way
this isn't a
replacement for
the subtitles
I sometimes
turn on the
subtitles anyway
or not
but at least
I have the
choice of
doing it
where to get
them if you
don't have
them
there is a
large repository
of subtitles
at opensubtitles.org
and you'll need
an account
probably
so you select
the language
and the top tip
is to pick the
one with the
highest rating
and then you're
kind of
trying to pick
it if you have
a DVD version
you look for
ones from the
DVD if you've
got a Blu-ray
version you look
for ones from
the Blu-ray
version
one way to
once you have
your subtitle
files one way
to check them
is using the
mpv
media player
which is based
on mplayer
and mplayer2
and it supports
a wide variety
of formats
audios
video codecs
etc
it's released
under the
gplv2
plus
parts are
under the
lgplv2.1
plus
and optional
parts are
under the
gplv3
and links
to that in
the show notes
as I said
if you have
the srt
file with
the same
movie name
then it'll
play it
automatically
otherwise you
can add
the
dash dash
subs
dash files
equal option
as well
all of this
is in the
show notes
extensive
extensive
show notes
for this
episode
so you can
scrub through
left and
right page
up page
down
and
you can
toggle the
subtitles
using the
j key
so if you
you can
download a
series of
subtitles and
then use
the j to
jump in
between them
top tip
is
search for
people saying
the word
okay which
tends to be
a common
word in
many languages
and then you
will know if
your subtitles
are in sync
or out of
sync
you don't
usually have a
problem with
subtitles being
in sync if
you can get
them from
text from
the dvd
itself
but if you're
downloading them
from the
internet they
can be out
of sync
very easily
where they
don't match
up and
it's usually
usually the
movie itself
is correct but
the start
offset is
wrong because
maybe they're
you're getting
it from a
different region
so the
person who
the dvd
in the
netherlands
is different
to the dvd
in ireland
because the
local warnings
are different
at the
beginning but
what you can
do is
adjust them
usually then
when the
movie starts
the subtitles
are correct
from then
on so
it's just
the original
offset that's
wrong
there's this
really cool
program called
ff
subsync
which is a
language
agnostic
automatic
synchronization
of subtitles
with video
files
so the
subtitles
are aligned
to the
correct
starting
point
within the
video
it's
released
under
an
MIT
license
you
pip
install
it
ffsubsync
and then
you use
command
ffs
video.mb3
space
dash
i
unsynchronized
dot srt
space
dash
o
synchronized
dot srt
and what
that does
is it
looks to
the first
speaking
piece of
tech
of audio
in the
file and
tries to
match that
up
what I've
noticed
helps
that
works very
well
so long
as the
first
piece
of speech
is actually
coming from
the movie
and not
from a
government
warning about
ripping
DVDs
or not
or a
warning about
smoking or
tobacco or
something
so you
can use
a really
cool tool
called
Lossless
Cut
Lossless
Cut
aims to
be the
ultimate
cross-platform
ffmpeg
GUI
for extremely
fast and
lossless
operation on
video,
audio,
subtitles
and other
related
files
gpl
dash
2.0
license
brilliant
little
tool
use this
all the
time
couldn't
be
simpler
you just
drag in
the
drop in
the movie
and you
go start
point
end point
press delete
and away
she goes
outputs it
without
transcoding
or anything
just dumps
it so it'll
allow you to
quickly remove
additional trailer
ads at the
beginning so
that ff
subsync will
have a better
chance
working if
they're all
trimmed away
okay so
if that
fails to
match up
the
subtitles
you can
use the
mpv
keyboard
shortcuts
to move
to the
first speech
segment
and then
press
ctrl shift
left
or ctrl
shift right
to adjust
the subtitle
delay
so that the
next and the
previous
subtitles are
displayed
so what this
will do is
if you're
watching
subtitles with
mpv
and somebody
says hello
you hear them
speak and
then a
duration later
the text hello
comes up on
the screen
you can just
rewind to when
the person starts
saying hello
and then
press
ctrl shift
right
so that they
match up
now the
thing to do
then is scrub
through the
scrub means
fast forwarding
or moving
forward in the
movie towards
the end and
make sure that
the sync is
still there
within your
subtitle files
so that when
they say goodbye
at the end
that whatever
the word is
the subtitle
matches the
timing of the
dialogue that's
going on
because
sometimes there
are different
versions of the
movies or there
are cut scenes
or they're just
subtitles that
are very poor
and are where
somebody has
the first part
of the movie
done and the
second part is
just quotations
from
I don't know
what message
they're trying
to send
so check
your subtitles
the whole way
through
make sure
they match
but if
they are in
sync and
they're just
off by an
offset of
say in my
example I'm
using
148.416
is the
millisecond
offset
which turns out
to be minus
148.146
seconds
you can use
a command line
tool called
srt offset
which says
it is a
simple command
line tool
to offset
srt subtitle
files this
allows you to
adjust the
timing of
subtitle srt
files which
can be useful
when the
subtitles are
out of sync
with videos
and that's
an MIT
licensed
and the
command is
srt
dash
offset
space
dash
i
input
dot srt
space
dash
offset
the word
space
dash
and the
amount
in my
case
minus
148.416
space
dash
o
output
dot srt
so that's
a lot
about
subtitle
files
my advice
is
spend
a lot
of time
on this
one
if you're
going to
be watching
a three
or four
hour
movie
there's
nothing
more
annoying
than
this
part
wrong
so
I've
been
known
to
watch
minutes
of
a
movie
found
something
annoying
about it
go back
fix
the
srt
files
and
reimport
them
again
it's
it's
like
that
and
sometimes
it's
it's
actually
just
better
to
get
a
copy
you
know
if
you're
having
no
luck
with
this
DVD
copy
try
and
get
another
copy
from
somewhere
else
and
fortunately
I've
been
fortunate
enough
to have
friends
who
have
had
good
copies
most
of
the
time
sometimes
my
script
will
create
a
mix
down
track
but
has a
problem
adding
it
back
into
the
movie
as a
separate
audio
track
so to
get
around
that
I
use
avi
demux
which
is
a
free
video
editor
designed
for
simple
cutting
filtering
and
encoding
tasks
it's
released
under
the
gpl
v2
and
if
you
go
file
open
and
select
the
movie
then
you
go
to
audio
track
select
audio
track
several
will
appear
at least
one
and
then
you
tick
on
the
next
available
track
in
my
case
track
then
you
click
from
the
drop
down
menu
add
audio
track
and
then
you
pick
the
mixed
audio
in
my
case
it's
dot
tilde
nasty old
people
underscore
mixed
mb3
and
you
add
that
back
in
then
save
the
file
and
you'll
have
a
saved
track
that
will
have
the
audio
so
that's
it
and
in
conclusion
I'm
finding
it
much
easier
to
watch
a
movie
with
a
voice
over
track
than
I
thought
the
idea
of
somebody
talking
in
my
ear
I
would
have
thought
would
have
been
annoying
particularly
when I
heard it
in
Polish
initially
but at
some
point
you
don't even
notice it
after a while
if you're
conscious about it
yes it can be
irritating
but sometimes
if you focus on the
dialogue that's going on
on the screen
you just
see the actors
talking in their
language
and you know
what they're saying
so I
find it a really
good option
for some
movies
particularly
in if there's a
dance or dialogue
scene
and the talking
is fast
say a rap
scene where
they're rapping
over and back
at each other
I might
turn it off
and focus on
the
subtitles
in that case
but
you have the
options
it's with
MPV
it's
shift
octatorp
shift
hashed
sign
shift
three
and you'll
get it
and you can
switch
and toggle
subtitles
using the
J key
so that's it
hope you found
this interesting
there was quite a lot
of tools here
for messing around
with video
audio
and
subtitles
which
I think
over a year
I've gathered up
and found
all of them
super useful
particularly
audacity
caden live
it's amazing
that it can do
that
FFmpeg
fantastic
tool
no doubt
this will be
added
I know
there are
some
pull requests
for doing
automatic
translations
they're better
than not
so like
doing whisper
on it
and then
if you
point
whisper
which we
used to do
the
translations
in
HPR
you will get
80%
of the way
but with a
movie you're
going to be
losing a lot
of the nuance
and I find
that if you
can get
the original
subtitles
or subtitles
from somebody
who's very
good at doing
subtitles
you'll get a lot
more nuance
of the language
and the movie
in fact
will be a lot
more enjoyable
okay
that's it
hope you
use this
or you know
if somebody
comes on with a
better tool
please record a
show
on that
because I
would
much prefer
to be using
somebody's tool
than having to
maintain my own
but it
has opened
up a
wide variety
of movies
for me
now that
I would
normally
not even
bother
watching
or just
too intense
to watch
and I
can focus
on the
actors and
actresses
doing very
good jobs
okay
and with
that I'll
probably
give you some
of the
recommendations
for movies
that I've
been watching
lately
and at
some point
if you
actually have
recommendations
for your
region
movies from
your region
that say
top three
movies that
everybody knows
in your region
that you would
say would be
representative
of must
watch movies
feel free to
record a show
about that
here on
Hacker Public
Radio
perhaps it
might be
tomorrow's
episode
so tune in
tomorrow for
another exciting
episode of
Hacker Public
Radio
You have been listening to the Hacker Public Radio podcast, at hackerpublicradio.org.
Today's show was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast, then visit the HPR site to find out how easy it really is.
Hosting for HPR has been kindly provided by anhonesthost.com, the Internet Archive, rsync.net, and the HPR Community Content Delivery Network.
Unless otherwise stated, today's show is released under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.