Example: Aligning a demo corpus#

Note

See also our Google Colab notebook for running this example without installing or downloading anything locally. There is also NTT123’s Jupyter notebook for running the alignment example with a custom LibriSpeech dataset, created by NTT123.

Set up#

Important

Ensure you have installed MFA via Installation.

  1. Ensure you have downloaded the pretrained model via mfa model download acoustic english_mfa

  2. Ensure you have downloaded the pretrained US English dictionary via mfa model download dictionary english_us_mfa

  3. Download the English LibriSpeech demo corpus and extract it to somewhere on your computer

  1. Ensure you have downloaded the pretrained model via mfa model download acoustic japanese_mfa

  2. Ensure you have downloaded the pretrained Japanese dictionary via mfa model download dictionary japanese_mfa

  3. Download the Japanese JVS demo corpus and extract it to somewhere on your computer

  4. Install Japanese-specific dependencies via conda install -c conda-forge spacy sudachipy sudachidict-core

  1. Ensure you have downloaded the pretrained model via mfa model download acoustic mandarin_mfa

  2. Ensure you have downloaded the pretrained China Mandarin dictionary via mfa model download dictionary mandarin_china_mfa

  3. Download the Mandarin THCHS-30 demo corpus and extract it to somewhere on your computer

  4. Install Mandarin-specific dependencies via pip install spacy-pkuseg dragonmapper hanziconv

Important

This example assumes you have a directory named mfa_data in your home directory in which the demo corpus was extracted.

Alignment#

Aligning using pre-trained models#

In the same environment that you’ve installed MFA, enter the following command into the terminal:

mfa align ~/mfa_data/librispeech-demo-1.0.0 english_us_mfa english_mfa ~/mfa_data/aligned_librispeech_demo --clean

mfa align ~/mfa_data/japanese-jvs-demo-1.0.0 japanese_mfa japanese_mfa ~/mfa_data/aligned_jvs_demo --clean

mfa align ~/mfa_data/mandarin-thchs-30-demo-1.0.0 mandarin_china_mfa mandarin_mfa ~/mfa_data/aligned_thchs_30_demo --clean

Once the files are aligned we can take a look at the alignment_analysis.csv file in the output directory to see if there are any glaring issues in alignment. This file is sorted initially by the phone_duration_deviation column, which is the maximum z-scored duration for phones in the utterance. High values indication much longer or shorter phones than we would expect given the phone, i.e., a [ɾ] lasting 100ms is very unlikely given the usual duration is typically around 10-20ms.

~/mfa/data/aligned_librispeech_demo/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

26-495-0029

0.0

15.33

26

-54.456238788323546

-57.6560760433391

15.243577592668007

6.645144699335612

19-227-0027

0.0

15.645

19

-44.529997004792335

-46.942441727063674

13.854204008671575

12.191697302549363

19-198-0010

0.0

12.6

19

-43.50238715277778

-45.62727763510158

9.378276520846718

13.180338896436085

26-495-0004

0.0

14.055

26

-50.670524759957324

-50.98836160475208

9.132307439395934

11.955992448065652

19-227-0016

0.0

12.685

19

-42.93412012411348

-46.04780329319469

8.374659305119293

12.57889123443367
~/mfa/data/aligned_jvs_demo/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

jvs004_nonpara30_BASIC5000_1571

0.0

4.724208333333333

jvs004

-44.60921775688559

-53.64337515830994

11.069623295539825

20.73555387405615

jvs001_parallel100_VOICEACTRESS100_001

0.0

8.621041666666667

jvs001

-41.720485607598604

-45.22640653756949

6.211908079327884

27.302904438762084

jvs003_nonpara30_BASIC5000_2550

0.0

10.909041666666667

jvs003

-40.970862454170486

-45.1975002500746

5.892864020481638

23.07655949682694

jvs004_nonpara30_BASIC5000_1560

0.0

9.180166666666667

jvs004

-46.13757829520697

-50.26279640197754

5.803162260964776

21.536162380195314

jvs003_nonpara30_BASIC5000_0440

0.0

9.649583333333334

jvs003

-41.17138924870466

-45.75857627223915

4.7608877931967415

23.593413391753792

The above table shows the five files with highest phone duration deviation, which we can then take a look more closely and see what’s causing the alignment issues.

For example, looking at jvs004_nonpara30_BASIC5000_1571 in a program like Praat, we can see the issue is due to an out of vocabulary item “1泊”. The original transcript is:

税、その他全て込みだと、１泊いくらですか。

Which is tokenized by MFA using sudachipy to the following sequence of words:

税 その 他 全て込み だ と １泊 いくら です か。

Every word other than “１泊” is present in japanese_mfa dictionary, but the lack of “１泊” causes a small section of the previous word “と” to be aligned as “spn”, and then the following word “いくら” takes up the actual time span of “いっぱく” in addition to the time span corresponding to “いくら”, since both “いっぱく” and “いくら” start with [i] followed by a stop closure.

There are a couple of ways to fix this issue. The most straight-forward way would be to add a dictionary entry for “１泊” as i pː a k ɯ or change the input transcript to use “一泊” instead of “１泊”, as the correct pronunciation is present for “一泊”.

However, this fix will only affect this one particular word/utterance, and there are likely many more out of vocabulary items. The Japanese alignment has the capability of using the katakana generated by the sudachipy morphological parser and use the katakana forms as input to grapheme-to-phoneme models on the fly, so for the purposes of this example, we’ll skip over adding words to the Japanese dictionary and just focus on using G2P models at alignment time.

~/mfa/data/aligned_thchs_30_demo/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

A4_8

0.0

12.4375

A4

-49.248455084405144

-55.64855964978536

6.169737157239693

8.524069008855722

C12_515

0.0

11.69

C12

-49.446983265611635

-51.683270083533394

5.940588177352259

10.706482170359841

A4_25

0.0

10.5

A4

-48.68171130952381

-50.71304063911898

5.613372337759242

6.7809069072769255

B34_251

0.0

9.25

B34

-40.79677787162162

-41.75823864792333

5.540030677004922

8.962213255170383

A4_6

0.0

9.625

A4

-50.510534300363446

-53.57253624076274

5.429628766470327

5.500242213510484

Adding words to the dictionary#

First we’ll need the pretrained G2P model. These are installed via the mfa model download command:

mfa model download g2p english_us_mfa

You should be able to run mfa model inspect g2p english_us_mfa and it will output information about the english_us_mfa G2P model.

mfa model download g2p japanese_mfa

You should be able to run mfa model inspect g2p japanese_mfa and it will output information about the japanese_mfa G2P model.

mfa model download g2p mandarin_china_mfa

You should be able to run mfa model inspect g2p mandarin_china_mfa and it will output information about the mandarin_china_mfa G2P model.

Depending on your use case, you might have a list of words to run G2P over, or just a corpus of sound and transcription files. The mfa g2p command can process either:

mfa g2p ~/mfa_data/librispeech-demo-1.0.0 english_us_mfa ~/mfa_data/g2pped_oovs.txt --dictionary_path english_us_mfa --clean

For Japanese, G2P functionality is done as part of alignment by specifying --g2p_model_path.

mfa g2p ~/mfa_data/mandarin-thchs-30-demo-1.0.0 mandarin_china_mfa ~/mfa_data/g2pped_oovs.txt --dictionary_path mandarin_china_mfa --clean

Running the above will output a text file in the format that MFA uses (Pronunciation dictionary format) with all the OOV words (ignoring bracketed words like <cutoff>). I recommend looking over the pronunciations generated and make sure that they look sensible. For languages where the orthography is not transparent, it may be helpful to include --num_pronunciations 3 so that more pronunciations are generated than just the most likely one. For more details on running G2P, see Generate pronunciations for words (mfa g2p).

Once you have looked over the dictionary, you can save the new pronunciations via:

mfa model add_words english_us_mfa ~/mfa_data/g2pped_oovs.txt

For Japanese, G2P functionality is done as part of alignment by specifying --g2p_model_path.

mfa model add_words mandarin_china_mfa ~/mfa_data/g2pped_oovs.txt

The new pronunciations will be available when you use the dictionary identifier in an MFA command, i.e. the modified command from Aligning a speech corpus with existing pronunciation dictionary and acoustic model:

mfa align ~/mfa_data/librispeech-demo-1.0.0 english_us_mfa english_mfa ~/mfa_data/aligned_librispeech_demo_no_oovs --clean

mfa align ~/mfa_data/japanese-jvs-demo-1.0.0 japanese_mfa japanese_mfa ~/mfa_data/aligned_jva_demo_no_oovs --g2p_model_path japanese_mfa --clean

mfa align ~/mfa_data/mandarin-thchs-30-demo-1.0.0 mandarin_china_mfa mandarin_mfa ~/mfa_data/aligned_mandarin_demo_no_oovs --clean

Now let’s see if adding words to the dictionary helped improve the alignment issues we were seeing before.

~/mfa/data/aligned_librispeech_demo_no_oovs/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

26-495-0029

0.0

15.33

26

-54.456238788323546

-57.6560760433391

15.243577592668007

6.645144699335612

19-227-0027

0.0

15.645

19

-44.529997004792335

-46.942441727063674

13.854204008671575

12.191697302549363

19-198-0010

0.0

12.6

19

-43.50238715277778

-45.62727763510158

9.378276520846718

13.180338896436085

26-495-0004

0.0

14.055

26

-50.670524759957324

-50.98836160475208

9.132307439395934

11.955992448065652

19-227-0016

0.0

12.685

19

-42.93412012411348

-46.04780329319469

8.374659305119293

12.57889123443367
~/mfa/data/aligned_jvs_demo_no_oovs/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

jvs001_parallel100_VOICEACTRESS100_001

0.0

8.621041666666667

jvs001

-41.84382250580047

-45.2327318925124

6.20348375405817

27.293943985523807

jvs003_nonpara30_BASIC5000_2550

0.0

10.909041666666667

jvs003

-40.983222101283225

-45.076715024312335

5.908918038417016

23.07655949682694

jvs002_nonpara30_BASIC5000_0114

0.0

4.245291666666667

jvs002

-39.14056525735294

-42.082562075720894

5.185645170149811

20.482769181663702

jvs003_nonpara30_BASIC5000_0440

0.0

9.649583333333334

jvs003

-41.123874676165805

-45.70696529871981

4.736230315134372

23.619350634943906

jvs001_parallel100_VOICEACTRESS100_004

0.0

6.523541666666667

jvs001

-44.13319593558282

-47.53735481118256

4.546602511572727

26.105708678065128

We can see now that jvs004_nonpara30_BASIC5000_1571 no longer has the top phone duration deviation. Looking at in a program like Praat, we can see that “1泊” and “いくら” are now properly aligned at the word level, and the phones in “いくら” are properly aligned.

However, it is important to note that the pronunciation generated for “1泊” is not correct. The pronunciation generated is not i pː a k ɯ as it should be, but is instead i tɕ i h a k ɯ. The source of this error is due to the sudachipy parse gives the pronunciation form of “1泊” as “イチハク”　and not “イッパク”. However, there is enough of the correct pronunciation to ensure that this error does not affect the alignment of surrounding words.

~/mfa/data/aligned_thchs_30_demo_no_oovs/alignment_analysis.csv#

file

begin

end

speaker

overall_log_likelihood

speech_log_likelihood

phone_duration_deviation

snr

A4_8

0.0

12.4375

A4

-49.248455084405144

-55.64855964978536

6.169737157239693

8.524069008855722

C12_515

0.0

11.69

C12

-49.446983265611635

-51.683270083533394

5.940588177352259

10.706482170359841

A4_25

0.0

10.5

A4

-48.68171130952381

-50.71304063911898

5.613372337759242

6.7809069072769255

B34_251

0.0

9.25

B34

-40.79677787162162

-41.75823864792333

5.540030677004922

8.962213255170383

A4_6

0.0

9.625

A4

-50.510534300363446

-53.57253624076274

5.429628766470327

5.500242213510484

See also

Adapting the acoustic model#

In general, adapting a pretrained acoustic model to your specific data will improve alignments.

We can adapt our pretrained model via the mfa adapt command:

mfa adapt ~/mfa_data/librispeech-demo-1.0.0 english_us_mfa english_mfa ~/mfa_data/english_mfa_adapted.zip --clean

We can now use the adapted model to align the librispeech-demo corpus. Note the change from english_mfa to ~/mfa_data/english_mfa_adapted.zip below.

mfa align ~/mfa_data/librispeech-demo-1.0.0 english_us_mfa ~/mfa_data/english_mfa_adapted.zip ~/mfa_data/aligned_librispeech_demo_adapted --clean

mfa adapt ~/mfa_data/japanese-jvs-demo-1.0.0 japanese_mfa japanese_mfa ~/mfa_data/japanese_mfa_adapted.zip --g2p_model_path japanese_mfa --clean

We can now use the adapted model to align the japanese-jvs-demo corpus. Note the change from japanese_mfa to ~/mfa_data/japanese_mfa_adapted.zip below.

mfa align ~/mfa_data/japanese-jvs-demo-1.0.0 japanese_mfa ~/mfa_data/japanese_mfa_adapted.zip ~/mfa_data/aligned_jvs_demo_adapted --g2p_model_path japanese_mfa --clean

mfa adapt ~/mfa_data/mandarin-thchs-30-demo-1.0.0 mandarin_china_mfa mandarin_mfa ~/mfa_data/mandarin_mfa_adapted.zip --clean

We can now use the adapted model to align the mandarin-thchs-30-demo corpus. Note the change from mandarin_mfa to ~/mfa_data/mandarin_mfa_adapted.zip below.

mfa align ~/mfa_data/mandarin-thchs-30-demo-1.0.0 mandarin_china_mfa ~/mfa_data/mandarin_mfa_adapted.zip ~/mfa_data/aligned_thchs_30_demo_adapted --clean

See also