Analyzing alignment quality#
When exporting textgrids following alignment, an additional file named alignment_analysis.csv will be exported. I am still currently working to refine what are the best measures for analyzing alignments as it’s not entirely as straightforward as taking the overall alignment log-likelihood.
Alignment log-likelihood#
The first measure provided for each utterance is the alignment log-likelihood. This represents overall the objective measure that was optimized for alignment. However, it is extremely important to note that this log-likelihood is a relative measure for the best path of alignment for this particular utterance compared to other possible alignments.
A primary reason that such heavy caveats come with this metric is due to the use of speaker adaptation. MFA does two passes of alignment. The first uses a speaker-independent model to generate an initial alignment. This initial alignment is used to estimate per-speaker feature transforms that try to map the observed features into a common space. Depending on the amount of data for a particular speaker, and the amount of variability they exhibit (i.e., do they yell, do they get excited, do they whisper, did they have a cold, etc etc), speaker transforms have a variable effect on improving alignment. This variable improvement directly affects the log likelihood for a given utterance.
Additionally, log-likelihood reflects differences in the training data versus alignment data. Is the variety of the language the same? Does it have similar gender distribution? Does it have similar styles (conversational, scripted)? Does it have similar noise levels? All of these can affect the acoustics of phones and skew how “likely” a given phone at a given point in time is.
Speech log-likelihood#
The overall alignment log-likelihood represents the best path including all sections of silence. In general when we’re thinking about how good an alignment is, we don’t necessarily care how good of a match the silence intervals in a given utterance are to the trained silence model. So the speech log-likelihood measure takes out all log-likelihoods from silence intervals and is the average of per-phone log-likelihoods in the utterance.
Phone duration deviation#
Stepping back from log-likelihoods generated by the model, we can take a look at statistics of the duration of phones in the aligned corpus. By calculating the mean and standard deviation of durations per phone, we can z-score the individual phone’s duration to see how unexpected it is relative to the corpus overall. The phone duration deviation measure is the maximum of the absolute z-score of each phone’s log-duration.
We use the absolute value of the z-score because often excessive durations due to misalignment will also result in excessively small durations on other phones. The average of raw z-scores in these cases will trend towards zero, when really we want these deviations to aggregate to utterances that clearly had something go wrong. The log-duration is used to better model the distribution of durations, as they are bounded by 0 (i.e., always positive), and skewed heavily towards lower durations with a long tail of longer durations, so a log-normal distribution provides a distribution that matches the assumptions of z-scores.
It is important to note that there stylistic and speaker influences on duration, and statistics are gathered for the whole corpus, not normalized per speaker, so false positives are likely to pop up when sorting by this metric. Normalizing per-speaker, however, might minimize the magnitude of duration deviation if a given speaker’s utterances are all poorly aligned. This would increase the likelihood of false negatives, and false positives are more acceptable than false negatives.
Max running short intervals#
This measure provides a sense of how many short intervals (10 ms or less) in a row are in the file. For long-running stretches of 10ms intervals in a row, this indicates that significant parts of the transcript were not correctly aligned or shoved together at the beginning/end of the file.
Intensity deviation#
Similar to Phone duration deviation, intensity deviation first calculates the mean and standard deviation of intensity over phones. The mean and standard deviation are then used to create z-scores of individual interval intensities to allow for querying and search of phones/utterances that unexpectedly high or low intensities for a given interval, which could indicate an issue in alignment, such as if all the speech intervals are aligned to stretches of silence.
However, there can be valid reasons for intensity to deviate related to speech style in particular, either if the speaker is excited or is whispering or is otherwise adopting an affect outside the rest of the speech.
Signal-to-Noise Ratio (SNR)#
Signal-to-Noise Ratio (SNR) is calculated as the ratio of the mean intensity of speech phones to the mean intensity of the silence in an utterance. If there is no silence aligned, then the calculation will use the minimum intensity in the utterance, so it may not be entirely accurate.
SNR can be used in certain circumstances to help find alignment or transcript errors, such as when a word is missing from a transcript, and that section of speech is aligned as silence, resulting in a lower than expected SNR.
It is important to note that SNR can vary due to recording conditions and may not be indicative of alignment errors alone. Knowing the source data is particularly important for utilizing this metric to find errors, such as knowing that the data should be generally clean speech in quiet recording conditions vs uncontrolled or variable recording conditions.
Ideas for the future that need a lot more thinking before I implement them#
Use the alignment best path from the speaker adapted pass with a lattice and scores generated using the speaker-independent first-pass alignment model
This might help get around the variable optimizations that are speaker dependent