Recognizers, verifiers and pitch trackers¶
kudio.core.wer
¶
Word and character error rates: how far a recogniser's transcript is from
the reference transcript, counted in words (wer) or in characters
(cer), with the counts behind each from word_errors and
char_errors.
The hypothesis, what the recogniser wrote, is aligned with the reference, what was said, so that as few words as possible are substituted (S), deleted (D) or inserted (I), and the rate is those edits over the length of the reference:
H being the hits, the reference words the alignment pairs with the same word. The fewest edits is the Levenshtein distance between the two word sequences, the minimum edit distance of [JM] Sec. 2.9 with every edit costing 1, on which [JM] Sec. 16.6 defines WER. CER is the same count over characters. A corpus is scored as Kaldi's compute-wer and jiwer score one: the errors of all its pairs over the reference words of all of them. The sources:
[JM] D. Jurafsky and J. H. Martin, "Speech and Language Processing",
3rd ed., draft of 19 August 2026: Ch. 2, "Words and Tokens",
Sec. 2.9, minimum edit distance, pp. 28-32; Ch. 16, "Automatic
Speech Recognition", Sec. 16.6, word error rate, pp. 21-24.
https://web.stanford.edu/~jurafsky/slp3/
[SCTK] NIST Speech Recognition Scoring Toolkit, the sclite manual,
doc/sclite.htm, master branch as read on 2026-11-12.
https://github.com/usnistgov/SCTK
[Kaldi] src/bin/compute-wer.cc, src/util/edit-distance-inl.h,
src/util/kaldi-holder-inl.h and
egs/wsj/s5/steps/scoring/score_kaldi_cer.sh, master branch as
read on 2026-11-12. https://github.com/kaldi-asr/kaldi
[jiwer] jiwer 2.3.0, jiwer/measures.py; jiwer 3.1.0 and 4.0.0,
src/jiwer/process.py, transformations.py and transforms.py.
https://github.com/jitsi/jiwer. Its alignments come from
rapidfuzz, rapidfuzz/distance/Levenshtein_impl.hpp in
https://github.com/rapidfuzz/rapidfuzz-cpp, main branch as read
on 2026-11-12.
[ESPnet] espnet2/legacy/nets/e2e_asr_common.py (ErrorCalculator),
egs2/TEMPLATE/asr1/asr.sh and espnet2/text/char_tokenizer.py,
master branch as read on 2026-11-12.
https://github.com/espnet/espnet
On the worked examples of [JM] these give the published counts: 6 substitutions, 1 deletion and 3 insertions in the 13 reference words of the CallHome utterance in Sec. 16.6, a WER of 76.9%, and the 5 edits between "intention" and "execution" in Sec. 2.9, split as its Fig. 2.17 splits them.
The total is fixed, the split is not. Reference "a c" against hypothesis
"d d a" is three edits either as two substitutions and an insertion, or as
two insertions, a hit on "a" and a deletion. Of the alignments with the
fewest errors I take the one with the most hits. Since
2H + S = N + M - (S + D + I), M being the length of the hypothesis, that
is also the one with the fewest substitutions, and it belongs to the pair:
the four counts are the same whichever way the table is filled. A tool that
builds one alignment instead picks among equally cheap steps by a fixed
order, and its counts follow that order. compute-wer takes the diagonal only
when it is strictly cheaper than both other steps, then a deletion only when
it is strictly cheaper than an insertion, and an insertion otherwise, so it
reports the pair above as 2 substitutions and 1 insertion where kudio reports
1 hit, 1 deletion and 2 insertions. jiwer counts the alignment rapidfuzz
recovers from its own table, with an order of preference of its own. The WER
agrees; a table of S, D and I from two tools need not. How often they part depends on how often
the text has ties: never in the 5,000 utterances of the simulated corpus in
tools/validation/wer.py (100,000 words, about 10% of them wrong), and on
618 of 18,420 random pairs of up to 12 words drawn from four, for
compute-wer.
sclite can count more errors. It minimises a weighted distance: 0 for a
hit, 3 for an insertion or a deletion and 4 for a substitution ([SCTK],
"Dynamic Programming string alignment"), which is 3 (S + D + I) + S.
Among the alignments with the fewest errors that picks the one with the
fewest substitutions, so wherever sclite's alignment has the fewest errors,
its four counts are the ones here. But it trades one more error for five
fewer substitutions. Against the reference "one two three four five", the
hypothesis "four five six seven eight" is 5 substitutions here, a WER of 1.0;
sclite's weights pair up the two "four five" at a cost of 18 against 20, for
3 deletions and 3 insertions, 6 errors and a WER of 1.2. That happens on none
of the simulated utterances, and on 59 of the 18,420 short pairs, with 711
more where it depends on how sclite breaks its own ties.
Words and characters. A word is whatever str.split() separates: runs
of any whitespace split, tabs, line breaks and the full-width space U+3000
among them. compute-wer splits on ASCII whitespace. jiwer splits on the
space character, after it collapses runs of two or more whitespace characters
into one space, so a lone tab or line break there joins two words into one:
"hello world" against "hello\nworld" is a WER of 1.0 in jiwer and 0 here. A
character is a Unicode code point, as Python's str holds it.
For CER the tools disagree about spaces. Kaldi's score_kaldi_cer.sh splits
each word into its characters and drops the spaces, and so does ESPnet's
ErrorCalculator, which logs CER during training. jiwer counts every character
from the first that is not whitespace to the last, spaces included, and so
does the CER that ESPnet's asr.sh reports, which turns each space into a
<space> token for sclite. I leave whitespace out by default. CER is the
measure for Chinese and Japanese, which are written without spaces; there a
space is layout, and counting it would make "我 愛 你" against "我愛你" 2
errors in 5 characters where nothing was misheard. spaces=True counts one
space between each two words however wide the gap, which is what jiwer counts
on text with single spaces between its words; jiwer counts each extra space,
and a tab, as a character of its own.
What to watch for:
- Nothing is normalised. Case, punctuation, digits against numbers written out, contractions and filler words all count as they stand: "Hello, world." against "hello world" is 2 substitutions, a WER of 1.0. Normalise both sides the same way before scoring, and say how next to the number. Lower-casing and turning punctuation into spaces, for example:
import re
def normalise(text):
return re.sub(r"[^\w\s']", " ", text.lower())
kudio.wer([normalise(t) for t in refs], [normalise(t) for t in hyps])
- Unicode is taken as it comes, with no normalisation form applied: "é" as one
code point and as "e" followed by a combining accent are different strings,
and
cer("café", "cafe\u0301")is 2 errors in 4 characters, 0.5. Rununicodedata.normalize("NFC", text)over both sides when they may come from different sources; "NFKC" also folds full-width letters and digits (A, 1) into ASCII. Simplified and traditional characters stay different too: "我爱你" against "我愛你" is a CER of 1/3. And a code point is not always one letter on screen: an accent written as a combining mark, a Devanagari syllable or an emoji sequence counts as several. - A corpus rate is pooled, not averaged. One error in a 2-word utterance and none in an 18-word one is 1/20 = 0.05 pooled, and 0.25 as the mean of the two utterances' rates, which gives the short one the weight of the long one. Pass the whole corpus in one call rather than averaging calls.
- WER is not a share of the words and has no upper bound, since insertions can outnumber the reference: "yes" against "yes yes yes" is 2 insertions over 1 word, a WER of 2.0.
- Text without spaces between its words is one word to
wer: "今天天氣很好" against "今天天氣不好" is a WER of 1.0 and a CER of 1/6. Usecerfor Chinese and Japanese, or segment the words first and join them with spaces. - An empty reference has nothing to divide by. Inside a corpus that does not
matter: its insertions count, as in compute-wer and jiwer 4.0, and
wer(["a b", ""], ["a b", "c"])is 1/2. When the whole reference side is empty,werandcerwarn and return nan; jiwer 4.0 returns the number of insertions there (0 for an empty pair), jiwer 2.3 to 3.1 raise as soon as any one reference is empty, and compute-wer prints nan or inf.word_errorsandchar_errorsreturn the counts either way. - A list is a corpus of transcripts, paired in order. A sentence already cut
into words,
["the", "cat"], would be read as two one-word transcripts, each aligned on its own; join the tokens with spaces first. A dict is read as utterance id to transcript and paired by key, as compute-wer pairs them. A pandas Series is paired by position, like a list, so two Series have to carry the same index in the same order or the call is refused: the same two transcripts with the hypotheses in the other order, their index following them, would score 1.0 by position and 0 by index..to_dict()of both pairs them by index. A DataFrame is refused as well, since it iterates over its column names; pass its column of transcripts. - The work grows with the product of the two lengths of each pair. On a desktop CPU the simulated corpus, 100,000 words in 5,000 utterances, is scored in under 0.1 s, and its 530,000 characters in about 0.7 s; one pair of 10,000-word transcripts takes about 0.3 s, and one of 100,000 words about 25 s.
Classes:
| Name | Description |
|---|---|
ErrorCounts |
The counts behind a WER or a CER, from |
Functions:
| Name | Description |
|---|---|
word_errors |
Hits, substitutions, deletions and insertions of words in |
char_errors |
Hits, substitutions, deletions and insertions of characters in |
wer |
Word error rate of hypothesis against reference, |
cer |
Character error rate of hypothesis against reference, |
ErrorCounts
¶
Bases: NamedTuple
The counts behind a WER or a CER, from word_errors or
char_errors, summed over every pair of a corpus.
hits + substitutions + deletions is the length of the reference and
hits + substitutions + insertions that of the hypothesis; the rate is
errors / reference_length. Of the alignments with the fewest errors,
the counts are those of the one with the most hits. Why, and why
another tool's split can differ while its total agrees:
https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word
Attributes:
| Name | Type | Description |
|---|---|---|
hits |
int
|
reference tokens aligned with the same token of the hypothesis |
substitutions |
int
|
reference tokens aligned with a different token |
deletions |
int
|
reference tokens with nothing aligned to them |
insertions |
int
|
hypothesis tokens with nothing aligned to them |
reference_length |
int
|
tokens in the reference, |
errors |
int
|
|
word_errors
¶
word_errors(
reference: Union[str, Sequence[str], Mapping[Any, str]],
hypothesis: Union[
str, Sequence[str], Mapping[Any, str]
],
) -> ErrorCounts
Hits, substitutions, deletions and insertions of words in hypothesis against reference, summed over the pairs of a corpus.
>>> kudio.word_errors("the cat sat on the mat", "the cat sat on mat")
ErrorCounts(hits=5, substitutions=0, deletions=1, insertions=0, reference_length=6)
Words are split on whitespace and compared as they stand, case and punctuation included. Of the alignments with the fewest errors, the counts are those of the one with the most hits. Why, and how another tool can split the same total differently: https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what was said: one transcript, a sequence of them, or a dict from utterance id to transcript. |
required |
hypothesis
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what the recogniser wrote, in the same form as reference: paired in order for sequences, by key for dicts. |
required |
Returns:
| Type | Description |
|---|---|
ErrorCounts
|
an |
Raises:
| Type | Description |
|---|---|
ValueError
|
for a side that is not text, a sequence of text or a dict of text (bytes, None, a set, a number, a DataFrame or another 2-d table); for a transcript inside one that is not a str (None, NaN, bytes, a list of tokens); for one side a single str and the other a collection; and for sides that do not pair up, sequences of different lengths, dicts with different keys or two pandas Series with different indexes. |
Source code in kudio/core/wer.py
char_errors
¶
char_errors(
reference: Union[str, Sequence[str], Mapping[Any, str]],
hypothesis: Union[
str, Sequence[str], Mapping[Any, str]
],
*,
spaces: bool = False,
) -> ErrorCounts
Hits, substitutions, deletions and insertions of characters in hypothesis against reference, summed over the pairs of a corpus.
>>> kudio.char_errors("我愛你", "我恨你")
ErrorCounts(hits=2, substitutions=1, deletions=0, insertions=0, reference_length=3)
A character is a Unicode code point, and nothing is normalised: an
accent written as a combining mark is a character of its own, so
cer("café", "cafe\u0301") is 0.5; a full-width letter is not its
ASCII one; and "我爱你" against "我愛你" is 1/3. unicodedata.normalize
over both sides takes care of the first with "NFC" and of the second as
well with "NFKC"; nothing folds simplified into traditional.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what was said, in any form |
required |
hypothesis
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what the recogniser wrote, in the same form. |
required |
spaces
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
ErrorCounts
|
an |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/wer.py
wer
¶
wer(
reference: Union[str, Sequence[str], Mapping[Any, str]],
hypothesis: Union[
str, Sequence[str], Mapping[Any, str]
],
) -> float
Word error rate of hypothesis against reference,
(S + D + I) / N, as a fraction (0.25, not 25%). 0 is a perfect
transcript and lower is better; insertions can take it above 1.
>>> kudio.wer("the cat sat on the mat", "the cat sat on mat")
0.16666666666666666
>>> kudio.wer(["a b", "c d e"], ["a x", "c d e"]) # 1 error, 5 words
0.2
A sequence or a dict of transcripts is one corpus: the errors of all its pairs over the reference words of all of them. Words are split on whitespace and nothing is normalised, so lower-case and strip punctuation first if case and punctuation should not count; one way is at https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what was said: one transcript, a sequence of them, or a dict from utterance id to transcript. |
required |
hypothesis
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what the recogniser wrote, in the same form: paired in order for sequences, by key for dicts. |
required |
Returns:
| Type | Description |
|---|---|
float
|
the rate. nan, with a warning, when the reference side has no words at all (an empty reference, references that are all empty, or no pairs), since a rate needs at least 1 reference word to divide by. An empty reference among others is scored, its insertions counted. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/wer.py
cer
¶
cer(
reference: Union[str, Sequence[str], Mapping[Any, str]],
hypothesis: Union[
str, Sequence[str], Mapping[Any, str]
],
*,
spaces: bool = False,
) -> float
Character error rate of hypothesis against reference,
(S + D + I) / N over characters, as a fraction. 0 is a perfect
transcript and lower is better; insertions can take it above 1.
>>> kudio.cer("今天天氣很好", "今天天氣不好")
0.16666666666666666
>>> kudio.cer("the cat sat", "the hat sat") # 1 of 9 letters
0.1111111111111111
>>> kudio.cer("the cat sat", "the hat sat", spaces=True) # 1 of 11
0.09090909090909091
A corpus is pooled as in wer. The two settings of spaces give
different numbers on the same text, since the spaces join the
denominator too, so say which one a table reports.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what was said, in any form |
required |
hypothesis
|
Union[str, Sequence[str], Mapping[Any, str]]
|
what the recogniser wrote, in the same form. |
required |
spaces
|
bool
|
|
False
|
Returns:
| Type | Description |
|---|---|
float
|
the rate. nan, with a warning, when the reference side has no characters other than whitespace, since a rate needs at least 1 to divide by. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/wer.py
kudio.core.verification
¶
Detection measures for speaker verification, and for any task scored trial by trial: the equal error rate and the minimum detection cost of NIST's speaker recognition evaluations, and the calibration measures that read the scores as likelihood ratios.
A verification system gives every trial, an enrolled speaker against a test recording, a score that is higher the more it believes the two are one talker, and a threshold turns the scores into decisions: a target trial scored below it is a miss, a non-target trial at or above it a false alarm. Each distinct score is a threshold of its own, so a list of trials is a set of operating points (P_miss, P_fa), the points of its DET curve (Martin et al. 1997), and both measures read one number off them:
eer, the error rate at which P_miss and P_fa are equal;min_dcf, the lowest weighted sum of the two over every threshold, NIST's C_Det divided by the cost of deciding without the scores.
det_curve returns the operating points themselves, for a plot or for
choosing a threshold.
Both judge the ranking alone. Where the scores are meant as natural-log likelihood ratios, as NIST's plans since SRE 2016 ask for, the decisions they make themselves can be costed as well:
act_dcf, the normalized cost at the threshold Bayes' rule sets for the application, andc_primary, its mean at SRE 2016's two priors;cllr, Brümmer and du Preez's C_llr, the cost over every application at once in bits, andmin_cllr, what the ranking would cost after the best calibration it allows.
Which EER. P_miss moves in steps of 1/T and P_fa in steps of 1/N, for T target and N non-target trials, and as a rule no operating point has the two equal. Every EER is therefore a rule for reading between the points, and the rules in use part on small lists. The default joins successive operating points with straight lines and takes the rate where the line crosses P_miss = P_fa. I chose it for three reasons:
- It is the rule of the scorers the evaluations published. The VoxSRC 2020-2023 scorer finds the crossing on scikit-learn's ROC curve with brentq, and NIST's SRE 2016 scoring script interpolates between the two operating points either side of it, which is the same crossing unless a target and a non-target share a score. With such ties that script's EER moves with the order of the list; the second point under "What to watch for" has by how much.
- A real threshold reaches it, unless the trials tied at the crossing include both a target and a non-target: one of the operating points there has one rate equal to the EER and the other at or below it.
- Across a run of tied scores the line is what accepting each tied trial with one and the same probability gives on average, so the order of the list cannot move the result.
method='rocch' gives the ROCCH-EER of the BOSARIS toolkit instead
(Brümmer and de Villiers, Sec. 2.6.2): the crossing of the convex hull of the
operating points, which they propose as a well-defined version of the EER. It
is the largest, over every prior p, of the lowest p P_miss + (1 - p) P_fa any
threshold gives (their Eq. 26), and it is never above the default.
Eleven trials show how far apart the rules can be. Ranked from the lowest score they are four non-targets, three targets, two non-targets and two targets, and the scorers read:
kudio's default, NIST's SRE 2016 script, the VoxSRC scorer 1/3 = 0.333
kudio, method='rocch' 3/14 = 0.214
Kaldi's compute-eer 0.4
SpeechBrain's EER 0.367
voxceleb_trainer's tuneThresholdfromScore 0.6
Kaldi counts up through the target scores and reports the miss rate at the
first one where the false-alarm rate is no higher than the miss rate.
SpeechBrain takes the mean of the two rates where they are closest.
voxceleb_trainer takes the larger of the two where they are closest, among
the points scikit-learn's roc_curve keeps by default; that drops the
points inside straight runs, here the one at P_fa = 1/3, P_miss = 0.4. On a
list of 40,000 trials, half of them targets, with Gaussian scores, the
default reads 1.090% and the hull 1.081%, and on that list and five more
like it the three other scorers stay within 0.01 percentage points of the
default.
Which cost. NIST's plans write the cost of a threshold as:
and divide it by C_Default = min(c_miss p_target, c_fa (1 - p_target)), the
cost of always rejecting or always accepting (SRE 2008 and 2010, Sec. 3; SRE
2016 and 2019, Sec. 3.1, Eqs. 2-4). min_dcf is the lowest of these
over every threshold, which is the minimum NIST reports beside the actual
cost, "assuming perfect calibration" in the words of the SRE 2010 plan. The
evaluations do not agree on the parameters, which is why p_target has no
default:
SRE 2008, and SRE 2010's historical cost c_miss 10, c_fa 1, p_target 0.01
SRE 2010, core condition c_miss 1, c_fa 1, p_target 0.001
SRE 2016 and the SRE 2019 CTS challenge c_miss 1, c_fa 1, p_target 0.01 and 0.005
SRE 2019 audio-visual, VoxSRC 2020-2023 c_miss 1, c_fa 1, p_target 0.05
The scorers' defaults differ as well: 0.01 in Kaldi's compute_min_dcf.py
and SpeechBrain's minDCF, 0.05 in the VoxSRC scorer and voxceleb_trainer.
SpeechBrain's minDCF returns C_Det before the division. Its VoxCeleb
recipes print that times 100, which is the normalized cost at its default
p_target = 0.01 with unit costs and nowhere else: at 0.05 it is five times
the normalized cost.
What to watch for:
- A higher score has to mean "more likely the same speaker". Distances go in
as their negatives. A list ranked exactly the wrong way round reads 1.0 by
default and 0.5 with
method='rocch', whose hull always contains the line of guessing. - Only distinct scores are thresholds. Kaldi's
compute_min_dcf.py, which the VoxSRC scorer and voxceleb_trainer copy, and NIST's SRE 2016 script rank the trials one at a time, and so also put operating points between two trials with the same score, which no threshold reaches and which move with the order of the list. On 22,000 trials whose scores were rounded to a tenth of their standard deviation, listed non-targets first, that rule gives a min DCF of 0.400 at p_target = 0.05, where no threshold does better than 0.436. NIST's script reads the EER of the same list as 0.05985 when its sort leaves the tied trials non-targets first and 0.07050 when it leaves them targets first; the default reads 0.06535 in any order. - The rates move in steps of 1/T and 1/N, and at a small p_target the false alarms decide. With unit costs the normalized cost is P_miss + beta P_fa, beta = (1 - p_target) / p_target, so at 0.001 each false alarm among 10,000 non-target trials adds 0.0999. A min DCF at that prior read off fewer non-target trials is the count of a handful of errors.
p_targetis the prior of the application the cost stands for, not the share of target trials in the list; the SRE 2010 plan says as much of its 0.001.- One cost has more than one spelling. Costs c_miss and c_fa at prior p give the same normalized cost as unit costs at the effective prior c_miss p / (c_miss p + c_fa (1 - p)) (Brümmer and de Villiers, Eq. 10), so SRE 2008's (10, 1, 0.01) is p_target = 0.0917 with unit costs, not 0.01. The copy of NIST's SRE 2016 script that WeSpeaker distributes prints the cost at (1, 1, 0.01) under the name minDCF08. On the first list of 40,000 trials above, the two read 0.059 and 0.143.
- The minimum is found with the labels in hand, so it is the cost of the best
threshold for this very list. A threshold for new trials is chosen on a
development list, with
det_curve. NIST's primary costs in SRE 2016 and 2019 are actual costs, at the threshold log(beta) on calibrated log-likelihood ratios, and SRE 2016 and the 2019 CTS challenge average them over partitions of the list;min_dcfpools the list as it comes and needs no calibration.
Sources: NIST, "The NIST Year 2008 Speaker Recognition Evaluation Plan"
(3 April 2008); "The NIST Year 2010 Speaker Recognition Evaluation Plan"
(version r6, 21 April 2010); "NIST 2016 Speaker Recognition Evaluation Plan"
(4 August 2016); "NIST 2019 Speaker Recognition Evaluation: CTS Challenge"
(22 July 2019); "NIST 2019 Speaker Recognition Evaluation Plan" (16 August
2019, the audio-visual evaluation). N. Brümmer and E. de Villiers, "The
BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF",
NIST SRE'11 Analysis Workshop, December 2011, arXiv:1304.2865. A. Martin,
G. Doddington, T. Kamm, M. Ordowski and M. Przybocki, "The DET curve in
assessment of detection task performance", Proc. Eurospeech 1997,
pp. 1895-1898, doi:10.21437/Eurospeech.1997-504. [BD06] N. Brümmer and
J. du Preez, "Application-independent evaluation of speaker detection",
Computer Speech and Language 20(2-3), pp. 230-275, 2006,
doi:10.1016/j.csl.2005.08.001. The other scorers are the
code they published: Kaldi's src/ivectorbin/compute-eer.cc and
egs/sre08/v1/sid/compute_min_dcf.py, the VoxSRC 2020-2023 validation
toolkits (compute_EER.py, compute_min_dcf.py),
clovaai/voxceleb_trainer's tuneThreshold.py, SpeechBrain's
speechbrain/utils/metric_stats.py, and NIST's SRE 2016 scoring script
(version 4.1, as WeSpeaker distributes it in wespeaker/utils/score_metrics.py,
last changed in commit 3556586 of 2022-03-25; its min_dcf_08 is
compute_c_norm(fnr, fpr, p_target=0.01, c_miss=1, c_fa=1)).
Their numbers above come from those rules restated in
tools/validation/verification.py, which reruns every comparison here;
tools/validation/calibration.py does the same for the calibration
measures, with PYLLR's rules (github.com/bsxfan/PYLLR, commit 442b10b).
Each call sorts the list once, and a million trials take about a tenth of a
second on a desktop CPU.
Classes:
| Name | Description |
|---|---|
DetCurve |
The operating points of a trial list, one per threshold, in rising |
Functions:
| Name | Description |
|---|---|
det_curve |
Every operating point of a trial list: for each threshold, the miss |
eer |
Equal error rate of a trial list: the rate at which misses and false |
min_dcf |
Minimum normalized detection cost of a trial list, as NIST's speaker |
cllr |
Log-likelihood-ratio cost of a trial list, in bits: how much the |
min_cllr |
C_llr of the best calibration the ranking allows, in bits: what |
act_dcf |
Actual normalized detection cost: the cost of the decisions the LLRs |
c_primary |
NIST's primary cost of SRE 2016 and the SRE 2019 CTS challenge: the |
Attributes:
| Name | Type | Description |
|---|---|---|
C_PRIMARY_PRIORS |
The two priors NIST's SRE 2016 and the SRE 2019 CTS challenge average |
C_PRIMARY_PRIORS
module-attribute
¶
The two priors NIST's SRE 2016 and the SRE 2019 CTS challenge average their primary cost over, with unit costs (Sec. 3.1, Table 1 of the 2016 plan).
DetCurve
¶
Bases: NamedTuple
The operating points of a trial list, one per threshold, in rising order of threshold.
A trial is accepted when its score is at or above the threshold. The
first point accepts every trial (P_miss 0, P_fa 1) and the last, at
inf, rejects every trial (P_miss 1, P_fa 0); in between there is one
point per distinct score. P_miss never falls and P_fa never rises from
one point to the next.
Attributes:
| Name | Type | Description |
|---|---|---|
thresholds |
ndarray
|
the thresholds, float64: every distinct score, then |
p_miss |
ndarray
|
share of target trials scored below each threshold |
p_fa |
ndarray
|
share of non-target trials scored at or above each threshold |
det_curve
¶
det_curve(scores, labels) -> DetCurve
Every operating point of a trial list: for each threshold, the miss and false-alarm rates of accepting the trials scored at or above it.
>>> curve = kudio.det_curve(dev_scores, dev_labels)
>>> worst = np.maximum(curve.p_miss, curve.p_fa)
>>> curve.thresholds[np.argmin(worst)]
>>> cost = 0.05 * curve.p_miss + 0.95 * curve.p_fa
>>> curve.thresholds[np.argmin(cost)]
The thresholds are the distinct scores, so trials with the same score are
accepted or rejected together; no point sits between them. The order is
rising threshold, the reverse of scikit-learn's roc_curve, and
nothing is dropped: the points inside a straight run are kept, and so are
the two ends, accepting everything and rejecting everything.
scikit-learn's det_curve rises as well, but keeps one point where
P_miss is 0 and one where P_fa is lowest and leaves out the points beyond
them, so whether it has either end depends on the list and on the
release: 1.6.1 never returns the one at inf, 1.7.2 can. For a DET
plot, put both rates through scipy.stats.norm.ppf; the ends map to
infinity, which matplotlib leaves out.
The first threshold in the example is the one whose larger error rate is
the smallest. That rate is the EER of eer, unless the trials tied
at the crossing include both a target and a non-target; then no single
threshold reaches the EER, and this one comes closest. The second is the threshold of min_dcf at
p_target 0.05 with unit costs. Chosen on a development list, either can
be used on new trials.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
one score per trial, higher for "more likely a target".
Integer scores are taken as the numbers they are, and every score is
compared as float64: integers past 2**53 that float64 cannot tell
apart count as tied, so that |
required | |
labels
|
one per trial, True or 1 for a target trial and False or 0 for a non-target one. |
required |
Returns:
| Type | Description |
|---|---|
DetCurve
|
a |
Raises:
| Type | Description |
|---|---|
ValueError
|
for scores that are not finite numbers float64 can
hold, labels other than booleans or 0/1, a two-dimensional array (a
|
Source code in kudio/core/verification.py
eer
¶
Equal error rate of a trial list: the rate at which misses and false alarms are equally likely. 0 is perfect separation, 0.5 is guessing.
>>> scores = list(range(11))
>>> labels = [0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1]
>>> kudio.eer(scores, labels)
0.3333333333333333
>>> kudio.eer(scores, labels, method='rocch')
0.21428571428571427
The result is a proportion; Kaldi's compute-eer, voxceleb_trainer and
the VoxSRC scorer print percent. Only the order of the scores matters,
so any strictly rising transformation of them leaves it unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
one score per trial, higher for "more likely a target". |
required | |
labels
|
one per trial, True or 1 for a target trial and False or 0 for a non-target one. |
required | |
method
|
str
|
|
'linear'
|
Returns:
| Type | Description |
|---|---|
float
|
the EER, 0 to 1 (0 to 0.5 with |
Raises:
| Type | Description |
|---|---|
ValueError
|
for an unknown method, and for the trial lists
|
Source code in kudio/core/verification.py
min_dcf
¶
Minimum normalized detection cost of a trial list, as NIST's speaker recognition evaluations define it. 0 is perfect; 1 is no better than deciding without the scores.
VoxSRC's cost, then SRE 2008's:
>>> kudio.min_dcf(scores, labels, p_target=0.05)
>>> kudio.min_dcf(scores, labels, p_target=0.01, c_miss=10)
The lowest, over every threshold, of:
(c_miss p_target P_miss + c_fa (1 - p_target) P_fa) / C_Default
C_Default = min(c_miss p_target, c_fa (1 - p_target))
The thresholds are the operating points of det_curve, including
accepting everything and rejecting everything, so the result never
exceeds 1. Tied scores are one threshold, as there.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
one score per trial, higher for "more likely a target". |
required | |
labels
|
one per trial, True or 1 for a target trial and False or 0 for a non-target one. |
required | |
p_target
|
float
|
prior probability of a target trial in the application the cost stands for, strictly between 0 and 1. It has no default, because the evaluations disagree: 0.001 in NIST's SRE 2010, 0.01 and 0.005 in SRE 2016 and the 2019 CTS challenge (whose primary cost averages the two), 0.05 in the SRE 2019 audio-visual evaluation and VoxSRC; the module docstring has the table. |
required |
c_miss
|
float
|
cost of a miss: 10 in SRE 2008 and in SRE 2010's historical cost, 1 in the later plans. |
1.0
|
c_fa
|
float
|
cost of a false alarm, 1 in every plan cited here. |
1.0
|
Returns:
| Type | Description |
|---|---|
float
|
the normalized minimum cost, 0 to 1. nan, with a warning, for a list without a target or without a non-target trial. |
Raises:
| Type | Description |
|---|---|
ValueError
|
for a p_target outside (0, 1), a cost that is not
positive and finite, a parameter that is not a number, weights
c_miss p_target or c_fa (1 - p_target) below 2.2e-308, the smallest
normal float64, and for the trial lists |
Source code in kudio/core/verification.py
cllr
¶
Log-likelihood-ratio cost of a trial list, in bits: how much the scores, read as likelihood ratios, cost over every application at once. 0 is perfect; 1 is what LLRs of 0 throughout cost, a system that always says it cannot tell.
Brümmer and du Preez's C_llr [BD06]:
It charges for discrimination and for calibration together: a list ranked
perfectly still costs where its LLRs say less, or more, than the
evidence holds. min_cllr is what the same ranking costs once
calibrated as well as it can be, and the difference is the cost of
calibration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
llrs
|
one natural-log likelihood ratio per trial, as BOSARIS and NIST's SRE plans take them; a base-10 LLR, the forensic custom, has to be multiplied by ln 10 = 2.3026 first. Raw scores, cosine similarities say, are not LLRs, and their C_llr says little: calibrate them first. +inf and -inf are LLRs of certainty, and a certainty that is wrong costs inf. |
required | |
labels
|
as |
required |
Returns:
| Type | Description |
|---|---|
float
|
C_llr in bits. nan, with a warning, for a list without a target or without a non-target trial. |
Raises:
| Type | Description |
|---|---|
ValueError
|
for an llr that is NaN, and for the lists
|
Source code in kudio/core/verification.py
min_cllr
¶
C_llr of the best calibration the ranking allows, in bits: what
cllr would be if the scores were turned into LLRs by the one
rising mapping that suits this list best.
The mapping is the pool-adjacent-violators fit of the target share
against the scores, which gives each trial a posterior probability; less
the log of the list's own prior odds, T/N, those are LLRs, and their
C_llr is this [BD06]. The scores only have to rank the trials, so raw
scores will do. Tied scores share one LLR, as they share one threshold
in det_curve.
BOSARIS's min_cllr.m sorts the non-targets of a tie before its
targets and so lets the fit split ties, which reads lower where targets
and non-targets share scores: by up to 0.30 bits on the rounded lists of
kudio's validation script. It also adds 1e-6 i / N to the i-th LLR to
keep them strictly rising, which moves the result by under 1e-7 bits
there. PYLLR, by the same author, keeps ties together, as here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
llrs
|
one score per trial, higher for "more likely a target"; LLRs or not. +inf and -inf rank like any other score. |
required | |
labels
|
as |
required |
Returns:
| Type | Description |
|---|---|
float
|
min C_llr in bits, from 0 to 1, never above |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/verification.py
act_dcf
¶
Actual normalized detection cost: the cost of the decisions the LLRs themselves make, at the threshold Bayes' rule sets for the application. 0 is perfect; 1 is no better than deciding without the scores.
A trial is accepted when its LLR is at or above
log(beta), beta = c_fa (1 - p_target) / (c_miss p_target), the
threshold of NIST's SRE 2016 and 2019 CTS plans (Sec. 3.1), and the cost
is min_dcf's, normalized the same way, at that one threshold
instead of the best. A trial exactly at the threshold is accepted, as
PYLLR's fast_Bayes_error_rate accepts it. The gap to min_dcf
is what calibration costs at that operating point.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
llrs
|
natural-log likelihood ratios, as |
required | |
p_target
|
float
|
as |
required |
c_miss
|
float
|
as |
1.0
|
c_fa
|
float
|
as |
1.0
|
Returns:
| Type | Description |
|---|---|
float
|
the normalized actual cost, from 0 up; above 1 when the LLRs decide worse than the prior alone. nan, with a warning, for a list without a target or without a non-target trial. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/verification.py
c_primary
¶
NIST's primary cost of SRE 2016 and the SRE 2019 CTS challenge: the
mean of act_dcf at p_target 0.01 and 0.005, with unit costs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
minimum
|
bool
|
average |
False
|
Returns:
| Type | Description |
|---|---|
float
|
the mean of the two normalized costs. nan, with a warning, as
The evaluations average the cost over partitions of the trial list, language and gender among them, with each partition's trials weighted equally (SRE 2016 plan, Sec. 3.1); this pools the list as it comes. To match an official number, compute the cost per partition and average them as the plan says. |
Source code in kudio/core/verification.py
kudio.core.diarization
¶
Who spoke when: the diarization error rate, the Jaccard error rate and the detection error rate of speech activity, against a reference of speaker turns.
DER is the error measure of NIST's Rich Transcription evaluations [RT09],
scored by its md-eval.pl and by dscore, the scorer of DIHARD II and III
[DIH19]. The time scored is cut into regions where the set of speakers stays
the same, and each reference speaker is mapped to at most one system speaker,
the mapping that makes the time they speak together the longest. A region of
d seconds with N_ref reference speakers, N_sys system speakers and N_correct
reference speakers whose mapped system speaker is talking there counts:
missed d max(0, N_ref - N_sys)
false alarm d max(0, N_sys - N_ref)
confusion d (min(N_ref, N_sys) - N_correct)
total d N_ref
and DER is the three errors over the total, so overlapping speech counts once for every speaker in it. It can pass 1: a system that talks where nobody does adds false alarm without bound.
JER, introduced for DIHARD II [DIH19], scores each reference speaker on
its own: the speaker's Jaccard error, 1 minus the time it shares with its
system speaker over the union of the two, averaged over the reference
speakers, so that every speaker weighs the same whatever they said. A
reference speaker left without a system speaker scores 1. The mapping is the
one dscore takes: the Hungarian assignment with the smallest summed Jaccard
error. pyannote.metrics' JaccardErrorRate maps by the longest shared time
instead, as for DER, and the two can part where a speaker's best overlap and
best Jaccard index point at different system speakers.
Detection error rate scores speech activity alone: the missed and
false-alarm speech over the reference speech, every speaker merged into one,
as pyannote.metrics' DetectionErrorRate takes it. The labels of a VAD are
enough.
[RT09] NIST, "The 2009 (RT-09) Rich Transcription Meeting Recognition
Evaluation Plan", 2009, Sec. 6.1.
[DIH19] N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy
and M. Liberman, "Second DIHARD Challenge Evaluation Plan",
version 1.1, 2019; the scorer is dscore,
github.com/nryant/dscore.
What to watch for:
- The collar is per side, in seconds, as in md-eval.pl and dscore:
collar=0.25leaves 0.25 s either side of every reference turn boundary unscored, 0.5 s in all. pyannote.metrics'collaris the total, so its 0.5 is this 0.25. DIHARD scores with no collar and with overlapping speech; the older NIST evaluations and many papers use 0.25 s and drop the overlap, and the two settings give very different numbers, so say which. - Without uem the whole of time is scored, which counts only where someone, in the reference or the system, is talking. Give the scoring regions where only part of a recording was annotated.
- dscore computes JER on 10 ms frames, so its JER and the exact one here agree to the last digit only where every boundary is on that grid.
- The speakers are mapped on all the time evaluated, before the collars and the overlap are taken out, as md-eval.pl maps them. pyannote.metrics maps on the time it scores, and with a collar or without the overlap the two can part: reference speakers A and B talking together for 10 s, then C alone for 2 s, against one system speaker throughout, score a DER of 1 here and 0 there once the overlap is left out.
- Without uem, all of time is scored, as dscore and pyannote.metrics score it. md-eval.pl run without a UEM scores only from the reference's first turn to its last, so system speech before or after those is not counted.
- Turns of one speaker that overlap or touch are one stretch of speech here. md-eval.pl refuses an RTTM with overlapping turns of one speaker, and pyannote.metrics counts the time they share twice. Collars are cut around every turn's boundaries as given.
Classes:
| Name | Description |
|---|---|
Turn |
A stretch of one speaker's speech, in seconds. |
DiarizationErrors |
The seconds behind a DER, summed over the recordings of a corpus. |
Functions:
| Name | Description |
|---|---|
load_rttm |
The SPEAKER lines of an RTTM file, as turns by recording. |
diarization_errors |
The seconds of missed speech, false alarm and speaker confusion behind |
der |
Diarization error rate of hypothesis against reference, as a |
jer |
Jaccard error rate of hypothesis against reference, as a fraction: |
detection_error_rate |
Missed and false-alarm speech over the reference speech, every speaker |
Turn
¶
Bases: NamedTuple
A stretch of one speaker's speech, in seconds.
DiarizationErrors
¶
Bases: NamedTuple
The seconds behind a DER, summed over the recordings of a corpus.
Attributes:
| Name | Type | Description |
|---|---|---|
missed |
float
|
reference speech no system speaker covers, counted per speaker |
false_alarm |
float
|
system speech beyond the reference speakers present, per speaker |
confusion |
float
|
reference speech given to the wrong system speaker |
total |
float
|
reference speech scored, counted once for each speaker in it |
der |
float
|
|
load_rttm
¶
load_rttm(path) -> Dict[str, List[Turn]]
The SPEAKER lines of an RTTM file, as turns by recording.
Each line SPEAKER <file> <channel> <onset> <duration> <NA> <NA>
<speaker> ... becomes Turn(onset, onset + duration, speaker) under
<file>; lines of other types are skipped, and so are blank lines and
lines starting with ;;.
Raises:
| Type | Description |
|---|---|
ValueError
|
for a SPEAKER line with fewer than 8 fields, or an onset or duration that is not a number, naming the line. |
Source code in kudio/core/diarization.py
diarization_errors
¶
diarization_errors(
reference,
hypothesis,
*,
collar: float = 0.0,
skip_overlap: bool = False,
uem=None,
) -> DiarizationErrors
The seconds of missed speech, false alarm and speaker confusion behind
der, and the reference speech they are counted against.
>>> kudio.diarization_errors(reference, hypothesis)
DiarizationErrors(missed=1.2, false_alarm=0.8, confusion=3.1, total=60.0)
Arguments as der. Summed over the recordings of a corpus, each
with its own speaker mapping.
Source code in kudio/core/diarization.py
der
¶
der(
reference,
hypothesis,
*,
collar: float = 0.0,
skip_overlap: bool = False,
uem=None,
) -> float
Diarization error rate of hypothesis against reference, as a fraction: missed speech, false alarm and speaker confusion over the reference speech. 0 is perfect; there is no upper bound.
>>> kudio.der(reference, hypothesis) # DIHARD
>>> kudio.der(reference, hypothesis, collar=0.25, skip_overlap=True)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
required | ||
hypothesis
|
the system's turns, in the same form; its speaker labels need not match the reference's. |
required | |
collar
|
float
|
seconds left unscored on each side of every reference turn
boundary, as md-eval.pl's |
0.0
|
skip_overlap
|
bool
|
leave out the time where two or more reference
speakers talk at once, as md-eval.pl's |
False
|
uem
|
the scoring regions, |
None
|
Returns:
| Type | Description |
|---|---|
float
|
the DER, pooled over the recordings of a corpus. nan, with a warning, when the reference has no speech in the time scored. |
Raises:
| Type | Description |
|---|---|
ValueError
|
for a turn that is not |
Source code in kudio/core/diarization.py
jer
¶
Jaccard error rate of hypothesis against reference, as a fraction: 1 minus each reference speaker's Jaccard index with its system speaker, averaged over the reference speakers. 0 is perfect and 1 the worst.
Each reference speaker is paired with at most one system speaker, by the Hungarian assignment with the smallest summed Jaccard error, as dscore pairs them, and one left over scores 1. Overlapping speech counts for every speaker in it, and there is no collar, as in DIHARD.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
as |
required | |
hypothesis
|
as |
required | |
uem
|
as |
None
|
Returns:
| Type | Description |
|---|---|
float
|
the JER; over a corpus, the mean over the reference speakers of every recording, as dscore pools it. nan, with a warning, when there is no reference speaker with speech in the time scored. dscore gives such a recording 1 when the system talks there and 0 when it does not, and leaves it out of the pooled figure, as this does. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/diarization.py
detection_error_rate
¶
Missed and false-alarm speech over the reference speech, every speaker merged into one: how well speech was found, whoever spoke.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
as |
required | |
hypothesis
|
as |
required | |
collar
|
float
|
as |
0.0
|
uem
|
as |
None
|
Returns:
| Type | Description |
|---|---|
float
|
the rate, from 0 up. nan, with a warning, when the reference has no speech in the time scored. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/diarization.py
kudio.core.f0eval
¶
Pitch-tracking errors: how far an estimated F0 track is from a reference track on the same frames.
These are the numbers pitch trackers are compared by on corpora recorded with
a laryngograph, such as Keele and PTDB-TUG, and the F0 RMSE that speech
synthesis papers report. Each function takes two tracks, one F0 value per
frame in Hz with 0 or nan where the frame is unvoiced, or two
kudio.PitchTrack objects, and compares frame i of one with frame i
of the other. Every frame falls in one of four cases ([R76], p. 407):
unvoiced in both; voiced in the reference only, a V/U error; voiced in the
estimate only, a U/V error; or voiced in both, where the estimate is either
close or a gross error. With N frames, N_VV of them voiced in both and N_GE
gross errors among those:
function what it takes over
gross_pitch_error GPE N_GE: |est - ref| > 0.2 ref N_VV
voicing_decision_error VDE N_VU + N_UV N
f0_frame_error FFE N_VU + N_UV + N_GE N
fine_pitch_error FPE SD of (est - ref) / ref N_VV - N_GE
f0_rmse RMS of 1200 log2(est / ref), cents N_VV
GPE, VDE and FFE are those of Chu and Alwan [CA09], eqs. 1 to 4, and FFE = VDE + (N_VV / N) GPE, their eq. 5. FPE is Drugman and Alwan's [DA11]: the standard deviation of the relative error over the frames voiced in both without a gross error. The first three come back as fractions, and FPE as a fraction of the reference F0; times 100 they are the percentages the papers print:
```pycon
>>> ref = [0, 100, 102, 104, 0, 0, 120, 121] # Hz, 0 where unvoiced
>>> est = [0, 101, 204, 0, 0, 98, 119, 0]
>>> kudio.gross_pitch_error(ref, est) # frame 2 an octave up, of 3
0.3333333333333333
>>> kudio.voicing_decision_error(ref, est) # frames 3, 5 and 7, of 8
0.375
>>> kudio.f0_frame_error(ref, est) # all four, of 8
0.5
[`f0_errors`][kudio.core.f0eval.f0_errors] returns all five at once, with the counts behind them.
Sources, and what each one counts:
```python
[R76] L. R. Rabiner, M. J. Cheng, A. E. Rosenberg and C. A. McGonegal,
"A comparative performance study of several pitch detection
algorithms", IEEE Trans. Acoust., Speech, Signal Process. 24(5),
pp. 399-418, 1976, doi:10.1109/TASSP.1976.1162846. A gross error
is a pitch-period error of 1 ms or more (10 samples at their
10 kHz), counted per utterance: at 100 Hz an estimate 9.1% low or
11.1% high, at 200 Hz one 16.7% low or 25% high. The fine errors,
the rest, are measured by their mean, which came out near 0 for
every detector and is not tabulated, and by their standard
deviation, in samples; V/U errors as a share of the reference's
voiced frames, and U/V errors in the same form.
[NIZ03] T. Nakatani, T. Irino and P. Zolfaghari, "Dominance spectrum
based V/UV classification and F0 estimation", Proc. Eurospeech
2003, pp. 2313-2316, doi:10.21437/Eurospeech.2003-210. A gross
error is an estimate more than 5% off, and the fine error is the
RMS of the relative error over the other frames.
[CA09] W. Chu and A. Alwan, "Reducing F0 frame error of F0 tracking
algorithms under noisy conditions with an unvoiced/voiced
classification frontend", Proc. ICASSP 2009, pp. 3969-3972,
doi:10.1109/ICASSP.2009.4960497. GPE over N_VV, its threshold
usually 20%, and VDE and FFE over all N frames.
[DA11] T. Drugman and A. Alwan, "Joint robust voicing detection and
pitch estimation based on residual harmonics", Proc. Interspeech
2011, pp. 1973-1976, doi:10.21437/Interspeech.2011-519. GPE, VDE
and FFE as in [CA09], and FPE as the standard deviation, in
percent, of the relative error of the frames under 20%.
[CA12] W. Chu and A. Alwan, "SAFE: a statistical approach to F0
estimation under clean and noisy conditions", IEEE Trans. Audio,
Speech, Lang. Process. 20(3), pp. 933-944, 2012,
doi:10.1109/TASL.2011.2168518. GPE as in [CA09], with every
tracker's voicing replaced by the reference's: a frame the
reference calls voiced and the tracker unvoiced gets an F0
interpolated from the nearest frames both call voiced (their
eq. 20), and a frame the tracker calls voiced against the
reference is set to 0. The fine errors are a mean (MFPE, eq. 23)
and a root mean square about zero (SDFPE, eq. 24, though they
call it a standard deviation), in Hz.
[CH08] A. Camacho and J. G. Harris, "A sawtooth waveform inspired pitch
estimator for speech and music", J. Acoust. Soc. Am. 124(3),
pp. 1638-1652, 2008, doi:10.1121/1.2951592, sec. III.C. A gross
error rate with the 20% rule, counted only at the times the
reference and every compared algorithm that gave an estimate
every millisecond called the sound pitched, and on the corpora
whose reference moves, after shifting each estimate by up to
100 ms to its best alignment with it.
[P11] G. Pirker, M. Wohlmayr, S. Petrik and F. Pernkopf, "A pitch
tracking corpus with evaluation on multipitch tracking scenario",
Proc. Interspeech 2011, pp. 1509-1512,
doi:10.21437/Interspeech.2011-317. PTDB-TUG. Its benchmark, two
talkers at once, counts gross errors (over 20%) as a share of all
frames, and cuts each utterance 70 ms before its first and after
its last reference pitch value.
[PMA95] F. Plante, G. F. Meyer and W. A. Ainsworth, "A pitch extraction
reference database", Proc. Eurospeech 1995, pp. 837-840,
doi:10.21437/Eurospeech.1995-191. Keele: a reference value every
10 ms from the laryngograph, 0 where unvoiced and negative where
the frame is uncertain, and uncertain frames are to be left out
of comparisons.
[K18] J. W. Kim, J. Salamon, P. Li and J. P. Bello, "CREPE: a
convolutional representation for pitch estimation", Proc. ICASSP
2018, pp. 161-165, doi:10.1109/ICASSP.2018.8461329. Raw pitch
accuracy within 50 cents, and within 25 and 10, from mir_eval.
[R14] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto,
D. Liang and D. P. W. Ellis, "mir_eval: a transparent
implementation of common MIR metrics", Proc. ISMIR 2014,
pp. 367-372, doi:10.5281/zenodo.1416528. The MIREX melody
measures: raw pitch accuracy over the reference's voiced frames,
within half a semitone; voicing recall and false alarm; and
overall accuracy over all frames.
Where the papers leave a choice open, I made it this way. The 20% is
measured against the reference and is strict, as [CA09]'s eq. 3 writes it,
so an estimate exactly 20% off is not a gross error: 120 Hz against 100 is
not. Exactly means in the binary values the arrays hold. Most decimals have
no exact binary form, and the comparison is rounded like any other, so two
decimals 20% apart can land either side: 120.12 Hz against 100.1 is a gross
error, 120.24 against 100.2 is not. FPE divides by the number of fine
frames, as [R76]'s eq. 3 does for its standard deviation; [DA11] does not
say. The RMSE counts every frame voiced in both, gross errors included. On
tracks built so that the 20% rule and a cent tolerance draw the same line,
mir_eval 0.8.2 gives the same numbers: 1 - overall_accuracy is FFE,
1 - raw_pitch_accuracy over the frames voiced in both is GPE, and with an
unlimited cent tolerance (cent_tolerance=np.inf) 1 - overall_accuracy
is VDE.
tools/validation/f0eval.py repeats that check on the tests' pair, 7
pairs at the edges of what can be scored and 400 random ones, and checks
every measure against a frame-by-frame implementation of its own as well.
With numpy 2.3.5 the rates agree to 1.1e-16 and the RMSE to 4.5e-13 cents.
What to watch for:
- The denominators are where papers part. GPE here is over N_VV. [CA12] replaced every tracker's voicing with the reference's, interpolating an F0 where a tracker gave none, [CH08] counted only where the reference and every compared algorithm called the sound pitched, and [P11] counts gross errors over all N frames, which is (N_VV / N) times the GPE here. The same two tracks give a different number under each; say which one a table holds.
- GPE on its own rewards a tracker for giving up. Calling a hard frame unvoiced takes it out of N_VV: turning one gross error into a V/U error lowers GPE, adds 1/N to VDE and leaves FFE where it was, which is why [CA09] proposed FFE. Report GPE with VDE, or FFE.
- VDE and FFE count every frame, so frames that both tracks call unvoiced dilute them: 300 frames with 30 voicing errors read 0.10, and with 100 frames of silence added at each end, 0.06. [P11] cut its utterances to 70 ms around the first and last pitch value. Trim every system's tracks the same way.
- Pooling and averaging are different numbers. Concatenate the tracks of a test set and score them once to pool their frames; the mean of per-file scores weighs a short file like a long one. A file with 5 gross errors in 10 frames voiced in both and one with 10 in 1000 average to a GPE of 0.255 and pool to 0.0149.
- 20% is 315.6 cents above the reference and 386.3 below it. Raw pitch accuracy, as MIREX and mir_eval score it [R14], takes a symmetric tolerance, 50 cents by default (2.9% up, 2.8% down), counts over the reference's voiced frames and reads an estimate's pitch whether or not it is called voiced, so 1 - RPA is not GPE.
- Negative values are refused, because they mean two things. In Keele's
reference [PMA95] they mark uncertain frames, which are to be dropped from
both tracks (
keep = ref >= 0). In MIREX files a negative estimate is an unvoiced frame with a pitch guess, whichnp.maximum(est, 0)reads as unvoiced. A masked array with masked frames is refused for the same reason:np.ma.filled(est, 0)reads them as unvoiced, andkeep = ~np.ma.getmaskarray(est)drops them from both tracks. - Nothing is shifted. A lag of one 10 ms frame on a voice gliding 2 octaves per second puts every frame 24 cents off, and [CH08] shifted the estimates by up to 100 ms to their best alignment before counting. Synthesized speech is seldom on the reference's frames at all and has to be lined up with it first, by DTW for instance; the alignment is then part of the number.
f0_rmsecounts gross errors at full size: one octave error among 100 frames voiced in both, the rest exact, is 120 cents. FPE leaves the gross errors out, and being a standard deviation it leaves a constant offset out too: an estimate 1% sharp on every frame has an FPE of 0 and an RMSE of 17.2 cents. An RMSE in Hz is another number, which weighs a high voice more (a semitone is 5.9 Hz at 100 Hz and 11.9 Hz at 200 Hz); an RMSE of ln F0 is the RMSE in cents divided by 1731.2.- FPE is relative, and a standard deviation that divides by the number of
fine frames; MATLAB's
stddivides by one fewer, which reads 0.5% higher at 100 frames. The fine errors of [CA12] are in Hz, and their SDFPE, like [NIZ03]'s RMS, takes a constant offset in; those of [R76] are in samples of pitch period; none of them is this number.
Classes:
| Name | Description |
|---|---|
F0Errors |
Every measure of |
Functions:
| Name | Description |
|---|---|
gross_pitch_error |
Gross pitch error: the share of the frames voiced in both tracks where |
voicing_decision_error |
Voicing decision error: the share of all frames where est and ref |
f0_frame_error |
F0 frame error: the share of all frames with any error, a voicing |
fine_pitch_error |
Fine pitch error: how much est scatters around ref on the frames |
f0_rmse |
Root mean square F0 error in cents over the frames voiced in both. |
f0_errors |
GPE, VDE, FFE, FPE and the RMSE in cents of est against ref in one |
F0Errors
¶
Bases: NamedTuple
Every measure of kudio.core.f0eval for one pair of tracks, and
the frame counts behind them.
The counts say how much each figure rests on: a GPE of 0.5 with
voiced_both=2 is one frame in two. To pool a test set, concatenate
its tracks and score them once rather than averaging these.
Attributes:
| Name | Type | Description |
|---|---|---|
gpe |
float
|
gross pitch error, |
vde |
float
|
voicing decision error, |
ffe |
float
|
F0 frame error, |
fpe |
float
|
fine pitch error, the standard deviation of |
rmse_cents |
float
|
RMS error over the frames voiced in both, in cents |
frames |
int
|
frames compared, N |
voiced_both |
int
|
frames voiced in both tracks, N_VV |
gross |
int
|
frames voiced in both whose estimate is more than the threshold off, N_GE |
voiced_to_unvoiced |
int
|
frames voiced in the reference and not in the estimate, N_VU |
unvoiced_to_voiced |
int
|
frames voiced in the estimate and not in the reference, N_UV |
vde
instance-attribute
¶
voicing decision error, (voiced_to_unvoiced + unvoiced_to_voiced) / frames
ffe
instance-attribute
¶
F0 frame error, (gross + voiced_to_unvoiced + unvoiced_to_voiced) / frames
fpe
instance-attribute
¶
fine pitch error, the standard deviation of (est - ref) / ref over
the voiced_both - gross frames without a gross error
rmse_cents
instance-attribute
¶
RMS error over the frames voiced in both, in cents
gross
instance-attribute
¶
frames voiced in both whose estimate is more than the threshold off, N_GE
voiced_to_unvoiced
instance-attribute
¶
frames voiced in the reference and not in the estimate, N_VU
unvoiced_to_voiced
instance-attribute
¶
frames voiced in the estimate and not in the reference, N_UV
gross_pitch_error
¶
Gross pitch error: the share of the frames voiced in both tracks where est is more than threshold off ref.
N_GE / N_VV, [CA09] eq. 2: a frame voiced in both counts when
|est - ref| > threshold * ref, which is the 20% rule with the
default, measured against the reference: 125 Hz against 100 counts, 100
against 125 does not. Frames voiced in only one track do not enter it,
so a tracker lowers its GPE by calling the frames it gets wrong
unvoiced; voicing_decision_error counts those, and
f0_frame_error counts both.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required | |
threshold
|
float
|
how far off, as a fraction of the reference F0, an estimate has to be to count: 0.2, the 20% the papers use. |
0.2
|
Returns:
| Type | Description |
|---|---|
float
|
the fraction of the N_VV frames with a gross error, 0 to 1. nan, with a warning, when no frame is voiced in both. |
Raises:
| Type | Description |
|---|---|
ValueError
|
for tracks of different lengths, two PitchTracks whose frame times differ, a PitchTrack with a voiced frame that has no F0 or without one finite time per frame, negative or infinite values, masked frames, an input that is not one track of numbers, or a threshold outside 0 to 1. |
Source code in kudio/core/f0eval.py
voicing_decision_error
¶
Voicing decision error: the share of all frames where est and ref disagree on whether the frame is voiced.
(N_VU + N_UV) / N, [CA09] eq. 1. Every frame counts in N, so frames both tracks call unvoiced, such as silence before and after the speech, lower it; trim the tracks of every system compared the same way.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required |
Returns:
| Type | Description |
|---|---|
float
|
the fraction of all N frames with a voicing error, 0 to 1. nan, with a warning, for two empty tracks. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Source code in kudio/core/f0eval.py
f0_frame_error
¶
F0 frame error: the share of all frames with any error, a voicing error or a gross pitch error.
(N_VU + N_UV + N_GE) / N, [CA09] eq. 4, which is
VDE + (N_VV / N) * GPE. Calling a frame unvoiced instead of getting
its pitch wrong moves the error from GPE to VDE and leaves FFE where it
was, which is why it is one number to compare trackers by.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required | |
threshold
|
float
|
as in |
0.2
|
Returns:
| Type | Description |
|---|---|
float
|
the fraction of all N frames with an error, 0 to 1. nan, with a warning, for two empty tracks. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Source code in kudio/core/f0eval.py
fine_pitch_error
¶
Fine pitch error: how much est scatters around ref on the frames it gets right, as a fraction of the reference F0.
The standard deviation of (est - ref) / ref over the frames voiced in
both whose estimate is within threshold of the reference, as [DA11]
defines FPE; times 100 it is their percentage. It divides by the number
of those frames, as [R76]'s eq. 3 does. Being a standard deviation, it
does not see a constant offset: an estimate 1% sharp on every frame
scores 0 here and 17.2 cents in f0_rmse.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required | |
threshold
|
float
|
the line between fine and gross, as in
|
0.2
|
Returns:
| Type | Description |
|---|---|
float
|
the standard deviation, as a fraction. nan, with a warning, when fewer than 2 frames are voiced in both without a gross error. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Source code in kudio/core/f0eval.py
f0_rmse
¶
Root mean square F0 error in cents over the frames voiced in both.
sqrt(mean((1200 log2(est / ref)) ** 2)), the F0 RMSE that speech
synthesis papers report. A cent is a hundredth of a semitone, so the
same error counts the same in a low voice and a high one, which an RMSE
in Hz does not. Gross errors count at full size: one octave error among
100 frames, the rest exact, is 120 cents. Synthesized speech has to be
put on the reference's frames first, by DTW for instance; this compares
the frames as given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required |
Returns:
| Type | Description |
|---|---|
float
|
the RMSE in cents. nan, with a warning, when no frame is voiced in both. |
Raises:
| Type | Description |
|---|---|
ValueError
|
as |
Source code in kudio/core/f0eval.py
f0_errors
¶
f0_errors(ref, est, *, threshold: float = 0.2) -> F0Errors
GPE, VDE, FFE, FPE and the RMSE in cents of est against ref in one pass, with the frame counts they come from.
>>> kudio.f0_errors([0, 100, 102, 104, 0, 0, 120, 121],
... [0, 101, 204, 0, 0, 98, 119, 0])
F0Errors(gpe=0.3333333333333333, vde=0.375, ffe=0.5,
fpe=0.009166666666666667, rmse_cents=692.9421890055355,
frames=8, voiced_both=3, gross=1, voiced_to_unvoiced=2,
unvoiced_to_voiced=1)
The octave error at frame 2 is 1200 of the RMSE's cents and none of
FPE's, which is left with the 1% and -0.83% of the other two frames.
The numbers are those of gross_pitch_error,
voicing_decision_error, f0_frame_error,
fine_pitch_error and f0_rmse, and so are the rules for
what each one cannot score.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
the reference track, F0 in Hz frame by frame with 0 or nan
where unvoiced, or a |
required | |
est
|
the estimate, on the same frames, in the same form. |
required | |
threshold
|
float
|
as in |
0.2
|
Returns:
| Type | Description |
|---|---|
F0Errors
|
an |
Raises:
| Type | Description |
|---|---|
ValueError
|