跳轉到

Recognizers, verifiers and pitch trackers

kudio.core.wer

Word and character error rates: how far a recogniser's transcript is from the reference transcript, counted in words (wer) or in characters (cer), with the counts behind each from word_errors and char_errors.

The hypothesis, what the recogniser wrote, is aligned with the reference, what was said, so that as few words as possible are substituted (S), deleted (D) or inserted (I), and the rate is those edits over the length of the reference:

WER = (S + D + I) / N,        N = H + S + D

H being the hits, the reference words the alignment pairs with the same word. The fewest edits is the Levenshtein distance between the two word sequences, the minimum edit distance of [JM] Sec. 2.9 with every edit costing 1, on which [JM] Sec. 16.6 defines WER. CER is the same count over characters. A corpus is scored as Kaldi's compute-wer and jiwer score one: the errors of all its pairs over the reference words of all of them. The sources:

[JM]     D. Jurafsky and J. H. Martin, "Speech and Language Processing",
         3rd ed., draft of 19 August 2026: Ch. 2, "Words and Tokens",
         Sec. 2.9, minimum edit distance, pp. 28-32; Ch. 16, "Automatic
         Speech Recognition", Sec. 16.6, word error rate, pp. 21-24.
         https://web.stanford.edu/~jurafsky/slp3/
[SCTK]   NIST Speech Recognition Scoring Toolkit, the sclite manual,
         doc/sclite.htm, master branch as read on 2026-11-12.
         https://github.com/usnistgov/SCTK
[Kaldi]  src/bin/compute-wer.cc, src/util/edit-distance-inl.h,
         src/util/kaldi-holder-inl.h and
         egs/wsj/s5/steps/scoring/score_kaldi_cer.sh, master branch as
         read on 2026-11-12. https://github.com/kaldi-asr/kaldi
[jiwer]  jiwer 2.3.0, jiwer/measures.py; jiwer 3.1.0 and 4.0.0,
         src/jiwer/process.py, transformations.py and transforms.py.
         https://github.com/jitsi/jiwer. Its alignments come from
         rapidfuzz, rapidfuzz/distance/Levenshtein_impl.hpp in
         https://github.com/rapidfuzz/rapidfuzz-cpp, main branch as read
         on 2026-11-12.
[ESPnet] espnet2/legacy/nets/e2e_asr_common.py (ErrorCalculator),
         egs2/TEMPLATE/asr1/asr.sh and espnet2/text/char_tokenizer.py,
         master branch as read on 2026-11-12.
         https://github.com/espnet/espnet

On the worked examples of [JM] these give the published counts: 6 substitutions, 1 deletion and 3 insertions in the 13 reference words of the CallHome utterance in Sec. 16.6, a WER of 76.9%, and the 5 edits between "intention" and "execution" in Sec. 2.9, split as its Fig. 2.17 splits them.

The total is fixed, the split is not. Reference "a c" against hypothesis "d d a" is three edits either as two substitutions and an insertion, or as two insertions, a hit on "a" and a deletion. Of the alignments with the fewest errors I take the one with the most hits. Since 2H + S = N + M - (S + D + I), M being the length of the hypothesis, that is also the one with the fewest substitutions, and it belongs to the pair: the four counts are the same whichever way the table is filled. A tool that builds one alignment instead picks among equally cheap steps by a fixed order, and its counts follow that order. compute-wer takes the diagonal only when it is strictly cheaper than both other steps, then a deletion only when it is strictly cheaper than an insertion, and an insertion otherwise, so it reports the pair above as 2 substitutions and 1 insertion where kudio reports 1 hit, 1 deletion and 2 insertions. jiwer counts the alignment rapidfuzz recovers from its own table, with an order of preference of its own. The WER agrees; a table of S, D and I from two tools need not. How often they part depends on how often the text has ties: never in the 5,000 utterances of the simulated corpus in tools/validation/wer.py (100,000 words, about 10% of them wrong), and on 618 of 18,420 random pairs of up to 12 words drawn from four, for compute-wer.

sclite can count more errors. It minimises a weighted distance: 0 for a hit, 3 for an insertion or a deletion and 4 for a substitution ([SCTK], "Dynamic Programming string alignment"), which is 3 (S + D + I) + S. Among the alignments with the fewest errors that picks the one with the fewest substitutions, so wherever sclite's alignment has the fewest errors, its four counts are the ones here. But it trades one more error for five fewer substitutions. Against the reference "one two three four five", the hypothesis "four five six seven eight" is 5 substitutions here, a WER of 1.0; sclite's weights pair up the two "four five" at a cost of 18 against 20, for 3 deletions and 3 insertions, 6 errors and a WER of 1.2. That happens on none of the simulated utterances, and on 59 of the 18,420 short pairs, with 711 more where it depends on how sclite breaks its own ties.

Words and characters. A word is whatever str.split() separates: runs of any whitespace split, tabs, line breaks and the full-width space U+3000 among them. compute-wer splits on ASCII whitespace. jiwer splits on the space character, after it collapses runs of two or more whitespace characters into one space, so a lone tab or line break there joins two words into one: "hello world" against "hello\nworld" is a WER of 1.0 in jiwer and 0 here. A character is a Unicode code point, as Python's str holds it.

For CER the tools disagree about spaces. Kaldi's score_kaldi_cer.sh splits each word into its characters and drops the spaces, and so does ESPnet's ErrorCalculator, which logs CER during training. jiwer counts every character from the first that is not whitespace to the last, spaces included, and so does the CER that ESPnet's asr.sh reports, which turns each space into a <space> token for sclite. I leave whitespace out by default. CER is the measure for Chinese and Japanese, which are written without spaces; there a space is layout, and counting it would make "我 愛 你" against "我愛你" 2 errors in 5 characters where nothing was misheard. spaces=True counts one space between each two words however wide the gap, which is what jiwer counts on text with single spaces between its words; jiwer counts each extra space, and a tab, as a character of its own.

What to watch for:

  • Nothing is normalised. Case, punctuation, digits against numbers written out, contractions and filler words all count as they stand: "Hello, world." against "hello world" is 2 substitutions, a WER of 1.0. Normalise both sides the same way before scoring, and say how next to the number. Lower-casing and turning punctuation into spaces, for example:
import re
def normalise(text):
    return re.sub(r"[^\w\s']", " ", text.lower())
kudio.wer([normalise(t) for t in refs], [normalise(t) for t in hyps])
  • Unicode is taken as it comes, with no normalisation form applied: "é" as one code point and as "e" followed by a combining accent are different strings, and cer("café", "cafe\u0301") is 2 errors in 4 characters, 0.5. Run unicodedata.normalize("NFC", text) over both sides when they may come from different sources; "NFKC" also folds full-width letters and digits (A, 1) into ASCII. Simplified and traditional characters stay different too: "我爱你" against "我愛你" is a CER of 1/3. And a code point is not always one letter on screen: an accent written as a combining mark, a Devanagari syllable or an emoji sequence counts as several.
  • A corpus rate is pooled, not averaged. One error in a 2-word utterance and none in an 18-word one is 1/20 = 0.05 pooled, and 0.25 as the mean of the two utterances' rates, which gives the short one the weight of the long one. Pass the whole corpus in one call rather than averaging calls.
  • WER is not a share of the words and has no upper bound, since insertions can outnumber the reference: "yes" against "yes yes yes" is 2 insertions over 1 word, a WER of 2.0.
  • Text without spaces between its words is one word to wer: "今天天氣很好" against "今天天氣不好" is a WER of 1.0 and a CER of 1/6. Use cer for Chinese and Japanese, or segment the words first and join them with spaces.
  • An empty reference has nothing to divide by. Inside a corpus that does not matter: its insertions count, as in compute-wer and jiwer 4.0, and wer(["a b", ""], ["a b", "c"]) is 1/2. When the whole reference side is empty, wer and cer warn and return nan; jiwer 4.0 returns the number of insertions there (0 for an empty pair), jiwer 2.3 to 3.1 raise as soon as any one reference is empty, and compute-wer prints nan or inf. word_errors and char_errors return the counts either way.
  • A list is a corpus of transcripts, paired in order. A sentence already cut into words, ["the", "cat"], would be read as two one-word transcripts, each aligned on its own; join the tokens with spaces first. A dict is read as utterance id to transcript and paired by key, as compute-wer pairs them. A pandas Series is paired by position, like a list, so two Series have to carry the same index in the same order or the call is refused: the same two transcripts with the hypotheses in the other order, their index following them, would score 1.0 by position and 0 by index. .to_dict() of both pairs them by index. A DataFrame is refused as well, since it iterates over its column names; pass its column of transcripts.
  • The work grows with the product of the two lengths of each pair. On a desktop CPU the simulated corpus, 100,000 words in 5,000 utterances, is scored in under 0.1 s, and its 530,000 characters in about 0.7 s; one pair of 10,000-word transcripts takes about 0.3 s, and one of 100,000 words about 25 s.

Classes:

Name Description
ErrorCounts

The counts behind a WER or a CER, from word_errors or

Functions:

Name Description
word_errors

Hits, substitutions, deletions and insertions of words in

char_errors

Hits, substitutions, deletions and insertions of characters in

wer

Word error rate of hypothesis against reference,

cer

Character error rate of hypothesis against reference,

ErrorCounts

Bases: NamedTuple

The counts behind a WER or a CER, from word_errors or char_errors, summed over every pair of a corpus.

hits + substitutions + deletions is the length of the reference and hits + substitutions + insertions that of the hypothesis; the rate is errors / reference_length. Of the alignments with the fewest errors, the counts are those of the one with the most hits. Why, and why another tool's split can differ while its total agrees: https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word

Attributes:

Name Type Description
hits int

reference tokens aligned with the same token of the hypothesis

substitutions int

reference tokens aligned with a different token

deletions int

reference tokens with nothing aligned to them

insertions int

hypothesis tokens with nothing aligned to them

reference_length int

tokens in the reference, hits + substitutions + deletions

errors int

substitutions + deletions + insertions, the edit distance.

hits instance-attribute

hits: int

reference tokens aligned with the same token of the hypothesis

substitutions instance-attribute

substitutions: int

reference tokens aligned with a different token

deletions instance-attribute

deletions: int

reference tokens with nothing aligned to them

insertions instance-attribute

insertions: int

hypothesis tokens with nothing aligned to them

reference_length instance-attribute

reference_length: int

tokens in the reference, hits + substitutions + deletions

errors property

errors: int

substitutions + deletions + insertions, the edit distance.

word_errors

word_errors(
    reference: Union[str, Sequence[str], Mapping[Any, str]],
    hypothesis: Union[
        str, Sequence[str], Mapping[Any, str]
    ],
) -> ErrorCounts

Hits, substitutions, deletions and insertions of words in hypothesis against reference, summed over the pairs of a corpus.

>>> kudio.word_errors("the cat sat on the mat", "the cat sat on mat")
ErrorCounts(hits=5, substitutions=0, deletions=1, insertions=0, reference_length=6)

Words are split on whitespace and compared as they stand, case and punctuation included. Of the alignments with the fewest errors, the counts are those of the one with the most hits. Why, and how another tool can split the same total differently: https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word

Parameters:

Name Type Description Default
reference Union[str, Sequence[str], Mapping[Any, str]]

what was said: one transcript, a sequence of them, or a dict from utterance id to transcript.

required
hypothesis Union[str, Sequence[str], Mapping[Any, str]]

what the recogniser wrote, in the same form as reference: paired in order for sequences, by key for dicts.

required

Returns:

Type Description
ErrorCounts

an ErrorCounts. A corpus with no reference words, or no pairs at all, has its counts returned like any other; it is the rate that cannot be taken from them.

Raises:

Type Description
ValueError

for a side that is not text, a sequence of text or a dict of text (bytes, None, a set, a number, a DataFrame or another 2-d table); for a transcript inside one that is not a str (None, NaN, bytes, a list of tokens); for one side a single str and the other a collection; and for sides that do not pair up, sequences of different lengths, dicts with different keys or two pandas Series with different indexes.

Source code in kudio/core/wer.py
def word_errors(reference: Union[str, Sequence[str], Mapping[Any, str]],
                hypothesis: Union[str, Sequence[str], Mapping[Any, str]]
                ) -> ErrorCounts:
    """Hits, substitutions, deletions and insertions of words in
    *hypothesis* against *reference*, summed over the pairs of a corpus.

    >>> kudio.word_errors("the cat sat on the mat", "the cat sat on mat")
    ErrorCounts(hits=5, substitutions=0, deletions=1, insertions=0, reference_length=6)

    Words are split on whitespace and compared as they stand, case and
    punctuation included. Of the alignments with the fewest errors, the
    counts are those of the one with the most hits. Why, and how another
    tool can split the same total differently:
    https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word

    :param reference: what was said: one transcript, a sequence of them, or a
        dict from utterance id to transcript.
    :param hypothesis: what the recogniser wrote, in the same form as
        *reference*: paired in order for sequences, by key for dicts.
    :raises ValueError: for a side that is not text, a sequence of text or a
        dict of text (bytes, None, a set, a number, a DataFrame or another
        2-d table); for a transcript inside one that is not a str (None,
        NaN, bytes, a list of tokens); for one side a single str and the
        other a collection; and for sides that do not pair up, sequences of
        different lengths, dicts with different keys or two pandas Series
        with different indexes.
    :returns: an :class:`ErrorCounts`. A corpus with no reference words, or
        no pairs at all, has its counts returned like any other; it is the
        rate that cannot be taken from them.
    """
    return _tally(_pairs(reference, hypothesis, 'word_errors'), _words)

char_errors

char_errors(
    reference: Union[str, Sequence[str], Mapping[Any, str]],
    hypothesis: Union[
        str, Sequence[str], Mapping[Any, str]
    ],
    *,
    spaces: bool = False,
) -> ErrorCounts

Hits, substitutions, deletions and insertions of characters in hypothesis against reference, summed over the pairs of a corpus.

>>> kudio.char_errors("我愛你", "我恨你")
ErrorCounts(hits=2, substitutions=1, deletions=0, insertions=0, reference_length=3)

A character is a Unicode code point, and nothing is normalised: an accent written as a combining mark is a character of its own, so cer("café", "cafe\u0301") is 0.5; a full-width letter is not its ASCII one; and "我爱你" against "我愛你" is 1/3. unicodedata.normalize over both sides takes care of the first with "NFC" and of the second as well with "NFKC"; nothing folds simplified into traditional.

Parameters:

Name Type Description Default
reference Union[str, Sequence[str], Mapping[Any, str]]

what was said, in any form word_errors takes.

required
hypothesis Union[str, Sequence[str], Mapping[Any, str]]

what the recogniser wrote, in the same form.

required
spaces bool

False (default) leaves all whitespace out, as Kaldi's CER recipe and ESPnet's training-time CER do. True counts one space between each two words, however the gap is written, and none at either end, as jiwer counts text with single spaces.

False

Returns:

Type Description
ErrorCounts

an ErrorCounts of characters.

Raises:

Type Description
ValueError

as word_errors, and for spaces other than True or False.

Source code in kudio/core/wer.py
def char_errors(reference: Union[str, Sequence[str], Mapping[Any, str]],
                hypothesis: Union[str, Sequence[str], Mapping[Any, str]], *,
                spaces: bool = False) -> ErrorCounts:
    """Hits, substitutions, deletions and insertions of characters in
    *hypothesis* against *reference*, summed over the pairs of a corpus.

    >>> kudio.char_errors("我愛你", "我恨你")
    ErrorCounts(hits=2, substitutions=1, deletions=0, insertions=0, reference_length=3)

    A character is a Unicode code point, and nothing is normalised: an
    accent written as a combining mark is a character of its own, so
    ``cer("café", "cafe\\u0301")`` is 0.5; a full-width letter is not its
    ASCII one; and "我爱你" against "我愛你" is 1/3. ``unicodedata.normalize``
    over both sides takes care of the first with "NFC" and of the second as
    well with "NFKC"; nothing folds simplified into traditional.

    :param reference: what was said, in any form :func:`word_errors` takes.
    :param hypothesis: what the recogniser wrote, in the same form.
    :param spaces: ``False`` (default) leaves all whitespace out, as Kaldi's
        CER recipe and ESPnet's training-time CER do. ``True`` counts one
        space between each two words, however the gap is written, and none
        at either end, as jiwer counts text with single spaces.
    :raises ValueError: as :func:`word_errors`, and for *spaces* other than
        True or False.
    :returns: an :class:`ErrorCounts` of characters.
    """
    tokens = _characters(_spaces(spaces, 'char_errors'))
    return _tally(_pairs(reference, hypothesis, 'char_errors'), tokens)

wer

wer(
    reference: Union[str, Sequence[str], Mapping[Any, str]],
    hypothesis: Union[
        str, Sequence[str], Mapping[Any, str]
    ],
) -> float

Word error rate of hypothesis against reference, (S + D + I) / N, as a fraction (0.25, not 25%). 0 is a perfect transcript and lower is better; insertions can take it above 1.

>>> kudio.wer("the cat sat on the mat", "the cat sat on mat")
0.16666666666666666
>>> kudio.wer(["a b", "c d e"], ["a x", "c d e"])        # 1 error, 5 words
0.2

A sequence or a dict of transcripts is one corpus: the errors of all its pairs over the reference words of all of them. Words are split on whitespace and nothing is normalised, so lower-case and strip punctuation first if case and punctuation should not count; one way is at https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word

Parameters:

Name Type Description Default
reference Union[str, Sequence[str], Mapping[Any, str]]

what was said: one transcript, a sequence of them, or a dict from utterance id to transcript.

required
hypothesis Union[str, Sequence[str], Mapping[Any, str]]

what the recogniser wrote, in the same form: paired in order for sequences, by key for dicts.

required

Returns:

Type Description
float

the rate. nan, with a warning, when the reference side has no words at all (an empty reference, references that are all empty, or no pairs), since a rate needs at least 1 reference word to divide by. An empty reference among others is scored, its insertions counted.

Raises:

Type Description
ValueError
Source code in kudio/core/wer.py
def wer(reference: Union[str, Sequence[str], Mapping[Any, str]],
        hypothesis: Union[str, Sequence[str], Mapping[Any, str]]) -> float:
    """Word error rate of *hypothesis* against *reference*,
    ``(S + D + I) / N``, as a fraction (0.25, not 25%). 0 is a perfect
    transcript and lower is better; insertions can take it above 1.

    >>> kudio.wer("the cat sat on the mat", "the cat sat on mat")
    0.16666666666666666
    >>> kudio.wer(["a b", "c d e"], ["a x", "c d e"])        # 1 error, 5 words
    0.2

    A sequence or a dict of transcripts is one corpus: the errors of all its
    pairs over the reference words of all of them. Words are split on
    whitespace and nothing is normalised, so lower-case and strip
    punctuation first if case and punctuation should not count; one way is at
    https://recklight.github.io/Kudio/guide/task-metrics/#which-errors-and-what-counts-as-a-word

    :param reference: what was said: one transcript, a sequence of them, or a
        dict from utterance id to transcript.
    :param hypothesis: what the recogniser wrote, in the same form: paired in
        order for sequences, by key for dicts.
    :raises ValueError: as :func:`word_errors`.
    :returns: the rate. nan, with a warning, when the reference side has no
        words at all (an empty reference, references that are all empty, or
        no pairs), since a rate needs at least 1 reference word to divide
        by. An empty reference among others is scored, its insertions
        counted.
    """
    pairs = _pairs(reference, hypothesis, 'wer')
    return _rate(_tally(pairs, _words), len(pairs), 'wer', 'word', 'words')

cer

cer(
    reference: Union[str, Sequence[str], Mapping[Any, str]],
    hypothesis: Union[
        str, Sequence[str], Mapping[Any, str]
    ],
    *,
    spaces: bool = False,
) -> float

Character error rate of hypothesis against reference, (S + D + I) / N over characters, as a fraction. 0 is a perfect transcript and lower is better; insertions can take it above 1.

>>> kudio.cer("今天天氣很好", "今天天氣不好")
0.16666666666666666
>>> kudio.cer("the cat sat", "the hat sat")               # 1 of 9 letters
0.1111111111111111
>>> kudio.cer("the cat sat", "the hat sat", spaces=True)  # 1 of 11
0.09090909090909091

A corpus is pooled as in wer. The two settings of spaces give different numbers on the same text, since the spaces join the denominator too, so say which one a table reports.

Parameters:

Name Type Description Default
reference Union[str, Sequence[str], Mapping[Any, str]]

what was said, in any form wer takes.

required
hypothesis Union[str, Sequence[str], Mapping[Any, str]]

what the recogniser wrote, in the same form.

required
spaces bool

False (default) leaves all whitespace out, as Kaldi's CER recipe and ESPnet's training-time CER do, so a hypothesis spaced differently from its reference costs nothing. True counts one space between each two words, however the gap is written, and none at either end: a missing or an extra word boundary is then an error, as it is in jiwer.

False

Returns:

Type Description
float

the rate. nan, with a warning, when the reference side has no characters other than whitespace, since a rate needs at least 1 to divide by.

Raises:

Type Description
ValueError

as word_errors, and for spaces other than True or False.

Source code in kudio/core/wer.py
def cer(reference: Union[str, Sequence[str], Mapping[Any, str]],
        hypothesis: Union[str, Sequence[str], Mapping[Any, str]], *,
        spaces: bool = False) -> float:
    """Character error rate of *hypothesis* against *reference*,
    ``(S + D + I) / N`` over characters, as a fraction. 0 is a perfect
    transcript and lower is better; insertions can take it above 1.

    >>> kudio.cer("今天天氣很好", "今天天氣不好")
    0.16666666666666666
    >>> kudio.cer("the cat sat", "the hat sat")               # 1 of 9 letters
    0.1111111111111111
    >>> kudio.cer("the cat sat", "the hat sat", spaces=True)  # 1 of 11
    0.09090909090909091

    A corpus is pooled as in :func:`wer`. The two settings of *spaces* give
    different numbers on the same text, since the spaces join the
    denominator too, so say which one a table reports.

    :param reference: what was said, in any form :func:`wer` takes.
    :param hypothesis: what the recogniser wrote, in the same form.
    :param spaces: ``False`` (default) leaves all whitespace out, as Kaldi's
        CER recipe and ESPnet's training-time CER do, so a hypothesis spaced
        differently from its reference costs nothing. ``True`` counts one
        space between each two words, however the gap is written, and none
        at either end: a missing or an extra word boundary is then an error,
        as it is in jiwer.
    :raises ValueError: as :func:`word_errors`, and for *spaces* other than
        True or False.
    :returns: the rate. nan, with a warning, when the reference side has no
        characters other than whitespace, since a rate needs at least 1 to
        divide by.
    """
    tokens = _characters(_spaces(spaces, 'cer'))
    pairs = _pairs(reference, hypothesis, 'cer')
    return _rate(_tally(pairs, tokens), len(pairs), 'cer', 'character',
                 'characters other than whitespace')

kudio.core.verification

Detection measures for speaker verification, and for any task scored trial by trial: the equal error rate and the minimum detection cost of NIST's speaker recognition evaluations, and the calibration measures that read the scores as likelihood ratios.

A verification system gives every trial, an enrolled speaker against a test recording, a score that is higher the more it believes the two are one talker, and a threshold turns the scores into decisions: a target trial scored below it is a miss, a non-target trial at or above it a false alarm. Each distinct score is a threshold of its own, so a list of trials is a set of operating points (P_miss, P_fa), the points of its DET curve (Martin et al. 1997), and both measures read one number off them:

  • eer, the error rate at which P_miss and P_fa are equal;
  • min_dcf, the lowest weighted sum of the two over every threshold, NIST's C_Det divided by the cost of deciding without the scores.

det_curve returns the operating points themselves, for a plot or for choosing a threshold.

Both judge the ranking alone. Where the scores are meant as natural-log likelihood ratios, as NIST's plans since SRE 2016 ask for, the decisions they make themselves can be costed as well:

  • act_dcf, the normalized cost at the threshold Bayes' rule sets for the application, and c_primary, its mean at SRE 2016's two priors;
  • cllr, Brümmer and du Preez's C_llr, the cost over every application at once in bits, and min_cllr, what the ranking would cost after the best calibration it allows.

Which EER. P_miss moves in steps of 1/T and P_fa in steps of 1/N, for T target and N non-target trials, and as a rule no operating point has the two equal. Every EER is therefore a rule for reading between the points, and the rules in use part on small lists. The default joins successive operating points with straight lines and takes the rate where the line crosses P_miss = P_fa. I chose it for three reasons:

  • It is the rule of the scorers the evaluations published. The VoxSRC 2020-2023 scorer finds the crossing on scikit-learn's ROC curve with brentq, and NIST's SRE 2016 scoring script interpolates between the two operating points either side of it, which is the same crossing unless a target and a non-target share a score. With such ties that script's EER moves with the order of the list; the second point under "What to watch for" has by how much.
  • A real threshold reaches it, unless the trials tied at the crossing include both a target and a non-target: one of the operating points there has one rate equal to the EER and the other at or below it.
  • Across a run of tied scores the line is what accepting each tied trial with one and the same probability gives on average, so the order of the list cannot move the result.

method='rocch' gives the ROCCH-EER of the BOSARIS toolkit instead (Brümmer and de Villiers, Sec. 2.6.2): the crossing of the convex hull of the operating points, which they propose as a well-defined version of the EER. It is the largest, over every prior p, of the lowest p P_miss + (1 - p) P_fa any threshold gives (their Eq. 26), and it is never above the default.

Eleven trials show how far apart the rules can be. Ranked from the lowest score they are four non-targets, three targets, two non-targets and two targets, and the scorers read:

kudio's default, NIST's SRE 2016 script, the VoxSRC scorer    1/3 = 0.333
kudio, method='rocch'                                         3/14 = 0.214
Kaldi's compute-eer                                           0.4
SpeechBrain's EER                                             0.367
voxceleb_trainer's tuneThresholdfromScore                     0.6

Kaldi counts up through the target scores and reports the miss rate at the first one where the false-alarm rate is no higher than the miss rate. SpeechBrain takes the mean of the two rates where they are closest. voxceleb_trainer takes the larger of the two where they are closest, among the points scikit-learn's roc_curve keeps by default; that drops the points inside straight runs, here the one at P_fa = 1/3, P_miss = 0.4. On a list of 40,000 trials, half of them targets, with Gaussian scores, the default reads 1.090% and the hull 1.081%, and on that list and five more like it the three other scorers stay within 0.01 percentage points of the default.

Which cost. NIST's plans write the cost of a threshold as:

C_Det = c_miss p_target P_miss + c_fa (1 - p_target) P_fa

and divide it by C_Default = min(c_miss p_target, c_fa (1 - p_target)), the cost of always rejecting or always accepting (SRE 2008 and 2010, Sec. 3; SRE 2016 and 2019, Sec. 3.1, Eqs. 2-4). min_dcf is the lowest of these over every threshold, which is the minimum NIST reports beside the actual cost, "assuming perfect calibration" in the words of the SRE 2010 plan. The evaluations do not agree on the parameters, which is why p_target has no default:

SRE 2008, and SRE 2010's historical cost   c_miss 10, c_fa 1, p_target 0.01
SRE 2010, core condition                   c_miss 1,  c_fa 1, p_target 0.001
SRE 2016 and the SRE 2019 CTS challenge    c_miss 1,  c_fa 1, p_target 0.01 and 0.005
SRE 2019 audio-visual, VoxSRC 2020-2023    c_miss 1,  c_fa 1, p_target 0.05

The scorers' defaults differ as well: 0.01 in Kaldi's compute_min_dcf.py and SpeechBrain's minDCF, 0.05 in the VoxSRC scorer and voxceleb_trainer. SpeechBrain's minDCF returns C_Det before the division. Its VoxCeleb recipes print that times 100, which is the normalized cost at its default p_target = 0.01 with unit costs and nowhere else: at 0.05 it is five times the normalized cost.

What to watch for:

  • A higher score has to mean "more likely the same speaker". Distances go in as their negatives. A list ranked exactly the wrong way round reads 1.0 by default and 0.5 with method='rocch', whose hull always contains the line of guessing.
  • Only distinct scores are thresholds. Kaldi's compute_min_dcf.py, which the VoxSRC scorer and voxceleb_trainer copy, and NIST's SRE 2016 script rank the trials one at a time, and so also put operating points between two trials with the same score, which no threshold reaches and which move with the order of the list. On 22,000 trials whose scores were rounded to a tenth of their standard deviation, listed non-targets first, that rule gives a min DCF of 0.400 at p_target = 0.05, where no threshold does better than 0.436. NIST's script reads the EER of the same list as 0.05985 when its sort leaves the tied trials non-targets first and 0.07050 when it leaves them targets first; the default reads 0.06535 in any order.
  • The rates move in steps of 1/T and 1/N, and at a small p_target the false alarms decide. With unit costs the normalized cost is P_miss + beta P_fa, beta = (1 - p_target) / p_target, so at 0.001 each false alarm among 10,000 non-target trials adds 0.0999. A min DCF at that prior read off fewer non-target trials is the count of a handful of errors.
  • p_target is the prior of the application the cost stands for, not the share of target trials in the list; the SRE 2010 plan says as much of its 0.001.
  • One cost has more than one spelling. Costs c_miss and c_fa at prior p give the same normalized cost as unit costs at the effective prior c_miss p / (c_miss p + c_fa (1 - p)) (Brümmer and de Villiers, Eq. 10), so SRE 2008's (10, 1, 0.01) is p_target = 0.0917 with unit costs, not 0.01. The copy of NIST's SRE 2016 script that WeSpeaker distributes prints the cost at (1, 1, 0.01) under the name minDCF08. On the first list of 40,000 trials above, the two read 0.059 and 0.143.
  • The minimum is found with the labels in hand, so it is the cost of the best threshold for this very list. A threshold for new trials is chosen on a development list, with det_curve. NIST's primary costs in SRE 2016 and 2019 are actual costs, at the threshold log(beta) on calibrated log-likelihood ratios, and SRE 2016 and the 2019 CTS challenge average them over partitions of the list; min_dcf pools the list as it comes and needs no calibration.

Sources: NIST, "The NIST Year 2008 Speaker Recognition Evaluation Plan" (3 April 2008); "The NIST Year 2010 Speaker Recognition Evaluation Plan" (version r6, 21 April 2010); "NIST 2016 Speaker Recognition Evaluation Plan" (4 August 2016); "NIST 2019 Speaker Recognition Evaluation: CTS Challenge" (22 July 2019); "NIST 2019 Speaker Recognition Evaluation Plan" (16 August 2019, the audio-visual evaluation). N. Brümmer and E. de Villiers, "The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF", NIST SRE'11 Analysis Workshop, December 2011, arXiv:1304.2865. A. Martin, G. Doddington, T. Kamm, M. Ordowski and M. Przybocki, "The DET curve in assessment of detection task performance", Proc. Eurospeech 1997, pp. 1895-1898, doi:10.21437/Eurospeech.1997-504. [BD06] N. Brümmer and J. du Preez, "Application-independent evaluation of speaker detection", Computer Speech and Language 20(2-3), pp. 230-275, 2006, doi:10.1016/j.csl.2005.08.001. The other scorers are the code they published: Kaldi's src/ivectorbin/compute-eer.cc and egs/sre08/v1/sid/compute_min_dcf.py, the VoxSRC 2020-2023 validation toolkits (compute_EER.py, compute_min_dcf.py), clovaai/voxceleb_trainer's tuneThreshold.py, SpeechBrain's speechbrain/utils/metric_stats.py, and NIST's SRE 2016 scoring script (version 4.1, as WeSpeaker distributes it in wespeaker/utils/score_metrics.py, last changed in commit 3556586 of 2022-03-25; its min_dcf_08 is compute_c_norm(fnr, fpr, p_target=0.01, c_miss=1, c_fa=1)). Their numbers above come from those rules restated in tools/validation/verification.py, which reruns every comparison here; tools/validation/calibration.py does the same for the calibration measures, with PYLLR's rules (github.com/bsxfan/PYLLR, commit 442b10b). Each call sorts the list once, and a million trials take about a tenth of a second on a desktop CPU.

Classes:

Name Description
DetCurve

The operating points of a trial list, one per threshold, in rising

Functions:

Name Description
det_curve

Every operating point of a trial list: for each threshold, the miss

eer

Equal error rate of a trial list: the rate at which misses and false

min_dcf

Minimum normalized detection cost of a trial list, as NIST's speaker

cllr

Log-likelihood-ratio cost of a trial list, in bits: how much the

min_cllr

C_llr of the best calibration the ranking allows, in bits: what

act_dcf

Actual normalized detection cost: the cost of the decisions the LLRs

c_primary

NIST's primary cost of SRE 2016 and the SRE 2019 CTS challenge: the

Attributes:

Name Type Description
C_PRIMARY_PRIORS

The two priors NIST's SRE 2016 and the SRE 2019 CTS challenge average

C_PRIMARY_PRIORS module-attribute

C_PRIMARY_PRIORS = (0.01, 0.005)

The two priors NIST's SRE 2016 and the SRE 2019 CTS challenge average their primary cost over, with unit costs (Sec. 3.1, Table 1 of the 2016 plan).

DetCurve

Bases: NamedTuple

The operating points of a trial list, one per threshold, in rising order of threshold.

A trial is accepted when its score is at or above the threshold. The first point accepts every trial (P_miss 0, P_fa 1) and the last, at inf, rejects every trial (P_miss 1, P_fa 0); in between there is one point per distinct score. P_miss never falls and P_fa never rises from one point to the next.

Attributes:

Name Type Description
thresholds ndarray

the thresholds, float64: every distinct score, then inf

p_miss ndarray

share of target trials scored below each threshold

p_fa ndarray

share of non-target trials scored at or above each threshold

thresholds instance-attribute

thresholds: ndarray

the thresholds, float64: every distinct score, then inf

p_miss instance-attribute

p_miss: ndarray

share of target trials scored below each threshold

p_fa instance-attribute

p_fa: ndarray

share of non-target trials scored at or above each threshold

det_curve

det_curve(scores, labels) -> DetCurve

Every operating point of a trial list: for each threshold, the miss and false-alarm rates of accepting the trials scored at or above it.

>>> curve = kudio.det_curve(dev_scores, dev_labels)
>>> worst = np.maximum(curve.p_miss, curve.p_fa)
>>> curve.thresholds[np.argmin(worst)]
>>> cost = 0.05 * curve.p_miss + 0.95 * curve.p_fa
>>> curve.thresholds[np.argmin(cost)]

The thresholds are the distinct scores, so trials with the same score are accepted or rejected together; no point sits between them. The order is rising threshold, the reverse of scikit-learn's roc_curve, and nothing is dropped: the points inside a straight run are kept, and so are the two ends, accepting everything and rejecting everything. scikit-learn's det_curve rises as well, but keeps one point where P_miss is 0 and one where P_fa is lowest and leaves out the points beyond them, so whether it has either end depends on the list and on the release: 1.6.1 never returns the one at inf, 1.7.2 can. For a DET plot, put both rates through scipy.stats.norm.ppf; the ends map to infinity, which matplotlib leaves out.

The first threshold in the example is the one whose larger error rate is the smallest. That rate is the EER of eer, unless the trials tied at the crossing include both a target and a non-target; then no single threshold reaches the EER, and this one comes closest. The second is the threshold of min_dcf at p_target 0.05 with unit costs. Chosen on a development list, either can be used on new trials.

Parameters:

Name Type Description Default
scores

one score per trial, higher for "more likely a target". Integer scores are taken as the numbers they are, and every score is compared as float64: integers past 2**53 that float64 cannot tell apart count as tied, so that scores >= threshold accepts exactly the trials each point counts.

required
labels

one per trial, True or 1 for a target trial and False or 0 for a non-target one.

required

Returns:

Type Description
DetCurve

a DetCurve. For a list with no target trials p_miss is nan throughout, and with no non-target trials p_fa is, with a warning; an empty list gives the single point at inf, both rates nan.

Raises:

Type Description
ValueError

for scores that are not finite numbers float64 can hold, labels other than booleans or 0/1, a two-dimensional array (a (n, 1) or (1, n) one is read as n trials), a masked array with trials masked, or scores and labels of different lengths.

Source code in kudio/core/verification.py
def det_curve(scores, labels) -> DetCurve:
    """Every operating point of a trial list: for each threshold, the miss
    and false-alarm rates of accepting the trials scored at or above it.

    >>> curve = kudio.det_curve(dev_scores, dev_labels)      # doctest: +SKIP
    >>> worst = np.maximum(curve.p_miss, curve.p_fa)         # doctest: +SKIP
    >>> curve.thresholds[np.argmin(worst)]                   # doctest: +SKIP
    >>> cost = 0.05 * curve.p_miss + 0.95 * curve.p_fa       # doctest: +SKIP
    >>> curve.thresholds[np.argmin(cost)]                    # doctest: +SKIP

    The thresholds are the distinct scores, so trials with the same score are
    accepted or rejected together; no point sits between them. The order is
    rising threshold, the reverse of scikit-learn's ``roc_curve``, and
    nothing is dropped: the points inside a straight run are kept, and so are
    the two ends, accepting everything and rejecting everything.
    scikit-learn's ``det_curve`` rises as well, but keeps one point where
    P_miss is 0 and one where P_fa is lowest and leaves out the points beyond
    them, so whether it has either end depends on the list and on the
    release: 1.6.1 never returns the one at ``inf``, 1.7.2 can. For a DET
    plot, put both rates through ``scipy.stats.norm.ppf``; the ends map to
    infinity, which matplotlib leaves out.

    The first threshold in the example is the one whose larger error rate is
    the smallest. That rate is the EER of :func:`eer`, unless the trials tied
    at the crossing include both a target and a non-target; then no single
    threshold reaches the EER, and this one comes closest. The second is the threshold of :func:`min_dcf` at
    p_target 0.05 with unit costs. Chosen on a development list, either can
    be used on new trials.

    :param scores: one score per trial, higher for "more likely a target".
        Integer scores are taken as the numbers they are, and every score is
        compared as float64: integers past 2**53 that float64 cannot tell
        apart count as tied, so that ``scores >= threshold`` accepts exactly
        the trials each point counts.
    :param labels: one per trial, True or 1 for a target trial and False or
        0 for a non-target one.
    :raises ValueError: for scores that are not finite numbers float64 can
        hold, labels other than booleans or 0/1, a two-dimensional array (a
        ``(n, 1)`` or ``(1, n)`` one is read as n trials), a masked array
        with trials masked, or scores and labels of different lengths.
    :returns: a :class:`DetCurve`. For a list with no target trials p_miss
        is nan throughout, and with no non-target trials p_fa is, with a
        warning; an empty list gives the single point at ``inf``, both rates
        nan.
    """
    counts = _counted(scores, labels, 'det_curve')
    why = _unscorable(counts, 'det_curve')
    if why is not None:
        nan = ("both rates" if not counts.targets and not counts.nontargets
               else "p_miss" if not counts.targets else "p_fa")
        warn(f"{why}; {nan} came out nan")
    p_miss = (counts.misses / counts.targets if counts.targets
              else np.full(counts.misses.size, np.nan))
    p_fa = (counts.false_alarms / counts.nontargets if counts.nontargets
            else np.full(counts.false_alarms.size, np.nan))
    return DetCurve(counts.thresholds, p_miss, p_fa)

eer

eer(scores, labels, *, method: str = 'linear') -> float

Equal error rate of a trial list: the rate at which misses and false alarms are equally likely. 0 is perfect separation, 0.5 is guessing.

>>> scores = list(range(11))
>>> labels = [0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1]
>>> kudio.eer(scores, labels)
0.3333333333333333
>>> kudio.eer(scores, labels, method='rocch')
0.21428571428571427

The result is a proportion; Kaldi's compute-eer, voxceleb_trainer and the VoxSRC scorer print percent. Only the order of the scores matters, so any strictly rising transformation of them leaves it unchanged.

Parameters:

Name Type Description Default
scores

one score per trial, higher for "more likely a target".

required
labels

one per trial, True or 1 for a target trial and False or 0 for a non-target one.

required
method str

'linear' (default): straight lines join successive operating points, and the EER is where they cross P_miss = P_fa, as the VoxSRC 2020-2023 scorer computes it, and NIST's SRE 2016 script on lists where no target and non-target share a score; with such ties that script's result moves with the order of the list. Unless the trials tied at the crossing include both a target and a non-target, some threshold has one error rate equal to it and the other at or below it. 'rocch': the crossing of the convex hull of the operating points, BOSARIS's ROCCH-EER, never above the default. The module docstring has why the default is the default and how far apart the two are.

'linear'

Returns:

Type Description
float

the EER, 0 to 1 (0 to 0.5 with 'rocch'). nan, with a warning, for a list without a target or without a non-target trial.

Raises:

Type Description
ValueError

for an unknown method, and for the trial lists det_curve refuses.

Source code in kudio/core/verification.py
def eer(scores, labels, *, method: str = 'linear') -> float:
    """Equal error rate of a trial list: the rate at which misses and false
    alarms are equally likely. 0 is perfect separation, 0.5 is guessing.

    >>> scores = list(range(11))
    >>> labels = [0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1]
    >>> kudio.eer(scores, labels)                           # doctest: +SKIP
    0.3333333333333333
    >>> kudio.eer(scores, labels, method='rocch')           # doctest: +SKIP
    0.21428571428571427

    The result is a proportion; Kaldi's ``compute-eer``, voxceleb_trainer and
    the VoxSRC scorer print percent. Only the order of the scores matters,
    so any strictly rising transformation of them leaves it unchanged.

    :param scores: one score per trial, higher for "more likely a target".
    :param labels: one per trial, True or 1 for a target trial and False or
        0 for a non-target one.
    :param method: ``'linear'`` (default): straight lines join successive
        operating points, and the EER is where they cross P_miss = P_fa, as
        the VoxSRC 2020-2023 scorer computes it, and NIST's SRE 2016 script
        on lists where no target and non-target share a score; with such
        ties that script's result moves with the order of the list. Unless the trials tied at the crossing include both a target and a
        non-target, some threshold has one error rate equal to it and the
        other at or below it. ``'rocch'``: the crossing of the convex hull of
        the operating points, BOSARIS's ROCCH-EER, never above the default.
        The module docstring has why the default is the default and how far
        apart the two are.
    :raises ValueError: for an unknown *method*, and for the trial lists
        :func:`det_curve` refuses.
    :returns: the EER, 0 to 1 (0 to 0.5 with ``'rocch'``). nan, with a
        warning, for a list without a target or without a non-target trial.
    """
    if method not in _EER_METHODS:
        raise ValueError(f"eer(): method has to be one of {_EER_METHODS}, got "
                         f"{method!r}")
    counts = _counted(scores, labels, 'eer')
    why = _unscorable(counts, 'eer')
    if why is not None:
        warn(f"{why}; scored nan")
        return float('nan')
    # D = misses N - false alarms T has the sign of P_miss - P_fa and rises
    # strictly from point to point. It and the hull's cross products are at
    # most 2 T N <= n**2 / 2, so int64 holds them exactly below 2**32 trials.
    gap = counts.misses * counts.nontargets - counts.false_alarms * counts.targets
    if method == 'rocch':
        return float(_hull_crossing(counts, gap)[0])
    k = int(np.searchsorted(gap, 0))        # the first point with D >= 0
    if gap[k] == 0:
        return float(Fraction(int(counts.misses[k]), counts.targets))
    return float(_crossing(counts, k - 1, k))

min_dcf

min_dcf(
    scores,
    labels,
    *,
    p_target: float,
    c_miss: float = 1.0,
    c_fa: float = 1.0,
) -> float

Minimum normalized detection cost of a trial list, as NIST's speaker recognition evaluations define it. 0 is perfect; 1 is no better than deciding without the scores.

VoxSRC's cost, then SRE 2008's:

>>> kudio.min_dcf(scores, labels, p_target=0.05)
>>> kudio.min_dcf(scores, labels, p_target=0.01, c_miss=10)

The lowest, over every threshold, of:

(c_miss p_target P_miss + c_fa (1 - p_target) P_fa) / C_Default
C_Default = min(c_miss p_target, c_fa (1 - p_target))

The thresholds are the operating points of det_curve, including accepting everything and rejecting everything, so the result never exceeds 1. Tied scores are one threshold, as there.

Parameters:

Name Type Description Default
scores

one score per trial, higher for "more likely a target".

required
labels

one per trial, True or 1 for a target trial and False or 0 for a non-target one.

required
p_target float

prior probability of a target trial in the application the cost stands for, strictly between 0 and 1. It has no default, because the evaluations disagree: 0.001 in NIST's SRE 2010, 0.01 and 0.005 in SRE 2016 and the 2019 CTS challenge (whose primary cost averages the two), 0.05 in the SRE 2019 audio-visual evaluation and VoxSRC; the module docstring has the table.

required
c_miss float

cost of a miss: 10 in SRE 2008 and in SRE 2010's historical cost, 1 in the later plans.

1.0
c_fa float

cost of a false alarm, 1 in every plan cited here.

1.0

Returns:

Type Description
float

the normalized minimum cost, 0 to 1. nan, with a warning, for a list without a target or without a non-target trial.

Raises:

Type Description
ValueError

for a p_target outside (0, 1), a cost that is not positive and finite, a parameter that is not a number, weights c_miss p_target or c_fa (1 - p_target) below 2.2e-308, the smallest normal float64, and for the trial lists det_curve refuses.

Source code in kudio/core/verification.py
def min_dcf(scores, labels, *, p_target: float, c_miss: float = 1.0,
            c_fa: float = 1.0) -> float:
    """Minimum normalized detection cost of a trial list, as NIST's speaker
    recognition evaluations define it. 0 is perfect; 1 is no better than
    deciding without the scores.

    VoxSRC's cost, then SRE 2008's:

    >>> kudio.min_dcf(scores, labels, p_target=0.05)              # doctest: +SKIP
    >>> kudio.min_dcf(scores, labels, p_target=0.01, c_miss=10)   # doctest: +SKIP

    The lowest, over every threshold, of::

        (c_miss p_target P_miss + c_fa (1 - p_target) P_fa) / C_Default
        C_Default = min(c_miss p_target, c_fa (1 - p_target))

    The thresholds are the operating points of :func:`det_curve`, including
    accepting everything and rejecting everything, so the result never
    exceeds 1. Tied scores are one threshold, as there.

    :param scores: one score per trial, higher for "more likely a target".
    :param labels: one per trial, True or 1 for a target trial and False or
        0 for a non-target one.
    :param p_target: prior probability of a target trial in the application
        the cost stands for, strictly between 0 and 1. It has no default,
        because the evaluations disagree: 0.001 in NIST's SRE 2010, 0.01 and
        0.005 in SRE 2016 and the 2019 CTS challenge (whose primary cost
        averages the two), 0.05 in the SRE 2019 audio-visual evaluation and
        VoxSRC; the module docstring has the table.
    :param c_miss: cost of a miss: 10 in SRE 2008 and in SRE 2010's
        historical cost, 1 in the later plans.
    :param c_fa: cost of a false alarm, 1 in every plan cited here.
    :raises ValueError: for a *p_target* outside (0, 1), a cost that is not
        positive and finite, a parameter that is not a number, weights
        c_miss p_target or c_fa (1 - p_target) below 2.2e-308, the smallest
        normal float64, and for the trial lists :func:`det_curve` refuses.
    :returns: the normalized minimum cost, 0 to 1. nan, with a warning, for
        a list without a target or without a non-target trial.
    """
    weight_miss, weight_fa = _weights(p_target, c_miss, c_fa, 'min_dcf')
    counts = _counted(scores, labels, 'min_dcf')
    why = _unscorable(counts, 'min_dcf')
    if why is not None:
        warn(f"{why}; scored nan")
        return float('nan')
    # rates first: they are exactly 0 and 1 at the two ends, so the cheaper
    # end costs exactly C_Default and the result cannot round past 1
    cost = (weight_miss * (counts.misses / counts.targets)
            + weight_fa * (counts.false_alarms / counts.nontargets))
    return float(np.min(cost)) / min(weight_miss, weight_fa)

cllr

cllr(llrs, labels) -> float

Log-likelihood-ratio cost of a trial list, in bits: how much the scores, read as likelihood ratios, cost over every application at once. 0 is perfect; 1 is what LLRs of 0 throughout cost, a system that always says it cannot tell.

>>> kudio.cllr(llrs, labels)

Brümmer and du Preez's C_llr [BD06]:

C_llr = ( mean_tar log2(1 + exp(-llr)) + mean_non log2(1 + exp(llr)) ) / 2

It charges for discrimination and for calibration together: a list ranked perfectly still costs where its LLRs say less, or more, than the evidence holds. min_cllr is what the same ranking costs once calibrated as well as it can be, and the difference is the cost of calibration.

Parameters:

Name Type Description Default
llrs

one natural-log likelihood ratio per trial, as BOSARIS and NIST's SRE plans take them; a base-10 LLR, the forensic custom, has to be multiplied by ln 10 = 2.3026 first. Raw scores, cosine similarities say, are not LLRs, and their C_llr says little: calibrate them first. +inf and -inf are LLRs of certainty, and a certainty that is wrong costs inf.

required
labels

as det_curve.

required

Returns:

Type Description
float

C_llr in bits. nan, with a warning, for a list without a target or without a non-target trial.

Raises:

Type Description
ValueError

for an llr that is NaN, and for the lists det_curve refuses on other grounds.

Source code in kudio/core/verification.py
def cllr(llrs, labels) -> float:
    """Log-likelihood-ratio cost of a trial list, in bits: how much the
    scores, read as likelihood ratios, cost over every application at once.
    0 is perfect; 1 is what LLRs of 0 throughout cost, a system that always
    says it cannot tell.

    >>> kudio.cllr(llrs, labels)                                  # doctest: +SKIP

    Brümmer and du Preez's C_llr [BD06]::

        C_llr = ( mean_tar log2(1 + exp(-llr)) + mean_non log2(1 + exp(llr)) ) / 2

    It charges for discrimination and for calibration together: a list ranked
    perfectly still costs where its LLRs say less, or more, than the
    evidence holds. :func:`min_cllr` is what the same ranking costs once
    calibrated as well as it can be, and the difference is the cost of
    calibration.

    :param llrs: one natural-log likelihood ratio per trial, as BOSARIS and
        NIST's SRE plans take them; a base-10 LLR, the forensic custom, has
        to be multiplied by ln 10 = 2.3026 first. Raw scores, cosine
        similarities say, are not LLRs, and their C_llr says little:
        calibrate them first. +inf and -inf are LLRs of certainty, and a
        certainty that is wrong costs inf.
    :param labels: as :func:`det_curve`.
    :raises ValueError: for an llr that is NaN, and for the lists
        :func:`det_curve` refuses on other grounds.
    :returns: C_llr in bits. nan, with a warning, for a list without a
        target or without a non-target trial.
    """
    lists = _llr_lists(llrs, labels, 'cllr')
    if lists is None:
        return float('nan')
    targets, nontargets = lists
    # log(1 + exp(x)) without overflow: logaddexp(0, x)
    return float((np.mean(np.logaddexp(0.0, -targets))
                  + np.mean(np.logaddexp(0.0, nontargets))) / (2.0 * math.log(2.0)))

min_cllr

min_cllr(llrs, labels) -> float

C_llr of the best calibration the ranking allows, in bits: what cllr would be if the scores were turned into LLRs by the one rising mapping that suits this list best.

>>> kudio.min_cllr(scores, labels)

The mapping is the pool-adjacent-violators fit of the target share against the scores, which gives each trial a posterior probability; less the log of the list's own prior odds, T/N, those are LLRs, and their C_llr is this [BD06]. The scores only have to rank the trials, so raw scores will do. Tied scores share one LLR, as they share one threshold in det_curve.

BOSARIS's min_cllr.m sorts the non-targets of a tie before its targets and so lets the fit split ties, which reads lower where targets and non-targets share scores: by up to 0.30 bits on the rounded lists of kudio's validation script. It also adds 1e-6 i / N to the i-th LLR to keep them strictly rising, which moves the result by under 1e-7 bits there. PYLLR, by the same author, keeps ties together, as here.

Parameters:

Name Type Description Default
llrs

one score per trial, higher for "more likely a target"; LLRs or not. +inf and -inf rank like any other score.

required
labels

as det_curve.

required

Returns:

Type Description
float

min C_llr in bits, from 0 to 1, never above cllr of the same list. nan, with a warning, for a list without a target or without a non-target trial.

Raises:

Type Description
ValueError

as cllr.

Source code in kudio/core/verification.py
def min_cllr(llrs, labels) -> float:
    """C_llr of the best calibration the ranking allows, in bits: what
    :func:`cllr` would be if the scores were turned into LLRs by the one
    rising mapping that suits this list best.

    >>> kudio.min_cllr(scores, labels)                            # doctest: +SKIP

    The mapping is the pool-adjacent-violators fit of the target share
    against the scores, which gives each trial a posterior probability; less
    the log of the list's own prior odds, T/N, those are LLRs, and their
    C_llr is this [BD06]. The scores only have to rank the trials, so raw
    scores will do. Tied scores share one LLR, as they share one threshold
    in :func:`det_curve`.

    BOSARIS's ``min_cllr.m`` sorts the non-targets of a tie before its
    targets and so lets the fit split ties, which reads lower where targets
    and non-targets share scores: by up to 0.30 bits on the rounded lists of
    kudio's validation script. It also adds ``1e-6 i / N`` to the i-th LLR to
    keep them strictly rising, which moves the result by under 1e-7 bits
    there. PYLLR, by the same author, keeps ties together, as here.

    :param llrs: one score per trial, higher for "more likely a target";
        LLRs or not. +inf and -inf rank like any other score.
    :param labels: as :func:`det_curve`.
    :raises ValueError: as :func:`cllr`.
    :returns: min C_llr in bits, from 0 to 1, never above :func:`cllr` of
        the same list. nan, with a warning, for a list without a target or
        without a non-target trial.
    """
    lists = _llr_lists(llrs, labels, 'min_cllr')
    if lists is None:
        return float('nan')
    targets, nontargets = lists
    scores = np.concatenate([targets, nontargets])
    is_target = np.concatenate([np.ones(targets.size, dtype=bool),
                                np.zeros(nontargets.size, dtype=bool)])
    hits, sizes = _pav_blocks(scores, is_target)
    misses = sizes - hits
    T, N = targets.size, nontargets.size
    # a target in a block with share t/n costs log(1 + (n - t) T / (t N)),
    # a non-target log(1 + t N / ((n - t) T)); a block holds a target only
    # if t > 0 and a non-target only if n - t > 0, so neither divides by 0
    with_targets = hits > 0
    with_misses = misses > 0
    tar = np.sum(hits[with_targets] * np.log1p(
        (misses[with_targets] * T) / (hits[with_targets] * N)))
    non = np.sum(misses[with_misses] * np.log1p(
        (hits[with_misses] * N) / (misses[with_misses] * T)))
    return float((tar / T + non / N) / (2.0 * math.log(2.0)))

act_dcf

act_dcf(
    llrs,
    labels,
    *,
    p_target: float,
    c_miss: float = 1.0,
    c_fa: float = 1.0,
) -> float

Actual normalized detection cost: the cost of the decisions the LLRs themselves make, at the threshold Bayes' rule sets for the application. 0 is perfect; 1 is no better than deciding without the scores.

>>> kudio.act_dcf(llrs, labels, p_target=0.01)

A trial is accepted when its LLR is at or above log(beta), beta = c_fa (1 - p_target) / (c_miss p_target), the threshold of NIST's SRE 2016 and 2019 CTS plans (Sec. 3.1), and the cost is min_dcf's, normalized the same way, at that one threshold instead of the best. A trial exactly at the threshold is accepted, as PYLLR's fast_Bayes_error_rate accepts it. The gap to min_dcf is what calibration costs at that operating point.

Parameters:

Name Type Description Default
llrs

natural-log likelihood ratios, as cllr.

required
p_target float

as min_dcf; 0.01 and 0.005 in SRE 2016.

required
c_miss float

as min_dcf.

1.0
c_fa float

as min_dcf.

1.0

Returns:

Type Description
float

the normalized actual cost, from 0 up; above 1 when the LLRs decide worse than the prior alone. nan, with a warning, for a list without a target or without a non-target trial.

Raises:

Type Description
ValueError

as min_dcf, and for an llr that is NaN.

Source code in kudio/core/verification.py
def act_dcf(llrs, labels, *, p_target: float, c_miss: float = 1.0,
            c_fa: float = 1.0) -> float:
    """Actual normalized detection cost: the cost of the decisions the LLRs
    themselves make, at the threshold Bayes' rule sets for the application.
    0 is perfect; 1 is no better than deciding without the scores.

    >>> kudio.act_dcf(llrs, labels, p_target=0.01)                # doctest: +SKIP

    A trial is accepted when its LLR is at or above
    ``log(beta)``, ``beta = c_fa (1 - p_target) / (c_miss p_target)``, the
    threshold of NIST's SRE 2016 and 2019 CTS plans (Sec. 3.1), and the cost
    is :func:`min_dcf`'s, normalized the same way, at that one threshold
    instead of the best. A trial exactly at the threshold is accepted, as
    PYLLR's ``fast_Bayes_error_rate`` accepts it. The gap to :func:`min_dcf`
    is what calibration costs at that operating point.

    :param llrs: natural-log likelihood ratios, as :func:`cllr`.
    :param p_target: as :func:`min_dcf`; 0.01 and 0.005 in SRE 2016.
    :param c_miss: as :func:`min_dcf`.
    :param c_fa: as :func:`min_dcf`.
    :raises ValueError: as :func:`min_dcf`, and for an llr that is NaN.
    :returns: the normalized actual cost, from 0 up; above 1 when the LLRs
        decide worse than the prior alone. nan, with a warning, for a list
        without a target or without a non-target trial.
    """
    weight_miss, weight_fa = _weights(p_target, c_miss, c_fa, 'act_dcf')
    lists = _llr_lists(llrs, labels, 'act_dcf')
    if lists is None:
        return float('nan')
    targets, nontargets = lists
    threshold = math.log(weight_fa / weight_miss)
    p_miss = np.count_nonzero(targets < threshold) / targets.size
    p_fa = np.count_nonzero(nontargets >= threshold) / nontargets.size
    return (weight_miss * p_miss + weight_fa * p_fa) / min(weight_miss, weight_fa)

c_primary

c_primary(llrs, labels, *, minimum: bool = False) -> float

NIST's primary cost of SRE 2016 and the SRE 2019 CTS challenge: the mean of act_dcf at p_target 0.01 and 0.005, with unit costs.

>>> kudio.c_primary(llrs, labels)
>>> kudio.c_primary(scores, labels, minimum=True)

Parameters:

Name Type Description Default
minimum bool

average min_dcf instead, the "minCprimary" that NIST's scoring script prints beside it.

False

Returns:

Type Description
float

the mean of the two normalized costs. nan, with a warning, as act_dcf gives it.

The evaluations average the cost over partitions of the trial list, language and gender among them, with each partition's trials weighted equally (SRE 2016 plan, Sec. 3.1); this pools the list as it comes. To match an official number, compute the cost per partition and average them as the plan says.

Source code in kudio/core/verification.py
def c_primary(llrs, labels, *, minimum: bool = False) -> float:
    """NIST's primary cost of SRE 2016 and the SRE 2019 CTS challenge: the
    mean of :func:`act_dcf` at p_target 0.01 and 0.005, with unit costs.

    >>> kudio.c_primary(llrs, labels)                             # doctest: +SKIP
    >>> kudio.c_primary(scores, labels, minimum=True)             # doctest: +SKIP

    :param minimum: average :func:`min_dcf` instead, the "minCprimary" that
        NIST's scoring script prints beside it.
    :returns: the mean of the two normalized costs. nan, with a warning, as
        :func:`act_dcf` gives it.

    The evaluations average the cost over partitions of the trial list,
    language and gender among them, with each partition's trials weighted
    equally (SRE 2016 plan, Sec. 3.1); this pools the list as it comes. To
    match an official number, compute the cost per partition and average
    them as the plan says.
    """
    if not isinstance(minimum, (bool, np.bool_)):
        raise ValueError(f"c_primary(): minimum has to be True or False, got "
                         f"{minimum!r}")
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        measure = min_dcf if minimum else act_dcf
        try:
            costs = [measure(llrs, labels, p_target=p) for p in C_PRIMARY_PRIORS]
        except ValueError as e:
            text = str(e)
            raise ValueError("c_primary(): " + text.split("(): ", 1)[-1]) from None
    if caught:
        # the two costs fail for one reason; say it once, by this name
        text = str(caught[0].message)
        warn("c_primary(): " + text.split("(): ", 1)[1] if "(): " in text else text)
        return float('nan')
    return (costs[0] + costs[1]) / 2.0

kudio.core.diarization

Who spoke when: the diarization error rate, the Jaccard error rate and the detection error rate of speech activity, against a reference of speaker turns.

DER is the error measure of NIST's Rich Transcription evaluations [RT09], scored by its md-eval.pl and by dscore, the scorer of DIHARD II and III [DIH19]. The time scored is cut into regions where the set of speakers stays the same, and each reference speaker is mapped to at most one system speaker, the mapping that makes the time they speak together the longest. A region of d seconds with N_ref reference speakers, N_sys system speakers and N_correct reference speakers whose mapped system speaker is talking there counts:

missed       d max(0, N_ref - N_sys)
false alarm  d max(0, N_sys - N_ref)
confusion    d (min(N_ref, N_sys) - N_correct)
total        d N_ref

and DER is the three errors over the total, so overlapping speech counts once for every speaker in it. It can pass 1: a system that talks where nobody does adds false alarm without bound.

JER, introduced for DIHARD II [DIH19], scores each reference speaker on its own: the speaker's Jaccard error, 1 minus the time it shares with its system speaker over the union of the two, averaged over the reference speakers, so that every speaker weighs the same whatever they said. A reference speaker left without a system speaker scores 1. The mapping is the one dscore takes: the Hungarian assignment with the smallest summed Jaccard error. pyannote.metrics' JaccardErrorRate maps by the longest shared time instead, as for DER, and the two can part where a speaker's best overlap and best Jaccard index point at different system speakers.

Detection error rate scores speech activity alone: the missed and false-alarm speech over the reference speech, every speaker merged into one, as pyannote.metrics' DetectionErrorRate takes it. The labels of a VAD are enough.

[RT09]  NIST, "The 2009 (RT-09) Rich Transcription Meeting Recognition
        Evaluation Plan", 2009, Sec. 6.1.
[DIH19] N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy
        and M. Liberman, "Second DIHARD Challenge Evaluation Plan",
        version 1.1, 2019; the scorer is dscore,
        github.com/nryant/dscore.

What to watch for:

  • The collar is per side, in seconds, as in md-eval.pl and dscore: collar=0.25 leaves 0.25 s either side of every reference turn boundary unscored, 0.5 s in all. pyannote.metrics' collar is the total, so its 0.5 is this 0.25. DIHARD scores with no collar and with overlapping speech; the older NIST evaluations and many papers use 0.25 s and drop the overlap, and the two settings give very different numbers, so say which.
  • Without uem the whole of time is scored, which counts only where someone, in the reference or the system, is talking. Give the scoring regions where only part of a recording was annotated.
  • dscore computes JER on 10 ms frames, so its JER and the exact one here agree to the last digit only where every boundary is on that grid.
  • The speakers are mapped on all the time evaluated, before the collars and the overlap are taken out, as md-eval.pl maps them. pyannote.metrics maps on the time it scores, and with a collar or without the overlap the two can part: reference speakers A and B talking together for 10 s, then C alone for 2 s, against one system speaker throughout, score a DER of 1 here and 0 there once the overlap is left out.
  • Without uem, all of time is scored, as dscore and pyannote.metrics score it. md-eval.pl run without a UEM scores only from the reference's first turn to its last, so system speech before or after those is not counted.
  • Turns of one speaker that overlap or touch are one stretch of speech here. md-eval.pl refuses an RTTM with overlapping turns of one speaker, and pyannote.metrics counts the time they share twice. Collars are cut around every turn's boundaries as given.

Classes:

Name Description
Turn

A stretch of one speaker's speech, in seconds.

DiarizationErrors

The seconds behind a DER, summed over the recordings of a corpus.

Functions:

Name Description
load_rttm

The SPEAKER lines of an RTTM file, as turns by recording.

diarization_errors

The seconds of missed speech, false alarm and speaker confusion behind

der

Diarization error rate of hypothesis against reference, as a

jer

Jaccard error rate of hypothesis against reference, as a fraction:

detection_error_rate

Missed and false-alarm speech over the reference speech, every speaker

Turn

Bases: NamedTuple

A stretch of one speaker's speech, in seconds.

DiarizationErrors

Bases: NamedTuple

The seconds behind a DER, summed over the recordings of a corpus.

Attributes:

Name Type Description
missed float

reference speech no system speaker covers, counted per speaker

false_alarm float

system speech beyond the reference speakers present, per speaker

confusion float

reference speech given to the wrong system speaker

total float

reference speech scored, counted once for each speaker in it

der float

(missed + false_alarm + confusion) / total; nan for no total.

missed instance-attribute

missed: float

reference speech no system speaker covers, counted per speaker

false_alarm instance-attribute

false_alarm: float

system speech beyond the reference speakers present, per speaker

confusion instance-attribute

confusion: float

reference speech given to the wrong system speaker

total instance-attribute

total: float

reference speech scored, counted once for each speaker in it

der property

der: float

(missed + false_alarm + confusion) / total; nan for no total.

load_rttm

load_rttm(path) -> Dict[str, List[Turn]]

The SPEAKER lines of an RTTM file, as turns by recording.

>>> reference = kudio.load_rttm("ref.rttm")

Each line SPEAKER <file> <channel> <onset> <duration> <NA> <NA> <speaker> ... becomes Turn(onset, onset + duration, speaker) under <file>; lines of other types are skipped, and so are blank lines and lines starting with ;;.

Raises:

Type Description
ValueError

for a SPEAKER line with fewer than 8 fields, or an onset or duration that is not a number, naming the line.

Source code in kudio/core/diarization.py
def load_rttm(path) -> Dict[str, List[Turn]]:
    """The SPEAKER lines of an RTTM file, as turns by recording.

    >>> reference = kudio.load_rttm("ref.rttm")              # doctest: +SKIP

    Each line ``SPEAKER <file> <channel> <onset> <duration> <NA> <NA>
    <speaker> ...`` becomes ``Turn(onset, onset + duration, speaker)`` under
    ``<file>``; lines of other types are skipped, and so are blank lines and
    lines starting with ``;;``.

    :raises ValueError: for a SPEAKER line with fewer than 8 fields, or an
        onset or duration that is not a number, naming the line.
    """
    out: Dict[str, List[Turn]] = {}
    for number, line in enumerate(Path(path).read_text(encoding='utf-8').splitlines(), 1):
        fields = line.split()
        if not fields or fields[0].startswith(';;') or fields[0] != 'SPEAKER':
            continue
        if len(fields) < 8:
            raise ValueError(f"load_rttm(): line {number} of {path} has "
                             f"{len(fields)} fields, and a SPEAKER line has 10")
        try:
            onset, duration = float(fields[3]), float(fields[4])
        except ValueError:
            raise ValueError(f"load_rttm(): line {number} of {path}: the onset "
                             f"and duration have to be numbers, got "
                             f"{fields[3]!r} and {fields[4]!r}") from None
        out.setdefault(fields[1], []).append(Turn(onset, onset + duration, fields[7]))
    return out

diarization_errors

diarization_errors(
    reference,
    hypothesis,
    *,
    collar: float = 0.0,
    skip_overlap: bool = False,
    uem=None,
) -> DiarizationErrors

The seconds of missed speech, false alarm and speaker confusion behind der, and the reference speech they are counted against.

>>> kudio.diarization_errors(reference, hypothesis)
DiarizationErrors(missed=1.2, false_alarm=0.8, confusion=3.1, total=60.0)

Arguments as der. Summed over the recordings of a corpus, each with its own speaker mapping.

Source code in kudio/core/diarization.py
def diarization_errors(reference, hypothesis, *, collar: float = 0.0,
                       skip_overlap: bool = False, uem=None) -> DiarizationErrors:
    """The seconds of missed speech, false alarm and speaker confusion behind
    :func:`der`, and the reference speech they are counted against.

    >>> kudio.diarization_errors(reference, hypothesis)      # doctest: +SKIP
    DiarizationErrors(missed=1.2, false_alarm=0.8, confusion=3.1, total=60.0)

    Arguments as :func:`der`. Summed over the recordings of a corpus, each
    with its own speaker mapping.
    """
    collar, skip_overlap = _check_options(collar, skip_overlap, 'diarization_errors')
    sums = [0.0, 0.0, 0.0, 0.0]
    for _, ref, hyp, regions in _recordings(reference, hypothesis, uem,
                                            'diarization_errors'):
        for k, value in enumerate(_errors(ref, hyp, regions, collar, skip_overlap)):
            sums[k] += value
    return DiarizationErrors(*sums)

der

der(
    reference,
    hypothesis,
    *,
    collar: float = 0.0,
    skip_overlap: bool = False,
    uem=None,
) -> float

Diarization error rate of hypothesis against reference, as a fraction: missed speech, false alarm and speaker confusion over the reference speech. 0 is perfect; there is no upper bound.

>>> kudio.der(reference, hypothesis)                     # DIHARD
>>> kudio.der(reference, hypothesis, collar=0.25, skip_overlap=True)

Parameters:

Name Type Description Default
reference

what was said when: a sequence of turns (start, end, speaker) in seconds, Turn or kudio Label (its text the speaker), or a dict of such sequences by recording, as load_rttm returns.

required
hypothesis

the system's turns, in the same form; its speaker labels need not match the reference's.

required
collar float

seconds left unscored on each side of every reference turn boundary, as md-eval.pl's -c and dscore's --collar take it. pyannote.metrics' collar is the total, twice this.

0.0
skip_overlap bool

leave out the time where two or more reference speakers talk at once, as md-eval.pl's -1.

False
uem

the scoring regions, [(start, end), ...] for one recording or a dict of them by recording; all of time if None.

None

Returns:

Type Description
float

the DER, pooled over the recordings of a corpus. nan, with a warning, when the reference has no speech in the time scored.

Raises:

Type Description
ValueError

for a turn that is not (start, end, speaker) of finite numbers with end not before start, a collar below 0, dicts whose recordings do not pair up, and a dict paired with a list.

Source code in kudio/core/diarization.py
def der(reference, hypothesis, *, collar: float = 0.0, skip_overlap: bool = False,
        uem=None) -> float:
    """Diarization error rate of *hypothesis* against *reference*, as a
    fraction: missed speech, false alarm and speaker confusion over the
    reference speech. 0 is perfect; there is no upper bound.

    >>> kudio.der(reference, hypothesis)                     # DIHARD
    >>> kudio.der(reference, hypothesis, collar=0.25, skip_overlap=True)

    :param reference: what was said when: a sequence of turns
        ``(start, end, speaker)`` in seconds, :class:`Turn` or kudio
        :class:`~kudio.core.dataset.Label` (its text the speaker), or a dict
        of such sequences by recording, as :func:`load_rttm` returns.
    :param hypothesis: the system's turns, in the same form; its speaker
        labels need not match the reference's.
    :param collar: seconds left unscored on each side of every reference turn
        boundary, as md-eval.pl's ``-c`` and dscore's ``--collar`` take it.
        pyannote.metrics' collar is the total, twice this.
    :param skip_overlap: leave out the time where two or more reference
        speakers talk at once, as md-eval.pl's ``-1``.
    :param uem: the scoring regions, ``[(start, end), ...]`` for one
        recording or a dict of them by recording; all of time if None.
    :raises ValueError: for a turn that is not ``(start, end, speaker)`` of
        finite numbers with end not before start, a collar below 0, dicts
        whose recordings do not pair up, and a dict paired with a list.
    :returns: the DER, pooled over the recordings of a corpus. nan, with a
        warning, when the reference has no speech in the time scored.
    """
    collar, skip_overlap = _check_options(collar, skip_overlap, 'der')
    sums = [0.0, 0.0, 0.0, 0.0]
    for _, ref, hyp, regions in _recordings(reference, hypothesis, uem, 'der'):
        for k, value in enumerate(_errors(ref, hyp, regions, collar, skip_overlap)):
            sums[k] += value
    errors = DiarizationErrors(*sums)
    if errors.total <= 0.0:
        warn("der(): the reference has no speech in the time scored, so there "
             "is nothing to divide the errors by; scored nan")
        return float('nan')
    return errors.der

jer

jer(reference, hypothesis, *, uem=None) -> float

Jaccard error rate of hypothesis against reference, as a fraction: 1 minus each reference speaker's Jaccard index with its system speaker, averaged over the reference speakers. 0 is perfect and 1 the worst.

>>> kudio.jer(reference, hypothesis)

Each reference speaker is paired with at most one system speaker, by the Hungarian assignment with the smallest summed Jaccard error, as dscore pairs them, and one left over scores 1. Overlapping speech counts for every speaker in it, and there is no collar, as in DIHARD.

Parameters:

Name Type Description Default
reference

as der.

required
hypothesis

as der.

required
uem

as der.

None

Returns:

Type Description
float

the JER; over a corpus, the mean over the reference speakers of every recording, as dscore pools it. nan, with a warning, when there is no reference speaker with speech in the time scored. dscore gives such a recording 1 when the system talks there and 0 when it does not, and leaves it out of the pooled figure, as this does.

Raises:

Type Description
ValueError

as der.

Source code in kudio/core/diarization.py
def jer(reference, hypothesis, *, uem=None) -> float:
    """Jaccard error rate of *hypothesis* against *reference*, as a fraction:
    1 minus each reference speaker's Jaccard index with its system speaker,
    averaged over the reference speakers. 0 is perfect and 1 the worst.

    >>> kudio.jer(reference, hypothesis)                     # doctest: +SKIP

    Each reference speaker is paired with at most one system speaker, by the
    Hungarian assignment with the smallest summed Jaccard error, as dscore
    pairs them, and one left over scores 1. Overlapping speech counts for
    every speaker in it, and there is no collar, as in DIHARD.

    :param reference: as :func:`der`.
    :param hypothesis: as :func:`der`.
    :param uem: as :func:`der`.
    :raises ValueError: as :func:`der`.
    :returns: the JER; over a corpus, the mean over the reference speakers of
        every recording, as dscore pools it. nan, with a warning, when there
        is no reference speaker with speech in the time scored. dscore gives
        such a recording 1 when the system talks there and 0 when it does
        not, and leaves it out of the pooled figure, as this does.
    """
    errors = []
    for _, ref_turns, hyp_turns, regions in _recordings(reference, hypothesis, uem, 'jer'):
        ref, hyp = _speakers(ref_turns), _speakers(hyp_turns)
        if regions is not None:
            ref = {s: _intersect(spans, regions) for s, spans in ref.items()}
            hyp = {s: _intersect(spans, regions) for s, spans in hyp.items()}
        ref = {s: spans for s, spans in ref.items() if spans}
        hyp = {s: spans for s, spans in hyp.items() if spans}
        if not ref:
            continue
        ref_names, hyp_names = list(ref), list(hyp)
        ref_len = np.array([_length(ref[s]) for s in ref_names])
        hyp_len = np.array([_length(hyp[s]) for s in hyp_names])
        shared = np.array([[_length(_intersect(ref[a], hyp[b])) for b in hyp_names]
                           for a in ref_names]).reshape(len(ref_names), len(hyp_names))
        union = ref_len[:, None] + hyp_len[None, :] - shared
        table = 1.0 - shared / union
        mapping = _assign(table, maximize=False)
        for i in range(len(ref_names)):
            errors.append(float(table[i, mapping[i]]) if i in mapping else 1.0)
    if not errors:
        warn("jer(): the reference has no speaker with speech in the time scored, "
             "so there is no speaker to average over; scored nan")
        return float('nan')
    return float(np.mean(errors))

detection_error_rate

detection_error_rate(
    reference, hypothesis, *, collar: float = 0.0, uem=None
) -> float

Missed and false-alarm speech over the reference speech, every speaker merged into one: how well speech was found, whoever spoke.

>>> labels = kudio.vad(y, sr)
>>> kudio.detection_error_rate(truth, labels)

Parameters:

Name Type Description Default
reference

as der; the speaker labels are not read.

required
hypothesis

as der.

required
collar float

as der, around the boundaries of the reference's turns.

0.0
uem

as der.

None

Returns:

Type Description
float

the rate, from 0 up. nan, with a warning, when the reference has no speech in the time scored.

Raises:

Type Description
ValueError

as der.

Source code in kudio/core/diarization.py
def detection_error_rate(reference, hypothesis, *, collar: float = 0.0,
                         uem=None) -> float:
    """Missed and false-alarm speech over the reference speech, every speaker
    merged into one: how well speech was found, whoever spoke.

    >>> labels = kudio.vad(y, sr)                            # doctest: +SKIP
    >>> kudio.detection_error_rate(truth, labels)            # doctest: +SKIP

    :param reference: as :func:`der`; the speaker labels are not read.
    :param hypothesis: as :func:`der`.
    :param collar: as :func:`der`, around the boundaries of the reference's
        turns.
    :param uem: as :func:`der`.
    :raises ValueError: as :func:`der`.
    :returns: the rate, from 0 up. nan, with a warning, when the reference
        has no speech in the time scored.
    """
    collar, _ = _check_options(collar, False, 'detection_error_rate')
    sums = [0.0, 0.0, 0.0, 0.0]
    for _, ref, hyp, regions in _recordings(reference, hypothesis, uem,
                                            'detection_error_rate'):
        ref = [Turn(t.start, t.end, 'speech') for t in ref]
        hyp = [Turn(t.start, t.end, 'speech') for t in hyp]
        for k, value in enumerate(_errors(ref, hyp, regions, collar, False)):
            sums[k] += value
    missed, false_alarm, _, total = sums
    if total <= 0.0:
        warn("detection_error_rate(): the reference has no speech in the time "
             "scored, so there is nothing to divide the errors by; scored nan")
        return float('nan')
    return (missed + false_alarm) / total

kudio.core.f0eval

Pitch-tracking errors: how far an estimated F0 track is from a reference track on the same frames.

These are the numbers pitch trackers are compared by on corpora recorded with a laryngograph, such as Keele and PTDB-TUG, and the F0 RMSE that speech synthesis papers report. Each function takes two tracks, one F0 value per frame in Hz with 0 or nan where the frame is unvoiced, or two kudio.PitchTrack objects, and compares frame i of one with frame i of the other. Every frame falls in one of four cases ([R76], p. 407): unvoiced in both; voiced in the reference only, a V/U error; voiced in the estimate only, a U/V error; or voiced in both, where the estimate is either close or a gross error. With N frames, N_VV of them voiced in both and N_GE gross errors among those:

function                    what it takes                       over
gross_pitch_error      GPE  N_GE: |est - ref| > 0.2 ref         N_VV
voicing_decision_error VDE  N_VU + N_UV                         N
f0_frame_error         FFE  N_VU + N_UV + N_GE                  N
fine_pitch_error       FPE  SD of (est - ref) / ref             N_VV - N_GE
f0_rmse                     RMS of 1200 log2(est / ref), cents  N_VV

GPE, VDE and FFE are those of Chu and Alwan [CA09], eqs. 1 to 4, and FFE = VDE + (N_VV / N) GPE, their eq. 5. FPE is Drugman and Alwan's [DA11]: the standard deviation of the relative error over the frames voiced in both without a gross error. The first three come back as fractions, and FPE as a fraction of the reference F0; times 100 they are the percentages the papers print:

```pycon
>>> ref = [0, 100, 102, 104, 0, 0, 120, 121]      # Hz, 0 where unvoiced
>>> est = [0, 101, 204, 0, 0, 98, 119, 0]
>>> kudio.gross_pitch_error(ref, est)     # frame 2 an octave up, of 3
0.3333333333333333
>>> kudio.voicing_decision_error(ref, est)   # frames 3, 5 and 7, of 8
0.375
>>> kudio.f0_frame_error(ref, est)           # all four, of 8
0.5
[`f0_errors`][kudio.core.f0eval.f0_errors] returns all five at once, with the counts behind them.

Sources, and what each one counts:

```python
[R76]   L. R. Rabiner, M. J. Cheng, A. E. Rosenberg and C. A. McGonegal,
        "A comparative performance study of several pitch detection
        algorithms", IEEE Trans. Acoust., Speech, Signal Process. 24(5),
        pp. 399-418, 1976, doi:10.1109/TASSP.1976.1162846. A gross error
        is a pitch-period error of 1 ms or more (10 samples at their
        10 kHz), counted per utterance: at 100 Hz an estimate 9.1% low or
        11.1% high, at 200 Hz one 16.7% low or 25% high. The fine errors,
        the rest, are measured by their mean, which came out near 0 for
        every detector and is not tabulated, and by their standard
        deviation, in samples; V/U errors as a share of the reference's
        voiced frames, and U/V errors in the same form.
[NIZ03] T. Nakatani, T. Irino and P. Zolfaghari, "Dominance spectrum
        based V/UV classification and F0 estimation", Proc. Eurospeech
        2003, pp. 2313-2316, doi:10.21437/Eurospeech.2003-210. A gross
        error is an estimate more than 5% off, and the fine error is the
        RMS of the relative error over the other frames.
[CA09]  W. Chu and A. Alwan, "Reducing F0 frame error of F0 tracking
        algorithms under noisy conditions with an unvoiced/voiced
        classification frontend", Proc. ICASSP 2009, pp. 3969-3972,
        doi:10.1109/ICASSP.2009.4960497. GPE over N_VV, its threshold
        usually 20%, and VDE and FFE over all N frames.
[DA11]  T. Drugman and A. Alwan, "Joint robust voicing detection and
        pitch estimation based on residual harmonics", Proc. Interspeech
        2011, pp. 1973-1976, doi:10.21437/Interspeech.2011-519. GPE, VDE
        and FFE as in [CA09], and FPE as the standard deviation, in
        percent, of the relative error of the frames under 20%.
[CA12]  W. Chu and A. Alwan, "SAFE: a statistical approach to F0
        estimation under clean and noisy conditions", IEEE Trans. Audio,
        Speech, Lang. Process. 20(3), pp. 933-944, 2012,
        doi:10.1109/TASL.2011.2168518. GPE as in [CA09], with every
        tracker's voicing replaced by the reference's: a frame the
        reference calls voiced and the tracker unvoiced gets an F0
        interpolated from the nearest frames both call voiced (their
        eq. 20), and a frame the tracker calls voiced against the
        reference is set to 0. The fine errors are a mean (MFPE, eq. 23)
        and a root mean square about zero (SDFPE, eq. 24, though they
        call it a standard deviation), in Hz.
[CH08]  A. Camacho and J. G. Harris, "A sawtooth waveform inspired pitch
        estimator for speech and music", J. Acoust. Soc. Am. 124(3),
        pp. 1638-1652, 2008, doi:10.1121/1.2951592, sec. III.C. A gross
        error rate with the 20% rule, counted only at the times the
        reference and every compared algorithm that gave an estimate
        every millisecond called the sound pitched, and on the corpora
        whose reference moves, after shifting each estimate by up to
        100 ms to its best alignment with it.
[P11]   G. Pirker, M. Wohlmayr, S. Petrik and F. Pernkopf, "A pitch
        tracking corpus with evaluation on multipitch tracking scenario",
        Proc. Interspeech 2011, pp. 1509-1512,
        doi:10.21437/Interspeech.2011-317. PTDB-TUG. Its benchmark, two
        talkers at once, counts gross errors (over 20%) as a share of all
        frames, and cuts each utterance 70 ms before its first and after
        its last reference pitch value.
[PMA95] F. Plante, G. F. Meyer and W. A. Ainsworth, "A pitch extraction
        reference database", Proc. Eurospeech 1995, pp. 837-840,
        doi:10.21437/Eurospeech.1995-191. Keele: a reference value every
        10 ms from the laryngograph, 0 where unvoiced and negative where
        the frame is uncertain, and uncertain frames are to be left out
        of comparisons.
[K18]   J. W. Kim, J. Salamon, P. Li and J. P. Bello, "CREPE: a
        convolutional representation for pitch estimation", Proc. ICASSP
        2018, pp. 161-165, doi:10.1109/ICASSP.2018.8461329. Raw pitch
        accuracy within 50 cents, and within 25 and 10, from mir_eval.
[R14]   C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto,
        D. Liang and D. P. W. Ellis, "mir_eval: a transparent
        implementation of common MIR metrics", Proc. ISMIR 2014,
        pp. 367-372, doi:10.5281/zenodo.1416528. The MIREX melody
        measures: raw pitch accuracy over the reference's voiced frames,
        within half a semitone; voicing recall and false alarm; and
        overall accuracy over all frames.

Where the papers leave a choice open, I made it this way. The 20% is measured against the reference and is strict, as [CA09]'s eq. 3 writes it, so an estimate exactly 20% off is not a gross error: 120 Hz against 100 is not. Exactly means in the binary values the arrays hold. Most decimals have no exact binary form, and the comparison is rounded like any other, so two decimals 20% apart can land either side: 120.12 Hz against 100.1 is a gross error, 120.24 against 100.2 is not. FPE divides by the number of fine frames, as [R76]'s eq. 3 does for its standard deviation; [DA11] does not say. The RMSE counts every frame voiced in both, gross errors included. On tracks built so that the 20% rule and a cent tolerance draw the same line, mir_eval 0.8.2 gives the same numbers: 1 - overall_accuracy is FFE, 1 - raw_pitch_accuracy over the frames voiced in both is GPE, and with an unlimited cent tolerance (cent_tolerance=np.inf) 1 - overall_accuracy is VDE. tools/validation/f0eval.py repeats that check on the tests' pair, 7 pairs at the edges of what can be scored and 400 random ones, and checks every measure against a frame-by-frame implementation of its own as well. With numpy 2.3.5 the rates agree to 1.1e-16 and the RMSE to 4.5e-13 cents.

What to watch for:

  • The denominators are where papers part. GPE here is over N_VV. [CA12] replaced every tracker's voicing with the reference's, interpolating an F0 where a tracker gave none, [CH08] counted only where the reference and every compared algorithm called the sound pitched, and [P11] counts gross errors over all N frames, which is (N_VV / N) times the GPE here. The same two tracks give a different number under each; say which one a table holds.
  • GPE on its own rewards a tracker for giving up. Calling a hard frame unvoiced takes it out of N_VV: turning one gross error into a V/U error lowers GPE, adds 1/N to VDE and leaves FFE where it was, which is why [CA09] proposed FFE. Report GPE with VDE, or FFE.
  • VDE and FFE count every frame, so frames that both tracks call unvoiced dilute them: 300 frames with 30 voicing errors read 0.10, and with 100 frames of silence added at each end, 0.06. [P11] cut its utterances to 70 ms around the first and last pitch value. Trim every system's tracks the same way.
  • Pooling and averaging are different numbers. Concatenate the tracks of a test set and score them once to pool their frames; the mean of per-file scores weighs a short file like a long one. A file with 5 gross errors in 10 frames voiced in both and one with 10 in 1000 average to a GPE of 0.255 and pool to 0.0149.
  • 20% is 315.6 cents above the reference and 386.3 below it. Raw pitch accuracy, as MIREX and mir_eval score it [R14], takes a symmetric tolerance, 50 cents by default (2.9% up, 2.8% down), counts over the reference's voiced frames and reads an estimate's pitch whether or not it is called voiced, so 1 - RPA is not GPE.
  • Negative values are refused, because they mean two things. In Keele's reference [PMA95] they mark uncertain frames, which are to be dropped from both tracks (keep = ref >= 0). In MIREX files a negative estimate is an unvoiced frame with a pitch guess, which np.maximum(est, 0) reads as unvoiced. A masked array with masked frames is refused for the same reason: np.ma.filled(est, 0) reads them as unvoiced, and keep = ~np.ma.getmaskarray(est) drops them from both tracks.
  • Nothing is shifted. A lag of one 10 ms frame on a voice gliding 2 octaves per second puts every frame 24 cents off, and [CH08] shifted the estimates by up to 100 ms to their best alignment before counting. Synthesized speech is seldom on the reference's frames at all and has to be lined up with it first, by DTW for instance; the alignment is then part of the number.
  • f0_rmse counts gross errors at full size: one octave error among 100 frames voiced in both, the rest exact, is 120 cents. FPE leaves the gross errors out, and being a standard deviation it leaves a constant offset out too: an estimate 1% sharp on every frame has an FPE of 0 and an RMSE of 17.2 cents. An RMSE in Hz is another number, which weighs a high voice more (a semitone is 5.9 Hz at 100 Hz and 11.9 Hz at 200 Hz); an RMSE of ln F0 is the RMSE in cents divided by 1731.2.
  • FPE is relative, and a standard deviation that divides by the number of fine frames; MATLAB's std divides by one fewer, which reads 0.5% higher at 100 frames. The fine errors of [CA12] are in Hz, and their SDFPE, like [NIZ03]'s RMS, takes a constant offset in; those of [R76] are in samples of pitch period; none of them is this number.

Classes:

Name Description
F0Errors

Every measure of kudio.core.f0eval for one pair of tracks, and

Functions:

Name Description
gross_pitch_error

Gross pitch error: the share of the frames voiced in both tracks where

voicing_decision_error

Voicing decision error: the share of all frames where est and ref

f0_frame_error

F0 frame error: the share of all frames with any error, a voicing

fine_pitch_error

Fine pitch error: how much est scatters around ref on the frames

f0_rmse

Root mean square F0 error in cents over the frames voiced in both.

f0_errors

GPE, VDE, FFE, FPE and the RMSE in cents of est against ref in one

F0Errors

Bases: NamedTuple

Every measure of kudio.core.f0eval for one pair of tracks, and the frame counts behind them.

The counts say how much each figure rests on: a GPE of 0.5 with voiced_both=2 is one frame in two. To pool a test set, concatenate its tracks and score them once rather than averaging these.

Attributes:

Name Type Description
gpe float

gross pitch error, gross / voiced_both

vde float

voicing decision error, (voiced_to_unvoiced + unvoiced_to_voiced) / frames

ffe float

F0 frame error, (gross + voiced_to_unvoiced + unvoiced_to_voiced) / frames

fpe float

fine pitch error, the standard deviation of (est - ref) / ref over

rmse_cents float

RMS error over the frames voiced in both, in cents

frames int

frames compared, N

voiced_both int

frames voiced in both tracks, N_VV

gross int

frames voiced in both whose estimate is more than the threshold off, N_GE

voiced_to_unvoiced int

frames voiced in the reference and not in the estimate, N_VU

unvoiced_to_voiced int

frames voiced in the estimate and not in the reference, N_UV

gpe instance-attribute

gpe: float

gross pitch error, gross / voiced_both

vde instance-attribute

vde: float

voicing decision error, (voiced_to_unvoiced + unvoiced_to_voiced) / frames

ffe instance-attribute

ffe: float

F0 frame error, (gross + voiced_to_unvoiced + unvoiced_to_voiced) / frames

fpe instance-attribute

fpe: float

fine pitch error, the standard deviation of (est - ref) / ref over the voiced_both - gross frames without a gross error

rmse_cents instance-attribute

rmse_cents: float

RMS error over the frames voiced in both, in cents

frames instance-attribute

frames: int

frames compared, N

voiced_both instance-attribute

voiced_both: int

frames voiced in both tracks, N_VV

gross instance-attribute

gross: int

frames voiced in both whose estimate is more than the threshold off, N_GE

voiced_to_unvoiced instance-attribute

voiced_to_unvoiced: int

frames voiced in the reference and not in the estimate, N_VU

unvoiced_to_voiced instance-attribute

unvoiced_to_voiced: int

frames voiced in the estimate and not in the reference, N_UV

gross_pitch_error

gross_pitch_error(
    ref, est, *, threshold: float = 0.2
) -> float

Gross pitch error: the share of the frames voiced in both tracks where est is more than threshold off ref.

N_GE / N_VV, [CA09] eq. 2: a frame voiced in both counts when |est - ref| > threshold * ref, which is the 20% rule with the default, measured against the reference: 125 Hz against 100 counts, 100 against 125 does not. Frames voiced in only one track do not enter it, so a tracker lowers its GPE by calling the frames it gets wrong unvoiced; voicing_decision_error counts those, and f0_frame_error counts both.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required
threshold float

how far off, as a fraction of the reference F0, an estimate has to be to count: 0.2, the 20% the papers use.

0.2

Returns:

Type Description
float

the fraction of the N_VV frames with a gross error, 0 to 1. nan, with a warning, when no frame is voiced in both.

Raises:

Type Description
ValueError

for tracks of different lengths, two PitchTracks whose frame times differ, a PitchTrack with a voiced frame that has no F0 or without one finite time per frame, negative or infinite values, masked frames, an input that is not one track of numbers, or a threshold outside 0 to 1.

Source code in kudio/core/f0eval.py
def gross_pitch_error(ref, est, *, threshold: float = 0.2) -> float:
    """Gross pitch error: the share of the frames voiced in both tracks where
    *est* is more than *threshold* off *ref*.

    N_GE / N_VV, [CA09] eq. 2: a frame voiced in both counts when
    ``|est - ref| > threshold * ref``, which is the 20% rule with the
    default, measured against the reference: 125 Hz against 100 counts, 100
    against 125 does not. Frames voiced in only one track do not enter it,
    so a tracker lowers its GPE by calling the frames it gets wrong
    unvoiced; :func:`voicing_decision_error` counts those, and
    :func:`f0_frame_error` counts both.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :param threshold: how far off, as a fraction of the reference F0, an
        estimate has to be to count: 0.2, the 20% the papers use.
    :raises ValueError: for tracks of different lengths, two PitchTracks
        whose frame times differ, a PitchTrack with a voiced frame that has
        no F0 or without one finite time per frame, negative or infinite
        values, masked frames, an input that is not one track of numbers, or
        a *threshold* outside 0 to 1.
    :returns: the fraction of the N_VV frames with a gross error, 0 to 1.
        nan, with a warning, when no frame is voiced in both.
    """
    fn = 'gross_pitch_error'
    threshold = _checked_threshold(threshold, fn)
    r, e = _pair(ref, est, fn)
    if not r.size:
        return _scored_nan(fn, "got two empty tracks")
    f = _frames(r, e, threshold)
    if not f.ref_hz.size:
        return _scored_nan(fn, f"{f.why_none}, and GPE needs at least 1 frame "
                               f"voiced in both")
    return _gpe(f)

voicing_decision_error

voicing_decision_error(ref, est) -> float

Voicing decision error: the share of all frames where est and ref disagree on whether the frame is voiced.

(N_VU + N_UV) / N, [CA09] eq. 1. Every frame counts in N, so frames both tracks call unvoiced, such as silence before and after the speech, lower it; trim the tracks of every system compared the same way.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required

Returns:

Type Description
float

the fraction of all N frames with a voicing error, 0 to 1. nan, with a warning, for two empty tracks.

Raises:

Type Description
ValueError
Source code in kudio/core/f0eval.py
def voicing_decision_error(ref, est) -> float:
    """Voicing decision error: the share of all frames where *est* and *ref*
    disagree on whether the frame is voiced.

    (N_VU + N_UV) / N, [CA09] eq. 1. Every frame counts in N, so frames both
    tracks call unvoiced, such as silence before and after the speech, lower
    it; trim the tracks of every system compared the same way.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :raises ValueError: as :func:`gross_pitch_error`.
    :returns: the fraction of all N frames with a voicing error, 0 to 1.
        nan, with a warning, for two empty tracks.
    """
    fn = 'voicing_decision_error'
    r, e = _pair(ref, est, fn)
    if not r.size:
        return _scored_nan(fn, "got two empty tracks")
    ref_on, est_on = r > 0.0, e > 0.0
    return int(np.count_nonzero(ref_on != est_on)) / r.size

f0_frame_error

f0_frame_error(
    ref, est, *, threshold: float = 0.2
) -> float

F0 frame error: the share of all frames with any error, a voicing error or a gross pitch error.

(N_VU + N_UV + N_GE) / N, [CA09] eq. 4, which is VDE + (N_VV / N) * GPE. Calling a frame unvoiced instead of getting its pitch wrong moves the error from GPE to VDE and leaves FFE where it was, which is why it is one number to compare trackers by.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required
threshold float 0.2

Returns:

Type Description
float

the fraction of all N frames with an error, 0 to 1. nan, with a warning, for two empty tracks.

Raises:

Type Description
ValueError
Source code in kudio/core/f0eval.py
def f0_frame_error(ref, est, *, threshold: float = 0.2) -> float:
    """F0 frame error: the share of all frames with any error, a voicing
    error or a gross pitch error.

    (N_VU + N_UV + N_GE) / N, [CA09] eq. 4, which is
    ``VDE + (N_VV / N) * GPE``. Calling a frame unvoiced instead of getting
    its pitch wrong moves the error from GPE to VDE and leaves FFE where it
    was, which is why it is one number to compare trackers by.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :param threshold: as in :func:`gross_pitch_error`.
    :raises ValueError: as :func:`gross_pitch_error`.
    :returns: the fraction of all N frames with an error, 0 to 1. nan, with
        a warning, for two empty tracks.
    """
    fn = 'f0_frame_error'
    threshold = _checked_threshold(threshold, fn)
    r, e = _pair(ref, est, fn)
    if not r.size:
        return _scored_nan(fn, "got two empty tracks")
    return _ffe(_frames(r, e, threshold))

fine_pitch_error

fine_pitch_error(
    ref, est, *, threshold: float = 0.2
) -> float

Fine pitch error: how much est scatters around ref on the frames it gets right, as a fraction of the reference F0.

The standard deviation of (est - ref) / ref over the frames voiced in both whose estimate is within threshold of the reference, as [DA11] defines FPE; times 100 it is their percentage. It divides by the number of those frames, as [R76]'s eq. 3 does. Being a standard deviation, it does not see a constant offset: an estimate 1% sharp on every frame scores 0 here and 17.2 cents in f0_rmse.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required
threshold float

the line between fine and gross, as in gross_pitch_error.

0.2

Returns:

Type Description
float

the standard deviation, as a fraction. nan, with a warning, when fewer than 2 frames are voiced in both without a gross error.

Raises:

Type Description
ValueError
Source code in kudio/core/f0eval.py
def fine_pitch_error(ref, est, *, threshold: float = 0.2) -> float:
    """Fine pitch error: how much *est* scatters around *ref* on the frames
    it gets right, as a fraction of the reference F0.

    The standard deviation of ``(est - ref) / ref`` over the frames voiced in
    both whose estimate is within *threshold* of the reference, as [DA11]
    defines FPE; times 100 it is their percentage. It divides by the number
    of those frames, as [R76]'s eq. 3 does. Being a standard deviation, it
    does not see a constant offset: an estimate 1% sharp on every frame
    scores 0 here and 17.2 cents in :func:`f0_rmse`.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :param threshold: the line between fine and gross, as in
        :func:`gross_pitch_error`.
    :raises ValueError: as :func:`gross_pitch_error`.
    :returns: the standard deviation, as a fraction. nan, with a warning,
        when fewer than 2 frames are voiced in both without a gross error.
    """
    fn = 'fine_pitch_error'
    threshold = _checked_threshold(threshold, fn)
    r, e = _pair(ref, est, fn)
    if not r.size:
        return _scored_nan(fn, "got two empty tracks")
    f = _frames(r, e, threshold)
    if not f.ref_hz.size:
        return _scored_nan(fn, f"{f.why_none}, and FPE needs {_FINE_NEED}")
    fine = _fine(f)
    if fine.size < _MIN_FINE:
        return _scored_nan(fn, f"FPE needs {_FINE_NEED} to take a standard "
                               f"deviation over, and the pair has {fine.size}")
    return float(np.std(fine))

f0_rmse

f0_rmse(ref, est) -> float

Root mean square F0 error in cents over the frames voiced in both.

sqrt(mean((1200 log2(est / ref)) ** 2)), the F0 RMSE that speech synthesis papers report. A cent is a hundredth of a semitone, so the same error counts the same in a low voice and a high one, which an RMSE in Hz does not. Gross errors count at full size: one octave error among 100 frames, the rest exact, is 120 cents. Synthesized speech has to be put on the reference's frames first, by DTW for instance; this compares the frames as given.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required

Returns:

Type Description
float

the RMSE in cents. nan, with a warning, when no frame is voiced in both.

Raises:

Type Description
ValueError

as gross_pitch_error, threshold aside.

Source code in kudio/core/f0eval.py
def f0_rmse(ref, est) -> float:
    """Root mean square F0 error in cents over the frames voiced in both.

    ``sqrt(mean((1200 log2(est / ref)) ** 2))``, the F0 RMSE that speech
    synthesis papers report. A cent is a hundredth of a semitone, so the
    same error counts the same in a low voice and a high one, which an RMSE
    in Hz does not. Gross errors count at full size: one octave error among
    100 frames, the rest exact, is 120 cents. Synthesized speech has to be
    put on the reference's frames first, by DTW for instance; this compares
    the frames as given.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :raises ValueError: as :func:`gross_pitch_error`, *threshold* aside.
    :returns: the RMSE in cents. nan, with a warning, when no frame is voiced
        in both.
    """
    fn = 'f0_rmse'
    r, e = _pair(ref, est, fn)
    if not r.size:
        return _scored_nan(fn, "got two empty tracks")
    f = _frames(r, e)
    if not f.ref_hz.size:
        return _scored_nan(fn, f"{f.why_none}, and the RMSE needs at least 1 "
                               f"frame voiced in both")
    return _rmse_cents(f)

f0_errors

f0_errors(ref, est, *, threshold: float = 0.2) -> F0Errors

GPE, VDE, FFE, FPE and the RMSE in cents of est against ref in one pass, with the frame counts they come from.

>>> kudio.f0_errors([0, 100, 102, 104, 0, 0, 120, 121],
...                 [0, 101, 204, 0, 0, 98, 119, 0])
F0Errors(gpe=0.3333333333333333, vde=0.375, ffe=0.5,
         fpe=0.009166666666666667, rmse_cents=692.9421890055355,
         frames=8, voiced_both=3, gross=1, voiced_to_unvoiced=2,
         unvoiced_to_voiced=1)

The octave error at frame 2 is 1200 of the RMSE's cents and none of FPE's, which is left with the 1% and -0.83% of the other two frames. The numbers are those of gross_pitch_error, voicing_decision_error, f0_frame_error, fine_pitch_error and f0_rmse, and so are the rules for what each one cannot score.

Parameters:

Name Type Description Default
ref

the reference track, F0 in Hz frame by frame with 0 or nan where unvoiced, or a kudio.PitchTrack.

required
est

the estimate, on the same frames, in the same form.

required
threshold float 0.2

Returns:

Type Description
F0Errors

an F0Errors. A field that cannot be scored is nan, and one warning names them all: every rate for two empty tracks; gpe, fpe and rmse_cents when no frame is voiced in both; fpe alone when fewer than 2 frames are voiced in both without a gross error.

Raises:

Type Description
ValueError
Source code in kudio/core/f0eval.py
def f0_errors(ref, est, *, threshold: float = 0.2) -> F0Errors:
    """GPE, VDE, FFE, FPE and the RMSE in cents of *est* against *ref* in one
    pass, with the frame counts they come from.

    >>> kudio.f0_errors([0, 100, 102, 104, 0, 0, 120, 121],
    ...                 [0, 101, 204, 0, 0, 98, 119, 0])  # doctest: +NORMALIZE_WHITESPACE
    F0Errors(gpe=0.3333333333333333, vde=0.375, ffe=0.5,
             fpe=0.009166666666666667, rmse_cents=692.9421890055355,
             frames=8, voiced_both=3, gross=1, voiced_to_unvoiced=2,
             unvoiced_to_voiced=1)

    The octave error at frame 2 is 1200 of the RMSE's cents and none of
    FPE's, which is left with the 1% and -0.83% of the other two frames.
    The numbers are those of :func:`gross_pitch_error`,
    :func:`voicing_decision_error`, :func:`f0_frame_error`,
    :func:`fine_pitch_error` and :func:`f0_rmse`, and so are the rules for
    what each one cannot score.

    :param ref: the reference track, F0 in Hz frame by frame with 0 or nan
        where unvoiced, or a :class:`kudio.PitchTrack`.
    :param est: the estimate, on the same frames, in the same form.
    :param threshold: as in :func:`gross_pitch_error`.
    :raises ValueError: as :func:`gross_pitch_error`.
    :returns: an :class:`F0Errors`. A field that cannot be scored is nan, and
        one warning names them all: every rate for two empty tracks; gpe,
        fpe and rmse_cents when no frame is voiced in both; fpe alone when
        fewer than 2 frames are voiced in both without a gross error.
    """
    fn = 'f0_errors'
    threshold = _checked_threshold(threshold, fn)
    r, e = _pair(ref, est, fn)
    nan = float('nan')
    if not r.size:
        warn(f"{fn}(): got two empty tracks; scored nan")
        return F0Errors(nan, nan, nan, nan, nan, 0, 0, 0, 0, 0)
    f = _frames(r, e, threshold)
    counts = dict(frames=f.frames, voiced_both=int(f.ref_hz.size),
                  gross=int(np.count_nonzero(f.gross)),
                  voiced_to_unvoiced=f.voiced_to_unvoiced,
                  unvoiced_to_voiced=f.unvoiced_to_voiced)
    if not f.ref_hz.size:
        warn(f"{fn}(): {f.why_none}, so gpe, fpe and rmse_cents have "
             f"no frame to measure; scored nan")
        return F0Errors(gpe=nan, vde=_vde(f), ffe=_ffe(f), fpe=nan,
                        rmse_cents=nan, **counts)
    fine = _fine(f)
    if fine.size < _MIN_FINE:
        warn(f"{fn}(): fpe needs {_FINE_NEED}, and the pair has "
             f"{fine.size}; scored nan")
        fpe = nan
    else:
        fpe = float(np.std(fine))
    return F0Errors(gpe=_gpe(f), vde=_vde(f), ffe=_ffe(f), fpe=fpe,
                    rmse_cents=_rmse_cents(f), **counts)