Performance assessment

class koogu.utils.assessments.PrecisionRecall(audio_annot_list, raw_results_root, annots_root, annotation_reader=None, thresholds=None, post_process_detections=False, **kwargs)

Bases: BaseMetric

Class for assessing precision-recall values.

Parameters:

  • audio_annot_list – A list containing pairs (tuples or sub-lists) of relative paths to audio files and the corresponding annotation files. Alternatively, you could also specify (path to) a 2-column csv file containing these pairs of entries (in the same order). Only use the csv option if the paths are simple (i.e., the filenames do not contain commas or other special characters).

  • raw_results_root – The full paths of the raw result container files whose filenames will be derived from the audio files listed in audio_annot_list will be resolved using this as base directory.

  • annots_root – The full paths of annotations files listed in audio_annot_list will be resolved using this as base directory.

  • annotation_reader – If not None, must be an annotation reader instance from annotations. Defaults to Raven Reader.

  • thresholds – If not None, must be either a scalar quantity or a list of non-decreasing values (float values in the range 0-1) at which precision and recall value(s) will be assessed. If None, will default to the range 0-1 with an interval of 0.05.

  • post_process_detections – If True (default: False), a post-processing algorithm will be applied to the raw detections before computing performance stats.

Optional parameters

Parameters:

  • suppress_nonmax – If True (default: False), only the top-scoring class per clip will be considered. When post-processing is enabled, the parameter is handled directly in koogu.utils.detections.postprocess_detections().

  • squeeze_min_dur – (default: None). If set (duration in seconds), an algorithm “to squeeze together” temporally overlapping regions from successive raw clips will be applied. The ‘squeezing’ will be restricted to produce detections that are at least as long as the specified value. The value must be smaller than the duration of the model inputs. Parameter used only when post-processing is enabled, and converts the duration to number of samples before passing it to koogu.utils.detections.postprocess_detections().

Parameters specific to

can also be specified, and will be passed as-is to the respective functions.

All other kwargs parameters (if any) will be passed as-is to the base class.

When calling assess(), passing return_counts=True will return the per-class counts for the numerators and denominators of precision and recall. Otherwise, per-class and overall precision-recall values will be returned.

See also

koogu.data.annotations

assess(**kwargs)

Perform the desired assessments.

class koogu.utils.assessments.BaseMetric(audio_annot_list, raw_results_root, annots_root, annotation_reader=None, reject_classes=None, remap_labels_dict=None, negative_class_label=None, **kwargs)

Base class for implementing performance assessment logic.

Parameters:

  • audio_annot_list – A list containing pairs (tuples or sub-lists) of relative paths to audio files and the corresponding annotation files. Alternatively, you could also specify (path to) a 2-column csv file containing these pairs of entries (in the same order). Only use the csv option if the paths are simple (i.e., the filenames do not contain commas or other special characters).

  • raw_results_root – The full paths of the raw result container files whose filenames will be derived from the audio files listed in audio_annot_list will be resolved using this as base directory.

  • annots_root – The full paths of annotations files listed in audio_annot_list will be resolved using this as base directory.

  • annotation_reader – If not None, must be an annotation reader instance from annotations. Defaults to Raven Reader.

  • reject_classes – Name (case sensitive) of the class (like ‘Noise’ or ‘Other’) for which performance assessments are not to be computed. Can specify multiple classes for rejection, as a list.

  • remap_labels_dict – If not None, must be a Python dictionary describing mapping of class labels. For details, see similarly named parameter to the constructor of koogu.utils.detections.LabelHelper.

  • negative_class_label – A string (e.g. ‘Other’, ‘Noise’) which will be used as a label to identify the negative class clips (those that did not match any annotations), if an inherited class deals with those. If specified, will be used in conjunction with remap_labels_dict.

assess(**kwargs)

Perform the desired assessments.