Quick-start guide

We present here a recipe for a full bioacoustics ML workflow, from data pre-processing to training, to performance assessments, and finally, to using a trained model for analyzing soundscape/field recordings.

As an example, we considered the North Atlantic Right Whale (NARW) up-call dataset from the DCLDE 2013 challenge. The dataset contained 7 days of round-the-clock recordings out of which recordings from the first 4 days were earmarked as a training set and recordings from the remaining 3 days were set aside as a test set. Each audio file was 15 minutes in duration, and files from each day were organized in day-specific subdirectories. The original dataset contained annotations in the legacy Xbat format, which we converted to RavenPro selection table format for compatibility with Koogu. A sample of the dataset, with converted annotations, can be accessed here.

You may test the below code snippets yourself, using the sample dataset. Once, you have it working, you could modify the program to suit your own dataset.

The code sections below expect the training and test audio files and corresponding annotation files to be organized in a directory structure as shown below:

📁 projects
└─ 📁 NARW
   └─ 📁 data
      ├─ 📁 train_audio
      ├─ 📁 train_annotations
      ├─ 📁 test_audio
      └─ 📁 test_annotations

Imports

First, import the necessary modules and functions from the Koogu package.

from koogu.data import feeder
from koogu.model import architectures
from koogu import prepare, train, assessments, recognize

from matplotlib import pyplot as plt           # used for plotting graphs

1. Data preparation

Point out where to fetch the training dataset from.

We also need to specify which annotation files correspond to which audio files (or, in this example, to sub-directories containing a collection of files).

# The root directories under which the training data (audio files and
# corresponding annotation files) are available.
audio_root = '/home/shyam/projects/NARW/data/train_audio'
annots_root = '/home/shyam/projects/NARW/data/train_annotations'

# Map audio files (or containing folders) to respective annotation files
audio_annot_list = [
    ['NOPP6_EST_20090328', 'NOPP6_20090328_RW_upcalls.selections.txt'],
    ['NOPP6_EST_20090329', 'NOPP6_20090329_RW_upcalls.selections.txt'],
    ['NOPP6_EST_20090330', 'NOPP6_20090330_RW_upcalls.selections.txt'],
    ['NOPP6_EST_20090331', 'NOPP6_20090331_RW_upcalls.selections.txt'],
]

Define parameters for preparing training audio, and for converting them to spectrograms.

data_settings = {
    # Settings for handling raw audio
    'audio_settings': {
        'clip_length': 2.0,
        'clip_advance': 0.4,
        'desired_fs': 1000
    },

    # Settings for converting audio to a time-frequency representation
    'spec_settings': {
        'win_len': 0.128,
        'win_overlap_prc': 0.75,
        'bandwidth_clip': [46, 391]
    }
}

1.1 Preprocess

The preprocessing step will split up the audio files into clips (defined by data_settings['audio_settings']), match available annotations to the clips, and mark each clip to indicate if it matched one or more annotations.

We believe that the available annotations in the training set covered almost all occurrences of the target NARW up-calls in the recordings, with no (or only a small number of) missed calls. As such, we can consider all un-annotated time periods in the recordings as inputs for the negative class (by setting the parameter negative_class_label).

# Path to the directory where pre-processed data will be written.
# Directory will be created if it doesn't exist.
prepared_audio_dir = '/home/shyam/projects/NARW/prepared_data'

# Convert audio files into prepared data
clip_counts = prepare.from_selection_table_map(
    data_settings['audio_settings'],
    audio_annot_list,
    audio_root, annots_root,
    output_root=prepared_audio_dir,
    negative_class_label='Other')

1.2. Feeder setup

Now, we define a feeder that efficiently feeds all the pre-processed clips, in batches, to the training/validation pipeline. The feeder is also transforms the audio clips into spectrograms.

Typically, model training is performed on computers having one or more GPUs. While the GPUs consume data at extreme speeds during training, it is imperative that the mechanism to feed the training data doesn’t keep the GPUs waiting for inputs. The feeders provided in Koogu utilize all available CPU cores to ensure that GPU utilization remains high during training.

data_feeder = feeder.SpectralDataFeeder(
    prepared_audio_dir,                        # where the prepared clips are at
    data_settings['audio_settings']['desired_fs'],
    data_settings['spec_settings'],
    validation_split=0.15,                     # set aside 15% for validation
    max_clips_per_class=20000                  # use up to 20k inputs per class
)

The considered sample dataset contains very many annotated calls, covering a reasonable range of input variations. As such, in this example we do not employ any data augmentation techniques. However, you could easily add some of the pre-canned data augmentations when you adopt this example to work with your dataset.

2. Training

First, describe the architecture of the model that is to be used. With Koogu, you do not need to write lot’s of code to build custom models; simply chose an exiting/available architecture (e.g., ConvNet, DenseNet) and specify how you’d want it customized.

In this example, we use a light-weight custom DenseNet architecture.

model = architectures.DenseNet(
    [4, 4, 4],                                 # 3 dense-blocks, 4 layers each
    preproc=[ ('Conv2D', {'filters': 16}) ],   # Add a 16-filter pre-conv layer
    dense_layers=[32]                          # End with a 32-node dense layer
)

The training process can be controlled, along with hyperparameter and regularization settings, by setting appropriate values in the Python dictionary that is input to train(). See the function API documentation for all available options.

# Settings that control the training process
training_settings = {
    'batch_size': 64,
    'epochs': 50,                              # run for 50 epochs

    # Start with a learning rate of 0.01, and drop it to a tenth of its value,
    # successively, at epochs 20 & 40.
    'learning_rate': 0.01,
    'lr_change_at_epochs': [20, 40],
    'lr_update_factors': [1.0, 1e-1, 1e-2],    # up to 20, beyond 20, beyond 40

    'dropout_rate': 0.05                       # Helps model generalize better
}

# Path to the directory where model files will be written
model_dir = '/home/shyam/projects/NARW/models/my_first_model'

# Perform training
history = train(
    data_feeder,
    model_dir,
    data_settings,
    model,
    training_settings
)

You can visualize how well the training progressed by plotting the contents of the history variable returned.

# Plot training & validation history
fig, ax = plt.subplots(2, sharex=True, figsize=(12, 9))
ax[0].plot(
    history['train_epochs'], history['binary_accuracy'], 'r',
    history['eval_epochs'], history['val_binary_accuracy'], 'g')
ax[0].set_ylabel('Accuracy')
ax[1].plot(
    history['train_epochs'], history['loss'], 'r',
    history['eval_epochs'], history['val_loss'], 'g')
ax[1].set_yscale('log')
ax[1].set_xlabel('Epoch')
ax[1].set_ylabel('Loss')
plt.show()

You may tune the training parameters above and repeat the training step until the training and validation accuracy/loss reach desired levels.

3. Performance assessment

3.1. Run on test dataset

If you have a test dataset available for assessing performance, you can easily run the trained model on that dataset. Simply point out where to fetch the test dataset from.

Similar to how training annotation data were presented (by associating annotation files to audio files), we also need to specify which test annotation files correspond to which test audio files (or, in this example, to sub-directories containing a collection of test files).

# The root directories under which the test data (audio files and
# corresponding annotation files) are available.
test_audio_root = '/home/shyam/projects/NARW/data/test_audio'
test_annots_root = '/home/shyam/projects/NARW/data/test_annotations'

# Map audio files to corresponding annotation files
test_audio_annot_list = [
    ['NOPP6_EST_20090401', 'NOPP6_20090401_RW_upcalls.selections.txt'],
    ['NOPP6_EST_20090402', 'NOPP6_20090402_RW_upcalls.selections.txt'],
    ['NOPP6_EST_20090403', 'NOPP6_20090403_RW_upcalls.selections.txt'],
]

Now apply the trained model to this test dataset. During testing, it is useful to save raw per-clip recognition scores which can be subsequently analyzed for assessing the model’s recognition performance.

# Directory in which raw detection scores will be saved
raw_detections_root = '/home/shyam/projects/NARW/test_audio_raw_detections'

# Run the model (detector/classifier)
recognize(
    model_dir,
    test_audio_root,
    raw_detections_dir=raw_detections_root,
    batch_size=64,     # Increasing this may improve speed if there's enough RAM
    recursive=True,    # Process subdirectories also
    show_progress=True
)

The recognize() function supports many customizations. See function API documentation for more details.

3.2. Determine performance

Now, compute performance metrics.

# Initialize a metric object with the above info
metric = assessments.PrecisionRecall(
    test_audio_annot_list,
    raw_detections_root, test_annots_root)
# The metric supports several options (including setting explicit thresholds).
# Refer to class documentation for more details.

# Run the assessments and gather results
per_class_pr, overall_pr = metric.assess()

And, visualize the assessments.

# Plot PR curves.
for class_name, pr in per_class_pr.items():
    print('-----', class_name, '-----')
    plt.plot(pr['recall'], pr['precision'], 'rd-')
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.grid()
    plt.show()

# Similarly, you could plot the contents of 'overall_pr' too

By analyzing the precision-recall curve, you can pick an operational threshold that yields the desired precision vs. recall trade-off.

4. Use the trained model

Once you are settled on a choice of detection threshold that yields a suitable precision-recall trade-off, you can apply the trained model on new recordings.

Koogu supports two ways of using a trained model.

4.1. Batch processing

In most common applications, one would want to be able to batch process large collections of audio files with a trained model.

In this mode, automatic recognition results are written out in Raven Pro selection table format after applying an algorithm to group together detections of the same class across contiguous clips.

# Path to directory containing audio files (may contain subdirectories too)
field_recordings_root = '/home/shyam/projects/NARW/field_recordings'
field_rec_detections_root = '/home/shyam/projects/NARW/field_rec_detections'

chosen_threshold = 0.75

recognize(
    model_dir,
    field_recordings_root,
    output_dir=field_rec_detections_root,
    threshold=chosen_threshold,
    reject_class='Other',                      # Only output target class dets
    #clip_advance=0.5,                         # Can use different clip advance
    batch_size=64,                             # Can go higher on good computers
    num_fetch_threads=4,                       # Parallel-process for speed
    recursive=True,                            # Process subdirectories also
    show_progress=True
)

The recognize() function supports many customizations. See function API documentation for more details.

4.2 Custom processing

Sometimes, one may need to process audio data that is not available in the form of audio files (or in unsupported formats). For example, one may want to apply a trained model to live-stream acoustic feeds. Koogu facilitates such use of a trained model via an additional interface in which you implement the task of preparing the data (breaking up into clips) in the format that a model expects. Then, you simply pass the clips to analyze_clips().

from koogu.model import TrainedModel
from koogu.inference import analyze_clips

# Load the trained model
trained_model = TrainedModel(model_dir)

# Read in the audio samples from a file (using one of SoundFile, AudioRead,
# scipy.io.wavfile, etc.), or buffer-in from a live stream.

# As with the model trained in the above example, you may need to resample the
# new data to 1 kHz, and then break them up into clips of length 2 s to match
# the trained model's input size.

not_end = True

while not_end:

    my_clips = ...
    # say we got 6 clips, making it a 6 x 2000 numpy array

    # Run detections and get per-clip scores for each class
    scores, processing_time = analyze_clips(trained_model, my_clips)
    # Given 6 clips, we get 'scores' to be a 6 x 2 array

    # ... do something with the results
    ...