Testing with Custom Data Randomization for Better Reproducible Results

[ab-tools]

Hello,

as per discussion here I've set shuffle = False in the data loader for training in an attempt to get better reproducible results.

But with all negative samples at the start and positive at the end of each epoch results were really bad.

Therefore, I've implemented a custom randomization of the training data which is done only once before training to keep it reproducible:
This seems to work, but looks like I still have a mistake in there as prediction values are not between 0 and 1 as expected, but between 0.000 and 0.001 instead. Maybe you have an idea, @dscripka, what I'm doing wrong here?

In the following the full Python code I'm using. Let's start with loading the training data and generating the negative and positive feature sets in a first Python session:

import os
import math
import random
import collections
import numpy as np
from numpy.lib.format import open_memmap
from pathlib import Path
from tqdm import tqdm
import openwakeword
import openwakeword.data
import openwakeword.utils
import openwakeword.metrics

import scipy
import datasets
import matplotlib.pyplot as plt
import torch
from torch import nn
import IPython.display as ipd



# Create audio pre-processing object to get openWakeWord audio embeddings

F = openwakeword.utils.AudioFeatures(inference_framework = "onnx", device = "gpu")



# Get negative example paths, filtering out clips that are too long or too short

negative_clips, negative_durations = openwakeword.data.filter_audio_paths(
    [
        "fma_large",
        "fsd50k/dev_audio",
        "fsd50k/eval_audio",
        "cv13/de_dev",
        "cv13/de_test",
        "cv13/de_train",
        "cv13/en_dev",
        "cv13/en_test"
    ],
    min_length_secs = 0.75, # minimum clip length in seconds
    max_length_secs = 60*30, # maximum clip length in seconds
    duration_method = "size" # use the file header to calculate duration
)

print(f"{len(negative_clips)} negative clips after filtering, representing ~{sum(negative_durations)//3600} hours")

# Shuffle negative clips
seed = os.getrandom(8)
random.seed(seed)
random.shuffle(negative_clips)
random.seed(seed)
random.shuffle(negative_durations)



# Use HuggingFace datasets to load files from disk by batches

audio_dataset = datasets.Dataset.from_dict({"audio": negative_clips})
audio_dataset = audio_dataset.cast_column("audio", datasets.Audio(sampling_rate=16000))



# Get audio embeddings (features) for negative clips and save to .npy file
# Process files by batch and save to Numpy memory mapped file so that
# an array larger than the available system memory can be created

batch_size = 1024 # number of files to load, compute features, and write to mmap at a time
clip_size = 2  # the desired window size (in seconds) for the trained openWakeWord model
N_total = int(sum(negative_durations)//clip_size) # maximum number of rows in mmap file
n_feature_cols = F.get_embedding_shape(clip_size)

output_file = "negative_features_random.npy"
output_array_shape = (N_total, n_feature_cols[0], n_feature_cols[1])
fp = open_memmap(output_file, mode='w+', dtype=np.float32, shape=output_array_shape)

row_counter = 0
for i in tqdm(np.arange(0, audio_dataset.num_rows, batch_size)):
    # Load data in batches and shape into rectangular array
    wav_data = [(j["array"]*32767).astype(np.int16) for j in audio_dataset[i:i+batch_size]["audio"]]
    wav_data = openwakeword.data.stack_clips(wav_data, clip_size=16000*clip_size).astype(np.int16)
    
    # Compute features (increase ncpu argument for faster processing)
    features = F.embed_clips(x=wav_data, batch_size=2048, ncpu=12)
    
    # Save computed features to mmap array file (stopping once the desired size is reached)
    if row_counter + features.shape[0] > N_total:
        fp[row_counter:min(row_counter+features.shape[0], N_total), :, :] = features[0:N_total - row_counter, :, :]
        fp.flush()
        break
    else:
        fp[row_counter:row_counter+features.shape[0], :, :] = features
        row_counter += features.shape[0]
        fp.flush()

# Trip empty rows from the mmapped array
openwakeword.data.trim_mmap(output_file)



# Get positive example paths, filtering out clips that are too long or too short

positive_clips, durations = openwakeword.data.filter_audio_paths(
    [
        "synthetic_speech_dataset_generation/out_vits_1",
        "synthetic_speech_dataset_generation/out_vits_2",
        "synthetic_speech_dataset_generation/out_vits_3",
        "synthetic_speech_dataset_generation/out_vits_4"
    ],
    min_length_secs = 0.75, # minimum clip length in seconds
    max_length_secs = 1.75, # maximum clip length in seconds
    duration_method = "size" # use the file header to calculate duration
)

print(f"{len(positive_clips)} positive clips after filtering")

# Shuffle positive clips
seed = os.getrandom(8)
random.seed(seed)
random.shuffle(positive_clips)
random.seed(seed)
random.shuffle(durations)



# Define starting point for each positive clip based on its length, so that each one ends 
# between 0-200 ms from the end of the total window size chosen for the model.
# This results in the model being most confident in the prediction right after the
# end of the wakeword in the audio stream, reducing latency in operation.

# Get start and end positions for the positive audio in the full window
sr = 16000
total_length_seconds = 2 # must be the some window length as that used for the negative examples
total_length = int(sr*total_length_seconds)

jitters = (np.random.uniform(0, 0.2, len(positive_clips))*sr).astype(np.int32)
starts = [total_length - (int(np.ceil(i*sr))+j) for i,j in zip(durations, jitters)]
ends = [int(i*sr) + j for i, j in zip(durations, starts)]

# Create generator to mix the positive audio with background audio
batch_size = 12
mixing_generator = openwakeword.data.mix_clips_batch(
    foreground_clips = positive_clips,
    background_clips = negative_clips,
    combined_size = total_length,
    batch_size = batch_size,
    snr_low = 5,
    snr_high = 15,
    start_index = starts,
    volume_augmentation=True, # randomly scale the volume of the audio after mixing
)



# Iterate through the mixing generator, computing audio features for positive examples and saving them

N_total = len(positive_clips) # maximum number of rows in mmap file
n_feature_cols = F.get_embedding_shape(total_length_seconds)

output_file = "positive_features_random.npy"
output_array_shape = (N_total, n_feature_cols[0], n_feature_cols[1])

fp = open_memmap(output_file, mode='w+', dtype=np.float32, shape=output_array_shape)

row_counter = 0
for batch in tqdm(mixing_generator, total=N_total//batch_size):
    batch, lbls, background = batch[0], batch[1], batch[2]
    
    # Compute audio features
    features = F.embed_clips(batch, batch_size=2048, ncpu=12)
    
    # Save computed features
    fp[row_counter:row_counter+features.shape[0], :, :] = features
    row_counter += features.shape[0]
    fp.flush()
    
    if row_counter >= N_total:
        break

# Trip empty rows from the mmapped array
openwakeword.data.trim_mmap(output_file)

The result of this are two NPY files, negative_features_random.npy and positive_features_random.npy, containing the feature sets accordingly.

In the script above the negative and positive samples are internally already in random order, now we just need to make sure they are also mixed during actual training. I'm therefore preparing the X and Y training data source sets (as per the original training notebook) in randomized order like this:

# Mix positive and negative features and save to combined X/Y features

total_length_seconds = 2
negative_features = np.load("negative_features_random.npy", "r")
positive_features = np.load("positive_features_random.npy", "r")

N_total = len(negative_features) + len(positive_features) # maximum number of rows in mmap file
n_feature_cols = F.get_embedding_shape(total_length_seconds)

output_file = "X_features_random.npy"
output_array_shape = (N_total, n_feature_cols[0], n_feature_cols[1])

fp = open_memmap(output_file, mode='w+', dtype=np.float32, shape=output_array_shape)

# All 0 now as we will set 1 for positive features in the loop below
Y = np.array([0]*len(negative_features) + [0]*len(positive_features)).astype(np.float32)[...,None]

object_counter = 0
row_counter = 0
negative_pos = 0
negatives_per_positive = math.floor(len(negative_features) / len(positive_features))
for i in tqdm(range(len(positive_features)), total=len(positive_features)):
    fp[row_counter:row_counter+positive_features[i].shape[0], :, :] = positive_features[i]
    row_counter += positive_features[i].shape[0]
    
    Y[object_counter] = 1
    object_counter += 1
    
    # After each positive feature and couple negative
    for j in range(negatives_per_positive):
        fp[row_counter:row_counter+negative_features[negative_pos].shape[0], :, :] = negative_features[negative_pos]
        row_counter += negative_features[negative_pos].shape[0]
        negative_pos += 1
        object_counter += 1
    
    fp.flush()

# Add last few negative features possibly left
for i in range(negative_pos, len(negative_features)):
    fp[row_counter:row_counter+negative_features[negative_pos].shape[0], :, :] = negative_features[negative_pos]
    row_counter += negative_features[negative_pos].shape[0]
    negative_pos += 1
    object_counter += 1

fp.flush()
np.save("Y_features_random.npy", Y)

Now we have the X and Y training feature sets prepared and can just load them for actual training:

# Load the data prepared in previous steps

X = np.load("X_features_random.npy", "r")
Y = np.load("Y_features_random.npy", "r")

# Make Pytorch dataloader
batch_size = 1024
training_data = torch.utils.data.DataLoader(
    torch.utils.data.TensorDataset(torch.from_numpy(X), torch.from_numpy(Y)),
    batch_size = batch_size,
    shuffle = False
)


# Define fully-connected network in PyTorch

layer_dim = 32
fcn = nn.Sequential(
                    nn.Flatten(),
                    nn.Linear(X.shape[1]*X.shape[2], layer_dim), # since the input is flattened, it's timesteps*feature columns
                    nn.LayerNorm(layer_dim),
                    nn.ReLU(),
                    nn.Linear(layer_dim, layer_dim),
                    nn.LayerNorm(layer_dim),
                    nn.ReLU(),
                    nn.Linear(layer_dim, 1),
                    nn.Sigmoid(),
                )
fcn.cuda()

loss_function = torch.nn.functional.binary_cross_entropy
optimizer = torch.optim.Adam(fcn.parameters(), lr=0.001)


# Define training loop, metrics, and logging

n_epochs = 10
history = collections.defaultdict(list)
for i in tqdm(range(n_epochs), total=n_epochs):
    for batch in training_data:
        # Get data for batch
        x, y = batch[0], batch[1]
        
        # Get weights for classes, and assign 10x higher weight to negative class
        # to help the model learn to not have too many false-positives
        # As you have more data (both positive and negative), this is less important
        weights = torch.ones(y.shape[0])
        weights[y.flatten() == 1] = 0.1
        
        # Zero gradients
        optimizer.zero_grad()
        
        # Run forward pass
        predictions = fcn(x.cuda())
        
        # Update model parameters
        temp = weights[..., None]
        loss = loss_function(predictions, y.cuda(), temp.cuda())
        loss.backward()
        optimizer.step()
        
        # Log metrics
        try:
            history['loss'].append(float(loss.detach().cpu().numpy()))
        except AttributeError:
            history['loss'].append(float(0))
        
        try:
            tp = sum(predictions.flatten()[y.flatten() == 1] >= 0.5).cpu()
            fn = sum(predictions.flatten()[y.flatten() == 1] < 0.5).cpu()
            history['recall'].append(float(tp/(tp+fn).detach().numpy()))
        except AttributeError:
            history['recall'].append(float(0))


# Export model to ONNX format

output_path = "trained-model-cuda10-random1.onnx"
torch.onnx.export(fcn, args=torch.zeros((1, 16, 96)).cuda(), verbose=True, f=output_path) # the 'args' is the shape of a single example

After training finished I'm plotting the graph and as you can already see here I need to use 0,0.001 as the range for the y axis:

plt.figure()
plt.plot(history['loss'], label="loss")
plt.plot(history['recall'], label="recall")
plt.legend()
plt.ylim(0,0.001)
plt.savefig('trained-model-cuda10-random1.png')

The result looks like this:

And just to compare, without my "manual randomization" and leaving shuffle = True in the data loader, loss/recall plot looks like this:

So it seems that something is clearly wrong (like inverted?) with the training feature sets, but I can't really figure out what I'm doing wrong - must be something super simple as so often. ;-)

Interestingly though, the model does still work "OK". Let's load the exported model again for a test:

oww = openwakeword.Model(
    wakeword_models=["trained-model-cuda10-random1.onnx"],
    enable_speex_noise_suppression=False,
    vad_threshold=0.5,
    inference_framework="onnx"
)

And test again a single positive sample:

# Do a quick test prediction on the test clip to confirm that the behavior is still as expected
scores = oww.predict_clip("positive-test-sample.wav")

plt.figure()
_ = plt.plot([i["trained-model-cuda10-random1"] for i in scores])
_ = plt.ylim(0,0.001)
plt.savefig('trained-model-cuda10-random1-positive.png')

Again you will notice that y axis range has to be 0,0.001 instead of 0,1.

It does recognize the single positive hit in the sample correctly:

And now we calculate the false accept rate:

scores = oww.predict_clip("santa_barbara_corpus_test_clip.wav")

false_accepts = openwakeword.metrics.get_false_positives(
    [i["trained-model-cuda10-random1"] for i in scores], threshold=0.0005
)

print(f"False-accepts: {false_accepts} in one hour sample data")

plt.figure()
_ = plt.plot([i["trained-model-cuda10-random1"] for i in scores])
_ = plt.ylim(0,0.001)
plt.savefig(f"trained-model-cuda10-random1-falseaccepts-{false_accepts}.png")

The result is 12. That's not great, but I also have seen worse. The resulting graph is interesting though:

As you can see this is not only between 0 and 0.001 again, but there is also a constant "base" around 0.0002 for some reason.

For comparison again the resulting graph when run without my "manual randomization" and shuffle = True:

So it's clear that something is going wrong with my data randomization function above, which I could not figure out yet...
Maybe you have an idea or perhaps even seen something similar before?

Thanks a lot
Andreas

[ab-tools]

Hello @dscripka,

did you by chance had the option to take a look on this topic?

I think the openWakeWord training would benefit a lot from using a custom, deterministic randomization like I tried above instead of asking the PyTorch data loader to shuffle everything which leads to considerable different training results each time you run the training.

I still could not find out what I'm doing wrong in the code above, therefore I would much appreciate your opinion/support on this.

Best regards and thanks a lot
Andreas

[dscripka]

Hey @ab-tools, apologies for the late reply.

It's difficult to tell what could be the issue with your code above. Generally, though, if your loss/recall curves look like that usually the issue is with the data preparation. Ultimately, I'm not sure that shuffling of the data is the root cause of training instabilities as if you train for a sufficient number of epochs each data point will be seen multiple times be the model, averaging out any ordering effects. In my tests, the dominant factor for reproducibility seems to be just the network initialization. Have you tried setting the seed for Pytorch, as discussed here: https://discuss.pytorch.org/t/reproducibility-dataloader-shuffle-true-using-seeds/173836?

Additionally, another factor to consider is the distribution of labels in each batch. I see in your code above that you have a batch size of 1024, but depending on how many positive and negative examples you have there can be a wide range of class ratios per batch. I find that you need at least 30 positive examples per batch for the model to train well, but this varies substantially depending on the particular data you are using.

I'm still actively working on an entirely new training process (hopefully ready to release in the next ~1 week) that automates a wide range of these steps, which should significantly will reduce the amount of effort for training new models. In my tests thus far, this doesn't eliminate the variability across training runs, but it does seem to reduce it to the point where practically it may not matter.

[ab-tools]

Thanks, @dscripka, much appreciating you taking the time to look into this!

If you say that you will release your new training notebook within just a week anyway and it also reduces variability across training runs, I think I'm just going to wait for this and try again then.

Would be great if you could post a reply here as soon as I can give it a try - looking forward to that!

Best regards and thanks a lot for all your efforts
Andreas

[dscripka]

@ab-tools, the new training process is almost ready! I'll reply back to this discussion when I've confirmed everything is working correctly in Google Colab.

#48

[ab-tools]

Hello @dscripka,

let me first emphasize again how much I appreciate your effort on this and also taking the time to reply to me here again!

I see that you reduced the standard clip size from 3 to 2 sec. which I think it's a good idea as most wake words are not that long.

I also think it's a great idea to provide prepared sets for negative training and false positive rate estimation as these can surely be used for almost all models in the same way.

I do realize though that we still need negative audio samples for augmenting the auto-generated synthetic positive samples and I couldn't hold back with at least trying this part (downloading/preparing the negative samples for augmentation) of your new script directly. ;-)

Two remarks about this part:

1. "full-scale training" vs. "small test training":

You did a great job to prepare your new notebook for "full-scale training" and not just a small test bench, making it highly configurable with the YAML file and thus allowing to do a "small test training" (which will be relatively fast) before performing a "full-scale training" which takes a long time.

I do think, however, it's a pity that you did not take the same approach also for the negative audio samples:
Here you only auto-download and process a small subset again with the note that for "full-scale training" you should use much more than this.

Personally, I would suggest to use the same approach for this step as for the rest of your notebook - make it configurable in the YAML file:
By default, it should download the full negative audio sample (Audioset Dataset, FMA, FSD50k, Freesound, etc.) as you would recommend it to use for "full-scale training". And to allow a "small test training" like you do for the rest of your notebook (which I think is a very good idea), there should be a configuration option to limit the amount of negative audio samples used.

2. Strange WAV audio format used:

I noticed that the generated 16k WAV audio samples use a pretty strange/big WAV audio format and I wonder if that is required.

With your "old" notebook the WAV audio samples were converted with SOX/FFMPEG, I think, and the result was codec "PCM Audio", 16000 hz, 1 channel at 256 kb/s. With your new notebook, however, I get codec "0x0003 (IEEE FLOAT)", 16000 hz, 1 channel at 1024 kb/s - which leads to WAV files being 4 times the size.

For the small test set surely no issue, but when downloading the full audio data set, I would prefer not to quadruple the file size, if possible. ;-)
Is there a specific need to use this 4 times larger WAV format for your new notebook or can we use the 256 kb/s variant again?

For reference, to check the WAV audio format used, I use this little tool here:
https://www.headbands.com/gspot/v26x/index.htm

Again, thanks a lot for your efforts, David
Andreas

[StuartIanNaylor]

I guess it fits better as the chunk size is 0.080msec or 1280 samples @16k so = 25 chunks, 3 secs would have a remainder.
The whitespace around KW can be an equally important, but also to much with a short KW then you detect whitespace.
I have been playing with the /examples/detect_from_microphone.py
on a Pi02 but kept getting buffer underruns.
So swapped out to sounddevice with a callback.

# Copyright 2022 David Scripka. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Imports
import sounddevice as sd
import numpy as np
from openwakeword.model import Model
import argparse
import threading

# Parse input arguments
parser=argparse.ArgumentParser()
parser.add_argument(
    "--chunk_size",
    help="How much audio (in number of samples) to predict on at once",
    type=int,
    default=1280,
    required=False
)
parser.add_argument(
    "--model_path",
    help="The path of a specific model to load",
    type=str,
    default="",
    required=False
)
parser.add_argument(
    "--inference_framework",
    help="The inference framework to use (either 'onnx' or 'tflite'",
    type=str,
    default='tflite',
    required=False
)

args=parser.parse_args()

CHANNELS = 1
RATE = 16000
CHUNK = args.chunk_size
sd.default.samplerate = RATE
sd.default.channels = CHANNELS
sd.default.dtype= ('int16', 'int16')

def sd_callback(rec, frames, time, status):
    audio = np.frombuffer(rec, dtype=np.int16)
    prediction = owwModel.predict(audio)

    # Column titles
    n_spaces = 16
    output_string_header = """
        Model Name         | Score | Wakeword Status
        --------------------------------------
        """

    for mdl in owwModel.prediction_buffer.keys():
        # Add scores in formatted table
        scores = list(owwModel.prediction_buffer[mdl])
        curr_score = format(scores[-1], '.20f').replace("-", "")

        output_string_header += f"""{mdl}{" "*(n_spaces - len(mdl))}   | {curr_score[0:5]} | {"--"+" "*20 if scores[-1] <= 0.5 else "Wakeword Detected!"}
        """

    # Print results table
    print("\033[F"*(4*n_models+1))
    print(output_string_header, "                             ", end='\r')

# Load pre-trained openwakeword models
if args.model_path != "":
    owwModel = Model(wakeword_models=[args.model_path], inference_framework=args.inference_framework)
else:
    owwModel = Model(inference_framework=args.inference_framework)

n_models = len(owwModel.models.keys())

# Start streaming from microphone
with sd.InputStream(channels=CHANNELS,
                    samplerate=RATE,
                    blocksize=int(CHUNK),
                    callback=sd_callback):
    threading.Event().wait()

You get

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
        Model Name         | Score | Wakeword Status
        --------------------------------------
        alexa              | 0.000 | --
        hey_mycroft        | 0.000 | --
        hey_jarvis         | 0.000 | --
        hey_rhasspy        | 0.000 | --
        1_minute_timer     | 0.000 | --
        5_minute_timer     | 0.000 | --
        10_minute_timer    | 0.000 | --
        20_minute_timer    | 0.000 | --
        30_minute_timer    | 0.000 | --
        1_hour_timer       | 0.000 | --
        weather            | 0.000 | --

Alexa to hey_rhasspy seem to work really well, timers seem very hard to activate and weather not at all.

[dscripka]

@ab-tools thank you for taking a look, and for the feedback, this is very helpful!

The clip size (total_length in the training config file) is arbitrary now, and the user can change it to whatever length makes the most sense for their target wake word/phrase and the rest of the training script will update accordingly. I agree that for many wake words 2 seconds is sufficient, but for longer phrases 3-4 seconds is likely better.

And you are correct about the WAV format, that was a mistake! 16-bit PCM is the desired format, I forgot to make the conversion prior to saving. Full 64-bit float is not necessary and does take up too much space. I have updated the notebook accordingly for the PR.

As for 'full-scale' vs 'small-test training', the trade-offs here are difficult. Full downloads of all of the datasets mentioned will take tens of hours to days depending on download speed, require potentially hundreds of GB's of hard-drive space, and depending on hardware could take several days to process. Thus, I didn't want to make that the default behavior. The full datasets are available to download from the exact same links, and very minor changes to the example downloading/processing code. Additionally, depending on the needs of the target deployment environment, some datasets may not even be necessary (e.g., if the user does not intend for the model to handle background music, downloading and using the FMA dataset can be skipped entirely). But I take your point that some users will want to use as much data as possible when training; perhaps I can make a separate (optional) download script for this purpose.

[ab-tools]

Perfect, thanks for quickly fixing the WAV format, @dscripka!

Regarding the "full-scale" dataset my idea would be to create a common baseline for training:
You seem to have a lot of understanding/experience and your scripts here are a great starting point for other people to get into this - like myself.

When your script provides a clear baseline, e. g. also which concrete negative audio data should be used for augmentation - because this worked well in your experience - others can try with exactly the same set like yourself and we can collect our results together and perhaps can come up with an even better combination at the end. But this only works if we start with the same set as a baseline.

It's totally enough, of course, if you provide this "full-scale" training part in a separate script though.

By the way, as I understand there is no need to split between training and validation for negative audio augmentation samples, so I would suggest to simply download the FMA file manually and extracting/converting everything, instead of using this "rudraml/fma" dataset which split things into "training" and "validation".

Thanks again
Andreas