During my time with Natura Bolivia I was tasked with one main task:

Create a end to end machine learning system for bioacoustics

This included:

1. Unifying and organizing all audio data
2. Creating an interface for labeling data
3. Integrate systems for training models and evaluating predictions
4. Build a system for reporting on the results from labeling and predictions

Now this is no small task, certainly not something a single developer splitting their time between other project should be perusing for a 6 month contract. But...

Luckily much of the needed infrastructure to make this work already exists, so despite daunting requirements I was able to build a system for Natura that meets all of these requirements while being robust and business ready.

What this is about

This article talks about generally how I was able to make this project a reality, some of the important lessons learned along the way, and should act as a general blueprint to help others understand how they can do the same. It will also feature a number of photos I'm fond of from my adventures throughout Bolivia :)

The central point: Label Studio

Label Studio stands out as a truly incredible piece of open source software. Out of the box it gives you:

1. Cloud storage for your data
2. A machine learning interface for integrating model results
3. Flexible labeling interface definitions

This is about 50% of what people really want, it's labels on your data in a very verbose format and a website that's reliable, quick and flexible. But it only manages to check 1 of the 4 boxes on our list. But importantly it works with every other key part of the system because of it's great integration with many other tools.

Downsides to Label Studio

Label studio is not cool, it's

Lesson 1: Metadata is important

Without metadata, you really cannot do much with any discoveries made.

This should be a fairly obvious point (in retrospect for me haha) but there is a lot of interesting metadata that can make anything you collect or report massively more interesting.

So for us, we did end up making a little CLI tool for handling our metadata. It's flow looked like:

1. Read metadata from audio files and generate three spreadsheet tables:
   1. One containing recording metadata for individual recordings
   2. One containing recordings that we failed to read
   3. One containing a row for each recording device

Item c here is important because the user fills out most of the information. This is where things like location, time of deployments, and observations about the sensor become relevant. This information is vital for understanding recording contents and extracting any useful data.

2. Create label studio ready tasks; Label studios term for a atomic unit for labeling. This includes:
   1. Validating user input data
   2. Splitting audio recordings for desired length for a given task
   3. Generating a new directory containing all data, ready for upload
3. Upload data to a new storage bucket. Each round of monitoring was decided to be given it's own bucket for organization and ease of separation for creating labeling projects. Optionally, this included running a specified google cloud job on the uploaded bucket. In our case, we defined a simple job using the Perch model zoo by google to generate some embeddings that will be used for downstream transfer learning.

Lesson 2: Learn Actively

About 75% of our labels were classified as possible for machine learning to perform decently at (generally if over 100 example of the species were present in our data). This means that by using active learning, a method encouraged by our machine learning model of choice: Perch we can help find some proportion of that 75% without needing to search.

This can be a huge productivity gain for employees. They can focus on the 41% (Common Species) and only audit the 38% (No Species). While this is far from complete automation it means:

1. Labeling efforts are more targeted, if resources are limited
2. Labeling efforts are reduced if a full project can be filled

Weaknesses to Active Learning

Active learning is inherently biased this means that the items it will sample will inherently skew your perception of the underlying data. There is a real chance that, even if Perch is good, it will not be able to distinguish between certain species, the difference simply does not exist in the embedding it generates. I have had multiple colleges with this exact problem.

So, for us it was a very important step to define a protocol for how labeling was conducted:

Mainly this was a set of criteria we would look for from a set of random samples, to make sure we were accurately representing the species that did exist in the data, even if models inherently would miss some.

Like all software engineering projects I was expecting to have to tamper some expectations on performance, but I found it surprising how many colleges were hoping for a solution that would work out of the box. The thing I like about machine learning vs AI is that it's much easier to evaluate the areas it will be weak in. There is a much smaller latent space, and a usually a much more directed goal that can help defend the use cases for a particular model.

Wait, isn't active learning a enterprise feature?

In Label Studio, they have a feature that can automatically decide the next task to be labeled based off active model predictions. This is really neat, but not all that required. For us, we found that doing the following worked great and gave us a lot of flexibility in the choice of what we labeled:

1. Train a new model:
   This would be trained on all new labels that have been created by a user, allowing us to incorporate the new data. This can be done from directly within the Label Studio interface
2. Create predictions for all tasks:
   For a given project in label studio, you can pull down the full set of embedding, generate a new set of predictions using the new model, and upload those back to the platform
3. Filter based on those predictions:
   Label Studios filtering is decent for this. It can let you search for:
   1. A specific label appearing in a certain task
   2. A aggregate confidence for the model prediction at that location

This generally for us managed to get the job done without being over-complicated

Wait, isn't that costly?

Because we are doing transfer learning the two models we decided to use were simply:

1. A logistic regression model
   This model is extremely fast, harder to over/under-fit and works well enough to get decent results for active learning. Importantly it is fast enough that it did not cost much to run repeatedly throughout our project. I found myself running it once every couple of days once I had gone through more of it's predictions.
2. A MLP model
   This model is still lightweight by comparison to anything you hear about in machine learning currently. Because of this and our reliance on the upstream Perch model we were able to get away with not spending very much on compute for our results. This freed up our team who is not super technical to play around with the model parameters within experiment systems like Aim. Depending on the size of the project, doing parameter optimization could have even been an option without incurring crazy costs.

Lesson 3: Standing on the shoulders of your colleagues

Another nice feature label studio has out of the box supports not only exporting your data, but then re-importing those as completed tasks in other projects. Meaning that the active learning harness Label Studio gives you, and their projects in general are capable of referencing historical data.

It's worth noting that when Label Studio exports a project, it will only export labels that have been annotated by a human. No model predictions, or tasks with no annotations will be exported.

For us it did impact the design decisions around how embeddings were stored.

Each embedding was stored alongside the audio files inside of individual buckets for a project. This way we won't leave any embeddings laying around, won't need a database to store them, and can download them fairly easily. Because each of our survey rounds was split without being very large, downloading these indivially was fast enough inside the same google regions that we had no issues. For a truly large project I imagine a dedicated vector database would have served this better.

Along with easy access to historical data to kickstart active learning and historical references in projects it gives our team a lot of flexibility in what they can import, it can allow for easy segregation between regions for different projects, and allow you to experiment with including certain sets of data for training to compose larger projects easily.

Lesson 4: Experiment!

Aim is a tool that let's you create and run experiments with model trainings. It's basically a free and open source version of what Weights and Biases does. This allowed our teams to make models they could actually trust, and really easily dive into the different performance metric differences of model runs. But it has an issue running in google cloud:

Aim uses a shared file for it's database, and that does not tend to play well with google cloud run. There, you don't have shared storage or even persistent storage. So we had to have a different, rather hacky solution:
Because we only ever had one machine learning model running at a time (limited through the maximum number of allowed concurrent jobs in google cloud run) we simply would...

1. Copy the data from google cloud storage down into the cloud run job
2. Run our training and report metrics directly into the files using Aims SDK
3. Then simply upload those back into a shared google cloud storage that the Aim dashboard could read from

This worked for us, but not very well. For a more long term solution I would recommend a docker compose service that runs with persistent memory, or to use a service like Weights and Biases that is less tied to persistent files with locks.

Conclusions

I think my biggest takeaway is that there exists free tools to build excellent machine learning infrastructure.

But, there is a issue: on order to get beyond basic labeling you need dedicated IT staff. I think the truth is that label studio on it's own gets you 90% of the way there, while being flexible. It is central to much of this process and gives you solid standards for implementing the rest of it. But that last 10% is critically important for operations like this to succeed. I would love to sit down at some point and make a more complete template that would cover most of the basics that label studio does not do on it's own:

1. Model experminets
2. Cloud Infra ready model harness
3. Results reporting
4. Project infrastructure

That way, teams could get a little closer to this goal. I would even love to make a set of model harnesses for specifically bio-acoustics. But I'm not sure if people would be interested in that really.

Hope you enjoyed :) - Carter