Turn Your Raspberry Pi Into a Smart AI Audio Classifier

Summary:

I am back with another Raspberry Pi automation project. This time I built a smart audio classifier utilizing YAMNet sound classification model and Adafruit digital MEMS microphones .The project can be useful for different applications including speech to text recognition, smart home security and industrial monitoring.

Hardware Setup:

 The following bill of materials is required:

•  Raspberry Pi (3/4/5)
•  Adafruit I2S MEMS Microphone Breakout - SPH0645LM4H (2 for stereo setup) 
• Flexible silicon jumper wires  

Wiring of I2S Mics to Raspberry Pi I2S interface 

Reference: https://learn.adafruit.com/adafruit-i2s-mems-microphone-breakout/raspberry-pi-wiring-test

 The following connections are made between the Raspberry Pi and the two digital mics:

• Mics 3V to Pi 3.3V
• Mics GND to Pi GND
• Mics BCLK to BCM 18 (pin 12)
• Mics DOUT to BCM 20 (pin 38)
• Mics LRCL to BCM 19 (pin 35)
• Left Mic SEL to Pi GND
• Right Mic SEL to Pi 3.3V

Software Configuration and Debugging:

After assembling the setup I ran some checks to make sure the Pi can record audio and the audio quality is acceptable. To enable I2S interface on the Pi, the following line needs to be added to the Pi config.txt (or /boot/firmware/config.txt). A reboot is required for the new setting to take effect:

 dtoverlay=googlevoicehat-soundcard # enable i2s overlay

The above will assign specific GPIO pins for the I2S clocks (usually Pin 18 for BCLK, Pin 19 for LRCLK) and data lines (Pin 20 for Data In). To confirm a new audio recording device is registered in ALSA (Advanced Linux Sound Architecture) inspect the output of the following command:

 arecord -l

 It should print something like below. Note that the card number could be different:

**** List of CAPTURE Hardware Devices ****
card 2: sndrpigooglevoi [snd_rpi_googlevoicehat_soundcar], device 0: Google voiceHAT SoundCard HiFi voicehat-hifi-0 [Google voiceHAT SoundCard HiFi voicehat-hifi-0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0 

To check the parameters associated with the audio device and its driver, I ran the following command:

 arecord -D hw:2,0 --dump-hw-params

The important output from this command tell the us the default configured sample rate, number of bits and channels

To record a stereo audio signal and save it into a wav file, I ran the following command:

arecord -D plughw:2 -c2 -r 48000 -f S32_LE -t wav -V stereo -v file_stereo.wav

Here is the exact breakdown of every argument in that line:

• arecord The core ALSA command-line utility used for recording sound.

• -D plughw:2 (Device) Tells ALSA which specific hardware device to listen to.

• -c2 (Channels) Sets the number of recording channels to 2 (Stereo)

• -r 48000 (Rate) Sets the sample rate to 48,000 Hz (48 kHz)

• -f S32_LE (Format) Sets the digital audio format to Signed 32-bit Little Endian

• -t wav (Type) Tells ALSA to package the recorded raw data into a standard .wav file container.

• -V stereo (VU Meter) Displays a visual volume meter (a VU meter)

• -v (Verbose) Turns on verbose mode.

• file_stereo.wav The name of the file that will be saved to your Raspberry Pi's current directory when you stop the recording

In parallel, I attached a logic analyzer to inspect the I2S bit clock, left-right clock and data signals. From the capture below bit clock frequency is measured to be approx. 3.1 MHz which is close to microphone datasheet number. The word select signal frequency matches the desired sample rate of 48 kHz and the data line pulses are visible in both phases of the word select signal meaning both channels data are being transmitted from the left and right microphones respectively.

Capture of I2S (BLCK, LRCLK and DOUT) signals

Running Audio Classification with Python:

On software side, I used LiteRT Python package (successor to TensorFlow Lite) for running the YAMNet machine learning model on the Raspberry Pi. I built a minimal python script to load .wav audio files and run inference to verify that model can closely predict the audio events. You can find the python script and audio files I tested with in this Github repo.

I plotted the spectrogram of two audio files for testing to visualize what the front end of YAMNet mode will use for inferring the audio class. The model expects the audio frames to come in size of 15600 samples with sampling rate of 16 kHz. Only mono data is supported. 

Mel Spectrograms of two audio samples to be classified (left: miaow_16k.wav | right: speech_whistling2.wav)

Running the inference on the two samples yielded the following results. The Python script divided the audio files into required size for the model (i.e 0.96 sec) then inference ran individually on each frame. The class with highest confidence score was then qualified and printed along with the time it took to execute the inference in Python. After all audio frames are processed, the overall class is selected based on mean score of all detected classes. In summary, After listening the to the audio samples myself, the predicted classes are to a good extent accurate.

Wearables Audio Classification Setup:

Putting it all together, I wanted to see how good is the model with predicitng audio events based on  recordings from wearable devices (e.g headphones, hearing aids). It is of great potential that AI can augment human in the loop in manual testing where someone needs to have the golden ears to judge the quality of the audio. This is just a simple experiment however and the goal is not to have something very professional and application ready. 

To couple my headphones to the MEMS microphones, I created a simplified acoustic coupler using a 5ml plastic jar. The jar has two holes one against the microphone sound part with some space. The second is against the headphone audio output. With that the microphone should be able to capture the audio ouput of the headphones without much distraction from the surrounding acoustic environment. To play audio into my headset I used Bluetooth on the Raspberry Pi to connect and play the two audio samples I tested with the model in previous section. By comparing the model results with original vs. recorded samples we get a feeling on how well the overall solution work together (i.e. hardware recording + audio classification with machine learning)

Comparing the audio classes detected on recorded audio signal and comparing them to original audio yield close results overall. Optimizing the recording setup and post-processing audio after recording to remove noise artifacts and slight DC offset would improve the results.

Front and side views of headphones attached to DIY coupler with microphones also attached at the bottom

  

 

YAMNet predictions on original and recorded data for miaow_16k sample

 

YAMNet predictions on original and recorded data for speech_whistling2