Preface
This book is about TinyML, a fast-growing field at the unique intersection of machine learning (ML) and embedded systems to make AI work with extremely low-powered devices, such as microcontrollers.
TinyML is an exciting field full of opportunities. With a small budget, we can give life to objects that interact with the world around us smartly and transform the way we live for the better. However, this field can be hard to approach if we come from an ML background with little familiarity with embedded systems, such as microcontrollers. Therefore, this book aims to dispel these barriers and make TinyML also accessible to developers with no embedded programming experience through practical examples. Each chapter will be a self-contained project to learn how to use some of the technologies at the heart of TinyML, interface with electronic components such as sensors, and deploy ML models on memory-constrained devices.
TinyML Cookbook starts with a practical introduction to this multidisciplinary field to get you up to speed with some of the fundamentals for deploying intelligent applications on Arduino Nano 33 BLE Sense and Raspberry Pi Pico. As you progress, you'll tackle various problems that you may encounter while prototyping microcontrollers, such as controlling the LED state with GPIO and a push-button and supplying power to microcontrollers with batteries. After that, you'll cover recipes relating to temperature, humidity, and the three V (voice, vision, and vibration) sensors to gain the necessary skills to implement end-to-end smart applications in different scenarios. Then, you'll learn best practices to build tiny models for memory-constrained microcontrollers. Finally, you'll explore two of the most recent technologies, microTVM and microNPU, which will help you step up your TinyML game.
By the end of this book, you'll be well versed in best practices and ML frameworks to develop ML apps easily on microcontrollers and have a clear understanding of the key aspects to consider during the development phase.
Who this book is for
This book is for ML developers/engineers interested in developing ML applications on microcontrollers through practical examples quickly. The book will help you expand your knowledge of the revolution of TinyML by building end-to-end smart projects with real-world data sensors on the Arduino Nano 33 BLE Sense and the Raspberry Pi Pico. Basic familiarity with C/C++, Python programming, and a command-line interface (CLI) is required. However, no prior knowledge of microcontrollers is necessary.
What this book covers
Chapter 1, Getting Started with TinyML, provides an overview of TinyML, presenting the opportunities and challenges to bring ML on extremely low-power microcontrollers. This chapter focuses on the fundamental elements behind ML, power consumption, and microcontrollers that make TinyML unique and different from conventional ML in the cloud, desktop, or even smartphones.
Chapter 2, Prototyping with Microcontrollers, presents concise and straightforward recipes to deal with the relevant microcontroller programming basics. We will deal with code debugging and how to transmit data to the Arduino serial monitor. After that, we will discover how to program GPIO peripherals with the ARM Mbed API and use the breadboard to connect external components, such as LEDs and push-buttons. In the end, we will see how to power the Arduino Nano 33 BLE Sense and the Raspberry Pi Pico with batteries.
Chapter 3, Building a Weather Station with TensorFlow Lite for Microcontrollers, guides you through all the development stages of a TensorFlow-based application for microcontrollers and teaches you how to acquire temperature and humidity sensor data. The application developed in the chapter is an ML-based weather station for snow forecasts.
In the first part, we will focus on dataset preparation by acquiring historical weather data from WorldWeatherOnline. After that, we will present the relevant basics to train and test a model with TensorFlow. In the end, we will deploy the model on the Arduino Nano 33 BLE Sense and the Raspberry Pi Pico with TensorFlow Lite for Microcontrollers.
Chapter 4, Voice Controlling LEDs with Edge Impulse, shows how to develop an end-to-end keyword spotting (KWS) application with Edge Impulse and get familiar with audio data acquisition and analog-to-digital (ADC) peripherals. The application considered for this chapter voice controls the LED emitting color (red, green, and blue) and the number of times to make it blink (one, two, and three).
In the first part, we will focus on the dataset preparation, showing how to acquire audio data with a mobile phone. After that, we will design a model using the mel-frequency cepstral coefficient (MFCC) features and optimize the performance with EON Tuner. In the end, we will finalize the KWS application on the Arduino Nano 33 BLE Sense and the Raspberry Pi Pico.
Chapter 5, Indoor Scene Classification with TensorFlow Lite for Microcontrollers and the Arduino Nano, aims to show you how to apply transfer learning with TensorFlow and get familiar with the best practices to use a camera module with a microcontroller. For the purpose of this chapter, we will develop an application to recognize indoor environments with the Arduino Nano 33 BLE Sense and the OV7670 camera module.
In the first part, we will see how to acquire images from the OV7670 camera module. After that, we will focus on the model design, applying transfer learning with Keras to recognize kitchen and bathroom rooms. In the end, we will deploy the quantized TensorFlow Lite model on the Arduino Nano 33 BLE Sense with the help of TensorFlow Lite for Microcontrollers.
Chapter 6, Building a Gesture-Based Interface for YouTube Playback, aims to develop an end-to-end gesture recognition application with Edge Impulse and the Raspberry Pi Pico to get acquainted with inertial sensors, teach you how to use I2C peripherals, and write a multithreading application in Arm Mbed OS.
In the first part, we will collect the accelerometer data through the Edge Impulse data forwarder to prepare the dataset. After that, we will design a model using features in the frequency domain to recognize three gestures. In the end, we will deploy the application on the Raspberry Pi Pico and implement a Python program with the PyAutoGUI library to build a touchless interface for YouTube video playback.
Chapter 7, Running a Tiny CIFAR-10 Model on a Virtual Platform with the Zephyr OS, provides best practices to build tiny models for memory-constrained microcontrollers. In this chapter, we will be designing a model for the CIFAR-10 image classification dataset on a virtual Arm Cortex-M3-based microcontroller.
In the first part, we will install Zephyr, the primary framework used in this chapter to accomplish our task. After that, we will design a tiny quantized CIFAR-10 model with TensorFlow. This model will fit on a microcontroller with only 256 KB of program memory and 64 KB of RAM. In the end, we will build an image classification application with TensorFlow Lite for Microcontrollers and the Zephyr OS and run it on a virtual platform using Quick Emulator (QEMU).
Chapter 8, Toward the Next TinyML Generation with microNPU, helps familiarize you with microNPU, a new class of processors for ML workloads on edge devices. In this chapter, we will be running a quantized CIFAR-10 model on a virtual Arm Ethos-U55 microNPU with the help of TVM.
In the first part, we will learn how the Arm Ethos-U55 microNPU works and install the software dependencies to build and run the model on the Arm Corstone-300 fixed virtual platform. After that, we will use the TVM compiler to convert a pretrained TensorFlow Lite model to C code. In the end, we will show how to compile and deploy the code generated by TVM into Arm Corstone-300 to perform inference with the Arm Ethos-U55 microNPU.
To get the most out of this book
You will need a computer (either a laptop or desktop) with an x86-64 architecture and at least one USB port for programming the Arduino Nano 33 BLE Sense and the Raspberry Pi Pico microcontroller boards. For the first six chapters, you can use Ubuntu 18.04 (or later) or Windows (for example, Windows 10) as an OS. However, you will need Ubuntu 18.04 (or later) for Chapter 7, Running a Tiny CIFAR-10 Model on a Virtual Platform with the Zephyr OS, and Chapter 8, Toward the Next TinyML Generation with microNPU.
The only software prerequisites for your computer are:
- Python (Python 3.7 recommended)
- Text editor (for example, gedit on Ubuntu)
- Media player (for example, VLC)
- Image viewer (for example, the default app in Ubuntu or Windows 10)
- Web browser (for example, Google Chrome)
During our TinyML journey, we will require different software tools to cover ML development and embedded programming. Thanks to Arduino, Edge Impulse, and Google, these tools will be in the cloud, browser-based, and with a free plan for our usage.
Arduino Nano 33 BLE Sense and Raspberry Pi Pico programs will be developed directly in the web browser with the Arduino Web Editor (https://create.arduino.cc). However, the Arduino Web Editor has a limit of 200 seconds of compilation time per day. Therefore, you may consider upgrading to any paid plan or using the free local Arduino IDE (https://www.arduino.cc/en/software) to get unlimited compilation time. If you are interested in the free local Arduino IDE, we have provided on GitHub (https://github.com/PacktPublishing/TinyML-Cookbook/blob/main/Docs/setup_local_arduino_ide.md) the instructions to set it up.
The following table summarizes the hardware devices and software tools covered in each chapter:
The projects may require sensors and additional electronic components to build realistic TinyML prototypes and experience the complete development workflow. All the components are listed at the beginning of each chapter and in the README.md file on GitHub (https://github.com/PacktPublishing/TinyML-Cookbook). Since you will build real electronic circuits, we require an electronic components kit that includes at least a solderless breadboard, colored LEDs, resistors, push-buttons, and jumper wires. Don't worry if you are an electronics beginner. You will learn more about these components in the first two chapters of this book. Furthermore, we have prepared a beginner shopping list on GitHub so you know precisely what to buy: https://github.com/PacktPublishing/TinyML-Cookbook/blob/main/Docs/shopping_list.md.
If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.
Download the example code files
You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/TinyML-Cookbook. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801814973_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Enter the ~/project_npu folder and create three folders, named binaries, src, and sw_libs."
A block of code is set as follows:
export PATH=~/project_npu/binaries/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4:$PATH
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
[default]
exten => s,1,Dial(Zap/1|30)
exten => s,2,Voicemail(u100)
exten => s,102,Voicemail(b100)
exten => i,1,Voicemail(s0)
Any command-line input or output is written as follows:
$ cd ~/project_npu
$ mkdir binaries
$ mkdir src
Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Click on Corstone-300 Ecosystem FVPs and then click on the Download Linux button."
Tips or Important Notes
Appear like this.
Sections
In this book, you will find several headings that appear frequently (Getting ready, How to do it..., and There's more...).
To give clear instructions on how to complete a recipe, use these sections as follows:
Getting ready
This section tells you what to expect in the recipe and describes how to set up any software or any preliminary settings required for the recipe.
How to do it…
This section contains the steps required to follow the recipe.
There's more…
This section consists of additional information about the recipe in order to make you more knowledgeable about the recipe.
Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Share Your Thoughts
Once you've read TinyML Cookbook, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.
Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.