Ethos-u ML Evaluation kit examples

Clone this repo:
  1. 8c61c0a MLECO-2985 Adding Corstone-310 support by Kshitij Sisodia · 2 days ago main
  2. b76b855 Minor change to help build API libs independently. by Richard Burton · 2 days ago
  3. 937052d MLECO-3101: Minor changes for CMSIS pack by Kshitij Sisodia · 5 days ago
  4. 2c2cacd MLECO-3202: Reduce Profiler's heap usage by Richard Burton · 6 days ago
  5. 213a543 MLECO-3186: Each use case should same namespace convention as KWS and ASR by Liam Barry · 9 days ago

NOTE: The default branch for this repository is now the main branch, the master branch is deprecated, and will be removed after 22.05 release. Please ensure any changes are added to the main branch.

Arm® ML embedded evaluation kit

This repository is for building and deploying Machine Learning (ML) applications targeted for Arm® Cortex®-M and Arm® Ethos™-U NPU. To run evaluations using this software, we suggest using:

Overview of the evaluation kit

The purpose of this evaluation kit is to allow users to develop software and test the performance of the Ethos-U NPU and Cortex-M55 CPU. The Ethos-U NPU is a new class of machine learning (ML) processor, specifically designed to accelerate computation for ML workloads in constrained embedded and IoT devices. The product is optimized to execute mathematical operations efficiently that are commonly used in ML algorithms, such as convolutions or activation functions.

ML use cases

The evaluation kit adds value by providing ready to use ML applications for the embedded stack. As a result, you can experiment with the already developed software use cases and create your own applications for Cortex-M CPU and Ethos-U NPU. The example application at your disposal and the utilized models are listed in the table below.

ML applicationDescriptionNeural Network Model
Image classificationRecognize the presence of objects in a given imageMobilenet V2
Keyword spotting(KWS)Recognize the presence of a key word in a recordingMicroNet
Automated Speech Recognition(ASR)Transcribe words in a recordingWav2Letter
KWS and ASRUtilise Cortex-M and Ethos-U to transcribe words in a recording after a keyword was spottedMicroNet Wav2Letter
Anomaly DetectionDetecting abnormal behavior based on a sound recording of a machineMicroNet
Visual Wake WordRecognize if person is present in a given imageMicroNet
Noise ReductionRemove noise from audio while keeping speech intactRNNoise
Generic inference runnerCode block allowing you to develop your own use case for Ethos-U NPUYour custom model
Object detectionDetects and draws face bounding box in a given imageYolo Fastest

The above use cases implement end-to-end ML flow including data pre-processing and post-processing. They will allow you to investigate embedded software stack, evaluate performance of the networks running on Cortex-M55 CPU and Ethos-U NPU by displaying different performance metrics such as inference cycle count estimation and results of the network execution.

Software and hardware overview

The evaluation kit is based on the Arm® Corstone™-300 reference package. Arm® Corstone™-300 helps you build SoCs quickly on the Arm® Cortex™-M55 and Arm® Ethos™-U55 designs. Arm® Corstone™-300 design implementation is publicly available on an Arm MPS3 FPGA board, or as a Fixed Virtual Platform of the MPS3 development board.

The Ethos-U NPU software stack is described here.

All ML use cases, albeit illustrating a different application, have common code such as initializing the Hardware Abstraction Layer (HAL). The application common code can be run on x86 or Arm Cortex-M architecture thanks to the HAL. For the ML application-specific part, Google® TensorFlow™ Lite for Microcontrollers inference engine is used to schedule the neural networks models executions. TensorFlow Lite for Microcontrollers is integrated with the Ethos-U NPU driver and delegates execution of certain operators to the NPU or, if the neural network model operators are not supported on NPU, to the CPU. CMSIS-NN is used to optimise CPU workload execution with int8 data type. Common ML application functions will help you to focus on implementing logic of your custom ML use case: you can modify only the use case code and leave all other components unchanged. Supplied build system will discover new ML application code and automatically include it into compilation flow.

APIs

To run an ML application on the Cortex-M and Ethos-U NPU, please, follow these steps:

  1. Setup your environment by installing the required prerequisites.
  2. Generate an optimized neural network model for Ethos-U with a Vela compiler by following instructions here.
  3. Configure the build system.
  4. Compile the project with a make command.
  5. If using a FVP, launch the desired application on the FVP. If using the FPGA option, load the image on the FPGA and launch the application.

To get familiar with these steps, you can follow the quick start guide.

Note: The default flow assumes Arm® Ethos™-U55 NPU usage, configured to use 128 Multiply-Accumulate units and is sharing SRAM with the Arm® Cortex®-M55.

Ml embedded evaluation kit supports:

Ethos™-U NPUDefault MACs/ccOther MACs/cc supportedDefault Memory ModeOther Memory Modes supported
Ethos™-U5512832, 64, 256Shared_SramSram_Only
Ethos™-U65256512Dedicated_SramShared_Sram

For more information see Building.

For more details see full documentation:

Contribution guidelines

Contributions are only accepted under the following conditions:

  • The contribution have certified origin and give us your permission. To manage this process we use Developer Certificate of Origin (DCO) V1.1. To indicate that contributors agree to the the terms of the DCO, it's necessary "sign off" the contribution by adding a line with name and e-mail address to every git commit message:

    Signed-off-by: John Doe <john.doe@example.org>
    

    This can be done automatically by adding the -s option to your git commit command. You must use your real name, no pseudonyms or anonymous contributions are accepted.

  • You give permission according to the Apache License 2.0.

    In each source file, include the following copyright notice:

    /*
    * Copyright (c) <years additions were made to project> <your name>, Arm Limited. All rights reserved.
    * SPDX-License-Identifier: Apache-2.0
    *
    * Licensed under the Apache License, Version 2.0 (the "License");
    * you may not use this file except in compliance with the License.
    * You may obtain a copy of the License at
    *
    *     http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing, software
    * distributed under the License is distributed on an "AS IS" BASIS,
    * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    * See the License for the specific language governing permissions and
    * limitations under the License.
    */
    

Coding standards and guidelines

This repository follows a set of guidelines, best practices, programming styles and conventions, see:

Code Reviews

Contributions must go through code review. Code reviews are performed through the mlplatform.org Gerrit server. Contributors need to signup to this Gerrit server with their GitHub account credentials. In order to be merged a patch needs to:

  • get a "+1 Verified" from the pre-commit job.
  • get a "+2 Code-review" from a reviewer, it means the patch has the final approval.

Testing

Prior to submitting a patch for review please make sure that all build variants works and unit tests pass. Contributions go through testing at the continuous integration system. All builds, tests and checks must pass before a contribution gets merged to the main branch.

Communication

Please, if you want to start public discussion, raise any issues or questions related to this repository, use https://discuss.mlplatform.org/c/ml-embedded-evaluation-kit forum.

Licenses

The ML Embedded applications samples are provided under the Apache 2.0 license, see License Apache 2.0.

Application input data sample files are provided under their original license:

LicenceProvenience
Automatic Speech Recognition SamplesCreative Commons Attribution 4.0 International Public Licensehttp://www.openslr.org/12/
Image Classification SamplesCreative Commons Attribution 1.0https://www.pexels.com
Keyword Spotting SamplesCreative Commons Attribution 4.0 International Public Licensehttp://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
Keyword Spotting and Automatic Speech Recognition SamplesCreative Commons Attribution 4.0 International Public Licensehttp://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
Visual Wake Word SamplesCreative Commons Attribution 1.0https://www.pexels.com
Noise Reduction SamplesCreative Commons Attribution 4.0 International Public Licensehttps://datashare.ed.ac.uk/handle/10283/2791/
Object Detection SamplesCreative Commons Attribution 1.0https://www.pexels.com