This section provides useful details on how the Machine Learning use-cases of the evaluation kit use the system memory.
Although the guidance provided here is concerning the Arm® Corstone™-300 system, it is fairly generic and is applicable for other platforms too. The Arm® Corstone™-300 is composed of both the Arm® Cortex™-M55 and the Arm® Ethos™-U NPU. The memory map for the Arm® Cortex™-M55 core can be found in the Appendix.
The Arm® Ethos™-U NPU interacts with the system through two AXI interfaces. The first one, is envisaged to be the higher-bandwidth, lower-latency, interface. In a typical system, this is wired to an SRAM as it is required to service frequent read and write traffic.
The second interface is expected to have a higher-latency, lower-bandwidth characteristic, and is typically wired to a flash device servicing read-only traffic. In this configuration, the Arm® Cortex™-M55 CPU and Arm® Ethos™-U NPU read the contents of the neural network model, or the .tflite file, from the flash memory region. With the Arm® Ethos™-U NPU requesting these read transactions over its second AXI bus.
The input and output tensors, along with any intermediate computation buffers, are placed on SRAM. Therefore, both the Arm® Cortex™-M55 CPU and Arm® Ethos™-U NPU would be reading, or writing, to this region when running an inference. The Arm® Ethos™-U NPU requests these Read and Write transactions over the first AXI bus.
Embedded target platforms supported have a description in the form of CMake files. These files have definitions that describe the memory regions and the peripheral base addresses.
See the example for Arm® Corstone™-300 description file corstone-sse-300.cmake. For the discussion on this page, it is useful to note the following definitions:
set(ISRAM0_SIZE "0x00200000" CACHE STRING "ISRAM0 size: 2 MiB")
set(ISRAM1_SIZE "0x00200000" CACHE STRING "ISRAM1 size: 2 MiB")
...
# SRAM size reserved for activation buffers
math(EXPR ACTIVATION_BUF_SRAM_SZ "${ISRAM0_SIZE} + ${ISRAM1_SIZE}" OUTPUT_FORMAT HEXADECIMAL)
This will set ACTIVATION_BUF_SRAM_SZ to be 4 MiB for Arm® Corstone™-300 target platform. As mentioned in the comments within the file, this size is directly linked to the size mentioned in the linker scripts, and therefore, it should not be changed without corresponding changes in the linker script too. For example, a snippet from the scatter file for Corstone™-300 shows:
;-----------------------------------------------------
; SSE-300's internal SRAM of 4MiB - reserved for
; activation buffers.
; This region should have 3 cycle read latency from
; both Cortex-M55 and Ethos-U NPU
;-----------------------------------------------------
isram.bin 0x31000000 UNINIT ALIGN 16 0x00400000
{
...
}
If the usable size of the internal SRAM was to be increased/decreased, the change should be made in both the linker script as well as the corstone-300.cmake definition.
Other than the obvious link between the linker script and the target profile description in CMake files, there are other parameters linked to what the reserved space for activation buffers is. These are:
The file default_vela.ini contains a parameter called arena_cache_size under Shared_Sram memory mode. For example:
[Memory_Mode.Shared_Sram] const_mem_area=Axi1 arena_mem_area=Axi0 cache_mem_area=Axi0 arena_cache_size=4194304
This size of 4 MiB here is provided here to allow the default vela optimisation process to use this size as a hint for the available SRAM size for use by the CPU and the NPU.
In every usecase.cmake file (present within each use case's source directory), there is a parameter called ${use_case}_ACTIVATION_BUF_SZ set to a fixed value by default. This default value should be less than the ACTIVATION_BUF_SRAM_SZ if the activation buffer needs to be reserved in the target platform's SRAM region.
When the neural network model is compiled with Vela, a summary report that includes memory usage is generated. For example, compiling the keyword spotting model ds_cnn_clustered_int8 with the Vela command:
vela \ --accelerator-config=ethos-u55-128 \ --optimise Performance \ --config scripts/vela/default_vela.ini --memory-mode=Shared_Sram --system-config=Ethos_U55_High_End_Embedded ds_cnn_clustered_int8.tflite
It produces, among others, the following output:
Total SRAM used 146.31 KiB Total Off-chip Flash used 452.42 KiB
The Total SRAM used here shows the required memory to store the tensor arena for the TensorFlow Lite Micro framework. This is the amount of memory required to store the input, output, and intermediate buffers. In the preceding example, the tensor arena requires 146.31 KiB of available SRAM.
Note: Vela can only estimate the SRAM required for graph execution. It has no way of estimating the memory used by internal structures from TensorFlow Lite Micro framework.
Therefore, we recommend that you top this memory size by at least 2KiB. We also recommend that you also carve out the tensor arena of this size, and then place it on the SRAM of the target system.
The Total Off-chip Flash parameter indicates the minimum amount of flash required to store the neural network model. In the preceding example, the system must have a minimum of 452.42 KiB of available flash memory to store the .tflite file contents.
Note: The Arm® Corstone™-300 system uses the DDR region as a flash memory. The timing adapter sets up the AXI bus that is wired to the DDR to mimic both bandwidth and latency characteristics of a flash memory device.
The preceding example outlines a typical configuration for Ethos-U55 NPU, and this corresponds to the default Vela memory mode setting. Evaluation kit supports all the Ethos-U NPU memory modes:
| Ethos™-U NPU | Default Memory Mode | Other Memory Modes supported |
|---|---|---|
| Ethos™-U55 | Shared_Sram | Sram_Only |
| Ethos™-U65 | Dedicated_Sram | Shared_Sram |
For further information on the default settings, please refer to: default_vela.ini.
For Ethos-U55 NPU, the system SRAM can also be used to store the neural network model along with the tensor arena. Vela supports optimizing the model for this configuration with its Sram_Only memory mode. Although the Vela settings for this configurations suggests that only AXI0 bus is used, when compiling the model a warning is generated, for example:
vela \ --accelerator-config=ethos-u55-128 \ --optimise Performance \ --config scripts/vela/default_vela.ini --memory-mode=Sram_Only --system-config=Ethos_U55_High_End_Embedded ds_cnn_clustered_int8.tflite Info: Changing const_mem_area from Sram to OnChipFlash. This will use the same characteristics as Sram.
This means that the neural network model is always placed in the flash region. In this case, timing adapters for the AXI buses are set the same values to mimic both bandwidth and latency characteristics of a SRAM memory device. See Ethos-U55 NPU timing adapter default configuration.
For script snippets, please refer to: Memory constraints.
Note:
- The
Shared_Srammemory mode represents the default configuration.- The
Dedicated_Srammemory mode is only applicable for the Arm® Ethos™-U65.- The
Sram_onlymemory mode is only applicable for the Arm® Ethos™-U55.
The evaluation kit uses the name activation buffer for the tensor arena in the TensorFlow Lite Micro framework. Every use-case application has a corresponding <use_case_name>_ACTIVATION_BUF_SZ parameter that governs the maximum available size of the activation buffer for that particular use-case.
The linker script is set up to place this memory region in SRAM for Ethos-U55 and in flash for Ethos-U65. Every target platform has a profile definition in the form of a CMake file.
For further information and an example, please refer to: Corstone-300 profile.
The parameter ACTIVATION_BUF_SRAM_SZ defines the maximum SRAM size available for the platform. This is propagated through the build system.
The neural network model is always placed in the flash region (even in case of Sram_Only memory mode as mentioned earlier).
The following numbers have been obtained from Vela for the Shared_Sram memory mode, along with the SRAM and flash memory requirements for the different use-cases of the evaluation kit.
Note: The SRAM usage does not include memory used by TensorFlow Lite Micro and must be topped up as explained under Total SRAM used.
Keyword spotting model requires
Image classification model requires
Automated speech recognition requires
Both the MPS3 Fixed Virtual Platform (FVP) and the MPS3 FPGA platform share the linker script for Arm® Corstone™-300 design. The CMake configuration parameter TARGET_SUBSYSTEM sets the design, which is described in: build options.
The memory map exposed by this design is located in the Appendix, and can be used as a reference when editing the linker script. This is useful to make sure that the region boundaries are respected. The snippet from the scatter file is as follows:
;--------------------------------------------------------- ; First load region (ITCM) ;--------------------------------------------------------- LOAD_REGION_0 0x00000000 0x00080000 { ... ;----------------------------------------------------- ; SSE-300's internal SRAM of 4MiB - reserved for ; activation buffers. ; This region should have 3 cycle read latency from ; both Cortex-M55 and Ethos-U NPU ;----------------------------------------------------- isram.bin 0x31000000 UNINIT ALIGN 16 0x00400000 { ; Cache area (if used) *.o (.bss.NoInit.ethos_u_cache) ; activation buffers a.k.a tensor arena when memory mode sram only *.o (.bss.NoInit.activation_buf_sram) } } ;--------------------------------------------------------- ; Second load region (DDR) ;--------------------------------------------------------- LOAD_REGION_1 0x70000000 0x02000000 { ;----------------------------------------------------- ; 32 MiB of DDR space for neural network model, ; input vectors and labels. If the activation buffer ; size required by the network is bigger than the ; SRAM size available, it is accommodated here. ;----------------------------------------------------- ddr.bin 0x70000000 ALIGN 16 0x02000000 { ; nn model's baked in input matrices *.o (ifm) ; nn model's default space *.o (nn_model) ; labels *.o (labels) ; activation buffers a.k.a tensor arena when memory mode dedicated sram *.o (activation_buf_dram) } ... }
Note: With Arm® Corstone™-300 FPGA and FVP implementations, only the BRAM, internal SRAM, and DDR memory regions are accessible to the Arm® Ethos™-U NPU block.
In the preceding snippet, the internal SRAM region memory is utilized by the activation buffers with a limit of 4MiB. If used by a Vela optimized neural network model, then the Arm® Ethos™-U NPU writes to this block frequently.
A bigger region of memory for storing the neural network model is placed in the DDR, or flash, region under LOAD_REGION_1. The two load regions are necessary as the motherboard configuration controller of the MPS3 limits the load size at address 0x00000000 to 1MiB. This has implications on how the application is deployed on MPS3, as explained under the following section: Deployment on MPS3.
The next section of the documentation covers: Troubleshooting.