The Tensor Operator Set Architecture (TOSA) Specification https://git.mlplatform.org/tosa/specification.git/ is a set of operators with defined accuracy and compatibility constraints that Arm expects to be implemented on its Neural Processing Units (NPUs). Most operators from the common ML frameworks (TensorFlow, PyTorch, etc) should be expressible in TOSA. TOSA is focused on inference, leaving training to the original frameworks.
The TOSA Reference Model package provides a reference implementation and testing infrastructure for TOSA. The reference model consumes a FlatBuffers serialization of the network subgraph generated by the TOSA Serialization Library, along with input tensors for placeholder nodes in NumPy format. By default, the model validates and evaluates the network subgraph, and writes out the resulting output tensors in NumPy format.
This package also contains a conformance test suite that can be created and then used to validate an implementation of TOSA.
The TOSA Reference Model and testing suite requires the following tools:
The model includes the following git submodules:
The model is written using C++17 and has been primarily tested on Ubuntu x86_64 20.04 LTS Linux systems.
The testing infrastructure requires:
Check out the required git submodules with:
git submodule update --init --recursive
The TOSA Reference Model repository has branches (major.minor) and tags (major.minor.patch) that correspond to each TOSA version. The main branch is used as the active development branch for the next version.
Perform a check-out of a specific version before compilation or installation of the test infrastructure by using:
git checkout --recurse-submodules VERSION
Where VERSION can be for example: v0.23 or v0.23.0
The TOSA Reference Model build can be prepared by creating makefiles using CMake:
mkdir -p build cd build cmake ..
Optionally, -DCMAKE_BUILD_TYPE=Debug can be used on the cmake command to create a debug build. Next compile using make:
make
The resulting executable will be named: reference_model/tosa_reference_model. This executable can be disabled with -DBUILD_TOSA_REFERENCE_MODEL_EXECUTABLE=NO. A static library will also be generated by default as follows: reference_model/libtosa_reference_model_lib.a. To make this a shared library (.so) add the following option -DBUILD_SHARED_LIBS=YES.
CMake only needs to be re-run if the build environment changes (e.g., new dependencies or source files). Code changes that do not affect these build rules can be rebuilt simply using make.
The inputs to the TOSA Reference Model consist of a FlatBuffers file containing the serialized subgraph, a JSON test descriptor that describes a sequence of name, shape and NumPy file for each input and output tensor.
The JSON test descriptor must have the following field:
Note by default, all the files specified by "tosa_file", "ifm_file", "ofm_file" are relative to desc.json. This could be overwritten by --flatbuffer_dir=, if desired.
An example command is shown below:
./build/reference_model/tosa_reference_model \ --test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
Instead of drive model by JSON test descriptor, user can also drive model with --tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file= options directly.
In case where --test_desc= and other options are specified at the same time, JSON test descriptor will be initialized first. All other options (--tosa_file=, --ifm_name=, --ifm_file=, --ofm_name=, --ofm_file=) will overwrite whatever specified by JSON descriptor.
On a successful execution, the output tensors will be written in NumPy format into output tensors specified by "ofm_file".
For example, you can generate new output .npy by:
./build/reference_model/tosa_reference_model \ --test_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json \ --ofm_file=out.npy
In this case, the "ofm_file" field in desc.json will be ignored, and the one specified by --ofm_file= will be picked.
When using JSON-formatted FlatBuffers input (.json extension), the FlatBuffers schema file from the TOSA Serialization library must be specified using --operator_fbs=. When using the binary FlatBuffers format (.tosa), the schema is not necessary.
The TOSA Reference Model distribution contains several example networks with inputs and reference outputs generated by TensorFlow or TensorFlow Lite in the examples directory.
These examples can be run through the TOSA Reference model and should produce the equivalent TOSA-compliant reference output tensors. Please note that differences in floating-point ordering and rounding may cause small differences in output for floating-point tests and differences in quantized scaling between TensorFlow Lite and the TOSA Specification may cause differences in quantized integer tests.
As an alternative to the executable described above, the model_runner.h is provided which can be used to invoke the TOSA Reference Model easily within C++. A sample of this class and how it can used can be found in model_runner_simple_sample.cpp. This sample can be compiled by adding -DBUILD_MODEL_RUNNER_SAMPLE=YES to the CMake command and executed by running ./build/reference_model/model_runner_sample.
Unit test are generated by default for the ModelRunner. This executable can be disabled by adding-DBUILD_TOSA_REFERENCE_MODEL_TESTS=NO to the CMake command. This executable can be run using ./build/reference_model/unit_tests and requires the submodule doctest.
The debugging facility can be enabled by setting a debug scope and debug level on the command line. For most purposes, the following flags will work: -d ALL -l HIGH. Debug output can be directed to a file using the -o switch.
The TOSA Unit Test infrastructure can create and run self-contained tests for implementations of the Tensor Operator Set Architecture (TOSA) Specification. These test tools generate TOSA operators for verification of the TOSA reference model against existing frameworks or other operator implementations.
A conformance suite generator is included in the TOSA Unit Test infrastructure to provide coverage validation of an implementation of TOSA. Please see the TOSA Conformance Generator section below to find out how to create these.
The test building tools by default generate positive tests with random arguments and reference inputs for each TOSA operator. Positive tests are expected to run without error and will produce a result. The test builder can also generate negative tests for all the ERROR_IF conditions within the TOSA Specification by using the --test-type options. Negative tests may contain invalid arguments or inputs and are expected to be loaded and fail without producing a result. Other errors or unpredictable results are expected to be handled in a system dependent way and are not tested by the test builder tool.
The unit tests are typically structured as a combination of input placeholder nodes, const nodes, and attributes feeding into a single TOSA operator. The unit tests use a Python copy of the FlatBuffers schema written by flatc to verify TOSA.
Each test has a JSON file which provides machine-readable metadata for the test, including the TOSA flatbuffer file, names, shapes, and NumPy filenames for each input and output tensor. There is also a boolean value for whether a failure is expected because the test is expected to trigger an invalid set of operands or attributes. Extra metadata may be included such as tags for filtering, data generation or compliance validation.
The test runner tool can execute the unit tests on the TOSA Reference Model to generate reference output tensor values (for positive tests). The test runner is a modular tool which can be extended to run the same tests on additional tools or frameworks - such a tool or framework is called a System Under Test (SUT). The reference output NumPy files generated by this step will be compared with output of SUTs to validate them.
The test infrastructure needs installing before being used. It is recommended to create a python virtual environment and then install the TOSA Unit Test infrastructure from the root of the reference model:
pip install .
When installing without a python virtual environment, use the pip option --user to install it for the current user only.
If you need an editable environment for adding a System Under Test, then use one of the following scripts (due to a deficiency in python setup tools). Create a python virtual environment and then source:
# For bash . scripts/py-dev-env.bash # For other shells source scripts/py-dev-env.sh
This script enables creation of part or all of the TOSA conformance tests, to enable running of these tests against the reference model or implementation.
The Conformance Tests comprise of a wide coverage of tests for all TOSA operators created by using the Unit Test Builder.
The generation of conformance tests is dependent on the FlatBuffers command flatc - please follow the section on the FlatBuffers compiler below.
These are the main script options for controlling the types of tests produced:
--profile - controls the TOSA profile, tosa-bi - for base inference integer tests, tosa-mi - for the main inference floating point tests, or all - for both.--test-type - selects positive, negative or both types of test.--output-type - selects the output file type between json, binary or both. The default - json - converts NumPy data files and flatbuffer files into JSON for ease in viewing and comparison.An example to create the TOSA operator unit tests for ADD and SUB:
tosa_verif_conformance_generator \ --profile tosa-bi \ --ref-model-path reference_model/build/reference_model/tosa_reference_model \ --operators add sub
The above command will create some temporary files in a conformance_build directory, but will output the conformance unit tests into a conformance directory.
If you have a different build directory for the reference model, you may have to supply one or more of the following options for path locations:
--ref-model-path - path to the tosa_reference_model executable.--schema-path or --operator-fbs - path to the TOSA flatbuffer schema file (tosa.fbs)--flatc-path - path to the FlatBuffers compiler flatcThis is an example using the default locations:
tosa_verif_conformance_generator \ --ref-model-path reference_model/build/reference_model/tosa_reference_model \ --flatc-path reference_model/build/thirdparty/serialization_lib/third_party/flatbuffers/flatc \ --schema-path reference_model/thirdparty/serialization_lib/schema/tosa.fbs \ --operators abs
This next example will create all the conformance tests, using different temporary build and output directories:
tosa_verif_conformance_generator \ --ref-model-path reference_model/build/reference_model/tosa_reference_model \ --build-directory tmp_build \ --output-directory conf_tests
Use the Unit Test Runner to run the conformance tests on the reference model, or extend the runner to also test against your implementation (System Under Test) - see those sections below for more information.
Unit tests are mostly single operator tests, that can be created and used to check an implementation. These tests can be used to investigate TOSA or complement the conformance tests.
The test builder is invoked by tosa_verif_build_tests. The builder generates test outputs in ./vtest/<operator_name>/ by default. To restrict test generation to particular regular expression wildcard, use the --filter argument. The tool can be run with no arguments to generate all tests.
Inputs and certain attributes are created using a random number generator, while others are exhaustive (within reasonable bounds) where the combinatorics allow exhaustive tests. The test generation is deterministic for a given random seed, but additional tests can be generated using --seed. As many corner-case error are often uncovered using creative tensor shapes, the random seed parameter will help get coverage of additional shapes.
By default only the positive tests will be produced, use the argument --test-type both to build positive and negative tests.
Additional parameters on some operators can be found in the command line help.
The unit test running script takes self-contained unit tests from the builder and runs them on the reference model or on a System Under Test.
The --test or -t option is used to specify a directory containing a test. Shell wildcards can be used to run more than one test at a time. Tests will be run sequentially by default, but you may control how many tests are run in parallel using the --jobs or -j switch.
For example, to run all of the TOSA add operator tests on the reference model, eight at a time:
tosa_verif_run_tests -t vtest/add/add* -j 8
The default location that is used for the reference model is reference_model/build/reference_model/tosa_reference_model, use the option --ref-model-path if you run this from a different location.
You can also supply a list of tests in a file, one per line, using the --test-list or -T option. Alternatively use --recursive or -r to run all the tests that can be found in the specified -t directory.
Finally you can choose the type of test to run - positive, negative or both (default) -using the test-type option. To only run the positive tests:
tosa_verif_run_tests --test-type positive -t vtest/*/*
The test runner is reasonably quiet by default, so running a large number of tests without any obvious errors will show only 1 line of output per test completion. The -v switch will show the commands being run in the background, and display any errors from the reference model.
To enable debugging on the reference model, shortcut commands have been provided: --ref-debug=high and --ref-intermediates to turn on debugging and dump intermediate tensor values.
Additional Systems Under Test (SUTs), such as reference implementations of operators, full frameworks, and hardware implementations can be tested by the test runner.
To do this you need to define an SUT module by extending the TosaTestRunner class found in verif/runner/tosa_test_runner.py, and then supplying this to the TOSA Test Runner.
With each test is a desc.json file that contains input and output filename information which is read and supplied to the TosaTestRunner class.
A TOSA System Under Test will need to be able to read the following input files:
Utilize the TosaTestRunner class to convert these test artifacts into another format before giving them to your SUT.
For positive tests usually at least one results file should be produced by your SUT containing the resulting tensor in NumPy format. The expected filenames are supplied in the desc.json information.
Your python class extending the TosaTestRunner class must contain:
__init__(self,...) function that calls the super() class function.runTestGraph(self) function that invokes your SUT and then translates the return code into a TosaTestRunner.TosaGraphResult. It returns this result and an optional error message.Examples of implementations can be found:
verif/runner/tosa_refmodel_sut_run.py - the reference modelverif/tests/tosa_mock_sut_run.py - mock version for testingThere is a helper function run_sh_command in verif/runner/run_command.py that provides a robust way of calling shell commands from python that can be used to invoke your SUT.
The SUT can then be supplied to the test runner by using the --sut-module flag, the following invokes the reference model as the SUT (default behaviour when not supplied):
tosa_verif_run_tests --sut-module runner.tosa_refmodel_sut_run -t TEST
You can also pass arguments to your SUT, for example this will pass an argument called ARGUMENT with a value of VALUE to the mysut.tosa_mysut_sut_run module:
tosa_run_tests --sut-module mysut.tosa_mysut_sut_run \ --sut-module-args mysut.tosa_mysut_sut_run:ARGUMENT=VALUE \ -t TEST
You can repeat this switch multiple times to pass multiple different arguments.
For an example of how to read these arguments in your SUT module, please see the tosa_mock_sut_run.py file.
Included in the TOSA Unit Test infrastructure are scripts to enable the creation of TOSA unit tests for example frameworks. Included at the moment is support for TensorFlow and TensorFlow Lite.
Installation (via pip install) of the following python package is required to generate the tests:
tensorflowA built copy of the tensorflow framework from source is required to compile the tests to TOSA - see the online documentation https://www.tensorflow.org/install/source on how to do this. The following tools are used from this build:
tensorflow/basel-bin/tensorflow/compiler/mlir/lite/flatbuffer_translatetensorflow/basel-bin/tensorflow/compiler/mlir/tf-optThe command to generate the unit test framework models:
tosa_verif_framework_generator -o tests
Next to convert these models to TOSA and then run them on the reference model:
tosa_verif_framework_compiler_runner \ --tf-base-dir tensorflow \ --tools-base-dir reference_model \ --recursive \ --test tests
Included in this repository are some support utilities used by the test runner:
tosa_json2numpy - converts from JSON format to NumPy array or the reverse operation.tosa_json2fbbin - converts from JSON flatbuffer format to flatbuffer binary format or the reverse operation. This is dependent on the FlatBuffers command flatc - see the section on the FlatBuffers compiler below.tosa_verif_result_check - compares two results files.tosa_convert2conformance - converts a unit test into a conformance suitable test.tosa_schemavalidation - validates a test desc.json against the schema.Please see the respective --help of each utility for more information on using them standalone.
The FlatBuffers compiler tool (flatc) is only needed if you want to use tosa_json2fbbin to convert the TOSA flatbuffer binary test files (.tosa) to JSON or from JSON to binary. It is best to use the flatbuffer version that comes with the reference model. After following the reference model compilation instructions, you can build the FlatBuffers tool using:
# After compiling the reference model (in the build directory) cd thirdparty/serialization_lib/third_party/flatbuffers make flatc
The TOSA Reference Model and TOSA Unit Tests are licensed under Apache-2.0.
Copyright (c) 2020-2023 Arm Limited.