| commit | 5ffbc94a5bb055512ef9252d05c4d9429c4c3c0d | [log] [tgz] |
|---|---|---|
| author | Eric Kunze <eric.kunze@arm.com> | Mon Nov 01 13:51:58 2021 -0700 |
| committer | Eric Kunze <eric.kunze@arm.com> | Wed Nov 03 13:03:39 2021 -0700 |
| tree | 516a349adc6b6ec7c617ec8a7944757a2025e108 | |
| parent | f3e016f7aedad5bc995932e75821f2eeb2ec45ea [diff] |
Remove draft tag for 0.23 release Update to the corresponding serialization library Change-Id: I323bf306d409b51bbf53447927953edd0fd79983 Signed-off-by: Eric Kunze <eric.kunze@arm.com>
The Tensor Operator Set Architecture (TOSA) Specification https://git.mlplatform.org/tosa/specification.git/ is a set of operators with defined accuracy and compatibility constraints that Arm expects to be implemented on its Neural Processing Units (NPUs). Most operators from the common ML frameworks (TensorFlow, PyTorch, etc) should be expressible in TOSA. TOSA is focused on inference, leaving training to the original frameworks.
The TOSA Reference Model package provides a reference implementation and testing infrastructure for TOSA. The reference model consumes a FlatBuffers serialization of the network subgraph generated by the TOSA Serialization Library, along with input tensors for placeholder nodes in NumPy format. By default, the model validates and evalutes the network subgraph, and writes out the resulting output tensors in NumPy format.
The TOSA Reference Model and testing suite requires the following tools:
The model includes the following git submodules:
The model is written using C++17 and has been primarily tested on Ubuntu x86_64 18.04 LTS Linux systems.
The testing infrastructure requires:
Check out the required git submodules with:
$ git submodule update --init --recursive
The TOSA Reference Model build can be prepared by creating makefiles using CMake:
$ mkdir -p build $ cd build $ cmake ..
Optionally, -DCMAKE_BUILD_MODE=Debug can be used on the cmake command to create a debug build. Next compile using make:
$ make
The resulting executable will be named: reference_model/tosa_reference_model. CMake only needs to be re-run if the build environment changes (e.g., new dependencies or source files). Code changes that do not affect these build rules can be rebuilt simply using make.
The inputs to the TOSA Reference Model consist of a FlatBuffers file containing the serialized subgraph, a JSON test descriptor that describes a sequence of name, shape and numpy file for each input and output tensor.
The JSON test descriptor must have the following field:
Note by default, all the files specified by "tosa_file", "ifm_file", "ofm_file" are relative to desc.json. This could be overwritten by -Cflatbuffer_dir=, if desired.
An example command is shown below:
$ ./build/reference_model/tosa_reference_model \ -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json
Instead of drive model by JSON test descriptor, user can also drive model with -Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file= options directly.
In case where -Ctest_desc= and other options are specified at the same time, JSON test descriptor will be initialized first. All other options (-Ctosa_file=, -Cifm_name=, -Cifm_file=, -Cofm_name=, -Cofm_file=) will overwrite whatever specified by JSON descriptor.
On a successful execution, the output tensors will be written in NumPy format into output tensors specified by "ofm_file".
For example, you can generate new output .npy by:
$ ./build/reference_model/tosa_reference_model \ -Ctest_desc=examples/test_add_1x4x4x4_f32/flatbuffer-tflite/desc.json -Cofm_file=out.npy
In this case, the "ofm_file" field in desc.json will be ignored, and the one specified by -Cofm_file= will be picked.
When using JSON-formatted FlatBuffers input (.json extension), the FlatBuffers schema file from the TOSA Serialization library must be specified using -Coperator_fbs=. When using the binary FlatBuffers format (.tosa), the schema is not necessary.
The TOSA Reference Model distribution contains several example networks with inputs and reference outputs generated by TensorFlow or TensorFlow Lite in the examples directory.
These examples can be run through the TOSA Reference model and should produce the equivalent TOSA-compliant reference output tensors. Please note that differences in floating-point ordering and rounding may cause small differences in output for floating-point tests and differences in quantized scaling between TensorFlow Lite and the TOSA Specification may cause differences in quantized integer tests.
The debugging facility can be enabled by setting a debug scope and debug level on the command line. For most purposes, the following flags will work: -dALL -lHIGH. Debug output can be directed to a file using the -o switch.
The TOSA Unit Test infrastruture builds and runs self-contained tests for implementations of the Tensor Operator Set Architecture (TOSA) Specification. These tools directly generate TOSA operators for verification of the TOSA reference model against existing frameworks or other operator implementations.
The test builder tool generates tests with random arguments and reference inputs for each TOSA operator. Currently, the test builder focuses on generating a wide range of legal arguments to each operator, but it also has limited support for generating tests with illegal arguments in order to make sure such usages are properly detected.
The unit tests are typically structured as a combination of input placeholder nodes, const nodes, and attributes feeding into a single TOSA operator. The unit tests use a Python copy of the FlatBuffers schema written by flatc to verif/tosa.
Each test has a JSON file which provides machine-readable metadata for the test, including the .tosa flatbuffer file, names, shapes, and NumPy filenames for each input and output tensor. There is also a boolean value for whether a failure is expected because the test is expected to trigger an invalid set of operands or attributes.
The test runner tool executes the unit tests on the TOSA Reference Model to generate reference output tensor values (for legal tests). The test runner is a modular tool which can be exended to run the same tests on additional tools or frameworks. The reference output NumPy files are generated by this step and can be programatically compared with output of other tools. to validate those tools.
The test builder is in verif/tosa_verif_build_tests.py. The builder generates test outputs in ./vtest/<operator_name>/ by default. To restrict test generation to particular regular expression wildcard, use the --filter argument. The tool can be run with no arguments to generate all tests.
Inputs and certain attributes are created using a random number generator, while others are exhaustive (within reasonable bounds) where the combinatorics allow exhaustive tests. The test generation is deterministic for a given random seed, but additional tests can be generated using --seed. As many corner-case error are often uncovered using creative tensor shapes, the random seed parameter will help get coverage of additional shapes.
Additional parameters on some operators can be found in the command line help.
The unit test running script takes self-contained unit tests from the builder and runs them on the reference model. Shell wildcards can be used to run more than one test at a time and tests can be run in parallel using the -j switch. For example, to run all of the add operator tests:
$ ./verif/tosa_verif_run_ref.py -t vtest/add/add* -j 8
The test runner is quiet by default, so running a large number of tests without any obvious errors will show no output while the tests are running. The -v switch will show the command being run in the background.
To enable debugging on the reference model, shortcut commands have been provided: --ref-debug=high and --ref-intermediates to turn on debugging and dump intermediate tensor values.
Additional Systems Under Test (SUTs), such as reference implementations of operators, full frameworks, etc, can be defined by extending the TosaTestRunner class. The SUTs can then be enabled by using the --sut-module flag.
The TOSA Reference Model and TOSA Unit Tests are licensed under Apache-2.0.