IVGCVSW-5571 Expose the TfLite Delegate to the TfLite python API
* Implemented external delegate adaptor interface for TfLite
* Activated armnn logging for delegate
* Added logging info to indicate if gpu tuning is turned on
* Added pytests to ensure functionality of the external delegate adaptor
* Included the delegate directory into doxygen
* Added documentation on how to use the external delegate in python
Signed-off-by: Finn Williams <Finn.Williams@arm.com>
Signed-off-by: Jan Eilers <jan.eilers@arm.com>
Change-Id: Id3b4588fb0b9ac7e3f47ba2c19feead7beb58e18
diff --git a/delegate/CMakeLists.txt b/delegate/CMakeLists.txt
index 8383722..495c1e3 100644
--- a/delegate/CMakeLists.txt
+++ b/delegate/CMakeLists.txt
@@ -15,6 +15,7 @@
include/armnn_delegate.hpp
include/DelegateOptions.hpp
src/armnn_delegate.cpp
+ src/armnn_external_delegate.cpp
src/DelegateOptions.cpp
src/Activation.hpp
src/ArgMinMax.hpp
diff --git a/delegate/IntegrateDelegateIntoPython.md b/delegate/IntegrateDelegateIntoPython.md
new file mode 100644
index 0000000..69a5ca0
--- /dev/null
+++ b/delegate/IntegrateDelegateIntoPython.md
@@ -0,0 +1,121 @@
+# Integrate the TfLite delegate into a python script
+If you have built the TfLite delegate as a separate dynamic library then this tutorial will show you how you can
+integrate it in TfLite to run models using python.
+
+Here is an example python script showing how to do this. In this script we are making use of the
+[external adaptor](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate)
+tool of TfLite that allows you to load delegates at runtime.
+```python
+import numpy as np
+import tflite_runtime.interpreter as tflite
+
+# Load TFLite model and allocate tensors.
+# (if you are using the complete tensorflow package you can find load_delegate in tf.experimental.load_delegate)
+armnn_delegate = tflite.load_delegate( library="<your-armnn-build-dir>/delegate/libarmnnDelegate.so",
+ options={"backends": "CpuAcc,GpuAcc,CpuRef", "logging-severity":"info"})
+# Delegates/Executes all operations supported by ArmNN to/with ArmNN
+interpreter = tflite.Interpreter(model_path="<your-armnn-repo-dir>/delegate/python/test/test_data/mock_model.tflite",
+ experimental_delegates=[armnn_delegate])
+interpreter.allocate_tensors()
+
+# Get input and output tensors.
+input_details = interpreter.get_input_details()
+output_details = interpreter.get_output_details()
+
+# Test model on random input data.
+input_shape = input_details[0]['shape']
+input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
+interpreter.set_tensor(input_details[0]['index'], input_data)
+
+interpreter.invoke()
+
+# Print out result
+output_data = interpreter.get_tensor(output_details[0]['index'])
+print(output_data)
+```
+
+# Prepare the environment
+Pre-requisites:
+ * Dynamically build ArmNN Delegate library
+ * python3 (Depends on TfLite version)
+ * virtualenv
+ * numpy (Depends on TfLite version)
+ * tflite_runtime (>=2.0, depends on ArmNN Delegate)
+
+If you haven't built the delegate yet then take a look at the [build guide](BuildBuideNative.md).
+
+We recommend creating a virtual environment for this tutorial. For the following code to work python3 is needed. Please
+also check the documentation of the TfLite version you want to use. There might be additional prerequisites for the python
+version.
+```bash
+# Install python3 (We ended up with python3.5.3) and virtualenv
+sudo apt-get install python3-pip
+sudo pip3 install virtualenv
+
+# create a virtual environment
+cd your/tutorial/dir
+# creates a directory myenv at the current location
+virtualenv -p python3 myenv
+# activate the environment
+source myenv/bin/activate
+```
+
+Now that the environment is active we can install additional packages we need for our example script. As you can see
+in the python script at the start of this page, this tutorial uses the `tflite_runtime` rather than the whole tensorflow
+package. The `tflite_runtime` is a package that wraps the TfLite Interpreter. Therefore it can only be used to run inferences of
+TfLite models. But since ArmNN is only an inference engine itself this is a perfect match. The
+`tflite_runtime` is also much smaller than the whole tensorflow package and better suited to run models on
+mobile and embedded devices.
+
+At the time of writing, there are no packages of either `tensorflow` or `tflite_runtime` available on `pypi` that
+are built for an arm architecture. That means installing them using `pip` on your development board is currently not
+possible. The TfLite [website](https://www.tensorflow.org/lite/guide/python) points you at prebuilt `tflite_runtime`
+packages. However, that limits you to specific TfLite and Python versions. For this reason we will build the
+`tflite_runtime` from source.
+
+You will have downloaded the tensorflow repository in order to build the ArmNN delegate. In there you can find further
+instructions on how to build the `tflite_runtime` under `tensorflow/lite/tools/pip_package/README.md`. This tutorial
+uses bazel to build it natively but there are scripts for cross-compilation available as well.
+```bash
+# Add the directory where bazel is built to your PATH so that the script can find it
+PATH=$PATH:your/build/dir/bazel/output
+# Run the following script to build tflite_runtime natively.
+tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
+```
+The execution of the script creates a `.whl` file which can be used by `pip` to install the TfLite Runtime package.
+The build-script produces some output in which you can find the location where the `.whl` file was created. Then all that is
+left to do is to install all necessary python packages with `pip`.
+```bash
+pip install tensorflow/lite/tools/pip_package/gen/tflite_pip/python3/dist/tflite_runtime-2.3.1-py3-none-any.whl numpy
+```
+
+Your virtual environment is now all setup. Copy the final python script into a python file e.g.
+`ExternalDelegatePythonTutorial.py`. Modify the python script above and replace `<your-armnn-build-dir>` and
+`<your-armnn-repo-dir>` with the directories you have set up. If you've been using the [native build guide](BuildGuideNative.md)
+this will be `$BASEDIR/armnn/build` and `$BASEDIR/armnn`.
+
+Finally, execute the script:
+```bash
+python ExternalDelegatePythonTutorial.py
+```
+The output should look similar to this:
+```bash
+Info: ArmNN v23.0.0
+
+Info: Initialization time: 0.56 ms
+
+INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
+[[ 12 123 16 12 11 14 20 16 20 12]]
+Info: Shutdown time: 0.28 ms
+```
+
+For more details on what kind of options you can pass to the armnn delegate please check
+[armnn_delegate_adaptor.cpp](src/armnn_external_delegate.cpp).
+
+You can also test the functionality of the external delegate adaptor by running some unit tests:
+```bash
+pip install pytest
+cd armnn/delegate/python/test
+# You can deselect tests that require backends that your hardware doesn't support using markers e.g. `-m "not GpuAccTest`
+pytest --delegate-dir="<your-armnn-build-dir>/armnn/delegate/libarmnnDelegate.so" -m "not GpuAccTest"
+```
diff --git a/delegate/include/DelegateOptions.hpp b/delegate/include/DelegateOptions.hpp
index daf2015..6058061 100644
--- a/delegate/include/DelegateOptions.hpp
+++ b/delegate/include/DelegateOptions.hpp
@@ -6,6 +6,8 @@
#pragma once
#include <armnn/ArmNN.hpp>
+#include <armnn/Logging.hpp>
+#include <armnn/Optional.hpp>
#include <set>
#include <string>
@@ -17,10 +19,13 @@
class DelegateOptions
{
public:
- DelegateOptions(armnn::Compute computeDevice, const std::vector<armnn::BackendOptions>& backendOptions = {});
+ DelegateOptions(armnn::Compute computeDevice,
+ const std::vector<armnn::BackendOptions>& backendOptions = {},
+ armnn::Optional<armnn::LogSeverity> logSeverityLevel = armnn::EmptyOptional());
DelegateOptions(const std::vector<armnn::BackendId>& backends,
- const std::vector<armnn::BackendOptions>& backendOptions = {});
+ const std::vector<armnn::BackendOptions>& backendOptions = {},
+ armnn::Optional<armnn::LogSeverity> logSeverityLevel = armnn::EmptyOptional());
const std::vector<armnn::BackendId>& GetBackends() const { return m_Backends; }
@@ -28,6 +33,18 @@
const std::vector<armnn::BackendOptions>& GetBackendOptions() const { return m_BackendOptions; }
+ /// Appends a backend option to the list of backend options
+ void AddBackendOption(const armnn::BackendOptions& option) { m_BackendOptions.push_back(option); }
+
+ /// Sets the severity level for logging within ArmNN that will be used on creation of the delegate
+ void SetLoggingSeverity(const armnn::LogSeverity& level) { m_LoggingSeverity = level; }
+ void SetLoggingSeverity(const std::string& level) { m_LoggingSeverity = armnn::StringToLogLevel(level); }
+
+ /// Returns the severity level for logging within ArmNN
+ armnn::LogSeverity GetLoggingSeverity() { return m_LoggingSeverity.value(); }
+
+ bool IsLoggingEnabled() { return m_LoggingSeverity.has_value(); }
+
private:
/// Which backend to run Delegate on.
/// Examples of possible values are: CpuRef, CpuAcc, GpuAcc.
@@ -52,6 +69,9 @@
/// "TuningFile" : string [filenameString]
/// "KernelProfilingEnabled" : bool [true | false]
std::vector<armnn::BackendOptions> m_BackendOptions;
+
+ /// Severity level for logging within ArmNN that will be used on creation of the delegate
+ armnn::Optional<armnn::LogSeverity> m_LoggingSeverity;
};
} // namespace armnnDelegate
diff --git a/delegate/python/test/conftest.py b/delegate/python/test/conftest.py
new file mode 100644
index 0000000..780a049
--- /dev/null
+++ b/delegate/python/test/conftest.py
@@ -0,0 +1,30 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+import pytest
+import os
+
+
+@pytest.fixture(scope="module")
+def test_data_folder(request):
+ """
+ This fixture returns path to the folder with the shared test resources
+ """
+ return str(os.path.join(request.fspath.dirname, "test_data"))
+
+
+def pytest_addoption(parser):
+ """
+ Adds the program option 'delegate-dir' to pytest
+ """
+ parser.addoption("--delegate-dir",
+ action="append",
+ help="Directory of the armnn tflite delegate library",
+ required=True)
+
+
+def pytest_generate_tests(metafunc):
+ """
+ Makes the program option 'delegate-dir' available to all tests as a function fixture
+ """
+ if "delegate_dir" in metafunc.fixturenames:
+ metafunc.parametrize("delegate_dir", metafunc.config.getoption("delegate_dir"))
\ No newline at end of file
diff --git a/delegate/python/test/pytest.ini b/delegate/python/test/pytest.ini
new file mode 100644
index 0000000..719af3c
--- /dev/null
+++ b/delegate/python/test/pytest.ini
@@ -0,0 +1,9 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+[pytest]
+addopts = --strict-markers
+markers =
+ CpuRefTest: marks tests that require the CpuRef backend
+ CpuAccTest: marks tests that require the CpuAcc backend
+ GpuAccTest: marks tests that require the GpuAcc backend
\ No newline at end of file
diff --git a/delegate/python/test/test_data/mock_model.tflite b/delegate/python/test/test_data/mock_model.tflite
new file mode 100644
index 0000000..0b8944d
--- /dev/null
+++ b/delegate/python/test/test_data/mock_model.tflite
Binary files differ
diff --git a/delegate/python/test/test_external_delegate.py b/delegate/python/test/test_external_delegate.py
new file mode 100644
index 0000000..995de8c
--- /dev/null
+++ b/delegate/python/test/test_external_delegate.py
@@ -0,0 +1,65 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+import pytest
+import tflite_runtime.interpreter as tflite
+import os
+from utils import run_mock_model
+
+
+def test_external_delegate_unknown_options(delegate_dir):
+ print(delegate_dir)
+ with pytest.raises(ValueError):
+ tflite.load_delegate(
+ delegate_dir,
+ options={"wrong": "wrong"})
+
+
+def test_external_delegate_options_multiple_backends(delegate_dir):
+ tflite.load_delegate(
+ delegate_dir,
+ options={"backends": "GpuAcc,CpuAcc,CpuRef,Unknown"})
+
+
+@pytest.mark.GpuAccTest
+def test_external_delegate_options_gpu_tuning(delegate_dir, test_data_folder, tmp_path):
+
+ tuning_file = os.path.join(str(tmp_path), "test_gpu.tuning")
+ # cleanup previous test run if necessary
+ if os.path.exists(tuning_file):
+ os.remove(tuning_file)
+
+ # with tuning level 2 a tuning file should be created
+ armnn_delegate = tflite.load_delegate(
+ delegate_dir,
+ options={
+ "backends": "GpuAcc",
+ "gpu-tuning-level": "2",
+ "gpu-tuning-file": tuning_file,
+ "logging-severity": "info"})
+
+ run_mock_model(armnn_delegate, test_data_folder)
+
+ # destroy delegate, otherwise tuning file won't be written to file
+ armnn_delegate.__del__()
+ assert (os.path.exists(tuning_file))
+
+ # if no tuning level is provided it defaults to 0 which means it will use the tuning parameters from a tuning
+ # file if one is provided
+ armnn_delegate2 = tflite.load_delegate(
+ delegate_dir,
+ options={
+ "backends": "GpuAcc",
+ "gpu-tuning-file": tuning_file,
+ "logging-severity": "info"})
+
+ run_mock_model(armnn_delegate2, test_data_folder)
+
+ # cleanup
+ os.remove(tuning_file)
+
+def test_external_delegate_options_wrong_logging_level(delegate_dir):
+ with pytest.raises(ValueError):
+ tflite.load_delegate(
+ delegate_dir,
+ options={"logging-severity": "wrong"})
diff --git a/delegate/python/test/utils.py b/delegate/python/test/utils.py
new file mode 100644
index 0000000..3adc24f
--- /dev/null
+++ b/delegate/python/test/utils.py
@@ -0,0 +1,24 @@
+# Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+import tflite_runtime.interpreter as tflite
+import numpy as np
+import os
+
+
+def run_mock_model(delegate, test_data_folder):
+ model_path = os.path.join(test_data_folder, 'mock_model.tflite')
+ interpreter = tflite.Interpreter(model_path=model_path,
+ experimental_delegates=[delegate])
+ interpreter.allocate_tensors()
+
+ # Get input and output tensors.
+ input_details = interpreter.get_input_details()
+ output_details = interpreter.get_output_details()
+
+ # Test model on random input data.
+ input_shape = input_details[0]['shape']
+ input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
+ interpreter.set_tensor(input_details[0]['index'], input_data)
+
+ interpreter.invoke()
\ No newline at end of file
diff --git a/delegate/src/DelegateOptions.cpp b/delegate/src/DelegateOptions.cpp
index af78685..3ec2d20 100644
--- a/delegate/src/DelegateOptions.cpp
+++ b/delegate/src/DelegateOptions.cpp
@@ -9,14 +9,16 @@
{
DelegateOptions::DelegateOptions(armnn::Compute computeDevice,
- const std::vector<armnn::BackendOptions>& backendOptions)
- : m_Backends({computeDevice}), m_BackendOptions(backendOptions)
+ const std::vector<armnn::BackendOptions>& backendOptions,
+ const armnn::Optional<armnn::LogSeverity> logSeverityLevel)
+ : m_Backends({computeDevice}), m_BackendOptions(backendOptions), m_LoggingSeverity(logSeverityLevel)
{
}
DelegateOptions::DelegateOptions(const std::vector<armnn::BackendId>& backends,
- const std::vector<armnn::BackendOptions>& backendOptions)
- : m_Backends(backends), m_BackendOptions(backendOptions)
+ const std::vector<armnn::BackendOptions>& backendOptions,
+ const armnn::Optional<armnn::LogSeverity> logSeverityLevel)
+ : m_Backends(backends), m_BackendOptions(backendOptions), m_LoggingSeverity(logSeverityLevel)
{
}
diff --git a/delegate/src/armnn_delegate.cpp b/delegate/src/armnn_delegate.cpp
index 5139adb..6250a5f 100644
--- a/delegate/src/armnn_delegate.cpp
+++ b/delegate/src/armnn_delegate.cpp
@@ -120,6 +120,12 @@
: m_Runtime(nullptr, nullptr),
m_Options(std::move(options))
{
+ // Configures logging for ARMNN
+ if (options.IsLoggingEnabled())
+ {
+ armnn::ConfigureLogging(true, true, options.GetLoggingSeverity());
+ }
+
// Create ArmNN Runtime
armnn::IRuntime::CreationOptions runtimeOptions;
diff --git a/delegate/src/armnn_external_delegate.cpp b/delegate/src/armnn_external_delegate.cpp
new file mode 100644
index 0000000..53b1725
--- /dev/null
+++ b/delegate/src/armnn_external_delegate.cpp
@@ -0,0 +1,148 @@
+//
+// Copyright © 2020 Arm Ltd and Contributors. All rights reserved.
+// SPDX-License-Identifier: MIT
+//
+#include "armnn_delegate.hpp"
+#include <armnn/Logging.hpp>
+
+#include <iostream>
+#include <tensorflow/lite/minimal_logging.h>
+
+namespace tflite
+{
+
+/**
+ * This file defines two symbols that need to be exported to use the TFLite external delegate provider. This is a plugin
+ * that can be used for fast integration of delegates into benchmark tests and other tools. It allows loading of
+ * a dynamic delegate library at runtime.
+ *
+ * The external delegate also has Tensorflow Lite Python bindings. Therefore the dynamic external delegate
+ * can be directly used with Tensorflow Lite Python APIs.
+ *
+ * See tensorflow/lite/delegates/external for details or visit the tensorflow guide
+ * [here](https://www.tensorflow.org/lite/performance/implementing_delegate#option_2_leverage_external_delegate)
+ */
+
+extern "C"
+{
+std::vector<std::string> gpu_options {"gpu-tuning-level",
+ "gpu-tuning-file",
+ "gpu-kernel-profiling-enabled"};
+
+
+/**
+ * Create an ArmNN delegate plugin
+ *
+ * Available options:
+ *
+ * Option key: "backends" \n
+ * Possible values: ["EthosNPU"/"GpuAcc"/"CpuAcc"/"CpuRef"] \n
+ * Descriptions: A comma separated list without whitespaces of
+ * backends which should be used for execution. Falls
+ * back to next backend in list if previous doesn't
+ * provide support for operation. e.g. "GpuAcc,CpuAcc"
+ *
+ * Option key: "logging-severity" \n
+ * Possible values: ["trace"/"debug"/"info"/"warning"/"error"/"fatal"] \n
+ * Description: Sets the logging severity level for ArmNN. Logging
+ * is turned off if this option is not provided.
+ *
+ * Option key: "gpu-tuning-level" \n
+ * Possible values: ["0"/"1"/"2"/"3"] \n
+ * Description: 0=UseOnly(default), 1=RapidTuning, 2=NormalTuning,
+ * 3=ExhaustiveTuning. Requires option gpu-tuning-file.
+ * 1,2 and 3 will create a tuning-file, 0 will apply the
+ * tunings from an existing file
+ *
+ * Option key: "gpu-tuning-file" \n
+ * Possible values: [filenameString] \n
+ * Description: File name for the tuning file.
+ *
+ * Option key: "gpu-kernel-profiling-enabled" \n
+ * Possible values: ["true"/"false"] \n
+ * Description: Enables GPU kernel profiling
+ *
+ *
+ * @param[in] option_keys Delegate option names
+ * @param[in] options_values Delegate option values
+ * @param[in] num_options Number of delegate options
+ * @param[in,out] report_error Error callback function
+ *
+ * @return An ArmNN delegate if it succeeds else NULL
+ */
+TfLiteDelegate* tflite_plugin_create_delegate(char** options_keys,
+ char** options_values,
+ size_t num_options,
+ void (*report_error)(const char*))
+{
+ // Returning null indicates an error during delegate creation so we initialize with that
+ TfLiteDelegate* delegate = nullptr;
+ try
+ {
+ // (Initializes with CpuRef backend)
+ armnnDelegate::DelegateOptions options = armnnDelegate::TfLiteArmnnDelegateOptionsDefault();
+ for (size_t i = 0; i < num_options; ++i)
+ {
+ // Process backends
+ if (std::string(options_keys[i]) == std::string("backends"))
+ {
+ // The backend option is a comma separated string of backendIDs that needs to be split
+ std::vector<armnn::BackendId> backends;
+ char* pch;
+ pch = strtok(options_values[i],",");
+ while (pch != NULL)
+ {
+ backends.push_back(pch);
+ pch = strtok (NULL, ",");
+ }
+ options.SetBackends(backends);
+ }
+ // Process logging level
+ else if (std::string(options_keys[i]) == std::string("logging-severity"))
+ {
+ options.SetLoggingSeverity(options_values[i]);
+ }
+ // Process GPU backend options
+ else if (std::string(options_keys[i]) == std::string("gpu-tuning-level"))
+ {
+ armnn::BackendOptions option("GpuAcc", {{"TuningLevel", atoi(options_values[i])}});
+ options.AddBackendOption(option);
+ }
+ else if (std::string(options_keys[i]) == std::string("gpu-tuning-file"))
+ {
+ armnn::BackendOptions option("GpuAcc", {{"TuningFile", std::string(options_values[i])}});
+ options.AddBackendOption(option);
+ }
+ else if (std::string(options_keys[i]) == std::string("gpu-kernel-profiling-enabled"))
+ {
+ armnn::BackendOptions option("GpuAcc", {{"KernelProfilingEnabled", (*options_values[i] != '0')}});
+ options.AddBackendOption(option);
+ }
+ else
+ {
+ throw armnn::Exception("Unknown option for the ArmNN Delegate given: " + std::string(options_keys[i]));
+ }
+ }
+ delegate = TfLiteArmnnDelegateCreate(options);
+ }
+ catch (const std::exception& ex)
+ {
+ if(report_error)
+ {
+ report_error(ex.what());
+ }
+ }
+ return delegate;
+}
+
+/** Destroy a given delegate plugin
+ *
+ * @param[in] delegate Delegate to destruct
+ */
+void tflite_plugin_destroy_delegate(TfLiteDelegate* delegate)
+{
+ armnnDelegate::TfLiteArmnnDelegateDelete(delegate);
+}
+
+} // extern "C"
+} // namespace tflite
\ No newline at end of file