Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 1 | /** |
Anthony Barbier | 79c6178 | 2017-06-23 11:48:24 +0100 | [diff] [blame] | 2 | @page tests Validation and benchmarks tests |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 3 | |
| 4 | @tableofcontents |
| 5 | |
| 6 | @section building_test_dependencies Building dependencies |
| 7 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 8 | The tests currently make use of Boost (Test and Program options) for |
| 9 | validation. Below are instructions about how to build these 3rd party |
| 10 | libraries. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 11 | |
Anthony Barbier | 79c6178 | 2017-06-23 11:48:24 +0100 | [diff] [blame] | 12 | @note By default the build of the validation and benchmark tests is disabled, to enable it use `validation_tests=1` and `benchmark_tests=1` |
| 13 | |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 14 | @subsection building_boost Building Boost |
| 15 | |
| 16 | First follow the instructions from the Boost library on how to setup the Boost |
| 17 | build system |
| 18 | (http://www.boost.org/doc/libs/1_64_0/more/getting_started/index.html). |
| 19 | Afterwards the required libraries can be build with: |
| 20 | |
| 21 | ./b2 --with-program_options --with-test link=static \ |
| 22 | define=BOOST_TEST_ALTERNATIVE_INIT_API |
| 23 | |
| 24 | Additionally, depending on your environment, it might be necessary to specify |
| 25 | the ```toolset=``` option to choose the right compiler. Moreover, |
| 26 | ```address-model=32``` can be used to force building for 32bit and |
| 27 | ```target-os=android``` must be specified to build for Android. |
| 28 | |
| 29 | After executing the build command the libraries |
| 30 | ```libboost_program_options.a``` and ```libboost_unit_test_framework.a``` can |
| 31 | be found in ```./stage/lib```. |
| 32 | |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 33 | @section tests_running_tests Running tests |
| 34 | @subsection tests_running_tests_benchmarking Benchmarking |
| 35 | @subsubsection tests_running_tests_benchmarking_filter Filter tests |
| 36 | All tests can be run by invoking |
| 37 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 38 | ./arm_compute_benchmark ./data |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 39 | |
| 40 | where `./data` contains the assets needed by the tests. |
| 41 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 42 | If only a subset of the tests has to be executed the `--filter` option takes a |
| 43 | regular expression to select matching tests. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 44 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 45 | ./arm_compute_benchmark --filter='NEON/.*AlexNet' ./data |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 46 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 47 | Additionally each test has a test id which can be used as a filter, too. |
| 48 | However, the test id is not guaranteed to be stable when new tests are added. |
| 49 | Only for a specific build the same the test will keep its id. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 50 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 51 | ./arm_compute_benchmark --filter-id=10 ./data |
| 52 | |
| 53 | All available tests can be displayed with the `--list-tests` switch. |
| 54 | |
| 55 | ./arm_compute_benchmark --list-tests |
| 56 | |
| 57 | More options can be found in the `--help` message. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 58 | |
| 59 | @subsubsection tests_running_tests_benchmarking_runtime Runtime |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 60 | By default every test is run once on a single thread. The number of iterations |
| 61 | can be controlled via the `--iterations` option and the number of threads via |
| 62 | `--threads`. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 63 | |
Moritz Pflanzer | 2b26b85 | 2017-07-21 10:09:30 +0100 | [diff] [blame] | 64 | @subsubsection tests_running_tests_benchmarking_output Output |
| 65 | By default the benchmarking results are printed in a human readable format on |
| 66 | the command line. The colored output can be disabled via `--no-color-output`. |
| 67 | As an alternative output format JSON is supported and can be selected via |
| 68 | `--log-format=json`. To write the output to a file instead of stdout the |
| 69 | `--log-file` option can be used. |
Anthony Barbier | 6ff3b19 | 2017-09-04 18:44:23 +0100 | [diff] [blame] | 70 | |
| 71 | @subsection tests_running_tests_validation Validation |
| 72 | @subsubsection tests_running_tests_validation_filter Filter tests |
| 73 | All tests can be run by invoking |
| 74 | |
| 75 | ./arm_compute_validation -- ./data |
| 76 | |
| 77 | where `./data` contains the assets needed by the tests. |
| 78 | |
| 79 | As running all tests can take a lot of time the suite is split into "precommit" and "nightly" tests. The precommit tests will be fast to execute but still cover the most important features. In contrast the nightly tests offer more extensive coverage but take longer. The different subsets can be selected from the command line as follows: |
| 80 | |
| 81 | ./arm_compute_validation -t @precommit -- ./data |
| 82 | ./arm_compute_validation -t @nightly -- ./data |
| 83 | |
| 84 | Additionally it is possible to select specific suites or tests: |
| 85 | |
| 86 | ./arm_compute_validation -t CL -- ./data |
| 87 | ./arm_compute_validation -t NEON/BitwiseAnd/RunSmall/_0 -- ./data |
| 88 | |
| 89 | All available tests can be displayed with the `--list_content` switch. |
| 90 | |
| 91 | ./arm_compute_validation --list_content -- ./data |
| 92 | |
| 93 | For a complete list of possible selectors please see: http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/runtime_config/test_unit_filtering.html |
| 94 | |
| 95 | @subsubsection tests_running_tests_validation_verbosity Verbosity |
| 96 | There are two separate flags to control the verbosity of the test output. `--report_level` controls the verbosity of the summary produced after all tests have been executed. `--log_level` controls the verbosity of the information generated during the execution of tests. All available settings can be found in the Boost documentation for [--report_level](http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/report_level.html) and [--log_level](http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html), respectively. |
| 97 | */ |