EL3 Firmware Test Framework User Guide ====================================== Contents : 1. Introduction 2. Host machine requirements 3. Tools 4. Building the Test Framework 5. Running the Test Framework 6. Global overview of the Test Framework's behaviour - - - - - - - - - - - - - - - - - - 1. Introduction ---------------- The EL3 Firmware Test Framework is a framework used to develop baremetal tests. It is currently supported on the Juno ARM development platform and the ARM Fixed Virtual Platform (FVP) models. It runs in Normal World, at exception level EL2. Its main purpose is to test the implementation of the EL3 Firmware. This document describes how to build the EL3 Firmware Test Framework for the Juno board and the FVP models. It also gives a global overview of what it does. 2. Host machine requirements ----------------------------- The minimum recommended machine specification for building the software and running the FVP models is a dual-core processor running at 2GHz with 12GB of RAM. For best performance, use a machine with a quad-core processor running at 2.6GHz with 16GB of RAM. The software has been tested on Ubuntu 12.04.04 (64-bit). Packages used for building the software were installed from that distribution unless otherwise specified. 3. Tools --------- The following tools are required to use the EL3 Firmware Test Framework: * Baremetal GNU GCC tools. Verified packages can be downloaded from [Linaro] [Linaro Toolchain]. The rest of this document assumes that the `gcc-linaro-aarch64-none-elf-4.9-2014.07_linux.tar.xz` tools are used. wget http://releases.linaro.org/14.07/components/toolchain/binaries/gcc-linaro-aarch64-none-elf-4.9-2014.07_linux.tar.xz tar -xf gcc-linaro-aarch64-none-elf-4.9-2014.07_linux.tar.xz * Perl module to interface to the libxml2 library. This is provided by the package `libxml-libxml-perl`. This is required by the perl script `tools/generate_test_list/generate_test_list.pl`, which is run during the Test Framework's build. * (Optional) For debugging, ARM [Development Studio 5 (DS-5)][DS-5] v5.19. 4. Building the Test framework ------------------------------- Two platforms are supported at the moment: * FVP models: Foundation, Base AEM, Base Cortex * Juno board To build the software for one of these two platforms, follow these steps: 1. Specify the cross-compiler prefix, the targeted platform and build: CROSS_COMPILE=/bin/aarch64-none-elf- \ make PLAT= all ... where `` is either `fvp` or `juno`. By default this produces a release version of the build. To produce a debug version instead, CROSS_COMPILE=/bin/aarch64-none-elf- \ make PLAT= DEBUG=1 all To make the build verbose, use: CROSS_COMPILE=/bin/aarch64-none-elf- \ make PLAT= V=1 all 2. The build process creates products in a `build` directory tree. The resulting binary is in `build///tftf.bin` where `` is either `debug` or `release`. The resulting ELF file is `build///tftf/tftf.elf` The resulting binary `tftf.bin` can then be put into the FIP image. In ARM Trusted Firmware's boot flow, the TFTF replaces the BL3-3 image (i.e. UEFI). Therefore, one way to build such a FIP is to run the command `make fip` from ARM Trusted Firmware's root directory and setting the `BL33` variable to point to the location of the `tftf.bin`. E.g.: cd / CROSS_COMPILE=/bin/aarch64-none-elf- \ BL33=/build///tftf.bin \ make PLAT= fip ### Summary of build options The Test Framework build system supports the following build options. Unless mentioned otherwise, these options are expected to be specified at the build command line and are not to be modified in any component makefiles. Note that the build system doesn't track dependency for build options. Therefore, if any of the build options are changed from a previous build, a clean build must be performed. * `CROSS_COMPILE`: Prefix to toolchain binaries. Please refer to examples in this document for usage. * `DEBUG`: Choose between a debug and release build. It can take either 0 (release) or 1 (debug) as values. 0 is the default. * `NEW_TEST_SESSION`: Choose whether a new test session should be started every time or whether the framework should determine whether a previous session was interrupted and resume it. It can take either 1 (always start new session) or 0 (resume session as appropriate). 1 is the default. * `PLAT`: Choose a platform to build the Test Framework for. The chosen platform name must be the name of one of the directories under the `plat/` directory other than `common`. * `SHELL_COLOR`: Choose whether text messages should use shell's color escape sequences to ease identifying which CPU displays it. If enabled, this makes each CPU write part of the message in a different color. It can take either 0 (disabled) or 1 (enabled) as values. 0 is the default. * `TEST_REPORTS`: List of desired test reports. Test reports are described by a space-separated list of colon-separated pairs of report destination and report type. Valid destinations are: 'uart' and 'semihosting'. Valid report types are 'raw' (text output) or 'junit' (XML Junit format). The default is 'uart:raw', that is a text report printed on the UART. Here's an example of multiple reports: 'uart:raw semihosting:junit'. The files stored on semihosting are named 'tftf_report_junit.xml' and 'tftf_report_raw.txt'. * `TESTS_FILE`: Path to the XML file listing the tests to run. Default is `plat//tests.xml` if it exists, otherwise it falls back to `tests/tests-common.xml`. * `V`: Verbose build. If assigned anything other than 0, the build commands are printed. Default is 0. * `USE_NVM`: Used to select the location of test results. It can take either 0 (RAM) or 1 (Secondary memory like flash) as test results storage. Default value is 0, as writing to the flash significantly slows tests down. * `FIRMWARE_UPDATE`: Whether the Firmware Update non-secure images (i.e. `NS_BL1U` and `NS_BL2U` images) should be built. The default value is 0. The platform makefile is free to override this value if Firmware Update is supported on this platform. 5. Running the Test framework ------------------------------ Refer to the sections `Running the software on FVP` and `Running the software on Juno` in [ARM Trusted Firmware User Guide]. The same instructions apply to run the Test Framework on those 2 platforms. The only difference is that you don't need the following software: * UEFI bootloader. The TFTF replaces UEFI in the boot flow. * Linux Kernel * Device tree * Filesystem In other words, only the following software is needed: * BL1 bootloader image * The FIP image containing the following bootloader images: * BL2 * BL3-0 if required by the platform (e.g. Juno) * BL3-1 * optionally BL3-2 * the TFTF (standing as the BL3-3 image) 6. Global overview of the Test Framework's behaviour ----------------------------------------------------- The EL3 firmware is expected to hand over to the Test Framework with all secondary cores powered down, i.e. only the primary core should enter the Test Framework. The primary CPU initialises the platform and the Test Framework's internal data structures. Then the test session begins. The Test Framework executes the tests one after the other. Tests results are saved in non-volatile memory as we go along. Once all tests have completed, a report is generated. The Test Framework currently supports 2 report formats: * Raw output, i.e. text messages over serial console * Junit output. XML file generated over semihosting. Currently this is used to interface tests results with Jenkins. The report format to use is chosen at build time. For more information, refer to the section `Summary of build options` above in this document. If the chosen report format is Junit, the TFTF will produce a file called `tftf_report_junit.xml`. Note that the Junit output requires semihosting support. This is supported on the FVP models but not on Juno. - - - - - - - - - - - - - - - - - - - - - - - - - - _Copyright (c) 2014, ARM Limited and Contributors. All rights reserved._ [ARM FVP website]: http://www.arm.com/fvp [Linaro Toolchain]: http://releases.linaro.org/14.07/components/toolchain/binaries/ [DS-5]: http://www.arm.com/products/tools/software-tools/ds-5/index.php [ARM Trusted Firmware User Guide]: https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/user-guide.md