summaryrefslogtreecommitdiff
path: root/automated/utils
AgeCommit message (Collapse)Author
2017-11-29automated/utils/test-runner.py: Compile the pattern regex before useAníbal Limón
If a pattern regex is specified is used many times to parse the stdout log to get results so is better to do a previous compile to speed up parsing. Change-Id: I9b37f35b4eded276144508b8ec255938fad19618 Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
2017-11-29automated/utils/test-runner.py: Only use fixup dict if exists a map entryAníbal Limón
If no mapping exists in the fixup dictionary use the original result, sometimes you only need to map certain result values. There is the same logic in lava-dispatcher [1]. [1] https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/connection.py#n85 Change-Id: Iafd47ebd672fb27213e20727e3f4a099429c824e Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
2017-11-28Merge "test-runner: add support for test plan overlay"Dan Rue
2017-11-28test-runner: add support for test plan overlayChase Qi
On ERP testing, we need to skip disk partitioning test on boards that don't have the second hard drive installed. We may also need to amend test parameters or add new tests in the future. This patch add support for test plan overlay to: * skip tests. * amend test parameters. * add new tests. Change-Id: I9143b313b32f0c541fb08bacce80f371b1cf15e6 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-11-27plans: introduce linaro test plan v2Chase Qi
* Simplify test plan format. * Update test-runner for the change in test plan. * Update existing test plans to v2. Change-Id: Ic10e6eb3d17cbf54140831d0e030dcbbb95c75db Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-10-23test-runner: ResultParser: handle custom params from command lineNicolas Dechesne
When setting test params on the command line with -r, they are correctly passed down into run.sh script, however at the end of the test run when we print the result, the custom params from command line are ignored, and we incorrectly print the test params from the .yaml file. e.g. $ test-runner -d automated/linux/wlan-smoke/wlan-smoke.yaml -e -r BOOT=disabled incorrectly prints the following: name,test_case_id,result,measurement,units,test_params wlan-smoke,ip-link,pass,,,DEVICE=wlan0;BOOT=enabled wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;BOOT=enabled wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;BOOT=enabled wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;BOOT=enabled When parsing the result, we need to override the test params with any param from the command line. After this patch, the same command produces: name,test_case_id,result,measurement,units,test_params wlan-smoke,ip-link,pass,,,DEVICE=wlan0;BOOT=disabled wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;BOOT=disabled wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;BOOT=disabled wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;BOOT=disabled And for fun: $ test-runner -d automated/linux/wlan-smoke/wlan-smoke.yaml -e -r BOOT=disabled THEBUGISGONE=yes name,test_case_id,result,measurement,units,test_params wlan-smoke,ip-link,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled Change-Id: Ib561620f73e868d53653aa158f35e7804d7e13f3 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-10-11utils: filter 'find /lava-*' output to get test directory onlyChase Qi
Except LAVA test directories like '/lava-[0-9]+', the current approach also captures unwanted directories like '/lava-lxc' directory. When this case happen, it skips test attachment publishing as there is no 'serets' file in non-test directory. With '-regex' option, it filters find output and keep LAVA test directories only. Change-Id: I77f9fc490790a7eebbaaf353e9c4ed26ddad9879 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-09-21test-runner: add support for remote execution in LAVAChase Qi
When '-l' or '--lava_run' option specified, test-runner will use lava-test-case to send result to LAVA. It might be possible to detect if it is running in LAVA at running time by checking if 'lava-test-case' exists, but that is not reliable as local environment can also has the util installed. So the above option introduced with 'False' as default value. Change-Id: Ief0c9e75d12c07832214d2d4fa30836df964615a Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-08-29utils: fix test attachment urlChase Qi
Artifactorial returns the full url now, the prefix isn't needed any more. Refer to https://lkft.validation.linaro.org/scheduler/job/21790#L3996 Change-Id: I579e49cc36c8cd38ed4b5f348a2cd4e661f67681 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-08-02test-runner: handle incorrect test definition parametersMilosz Wasilewski
When custom parameters for test definition (-r) are not formatter properly, they should be ignored. Change-Id: I48fd26d2466ef79f0b301a2515b37c368537e2eb Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-08-02test-runner: add support for single test parametersMilosz Wasilewski
When using test-runner to execute single test (-d) it is now possible to specify additional parameters passed to the test on test-runner command line. Change-Id: I365a7fb084dbf7528bf14ce053a0856056c27150 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-07-20test-runner: fail gracefully if SSH connection is closedMilosz Wasilewski
Environment is collected after the tests are executed. In case SSH connection is closed during test execution, environment collection should fail gracefully so at least test results can be presented. Change-Id: I55e1874dae2f20f9960bf0a12d05a35a48d321fc Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-07-20test-runner: set a proper default for $HOMEMilosz Wasilewski
In case $HOME is not set, getting the environment variable returns None. This can't be concatenated with string without casting. This patch changes the default to empty string in case $HOME is not set. Change-Id: I4ba5c0e5a4952b31d6bac8e8cf225ef2ffa131d3 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-06-27test-runner: fix UnicodeEncodeError with Python2Chase Qi
decode('utf-8') was added to strip off b' prefix from Python3's output. The current code also runs that with Python2, it worked in most cases some how. Howerver, once test-runner output redirectory used, it hits UnicodeEncodeError when running tests include wget download step like ltp and libhugetlbfs tests. Usage: python2 automated/utils/test-runner.py \ -d automated/linux/libhugetlbfs/libhugetlbfs.yaml \ -o /root/output/libhugetlbfs \ > test-runner-stdout.log 2> test-runner-stderr.log Error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 11: ordinal not in range(128) This patch detects Python version at runtime, then uses decode('utf-8') for on Python3 pexpect and subprocess.check_output() output to remove prefix 'b'. Change-Id: Ie52192787a8e4cd0550e1cd05b1ca53a02f4ab63 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-06-14utils: add post-to-squad.pyChase Qi
Change-Id: I98e58c128def865c40ca84f6652963e0757259db Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-06-12test-runner: Report environmental informationDan Rue
Add a dictionary key named 'environment' that identifies the host running the tests, whether that host is local or remote. For example, on a HiSilicon D03 system, the following data structure will be added to each result in results.json: { "bios_version": "Hisilicon D03 UEFI 16.12 Release", "board_name": "D03", "board_vendor": "Huawei", "kernel": "4.9.0-20.gitedc2a1c.linaro.aarch64", "linux_distribution": "centos", "packages": [ "GeoIP-1.5.0-11.el7", "NetworkManager-1.4.0-20.el7_3", ... "yum-plugin-fastestmirror-1.1.31-40.el7", "zlib-1.2.7-17.el7" ], "uname": "Linux localhost.localdomain 4.9.0-20.gitedc2a1c.linaro.aarch64 #1 SMP Wed Dec 14 17:50:15 UTC 2016 aarch64 aarch64 aarch64 GNU/Linux" } Support exists for Ubuntu, Debian, CentOS, and Fedora. Other distributions will print a warning and report an empty package list. Environmental data is added automatically in each test and may be skipped by passing -e or --skip-environment. This change led to a slight refactor in the way remote commands were being handled. Instead of percolating remote vs local logic everywhere, run_command will run an arbitrary shell command either locally or on the target host, and return the output. This allows all shell commands, whether local or remote, to be handled consistently. Change-Id: Ia94695751a04a7fc862a299526d33784c57a9893 Signed-off-by: Dan Rue <dan.rue@linaro.org>
2017-06-09test-runner: Fix variable overloadingDan Rue
flake8 reported the following: test_runner.py:196:25: F402 import 'cmd' from line 4 shadowed by loop variable "cmd" is actually used three different ways in test-runner.py. First, it's an imported module. Second, it's a loop variable. Third, it's re-assigned inside the loop. Change-Id: If907182b4f8229e47321d3819772213bb88791dc Signed-off-by: Dan Rue <dan.rue@linaro.org>
2017-06-06test-runner: add support for parse patternChase Qi
LAVA V2 is able to catch and save result lines that send to stdout instantly. For tests that run for a long time and generates a plenty of results as it runs, like kselftest and ltp tests, LAVA's live result collection feature could be useful. In Linaro Test Suite, to support completely local run and reduce the complexity of log parser debugging, we parse test log in test script and save results in LAVA's format to result.txt file. Then use 'send-to-lava.sh [path]/result.txt' to send result lines to stdout after test finished. This should work for most test cases, but maybe not good enough for tests mentioned above as the tests may crash or hang or cause Kernel panic or system halt during test run, in those cases, the result sending line has no chance to get executed. Using parse pattern in test definition file is one way to enable LAVA's live result collection feature. This patch is used to add parse pattern support in test-runner so that we can run the test cases using parse pattern on the local. It also could be useful for pattern debugging on the local. Change-Id: I5756af84f653ab00fc52c62ff888b71fe8817944 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-30test-runner: Skip duplicate testsDan Rue
This commit adds checking to ensure that duplicate tests are not added to the test plan. Change-Id: I045c16467cc17812a4559a14ac66e55a9c33a189 Signed-off-by: Dan Rue <dan.rue@linaro.org>
2017-05-26automated: utils: use lava-test-case for result saving in LAVAChase Qi
In LAVA, use lava-test-case for result saving makes the upload helper script more reliable. For local run, saving file upload 'fail/skip' result to 'result.txt' isn't necessary as it is always skipped. And some test cases may don't use 'result.txt' at all, in these cases, we will need to handle file creating in the upload helper script which increases complicity. Change-Id: Ie9ed79b66bab6abc861c2a3da7a3c6fd21b19c88 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-25test-runner: add support for Python3Chase Qi
* Use os.chmod('file', 0o755) to change file permission. It is supported in Python3 and higher version Python2(tested with v2.7.12 on Ubuntu 16.04). To support lower version Python2, we will need to detect Python version at runtime and use different syntax respectively. That probably isn't necessary. * Use print("string") format for printing which is supported by both Python2 and Python3. * Cope with Python3's strict separation of strings and bytes. Change-Id: I11ab8b956a750e7d6bfafe6ead37686248572854 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-23test-runner: fix test kind selectionChase Qi
Change-Id: I89c8760d20b6d5fb9705adb4fe237916487b3a30 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-12Add a script to populate local DNS in NFSNeil Williams
When used with a Debian initramfs (jessie, stretch or later) when the kernel reports dns0 : 10.15.0.1 when mounting the NFS, use this data to populate /etc/resolv.conf Avoids overwriting an existing /etc/resolv.conf Usage: ./automated/utils/nfs-dns.sh eth0 Change-Id: Ie61fde7081f71110d8a4bc6ebe154ca1cfc5beac
2017-05-12utils: skip file uploading on empty tokenChase Qi
Change-Id: Ib6a4dc1be433b161d665ddcf07d1eb8c3382ca9f Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-11test-runner.py: use python2, not pythonNicolas Dechesne
test-runner.py does not work with python3. The first error is the following: File "/work/linaro/test-definitions/automated/bin/test-runner", line 201 os.chmod('%s/run.sh' % self.test['test_path'], 0755) ^ SyntaxError: invalid token Until we have support for python3, let's make sure that we use python2, this is especially needed when trying to use distro where python3 is the default (e.g. archlinux). Change-Id: Id387bf0d6092c6555081eb891de12ff0a4df3bb7 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-04-13utils: add support for LAVA's secrets featureChase Qi
* Use token from LAVA's secrets file if defined. * Renamed URL and TOKEN variables with ARTIFACTORIAL_ as prefix to make them more meaningful. Change-Id: I80775d0bf7492275a3f2ca73d54640ea488f4a35 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-04-06test-runner: set test case relative path for remoet exec onlyChase Qi
'tc_relative_dir' is used for partial file copy to target. It splits realpath to yaml file by test kind parameter and assumes tests are from 'automated' or 'manual' folder. Local exec copy the whole repo. It doesn't have the above assumption. Moving the 'tc_relative_dir' related lines to remote exec only. Change-Id: I95415fa93a94ddf654681ada704e4b2a5c12f061 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-29test-runner: run 'run.sh' under 'set -x'Chase Qi
When converting testdef to 'run.sh', LAVA has shell '-x' argument enabled. It can be useful for debugging. Change-Id: I1d1070b716713238e213b7140eb44526cdbcd7e1 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-17automated: test-runner: encapsulate the TestRun object into TestDefNicolas Dechesne
There is no need to create an additional object in main(). Instead we can create the 'runner' object in the TestDef init() and directly invoke the run() and check_result() function. Also rename TestDefinition.run() function as TestDefinition.mkrun(). This function really generates the run script, calling it run() is confusing, as this is not where the test runs. Change-Id: I4348e1e834e7083174aa60a8f4533747e49c3757 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-03-17automated: test-runner: mark test run as fail when no result foundChase Qi
Change-Id: I6cfc6f192a2392d8da13b788caaba97d08b50227 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-17test-runner: add remote execution featureMilosz Wasilewski
Test can be executed remotely using ssh. It is assumed that ssh communication is performed without passwords and public key is already added to authorized_keys. Most tests require root access to be executed, so there is a silent assumption that remote test execution is done using 'root' account. The account name might be different, but should be granted the same level of privileges. Change-Id: I41f1d32d20f15ac6cfa8fc9e99f527fe61d4743d Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-03-14test-runner: set PATH before running the testsNicolas Dechesne
Set PATH to a known value before starting test execution, to avoid PATH being polluted by the user environment. Change-Id: Ia7861932cd56f363d2d6cfa50ffd54c8acb1ea21 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-02-28automated: utils: improve upload-to-artifactorial.shChase Qi
* Only do attachment uploading in LAVA * Use basename of "${ATTACHMENT}" in case it includes a path Change-Id: I4893da2b16766a64152804be47ee8853a162534a Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-02-17automated: utils: add upload-to-artifactorial.shChase Qi
Change-Id: I969b285e1b3e092acbee0ca425f5bb1f81a97354 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-01-23test-runner: user $HOME instead of /rootNicolas Dechesne
Avoid hardcoding '/root', and use $HOME instead. A nice side effect is that it allows to use test-runner on the PC as standard user for local debug.. Change-Id: I2003565f9a9aa432dcf5e02cc49f7e87fe8ed946 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-01-19automated: add support for measurements with no unitNicolas Dechesne
It is possible to report measurements with no specific units, however the helper functions currently require the unit to be specified. This patch allows test case to provide only the measurement in add_metric() , effectively making the last arg (unit) optional. When processing the output , we then need to ensure whether or not the unit was specified in the first place. I needed this when porting sysbench test case onto 'automated' method (e.g. sysbench reports 'count of events'). With this change I can get the following test results from sysbench: <TEST_CASE_ID=total-time-2-threads RESULT=pass UNITS=s MEASUREMENT=4.0010> <TEST_CASE_ID=total-number-of-events-2-threads RESULT=pass UNITS= MEASUREMENT=8827> <TEST_CASE_ID=total-time-taken-by-event-execution-2-threads RESULT=pass UNITS= MEASUREMENT=7.9996> <TEST_CASE_ID=min-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.90> <TEST_CASE_ID=avg-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.91> <TEST_CASE_ID=max-2-threads RESULT=pass UNITS=ms MEASUREMENT=3.38> <TEST_CASE_ID=approx.-95-percentile-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.93> <ENDRUN sysbench 9ce36643-e7e9-481e-a45a-464cd07d8cd9> 2017-01-19 15:45:57,557 - RUNNER.ResultParser: INFO: Result files saved to: /home/ndec/output/sysbench_9ce36643-e7e9-481e-a45a-464cd07d8cd9 --- Printing result.csv --- name,test_case_id,result,measurement,units,test_params sysbench,total-time-2-threads,pass,4.0010,s,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,total-number-of-events-2-threads,pass,8827,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,total-time-taken-by-event-execution-2-threads,pass,7.9996,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,min-2-threads,pass,0.90,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,avg-2-threads,pass,0.91,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,max-2-threads,pass,3.38,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,approx.-95-percentile-2-threads,pass,0.93,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 Change-Id: Id41207c3237f597bc3db83a5d7d54ee35f6974e3 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-01-03get_vland_pci_dev.sh: return the PCI deviceJosep Puigdemont
Change-Id: If0ac3a702528d97f536513e35a1f65be43f15a27 Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
2016-12-08test-runner: fixed result parsing regexpMilosz Wasilewski
Change-Id: I59f130777f01f038899041d2647de7ba0d3d004b Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-12-05automated: test-runner: Added support for complex test namesMilosz Wasilewski
Test names can contain '=' now. The parsing is based on regex rather than simple line split with '=' as delimiter. This adresses libhugetlbfs test parsing problem. Change-Id: I1b6e03960456abf437a2720f6e5177556c766f5a Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-12-02test-runner: fix version check for manual testChase Qi
When '-k manual' specified, test-runner only copy 'manual' folder to output directory, which isn't a valid git repo. From there, 'git checkout' or 'git rev-parse HEAD' fails. Change-Id: I005501efacb46fb49729995d112f8498164d8ad0 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-12-01test-runner: save test case name to result filesChase Qi
Change-Id: If4d6e28a05e886e6036916b36749b231cd916c34 Signed-off-by: Chase Qi <chase.qi@linaro.org> Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-29test-runner: record default parametersMilosz Wasilewski
Default parameters from test definition are now recorded in the result Change-Id: Id8d0fc88c694d158f6406586f5c79dcc3a83c164 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-29test-runner: added support for test versionMilosz Wasilewski
Change-Id: Idc255ad3e01c5ab95f259221fc901b93c2362862 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-23automated: test-runner: added support for 'parameters' in result parserChase Qi
In test plan, we can use 'params' or 'parameters'. In result parser, 'params' is already supported, adding support for 'parameters' now. Change-Id: I999565eafb72ca8cb9c56a8072fbd53e9f8e0fdd Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-18automated: test-runner: added option to record manual resultsMilosz Wasilewski
Manual test cases can now be executed using test runner. The results are recorded the same way as for automated tests. Change-Id: Ifacbc7591f81bc4468c4dbe6550be0ba1ec3fd61 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-16httperf-runner.py: a script to test http performanceJosep Puigdemont
This script will try to find the "best" replies/s rate for a target http server. It uses httperf and a trial-and-error method to find the best rate. Change-Id: I5f76d04ae5db710869038b700d8d9e99efad5d5f Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
2016-11-10test-runner: save test parameters to result.csvChase Qi
Change-Id: If6759527cb0a7cc8556c06a91b680ce46d466972 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-07automated: test-runner: enhance result parserChase Qi
* Add test name and params to result.json * Collect all results in output/result.json * Add header and test name for output/result.csv Change-Id: Ie73af79be9e68780a8863e651dc30ed34bcbf4c3 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-07automated: send-to-lava: accept any character for measurement and unitsChase Qi
Change-Id: I61ade512f1f9f887e51cde8244ed51a40952df19 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-10-27automated/utils: add vland helper functions.Anders Roxell
get_vland_interface and get_vland_mac Change-Id: I32e84c985ca4e59eb695bcc8312da9d6de1643a4 Signed-off-by: Anders Roxell <anders.roxell@linaro.org>