Age | Commit message (Collapse) | Author |
|
If a pattern regex is specified is used many times to parse the stdout
log to get results so is better to do a previous compile to speed up
parsing.
Change-Id: I9b37f35b4eded276144508b8ec255938fad19618
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
|
|
If no mapping exists in the fixup dictionary use the original result,
sometimes you only need to map certain result values.
There is the same logic in lava-dispatcher [1].
[1]
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/connection.py#n85
Change-Id: Iafd47ebd672fb27213e20727e3f4a099429c824e
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
|
|
|
|
On ERP testing, we need to skip disk partitioning test on boards that
don't have the second hard drive installed. We may also need to amend
test parameters or add new tests in the future. This patch add support
for test plan overlay to:
* skip tests.
* amend test parameters.
* add new tests.
Change-Id: I9143b313b32f0c541fb08bacce80f371b1cf15e6
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
* Simplify test plan format.
* Update test-runner for the change in test plan.
* Update existing test plans to v2.
Change-Id: Ic10e6eb3d17cbf54140831d0e030dcbbb95c75db
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
When setting test params on the command line with -r, they are correctly passed
down into run.sh script, however at the end of the test run when we print the
result, the custom params from command line are ignored, and we incorrectly
print the test params from the .yaml file.
e.g.
$ test-runner -d automated/linux/wlan-smoke/wlan-smoke.yaml -e -r BOOT=disabled
incorrectly prints the following:
name,test_case_id,result,measurement,units,test_params
wlan-smoke,ip-link,pass,,,DEVICE=wlan0;BOOT=enabled
wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;BOOT=enabled
wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;BOOT=enabled
wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;BOOT=enabled
When parsing the result, we need to override the test params with any param from
the command line. After this patch, the same command produces:
name,test_case_id,result,measurement,units,test_params
wlan-smoke,ip-link,pass,,,DEVICE=wlan0;BOOT=disabled
wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;BOOT=disabled
wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;BOOT=disabled
wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;BOOT=disabled
And for fun:
$ test-runner -d automated/linux/wlan-smoke/wlan-smoke.yaml -e -r BOOT=disabled THEBUGISGONE=yes
name,test_case_id,result,measurement,units,test_params
wlan-smoke,ip-link,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled
wlan-smoke,wlan-boot,fail,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled
wlan-smoke,wlan-down,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled
wlan-smoke,wlan-up,pass,,,DEVICE=wlan0;THEBUGISGONE=yes;BOOT=disabled
Change-Id: Ib561620f73e868d53653aa158f35e7804d7e13f3
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
Except LAVA test directories like '/lava-[0-9]+', the current approach
also captures unwanted directories like '/lava-lxc' directory. When this
case happen, it skips test attachment publishing as there is no 'serets'
file in non-test directory.
With '-regex' option, it filters find output and keep LAVA test
directories only.
Change-Id: I77f9fc490790a7eebbaaf353e9c4ed26ddad9879
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
When '-l' or '--lava_run' option specified, test-runner will use
lava-test-case to send result to LAVA.
It might be possible to detect if it is running in LAVA at running time
by checking if 'lava-test-case' exists, but that is not reliable as
local environment can also has the util installed. So the above option
introduced with 'False' as default value.
Change-Id: Ief0c9e75d12c07832214d2d4fa30836df964615a
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Artifactorial returns the full url now, the prefix isn't needed any
more. Refer to
https://lkft.validation.linaro.org/scheduler/job/21790#L3996
Change-Id: I579e49cc36c8cd38ed4b5f348a2cd4e661f67681
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
When custom parameters for test definition (-r) are not formatter
properly, they should be ignored.
Change-Id: I48fd26d2466ef79f0b301a2515b37c368537e2eb
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
When using test-runner to execute single test (-d) it is now possible to
specify additional parameters passed to the test on test-runner command
line.
Change-Id: I365a7fb084dbf7528bf14ce053a0856056c27150
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
Environment is collected after the tests are executed. In case SSH
connection is closed during test execution, environment collection
should fail gracefully so at least test results can be presented.
Change-Id: I55e1874dae2f20f9960bf0a12d05a35a48d321fc
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
In case $HOME is not set, getting the environment variable returns None.
This can't be concatenated with string without casting. This patch
changes the default to empty string in case $HOME is not set.
Change-Id: I4ba5c0e5a4952b31d6bac8e8cf225ef2ffa131d3
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
decode('utf-8') was added to strip off b' prefix from Python3's output.
The current code also runs that with Python2, it worked in most cases
some how. Howerver, once test-runner output redirectory used, it hits
UnicodeEncodeError when running tests include wget download step like
ltp and libhugetlbfs tests.
Usage: python2 automated/utils/test-runner.py \
-d automated/linux/libhugetlbfs/libhugetlbfs.yaml \
-o /root/output/libhugetlbfs \
> test-runner-stdout.log 2> test-runner-stderr.log
Error: UnicodeEncodeError: 'ascii' codec can't encode character
u'\u2018' in position 11: ordinal not in range(128)
This patch detects Python version at runtime, then uses decode('utf-8')
for on Python3 pexpect and subprocess.check_output() output to remove
prefix 'b'.
Change-Id: Ie52192787a8e4cd0550e1cd05b1ca53a02f4ab63
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Change-Id: I98e58c128def865c40ca84f6652963e0757259db
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Add a dictionary key named 'environment' that identifies the host running the
tests, whether that host is local or remote. For example, on a HiSilicon D03
system, the following data structure will be added to each result in
results.json:
{
"bios_version": "Hisilicon D03 UEFI 16.12 Release",
"board_name": "D03",
"board_vendor": "Huawei",
"kernel": "4.9.0-20.gitedc2a1c.linaro.aarch64",
"linux_distribution": "centos",
"packages": [
"GeoIP-1.5.0-11.el7",
"NetworkManager-1.4.0-20.el7_3",
...
"yum-plugin-fastestmirror-1.1.31-40.el7",
"zlib-1.2.7-17.el7"
],
"uname": "Linux localhost.localdomain 4.9.0-20.gitedc2a1c.linaro.aarch64 #1 SMP Wed Dec 14 17:50:15 UTC 2016 aarch64 aarch64 aarch64 GNU/Linux"
}
Support exists for Ubuntu, Debian, CentOS, and Fedora. Other distributions will
print a warning and report an empty package list. Environmental data is added
automatically in each test and may be skipped by passing -e or
--skip-environment.
This change led to a slight refactor in the way remote commands were being
handled. Instead of percolating remote vs local logic everywhere, run_command
will run an arbitrary shell command either locally or on the target host, and
return the output. This allows all shell commands, whether local or remote, to
be handled consistently.
Change-Id: Ia94695751a04a7fc862a299526d33784c57a9893
Signed-off-by: Dan Rue <dan.rue@linaro.org>
|
|
flake8 reported the following:
test_runner.py:196:25: F402 import 'cmd' from line 4 shadowed by loop variable
"cmd" is actually used three different ways in test-runner.py. First, it's an
imported module. Second, it's a loop variable. Third, it's re-assigned inside
the loop.
Change-Id: If907182b4f8229e47321d3819772213bb88791dc
Signed-off-by: Dan Rue <dan.rue@linaro.org>
|
|
LAVA V2 is able to catch and save result lines that send to stdout
instantly. For tests that run for a long time and generates a plenty of
results as it runs, like kselftest and ltp tests, LAVA's live result
collection feature could be useful.
In Linaro Test Suite, to support completely local run and reduce the
complexity of log parser debugging, we parse test log in test script
and save results in LAVA's format to result.txt file. Then use
'send-to-lava.sh [path]/result.txt' to send result lines to stdout after
test finished. This should work for most test cases, but maybe not good
enough for tests mentioned above as the tests may crash or hang or cause
Kernel panic or system halt during test run, in those cases, the result
sending line has no chance to get executed.
Using parse pattern in test definition file is one way to enable LAVA's
live result collection feature. This patch is used to add parse pattern
support in test-runner so that we can run the test cases using parse
pattern on the local. It also could be useful for pattern debugging on
the local.
Change-Id: I5756af84f653ab00fc52c62ff888b71fe8817944
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
This commit adds checking to ensure that duplicate tests are not added to the
test plan.
Change-Id: I045c16467cc17812a4559a14ac66e55a9c33a189
Signed-off-by: Dan Rue <dan.rue@linaro.org>
|
|
In LAVA, use lava-test-case for result saving makes the upload helper
script more reliable.
For local run, saving file upload 'fail/skip' result to 'result.txt'
isn't necessary as it is always skipped. And some test cases may
don't use 'result.txt' at all, in these cases, we will need to handle
file creating in the upload helper script which increases complicity.
Change-Id: Ie9ed79b66bab6abc861c2a3da7a3c6fd21b19c88
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
* Use os.chmod('file', 0o755) to change file permission. It is
supported in Python3 and higher version Python2(tested with v2.7.12
on Ubuntu 16.04). To support lower version Python2, we will need to
detect Python version at runtime and use different syntax
respectively. That probably isn't necessary.
* Use print("string") format for printing which is supported by both
Python2 and Python3.
* Cope with Python3's strict separation of strings and bytes.
Change-Id: I11ab8b956a750e7d6bfafe6ead37686248572854
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Change-Id: I89c8760d20b6d5fb9705adb4fe237916487b3a30
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
When used with a Debian initramfs (jessie, stretch or later)
when the kernel reports dns0 : 10.15.0.1 when mounting the
NFS, use this data to populate /etc/resolv.conf
Avoids overwriting an existing /etc/resolv.conf
Usage: ./automated/utils/nfs-dns.sh eth0
Change-Id: Ie61fde7081f71110d8a4bc6ebe154ca1cfc5beac
|
|
Change-Id: Ib6a4dc1be433b161d665ddcf07d1eb8c3382ca9f
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
test-runner.py does not work with python3. The first error is the following:
File "/work/linaro/test-definitions/automated/bin/test-runner", line 201
os.chmod('%s/run.sh' % self.test['test_path'], 0755)
^
SyntaxError: invalid token
Until we have support for python3, let's make sure that we use python2, this is
especially needed when trying to use distro where python3 is the default
(e.g. archlinux).
Change-Id: Id387bf0d6092c6555081eb891de12ff0a4df3bb7
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
* Use token from LAVA's secrets file if defined.
* Renamed URL and TOKEN variables with ARTIFACTORIAL_ as prefix to
make them more meaningful.
Change-Id: I80775d0bf7492275a3f2ca73d54640ea488f4a35
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
'tc_relative_dir' is used for partial file copy to target. It splits
realpath to yaml file by test kind parameter and assumes tests are from
'automated' or 'manual' folder.
Local exec copy the whole repo. It doesn't have the above assumption.
Moving the 'tc_relative_dir' related lines to remote exec only.
Change-Id: I95415fa93a94ddf654681ada704e4b2a5c12f061
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
When converting testdef to 'run.sh', LAVA has shell '-x' argument
enabled. It can be useful for debugging.
Change-Id: I1d1070b716713238e213b7140eb44526cdbcd7e1
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
There is no need to create an additional object in main(). Instead we can create
the 'runner' object in the TestDef init() and directly invoke the run() and
check_result() function.
Also rename TestDefinition.run() function as TestDefinition.mkrun(). This
function really generates the run script, calling it run() is confusing, as this
is not where the test runs.
Change-Id: I4348e1e834e7083174aa60a8f4533747e49c3757
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
Change-Id: I6cfc6f192a2392d8da13b788caaba97d08b50227
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Test can be executed remotely using ssh. It is assumed that ssh
communication is performed without passwords and public key is already
added to authorized_keys.
Most tests require root access to be executed, so there is a silent
assumption that remote test execution is done using 'root' account. The
account name might be different, but should be granted the same level of
privileges.
Change-Id: I41f1d32d20f15ac6cfa8fc9e99f527fe61d4743d
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
Set PATH to a known value before starting test execution, to avoid PATH being
polluted by the user environment.
Change-Id: Ia7861932cd56f363d2d6cfa50ffd54c8acb1ea21
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
* Only do attachment uploading in LAVA
* Use basename of "${ATTACHMENT}" in case it includes a path
Change-Id: I4893da2b16766a64152804be47ee8853a162534a
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Change-Id: I969b285e1b3e092acbee0ca425f5bb1f81a97354
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Avoid hardcoding '/root', and use $HOME instead. A nice side effect is that it
allows to use test-runner on the PC as standard user for local debug..
Change-Id: I2003565f9a9aa432dcf5e02cc49f7e87fe8ed946
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
It is possible to report measurements with no specific units, however the helper
functions currently require the unit to be specified. This patch allows test
case to provide only the measurement in add_metric() , effectively making the
last arg (unit) optional.
When processing the output , we then need to ensure whether or not the unit was
specified in the first place.
I needed this when porting sysbench test case onto 'automated' method
(e.g. sysbench reports 'count of events'). With this change I can get the
following test results from sysbench:
<TEST_CASE_ID=total-time-2-threads RESULT=pass UNITS=s MEASUREMENT=4.0010>
<TEST_CASE_ID=total-number-of-events-2-threads RESULT=pass UNITS= MEASUREMENT=8827>
<TEST_CASE_ID=total-time-taken-by-event-execution-2-threads RESULT=pass UNITS= MEASUREMENT=7.9996>
<TEST_CASE_ID=min-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.90>
<TEST_CASE_ID=avg-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.91>
<TEST_CASE_ID=max-2-threads RESULT=pass UNITS=ms MEASUREMENT=3.38>
<TEST_CASE_ID=approx.-95-percentile-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.93>
<ENDRUN sysbench 9ce36643-e7e9-481e-a45a-464cd07d8cd9>
2017-01-19 15:45:57,557 - RUNNER.ResultParser: INFO: Result files saved to: /home/ndec/output/sysbench_9ce36643-e7e9-481e-a45a-464cd07d8cd9
--- Printing result.csv ---
name,test_case_id,result,measurement,units,test_params
sysbench,total-time-2-threads,pass,4.0010,s,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,total-number-of-events-2-threads,pass,8827,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,total-time-taken-by-event-execution-2-threads,pass,7.9996,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,min-2-threads,pass,0.90,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,avg-2-threads,pass,0.91,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,max-2-threads,pass,3.38,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
sysbench,approx.-95-percentile-2-threads,pass,0.93,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1
Change-Id: Id41207c3237f597bc3db83a5d7d54ee35f6974e3
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
|
|
Change-Id: If0ac3a702528d97f536513e35a1f65be43f15a27
Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
|
|
Change-Id: I59f130777f01f038899041d2647de7ba0d3d004b
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
Test names can contain '=' now. The parsing is based on regex rather
than simple line split with '=' as delimiter. This adresses libhugetlbfs
test parsing problem.
Change-Id: I1b6e03960456abf437a2720f6e5177556c766f5a
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
When '-k manual' specified, test-runner only copy 'manual' folder to
output directory, which isn't a valid git repo. From there, 'git
checkout' or 'git rev-parse HEAD' fails.
Change-Id: I005501efacb46fb49729995d112f8498164d8ad0
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Change-Id: If4d6e28a05e886e6036916b36749b231cd916c34
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
Default parameters from test definition are now recorded in the result
Change-Id: Id8d0fc88c694d158f6406586f5c79dcc3a83c164
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
Change-Id: Idc255ad3e01c5ab95f259221fc901b93c2362862
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
In test plan, we can use 'params' or 'parameters'. In result parser,
'params' is already supported, adding support for 'parameters' now.
Change-Id: I999565eafb72ca8cb9c56a8072fbd53e9f8e0fdd
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Manual test cases can now be executed using test runner. The results are
recorded the same way as for automated tests.
Change-Id: Ifacbc7591f81bc4468c4dbe6550be0ba1ec3fd61
Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
|
|
This script will try to find the "best" replies/s rate for a target
http server. It uses httperf and a trial-and-error method to find
the best rate.
Change-Id: I5f76d04ae5db710869038b700d8d9e99efad5d5f
Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
|
|
Change-Id: If6759527cb0a7cc8556c06a91b680ce46d466972
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
* Add test name and params to result.json
* Collect all results in output/result.json
* Add header and test name for output/result.csv
Change-Id: Ie73af79be9e68780a8863e651dc30ed34bcbf4c3
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
Change-Id: I61ade512f1f9f887e51cde8244ed51a40952df19
Signed-off-by: Chase Qi <chase.qi@linaro.org>
|
|
get_vland_interface and get_vland_mac
Change-Id: I32e84c985ca4e59eb695bcc8312da9d6de1643a4
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
|