summaryrefslogtreecommitdiff
path: root/automated/utils
AgeCommit message (Collapse)Author
2017-04-13utils: add support for LAVA's secrets featureChase Qi
* Use token from LAVA's secrets file if defined. * Renamed URL and TOKEN variables with ARTIFACTORIAL_ as prefix to make them more meaningful. Change-Id: I80775d0bf7492275a3f2ca73d54640ea488f4a35 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-04-06test-runner: set test case relative path for remoet exec onlyChase Qi
'tc_relative_dir' is used for partial file copy to target. It splits realpath to yaml file by test kind parameter and assumes tests are from 'automated' or 'manual' folder. Local exec copy the whole repo. It doesn't have the above assumption. Moving the 'tc_relative_dir' related lines to remote exec only. Change-Id: I95415fa93a94ddf654681ada704e4b2a5c12f061 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-29test-runner: run 'run.sh' under 'set -x'Chase Qi
When converting testdef to 'run.sh', LAVA has shell '-x' argument enabled. It can be useful for debugging. Change-Id: I1d1070b716713238e213b7140eb44526cdbcd7e1 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-17automated: test-runner: encapsulate the TestRun object into TestDefNicolas Dechesne
There is no need to create an additional object in main(). Instead we can create the 'runner' object in the TestDef init() and directly invoke the run() and check_result() function. Also rename TestDefinition.run() function as TestDefinition.mkrun(). This function really generates the run script, calling it run() is confusing, as this is not where the test runs. Change-Id: I4348e1e834e7083174aa60a8f4533747e49c3757 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-03-17automated: test-runner: mark test run as fail when no result foundChase Qi
Change-Id: I6cfc6f192a2392d8da13b788caaba97d08b50227 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-03-17test-runner: add remote execution featureMilosz Wasilewski
Test can be executed remotely using ssh. It is assumed that ssh communication is performed without passwords and public key is already added to authorized_keys. Most tests require root access to be executed, so there is a silent assumption that remote test execution is done using 'root' account. The account name might be different, but should be granted the same level of privileges. Change-Id: I41f1d32d20f15ac6cfa8fc9e99f527fe61d4743d Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-03-14test-runner: set PATH before running the testsNicolas Dechesne
Set PATH to a known value before starting test execution, to avoid PATH being polluted by the user environment. Change-Id: Ia7861932cd56f363d2d6cfa50ffd54c8acb1ea21 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-02-28automated: utils: improve upload-to-artifactorial.shChase Qi
* Only do attachment uploading in LAVA * Use basename of "${ATTACHMENT}" in case it includes a path Change-Id: I4893da2b16766a64152804be47ee8853a162534a Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-02-17automated: utils: add upload-to-artifactorial.shChase Qi
Change-Id: I969b285e1b3e092acbee0ca425f5bb1f81a97354 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-01-23test-runner: user $HOME instead of /rootNicolas Dechesne
Avoid hardcoding '/root', and use $HOME instead. A nice side effect is that it allows to use test-runner on the PC as standard user for local debug.. Change-Id: I2003565f9a9aa432dcf5e02cc49f7e87fe8ed946 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-01-19automated: add support for measurements with no unitNicolas Dechesne
It is possible to report measurements with no specific units, however the helper functions currently require the unit to be specified. This patch allows test case to provide only the measurement in add_metric() , effectively making the last arg (unit) optional. When processing the output , we then need to ensure whether or not the unit was specified in the first place. I needed this when porting sysbench test case onto 'automated' method (e.g. sysbench reports 'count of events'). With this change I can get the following test results from sysbench: <TEST_CASE_ID=total-time-2-threads RESULT=pass UNITS=s MEASUREMENT=4.0010> <TEST_CASE_ID=total-number-of-events-2-threads RESULT=pass UNITS= MEASUREMENT=8827> <TEST_CASE_ID=total-time-taken-by-event-execution-2-threads RESULT=pass UNITS= MEASUREMENT=7.9996> <TEST_CASE_ID=min-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.90> <TEST_CASE_ID=avg-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.91> <TEST_CASE_ID=max-2-threads RESULT=pass UNITS=ms MEASUREMENT=3.38> <TEST_CASE_ID=approx.-95-percentile-2-threads RESULT=pass UNITS=ms MEASUREMENT=0.93> <ENDRUN sysbench 9ce36643-e7e9-481e-a45a-464cd07d8cd9> 2017-01-19 15:45:57,557 - RUNNER.ResultParser: INFO: Result files saved to: /home/ndec/output/sysbench_9ce36643-e7e9-481e-a45a-464cd07d8cd9 --- Printing result.csv --- name,test_case_id,result,measurement,units,test_params sysbench,total-time-2-threads,pass,4.0010,s,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,total-number-of-events-2-threads,pass,8827,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,total-time-taken-by-event-execution-2-threads,pass,7.9996,,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,min-2-threads,pass,0.90,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,avg-2-threads,pass,0.91,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,max-2-threads,pass,3.38,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 sysbench,approx.-95-percentile-2-threads,pass,0.93,ms,DURATION=4;MAX_REQUESTS=50000;SKIP_INSTALL=False;TEST=cpu;FILE_TEST_MODE=seqrewr;TIMES=1 Change-Id: Id41207c3237f597bc3db83a5d7d54ee35f6974e3 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-01-03get_vland_pci_dev.sh: return the PCI deviceJosep Puigdemont
Change-Id: If0ac3a702528d97f536513e35a1f65be43f15a27 Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
2016-12-08test-runner: fixed result parsing regexpMilosz Wasilewski
Change-Id: I59f130777f01f038899041d2647de7ba0d3d004b Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-12-05automated: test-runner: Added support for complex test namesMilosz Wasilewski
Test names can contain '=' now. The parsing is based on regex rather than simple line split with '=' as delimiter. This adresses libhugetlbfs test parsing problem. Change-Id: I1b6e03960456abf437a2720f6e5177556c766f5a Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-12-02test-runner: fix version check for manual testChase Qi
When '-k manual' specified, test-runner only copy 'manual' folder to output directory, which isn't a valid git repo. From there, 'git checkout' or 'git rev-parse HEAD' fails. Change-Id: I005501efacb46fb49729995d112f8498164d8ad0 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-12-01test-runner: save test case name to result filesChase Qi
Change-Id: If4d6e28a05e886e6036916b36749b231cd916c34 Signed-off-by: Chase Qi <chase.qi@linaro.org> Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-29test-runner: record default parametersMilosz Wasilewski
Default parameters from test definition are now recorded in the result Change-Id: Id8d0fc88c694d158f6406586f5c79dcc3a83c164 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-29test-runner: added support for test versionMilosz Wasilewski
Change-Id: Idc255ad3e01c5ab95f259221fc901b93c2362862 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-23automated: test-runner: added support for 'parameters' in result parserChase Qi
In test plan, we can use 'params' or 'parameters'. In result parser, 'params' is already supported, adding support for 'parameters' now. Change-Id: I999565eafb72ca8cb9c56a8072fbd53e9f8e0fdd Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-18automated: test-runner: added option to record manual resultsMilosz Wasilewski
Manual test cases can now be executed using test runner. The results are recorded the same way as for automated tests. Change-Id: Ifacbc7591f81bc4468c4dbe6550be0ba1ec3fd61 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2016-11-16httperf-runner.py: a script to test http performanceJosep Puigdemont
This script will try to find the "best" replies/s rate for a target http server. It uses httperf and a trial-and-error method to find the best rate. Change-Id: I5f76d04ae5db710869038b700d8d9e99efad5d5f Signed-off-by: Josep Puigdemont <josep.puigdemont@linaro.org>
2016-11-10test-runner: save test parameters to result.csvChase Qi
Change-Id: If6759527cb0a7cc8556c06a91b680ce46d466972 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-07automated: test-runner: enhance result parserChase Qi
* Add test name and params to result.json * Collect all results in output/result.json * Add header and test name for output/result.csv Change-Id: Ie73af79be9e68780a8863e651dc30ed34bcbf4c3 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-11-07automated: send-to-lava: accept any character for measurement and unitsChase Qi
Change-Id: I61ade512f1f9f887e51cde8244ed51a40952df19 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-10-27automated/utils: add vland helper functions.Anders Roxell
get_vland_interface and get_vland_mac Change-Id: I32e84c985ca4e59eb695bcc8312da9d6de1643a4 Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
2016-10-24v2: send-to-lava.sh: move 'command -v' out of while loopChase Qi
Change-Id: Iff916da54cee93664f24bcdd1c14970b5a5db984 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-09-30send-to-lava: ignore results caseNaresh Kamboju
Change-Id: Id01743d67bc24ad2af7321db9afd8395afc07a73 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2016-09-02v2: test-runner: handle 3rd party python modulesChase Qi
Change-Id: I0a7b460cc891f2c4a927b11940699fde221d87e4 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2016-08-25v2: initial commitChase Qi
- Added utils, bin, and lib folders and related tools - Added Linux smoke test case as an example - Added test plan example that used to test test-runner Change-Id: I73693108db219cb67afc99525d80cab93f1eb9ea Signed-off-by: Chase Qi <chase.qi@linaro.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-20 06:52:10 +0000