docs: added page about test-runner

Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>

4 files changed, 184 insertions, 108 deletions

diff --git a/README.md b/README.md
index a82d168..160b357 100644
--- a/README.md
+++ b/README.md
@@ -1,109 +1,8 @@
-# Test Definitions
-A test suite works with and without [LAVA](http://lavasoftware.org/). The following two
-sets of automated tests are supported.
-
-- `automated/linux/`
-- `automated/android/`
-
-For each test case, both the test script and the corresponding test definition files are
-provided in the same folder and are named after the test case name. Test scripts are
-self-contained and work independently. Test definition files in YAML format are provided
-for test runs with local test-runner and within LAVA.
-
-## Installation
-Installing the latest development version:
-
-    git clone https://github.com/Linaro/test-definitions
-    cd ./test-definitions
-    . ./automated/bin/setenv.sh
-    pip install -r ${REPO_PATH}/automated/utils/requirements.txt
-
-If the above succeeds, try:
-
-    test-runner -h
-
-## Running test
-### Running test script
-#### linux
-
-    cd ./automated/linux/smoke/
-    ./smoke.sh
-
-Skip package installation:
-
-    ./smoke.sh -s true
-
-#### android
-
-    cd ./automated/android/dd-wr-speed/
-    ./dd-wr-speed.sh
-
-Specify SN when more than one device connected:
-
-    ./dd-wr-speed.sh -s "serial_no"
-
-Specify other params:
-
-    ./dd-wr-speed.sh -i "10" -p "/dev/block/mmcblk1p1"
+# Build status
+[![Build Status](https://travis-ci.com/Linaro/test-definitions.svg?branch=master)](https://travis-ci.com/Linaro/test-definitions)
 
-### Using test-runner
-#### single test run
-
-    test-runner -d ./automated/linux/smoke/smoke.yaml
-
-skip package install:
-
-    test-runner -d ./automated/linux/smoke/smoke.yaml -s
-
-#### running test plan
-
-Run a set of tests defined in agenda file:
-    
-    test-runner -p ./plans/linux-example.yaml
-
-Apply test plan overlay to skip, amend or add tests:
-    
-    test-runner -p ./plans/linux-example.yaml -O test-plan-overlay-example.yaml
-
-## Collecting result
-
-### Using test script
-Test script normally puts test log and parsed results to its own `output` directory. e.g.
-    
-    automated/linux/smoke/output
-
-### Using test-runner
-test-runner needs a separate directory outside the repo to store test and result files.
-The directory defaults to `$HOME/output` and can be changed with `-o <dir>`. test-runner
-converts test definition file to `run.sh` and then parses its stdout. Results
-will be saved to results.{json,csv} by test. e.g.
-    
-    /root/output/smoke_9879e7fd-a8b6-472d-b266-a20b05d52ed1/result.csv
-
-When using the same output directory for multiple tests, test-runner combines results
-from all tests and save them to `${OUTPUT}/results.{json,csv}`. e.g.
-    
-    /root/output/result.json
-
-## Contributing
-
-Please use Github for pull requests: https://github.com/Linaro/test-definitions/pulls
-
-https://git.linaro.org/qa/test-definitions.git is a read-only mirror. New changes in the
-github repo will be pushed to the mirror every 10 minutes.
-
-Refer to [test writing guidelines](docs/test-writing-guidelines.md) to modify
-or add test. 
-
-Changes need to be able to pass sanity check, which by default checks files in the most
-recent commit:
-
-    ./sanity-check.sh
+# Test Definitions
+A set of testing scripts designed to work with [LAVA](http://lavasoftware.org/).
+Also contains test-runner script that allows execoution outside LAVA.
 
-To develop locally, there are Dockerfiles in test/ that can be used to simulate
-target environments. The easiest way to use is to run `test.sh
-[debian|centos]`. test.sh will run validate.py, and then build the Docker
-environment specified, run plans/linux-example.yaml, and then drop into a bash
-shell inside the container so that things like /root/output can be inspected.
-It is not (yet) a pass/fail test; merely a development helper and validation
-environment.
+More details in [docs](docs/index.md)
diff --git a/docs/index.md b/docs/index.md
index 44651ea..2d91dbb 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -85,6 +85,8 @@ from all tests and save them to `${OUTPUT}/results.{json,csv}`. e.g.
 
     /root/output/result.json
 
+More details on test-runner usage in [test-runner docs](test-runner.md)
+
 ## Generating documentation
 
 [Full docs](https://test-definitions.readthedocs.io) are generated from existing
diff --git a/docs/test-runner.md b/docs/test-runner.md
new file mode 100644
index 0000000..1be2cbe
--- /dev/null
+++ b/docs/test-runner.md
@@ -0,0 +1,175 @@
+# Test Runner
+
+## Installation
+Installing the latest development version:
+
+    git clone https://github.com/Linaro/test-definitions
+    cd ./test-definitions
+    . ./automated/bin/setenv.sh
+    pip install -r ${REPO_PATH}/automated/utils/requirements.txt
+
+If the above succeeds, try:
+
+    test-runner -h
+
+### Using virtualenv
+test-runner can also be installed inside virtualenv if one doesn't wish
+to install directly on the system. This might be useful for hacking the tool
+or the tests themselves.
+
+    git clone https://github.com/Linaro/test-definitions
+    cd ./test-definitions
+    . ./automated/bin/setenv.sh
+    virtualenv -p python3 venv
+    . ./venv/bin/activate
+    pip install -r ${REPO_PATH}/automated/utils/requirements.txt
+
+If the above succeeds, try:
+
+    test-runner -h
+
+## Running automated tests
+test-runner can be used to execute automated tests on the local or remote
+target.
+
+### Running tests on the local target (cheating a bit)
+Running tests on local target might be tricky. By default test-runner
+requires to be executed as root. This limitation is lifted for manual tests.
+Passing *--kind manual* will result in test execution without root access.
+This only works if there is no need to install dependencies, so *--skip_install*
+is also required.
+
+    test-runner --kind manual --test_def automated/linux/smoke/smoke.yaml --skip_install
+
+### Running tests over ssh
+test-runner can connect to the target using ssh. It requires passwordless
+authentication. This means that:
+ * sshd on the target must accept certificate based authentication
+ * user's public key needs to be added to root's authorized_keys on the target
+
+#### ssh setup
+1. make sure sshd accepts passwordless connection. Content of /etc/sshd_config
+   ```
+   PubkeyAuthentication yes
+   AuthorizedKeysFile .ssh/authorized_keys
+   ```
+
+2. Copy your public key to /root/.ssh/authorized_keys. Public key is usually located
+   at ~/.ssh/id_rsa.pub
+
+#### Executing tests remotely
+
+    test-runner --target root@192.168.0.44 --test_def automated/linux/smoke/smoke.yaml
+
+### Running a test plan
+
+Run a set of tests defined in agenda file:
+
+    test-runner -p ./plans/linux-example.yaml
+
+Apply test plan overlay to skip, amend or add tests:
+
+    test-runner -p ./plans/linux-example.yaml -O test-plan-overlay-example.yaml
+
+
+## Running manual tests
+test-runner also allows to execute and record results for manual tests.
+It has a built-in simple shell for test execution.
+
+```
+2020-06-18 10:15:03,619 - RUNNER: INFO: Tests to run:
+{'path': 'manual/generic/kernel-version.yaml', 'uuid': '4ce9c17b-52f6-4eb9-8804-278d6191b454', 'timeout': None, 'skip_install': False}
+2020-06-18 10:15:07,482 - RUNNER.TestSetup: INFO: Test repo copied to: /home/milosz/output/kernel-version_4ce9c17b-52f6-4eb9-8804-278d6191b454
+kernel-version
+
+        Welcome to manual test executor. Type 'help' for available commands.
+        This shell is meant to be executed on your computer, not on the system
+        under test. Please execute the steps from the test case, compare to
+        expected result and record the test result as 'pass' or 'fail'. If there
+        is an issue that prevents from executing the step, please record the result
+        as 'skip'.
+
+linux-kernel-version > help
+
+Documented commands (type help <topic>):
+========================================
+EOF      description  fail  next  quit  start
+current  expected     help  pass  skip  steps
+
+linux-kernel-version > description
+Test if the kernel version is correct
+linux-kernel-version > steps
+0. uname -a
+1. Check the output of the kernel version matches the version displayed on the build page
+linux-kernel-version > expected
+0. Kernel version matches the version on build page
+linux-kernel-version > current
+0. uname -a
+linux-kernel-version > next
+1. Check the output of the kernel version matches the version displayed on the build page
+linux-kernel-version > pass
+Recording pass in /home/milosz/output/kernel-version_4ce9c17b-52f6-4eb9-8804-278d6191b454/stdout.log
+2020-06-18 10:16:06,439 - RUNNER.ResultParser: WARNING: All parameters for qa reports are not set, results will not be pushed to qa reports
+2020-06-18 10:16:06,439 - RUNNER.ResultParser: INFO: Result files saved to: /home/milosz/output/kernel-version_4ce9c17b-52f6-4eb9-8804-278d6191b454
+--- Printing result.csv ---
+name,test_case_id,result,measurement,units,test_params
+linux-kernel-version,linux-kernel-version,pass,,,
+```
+
+### Running single test
+
+    test-runner --test_def manual/generic/kernel-version.yaml --kind manual
+
+## Executing manual tests from test plan
+When test plan contains manual tests, test-runned will execute them one-by-one
+in the built-in shell. Results will be recorder the same way as for automated tests
+
+    test-runner --test_plan plans/linux-test-plan-example.yaml --kind manual
+
+
+## Collecting result
+
+## Using test-runner
+test-runner needs a separate directory outside the repo to store test and result files.
+The directory defaults to `$HOME/output` and can be changed with `-o <dir>`. test-runner
+converts test definition file to `run.sh` and then parses its stdout. Results
+will be saved to results.{json,csv} by test. e.g.
+
+    /root/output/smoke_9879e7fd-a8b6-472d-b266-a20b05d52ed1/result.csv
+
+When using the same output directory for multiple tests, test-runner combines results
+from all tests and save them to `${OUTPUT}/results.{json,csv}`. e.g.
+
+    /root/output/result.json
+
+### Exporting test results to SQUAD (aka qa-reports)
+test-runner is now able to upload test results to SQUAD. Example below:
+
+    test-runner --test_def manual/generic/kernel-version.yaml \
+        --kind manual \
+        --qa-reports-server https://staging-qa-reports.linaro.org \
+        --qa-reports-project mwasilew-example \
+        --qa-reports-group people \
+        --qa-reports-env laptop \
+        --qa-reports-build-version 1 \
+        --qa-reports-token ${token}
+
+#### SQUAD metadata
+test-runner can also submit metadata as part of the results. Metadata is
+usually used to describe the versions of software under test and test suites.
+Following options for metadata upload are available:
+
+```
+--qa-reports-disable-metadata
+                    Disable sending metadata to SQUAD. Default: false
+--qa-reports-metadata KEY=VALUE [KEY=VALUE ...]
+                    List of metadata key=value pairs to be sent to SQUAD
+--qa-reports-metadata-file QA_REPORTS_METADATA_FILE
+                    YAML file that defines metadata to be reported to SQUAD
+```
+
+Metadata file should be of format:
+```
+key1: value1
+key2: value2
+```
diff --git a/mkdocs_plugin/requirements.txt b/mkdocs_plugin/requirements.txt
index 53d448e..0e74602 100644
--- a/mkdocs_plugin/requirements.txt
+++ b/mkdocs_plugin/requirements.txt
@@ -1,5 +1,5 @@
 mkdocs-test-definitions-plugin
-mdutils
+mdutils==1.0.0
 git+https://github.com/mwasilew/mkdocs-plugin-tags.git
 mkdocs-exclude
 mkdocs-material