From e01e9386133050ad79bab2a757ca70d645f5091e Mon Sep 17 00:00:00 2001 From: Chase Qi Date: Mon, 27 Feb 2017 18:34:02 +0800 Subject: automated: doc: fix typo in filename Change-Id: Iaa4119406790c36215cbb84f108903342f8b3e11 Signed-off-by: Chase Qi --- automated/doc/test-writing-guidelines.rst | 239 +++++++++++++++++++++++++++++ automated/doc/test-writting-guidelines.rst | 239 ----------------------------- 2 files changed, 239 insertions(+), 239 deletions(-) create mode 100644 automated/doc/test-writing-guidelines.rst delete mode 100644 automated/doc/test-writting-guidelines.rst diff --git a/automated/doc/test-writing-guidelines.rst b/automated/doc/test-writing-guidelines.rst new file mode 100644 index 0000000..6a84562 --- /dev/null +++ b/automated/doc/test-writing-guidelines.rst @@ -0,0 +1,239 @@ +======================= +Test Writing Guidelines +======================= + +This document describes guidelines and is intended for anybody who wants to write +or modify a test case. It's not a definitive guide and it's not, by any means, a +substitute for common sense. + +General Rules +============= + +1. Simplicity +------------- + +It's worth keeping test cases as simple as possible. + +2. Code duplication +------------------- + +Whenever you are about to copy a large part of the code from one test case to +another, think if it is possible to move it to a library to reduce code +duplication and the cost of maintenance. + +3. Coding style +--------------- + +Use common sense and BE CONSISTENT. + +If you are editing code, take a few minutes to look at the code around you and +determine its style. + +The point of having style guidelines is to have a common vocabulary of coding so +people can concentrate on what you are saying, rather than on how you are saying +it. + +3.1 Shell coding style +~~~~~~~~~~~~~~~~~~~~~~ +When writing test cases in shell write in *portable shell* only. + +You can either try to run the test cases on Debian which has '/bin/sh' pointing +to 'dash' by default or install 'dash' on your favorite distribution and use +it to run the tests. + +Ref: `Shell Style Guide `_ + +3.2 Python coding style +~~~~~~~~~~~~~~~~~~~~~~~ +Please follow PEP 8 style guide whenever possible. + +Ref: `PEP 8 `_ +Easy-to-read version of PEP 8 available at `pep8.org `_ + +4. Commenting code +------------------ + +Use useful comments in your program to explain: + + * assumptions + * important decisions + * important details + * problems you're trying to solve + * problems you're trying to overcome in your program, etc. + +Code tells you how, comments should tell you why. + +5. License +---------- +Code contributed to this repository should be licensed under GPLv2+ (GNU GPL +version 2 or any later version). + +Writing a test case +=================== + +Linux +------ + +1. Structure +~~~~~~~~~~~~ + +Tests are generally placed under 'linux/' directory. Everything that relates to +the test goes under the same folder named with test case name. + +Define 'linux/test-case-name/output' folder in test case to save test output and +result. Using a dedicated folder is helpful to distinguish between test script +and test output. + +2. Installing dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The same test case should work on Debian/Ubuntu, Fedora/CentOS and OE based +distributions whenever possible. This can be achieved with install_deps() +function. The following is a simple example. "${SKIP_INSTALL}" should be set to +'true' on distributions that do not supported install_deps(). In the unsupported +case, if "${SKIP_INSTALL}" is 'true', install_deps() still will skip package +installation. + +Example 1:: + + install_deps "${pkgs}" "${SKIP_INSTALL}" + +Package name may vary by distribution. In this case, you will need to handle +package installation with separate lines. dist_name() function is designed to +detect the distribution ID at running time so that you can define package name +by distribution. Refer to the following example. + +Example 2:: + + dist_name + case "${dist}" in + debian|ubuntu) install_deps "lsb-release" "${SKIP_INSTALL}" ;; + fedora|centos) install_deps "redhat-lsb-core" "${SKIP_INSTALL}" ;; + *) warn_msg "Unsupported distro: ${dist}! Package installation skipped." ;; + esac + +Except automated package installation, you may also need to download and install +software manually. If you want to make these steps skippable, here is an +example. + +Example 3:: + + if [ "${SKIP_INSTALL}" = "true" ] || [ "${SKIP_INSTALL}" = "True" ]; then + dist_name + case "${dist}" in + debian|ubuntu) install_deps "${pkgs}" ;; + fedora|centos) install_deps "${pkgs}" ;; + *) warn_msg "Unsupported distro: ${dist}! Package installation skipped." ;; + esac + + # manually install steps. + git clone "${repo}" + cd "${dir}" + ./configure && make install + fi + +Hopefully, the above 3 examples cover most of the user cases. When +writing test cases, in general: + + * Define 'SKIP_INSTALL' variable with 'false' as default. + * Add parameter '-s ', so that user can modify 'SKIP_INSTALL'. + * Try to use the above functions, and give unknown distributions more care. + +3. Saving output +~~~~~~~~~~~~~~~~~ + +'test-case-name/output' directory is recommended to save test log and result +files. + +4. Parsing result +~~~~~~~~~~~~~~~~~ + +Saving parsed result in the same format is important for post process such as +sending to LAVA. The following result format should be followed. + + test-caes-id pass/fail/skip + test-case-id pass/fail/skip measurement + test-case-id pass/fail/skip measurement units + +'output/result.txt' file is recommended to save result. + +We encourage test writers to use the functions defined in 'sh-test-lib' to format +test result. + +Print "test-case pass/fail" by checking exit code:: + + check_return "${test_case_id}" + +Add a metric for performance test:: + + add_metic "${test-case-id}" "pass/fail/skip" "${measurement}" "${units}" + + +5. Running in LAVA +~~~~~~~~~~~~~~~~~~ + +LAVA is the foundation of test automation in Linaro. It is able to handle image +deployment and boot, and provides a test shell for test run. To run a test case +in LAVA, a definition file in YAML format is required. + +Bear in mind, do all the LAVA-specific steps in test definition file, and do not +use any LAVA-specific steps in test script, otherwise you may lock yourself out +of your own test case when LAVA isn't available or the board you want to test +wasn't deployed in LAVA. + +Test script should handle dependencies installation, test execution, result +parsing and other work in a self-contained way, and produce result.txt file with +a format that can be easily parsed and sent to LAVA. This is a more robust way. +Test case works with/without LAVA and can be tested locally. + +A general test definition file should contain the below keywords and steps:: + + metadata: + # Define parameters required by test case with default values. + params: + SKIP_INSTALL: False + run: + # A typical test run in LAVA requires the below steps. + steps: + # Enter the directory of the test case. + - cd ./automated/linux/smoke/ + # Run the test. + - ./smoke.sh -s "${SKIP_INSTALL}" + # Send the results in result.txt to LAVA. + - ../../utils/send-to-lava.sh ./output/result.txt + +Android specific +---------------- + +The above test writing guidelines also apply to Android test cases. The major +difference is that we run all Android test cases through adb shell. Compare with +local run, adb and adb shell enable us to do more. And this model is well +supported by LAVA V2 LXC protocol. + +A typical Android test case can be written with the following steps:: + + # Check adb connect with initialize_adb funtion + initialize_adb + # Install binaries and scripts + detect_abi + install "../../bin/${abi}/busybox" + install "./device-script.sh" + # Run test script through adb shell. + adb -s "${SN}" shell device-script.sh + # Pull output from device for parsing. + pull_output "${DEVICE_OUTPUT}" "${HOST_OUTPUT}" + +Test Contribution Checklist +=========================== + +* When applicable, check test cases with the following tools with line length + rule relaxed. + + - checkbashisms - check for bashisms in /bin/sh scripts. + - shellcheck - Shell script analysis tool. + - pep8 - check Python code against the style conventions in PEP 8. + - pyflakes - simple Python 2 source checker + - pylint - code analysis for Python + +* Run test cases on local system without LAVA. +* Optionally, run test cases in LAVA and provide job example. diff --git a/automated/doc/test-writting-guidelines.rst b/automated/doc/test-writting-guidelines.rst deleted file mode 100644 index 6a84562..0000000 --- a/automated/doc/test-writting-guidelines.rst +++ /dev/null @@ -1,239 +0,0 @@ -======================= -Test Writing Guidelines -======================= - -This document describes guidelines and is intended for anybody who wants to write -or modify a test case. It's not a definitive guide and it's not, by any means, a -substitute for common sense. - -General Rules -============= - -1. Simplicity -------------- - -It's worth keeping test cases as simple as possible. - -2. Code duplication -------------------- - -Whenever you are about to copy a large part of the code from one test case to -another, think if it is possible to move it to a library to reduce code -duplication and the cost of maintenance. - -3. Coding style ---------------- - -Use common sense and BE CONSISTENT. - -If you are editing code, take a few minutes to look at the code around you and -determine its style. - -The point of having style guidelines is to have a common vocabulary of coding so -people can concentrate on what you are saying, rather than on how you are saying -it. - -3.1 Shell coding style -~~~~~~~~~~~~~~~~~~~~~~ -When writing test cases in shell write in *portable shell* only. - -You can either try to run the test cases on Debian which has '/bin/sh' pointing -to 'dash' by default or install 'dash' on your favorite distribution and use -it to run the tests. - -Ref: `Shell Style Guide `_ - -3.2 Python coding style -~~~~~~~~~~~~~~~~~~~~~~~ -Please follow PEP 8 style guide whenever possible. - -Ref: `PEP 8 `_ -Easy-to-read version of PEP 8 available at `pep8.org `_ - -4. Commenting code ------------------- - -Use useful comments in your program to explain: - - * assumptions - * important decisions - * important details - * problems you're trying to solve - * problems you're trying to overcome in your program, etc. - -Code tells you how, comments should tell you why. - -5. License ----------- -Code contributed to this repository should be licensed under GPLv2+ (GNU GPL -version 2 or any later version). - -Writing a test case -=================== - -Linux ------- - -1. Structure -~~~~~~~~~~~~ - -Tests are generally placed under 'linux/' directory. Everything that relates to -the test goes under the same folder named with test case name. - -Define 'linux/test-case-name/output' folder in test case to save test output and -result. Using a dedicated folder is helpful to distinguish between test script -and test output. - -2. Installing dependencies -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The same test case should work on Debian/Ubuntu, Fedora/CentOS and OE based -distributions whenever possible. This can be achieved with install_deps() -function. The following is a simple example. "${SKIP_INSTALL}" should be set to -'true' on distributions that do not supported install_deps(). In the unsupported -case, if "${SKIP_INSTALL}" is 'true', install_deps() still will skip package -installation. - -Example 1:: - - install_deps "${pkgs}" "${SKIP_INSTALL}" - -Package name may vary by distribution. In this case, you will need to handle -package installation with separate lines. dist_name() function is designed to -detect the distribution ID at running time so that you can define package name -by distribution. Refer to the following example. - -Example 2:: - - dist_name - case "${dist}" in - debian|ubuntu) install_deps "lsb-release" "${SKIP_INSTALL}" ;; - fedora|centos) install_deps "redhat-lsb-core" "${SKIP_INSTALL}" ;; - *) warn_msg "Unsupported distro: ${dist}! Package installation skipped." ;; - esac - -Except automated package installation, you may also need to download and install -software manually. If you want to make these steps skippable, here is an -example. - -Example 3:: - - if [ "${SKIP_INSTALL}" = "true" ] || [ "${SKIP_INSTALL}" = "True" ]; then - dist_name - case "${dist}" in - debian|ubuntu) install_deps "${pkgs}" ;; - fedora|centos) install_deps "${pkgs}" ;; - *) warn_msg "Unsupported distro: ${dist}! Package installation skipped." ;; - esac - - # manually install steps. - git clone "${repo}" - cd "${dir}" - ./configure && make install - fi - -Hopefully, the above 3 examples cover most of the user cases. When -writing test cases, in general: - - * Define 'SKIP_INSTALL' variable with 'false' as default. - * Add parameter '-s ', so that user can modify 'SKIP_INSTALL'. - * Try to use the above functions, and give unknown distributions more care. - -3. Saving output -~~~~~~~~~~~~~~~~~ - -'test-case-name/output' directory is recommended to save test log and result -files. - -4. Parsing result -~~~~~~~~~~~~~~~~~ - -Saving parsed result in the same format is important for post process such as -sending to LAVA. The following result format should be followed. - - test-caes-id pass/fail/skip - test-case-id pass/fail/skip measurement - test-case-id pass/fail/skip measurement units - -'output/result.txt' file is recommended to save result. - -We encourage test writers to use the functions defined in 'sh-test-lib' to format -test result. - -Print "test-case pass/fail" by checking exit code:: - - check_return "${test_case_id}" - -Add a metric for performance test:: - - add_metic "${test-case-id}" "pass/fail/skip" "${measurement}" "${units}" - - -5. Running in LAVA -~~~~~~~~~~~~~~~~~~ - -LAVA is the foundation of test automation in Linaro. It is able to handle image -deployment and boot, and provides a test shell for test run. To run a test case -in LAVA, a definition file in YAML format is required. - -Bear in mind, do all the LAVA-specific steps in test definition file, and do not -use any LAVA-specific steps in test script, otherwise you may lock yourself out -of your own test case when LAVA isn't available or the board you want to test -wasn't deployed in LAVA. - -Test script should handle dependencies installation, test execution, result -parsing and other work in a self-contained way, and produce result.txt file with -a format that can be easily parsed and sent to LAVA. This is a more robust way. -Test case works with/without LAVA and can be tested locally. - -A general test definition file should contain the below keywords and steps:: - - metadata: - # Define parameters required by test case with default values. - params: - SKIP_INSTALL: False - run: - # A typical test run in LAVA requires the below steps. - steps: - # Enter the directory of the test case. - - cd ./automated/linux/smoke/ - # Run the test. - - ./smoke.sh -s "${SKIP_INSTALL}" - # Send the results in result.txt to LAVA. - - ../../utils/send-to-lava.sh ./output/result.txt - -Android specific ----------------- - -The above test writing guidelines also apply to Android test cases. The major -difference is that we run all Android test cases through adb shell. Compare with -local run, adb and adb shell enable us to do more. And this model is well -supported by LAVA V2 LXC protocol. - -A typical Android test case can be written with the following steps:: - - # Check adb connect with initialize_adb funtion - initialize_adb - # Install binaries and scripts - detect_abi - install "../../bin/${abi}/busybox" - install "./device-script.sh" - # Run test script through adb shell. - adb -s "${SN}" shell device-script.sh - # Pull output from device for parsing. - pull_output "${DEVICE_OUTPUT}" "${HOST_OUTPUT}" - -Test Contribution Checklist -=========================== - -* When applicable, check test cases with the following tools with line length - rule relaxed. - - - checkbashisms - check for bashisms in /bin/sh scripts. - - shellcheck - Shell script analysis tool. - - pep8 - check Python code against the style conventions in PEP 8. - - pyflakes - simple Python 2 source checker - - pylint - code analysis for Python - -* Run test cases on local system without LAVA. -* Optionally, run test cases in LAVA and provide job example. -- cgit v1.2.3