summaryrefslogtreecommitdiff
path: root/jenkins-helpers.sh
AgeCommit message (Collapse)Author
2021-08-04print_arch_for_label: Add support for tcwg-armv8_32 and tcwg-armv8_64Christophe Lyon
Change-Id: I28f86ab1626beefe8a237da464640ccf9ef830a4
2021-07-26round-robin.sh (clone_repo): Fix checkout of branchesMaxim Kuvyrkov
Change-Id: I750be137aa978be5334bdce258d674804650c6da
2021-07-26start-container-docker.sh: Use at most half of RAM on normal buildersMaxim Kuvyrkov
This reverts commit 73ab5ac992942292817b1441472d325ed4c553be. Change-Id: Ia4148af0ac04a5f75b63d5c5af5a334939a39a03
2021-07-20jenkins-helpers.sh: Fix typoMaxim Kuvyrkov
Change-Id: I94ca9a921d7c84ce7a2a04c5e14a3c82bda96c17
2021-07-20round-robin-bisect.sh: Simplify handling of baseline repos/remotesMaxim Kuvyrkov
Create baseline remote only when we are about to push to them. Use baseline repos as normal repos for the rest of the build process. Change-Id: I17e6c57c051ae1b94921ccbb9ee1874fe0c41409
2021-07-20jenkins-helpers.sh (git_set_remote): New functionMaxim Kuvyrkov
Use it in clone_or_update_repo_no_checkout() and git_init_linaro_local_remote(). Change-Id: Ibc6292eaaa977a20ff8b0604fde2a5011d44d9e2
2021-07-15jenkins-helpers.sh: Update only requested remoteMaxim Kuvyrkov
... in clone_or_update_repo_no_checkout. Otherwise we risk failing due to some other remote being broken. Change-Id: Ifa9189d5206a0f34c67c0e7dd352bbc0ea3a38d4
2021-07-14jenkins-helpers.sh (git_init_linaro_local_remote): Add special case ...Maxim Kuvyrkov
... for gcc-compare-results.git, for which we want to use the main repo to generate flaky test lists. Change-Id: I7b2e7a0e7401aa4982eebd11658c119e33ee6750
2021-06-19jenkins-helpers.sh: Bind-mount benchmark git cache for bmk containersMaxim Kuvyrkov
Change-Id: I8ed5cc026f3a1685fd44d9ecc34507ad9df5a0f7
2021-02-25Error if value is not provided for '==arr[key]'.Prathamesh Kulkarni
Currently, the script gives error $2 is unbound, if value is not passed. The patch modifies it to emit an error and exit. Change-Id: I7a63d510edca4e96a49833fb6e965d14ccb6f52c
2021-01-14tcwg_bmk-build.sh: Rename ${rr[target]} from "arm-eabi" to "arm_eabi"Maxim Kuvyrkov
... so that we can correctly extract it from ${rr[ci_config]}, which splits fields on "-". At the moment we detect target as "arm" and pass "--target arm-linux-gnueabihf" to abe.sh. Change-Id: Ie2fd34e6d2b627181e6134ea55f31b0c5a0763b0
2020-11-30jenkins-helpers.sh: Remove llvm-reference from print_bind_mountsChristophe Lyon
llvm-reference is not obsolete, which should use the monorepo cache with is in snapshots-refs. Change-Id: I9e9455a9e2db42798a2146945c270012c966896f
2020-10-23print_host_for_node: Make "ignore_fail" optionalDiana Picus
We have at least one place (start-container-docker.sh) where print_host_for_node is called with only one argument. This causes jobs to fail with this error message: "$2: unbound variable". This patch tries to address the issue by using the empty expansion if the parameter is not set. Change-Id: I0d695a136829cc819e5cfc487b34f0fa6bb370bc
2020-09-11jenkins-helpers.sh: Split $EXPECTED_EXIT into $INTERNAL_FAILMaxim Kuvyrkov
... and $EXTERNAL_FAIL. This allows us to distinguish between regressions that we should bisect and infra problems that we should skip bisects for. Change-Id: I8f3b82c86d95e27351b01a2b3d2467b8e15b309e
2020-09-03tcwg_bmk-build.sh, jenkins-helpers.sh: Add support for arm-eabi targetChristophe Lyon
This is useful to run benchmarks on stm32 boards (Cortex-M, bare-metal). Change-Id: I115b982effea9d0ddebe990d74f8bb6c72806778
2020-08-31Increase memory allocation for build containersDiana Picus
Building flang needs more than 1GB per core and trying with 2GB per core surpasses the 30000 threshold. Use unlimited instead. Change-Id: Ibe1a06d5e529eaad9ea3b4a9d6c569a1171bed9c
2020-08-12jenkins-helpers.sh: Add "failsafe" parameter to print_host_for_nodeChristophe Lyon
When we call print_host_for_node from print_node_with_least_containers, we don't want to fail early with an error if any of the tester nodes is offline, so keep iterating until we find a suitable one. Change-Id: I0e6159a54f40ba83a56af03289ea905ed97cb7e9
2020-05-22tcwg_bmk-build.sh/tcwg_gnu-build.sh:David Spickett
Mark expected exits It is expected that commands run by run_step can fail and return non zero. This causes the build scripts to exit early and print misleading traceback. So that we only catch unexpected exits, mark expected failures with a magic return code. Only print traceback when the code is different from this. Change-Id: I8b92ca1cb941481d4daa8ab87ee74b2b394b24ac
2020-05-20jenkins-helpers.sh: Fix run_step starting step logicDavid Spickett
This if was originally written with -o -a but changed to || and && to fix shellcheck warning: https://github.com/koalaman/shellcheck/wiki/SC2166 Now we find out why shellcheck warns about that because the replacement expression was incorrect despite looking the same. This: X -o Y -a Z Was converted to: X || Y && Z Which does: (X || Y) && Z But the orignal behaviour is: X || (Y && Z) Change-Id: I0bebd9c008e605953c0af8d26df15d811816ff14
2020-05-19jenkins-helpers.sh: Handle corrupted repos in clone_or_update_repo_no_checkoutMaxim Kuvyrkov
In [1] we have a binutils repo corrupted, which prevents its update and most git operations. Git fetch returns: error: object file .git/objects/32/d1f47a126567f24be18e7cadf5d2490968c986 is empty error: object file .git/objects/32/d1f47a126567f24be18e7cadf5d2490968c986 is empty fatal: loose object 32d1f47a126567f24be18e7cadf5d2490968c986 (stored in .git/objects/32/d1f47a126567f24be18e7cadf5d2490968c986) is corrupt fatal: The remote end hung up unexpectedly This kind of problem happens due to Lab power outages, jenkins restarts, jenkins node container restarts, and other infrastructure artifacts. The solution is to remove a repo where a basic "git status" command fails. [1] https://ci.linaro.org/job/tcwg_kernel-bisect-gnu-master-arm-next-allyesconfig/78/artifact/artifacts/build-baseline/03-build_abe-binutils/console.log/*view*/ Change-Id: I33cd22787a62c3759a950f3a1cc84d923f2f370c
2020-05-18tcwg_bmk-build/tcwg_gnu-build: Add traceback printingDavid Spickett
The line number you get from the previous message is confusing. For example: ERROR at run_step:524 Means that something in the function run_step, which was called from line 524, called exit. Instead, add a Python style traceback that shows the call stack with file, line and the content of that line. (though unfortunatley we can't get the line no. of the exit itself) Example traceback: ERROR Traceback (most recent call last): File: /tmp/foo.sh, line 37 foo # hey! File: /tmp/foo.sh, line 32 bar # hi there File: /tmp/foo.sh, line 1 (trap handler, 'exit' call line is unknown) Change-Id: Id6a2734e2b5ef74d05cd51cdbde9a95e5d5a4b44
2020-05-18jenkins-helpers.sh: Fix log_url unbound errorDavid Spickett
local only sets scope it needs a value to actually exist. Change-Id: I6a790da6022b4ad95e1c5534b325f4c170131b74
2020-05-15jenkins-helpers.sh: Clarify HANDLING failure messageDavid Spickett
When a step fails and we carry on you get this: RUNNING <step> CARRYING ON after failure in <step> HANDLING previous failure in <step2> RUNNING <step2> Which in context can be read to mean that <step> failed then <step2> *also* failed. (especially if you don't know to expect another RUN message) Now we'll see: HANDLING <step2> will handle previous failure RUNNING <step2> Change-Id: I52da2ce8ea976524f366fb590c230303f027c644
2020-05-15jenkins-helpers.sh: Log run_step_init start and stop stepsDavid Spickett
So that we can figure out why some jobs are skipping every step. Change-Id: I097199a889372a7bc1195a9cc2f931cd507a43d7
2020-05-15jenkins-helpers.sh: Add log URL when running a stepDavid Spickett
Change-Id: I5e21cc78e7138054b4df4f16f2dd36a7d9e4ba62
2020-05-14tcwg_bmk/round-robin: Add assert_with_msg functionDavid Spickett
This acts like assert but prints a message if the assert fails. This will be visible in the Jenkins logs and give us a unique string to look for in the abscence of traceback for shell scripts. Two instances of returning false in check_regression have had echos added to them, instead of asserts. This is because they are not hard failures, and may be handled by the caller. (run_step) I have only converted the tcwg-bmk relevant scripts to assert_with_msg so far, as a proof of concept. Change-Id: I3b1badf09e4856c80b973ca8b23088626ed66b0a
2020-05-05jenkins-helpers.sh: Fix print_node_with_least_containersChristophe Lyon
We iterate over $testers as an array of node names, but it is defined as a space-separated string, so we currently iterate only once and $tester contains all the relevant node names instead of only one at a time. This is silent because print_host_for_node considers only its first parameter, which happens to be the first node in the list. This is fine until that node is offline, so we exit the loop with an error during the first iteration. Fix by using $testers as a string variable. Change-Id: I9ffe1253894697f7324d43170e7bf48b566e324b
2020-05-05jenkins-helpers.sh: Add pwd check functionDavid Spickett
Used in other scripts to check that pwd isnt / before deleting things. Change-Id: Id3e284a486f300c97e92f730e84183340c999dd9
2020-04-29tcwg-benchmark.sh: Remove unused @hw_type@ substitution from $results_idMaxim Kuvyrkov
Change-Id: I6530286516fc131980e4379399ca0fff00883f8f
2020-04-28Shellcheck fixes round 3David Spickett
Now using the version that's present in the bionic images. Meaning we have more warnings but can also set the minimum level to ignore a lot of them. Fix the check for whether shellcheck supports --severity. (we have pipefail on, but we expect shellcheck to fail in this case) Change-Id: I296b8554591b7d327c75393e3924184abc6512b7
2020-04-27jenkins-helpers.sh: Rework print_host_for_node to rely only on sshMaxim Kuvyrkov
Which gives us control over TCWG "DNS" when combined with https://review.linaro.org/c/ci/dockerfiles/+/34893 . Change-Id: Ia9b483f195936fa27422ecaa24a9c3d16c1b446a
2020-04-23Shellcheck fixes round 2David Spickett
Also update the sanity check script to include .job files. Change-Id: Ibf72d9ff346ff7b012d4e282a6ac8f3b7e73590d
2020-04-20Shellcheck warning fixesDavid Spickett
(all error/style level warnings) SC1035: missing whitespace SC2068: double quote array expansions SC2145: argument mixes string and array SC1066: Don't use $ on the left side of assignments SC2126: Use grep -c instead of grep|wc Change-Id: Ia53070d3e2c9aea9f626fdb7b2b5bfc9e1f8e682
2020-03-11Add annotations to round-robin's "results" fileMaxim Kuvyrkov
Annotations provide human-readable comments and data on regressions. For this we ignore all lines in "results" file starting with "#". Change-Id: I31328fe2c13895a7379137b00891a9d2fc0a39d7
2020-02-06jenkins-helpers.sh: UpdateMaxim Kuvyrkov
Change-Id: I27f7ec387612258fb326b85ba0048cba72743816
2019-10-14jenkins-helpers.sh: Optimize disk I/O for tcwg_bmk-* jobsMaxim Kuvyrkov
Change-Id: I719bc32c69c9c5d8ef726b7096b70e1303812107
2019-10-14jenkins-helpers.sh: Increase ssh timeout in print_node_with_least_containersMaxim Kuvyrkov
... to give SSSD more time to gather itself. Change-Id: I8c2f0bb767590a3c95ea2424ab66a31ce9e0d6ca
2019-10-11jenkins-helpers.sh: Add a timeout for "ssh $board true" in wait_for_ssh_serverMaxim Kuvyrkov
I've noticed a strange failure in tcwg-benchmark script/job. When trying to reboot tcwg-tk1-03 the board got wedged (kernel soft lockup), and reboot was hanging. In this case the command, "ssh tcwg-tk1-03.tcwglab true" was also stuck for several hours. Avoid such hangs by adding a timeout. If we see more kernel lockups TK1s/TX1s on reboots, we'll add PDU power-cycling to the scripts. Change-Id: I1f0cdbf6ddb6a1b192b6b5081cc49889e5a62364
2019-10-11jenkins-helpers.sh: Fix creation of testing containers on tcwg-apm-*.tcwglabMaxim Kuvyrkov
Tcwg-apm-*.tcwglab machines (and, generally, most *.tcwglab configured by Lab) tend to use "dash" shell for infrastructure users tcwg-buildslave and tcwg-benchmark. Dash does not expand "*" wildcard, so we don't get ssh host keys inside containers, which cause ssh to fail. Fix by using more portable "find". Change-Id: I08127c973ce0c0f956465d4706ae1d3769888c54
2019-10-11jenkins-helpers.sh: Handle paths in run_step's argumentsMaxim Kuvyrkov
When we pass ${rr[top_artifacts]}/results_id to "run_step ... benchmark" we end up with artifacts directory for this step something like artifacts/11-benchmark-.../home/$USER/<path>/.../results_id/ -- because we include all arguments to the run_step in the pretty name of the step. This patch makes $pretty_step name absorb arguments up to "--" in its name, and also flattens any paths by changing "/" to "-". Change-Id: I700818409337b9884284d785641edae4db419462
2019-09-14tcwg-benchmark.sh: Bind-mount $WORKSPACE inside benchmarking containerMaxim Kuvyrkov
... to enable removal of stale/aborted benchmarking containers via $WORKSPACE/.lock cleanup. Stale benchmarking containers are dangerous due to perf workaround running inside, which steals samples from the "measuring" perf. Change-Id: I54dfbb42e5d04ff49c9bfb591df681cf16c5b781
2019-09-13Run benchmarks inside containersDiana Picus
This creates a new task ('bench') and starts a docker container to run the benchmarks in. It uses the same image as the 'build' task. Switching between aarch64 and aarch32 benchmarking breaks unless force install is requested because we only keep one version of the spec tools on the board. This will be fixed in a subsequent patch to bmk-scripts. Change-Id: Ie98cdf47aa3ed456e6349bac40d88bd7acaa7acd
2019-09-10jenkins-helpers.sh: Don't require $WORKSPACE set for "build" containersMaxim Kuvyrkov
It is useful to start "build" containers by developers for investigations without $WORKSPACE set. Change-Id: I7cd8decb5e2f05c676a60dcab0954d72c1e0851e
2019-09-02jenkins-helpers.sh: Fix print_bind_mounts()Maxim Kuvyrkov
... by looking at the right filesystem. Change-Id: I42e3a5d942b469e983a84e09fc7a11856b81abde
2019-08-30jenkins-helpers.sh: Bind-mount ssh host keysMaxim Kuvyrkov
Change-Id: I86b32b378afe82207211297d3cc4ea66e6f4c9ad
2019-08-29start-container-docker.sh: Fix container startup on remote targetMaxim Kuvyrkov
Fix conversion of bind-mount paths for remote host-container-enabled targets. At the moment we check whether current machine is host-container-enabled (and convert paths), rather than the target. Change-Id: Iccb48cc9533d9674a00064c55c47751f8e115ffc
2019-08-08jenkins-helpers.sh: Add entries for tcwg-amp-* machinesMaxim Kuvyrkov
Change-Id: I24dee66c8d0b0b585420096cfd309d0472f13c54
2019-05-21jenkins-helpers.sh: Explicitly delete .git/gc.logMaxim Kuvyrkov
Turns out that "--prune=all" is not enough to get rid of .git/gc.log. This file prevents "git gc", so remove it. Change-Id: I74c3cd96fca19ef9718506e585df71339f4523e7
2019-05-14jenkins-helpers: print_arch_for_label: Sort labels for easier maintenanceChristophe Lyon
Change-Id: I770158a3e1c7acc900093b8b722b36b2c220ebba
2019-05-14jenkins-helpers.sh: print_arch_for_label() Add support for tcwg-lc_64*Christophe Lyon
Change-Id: I586372c437f26b8a029e0b26b5a08d9401dca9fd
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-28 04:52:59 +0000