summaryrefslogtreecommitdiff
path: root/jenkins-helpers.sh
AgeCommit message (Collapse)Author
2019-08-30jenkins-helpers.sh: Bind-mount ssh host keysMaxim Kuvyrkov
Change-Id: I86b32b378afe82207211297d3cc4ea66e6f4c9ad
2019-08-29start-container-docker.sh: Fix container startup on remote targetMaxim Kuvyrkov
Fix conversion of bind-mount paths for remote host-container-enabled targets. At the moment we check whether current machine is host-container-enabled (and convert paths), rather than the target. Change-Id: Iccb48cc9533d9674a00064c55c47751f8e115ffc
2019-08-08jenkins-helpers.sh: Add entries for tcwg-amp-* machinesMaxim Kuvyrkov
Change-Id: I24dee66c8d0b0b585420096cfd309d0472f13c54
2019-05-21jenkins-helpers.sh: Explicitly delete .git/gc.logMaxim Kuvyrkov
Turns out that "--prune=all" is not enough to get rid of .git/gc.log. This file prevents "git gc", so remove it. Change-Id: I74c3cd96fca19ef9718506e585df71339f4523e7
2019-05-14jenkins-helpers: print_arch_for_label: Sort labels for easier maintenanceChristophe Lyon
Change-Id: I770158a3e1c7acc900093b8b722b36b2c220ebba
2019-05-14jenkins-helpers.sh: print_arch_for_label() Add support for tcwg-lc_64*Christophe Lyon
Change-Id: I586372c437f26b8a029e0b26b5a08d9401dca9fd
2019-04-27jenkins-helpers.sh: Avoid inconsistent state of remote by re-creating itMaxim Kuvyrkov
We can get into state when remote has no "fetch" configured, and "git remote set-branches" fails. Make repo updates more robust by recreating remote entries. Also makes for simpler code. Change-Id: I18b830ac177bac1917ebfc4c4bdcd7bb5aa60f57
2019-04-27jenkins-helpers.sh: Add close_baseline_repo helperMaxim Kuvyrkov
... and use it in round-robin.sh:clone_repo(). Change-Id: I6d10cc70430f1a834efb7d4b438c939aa61a6568
2019-04-27jenkins-helpers.sh: Generalize clone_or_update_repo*Maxim Kuvyrkov
... to handle non-origin remotes. This is a prerequisite for next change. Change-Id: Ic961c25c089e853f53423eb99dff1036c147baa9
2019-04-18jenkins-helpers.sh: Fix ccache pollution on high-executor-count nodesMaxim Kuvyrkov
Change-Id: Idc60ff9e1ce6a2db8a36a8c7352c1c132913156f
2019-04-16jenkins-helpers.sh: Improve handling of git errors in run_with_timeout_and_retryChristophe Lyon
When 'git clone' fails, it can leave a directory with inconsistent state, thus making 'git clone' retries fail immediately, which effectively makes the retry mechanism useless. This patch identifies 'git clone' cases and removes the target directory in case of error. Change-Id: I6b1feb1251cf321b95d217c661e82b3e3c1c2c4d
2019-03-27jenkins-helpers.sh: run_step: Use 'ts'Christophe Lyon
To add timestamps in console.log files. This will hopefully help finding which steps use most of the time. Change-Id: I25cd487f9ddee619cb8f6adb81777541ee9bf959
2019-03-25docker-run.sh, jenkins-helpers.sh: Fix use of flockChristophe Lyon
flock expects a real command as argument, not a shell function, so the scripts are currently broken: flock: failed to execute container_exec: No such file or directory This patch defines the JENKINS_FLOCK helper variable which is expanded by the remote_exec() shell helper. Change-Id: Ie5117d1116c9bbc9b556bc25946a7a0afc706ea2
2019-03-21start-container-docker.sh, jenkins-helpers.sh: Add --ssh_info optionChristophe Lyon
The new boolean --ssh_info option of start-container-docker.sh enables passing ssh_host and ssh_port variables to the command executed in the container via the environment. The remote_exec helper is enhanced to support passing environment variables. Change-Id: Ic068220d52541ada2ea6b4a26f40730788babf9d
2019-03-21jenkins-helpers.sh: Don't touch step's artifacts directory unnecessarilyMaxim Kuvyrkov
Change-Id: I00df2aaa065377884b60360060ed2234e6555e3a
2019-03-19Patch [1/3] Merge tcwg_gnu: jenkins-helpers.sh - s/tcwg_kernel-*/tcwg_*-*.Prathamesh Kulkarni
Change-Id: Ibc6dcd085f684eb6f09ec9a512aa5abd6b9d8200
2019-03-13print_nodes_in_labels: Handle empty labelsChristophe Lyon
Return an empty list instead of an error in case the label contains no slave. Change-Id: I5b1561dd86d62347b1db59b3204c1c955d181000
2019-03-06Use APM boards for cross-testing.Christophe Lyon
In print_tester_label_for_target(), do not use tk1/tx1 anymore, use APMs instead. However, this means we no longer use armv7 hardware to test arm-* configurations (as opposed to armv8l*). Change-Id: Iff08678c39ad15f59628e5a91f62d0812d0bb0b3
2019-02-26tcwg-dev-build: Enable builds using manifestsMaxim Kuvyrkov
If manifest.txt was uploaded in job's parameters, then pass --manifest option to MakeRelease.job. Fix MakeRelease.job to not pass --target when manifest is set, otherwise ABE complains. Change-Id: I2aa25d3840281159e2ed00cfaab3b9e47a661801
2019-02-22jenkins-helpers.sh: Simplify run_step* functionsMaxim Kuvyrkov
Remove copying of entries from base-artifacts/STEP/ to artifacts/STEP/ for skipped steps. We used to rely on $run_step_prev_artifacts/build-ok markers to detect pass/fail conditions of the previous step -- something that we now explicitly handle with skip_/reset_/stop_on_on_fail modes of run_step(). So remove unused feature. Change-Id: I06e4298aa06ffb2aaf677d2c51e89da61ae9b119
2019-02-19jenkins-helpers.sh: Remove stale locks in git repos.Maxim Kuvyrkov
We get locks in git repo when builds are interrupted during git remote update. Change-Id: I184f6e0f4e401237d85f0e0af328a67be654faec
2019-02-19convert_args_to_variables: Document "==" syntaxDiana Picus
Add a comment explaining "==arr[key] value" syntax for parameters. Change-Id: Id18bd5fc912297cd98e55161f8aa1fd30532e706
2019-02-14Set run_step_status to 0 before running step.Prathamesh Kulkarni
run_step_status is set to exit status of step only if it fails, else it holds the value of exit status of previous step. Avoid that by initializing it to 0. Change-Id: I1e7625ffc22ad2a44470892b11092e80f2b52eae
2019-02-08jenkins-helpers.sh: Workaround DNS issueMaxim Kuvyrkov
We don't have DNS alias entries tcwg-x86_64-build-09.tcwglab and tcwg-x86_64-build-10.tcwglab -- for build-09.tcwglab and build-10.tcwglab machines. Change-Id: Ib32956b92098ac610be6f178c5d24c23ffa994f0
2019-02-08jenkins-helpers.sh: Support custom docker root dirsMaxim Kuvyrkov
Several of our machines have docker root dir set to /home/docker instead of standard /var/lib/docker. The reason for using /home/docker is to allocate docker containers on the biggest disk partition. Change-Id: I6d16f561b5b8d303cb58d3a30f2847814eb55756
2019-02-07jenkins-helpers.sh: Improve remote_execMaxim Kuvyrkov
Don't add "exec" to the remote command unless necessary. We need this to trigger jenkins jobs via === remote_exec "ci.linaro.org:2222::-l $USER@linaro.org" build ... === Note that "build" is a command of jenkins-cli, not an executable, so prepending "exec" breaks the invocation. Change-Id: If122869dadb76711666aa49690f508757e7ebce9
2019-01-22tcwg_kernel-bisect.sh: Avoid absolute paths in manifestsMaxim Kuvyrkov
... and their use in reproduction instructions. Also - download build-parameter.sh in instructions that test.sh relies on - use curl instead of wget in instuctions - limit commit output to 1000 lines in the email body. Change-Id: Ib3b0cb94134a588e3a8c72fd7261b70d83935450
2019-01-22jenkins-helpers.sh: Improve run_step functionality.Maxim Kuvyrkov
This is partly fix, partly cleanup. At the moment in bisection mode artifacts/results are not populated until first step successfully executes. This means that for bisections of binutils, gcc, and llvm regressions the results file is empty if the build of these projects fails (i.e., no successful steps at all). This causes empty clauses in the bisection emails due to artifacts/results file being empty. We could have fixed this by adding === Starting Y build <value of successful previous step> === to the top of every build step, but instead we refactor and cleanup run_step support to handle constant-value result entries. In addition to current "stop_on_fail" mode we add "skip_on_fail" and "reset_on_fail" modes, which simplify error handling for most build steps. Change-Id: Ic2d1a15fa490ec25be741cdaf718442260c52e6d
2019-01-21round-robin.h: Record correct sha1 of base-artifacts/ in manifestMaxim Kuvyrkov
Git_rev_parse resolves references against refs/remotes/origin/$ref, which is what we want most of the time, but not for HEAD. Refs/remotes/origin/HEAD will, typically return sha1 of repo's master branch, not sha1 of current checkout. The fix is to use plain "git rev-parse". A future patch will rename git_rev_parse to git_remote_rev_parse or something similar. Change-Id: Ic3656ee9880862608928a7e6081fe6559e64c70e
2019-01-21jenkins-helpers: Enable clone_or_update_repo to fetch single branchMaxim Kuvyrkov
... to optimize clone and updates of base-artifacts. The base-artifacts repo has a lot of contineously re-written branches, but these branches are only a few commits deep (they store "stack" of current regressions). We want to fetch only the branch of base-artifacts for configuration we are testing, not the whole repo. This patch adds new parameter to clone_or_update_repo and clone_or_update_repo_no_checkout. Change-Id: I286a78e167ddc2e34a770fb6f8b60323a14fe8c6
2019-01-20jenkins-helpers.sh: Fix fresh_dir and use it in git_cleanMaxim Kuvyrkov
Find races with "rm -rf" and complains about missing files and directories that were just deleted. Ignore find's exit status and stderr. Change-Id: Ie79b9ef56e4932a694cdee16b31384efd99f1c1f
2019-01-17jenkins-helpers.sh: Switch to official LLVM git mirrorMaxim Kuvyrkov
Change-Id: Iea3db4df798980e53ae6fa34d56596ba4b6198b4
2019-01-17jenkins-helpers.sh: Fix "git gc" in clone_or_update_repoMaxim Kuvyrkov
Change-Id: I8f4711912ebf18504ae7cad79c755e482610f840
2019-01-16jenkins-helpers.sh: Speed-up "git gc" in clone_or_update_repoMaxim Kuvyrkov
Run "git gc" in --auto mode to skip repos which are in good shape. Change-Id: I3cd1adffda2736dd1f37d8460f1cf905a537f55e
2019-01-15jenkins-helpers.sh: Implement stack logic for __manifest_fileMaxim Kuvyrkov
In tcwg_kernel-bisect.sh we use manifest handling to put options for tcwg_kernel-build.sh script into build-parameters.sh file. This overwrites __manifest_file variable of bisect job, so we end up forwarding half of bisect manifest into build-parameters.sh. Fix this by adding push/pop logic for __manifest_file variable. Change-Id: I22adc9f0533024063398b5d7cb1a416cad2aa2d1
2019-01-15jenkins-helpers.sh: Workaround stale "git gc" lock.Maxim Kuvyrkov
If build times out during "git gc" or "git gc" is otherwise killed, then we have to clean up a stale lock. Change-Id: I2c60b0a35a0881c2b5b893f6f925ffab5f88fcd4
2019-01-13tcwg_kernel-bisect.sh: Add support to reproduce bisects.Maxim Kuvyrkov
Change-Id: Ie2e0f6db672207123a9903eb7f60de23f316f60a
2019-01-13jenkins-helpers.sh: New helper fresh_dirMaxim Kuvyrkov
Change-Id: I1513712e64500712a5540b9b5a5bd8e1228334c4
2019-01-11jenkins-helpers.sh: Run "git gc" when updating git repos.Maxim Kuvyrkov
This avoids git repos from growing too large. Change-Id: I2cd9b83180ca13557a0aa5558d5019a98b81833b
2019-01-11jenkins-helpers.sh: Add "^^ true/false" switch to manifest handlingMaxim Kuvyrkov
In tcwg_kernel-bisect.sh we need to call tcwg_kernel-build.sh with various options. We also generate manifest for each call so that we can reproduce /bisect/ runs. With current support we need to write something like this to handle reproductions of bisects: === if $reproduce_bisect; then ./tcwg_kernel-build.sh @@ /path/to/manifest.sh else ./tcwg_kernel-build.sh %% /path/to/manifest.sh --build options fi === which is cumbersome and obstructs script logic. This patch adds a "reproduction switch" -- "^^ true/false". When "^^ true %% manifest.sh --build options" it provided, then manifest.sh is sourced and all "--build options" are discarded to avoid changing build environment that manifest.sh sets up. When "^^ false %% manifest.sh --build options" is given, then manifest.sh is created and populated with "--build options". Change-Id: I38fcf37dc024891f5aa858dbd110082692dc09c9
2019-01-11jenkins-helpers.sh: Include contents of sourced config files in manifestMaxim Kuvyrkov
Change-Id: I12bfdcbc182ed807a6ea06f464122e9a8cecaa19
2019-01-11jenkins-helpers.sh: Create directory for manifest file.Maxim Kuvyrkov
Change-Id: I2a430189c09e5f1a4bcb51ecdad32e218ab7b844
2019-01-07git_init_linaro_local_remote to use toolchain/ci/* instead of the same repoKugan Vivekanandarajah
Change-Id: Iffb597ab8dbf676779c3a2135baf9d2f536cc808
2018-12-18tcwg_kernel-build.sh: Improve checking of parameters.Maxim Kuvyrkov
Change-Id: Iba94ba9ea421f2dc326b8909a71807ebe46a52b4
2018-12-18jenkins-helpers.sh: Add git_rev_parse_longMaxim Kuvyrkov
... which will be used for manifest generation. Change-Id: I5402707efc785f0dee507ce597f5ceefe88eaebc
2018-12-14jenkins-helpers.sh: Resolve absolute path to manifest file.Maxim Kuvyrkov
Change-Id: I3763506b1f6e2577022ae238d2a03c72acb05b4b
2018-12-14Alter timeout to work in the presence of set -ePeter Smith
The run_with_timeout_and_retry did not account for set -e so when the first command failed there was no retry. Changed to run the command in a subshell following the example of timeout in print_node_with_least_containers. Also renamed retval to ret to be more consistent with the rest of the script. Change-Id: I2f157b81914fc558d67e484e66c0108915c58dd8
2018-12-12tcwg_kernel*: Stop using and remove default configurationMaxim Kuvyrkov
Rely on manifest snippets in tcwg_kernel/* instead. Also make obligatory_variables handle bash arrays by using "declare -p". User-visible change is that now developers will need to explicitly add "@@ ./tcwg_kernel/gnu-master.sh", etc., which have been default in the past. Change-Id: I788e7cdca098fddeb67e9087252f9d53691407bf
2018-12-12convert_args_to_variables: Rework handling of configuration variablesMaxim Kuvyrkov
1. Add support for bash associative arrays to convert_args_to_variables via "==arr[elem] value" syntax. 2. Improve convert_args_to_variables to use "declare -g" directive to declare variables in the global scope -- instead of "eval var=val" approach. Change-Id: I351669734a3d0e3b6aa1edbf8fa9cb071487775a
2018-12-12Add timeout and retry utility and use it for git clone and fetchPeter Smith
Git has been observed to hang at the git fetch-pack stage when either the git or ssh protocol is used to both local and remote repositories. There are a number of possible reasons for this to happen and it is not unique to Linaro. To work around a hung job keeping a slot open for many hours we add a timeout and retry mechanism that will at a minimum fail fast. Change-Id: I7bcfc66f2ff3b2ceab7dad148effe3397705136e
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-17 03:08:14 +0000