summaryrefslogtreecommitdiff
path: root/jenkins.sh
AgeCommit message (Collapse)Author
2020-04-30Shellcheck fixes round 4David Spickett
Fixes for: https://github.com/koalaman/shellcheck/wiki/SC2207 Are using the bash 4.x 'mapfile' syntax where the input to the array is seperated by newlines. If it's separated by spaces we use 'read' instead. Support for selecting libc in MakeRelease.job was removed in 3fa23da28ad73d0fe4b512c4aab269b5280a6650. So remove references to "libc" which were expanding to nothing anyway. In tcwg-llvm-common.sh PROJECTS->BRANCHED_PROJECTS is not strictly needed. However shellcheck can't tell that the local PROJECTS in this file is different from the PROJECTS in tcwg-llvm-build.sh. Change-Id: Ie55968e3bf185d0bd5795fe48014636cda1009bc
2020-04-28Shellcheck fixes round 3David Spickett
Now using the version that's present in the bionic images. Meaning we have more warnings but can also set the minimum level to ignore a lot of them. Fix the check for whether shellcheck supports --severity. (we have pipefail on, but we expect shellcheck to fail in this case) Change-Id: I296b8554591b7d327c75393e3924184abc6512b7
2020-04-23Shellcheck fixes round 2David Spickett
Also update the sanity check script to include .job files. Change-Id: Ibf72d9ff346ff7b012d4e282a6ac8f3b7e73590d
2019-07-04jenkins.sh: Don't fail if check-artifacts.txt contains dj_sum entryChristophe Lyon
In such a case, the affectation to $sums fails, and because we are running under 'set -e' the script exits before completing the execution of exit_handler() and does not remove the lock. Change-Id: Ice52951ac61a72c19c93d894aab7d6dca96e8db5
2019-03-20Use --set buildconfig=bootstrap instead of --enable bootstrap.Prathamesh Kulkarni
The point of this patch is to replace --enable bootstrap from our existing scripts with --set buildconfig=bootstrap, so we can remove --enable bootstrap option from abe. A quick grep in jenkins-scripts repo shows that only jenkins.sh and round-robin.sh pass "--enable bootstrap" to abe. Change-Id: I76294650cfc388f2f218a0af482f36c4a8fbd9f1
2019-01-31Remove overriden old prebuilt qemu binariesOmair Javaid
We were overriding qemu with prebuilt binaries provided by Peter Maydell. Apparently we were doing so as those binaries show test improvements. As we have moved to docker containers we should update these binaries in docker files if needed. Change-Id: I2a95521d65682bffe2385fbc49aeb84058858f81
2018-05-28jenkins.sh: Fix typo when overriding the Linux kernel version.Christophe Lyon
Change-Id: Iec9b2ad4813707efbc97d888c31dff420265a921
2018-05-28jenkins.sh: grep in artifact_list only if it existsChristophe Lyon
It some cases of early build errors, the artifact_list files do not exist yet, making grep fail. This is causing trouble where the Jenkins job does not complete until it times out. For instance the console ends with: ERROR (#118): perform_build_steps (Step CHECKOUT failed) ERROR (#307): build_failure (Build process failed after 1 minutes) + jenkins.sh: 1: exit_handler + jenkins.sh: exit_handler : 111: true + jenkins.sh: exit_handler : 112: upload_to_logserver + jenkins.sh: upload_to_logserver : 44: test xdev-01.tcwglab '!=' x ++ jenkins.sh: upload_to_logserver : 45: mktemp -d + jenkins.sh: upload_to_logserver : 45: logs_dir=/tmp/tmp.KzE4tbrB88 + jenkins.sh: upload_to_logserver : 46: sums= + jenkins.sh: upload_to_logserver : 49: ssh dev-01.tcwglab mkdir -p '/home/tcwg-buildslave/logs/tcwg-backport-65ff96d8ab4119c637655776aa3df4d3a8131ce5^/0d431ccd6cc23cb1bd5153c6d4ea2c06b039b6c2/x86_64.aarch64-linux-gnu' + jenkins.sh: upload_to_logserver : 52: makelog_list=($(read_var ${user_workspace}/build-artifacts.txt 'log_make_[^=]*')) ++ jenkins.sh: upload_to_logserver : 52: read_var /home/tcwg-buildslave/workspace/tcwg-buildfarm/aarch64-linux-gnu/build-artifacts.txt 'log_make_[^=]*' ++ jenkins.sh: read_var : 37: local artifact_list=/home/tcwg-buildslave/workspace/tcwg-buildfarm/aarch64-linux-gnu/build-artifacts.txt ++ jenkins.sh: read_var : 38: local 'var=log_make_[^=]*' ++ jenkins.sh: read_var : 39: grep '^log_make_[^=]*=' /home/tcwg-buildslave/workspace/tcwg-buildfarm/aarch64-linux-gnu/build-artifacts.txt ++ jenkins.sh: read_var : 39: cut -d = -f 2- grep: /home/tcwg-buildslave/workspace/tcwg-buildfarm/aarch64-linux-gnu/build-artifacts.txt: No such file or directory and gets stuck here because there is still a background process waiting for the lock. This patch is an attempt to avoid that. Change-Id: I38dff9877bdb6e844bb59859383f1aa0110b1e68
2018-05-16jenkins.sh: Handle the case were we have nothing to upload to logserver.Christophe Lyon
If the build was already performed, we have nothing to upload to logserver, so make sure the EXIT handler does not fail in such a case. Change-Id: I3885ed08fcfe5204b5549eca0a9aa943d879fb0b
2018-05-16jenkins.sh: Fix handling of the case where the build failed.Christophe Lyon
In such a case, check-artifacts.txt does not exist, and the grep command fails, leaving the script waiting for input on stdin. Change-Id: I078eb8d819719240c92969690631fb73e6ec867e
2018-05-15jenkins.sh: Create $basedir/$dir on logserver.Christophe Lyon
Fix previous patch: after it, we dropped creation of $basedir/$dir and just kept creation of $(dirname $basedir/$dir), which is obviously not enough. Change-Id: I96fa60f66e8192803c30da31d304ab5ac4265f95
2018-05-15jenkins-logs.sh: Upload logs during EXIT trap handler.Christophe Lyon
We upload the logs and results at the very end of the script, but in some cases we exit earlier (for instance when the first cross-compiler build failed). In such cases we wouldn't upload the logs, relying on the Jenkins artifacts to debug the problem. This patch moves the upload code to a dedicated function, which is called by the EXIT trap handler. We must be careful to also remove the locks if needed, so we create an EXIT trap handler to perform both actions: upload the logs/results/manifest and clean the locks. This also fixes a potential problem where a build failure would not upload the manifest (in case the failing build went far enough to generate the manifest). Change-Id: I45d8db3993a986b529756454e5c022105f4c4deb
2018-05-15Revert "jenkins-logs.sh: Upload logs during EXIT trap handler."Christophe Lyon
This reverts commit 95beec124ec5f9bd95d88fd6c00ecd4a0e718dab. This patch seems to be causing problems: buildfarm jobs are stuck at "exit 0", and then get killed by a Jenkins timeout after 10h. Change-Id: I4784c61d863d33fa43a89ebf1b39a573c48a7873
2018-05-07jenkins-logs.sh: Upload logs during EXIT trap handler.Christophe Lyon
We upload the logs and results at the very end of the script, but in some cases we exit earlier (for instance when the first cross-compiler build failed). In such cases we wouldn't upload the logs, relying on the Jenkins artifacts to debug the problem. This patch moves the logs upload code to a dedicated function, which is called as the EXIT trap handler. Change-Id: I7106ff19d933bdf90b41cf7adb1c78e25f8bba3e
2018-03-08jenkins.sh: Add --send-results-to option.Christophe Lyon
And forward it to ABE. Change-Id: Iad877a0f00b06da3259e291974acd3a291efad08
2017-06-15jenkins.sh: Support logserver on non-default ssh portsMaxim Kuvyrkov
To switch to dedicated log-providing ssh server running inside a container. Change-Id: I584ff754dfb3cf075625dfaad70e048bed4c3bdc
2017-05-22jenkins.sh: Collect .sum files only if --runtests was provided.Christophe Lyon
Otherwise check-artifacts.txt does not exist and grep fails, making jenkins.sh fail too. If --runtests was not used, we do not want to fail for lack of .sum files. Change-Id: I6cb7736cf40ec4b036d5806b9ee89a6f3d8dbb53
2017-05-22jenkins.sh: Remove use of Jenkins env variables.Christophe Lyon
Since these variables are not propagated to the containers, stop using the empty values for: NODE_NAME, JOB_NAME, BUILD_USER_FIRST_NAME, BUILD_USER_LAST_NAME, BUILD_USER_ID. This patch removes the use of BUILD_URL only when printing it to the manifest, and does not remove the uses of BUILD_URL when acquiring the build lock, which should be done in a later patch. Change-Id: I3f4677cb174c3d8604fb0c57a4de39e38d4187ce
2017-04-25jenkins.sh: Use abe.sh --list-artifactsCharles Baylis
Use --list-artifacts to locate tarballs, manifests Abe's logs, and *.sum/*.log files. This eliminates use of find from jenkins.sh. Change-Id: I824eac39e2dbcf7f3ef3cc893b446c4964e29ad4
2017-04-25jenkins.sh: Remove unneeded indirectionMaxim Kuvyrkov
We now start test containers before calling jenkins.sh Change-Id: I88799d0837af8aa74fd48899e60ca60877bf6352
2017-03-22jenkins.sh: Use concise output for CPU infoMaxim Kuvyrkov
Change-Id: I7aaa68e9dfbcded3995127dcd099032232720bca
2017-03-07jenkins.sh: Fix handling of --extraconfig opton.Christophe Lyon
Change-Id: I499c4ed5d2f1c872de200ac2072e0ba0eebcfcec
2017-02-17jenkins.sh: Exit early when an error is detected.Christophe Lyon
Instead of just printing an help message and continuing. Change-Id: I6734639095589c861e72d8c0f4a444b86fefa1fc
2017-02-17jenkins.sh: Do not set WORKSPACE to a default value ($pwd).Christophe Lyon
WORKSPACE used to be set to $(pwd) if not already defined, but if jenkins.sh is called remotely via ssh for instance, WORKSPACE is undefined but the build succeeds in $HOME instead, which is misleading and can confuse other Jenkins script. This patch removes this default value and prints an error message in case $user_workspace (derived from $WORKSPACE, or defined by a command-line option) is undefined. Change-Id: I53a927a8f5e00430e2aefa22991daa42b2776b95
2016-12-14Update scripts from ABE. Merge remote-tracking branch 'jenkins-sh/master'Maxim Kuvyrkov
Change-Id: I8e66a03e2533620a6d941d2af4715502f87ef6d4
2016-12-12Make sure we have only one manifest file to deal with.Christophe Lyon
When jenkins.sh is called shortly before midnight, the 2 calls to abe.sh can generate 2 manifest files (different name because of different date). The code can handle one only. Change-Id: I30105f39a4c121cec095cd4cbe1000d5b6a96817
2016-12-04MakeRelease.job, jenkins.sh: Add --abedir option.Christophe Lyon
In MakeRelease.job, rename --abepath to --abedir for homogeneity. In jenkins.sh, replace --abe to --abedir, remove support for short option name (-c). Change-Id: Ic35f1a1a73e848da4da7221017db552d73d25db0
2016-11-17jenkins.sh: Handle expectedly-missing test logs.Maxim Kuvyrkov
When $runtests is false check.out and check.err are not generated. Change-Id: Ic5b0d3de2ad2edf711805099ec48e627d048d7cb
2016-11-16jenkins.sh: Make wget less verbose.Christophe Lyon
When downloading qemu. Change-Id: I840697b23eb58e7fa280fbca2fc569694eb6a5fa
2016-11-10Replace `...` with $(...)Charles Baylis
Shellcheck warns that backticks are deprecated, so replace them with the more modern $(...) syntax. This patch has been generated automatically with this shell command: perl -i -ape 's/`\(([^()`]*)\)`/\$($1)/; s/`([^(][^`]*)`/\$($1)/' *.sh lib/*.sh This leaves two more `...` in test.sh which will be replaced by hand in a later patch. Change-Id: Ia909ee0d7b2d4049c6cacf3de3babafb0f4d0f07
2016-11-06Remove non-functional --nodepends option. TCWG-859.Christophe Lyon
In fact, the option has been removed a long time ago, but some currently unused code in jenkins.sh was still using it. Change-Id: I1bf2810b31704a6693bec609b4620b1ae34f4b1f
2016-10-25jenkins.sh: Pass --testcontainer option to abe.shMaxim Kuvyrkov
... when --testcontainerfile option is given to jenkins.sh. It can take up to 20 minutes to start a testing container when the testing board is under heavy load, so it is best to parallelize it with the build of the toolchain. If starting a container exceeds time required to build the toolchain -- then we need to investigate and fix the bottlenecks on the tester side. Change-Id: Iba061228a9aec90e8d7f9e71162526685a48d8b4
2016-10-25jenkins.sh: Split build and test into 2 abe.sh invocationsMaxim Kuvyrkov
The purpose is two-fold. Firstly, this patch is a step towards separating build and test stages, which we need for testing of binary releases. Immediatelly, however, this patch makes --testcontainerfile jenkins.sh option possible. By splitting test from the build we give time for $testcontainerfile to appear once the test container is started. Change-Id: Ie85fffa8f46498f08fc2eb2fcdd5733586826a1e
2016-08-11jenkins.sh: Remove workaroundMaxim Kuvyrkov
Change-Id: I27d620b217008c8bd8d7d12b0c6f98bc670a0107
2016-08-05Cleanup usage of "fileserver" and "ref" options in scriptsMaxim Kuvyrkov
Remove --fileserver from jenkins.sh, but keep it in MakeRelease.job, since the latter is using fileserver as logserver. Also fix path to reference snapshots. Change-Id: Idbb77318810e5f9c0597a6182e2bc84d036e9400
2016-07-12Fetch gerrit patches in ABE's lib/checkout.sh rather than in jenkins.sh.Maxim Kuvyrkov
The gerrit patches are in refs/changes/*, which are not fetched by default by git clone. Since we are putting more and more stuff to gerrit, it makes sense for us to fetch these commits in ABE. The extra refs are going to be present in the reference git repos, so overhead will be negligible. Change-Id: I3043decb858231a3fcf2bf8a40ed12f4550e8739
2016-07-11Update qemu to be used for armeb-linux-gnueabihf testingMaxim Kuvyrkov
This qemu should no longer fail sync-4 test. Change-Id: I8f2a36fec3cd9b00d2b0b8a7eacc4e717df99859
2016-07-06jenkins.sh: Deploy custom QEMU only when running on x86_64Maxim Kuvyrkov
... otherwise aarch64 and armhf hosts try to execute x86_64-hosted qemu-* binaries. Change-Id: I1e7399c05aa409ad9fa35dcab47d13c325403ed5
2016-07-04jenkins.sh: Use custom qemuMaxim Kuvyrkov
This WIP qemu binaries provide significantly better results: 800 more PASSes, 400 less FAILs -- compared to Trusty qemu-armeb. The tarball also provides qemu-arm and qemu-aarch64, which perform just as well as qemu-armeb. Change-Id: I21113651a01ff4af4a569673fdb54741b41aff54
2016-06-24Print information about the build machine in jenkins.sh rather than ABE's ↵Christophe Lyon
configure. This information is more useful in the case of builds started by Jenkins than when started via the command line. Change-Id: I00952a4c58fb2f28113d7e3e94a81001d3184c33
2016-06-03jenkins.sh: Checkout same sources as will be used for the buildMaxim Kuvyrkov
Currently the changes and overrides to components are not passed to the --checkout all command. This patch fixes this. Change-Id: If902c3cc67b0de5790f2c91f9b7743882daf9e19
2016-05-16Update copyright dates, aways use bash instead of sh, since on some systems ↵Rob Savoye
it's dash instead. Change-Id: I2b819d18b52e35045684948084b125c35d4d43a6
2016-05-10jenkins.sh: Skip partial gdb .sum files.Yvan Roux
Change-Id: I1ba1ae046cd600042d2c473a3afd551c6a6688e7
2016-04-27jenkins.sh: Remove unused code, fix BZ #2204Maxim Kuvyrkov
$revision variable is not used anywhere in jenkins.sh Change-Id: If11fb5738bd6cee4b7a20b35fb63b0112a72797b
2016-04-07jenkins.sh: Reduce parallelist of builds in Cambirdge labMaxim Kuvyrkov
... to try workaround sanitizer testsuite instabilities and races. Note that the Hetzner builders have been renamed tcwg-ex40-XX -> ex40-XX, so the old stanza is no longer applicable. Also the change will only affect the 32-core builders, and the old 4-core builders will remain at -j4 parallelism. Change-Id: Ie7c1ef6a3d89264e5fbad5e06520c20d8582baee
2016-03-29Add support for --extraconfig <tool>=<path> option.Christophe Lyon
Change-Id: Iad2016cb3692c9320555a982311d5792b1a95ac0
2016-03-18Add jenkins.sh "--override" optionMaxim Kuvyrkov
... to specify non-default versions for glibc and other components. Change-Id: I7322c7fd624135d7551b1da3e2afd217e1dfebd6
2015-12-08Unambiguous manifest locationBernard Ogden
If building with such options as tarbin, we get a duplicate manifest inside destdir. Look for the manifest inside build, excluding destdir. There should always be exactly one such manifest. Symptom of multiple manifests is jenkins.sh exiting with error due to: + jenkins.sh: 405: test 'x/<some_path>/foo@bar-<date>-manifest.txt /<some other path>/foo@bar-<date>-manifest.txt' '!=' x + jenkins.sh: 406: echo node=tcwg_x86_64_ex40build_07 ./jenkins.sh: line 406: ${manifest}: ambiguous redirect Change-Id: Id3db8c084622efb0f09504eec51a7b4d1594272a
2015-12-06jenkins.sh: Follow abe.sh's style in handling multiple --excludecheck optionsMaxim Kuvyrkov
Change-Id: Ib08025be14712a6bc536055e87cc71d82164d5df
2015-12-04Check that expect works before starting buildCharles Baylis
There is an ongoing problem which means that expect is unable to spawn processes. Detect the condition before running make check, and try to output a little bit of debug info. Change-Id: Ib9432973d70562da6d31533c64214bcac67cfdec
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-27 10:12:01 +0000