summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-01-22tcwg_kernel-bisect.sh: Better handle regressions in merges commitsmaximMaxim Kuvyrkov
At the moment we check only first parent of first_bad to confirm that it is good. This patch extends that to checking/finding a good parent of merge commits. Change-Id: I0cb38223e70b10531555615ca39f9d584948c751
2019-01-22tcwg_kernel-bisect.sh: Avoid absolute paths in manifestsMaxim Kuvyrkov
... and their use in reproduction instructions. Also - download build-parameter.sh in instructions that test.sh relies on - use curl instead of wget in instuctions - limit commit output to 1000 lines in the email body. Change-Id: Ib3b0cb94134a588e3a8c72fd7261b70d83935450
2019-01-22tcwg_kernel-bisect.sh: Fetch build-parameters.sh in reproduction instructionsMaxim Kuvyrkov
... which is needed by test.sh Change-Id: I1b1a2c7b8e681308679d56a92edb1c745f3b5ee8
2019-01-22jenkins-helpers.sh: Improve run_step functionality.Maxim Kuvyrkov
This is partly fix, partly cleanup. At the moment in bisection mode artifacts/results are not populated until first step successfully executes. This means that for bisections of binutils, gcc, and llvm regressions the results file is empty if the build of these projects fails (i.e., no successful steps at all). This causes empty clauses in the bisection emails due to artifacts/results file being empty. We could have fixed this by adding === Starting Y build <value of successful previous step> === to the top of every build step, but instead we refactor and cleanup run_step support to handle constant-value result entries. In addition to current "stop_on_fail" mode we add "skip_on_fail" and "reset_on_fail" modes, which simplify error handling for most build steps. Change-Id: Ic2d1a15fa490ec25be741cdaf718442260c52e6d
2019-01-22tcwg_kernel-build.sh: Fix bisection of GCC regressionsMaxim Kuvyrkov
I forgot to change the name of GCC's build step from build_abe-gcc to build_abe-stage1 in commit 5d1de6f1 "tcwg_kernel-build.sh: Support building glibc and gcc-stage2", which caused GCC bisections to skip all build steps. We didn't have a GCC regression until last Friday. Change-Id: Ie1242ca5a78bc420d86f241241874b1eb64c7881
2019-01-22tcwg_kernel-bisect: Generalize bisect script for use in tcwg_bmk projectMaxim Kuvyrkov
Add $ci_project and $ci_config parameters to replace references to tcwg_kernel-specific variables. Note that linux-next rebase workaround doesn't depend on specifics of tcwg_kernel project, and would work just fine in other CI projects, if necessary. Change-Id: I81e637126bf0622ee1b67e05974c08fc51c313dd
2019-01-22tcwg_kernel*: Correct names of *_rev variables.Maxim Kuvyrkov
At the moment *_rev variables, such as current_rev, accept both git branches and git revisions (SHA1s) as values, which is confusing and makes it difficult to write code for manifest generation (manifests need all GIT references resolved at the moment of generation so that reproduction runs sees the "old" state of world). This change standardize the meaning of *_branch and *_rev suffices: *_branch variables accept either SHA1s or git branches and *_rev variables accept only SHA1s. With the established meanings, $bad_rev and $baseline_rev parameters of tcwg_kernel-bisect.sh script can only be SHA1s (which they are anyway in all reasonable cases), which allowed to simplify handling of a few things in tcwg_kernel-bisect.sh script. Change-Id: Ifc52385e4d3aeab2a3b6945fa6e1c88bcc2c7b6d
2019-01-22tcwg_kernel-bisect.sh: Re-arrange code.Maxim Kuvyrkov
Change-Id: I0fa117d18dcec89e2c5c7e1e65c9f32d28d51511
2019-01-21tcwg_kernel-bisect.sh: Improve handling of git-bisect edge casesMaxim Kuvyrkov
Move code for handling "last_good" revision to a better place, and re-trigger bisect to find true first_bad commits. Change-Id: I29ee1ef7c287dc9baa6f909142eb57e344cda523
2019-01-21tcwg_kernel-bisect.sh: Re-write linux-next workaround as a loop.Maxim Kuvyrkov
Change-Id: Ie4ff62b9b296efa745d10e4d51eebac349321cf1
2019-01-21tcwg_kernel-bisect.sh: Re-arrange code in linux-next workaroundMaxim Kuvyrkov
... in preparation to converting it into a loop. Make all 3 if-clauses follow the same template. Change-Id: If02069b536c5846db0466281ccc88d9bf0f67dff
2019-01-21tcwg_kernel-bisect.sh: Cleanup handling of $good_rev.Maxim Kuvyrkov
Make it explicit that good_rev is set to baseline_rev. Change-Id: I61bc702048aa43794661c04726bffda20995c3db
2019-01-21tcwg_kernel-bisect.sh: Update reproduction instructionsMaxim Kuvyrkov
It appears the simplest and most robust way to reproduce a build is to follow the bisection procedure: checkout revision by hand and run a test script to perform the build. Incidentally, this also allows testing of random commits in the repo, not just first_bad and last_good commits. Therefore, we reproduce the full baseline build, which initialize all repos, builds compilers, etc., and then use the test.sh script to replicate individual builds. Change-Id: I5f884b1a485f641900fe33dfec8974ef207920eb
2019-01-21tcwg_kernel-bisect.sh: Re-arrange code.Maxim Kuvyrkov
Change-Id: I05e62cba23e84c09377faadde56eea9513957d9b
2019-01-21tcwg_kernel-bisect.sh: Put bisect test script into artifacts/Maxim Kuvyrkov
The bisect script will be used in updated instructions to reproduce the build. Change-Id: I67b16e4f86f55b80dea39980028b244a95e2417b
2019-01-21tcwg_kernel-bisect.sh: Fix manifest handling in reproducer runsMaxim Kuvyrkov
1. Move build-parameters.sh to manifests/, so it survives artifacts cleaning. 2. Don't overwrite build-parameters.sh in reproducer runs -- use "^^ $reproduce_bisect" when generating it (see documentation of convert_args_to_variables). 3. Finally, don't pop the manifest filename when reproducing (since the manifest filename is not pushed when "^^ true"). Change-Id: I2c2cfc571dfc11fcee7b9720d6d30bc5ee9d6897
2019-01-21tcwg_kernel-bisect.sh: Simplify handling of artifact directoriesMaxim Kuvyrkov
... for the benefit of reproducer runs. Keep baseline directory in place since otherwise the reproducer run will try to move the symlink thinking it is a directory. Create all symlinks with "-f" to overwrite them in reproducer runs. Change-Id: If11b27fad7dc3112a4aa43bb021e1de8d3884d00
2019-01-21tcwg_kernel-bisect.sh: Remove --confirm_failure optionMaxim Kuvyrkov
Confirm_failure option was added to quickly detect failures due to infrastructure problems (e.g., git repos stop responding). It is saner, simpler and quicker to rely on "baseline" build to verify that infrastructure is OK, and reproduce "bad" build just before starting bisection. Change-Id: Iaf35b260f3ec0ccdf043ee8e03b33bde9ef4a649
2019-01-21round-robin.sh: Improve comment.Maxim Kuvyrkov
Change-Id: I2996f4e84f31e5acd7f50bc4d93b55fc53774386
2019-01-21tcwg_kernel-build.sh: Simplify reproduction logicMaxim Kuvyrkov
Remove "reproduce" mode and follow the manifest. Use dedicated variables for storing projects' SHA1s' in the manifest in clone_repo like we do in all other places. Change-Id: I5407c403cd553bb6103addda0829ba8d0f0869cb
2019-01-21round-robin.sh: Make clone_repo clean repos during bisect buildsMaxim Kuvyrkov
The bisect builds should use revisions of repos checked out by the bisect script, but we should still clean the directories from stale files. Change-Id: I5d3dd237d0f7ff165eeea1ccd8a2cdad1e9c091a
2019-01-21round-robin.h: Record correct sha1 of base-artifacts/ in manifestMaxim Kuvyrkov
Git_rev_parse resolves references against refs/remotes/origin/$ref, which is what we want most of the time, but not for HEAD. Refs/remotes/origin/HEAD will, typically return sha1 of repo's master branch, not sha1 of current checkout. The fix is to use plain "git rev-parse". A future patch will rename git_rev_parse to git_remote_rev_parse or something similar. Change-Id: Ic3656ee9880862608928a7e6081fe6559e64c70e
2019-01-21jenkins-helpers: Enable clone_or_update_repo to fetch single branchMaxim Kuvyrkov
... to optimize clone and updates of base-artifacts. The base-artifacts repo has a lot of contineously re-written branches, but these branches are only a few commits deep (they store "stack" of current regressions). We want to fetch only the branch of base-artifacts for configuration we are testing, not the whole repo. This patch adds new parameter to clone_or_update_repo and clone_or_update_repo_no_checkout. Change-Id: I286a78e167ddc2e34a770fb6f8b60323a14fe8c6
2019-01-20jenkins-helpers.sh: Fix fresh_dir and use it in git_cleanMaxim Kuvyrkov
Find races with "rm -rf" and complains about missing files and directories that were just deleted. Ignore find's exit status and stderr. Change-Id: Ie79b9ef56e4932a694cdee16b31384efd99f1c1f
2019-01-18TestRelease.job: execute all tests, update diagnostic format.Christophe Lyon
It's better to execute all tests and not stop on the first failing one, this patch fixes this. It also changes the diagnostic message to match other testsuites. Change-Id: I4bd3fb36c32bf5b0c32efcfdf016fdf477a19915
2019-01-17jenkins-helpers.sh: Switch to official LLVM git mirrorMaxim Kuvyrkov
Change-Id: Iea3db4df798980e53ae6fa34d56596ba4b6198b4
2019-01-17tcwg_kernel-bisect.sh: Update https:// links to artifactsMaxim Kuvyrkov
Change-Id: I95c37245a38bd4a8c8cb59969da3a94099f1560b
2019-01-17jenkins-helpers.sh: Fix "git gc" in clone_or_update_repoMaxim Kuvyrkov
Change-Id: I8f4711912ebf18504ae7cad79c755e482610f840
2019-01-16jenkins-helpers.sh: Speed-up "git gc" in clone_or_update_repoMaxim Kuvyrkov
Run "git gc" in --auto mode to skip repos which are in good shape. Change-Id: I3cd1adffda2736dd1f37d8460f1cf905a537f55e
2019-01-16tcwg_kernel-bisect.sh: Simplify artifacts/ directory handlingMaxim Kuvyrkov
Create symlinks for baseline, good, last_good, first_bad and bad builds instead of renaming directories. Change-Id: Id343b7167bed4f4eeeacc5093de010c89354754d
2019-01-16tcwg_kernel-bisect.sh: Handle bisect corner-casesMaxim Kuvyrkov
When git-bisect tests commit C as good then it assumes that all predecessors of C are good, which may not be the case when several unrelated failures occur. This patch improves handling of case when parent of $first_bad commit is bad itself. Change-Id: I8f6f55b4f3683d0e4f34d1714eeb74a46f4d6f0c
2019-01-16tcwg_kernel-build.sh: Stabilize kernel build for ccacheMaxim Kuvyrkov
Kernel likes to include auto-generated stuff into main headers, which completely breaks ccache optimizations. KBUILD_BUILD_TIMESTAMP is the big one, it's different between builds from the same sources. SUBLEVEL and EXTRAVERSION change often enough to break large bisections. Same for LOCALVERSION. We save kernel.release in artifacts/ to have a record of release strings. Change-Id: I06ed87c554481b5202cde517ee4fbaa6c25e454b
2019-01-15tcwg_kernel-bisect.sh: Fix linux-next workaroundMaxim Kuvyrkov
When trying to use linux mainline, the mainline/master branch might be ahead of linux-next repo, so the tip commit won't be present in the git clone. Fix this by fetching mainline repo and using merge-base of mainline and $bad_rev as tentative good commit. Change-Id: I21eaa98b2ebd156b846b109e8bb41e6528399f2a
2019-01-15tcwg_kernel-bisect.sh: Create symlinks for moved filesMaxim Kuvyrkov
Make sure that manifests/ directory has build-bad.sh and build-baseline.sh manifests, which are needed for reproducing bisects. Change-Id: I1e2a034286ca34aa898f69fdf726197ba63e1125
2019-01-15jenkins-helpers.sh: Implement stack logic for __manifest_fileMaxim Kuvyrkov
In tcwg_kernel-bisect.sh we use manifest handling to put options for tcwg_kernel-build.sh script into build-parameters.sh file. This overwrites __manifest_file variable of bisect job, so we end up forwarding half of bisect manifest into build-parameters.sh. Fix this by adding push/pop logic for __manifest_file variable. Change-Id: I22adc9f0533024063398b5d7cb1a416cad2aa2d1
2019-01-15jenkins-helpers.sh: Workaround stale "git gc" lock.Maxim Kuvyrkov
If build times out during "git gc" or "git gc" is otherwise killed, then we have to clean up a stale lock. Change-Id: I2c60b0a35a0881c2b5b893f6f925ffab5f88fcd4
2019-01-14round-robin.sh: Fix typo in reset_artifactsMaxim Kuvyrkov
Change-Id: I5d3a1a196e38e7fb578f4c9b76622edca346aaa4
2019-01-14tcwg_kernel-bisect.sh: Change in correct directory directoryMaxim Kuvyrkov
Change-Id: Id3a6201bc181fd044ce7f7d648abc15a1763481c
2019-01-14tcwg_kernel-bisect.sh: Simplify handling of email bodyMaxim Kuvyrkov
With updates to tcwg_kernel.yaml we no longer need to output email recipients and body into shell variable, and can just create files with text instead. Change-Id: I85f944aae9fa5ccc446bb26fdf871e65a9164ba1
2019-01-14tcwg_kernel-bisect.sh: Update handling of build-nameMaxim Kuvyrkov
... to match changes in tcwg_kernel.yaml. Change-Id: I03a63d5cfa538a9db8bcb3bf7b423f00f1e6478a
2019-01-13tcwg_kernel-bisect.sh: Fix handling of manifest filesMaxim Kuvyrkov
Move manifest files of individual builds to a dedicated manifests/ directory, which is preserved across runs. This fixes bisect reproductions, which previously broke due to manifests in [downloaded] artifacts being removed at the start of this script while cleaning artifacts directory. Change-Id: Ieffbb9d482943398a332cd51466f8116e905c170
2019-01-13tcwg_kernel-bisect.sh: Add support to reproduce bisects.Maxim Kuvyrkov
Change-Id: Ie2e0f6db672207123a9903eb7f60de23f316f60a
2019-01-13jenkins-helpers.sh: New helper fresh_dirMaxim Kuvyrkov
Change-Id: I1513712e64500712a5540b9b5a5bd8e1228334c4
2019-01-13tcwg_kernel-bisect.sh: Handle git-bisect's smartness.Maxim Kuvyrkov
Handle transitive assumptions of git-bisect about parent commits. See inline comment for details. Change-Id: I28e35b249e43325f37155059de4ec356cf140de4
2019-01-11tcwg_kernel-bisect.sh: Try harder to bisect linux-next failuresMaxim Kuvyrkov
Try mainline:master as last-resort baseline build. Change-Id: I3d3b03139e6052bb9254778e1f49d45319313ca8
2019-01-11tcwg_kernel-bisect.sh: Cleanup.Maxim Kuvyrkov
Change-Id: I7c68ba0d0a63c922511618350357a13089002127
2019-01-11tcwg_kernel-bisect.sh: Send emails with useful info when bisect failsMaxim Kuvyrkov
We are trying to reduce the number of failed bisect and make scripting more robust against history re-writes (which is the primary cause of failed bisects). Re-work logic to generate email body to include useful info about failed bisects. Change-Id: Ia25f3311016c0175f3b59e1e4c7308485b5276a4
2019-01-11tcwg_kernel-build.sh: Make sure manifest declares variables in the global ↵Maxim Kuvyrkov
namespace. Change-Id: Ic618e76d38a3dc446b6fd29e05caf9987770f01b
2019-01-11jenkins-helpers.sh: Run "git gc" when updating git repos.Maxim Kuvyrkov
This avoids git repos from growing too large. Change-Id: I2cd9b83180ca13557a0aa5558d5019a98b81833b
2019-01-11jenkins-helpers.sh: Add "^^ true/false" switch to manifest handlingMaxim Kuvyrkov
In tcwg_kernel-bisect.sh we need to call tcwg_kernel-build.sh with various options. We also generate manifest for each call so that we can reproduce /bisect/ runs. With current support we need to write something like this to handle reproductions of bisects: === if $reproduce_bisect; then ./tcwg_kernel-build.sh @@ /path/to/manifest.sh else ./tcwg_kernel-build.sh %% /path/to/manifest.sh --build options fi === which is cumbersome and obstructs script logic. This patch adds a "reproduction switch" -- "^^ true/false". When "^^ true %% manifest.sh --build options" it provided, then manifest.sh is sourced and all "--build options" are discarded to avoid changing build environment that manifest.sh sets up. When "^^ false %% manifest.sh --build options" is given, then manifest.sh is created and populated with "--build options". Change-Id: I38fcf37dc024891f5aa858dbd110082692dc09c9