| Age | Commit message (Collapse) | Author |
|---|
|
Be more deliberate in preserving "first-bad" commit in
the base-artifacts repo. Create a marker file reset-baseline
for such builds.
Also prepend the commit message title with number of pending
commits, which is roughly twice the number of pending regressions.
Change-Id: I14e16d893cab12637b71037665c8aee9b6ef24f1
|
|
Re-work commit title to handle multiple projects
Change-Id: I5532a36e961734246d6c1b028d70a3f9906b8180
|
|
Make update_baseline handle multiple projects being updated.
Change-Id: I9e6d2740154ac1987092c37ac09179e18b7f4da3
|
|
Now that we reliably reach check_regression (due to previous
improvements in run_step support) move creation of trigger
files here from clone_repo.
If updating multiple components -- create trigger files
for individual builds. I.e.,
trigger-build-llvm:
llvm_branch=SHA1_LLVM
linux_branch=baseline
trigger-build-linux:
llvm_branch=baseline
linux_branch=SHA1_LINUX
If updating a single component -- create trigger-bisect like before.
In the tcwg_kernel-bisect.sh script we start using trigger files
produced by the build script, thus simplifying and generalizing
bisect script.
Change-Id: Iaee02ce0f8fbb3f9f5dbee805089ff31f1859cb5
|
|
With simpler meaning of rr[COMPONENT_branch] to be either "baseline"
(instead of "default", overrides from ${rr[current_project]}, etc.)
we only need to resolve "baseline" value.
Change-Id: I5f32949587688d152402b6b64ed4a37ac062019e
|
|
This patch series is both a major cleanup and an optimization of
round-robin jobs. We switch from testing each component
(binutils, gcc, linux) in separate builds in round-robin fashion
to testing all components in the same build, and, if there's
a failure, re-triggering builds for individual components like
we are going now. Then if individual build fails, it triggers
bisect on that one component.
The cleanup part is that jenkins triggers are greatly simplified
and we can remove all trigger jobs (of which we had hundreds).
The optimization part is that successful builds (which is
the majority) now consume N times less resources, where N is
the number of component -- 3 in case of binutils, gcc, linux, --
because all components are updated and tested in the same build.
This patch is the first in the series, and it replaces uses of
$current_project and $current_branch with "baseline" keyword
of components' branches. I.e., for testing LLVM in LLVM+Kernel
jobs we currently use
==rr[llvm_branch] master ==rr[linux_branch] master \
==rr[current_project] llvm ==rr[current_branch] SHA1
and it will change to
==rr[llvm_branch] SHA1 ==rr[linux_branch] baseline
. IMO, the latter is significantly more intuitive.
Change-Id: Ib47dcac1571381f870150e4872ec37be5d900a07
|
|
"ERROR:" detects linker errors and "] Error " detects make errors
Change-Id: I91fe470d2cf81487945e2e55796c0dfd3f5dc74c
|
|
Replaces uses of ${rr[linux_version]} and $toolchain with
${rr[ci_config]}, which makes code more generic.
Change-Id: I27030c467b3bdb448aa2dbed36272b93726789cb
|
|
We don't have DNS alias entries tcwg-x86_64-build-09.tcwglab and
tcwg-x86_64-build-10.tcwglab -- for build-09.tcwglab and build-10.tcwglab
machines.
Change-Id: Ib32956b92098ac610be6f178c5d24c23ffa994f0
|
|
Several of our machines have docker root dir set to /home/docker
instead of standard /var/lib/docker.
The reason for using /home/docker is to allocate docker containers
on the biggest disk partition.
Change-Id: I6d16f561b5b8d303cb58d3a30f2847814eb55756
|
|
... and use ${rr[X]} instead of $X.
We make an exception for $current_project in tcwg_kernel-bisect.sh
to do a mechanical change in a separate patch.
Change-Id: Ib365552483d016e9096cc791f61a34015e50e32a
|
|
We [ab]used build_container_exec to ssh the target board to get
most recent ssh configuration. We have since started to require
local ssh access to work as well (e.g., for reboot functionality),
and implemented automatic updates of tcwg-buildslave and
tcwg-benchmark /home files.
Also, don't use ssh multiplexing (-Snone) to reliably forward
ssh agent to the benchmarking board. We have seen strange
ssh-agent failures over shared connections.
Change-Id: Iab09821272aa91f8a80b0e1b8fc5a9dd1a6bd3fb
|
|
Change-Id: I3af87f6a085ccdc781a67cf4462ebaf6abf393d7
|
|
Use ABE's internal linux sources for installation of kernel
headers. This simplifies tcwg_bmk project where we don't need
to track kernel sources.
Change-Id: I2a8eb6978f7b8783fefb6278b601353941d06aa0
|
|
Change-Id: Ia1382779ccd0a405d30b6fa6101cea987dbe2df0
|
|
Every configuration depends on baseline branches
linaro-local/ci/$ci_project/$ci_config existing in all relevant
repos, which is a pain to arrange by hand. We add new variable
rr[init_configuration], which defaults to 'false'.
For new configurations we set this parameter to true, which causes
the build script to use relevant upstream branches for components
and special "empty" branch for base-artifacts. The net result is
that after leaving new configuration for a day or two with
rr[init_configuration] set, the builds will initialize branches
in all relevant repos themselves, at which point we can turn
rr[init_configuration] back off.
Change-Id: Id1c78772619626f1a3e98a213e26046266f838e7
|
|
Don't add "exec" to the remote command unless necessary.
We need this to trigger jenkins jobs via
===
remote_exec "ci.linaro.org:2222::-l $USER@linaro.org" build ...
===
Note that "build" is a command of jenkins-cli, not an executable,
so prepending "exec" breaks the invocation.
Change-Id: If122869dadb76711666aa49690f508757e7ebce9
|
|
We were overriding qemu with prebuilt binaries provided by Peter Maydell.
Apparently we were doing so as those binaries show test improvements.
As we have moved to docker containers we should update these binaries
in docker files if needed.
Change-Id: I2a95521d65682bffe2385fbc49aeb84058858f81
|
|
Don't send regression emails for linux breakages with GNU toolchain
to Maxim K. and Christophe L..
Change-Id: Iad009ab97e9acfd6b3d6fb89f26489aecb9b7c9f
|
|
Change-Id: If07fb20b7afef172b948b06a48771f37dcd47e23
|
|
Add an excerpt from manifest listing git URLs and branches
(and avoid entries like rr[gcc_url]="no_gcc_url").
E.g.,
===
Configuration details:
rr[llvm_url]="https://github.com/llvm/llvm-project.git"
rr[llvm_branch]="master"
rr[linux_url]="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git"
rr[linux_branch]="master"
===
Also, don't include bisection range in the email body, since it's not
really useful.
Change-Id: I3a2d54968708db0fbd98bbc724449c06694fa92f
|
|
With improvements to manifest handling we haven't been using these
configuration files for a few weeks now.
Change-Id: I6815ad457119dee055e62b669958de058ac13e97
|
|
Linux-next:master rebases almost every day on top of the Linus's tree
(aka linux-next:stable), which creates problems for bisections.
We have tried to workaround these rebases, but results are still not
ideal. This patch reworks (and simplifies!) handling of linux-next.
The goal of testing linux-next:master is to detect regressions before they
reach Linus's tree (linux-next:stable). Therefore, we are really only
interested in regressions between linux-next:stable and linux-next:master
(regressions in Linus's tree will be bisected in another job anyway).
So the new workaround is to force baseline_rev to linux-next:stable
and be done with it!
This also allows us to remove confusing distinction between
baseline_rev and good_rev.
Change-Id: I96002f1d21f68e353c0b62837c4968bf956ecbf2
|
|
Add a quick exit path to the $artifacts/test.sh script, which
simplifies checking of first_bad's parents. Remove list of
skipped_revs since we are not using it.
Change-Id: I3d8601bafcbf1501239fa5a097830181b327454c
|
|
Setting $reproduce_bisect to true has the effect of bisect being
reproduced exactly as the original one, and expects manifests of
all test builds to be present. While this is useful, the common case
is to reproduce a bisect that went wrong, so the reproduction will
diverge from the initial bisection. Therefore to reproduce the common
case use (assuming original artifacts in archive directory)
./jenkins-scripts/tcwg_kernel-bisect.sh @@ archive/artifacts/manifests/bisect.sh -- @@ archive/artifacts/manifests/build-parameters.sh
and to reproduce bisect exactly use
./jenkins-scripts/tcwg_kernel-bisect.sh @@ archive/artifacts/manifests/bisect.sh --reproduce_bisect true
Change-Id: Ie6aa842d9949fad2298b1d56d74973e61541836a
|
|
We just need to keep this parameter if present, and pass it to run.sh
unchanged. Just take care that the path component in a ssh://
toolchain is always the last one.
Change-Id: I581af110e4df06adce5fcdfb3ae722ea74fc6451
|
|
At the moment we check only first parent of first_bad to confirm
that it is good. This patch extends that to checking/finding
a good parent of merge commits.
Change-Id: I0cb38223e70b10531555615ca39f9d584948c751
|
|
... and their use in reproduction instructions.
Also
- download build-parameter.sh in instructions that test.sh relies
on
- use curl instead of wget in instuctions
- limit commit output to 1000 lines in the email body.
Change-Id: Ib3b0cb94134a588e3a8c72fd7261b70d83935450
|
|
... which is needed by test.sh
Change-Id: I1b1a2c7b8e681308679d56a92edb1c745f3b5ee8
|
|
This is partly fix, partly cleanup. At the moment in bisection
mode artifacts/results are not populated until first step successfully
executes. This means that for bisections of binutils, gcc, and llvm
regressions the results file is empty if the build of these projects
fails (i.e., no successful steps at all). This causes empty clauses
in the bisection emails due to artifacts/results file being empty.
We could have fixed this by adding
===
Starting Y build
<value of successful previous step>
===
to the top of every build step, but instead we refactor and cleanup
run_step support to handle constant-value result entries.
In addition to current "stop_on_fail" mode we add "skip_on_fail" and
"reset_on_fail" modes, which simplify error handling for most build
steps.
Change-Id: Ic2d1a15fa490ec25be741cdaf718442260c52e6d
|
|
I forgot to change the name of GCC's build step from
build_abe-gcc to build_abe-stage1 in commit 5d1de6f1
"tcwg_kernel-build.sh: Support building glibc and gcc-stage2",
which caused GCC bisections to skip all build steps.
We didn't have a GCC regression until last Friday.
Change-Id: Ie1242ca5a78bc420d86f241241874b1eb64c7881
|
|
Add $ci_project and $ci_config parameters to replace references to
tcwg_kernel-specific variables.
Note that linux-next rebase workaround doesn't depend on specifics
of tcwg_kernel project, and would work just fine in other CI
projects, if necessary.
Change-Id: I81e637126bf0622ee1b67e05974c08fc51c313dd
|
|
At the moment *_rev variables, such as current_rev, accept
both git branches and git revisions (SHA1s) as values, which is
confusing and makes it difficult to write code for manifest generation
(manifests need all GIT references resolved at the moment of
generation so that reproduction runs sees the "old" state of
world).
This change standardize the meaning of *_branch and
*_rev suffices: *_branch variables accept either SHA1s or git
branches and *_rev variables accept only SHA1s.
With the established meanings, $bad_rev and $baseline_rev parameters of
tcwg_kernel-bisect.sh script can only be SHA1s (which they are anyway
in all reasonable cases), which allowed to simplify handling of a few
things in tcwg_kernel-bisect.sh script.
Change-Id: Ifc52385e4d3aeab2a3b6945fa6e1c88bcc2c7b6d
|
|
Change-Id: I0fa117d18dcec89e2c5c7e1e65c9f32d28d51511
|
|
Move code for handling "last_good" revision to a better place,
and re-trigger bisect to find true first_bad commits.
Change-Id: I29ee1ef7c287dc9baa6f909142eb57e344cda523
|
|
Change-Id: Ie4ff62b9b296efa745d10e4d51eebac349321cf1
|
|
... in preparation to converting it into a loop. Make all 3
if-clauses follow the same template.
Change-Id: If02069b536c5846db0466281ccc88d9bf0f67dff
|
|
Make it explicit that good_rev is set to baseline_rev.
Change-Id: I61bc702048aa43794661c04726bffda20995c3db
|
|
It appears the simplest and most robust way to reproduce a build
is to follow the bisection procedure: checkout revision by hand
and run a test script to perform the build. Incidentally, this
also allows testing of random commits in the repo, not just
first_bad and last_good commits.
Therefore, we reproduce the full baseline build, which initialize
all repos, builds compilers, etc., and then use the test.sh script
to replicate individual builds.
Change-Id: I5f884b1a485f641900fe33dfec8974ef207920eb
|
|
Change-Id: I05e62cba23e84c09377faadde56eea9513957d9b
|
|
The bisect script will be used in updated instructions
to reproduce the build.
Change-Id: I67b16e4f86f55b80dea39980028b244a95e2417b
|
|
1. Move build-parameters.sh to manifests/, so it survives artifacts
cleaning.
2. Don't overwrite build-parameters.sh in reproducer runs --
use "^^ $reproduce_bisect" when generating it (see documentation
of convert_args_to_variables).
3. Finally, don't pop the manifest filename when reproducing
(since the manifest filename is not pushed when "^^ true").
Change-Id: I2c2cfc571dfc11fcee7b9720d6d30bc5ee9d6897
|
|
... for the benefit of reproducer runs.
Create all symlinks with "-f" to overwrite them in reproducer runs.
Keep baseline directory in place since otherwise the reproducer run
will try to move the symlink thinking it is a directory and we end up
with two symlinks pointing to each other.
The baseline build is special (build-baseline.sh and build-baseline/)
-- it produces pre-requisites for "test" builds. We add build-SHA1
symlinks for the baseline build because it can happen that
last_good == baseline, so when creating symlink for last_good we need
build-<SHA1 of baseline> in place for such cases.
Change-Id: If11b27fad7dc3112a4aa43bb021e1de8d3884d00
|
|
Confirm_failure option was added to quickly detect failures
due to infrastructure problems (e.g., git repos stop responding).
It is saner, simpler and quicker to rely on "baseline" build
to verify that infrastructure is OK, and reproduce "bad" build
just before starting bisection.
Change-Id: Iaf35b260f3ec0ccdf043ee8e03b33bde9ef4a649
|
|
Change-Id: I2996f4e84f31e5acd7f50bc4d93b55fc53774386
|
|
Remove "reproduce" mode and follow the manifest. Use dedicated
variables for storing projects' SHA1s' in the manifest in clone_repo
like we do in all other places.
Change-Id: I5407c403cd553bb6103addda0829ba8d0f0869cb
|
|
The bisect builds should use revisions of repos checked out by
the bisect script, but we should still clean the directories from
stale files.
Change-Id: I5d3dd237d0f7ff165eeea1ccd8a2cdad1e9c091a
|
|
Git_rev_parse resolves references against refs/remotes/origin/$ref,
which is what we want most of the time, but not for HEAD.
Refs/remotes/origin/HEAD will, typically return sha1 of repo's
master branch, not sha1 of current checkout.
The fix is to use plain "git rev-parse".
A future patch will rename git_rev_parse to git_remote_rev_parse
or something similar.
Change-Id: Ic3656ee9880862608928a7e6081fe6559e64c70e
|
|
... to optimize clone and updates of base-artifacts.
The base-artifacts repo has a lot of contineously re-written
branches, but these branches are only a few commits deep
(they store "stack" of current regressions). We want to fetch
only the branch of base-artifacts for configuration we are testing,
not the whole repo.
This patch adds new parameter to clone_or_update_repo and
clone_or_update_repo_no_checkout.
Change-Id: I286a78e167ddc2e34a770fb6f8b60323a14fe8c6
|
|
Find races with "rm -rf" and complains about missing files
and directories that were just deleted. Ignore find's exit
status and stderr.
Change-Id: Ie79b9ef56e4932a694cdee16b31384efd99f1c1f
|
