Age | Commit message (Collapse) | Author |
|
... to dockerfiles.git/tcwg-base/tcwg-host/docker-wrapper .
Our workaround to dockerhub's pull limits in start-container-docker.sh
works well, but with one exception -- it doesn't properly work when
container is started on a remote host, which we do a lot in
tcwg-benchmark-* jobs.
To solve this we migrate workaround logic to docker wrapper script,
and add a new docker command for it -- "maybepull".
Change-Id: I131f1adc56052164a76c5dfaf9d07a4d2f24ec43
|
|
... to ignore "docker rmi" failures on used images.
Change-Id: I69fd40c45fe5f0b36e4b833097c2ac4e5a57b984
|
|
... and update "pull" timestamps in tcwg-start-container.sh
Change-Id: Ib5638a9fb22c341527443fd3ff2f8431f7aa1fe0
|
|
Change-Id: Ieb39c841151533d7553786f78f80fbb7a62857e2
|
|
We attempt to run all our build using current versions of docker images.
Unfortunately, now that dockerhub limits pull requests, we need to be
more considerate to when we pull the image or attempt to use the local
copy.
This is the first stab at the problem. We use stamp files under
/home/shared/docker/ to track times of last image pull and last image
use. We then run "docker pull" on images that are older than a day
to avoid using terribly-outdated local copies.
We also switch to deleting images that haven't been used for 3+ days in
tcwg-cleanup-stale-containers.sh -- instead of deleting all not-used-at-
the-moment images.
Change-Id: I3839fe030b30adaee318f7453c6d2e47185f4ffc
|
|
Fixes for:
https://github.com/koalaman/shellcheck/wiki/SC2207
Are using the bash 4.x 'mapfile' syntax where the
input to the array is seperated by newlines.
If it's separated by spaces we use 'read' instead.
Support for selecting libc in MakeRelease.job was
removed in 3fa23da28ad73d0fe4b512c4aab269b5280a6650.
So remove references to "libc" which were expanding
to nothing anyway.
In tcwg-llvm-common.sh PROJECTS->BRANCHED_PROJECTS
is not strictly needed. However shellcheck can't
tell that the local PROJECTS in this file is
different from the PROJECTS in tcwg-llvm-build.sh.
Change-Id: Ie55968e3bf185d0bd5795fe48014636cda1009bc
|
|
Don't add empty container names
to the list to remove. This was causing
us to run docker commands without
container names.
Change-Id: Iddf2c7aa268aec4a6254db1bf6e4c66850bb7dfa
|
|
Now using the version that's
present in the bionic images.
Meaning we have more warnings
but can also set the minimum
level to ignore a lot of them.
Fix the check for whether
shellcheck supports --severity.
(we have pipefail on, but we
expect shellcheck to fail in
this case)
Change-Id: I296b8554591b7d327c75393e3924184abc6512b7
|
|
First round of low hanging fruit
from the files with the fewest
warnings.
Change-Id: I60ad57b5bfb2f0be9bb6506252c820f44d055200
|
|
(all error/style level warnings)
SC1035: missing whitespace
SC2068: double quote array expansions
SC2145: argument mixes string and array
SC1066: Don't use $ on the left side of assignments
SC2126: Use grep -c instead of grep|wc
Change-Id: Ia53070d3e2c9aea9f626fdb7b2b5bfc9e1f8e682
|
|
Use Docker's internal command to cleanup images.
Change-Id: Ibb4605761d3c638fcf18d061763c1e2594c7b890
|
|
... due to present dangling volumes, which are expected on dev machines.
Change-Id: Iec4aefbc025ee0546458299db442597d1fc22397
|
|
The current code seems to miss some of the dependencies of dangling
images, so we attempt to delete a used image.
Rather than being more careful in not attempting to delete used images,
we non-forcefully attempt to delete all images, and ignore failures.
Change-Id: I240ab7faf6f21d7f60451375992805fcc9124f60
|
|
Add a few more messages, to make the Jenkins console easier to understand.
Change-Id: I2bab8aaf5333c4e7f8d8d8681986c961336f1fd6
|
|
process to kill.
Handle two cases where killall can fail:
- command not found, in which case we want an error so that we know we have to install killall
- no ssh-agent process running, which is OK
Change-Id: I0d6d7e40eed6db7c6250f2231f05b8cca73c8355
|
|
processes.
Without this patch, failure to kill such processes is not reported as
part of the script exit status. This happens for instance when the killall
command is not installed on the machine.
Change-Id: Ic0fa10ce6b69f12fb2382cfa3680fcc1c7d1b4bd
|
|
"Echo" concatenates all elements into a single string, so we
need to break them up for "grep" to match its pattern.
Change-Id: I954ca80fc6f0e289185780f76988688ce0f1ed78
|
|
Change-Id: I19ed211d69326ff34a239cb4cc80a6f2b72ccd63
|
|
... when no matches for "grep" or "killall" are found.
Change-Id: I800f06607e01c3a1d167ebdf9a2c3eaeb75559a1
|
|
Change-Id: Ia50672fbab9ade7922feb12a6199f4ae581c18cd
|
|
We now start to keep useful data in docker volumes named host-home,
home-$USER, etc.
Change-Id: I4df6355de4fca9bcdc3073edefb8a6be4d97a61c
|
|
threshold.
Implements TCWG-1404.
We want to make sure we do not have more containers left than a given
threshold, to make sure there is no problem with automatically created
containers becoming too numerous and not properly cleaned up.
Change-Id: Ia740b58d1b75cdcd1fa134b551613e9b043ada34
|
|
Change-Id: I6626b870109f56619930017e6b4e8f40d65e52f2
|
|
Change-Id: I0acd334a54ebf9160ec706bcc0024acebe76e627
|
|
Implements TCWG-1404.
Containers started by Jenkins jobs have names starting with a number,
we want to cleanup only those. This behaviour is hardcoded by this
patch, but controlled by a flag we can control later if needed.
Change-Id: Idbf8eb6cfc44ccd7cdaad08019522f670046573e
|
|
... instead of via local file socket. This works around problem of
tcwg-buildslave not being in "docker" group.
Change-Id: I4764b0cbc1ba8d6474874d4d1f4550e4cd5de651
|
|
Thanks, Charles!
Change-Id: Iba7f6ba6ad5f51eaf244c38ad68f37150d080a4b
|
|
Exit code is now set to non-zero only when unexpected events happen.
Expected cleanups do not increase exit code anymore.
Change-Id: I009825e7b8d7ba16303ec3dd2aeb7ae69110168b
|
|
Change-Id: Ie12e9a64a4ed1d21ab9c155c3f263467c950176d
|
|
Change-Id: Icbdf341f268ba3ff18552428e9466cc371450cad
|
|
Change-Id: I9169344fbf1657b22de80cd99f209618f2ce6bfd
|
|
Change-Id: I18aecdf987ab3a479e26ab7f7c812a17751bdfd7
|
|
E.g., DOCKER="docker -H build-02.tcwglab:2375" ./tcwg-cleanup-stale-containers.sh
Change-Id: I7424ab999d3305db9059dbb773a3f4371b4873a7
|
|
... instead of removing them. This improvement makes cleanup job
to stop running containers after [default] 10 hours, and then
remove stopped containers after 10 days.
Change-Id: I1411a67788890b56cf6d08ee265beccdcdb7bd74
|
|
Change-Id: I7519d6573aae3b95c304e821fcc5dcd40bfa6f58
|
|
Change-Id: I49fd34eb85acabd916e61f5eb7f95a3e4398f4fd
|
|
In our current workflow we should not generate stale volumes or
containers, but we can generate stale images. Add options to
allow cleaning up what we need, instead of all or nothing.
Change-Id: Id270a7e51f4c931c32a08d2f244e727a7d783750
|
|
... to avoid spurious warning from removing untagged images
that are used by non-running containers.
Change-Id: I6d8f665c2c1559cd7826236cbfac16cbc48e1f31
|
|
Change-Id: I68a2df464a9fe1a9563ad2e5b0df068890146082
|
|
... what can I say ... Monday morning.
Change-Id: Ic6b44cd5c0f9ccec69665fa997b96fdecfd89735
|
|
Change-Id: I9ed79a09f2a91d7ca66ae8055aab29b6a4bc702d
|
|
Change-Id: I553c12efa9d4933286612b91e3a0faf158e78911
|
|
... which we generate when commiting a container for debugging.
If the debug container is still present, then image removal will
[expectedly] fail, so only unused images will be removed.
Change-Id: If2790d19ff4d944077ebb6f47d36559df20060da
|
|
Change-Id: I9a763abb102ac33eae9a55fa36bd6754ca7eb61e
|
|
Change-Id: Ia093265c5220910f54d792c8b0fc27f3086bbbbc
|
|
Handling dangling volumes is straight-forward.
For unknown to me reasons "docker rm" return non-zero exit code when
successfully removing a container. Maybe it is proxying exit code from
the container's main process (sshd in our case).
In any case, we now switch to the meaning of the exit code to
zero -- nothing to do;
non-zero -- something was cleaned up.
Change-Id: I16eb5946055a828511d320dcfc28933c7af5a2d8
|
|
Change-Id: I086487f4e33e5ba0e9b02e973827a8363bb0fa1e
|
|
Change-Id: If3898928a9231b90e115e60fd400cbb29ff4b553
|
|
Change-Id: I7ab120d24e659bb238dfce6e6c044db69160e270
|
|
Change-Id: I2d92f4eb5b76a1a62cf09f1ed021756a2f9fcceb
|