README for test-manifest2.sh script
-----------------------------------
=== Purpose:
Test that builds can be reproduced by ABE using manifest files.
=== How to run the script
typical usage is:
$ bash abe-tests/test-manifest2.sh 2>&1 | tee mani2.log
This creates a directory called 'full-manifest-test', which is deleted at the
start if it already exists. Results are stored inside that directory, along
with files generated during the test.
The main results files are:
full-manifest-test/results_summary.txt
contains a list of tests and whether they passed or failed
full-manifest-test/results_report.txt
An expanded version of the summary which also includes selected results
files so that the causes of failures can be seen.
=== Options accepted by the script
--abe-path <path>
If set, the path is used for a checkout of ABE. If not set, then the
script checks out its own copy of ABE to test.
--ref-snapshots <path>
Path to git reference snapshots directory.
--abe-branch <branch name>
Name of branch to test. Defaults to master.
--display-report
Displays full test report at the end of test execution. The report is
always available in full-manifest-test/results_report.txt even if
this option is not specified.
--quiet
Suppresses ABE build output.
--debug
Enables bash command tracing.
=== Operations performed by the script:
1. clone abe repo, and checks out array branch
2. in a new directory, configures abe and builds aarch64-linux-gnu
toolchain and generates a tarball
3. examines build log and build dir to extract git revisions used in the
build.
4. extracts tarball
5. locates manifest file, and strips out the developer info section.
6. in a new directory, configures abe and builds a toolchain using the
stripped manifest file and generates a tarball
7. examines build log and build dir to extract git revisions used in the
build.
8. compares git revisions in the two builds
9. extracts build commands from the log and postprocesses them to normalize
build directory names and temporary identifiers using PIDs.
10. compares these post-processed log files
11. extracts tarball from 2nd build
12. locates manifest file, and strips out the developer info section.
13. compares stripped manifests of the two builds with different levels of
strictness.
=== Explanation of tests
. revisions diff
Tests the revisions used when building components which are stored in git.
Pass criteria: when building with a manifest, the same git revisions are used
for each component which is stored in git.
. run lines diff
Extracts the 'RUN:' lines from the build log, ignoring those used in the
checkout phase of the build process.
Pass criteria: the same command lines are used in the manifest build as were
used when the original build was done.
. references to build dir
checks that the stripped manifest does not contain references to the build
directory
Pass criteria: the strings "/build/" or "/build2/" do not appear in the
stripped manifest file.
. manifest diff
Compares the stripped manifest generated by the first build with the stripped
manifest generated by the second build.
Pass criteria (strict): the manifests are identical
Pass criteria (relaxed): the manifests contain the same lines, but the order
may be different
Pass criteria (super relaxed): the manifests contain the same lines, but the
order may be different, and lines may be duplicated
=== Log file contents
Where '1' appears in the filename, the file relates to a log generated from the
first build, which is used to create the manifest.
Where '2' appears in the filename, the file relates to a log generated from the
second build, which is done using the manifest created by the first build.
. full-manifest-test/files_1.txt
. full-manifest-test/files_2.txt
List of files in the tarball.
. full-manifest-test/files_diff.txt
Unified diff of the list of files.
. full-manifest-test/orig-manifest1.txt
. full-manifest-test/orig-manifest2.txt
The manifest output by each build, in unmodified form.
. full-manifest-test/relaxed_mani_diff.txt
Unified diff of sorted stripped manifest files.
. full-manifest-test/strict_mani_diff.txt
Unified diff of stripped manifest files.
. full-manifest-test/super_relaxed_mani_diff.txt
Unified diff of sorted, uniq'd stripped manifest files.
. full-manifest-test/results_summary.txt
Summary of results with PASSED/FAILED lines.
. full-manifest-test/revisions1.txt
. full-manifest-test/revisions2.txt
List of git revisions for components used in each build.
. full-manifest-test/revisions_diff.txt
Unified diff of revisions list.
. full-manifest-test/run_lines_1.txt
. full-manifest-test/run_lines_2.txt
List of RUN: lines which were logged during the build phase of abe build. These
are processed to normalise for expected differences in paths.
. full-manifest-test/run_lines_diff.txt
Unified diff of run_lines_[12].txt.
. full-manifest-test/stripped-manifest1.txt
. full-manifest-test/stripped-manifest2.txt
Manifests with 'developer info' section stripped out.
