diff options
Diffstat (limited to 'utils/mangle-jobs')
31 files changed, 0 insertions, 971 deletions
diff --git a/utils/mangle-jobs/README b/utils/mangle-jobs/README deleted file mode 100644 index 70f01da..0000000 --- a/utils/mangle-jobs/README +++ /dev/null @@ -1,92 +0,0 @@ -Intro ------ -mangle-jobs is a scripting framework to perform mass migration of Jenkins -job definitions. The motivataion for such script is that, the way -linaro-android-cloud-buildd is used, all (or large cluster) of jobs -represent the same type of build process, just parameterized with -specific settings (Android and toolchain versions, etc.) Thus, many -changes needed - for example, exporting additional build artifact, or -enabling some plugin - should be applied to all jobs (or large subset) -at the same time. Having an automated scripting framework for such -migrations saves from doing time-consuming and error-prone manual changes -via Jenkins UI for lot of jobs. The drawback is that migration is performed -on the XML-encoded content, so some development and XML processing -familiarity is required. However, due to XPath and LXML library which -implements its support, migration development is sufficiently easy, and -there's growing collection of existing migrations scripts, to serve -as example and templates for new usecases. - -mangle-jobs script ------------------- -mangle-jobs script allows to run a "mangle" (migration) script for all -jobs present in Jenkins and submit results back to server for permanent -storage. A mangle script operates on an XML representation of job config, -and can apply arbitrary Python algorithm to it (including conditionals -to apply to a subset of jobs). Besides production mode of updating jobs -in a networked Jenkins server, mangle-jobs also offers extensive dry-run -and testing capabilities, suitable for mangle script development and -pre-production testing. - -Mangle script development workflow ----------------------------------- -Basic mangle-jobs workflow is as follows: - -1. Figure out what XML changes must be performed. For example, select -a particular job in Jenkins, make a backup copy of its config, make -needed changes via Jenkins UI, and diff job config against the backup -copy. - -2. Find existing mangle script performing changes close to ones needed. -As noted above, there's a good selection of such scripts, and it makes -little sense to develop a new script from scratch, this is perfect -example when program-by-example technique helps. - -3. Make a copy of template you found and make needed changes. - -4. Take the config backup copy you made earlier and run mangle-jobs on it -in test mode: - -$ ./mangle-jobs <your_script.mangle> --file=<config_backup.xml> - -You will see a textual diff of old and new config, highlighting the changes -your script made. Verify that they are as expected, and repeat from step 3 -if needed. - -5. You are ready to perform en-masse pre-production test now. You should have -Jenkins username with appropriate permissions and its password or API key -(recommended) handy. The API key can found in Jenkins by clicking you username -in the top right corner, then Configure in menu. Password/API key will be -prompted on teh console. If you really need that, you can put the credential -in the file and refer to it with --passwd-file=<passwd_file> to avoid -interactive prompts. - -$ ./mangle-jobs --user=<user> <your_script.mangle> - -This will run your script repeatedly for each job store in Jenkins server and -will show aggregated diff output. Review it carefully to watch for anomalities -and corner cases. Repeat from step 3 if needed. - -6. Once you're absolutely sure that the changes performed by your mangle script -are correct, run it in the production update mode: - -$ ./mangle-jobs --user=<user> <your_script.mangle> --really - -Mangle script details ---------------------- -A mangle script is standalone Python module. It should define a function -mangle(), accepting single parameter - elemnt tree, parsed from XML job -config with lxml.etree.fromstring(). The function should perform needed -transformation on the tree (in place). - -To get speciifics of command-line arguments and options, run mangle-jobs -with --help option. - -Known issues ------------- -XML dialects produced by Jenkins and by mangle-jobs tool may not necessarily -match. This means that while they may match with respect to underlying XML -syntax (node structure and content), their textual representation may be -different, leading to spurious differences in test run output. There's -ongoing effort to make output XML rendition of mangle-jobs correspond to -the dialect used by Jenkins, but there still may be spurious change noise -in the output. diff --git a/utils/mangle-jobs/add-copy-to-slave.mangle b/utils/mangle-jobs/add-copy-to-slave.mangle deleted file mode 100644 index de28dde..0000000 --- a/utils/mangle-jobs/add-copy-to-slave.mangle +++ /dev/null @@ -1,25 +0,0 @@ -# Add copy-to-slave plugin setup for build dependencies - -#from lxml.etree import Element -from lxml.etree import fromstring - -new_node = """\ - <com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper> - <includes>**/*</includes> - <excludes></excludes> - <flatten>false</flatten> - <includeAntExcludes>false</includeAntExcludes> - <hudsonHomeRelative>false</hudsonHomeRelative> - <relativeTo>copyToSlave</relativeTo> - </com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper> -""" - -def mangle(tree): - if tree.xpath("//com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper"): - # Already there - return - tag = tree.xpath('/project/buildWrappers')[0] - node = fromstring(new_node) - tag.append(node) - node.getprevious().tail += " " - node.tail = "\n " diff --git a/utils/mangle-jobs/add-email-ext-notification.mangle b/utils/mangle-jobs/add-email-ext-notification.mangle deleted file mode 100644 index a0bea64..0000000 --- a/utils/mangle-jobs/add-email-ext-notification.mangle +++ /dev/null @@ -1,135 +0,0 @@ -# Add Email notifications - -from lxml.etree import fromstring -from mangle_helper import * - - -builders = 'linaro-android-builder-notifications@linaro.org' -restricted_builders = 'linaro-android-restricted-builder-notifications@linaro.org' - -publisher = """\ - <hudson.plugins.emailext.ExtendedEmailPublisher> - <recipientList>%s</recipientList> - <configuredTriggers> - <hudson.plugins.emailext.plugins.trigger.UnstableTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.UnstableTrigger> - <hudson.plugins.emailext.plugins.trigger.NotBuiltTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.NotBuiltTrigger> - <hudson.plugins.emailext.plugins.trigger.AbortedTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.AbortedTrigger> - <hudson.plugins.emailext.plugins.trigger.FailureTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.FailureTrigger> - <hudson.plugins.emailext.plugins.trigger.StillFailingTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.StillFailingTrigger> - <hudson.plugins.emailext.plugins.trigger.FixedTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.FixedTrigger> - <hudson.plugins.emailext.plugins.trigger.StillUnstableTrigger> - <email> - <recipientList/> - <subject>$PROJECT_DEFAULT_SUBJECT</subject> - <body>$PROJECT_DEFAULT_CONTENT</body> - <sendToDevelopers>false</sendToDevelopers> - <sendToRequester>false</sendToRequester> - <includeCulprits>false</includeCulprits> - <sendToRecipientList>true</sendToRecipientList> - <attachmentsPattern/> - <attachBuildLog>false</attachBuildLog> - <replyTo/> - </email> - </hudson.plugins.emailext.plugins.trigger.StillUnstableTrigger> - </configuredTriggers> - <contentType>default</contentType> - <defaultSubject>$DEFAULT_SUBJECT</defaultSubject> - <defaultContent>$DEFAULT_CONTENT</defaultContent> - <attachmentsPattern/> - <presendScript/> - <attachBuildLog>false</attachBuildLog> - <replyTo>$DEFAULT_RECIPIENTS</replyTo> - </hudson.plugins.emailext.ExtendedEmailPublisher> -""" - -def mangle(tree): - if tree.xpath("//hudson.plugins.emailext.ExtendedEmailPublisher"): - # Already there - return - tag = tree.xpath('/project/publishers')[0] - cfg = get_build_config(tree) - if cfg.get("BUILD_TYPE", "build-android") == "build-android-restricted": - node = fromstring(publisher % restricted_builders) - else: - node = fromstring(publisher % builders) - tag.append(node) - node.getprevious().tail += " " - node.tail = "\n " diff --git a/utils/mangle-jobs/add-snapshots-l-o-publisher.mangle b/utils/mangle-jobs/add-snapshots-l-o-publisher.mangle deleted file mode 100644 index 21cddb8..0000000 --- a/utils/mangle-jobs/add-snapshots-l-o-publisher.mangle +++ /dev/null @@ -1,68 +0,0 @@ -# Add SSH(SFTP) artifact publishing to snapshots.linaro.org - -#from lxml.etree import Element -from lxml.etree import fromstring - -publisher = """\ - <jenkins.plugins.publish__over__ssh.BapSshPublisherPlugin> - <consolePrefix>SSH: </consolePrefix> - <delegate> - <publishers> - <jenkins.plugins.publish__over__ssh.BapSshPublisher> - <configName>snapshots.linaro.org</configName> - <verbose>false</verbose> - <transfers> - <jenkins.plugins.publish__over__ssh.BapSshTransfer> - <remoteDirectory>${JOB_NAME}/${BUILD_NUMBER}</remoteDirectory> - <sourceFiles>build/out/target/*/*/*.img,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info</sourceFiles> - <excludes></excludes> - <removePrefix>build/out</removePrefix> - <remoteDirectorySDF>false</remoteDirectorySDF> - <flatten>false</flatten> - <cleanRemote>false</cleanRemote> - <execCommand></execCommand> - <execTimeout>120000</execTimeout> - <usePty>false</usePty> - </jenkins.plugins.publish__over__ssh.BapSshTransfer> - </transfers> - <useWorkspaceInPromotion>false</useWorkspaceInPromotion> - <usePromotionTimestamp>false</usePromotionTimestamp> - </jenkins.plugins.publish__over__ssh.BapSshPublisher> - <jenkins.plugins.publish__over__ssh.BapSshPublisher> - <configName>snapshots.linaro.org file-move</configName> - <verbose>false</verbose> - <transfers> - <jenkins.plugins.publish__over__ssh.BapSshTransfer> - <remoteDirectory></remoteDirectory> - <sourceFiles></sourceFiles> - <excludes></excludes> - <removePrefix></removePrefix> - <remoteDirectorySDF>false</remoteDirectorySDF> - <flatten>false</flatten> - <cleanRemote>false</cleanRemote> - <execCommand>reshuffle-files</execCommand> - <execTimeout>120000</execTimeout> - <usePty>false</usePty> - </jenkins.plugins.publish__over__ssh.BapSshTransfer> - </transfers> - <useWorkspaceInPromotion>false</useWorkspaceInPromotion> - <usePromotionTimestamp>false</usePromotionTimestamp> - </jenkins.plugins.publish__over__ssh.BapSshPublisher> - </publishers> - <continueOnError>false</continueOnError> - <failOnError>false</failOnError> - <alwaysPublishFromMaster>true</alwaysPublishFromMaster> - <hostConfigurationAccess class="jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin" reference="../.."/> - </delegate> - </jenkins.plugins.publish__over__ssh.BapSshPublisherPlugin> -""" - -def mangle(tree): - if tree.xpath("//jenkins.plugins.publish__over__ssh.BapSshPublisherPlugin"): - # Already there - return - tag = tree.xpath('/project/publishers')[0] - node = fromstring(publisher) - tag.append(node) - node.getprevious().tail += " " - node.tail = "\n " diff --git a/utils/mangle-jobs/artifacts-add.mangle b/utils/mangle-jobs/artifacts-add.mangle deleted file mode 100644 index 1352135..0000000 --- a/utils/mangle-jobs/artifacts-add.mangle +++ /dev/null @@ -1,19 +0,0 @@ -# Add Artifact archiving config - -from lxml.etree import fromstring - - -new_node = fromstring("""\ - <hudson.tasks.ArtifactArchiver> - <artifacts>build/out/*.xml</artifacts> - <latestOnly>false</latestOnly> - </hudson.tasks.ArtifactArchiver> -""") - -def mangle(tree): - if tree.xpath("//hudson.tasks.ArtifactArchiver"): - # Needed config already there - don't change - return False - - parent_node = tree.xpath('/project/publishers')[0] - parent_node.insert(0, new_node) diff --git a/utils/mangle-jobs/artifacts-remove.mangle b/utils/mangle-jobs/artifacts-remove.mangle deleted file mode 100644 index 0ec0810..0000000 --- a/utils/mangle-jobs/artifacts-remove.mangle +++ /dev/null @@ -1,13 +0,0 @@ -# Disable Jenkins builtin artifact archiving (part of migration to snapshots.linaro.org) - -from mangle_helper import * - - -def mangle(tree): - cfg = get_build_config(tree) - if cfg.get("BUILD_TYPE", "build-android") != "build-android": - return - tags = tree.xpath('//hudson.tasks.ArtifactArchiver') - if not tags: - return - tags[0].getparent().remove(tags[0]) diff --git a/utils/mangle-jobs/artifacts-set.mangle b/utils/mangle-jobs/artifacts-set.mangle deleted file mode 100644 index 4a5c707..0000000 --- a/utils/mangle-jobs/artifacts-set.mangle +++ /dev/null @@ -1,15 +0,0 @@ -# Update list of archived artifacts -# Note - since we stopped to use native Jenkins archiving, -# this script is no longer used - -new_value = "build/out/target/*/*/*.img," \ - "build/out/target/*/*/*.tar.bz2," \ - "build/out/target/*/*/MD5SUMS," \ - "build/out/*.tar.bz2," \ - "build/out/*.xml," \ - "build/out/*_config," \ - "build/out/lava-job-info" - -def mangle(tree): - tags = tree.xpath('/project/publishers/hudson.tasks.ArtifactArchiver/artifacts') - tags[0].text = new_value diff --git a/utils/mangle-jobs/build-expiration-set.mangle b/utils/mangle-jobs/build-expiration-set.mangle deleted file mode 100644 index d9d8a6d..0000000 --- a/utils/mangle-jobs/build-expiration-set.mangle +++ /dev/null @@ -1,17 +0,0 @@ -# Set expiration policy for a job - -from lxml.etree import fromstring -from mangle_helper import * - - -new_node = """\ - <logRotator> - <daysToKeep>90</daysToKeep> - <numToKeep>100</numToKeep> - <artifactDaysToKeep>-1</artifactDaysToKeep> - <artifactNumToKeep>-1</artifactNumToKeep> - </logRotator> -""" - -def mangle(tree): - add_or_replace_node(tree, "/project/logRotator", new_node) diff --git a/utils/mangle-jobs/build-steps-set.mangle b/utils/mangle-jobs/build-steps-set.mangle deleted file mode 100644 index 16f5ec3..0000000 --- a/utils/mangle-jobs/build-steps-set.mangle +++ /dev/null @@ -1,19 +0,0 @@ -# Set latest build steps config for all jobs, as provided in build.xml file - -from lxml.etree import fromstring - - -builders = open("builders.xml").read() - -def mangle(tree): - tags = tree.xpath("//jenkins.plugins.publish__over__ssh.BapSshPublisherPlugin") - for t in tags: - t.getparent().remove(t) - - # Replace existing <builders> content - node = fromstring(builders) - # Add our complete content after the current <builders> - tag = tree.xpath('/project/builders')[0] - tag.addnext(node) - # And now remove the original <builders> - tag.getparent().remove(tag) diff --git a/utils/mangle-jobs/build-timeout-add.mangle b/utils/mangle-jobs/build-timeout-add.mangle deleted file mode 100644 index 142f70a..0000000 --- a/utils/mangle-jobs/build-timeout-add.mangle +++ /dev/null @@ -1,22 +0,0 @@ -# Add build-timeout plugin setup for build dependencies - -#from lxml.etree import Element -from lxml.etree import fromstring - -new_node = """\ - <hudson.plugins.build__timeout.BuildTimeoutWrapper> - <timeoutMinutes>150</timeoutMinutes> - <failBuild>false</failBuild> - </hudson.plugins.build__timeout.BuildTimeoutWrapper> -""" - -def mangle(tree): - if tree.xpath("//hudson.plugins.build__timeout.BuildTimeoutWrapper"): - # Already there - return - tag = tree.xpath('/project/buildWrappers')[0] - node = fromstring(new_node) - tag.append(node) - if node.getprevious(): - node.getprevious().tail += " " - node.tail = "\n " diff --git a/utils/mangle-jobs/build-timeout-set.mangle b/utils/mangle-jobs/build-timeout-set.mangle deleted file mode 100644 index 9c4b113..0000000 --- a/utils/mangle-jobs/build-timeout-set.mangle +++ /dev/null @@ -1,15 +0,0 @@ -# Update default build timeout value to new default. -# Don't touch custom timeout times - -OLD_DEFAULT = "275" -NEW_DEFAULT = "375" - - -def mangle(tree): - tags = tree.xpath("//hudson.plugins.build__timeout.BuildTimeoutWrapper/timeoutMinutes") - if not tags: - return False - tag = tags[0] - if tag.text != OLD_DEFAULT: - return False - tag.text = NEW_DEFAULT diff --git a/utils/mangle-jobs/builders.xml b/utils/mangle-jobs/builders.xml deleted file mode 100644 index 4d89266..0000000 --- a/utils/mangle-jobs/builders.xml +++ /dev/null @@ -1,17 +0,0 @@ - <builders> - <hudson.plugins.shell.ShellWithStatus> - <command># Runs as ubuntu -rm -rf build-tools -bzr get lp:linaro-android-build-tools build-tools -sudo -H -E build-tools/node/build us-east-1.ec2-git-mirror.linaro.org "$CONFIG" - -time build-tools/utils/new-publish/publish -p2 $JOB_NAME/$BUILD_NUMBER "build/out/target/*/*/*.img,build/out/target/*/*/*.img.bz2,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info,build/out/linaro_kernel_build_cmds.sh,build/out/linaro_android_build_cmds.sh,build/out/BUILD-INFO.txt,build/out/*/BUILD-INFO.txt,build/out/*/*/BUILD-INFO.txt,build/out/*/*/*/BUILD-INFO.txt,build/out/target/product/*/howto/HOWTO_*.txt,build/out/target/product/*/HOWTO_*.txt" - -build-tools/node/lava-submit "$CONFIG" - -time build-tools/utils/new-publish/publish -p2 $JOB_NAME/$BUILD_NUMBER build/out/lava-job-info - -echo "Build finished" -</command> - </hudson.plugins.shell.ShellWithStatus> - </builders> diff --git a/utils/mangle-jobs/ci-tools-checkout.mangle b/utils/mangle-jobs/ci-tools-checkout.mangle deleted file mode 100644 index 46b9b5f..0000000 --- a/utils/mangle-jobs/ci-tools-checkout.mangle +++ /dev/null @@ -1,9 +0,0 @@ -# Update initial startup shell commands for a job - -old_cmd = 'bzr branch lp:~linaro-infrastructure/linaro-ci/lci-build-tools' -new_cmd = 'bzr branch lp:linaro-ci lci-build-tools' - -def mangle(tree): - tags = tree.xpath('/project/builders/hudson.tasks.Shell/command') - if tags: - tags[0].text = tags[0].text.replace(old_cmd, new_cmd) diff --git a/utils/mangle-jobs/copytoslave-remove.mangle b/utils/mangle-jobs/copytoslave-remove.mangle deleted file mode 100644 index 3ca70b5..0000000 --- a/utils/mangle-jobs/copytoslave-remove.mangle +++ /dev/null @@ -1,10 +0,0 @@ -# Remove CopyToSlave plugin config section - -from mangle_helper import * - - -def mangle(tree): - tags = tree.xpath('//com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper') - if not tags: - return - tags[0].getparent().remove(tags[0]) diff --git a/utils/mangle-jobs/cron-set-no-weekend.mangle b/utils/mangle-jobs/cron-set-no-weekend.mangle deleted file mode 100644 index b6091fe..0000000 --- a/utils/mangle-jobs/cron-set-no-weekend.mangle +++ /dev/null @@ -1,12 +0,0 @@ -# Set cron weekend spec to skip Sat - - -def mangle(tree): - nodes = tree.xpath("//hudson.triggers.TimerTrigger/spec") - if not nodes: - return False - spec = nodes[0].text.split() - if spec[-1] != "*": - return False - spec[-1] = "0-5" - nodes[0].text = " ".join(spec) diff --git a/utils/mangle-jobs/description-prepend.mangle b/utils/mangle-jobs/description-prepend.mangle deleted file mode 100644 index 8c08789..0000000 --- a/utils/mangle-jobs/description-prepend.mangle +++ /dev/null @@ -1,28 +0,0 @@ -# This script prepends a notice to each job's description -# (or replaces such notice if already there). - -from mangle_helper import * - - -PREPEND_TEXT = open("header.html").read() - -HEADER = "\n<!-- Automatically managed header - do not change, do not add anything before this! -->\n" -FOOTER = "<!-- End of automatically managed header -->" - -def mangle(tree): - cfg = get_build_config(tree) - build_type = cfg.get("BUILD_TYPE") - if not build_type: - build_type = cfg.get("SCRIPT_NAME", "build-android") - if build_type != "build-android": - return - - nodes = tree.xpath('/project/description') - if not nodes: return - descr = nodes[0].text - if descr is None: - descr = "" - if FOOTER in descr: - descr = descr[descr.find(FOOTER) + len(FOOTER):] - descr = HEADER + PREPEND_TEXT + "\n" + FOOTER + "\n\n" + descr - nodes[0].text = descr diff --git a/utils/mangle-jobs/email-ext-notification-set.mangle b/utils/mangle-jobs/email-ext-notification-set.mangle deleted file mode 100644 index 73b0dc0..0000000 --- a/utils/mangle-jobs/email-ext-notification-set.mangle +++ /dev/null @@ -1,20 +0,0 @@ -# -# Update email notification recepient list -# - -from mangle_helper import * - - -builders = 'linaro-android-builder-notifications@linaro.org' -restricted_builders = 'linaro-android-restricted-builder-notifications@linaro.org' - -def mangle(tree): - if not tree.xpath('//hudson.plugins.emailext.ExtendedEmailPublisher'): - # If node doesn't exist do nothing. - return - tags = tree.xpath('/project/publishers/hudson.plugins.emailext.ExtendedEmailPublisher/recipientList') - cfg = get_build_config(tree) - if cfg.get("BUILD_TYPE", "build-android") == "build-android-restricted": - tags[0].text = restricted_builders - else: - tags[0].text = builders diff --git a/utils/mangle-jobs/fingerprinter-add.mangle b/utils/mangle-jobs/fingerprinter-add.mangle deleted file mode 100644 index 61edd24..0000000 --- a/utils/mangle-jobs/fingerprinter-add.mangle +++ /dev/null @@ -1,25 +0,0 @@ -# Add Fingerprinting config - -from lxml.etree import fromstring - - -new_node = fromstring("""\ - <hudson.tasks.Fingerprinter> - <targets>build/fingerprints/*</targets> - <recordBuildArtifacts>false</recordBuildArtifacts> - </hudson.tasks.Fingerprinter> -""") - -def mangle(tree): - if tree.xpath("//hudson.tasks.Fingerprinter"): - # Needed config already there - don't change - return False - - parent_node = tree.xpath('/project/publishers')[0] - parent_node.append(new_node) - - # Adjust whitespacing around a node for pretty formatting - # You may need to update thelines below, based on a test run, - # if you reuse this script for another case. -# new_node.getprevious().tail += " " - new_node.tail = "\n " diff --git a/utils/mangle-jobs/header.html b/utils/mangle-jobs/header.html deleted file mode 100644 index 5ca927a..0000000 --- a/utils/mangle-jobs/header.html +++ /dev/null @@ -1,12 +0,0 @@ -<table width="65%" align="center"> -<tr><td> -<p style="background: #FFCCFF; padding: 15px"> -<span style="color: gray">New!</span><br /> -We now provide <tt>linaro_android_build_cmds.sh</tt> script (see downloads) -with exact commands to reproduce this build. Use it as a reference of -commands you need to execute, or just run to perform a build automatically. -There is also <tt>linaro_kernel_build_cmds.sh</tt> to rebuild just a kernel. -Alternatively, you can try instructions below. -</td></tr> -</p> -</table> diff --git a/utils/mangle-jobs/job-shell-commands-set.mangle b/utils/mangle-jobs/job-shell-commands-set.mangle deleted file mode 100644 index 81abacc..0000000 --- a/utils/mangle-jobs/job-shell-commands-set.mangle +++ /dev/null @@ -1,11 +0,0 @@ -# Update initial startup shell commands for a job - -new_cmd = '''\ -rm -rf build-tools -bzr get lp:linaro-android-build-tools build-tools -sudo -H -E build-tools/node/build us-east-1.ec2-git-mirror.linaro.org "$CONFIG" -''' - -def mangle(tree): - tags = tree.xpath('/project/builders/hudson.tasks.Shell/command') - tags[0].text = new_cmd diff --git a/utils/mangle-jobs/job-shell-commands-set2.mangle b/utils/mangle-jobs/job-shell-commands-set2.mangle deleted file mode 100644 index 5771450..0000000 --- a/utils/mangle-jobs/job-shell-commands-set2.mangle +++ /dev/null @@ -1,11 +0,0 @@ -# Update initial startup shell commands for a job - -new_cmd = '''\ -rm -rf build-tools -bzr get lp:linaro-android-build-tools build-tools -sudo -H -E build-tools/node/build us-east-1.ec2-git-mirror.linaro.org "$CONFIG" -''' - -def mangle(tree): - tags = tree.xpath('/project/builders/hudson.plugins.shell.ShellWithStatus/command') - tags[0].text = new_cmd diff --git a/utils/mangle-jobs/logparser-add.mangle b/utils/mangle-jobs/logparser-add.mangle deleted file mode 100644 index 597a66e..0000000 --- a/utils/mangle-jobs/logparser-add.mangle +++ /dev/null @@ -1,20 +0,0 @@ -# Add Log Parser plugin application to jobs - -#from lxml.etree import Element -from lxml.etree import fromstring - -def mangle(tree): - if tree.xpath("//hudson.plugins.logparser.LogParserPublisher"): - # Already there - return - tag = tree.xpath('//publishers')[0] - node = fromstring("""\ - <hudson.plugins.logparser.LogParserPublisher> - <unstableOnWarning>false</unstableOnWarning> - <failBuildOnError>false</failBuildOnError> - <parsingRulesPath>/var/lib/jenkins/userContent/android.parse</parsingRulesPath> - </hudson.plugins.logparser.LogParserPublisher> -""") - tag.append(node) - node.getprevious().tail += " " - node.tail = "\n " diff --git a/utils/mangle-jobs/mangle-jobs b/utils/mangle-jobs/mangle-jobs deleted file mode 100755 index 9c465e9..0000000 --- a/utils/mangle-jobs/mangle-jobs +++ /dev/null @@ -1,224 +0,0 @@ -#!/usr/bin/python -"""Helper to mass-edit jobs in jenkins. - -""" - -############################################################################### -# Copyright (c) 2011 Linaro -# All rights reserved. This program and the accompanying materials -# are made available under the terms of the Eclipse Public License v1.0 -# which accompanies this distribution, and is available at -# http://www.eclipse.org/legal/epl-v10.html -############################################################################### - -import base64 -from contextlib import nested -import json -import os -import sys -import copy -import re -from tempfile import NamedTemporaryFile -import urllib2 -import optparse -import getpass -from xml.dom import minidom - -from lxml.etree import fromstring, tostring - - -optparser = optparse.OptionParser(usage="%prog <mangle script>") -optparser.add_option("--url", default="http://localhost:8080/jenkins/", - help="Jenkins base url, default: %default") -optparser.add_option("--user", - help="Jenkins username") -optparser.add_option("--passwd-file", metavar="FILE", - help="File holding Jenkins password") -optparser.add_option("--really", action="store_true", - help="Actually perform changes") -optparser.add_option("--filter-jobname", - help="Process only jobs matching regex pattern") -optparser.add_option("--limit", type="int", default=-1, - help="Change at most LIMIT jobs") -optparser.add_option("--file", - help="Process a file instead of all jobs on a remote server") - -options, args = optparser.parse_args(sys.argv[1:]) -if len(args) != 1: - optparser.error("Wrong number of arguments") - -d = {} -execfile(args[0], d, d) -mangler = d['mangle'] - -password = None -if options.passwd_file: - password = open(options.passwd_file).read().strip() -elif not options.file: - password = getpass.getpass("Password/API Token:") - -if options.url[-1] != '/': - options.url += '/' - -auth_headers = { - 'Authorization': 'Basic %s' % ( - base64.encodestring('%s:%s' % (options.user, password))[:-1],), - } - -def _authJenkins(jenkins_path, data=None, extra_headers=None): - """Make an authenticated request to jenkins. - - @param jenkins_path: The path on the Jenkins instance to make the request - to. - @param data: Data to include in the request (if this is not None the - request will be a POST). - @param extra_headers: A dictionary of extra headers that will passed in - addition to Authorization. - @raises urllib2.HTTPError: If the response is not a HTTP 200. - @returns: the body of the response. - """ - headers = auth_headers.copy() - if extra_headers: - headers.update(extra_headers) - req = urllib2.Request( - options.url + jenkins_path, data, headers) - resp = urllib2.urlopen(req) - return resp.read() - -def getJobConfig(job_name): - return _authJenkins('job/' + job_name + '/config.xml') - -def postConfig(url, configXml, extra_headers=None): - headers = {'Content-Type': 'text/xml', } - if extra_headers is not None: - headers.update(extra_headers) - _authJenkins(url, configXml, headers) - -def render_xml(tree): - # Render XML to exact dialect used by Jenkins - # This involves some dirty magic - text = tostring(tree, xml_declaration=True, encoding='UTF-8') - # Roundtrip via minidom, this takes care of encoding " as entities - tree2 = minidom.parseString(text) - text = tree2.toxml('UTF-8') - - # expand empty tags - text = re.sub(r"<([-A-Za-z.]+)/>", "<\\1></\\1>", text) - - # Some CR noise should be entities - text = text.replace("\r", "
") - - # Finally, munge xml decl - line1, rest = text.split("><", 1) - line1 = line1.replace('"', "'") - r = line1 + ">\n<" + rest - - return r - -def show_diff(old, new): - with nested(NamedTemporaryFile(), NamedTemporaryFile()) as (a, b): - a.write(old) - b.write(new) - a.flush(); b.flush() - os.system('diff -u %s %s' % (a.name, b.name)) - print - -def indent_tree(elem, level=0): - "Indent XML tree for pretty-printing" - i = "\n" + level*" " - if len(elem): - if not elem.text or not elem.text.strip(): - elem.text = i + " " - if not elem.tail or not elem.tail.strip(): - elem.tail = i - for elem in elem: - indent_tree(elem, level+1) - if not elem.tail or not elem.tail.strip(): - elem.tail = i - else: - if level and (not elem.tail or not elem.tail.strip()): - elem.tail = i - -def normalize2text(tree): - """Return normalized text representation of XML tree, suitable for - diffing with normal diff tool.""" - normalized = copy.deepcopy(tree) - indent_tree(normalized) - return tostring(normalized) - -def match_job_name(job_name): - "Check if job name matches filters which may be specified on command line." - if not options.filter_jobname: - return True - neg = False - r = options.filter_jobname - if r[0] == "-": - neg = True - r = r[1:] - return bool(re.search(r, job_name)) ^ neg - -def get_csrf_token(): - try: - crumb_data = _authJenkins('crumbIssuer/api/xml') - except urllib2.HTTPError: - # Ignore errors for android-build which provides no crumb. - return None - tree = minidom.parseString(crumb_data) - crumb_tag = tree.getElementsByTagName('crumb')[0] - field_tag = tree.getElementsByTagName('crumbRequestField')[0] - crumb = str(crumb_tag.firstChild.wholeText) - field = str(field_tag.firstChild.wholeText) - return (field, crumb) - -def process_remote_jenkins(): - jobs = json.load(urllib2.urlopen(options.url + 'api/json?tree=jobs[name]')) - names = [job['name'] for job in jobs['jobs']] - names = [name for name in names if name == 'blank' or '_' in name] - limit = options.limit - - csrf_token = get_csrf_token() - if csrf_token is None: - extra_headers = None - else: - extra_headers = { csrf_token[0]: csrf_token[1], } - - for name in names: - if not match_job_name(name): - continue - if limit == 0: - break - limit -= 1 - print "Processing:" + name - sys.stdout.flush() - org_text = getJobConfig(name) - tree = fromstring(org_text) - org_normalized = normalize2text(tree) - - if mangler(tree) == False: - continue - - if not options.really: - new_normalized = normalize2text(tree) - show_diff(org_normalized, new_normalized) - else: - new_text = render_xml(tree) - if type(new_text) == type(u""): - new_text = new_text.encode("utf8") - postConfig(str('job/' + name + '/config.xml'), new_text, - extra_headers) - - -def main(): - if options.file: - text = open(options.file).read() - tree = fromstring(text) - - org_normalized = normalize2text(tree) - mangler(tree) - new_normalized = normalize2text(tree) - show_diff(org_normalized, new_normalized) - else: - process_remote_jenkins() - -if __name__ == "__main__": - main() diff --git a/utils/mangle-jobs/mangle_helper.py b/utils/mangle-jobs/mangle_helper.py deleted file mode 100644 index 363d2c2..0000000 --- a/utils/mangle-jobs/mangle_helper.py +++ /dev/null @@ -1,48 +0,0 @@ -import base64 -import lxml.etree - - -def get_build_config(tree): - tag = tree.xpath("/project/properties//defaultValue") - assert len(tag) == 1 - tag = tag[0] - d = {} - for l in base64.decodestring(tag.text).split("\n"): - l = l.strip() - if not l or l[0] == "#": - continue - k, v = l.split("=", 1) - d[k] = v - return d - - -def add_or_replace_node(tree, node_xpath, node_text): - new_node = lxml.etree.fromstring(node_text) - nodes = tree.xpath(node_xpath) - assert len(nodes) < 2, "Please use more selective XPath expression" - if nodes: - # First, add new node after existing - nodes[0].addnext(new_node) - # Then, delete the original node - nodes[0].getparent().remove(nodes[0]) - else: - parent_xpath, _ = node_xpath.rsplit("/", 1) - parent = tree.xpath(parent_xpath)[0] - parent.append(new_node) - - -def add_child(tree, node_xpath, node_text): - "Add new node (node_text) as last child of node_xpath." - nodes = tree.xpath(node_xpath) - assert len(nodes) == 1, "Found %d nodes with XPath %s, expected 1" % (len(nodes), node_xpath) - new_node = lxml.etree.fromstring(node_text) - nodes[0].append(new_node) - - -def add_sibling(tree, node_xpath, node_text): - "Add new node (node_text) as next sibling after node_xpath." - new_node = lxml.etree.fromstring(node_text) - nodes = tree.xpath(node_xpath) - assert len(nodes) > 0, "Node not found: %s" % node_xpath - assert len(nodes) < 2, "Please use more selective XPath expression than %s" % node_xpath - nodes[0].addnext(new_node) diff --git a/utils/mangle-jobs/null.mangle b/utils/mangle-jobs/null.mangle deleted file mode 100644 index e52032e..0000000 --- a/utils/mangle-jobs/null.mangle +++ /dev/null @@ -1,4 +0,0 @@ -# Make no changes, for identity testing - -def mangle(tree): - pass diff --git a/utils/mangle-jobs/project-based-sec-add.mangle b/utils/mangle-jobs/project-based-sec-add.mangle deleted file mode 100644 index d59faaf..0000000 --- a/utils/mangle-jobs/project-based-sec-add.mangle +++ /dev/null @@ -1,17 +0,0 @@ -# Add default project-based security setting for project (anon read access) - -from lxml.etree import fromstring -from mangle_helper import * - -new_node = """\ - <hudson.security.AuthorizationMatrixProperty> - <permission>hudson.model.Item.Read:anonymous</permission> - </hudson.security.AuthorizationMatrixProperty> -""" - -def mangle(tree): - if tree.xpath("//properties/hudson.security.AuthorizationMatrixProperty"): - # Needed config already there - don't change - return False - - add_child(tree, "/project/properties", new_node) diff --git a/utils/mangle-jobs/push-artifacts-set.mangle b/utils/mangle-jobs/push-artifacts-set.mangle deleted file mode 100644 index 67e0045..0000000 --- a/utils/mangle-jobs/push-artifacts-set.mangle +++ /dev/null @@ -1,26 +0,0 @@ -# -# This script is no longer used! -# -# See build-steps-set.mangle & builders.xml -# - - -new_value = "build/out/target/*/*/*.img," \ - "build/out/target/*/*/*.img.bz2," \ - "build/out/target/*/*/*.tar.bz2," \ - "build/out/target/*/*/MD5SUMS," \ - "build/out/*.tar.bz2," \ - "build/out/*.xml," \ - "build/out/*_config," \ - "build/out/lava-job-info," \ - "build/out/linaro_kernel_build_cmds.sh," \ - "build/out/linaro_android_build_cmds.sh," \ - "build/out/**/BUILD-INFO.txt," \ - "build/out/target/product/*/HOWTO_*.txt" \ - "build/out/target/product/*/howto/HOWTO_*.txt" - -def mangle(tree): - nodes = tree.xpath('//jenkins.plugins.publish__over__ssh.BapSshTransfer/sourceFiles') - # There're few such nodes, ours is first, but extra check to be sure - assert nodes[0].text.startswith('build/out/') - nodes[0].text = new_value diff --git a/utils/mangle-jobs/reset-cron.mangle b/utils/mangle-jobs/reset-cron.mangle deleted file mode 100644 index 3848ac4..0000000 --- a/utils/mangle-jobs/reset-cron.mangle +++ /dev/null @@ -1,8 +0,0 @@ -# Reset any triggers of a job, cron triggers in particular (but not limited) -# This is useful for sandboxes to avoid unexpected cron jobs runs - -def mangle(tree): - tag = tree.xpath('//triggers')[0] - for t in tag: - tag.remove(t) - tag.text = None diff --git a/utils/mangle-jobs/restricted-relabel.mangle b/utils/mangle-jobs/restricted-relabel.mangle deleted file mode 100644 index 131065e..0000000 --- a/utils/mangle-jobs/restricted-relabel.mangle +++ /dev/null @@ -1,11 +0,0 @@ -# Replace labels for restricted jobs - - -def mangle(tree): - nodes = tree.xpath("//assignedNode") - if not nodes: - return False - labels = nodes[0].text - if "private" not in labels: - return False - nodes[0].text = "ec2-restricted" diff --git a/utils/mangle-jobs/shell-status-set.mangle b/utils/mangle-jobs/shell-status-set.mangle deleted file mode 100644 index aab05e6..0000000 --- a/utils/mangle-jobs/shell-status-set.mangle +++ /dev/null @@ -1,11 +0,0 @@ -# Make 1st of 2 shell scripts we have run using shell-status plugin - -#from lxml.etree import Element -from lxml.etree import fromstring - -def mangle(tree): - tags = tree.xpath('//hudson.tasks.Shell') - if len(tags) != 2: - # Already applied - return - tags[0].tag = "hudson.plugins.shell.ShellWithStatus" diff --git a/utils/mangle-jobs/test.mangle b/utils/mangle-jobs/test.mangle deleted file mode 100644 index ea5dd84..0000000 --- a/utils/mangle-jobs/test.mangle +++ /dev/null @@ -1,7 +0,0 @@ -# Sample test mangle script - -def mangle(tree): - tags = tree.xpath('/project/description') - if not tags: return - tag = tags[0] - tag.text = 'Excitingly created.' |