Refactor and add roles for setup and provisioning

Add roles to handle build discovery and download, host provisioning (one role
per lab), and erp testing.

See README files in the playbook and roles for details.

Change-Id: Ia7b4cd999b274eecd67be43c77c0e9bfd8c34501
Signed-off-by: Dan Rue <dan.rue@linaro.org>

30 files changed, 652 insertions, 166 deletions

diff --git a/erp-playbook/.gitignore b/erp-playbook/.gitignore
index a8b42eb..f367774 100644
--- a/erp-playbook/.gitignore
+++ b/erp-playbook/.gitignore
@@ -1 +1,2 @@
 *.retry
+builds/
diff --git a/erp-playbook/LICENSE b/erp-playbook/LICENSE
new file mode 100644
index 0000000..963f5b0
--- /dev/null
+++ b/erp-playbook/LICENSE
@@ -0,0 +1,23 @@
+Copyright (c) 2017, Linaro Limited
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/erp-playbook/Makefile b/erp-playbook/Makefile
new file mode 100644
index 0000000..f5487b6
--- /dev/null
+++ b/erp-playbook/Makefile
@@ -0,0 +1,5 @@
+get-last-change:
+	sh local_scripts/check-for-changes.sh
+
+run-drue:
+	ansible-playbook -K -l erp-drue main.yml
diff --git a/erp-playbook/README.md b/erp-playbook/README.md
index 8294503..e5f7a5d 100644
--- a/erp-playbook/README.md
+++ b/erp-playbook/README.md
@@ -1,26 +1,55 @@
-ERP Test Harness
-================
+# ERP Test Harness
 
 This ansible playbook can be used to automate the ERP testing on a set of given
 hosts.
 
-Usage
------
+## Roles
 
-First, you will need a vault password. Ask the Linaro QA team, and then store
-it in '~/.vault_pass_erp.txt'.
-
-Modify hosts in 'hosts' file. Run `run.sh <build_number> <hostname>`.
-
-Ansible will ssh into the given host, install test-definitions prerequisites,
-clone test-definitions, run each test plan that is used for ERP testing, and
-post the results to https://qa-reports.linaro.org/ using the given build
-number.
+ERP testing is as easy as 1 2 3:
+1. Get a build
+2. Load a build on a host (provision)
+3. Install prerequisites on a host and run the tests autonomously
 
 The ansible run is idempotent; each time it is run, if the tests are already
 running on a host it will not do anything. If the tests are not running on a
 host, it will start them.
 
+### Find ERP build
+
+The `erp_get_build` role will ask jenkins for the latest build and then
+download the build locally. This role is required to be run before the
+subsequent roles.
+
+### Provision Host
+
+Depending on which lab a host is in (defined by their host group in `hosts`), a
+lab-specific role will be run to provision a host.
+
+The `erp_provision_austin_host` is used to provision hosts in the US Austin
+lab.
+
+The `erp_provision_cambridge_host` is used to provision hosts in the UK
+Cambridge lab.
+
+### Run ERP Test Suite
+
+The `erp_run_test_suite` will ssh into the given host, install test-definitions
+prerequisites, clone test-definitions, run each test plan that is used for ERP
+testing, and post the results to https://qa-reports.linaro.org/ using the given
+latest build number (discovered in `erp_get_build`).
+
+## Usage
+
+See the `Makefile` for example usecases.
+
 Note that 'hosts' and 'group_vars/all' files are encrypted using ansible vault.
 To use, the vault password will have to be available in ~/.vault_pass_erp.txt.
 Once set, use `ansible-vault edit` to view and modify.
+
+See `hosts.dist` and `group_vars/erp_all.dist` for examples of inventory
+definitions.
+
+To run against a given host or host group, run `ansible-playbook -l
+<hostname|hostgroup> main.yml`. If -l (limit) is not used, ansible will attempt
+to run against every host defined in `hosts`.
+
diff --git a/erp-playbook/ansible.cfg b/erp-playbook/ansible.cfg
index 2375d42..8b279b1 100644
--- a/erp-playbook/ansible.cfg
+++ b/erp-playbook/ansible.cfg
@@ -1,3 +1,4 @@
 [defaults]
 host_key_checking = False
 vault_password_file = ~/.vault_pass_erp.txt
+inventory = hosts
diff --git a/erp-playbook/group_vars/all b/erp-playbook/group_vars/all
deleted file mode 100644
index f7934bf..0000000
--- a/erp-playbook/group_vars/all
+++ /dev/null
@@ -1,36 +0,0 @@
-$ANSIBLE_VAULT;1.1;AES256
-65616530646134366630663835373033633064353533623664313363643131303166386432616633
-6166373066653736373431363462356234313536666436370a376639646264646131386138613031
-34623065306339373030656462396530303433393961383564336337356661326630646661373333
-6431653961336432660a343061623535633964653432353762346335343532653334363663323333
-62363066656563633239643661353139653339626632396565373830333339366433316436323365
-35636366623465356462333364303864396137666339343836343031646431346665316164663936
-61653661373166613731643961663965316363616665373137333135386333373531306661383636
-65316339353137363136646432343364636364613233623132393065366339333262353537393930
-37663563326630353661623462663536343433633565313965616430623532346161653238663265
-35363838343566303832383464643663383333356238633730363366636165313666313165633966
-32613462646539366563383263613236316538343162653032626266653734373265623632636565
-62366437613431316538633137623263326632383965316533393139663334306531363461353166
-35313562346430633734343034383065663533623932666139663039343163323233656265623334
-35373730353630643662386534633465303465626534306539663636633236346632636334343864
-32366639303661333936623232666462353733366436646465333231303765356134306339373837
-63393137643430363762316363363166386233393835646332646366663339343462376637656462
-36663833643833643532386333313136383233666661346162356261373531346631323463316530
-36386661666635353862616566386262393231386266353534666163636235323966316637356332
-62646633303066646239376232626536303830343534363464633638393138333038316632633237
-64633034363039636234636232373830346561653431313237316136633637393839393164313533
-62356136313563316234366661303534636330343362333863363535633361613739653339386532
-36396266343934653639393563303133366462666136626237613866336462316634396135333932
-63343865336163373434643130396239386466353964303334636436353434656135623335366163
-64663036336537353330653631323164333437353235386639333832633565303438303636346635
-62396262616333353137636534653339633333363039373233616433643534636434376435336338
-62613638363566323162343062666331356361666432336235393061333538383862636230326630
-34633663333035303430626137313463656464393934363962653734613239643134346137653031
-35323332313262363432353966636437383531353134633330613866386431386335346432616261
-39356430383234653834353234613366343035366239313639363434626661363833363831343466
-34626639616139313466386166613861366232356533653634633533383363313737313435396464
-35383664326134393464366530343165343831303738383264363238313031623939626238363366
-64653737616339303534623635653430383934636633373535653137623838386239626439666230
-61363130306333393164663335336331616637363463663330653231643834336134663065616233
-62653834636239376332336435616639633933326230313132313963323061356638363334376134
-333736313962613533373837303433626631
diff --git a/erp-playbook/group_vars/all_erp b/erp-playbook/group_vars/all_erp
new file mode 100644
index 0000000..778e916
--- /dev/null
+++ b/erp-playbook/group_vars/all_erp
@@ -0,0 +1,36 @@
+$ANSIBLE_VAULT;1.1;AES256
+65306636333130313337616532383532373133383337333464303031326236336439613233336336
+3234366462613236646432336233373466353839363434300a616264356635633636613835363262
+62373936633131613437356438336434623165643536386363306534323939663736643964353230
+3665306536336437330a376530353236316134653564343463306438666630333333366535393135
+33636561366436646135353132336664343239643363393831373035623135626164343234623665
+63643761393231393131666261396636336137656362613364313262616538343430303365646432
+31313637303531363835326430303363353362666366663566343563643363353834333263313932
+62333239613636356339316366613961356361393063636666323562303937396366646532356137
+31653965386237396530663537306635306263366562376262366632346164336435336566636562
+63373765653538326563323461666438333465616230346563656163363231386261306563366638
+64643632313466383963663835666132636566663739643137326435333465343736636663646439
+36343935306562333166613831623238343864346562336537613261333739366365393235393339
+66376130653032323363363530663430306638333631613231653730623161306530373234636134
+64613164613361316539613135663964303938386164373364323833373164356233343065396438
+36626130653162303562386532363664343335646336313563343432326539393133343530656232
+35373961663333353839383765656235613434383633313235373330303932343666396139636432
+34373261363166646136393564643432326163376137393932633031353166386630616164643637
+30356337356530303937376265306237633931626433353265663830383132616436636662333038
+32343032653064313363356362663531656134383537363364623338653862656164646264663135
+35343965316238633732653934633630356263356631663762366136613936306164643238393537
+39616663323936326130613464623762663339383534363963393265663834303437383865343631
+32653231346136323663663362373262626233643264306261646536356630383238393139326530
+65373636353966393630396462633462663239633432643833386634363430383964353461636230
+65653430353163643564653565333038343663393065373531613762343039353864343030393834
+38353165323339636234623334383863636332613831373732366332386664323133353839303138
+31646230303334636265633037383066396333333165303438663031393334646365306262373830
+38353733346264323862636261653233353834303832646461633832616634633738316238363466
+61316363366439663335336462383866343433373039303263346532396263346361313963653663
+61373161636532623461363265383762333731613631366335313336643035353037373864663539
+66616239626362323030623465363464346133356663356537613730346264633938643262663431
+61356238393862653366376339663466633339363061643530376539353639393761636233323631
+34666163303536656537386464356461356233623733373737626235303436396566356234323661
+39376563343037353161343131646665346234663538366531313932616564336231616266333166
+34336632303632623635303734653831613633613061346530343465366662383063353634356434
+393263613764353561666631396365393638
diff --git a/erp-playbook/group_vars/all_erp.dist b/erp-playbook/group_vars/all_erp.dist
new file mode 100644
index 0000000..66478c9
--- /dev/null
+++ b/erp-playbook/group_vars/all_erp.dist
@@ -0,0 +1,14 @@
+# Default username and password for each host in ../hosts
+# If a host needs a different settings, put it in ../hosts
+# These should match the preseed file used during host provisioning.
+ansible_user: linaro
+ansible_ssh_pass: linaro
+ansible_become_pass: root
+ansible_become: yes
+ansible_become_method: su
+
+squad_auth_tokens:
+  production: "MY_SECRET_TOKEN"
+  staging: "MY_SECRET_TOKEN"
+
+ansible_ssh_common_args: '-o UserKnownHostsFile=/dev/null'
diff --git a/erp-playbook/hosts b/erp-playbook/hosts
index 53831f1..a85797a 100644
--- a/erp-playbook/hosts
+++ b/erp-playbook/hosts
@@ -1,48 +1,85 @@
 $ANSIBLE_VAULT;1.1;AES256
-35666635663836313735346463633438346335393332373633623233383130373063383136356564
-6430623666396434616637363364376337626530343934630a336436396636326366313865663635
-37303763363539303865623236653235346633383539353939326436653864336634353937383863
-6633656463373936660a353066356331336630356638346435643338633136633636303135363431
-39356130383030313234643037666561363765393039626264383338626139326634386462626162
-66346538353963323035653461363564383665396662666634646464656266626164356336373563
-38376436333764396431646433316563303136343831633836633137643261393536303161666233
-36326630343461663666393963666265633364373065636333643131323437396539643763653337
-37636230363336653466303562666364363831626133346565313266376463393332643765323764
-30303833353031656133393965383262386433383564616430326130396664613032306434303166
-37326638653339326630326261343537343237373336663839306635333239656362643132333764
-34313362323238326330353632366239386538663538646265666232366161303939633164646636
-39303636643162356235373565323036653861616332666163303231343833333930663730666431
-38323538306534353762663333383861393338666461383833303231383232636335376632636661
-34373936653162383535316661343532656538633731646431313631316530353430626630353539
-38376336333731613834653036343533336337353563316564653434656266633739376665393630
-61383333663535663936336335656263666237323261333563356530616161646133636532626262
-66306134343033313139386436396435646435646566666566386339613764383738306664313233
-32623664633334373138316339313763326162313738376335393938643632363938323261356534
-39366363353230633031343963633563303431386238326338393439313437643737386130323065
-34353866393332666231656631313365313434646532343439636437383466323331643937386530
-66396232343432336464663036363861303364383261633633333837636533396632633434383533
-32633366353832363331303231353966383263623739383236393034363637376431346563333330
-63616539323933373163623533666437376537643234666561306231346165303961323033633237
-66343434333431343036373361353638396336303363633938313032633139666631663166653335
-34663433323832383364303863333365643564343139396431626361353666636439343065356634
-36623836363963356465353063393262333231353531323332393136326639623066666265663566
-38303362663033313865393433363436356537323438666539313163313764633861333365623965
-66653861393733636661326261666464373464623634353261306332646634373464663164643161
-66386434303631333839626562666263653832643739346439396130643831323830336632633865
-63306436643261376236336231623364643532636362353465666564343032353339336131343762
-34306139303337373439313062346533663538333366663033376534336136346563366138376338
-32653738383639663961623265633161393732386430326137353863623930623834386336346566
-33366235326663626139633733323930356538323436373761643035393730643966386531613234
-63393163323136643635393533363830656466396132666239343333346638613039626430313565
-63666364623330626632646538323539333032353263623265343437626533376165636231626437
-38313466663038346239353530613261653532393235366636343564333462313364616537626565
-66636265623031393235653766303766656265303033643835626139653162393565393632663838
-65343262313335643632393163353962653730333430353061626461363864356261363433323738
-36346137643665613663653166323062373138663530633931336530356166623331363335663236
-62613637666363626131326234333766346239353836343164393139366632393134663436616261
-30356339666233313332663262663335306264613464636264613562656539313137613737646162
-32646231316632323633353162393433323837626361323739386562373462366466613538306566
-31343964643233346432353139386266323862336261643532363936313339663033363461633135
-34333834383135653864393162363735336339373330623761393339363338383962323964333334
-39313531303861653631383537363663363338323862633935353439353166643563356661336664
-663264323163643761633937303731333036
+36333937643033626433363536306539613836356430393366353635393764616662393430376135
+3662653062616266643531646264326166313235353363310a663231326137323436376661636533
+31303232346561393361316431643134306537393532396639326535323239303662363232346661
+6539623434343666390a633861336131313335303237623762306135633430326661383836386430
+36383263356537646434376365336332346563623831386365346161353161346264633539646663
+61343334653462353536306266343730653839306533363236336464343435306666323966666566
+61336562373532323462326365333036383863343261366564333538343766663961303235663134
+39346461363439653535663634313638386263663132666465396631633234636434356539323037
+62373164376236306665383230653830336636653534316639333636373435386361623561373866
+37653533613037316532613633376466383361353037313037613431636437643133373233383666
+39363664316438613738616634386536653233313837666636373231313434636666376539663836
+30643032333331643939316164626130353232323366383431393662386638303032313836373838
+38316232393637383466373136353932323564623939376639313933666532643832333733396663
+61663132643031373466343131353431663735346330376235393831613636623134626330663265
+36323631636636386563353235313135393862383930353261636662626162663336623061313738
+30323764396535333166356237353164666133656639323430376637633363326235643166356336
+30643331353733306338373136346534326535326663613438366237613765626434346138376236
+62376365333532326365373735383139663366656362636230623133303732666661343262633364
+30306535646336613866643832633737623737316335313361306134653134373430333064383438
+61313862623362303331643338373637623532643235323061633866363738326430636530333031
+34366435613132343931373866373661623365343264323464393336393138663665356331653261
+66626235393432363038666432643634353864383561666232353137336466656330643165316130
+64646565653030343461353065393835663966643031623838303535386434346635376663303930
+65656339383235383664383366633561383739346235343539306539316232366366616134316161
+62396564383733633736626366343033636230333432636139366165333638346661353538383764
+35353138353332333432353139643864373530333635663234643561666266346364386336336166
+39323830396233616466663730366132646432373636356361306563316434643230623862636262
+31303631386564373862333134356230303537356436396635393630643838656637303139383165
+66633064336164336665646136323939643462303033306264313163646534393163313864333961
+37666462393634363930626434323230333731373337613666346634643537346534353563333634
+30626361323364336431306632393437653035613466633438613536633334316333343135303264
+36313662653332656531373435326536393163383961313962353233636433396461313137313637
+63303031666431316133666266343365326266316239346439653863616535393262623962373265
+64306533643730663133353834626338303761333662363566643139353735636635383164333663
+62373664386562626361373039306165316436323835616663663165633733613139663934316334
+66303730303063663336623632383665346239313232343032323530363637613261373637653235
+34363037303365323937653562663161306235633366326566656532663263323635636639336162
+37316132386135366535396132646332303065653366306561356639666564346438633830363164
+30313738316562353039653234303130393261336331636536653634383231653038373634643838
+37356131616530353537306162383463393762653261326430363830303137363865373539643837
+37363166653366643733363238373436356439306633623263356330363231346236313463623666
+64303836376130656639653431646662363239333438333536343334336439323633333430663238
+61653932316665663832353362343238343634346634633164346237356266643438653630383332
+33616538343164346139633064393263306362366532613665303565386265636137663330303164
+38383339306462623938636230306164333461316639613262393934306362326239656564373334
+32306139373030356466656532636134366635306338343933313066633263353932353231386166
+64303932383765396534363137336361616461383234373432373330383762393265363063386338
+35346162353266383334303565393163343233313566646637333437396461643136333038623964
+38653038383632323630343866343237666330383363303133366364656636646639346561636634
+35616430616435326665333231643963346131666336396339316462643935656261303465353933
+66313230373734343035386136376162383536663763623932326463316239643035376132656237
+32366138343938333462363364666161636639656366636563313133653564326361323866323264
+35303166616634643635613034343330656362306638316537303266336334623663303936636331
+39316539623930353530386637343661616131366165643735316531393033633163393138626331
+30643737653666633466633530336135383832656538366231303037396330616435646462653834
+36653338363739373533386339393964343234666166363331373734386364653666626237363037
+31393761653064363263643565303732373534656465623431343062636263393065663462353331
+37663433366135343139346561356664313633383161346436363935343564366232643438333439
+37323137376131373539643861633332643966666236653461366665303165663036616133633065
+32306162386564633337633162623963633063393038343465613664653664396237333264613633
+64613761616239363266666139316664656538336333393331396166383864353238386132386363
+66616235613030396538613332346566363236326330613831616266653065643432356262323938
+64353531373932656363306137373136366233356132623265346130623939323262343239303465
+31653031373934376338653031323763643261656363666266303238323061373333633364643661
+66323430363261333466393261356365346236356339613039623935613961353866666332313931
+39643562643839646136326132316531363666316263656232376537636364363231643566663566
+30613564383535306532353835373336636337646166386339383735313766653538363962366632
+30303864666638323665623162326564356430303433316537633835653134636466373165656437
+35383062646464626435323562373363333565373839666634393834323031336336386133386234
+32386232666633306632663163363338323233383364623361656231646330653034663638376366
+66613162346437303066373261356338383035316430653231363535396134313764383466343534
+30346337313139656235643338363832633135323539303561326334316366623463646436393962
+62633930646533363030313364383038386665333033363764636366613965623439303965383538
+34336635346464353837633131306664643236343562343536636539306637643631333831633762
+38316530623536376533356164643138616130393534626466363764383766396665363262306431
+35613838353431666535313930343730303837373833326537383930346162393337643464306661
+65393233653062333034336439306331346633333632303732613361623033343334353265626231
+37353131626233623735643266346332653037333030353363383138353232353562663832333431
+32353331336236323534326631663162663866633438633362656663373036316135333630633435
+64333265623831623838616165303963323665363162353765653437303566323963373639333632
+36663165333434616663353637656163303833656333363334386434643034363230343933316461
+61396163616638353234383264393264336363656265313562393439313330343335373239383063
+64633563656436623164393561343330356461373266653664396239383065666534363833366562
+39383638633636373830
diff --git a/erp-playbook/hosts.dist b/erp-playbook/hosts.dist
new file mode 100644
index 0000000..64876f6
--- /dev/null
+++ b/erp-playbook/hosts.dist
@@ -0,0 +1,30 @@
+# *_erp groups are groups of ERP boards, with the same user and passwords
+# all_erp is a meta-group that contains all erp boards.
+
+[local]
+localhost ansible_connection=local
+
+[cambridge_erp]
+# Cambridge Lab
+cb-r1-crb1s-02 ansible_host=10.101.2.100
+cb-r2-d03-04 ansible_host=10.101.2.111
+cb-r2-d05-01 ansible_host=10.101.2.107
+cb-r2-d05-03 ansible_host=10.101.2.22
+cb-r1-m1-c33n1 ansible_host=10.101.3.88
+cb-r1-m1-c34n1 ansible_host=10.101.3.89
+
+[open_estuary_erp]
+# Open Estuary Lab
+oe-d03-251 # Connection parameters in ~/.ssh/config
+
+[all_erp:children]
+# All erp hosts
+cambridge_erp
+open_estuary_erp
+
+[erp-drue]
+# Specific run group consisting of already defined hosts
+cb-r1-crb1s-02 # ThunderX
+cb-r2-d03-04 # D03
+cb-r2-d05-01 # D05
+cb-r1-m1-c33n1 # Moonshot
diff --git a/erp-playbook/local_scripts/check-for-changes.sh b/erp-playbook/local_scripts/check-for-changes.sh
new file mode 100644
index 0000000..5379148
--- /dev/null
+++ b/erp-playbook/local_scripts/check-for-changes.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+URLs=$(cat <<EOF
+    http://repo.linaro.org/debian/erp-17.08-stable/dists/jessie/main/binary-arm64/Packages.bz2
+    http://repo.linaro.org/debian/erp-17.08-staging/dists/jessie/main/binary-arm64/Packages.bz2
+    http://repo.linaro.org/debian/erp-17.08-staging/dists/jessie/main/debian-installer/binary-arm64/Packages.gz
+EOF
+)
+
+RESULT=""
+for url in $URLs; do
+
+    LAST_CHANGE="$(curl --silent -I ${url} | grep ^Last-Modified | sed 's/Last-Modified: //')"
+    # Convert to a sortable datetime
+    LAST_CHANGE=$(date -d "${LAST_CHANGE}" --iso-8601=seconds --utc)
+
+    if [ -n "${RESULT}" ]; then
+        RESULT="${RESULT}\n"
+    fi
+    RESULT="${RESULT}${LAST_CHANGE}"
+
+done
+
+#echo "${RESULT}"
+#echo ""
+echo "Last Change: $(echo "${RESULT}" | sort -n | tail -1)"
+
diff --git a/erp-playbook/main.yml b/erp-playbook/main.yml
index 1a1f093..8de38bc 100644
--- a/erp-playbook/main.yml
+++ b/erp-playbook/main.yml
@@ -1,61 +1,20 @@
----
-- hosts: erp
-  become: yes
-  become_method: su
+- hosts: all_erp
   vars:
     squad_environment: production
-
-  tasks:
-  - name: Verify build_number is set
-    assert:
-      that:
-        - "build_number > 0"
-
-  - name: Check for /root/test-definitions
-    stat:
-      path: /root/test-definitions
-    register: tests_run
-  - name: Verify host is clean
-    fail:
-      msg: Error - It looks like tests have already run on this host.
-    when: tests_run.stat.exists
-
-  - name: Package Prerequisites
-    apt:
-      update_cache: yes
-      cache_valid_time: 86400
-      pkg: "{{item}}"
-    with_items:
-      - git
-      - python-pip
-      - daemon
-      - sysstat # for troubleshooting
-      - strace # for troubleshooting
-
-  - name: Clone test-definitions repository
-    git:
-      repo: 'https://git.linaro.org/qa/test-definitions.git'
-      dest: /root/test-definitions
-      version: erp-17.08
-
-  - name: Install python prerequisites
-    pip:
-      requirements: /root/test-definitions/automated/utils/requirements.txt
-
-  - name: Install python prerequisites
-    pip:
-      name: python-slugify
-
-  - name: Copy test harness
-    template:
-      src: run_erp_suite.sh
-      dest: /root/run_erp_suite.sh
-      mode: 0755
-
-  - name: Run ERP test suite
-    # daemon --name ensures only one runs at a time
-    command: daemon --name=erp_suite --errlog=/root/run_erp_suite.stderr.log --dbglog=/root/run_erp_suite.stdout.log -- /root/run_erp_suite.sh {{build_number}}
-    environment:
-      SQUAD_AUTH_TOKEN: "{{squad[squad_environment]['auth_token']}}"
-
-
+    debian_installer_environment: staging
+
+  pre_tasks:
+    - set_fact:
+        erp_squad_auth_token: "{{squad_auth_tokens[squad_environment]}}"
+
+  roles:
+    - role: erp_get_build
+      delegate_to: localhost
+      run_once: true
+    - role: erp_provision_cambridge_host
+      delegate_to: localhost
+      when: "'cambridge_erp' in group_names"
+    - role: erp_provision_austin_host
+      delegate_to: qa-pxe
+      when: "'austin_erp' in group_names"
+    - role: erp_run_test_suite
diff --git a/erp-playbook/roles/erp_get_build/README.md b/erp-playbook/roles/erp_get_build/README.md
new file mode 100644
index 0000000..93d8dcc
--- /dev/null
+++ b/erp-playbook/roles/erp_get_build/README.md
@@ -0,0 +1,51 @@
+ERP Get Build
+=============
+
+Download an Enterprise Reference Platform (ERP) release.
+
+By default, this role will discover the latest staging ERP build from jenkins,
+set 'erp_latest_build' number, and download the build to ./builds/staging/.
+
+Requirements
+------------
+
+None
+
+Role Variables
+--------------
+
+In:
+
+| variable | description | default
+|----------|-------------|---------
+| erp_debian_installer_environment | [staging|release] | staging
+
+Out:
+
+| variable | description | example
+|----------|-------------|---------
+| erp_latest_build | Latest build number, based on debian_installer_environment | 430
+
+Dependencies
+------------
+
+None
+
+Example Playbook
+----------------
+
+Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too:
+
+    - hosts: servers
+      roles:
+        - role: get_erp_build
+
+License
+-------
+
+BSD
+
+Author Information
+------------------
+
+Dan Rue <dan.rue@linaro.org>
diff --git a/erp-playbook/roles/erp_get_build/defaults/main.yml b/erp-playbook/roles/erp_get_build/defaults/main.yml
new file mode 100644
index 0000000..ce959ba
--- /dev/null
+++ b/erp-playbook/roles/erp_get_build/defaults/main.yml
@@ -0,0 +1,14 @@
+---
+
+# Which debian installer to use. Defaults to staging, but may also be
+# 'release'.
+erp_debian_installer_environment: staging
+
+erp_debian_installer_jenkins_urls:
+  staging: "https://ci.linaro.org/view/reference-platform/job/96boards-reference-debian-installer-staging/api/json"
+  release: "https://ci.linaro.org/view/reference-platform/job/96boards-reference-debian-installer/api/json"
+
+erp_debian_installer_download_urls:
+  staging: "http://builds.96boards.org/snapshots/reference-platform/components/debian-installer-staging/"
+  release: "http://builds.96boards.org/snapshots/reference-platform/components/debian-installer/"
+
diff --git a/erp-playbook/roles/erp_get_build/meta/main.yml b/erp-playbook/roles/erp_get_build/meta/main.yml
new file mode 100644
index 0000000..6bca1d8
--- /dev/null
+++ b/erp-playbook/roles/erp_get_build/meta/main.yml
@@ -0,0 +1,12 @@
+galaxy_info:
+  author: Dan Rue <dan.rue@linaro.org>
+  description: Download an ERP release
+  company: Linaro
+
+  license: BSD
+
+  min_ansible_version: 2.2
+
+  galaxy_tags: []
+
+dependencies: []
diff --git a/erp-playbook/roles/erp_get_build/tasks/main.yml b/erp-playbook/roles/erp_get_build/tasks/main.yml
new file mode 100644
index 0000000..3a2fe3e
--- /dev/null
+++ b/erp-playbook/roles/erp_get_build/tasks/main.yml
@@ -0,0 +1,22 @@
+---
+- name: Get latest build number from Jenkins
+  uri:
+    url: "{{erp_debian_installer_jenkins_urls[erp_debian_installer_environment]}}"
+  register: jenkins_build_info
+- set_fact:
+    erp_latest_build: "{{jenkins_build_info['json']['lastCompletedBuild']['number']}}"
+
+- debug:
+    msg: "Latest debian installer {{erp_debian_installer_environment}} build is {{erp_latest_build}}"
+
+- name: Create local builds directory
+  file:
+    path: "./builds/debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}"
+    state: directory
+- name: Download build locally
+  get_url:
+    url: "{{erp_debian_installer_download_urls[erp_debian_installer_environment]}}/{{erp_latest_build}}/debian-installer/arm64/{{item}}"
+    dest: "./builds/debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}/{{item}}"
+  with_items:
+    - "initrd.gz"
+    - "linux"
diff --git a/erp-playbook/roles/erp_provision_austin_host/README.md b/erp-playbook/roles/erp_provision_austin_host/README.md
new file mode 100644
index 0000000..90c1f24
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_austin_host/README.md
@@ -0,0 +1,29 @@
+ERP Provision Austin Host
+=========================
+
+Provision a host in the Austin lab.
+
+Requirements
+------------
+
+This role expects erp_get_build to run first.
+
+Role Variables
+--------------
+
+None
+
+Dependencies
+------------
+
+erp_get_build role.
+
+License
+-------
+
+BSD
+
+Author Information
+------------------
+
+Dan Rue <dan.rue@linaro.org>
diff --git a/erp-playbook/roles/erp_provision_austin_host/defaults/main.yml b/erp-playbook/roles/erp_provision_austin_host/defaults/main.yml
new file mode 100644
index 0000000..e2d4b45
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_austin_host/defaults/main.yml
@@ -0,0 +1,2 @@
+---
+# defaults file for erp_provision_cambridge_lab
\ No newline at end of file
diff --git a/erp-playbook/roles/erp_provision_austin_host/meta/main.yml b/erp-playbook/roles/erp_provision_austin_host/meta/main.yml
new file mode 100644
index 0000000..809040d
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_austin_host/meta/main.yml
@@ -0,0 +1,12 @@
+galaxy_info:
+  author: Dan Rue <dan.rue@linaro.org>
+  description: Provision a host in the Cambridge lab.
+  company: Linaro
+
+  license: BSD
+
+  min_ansible_version: 2.2
+
+  galaxy_tags: []
+
+dependencies: []
diff --git a/erp-playbook/roles/erp_provision_austin_host/tasks/main.yml b/erp-playbook/roles/erp_provision_austin_host/tasks/main.yml
new file mode 100644
index 0000000..d99c73e
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_austin_host/tasks/main.yml
@@ -0,0 +1,54 @@
+---
+- name: Verify variables are set
+  assert:
+    that: "{{ item }} is defined"
+    msg: Run erp_get_build role first
+  with_items:
+    - erp_debian_installer_environment
+    - erp_latest_build
+
+- name: Create destination path
+  file:
+    path: "/var/lib/tftpboot/debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}"
+    state: directory
+- name: Copy over new build
+  copy:
+    src: "./builds/debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}/{{item}}"
+    dest: "/var/lib/tftpboot/debian-staging/{{erp_latest_build}}/{{item}}"
+  with_items:
+    - "initrd.gz"
+    - "linux"
+- name: Update default for host /var/lib/tftpboot/grub.cfg
+  blockinfile:
+    backup: yes
+    path: /var/lib/tftpboot/grub.cfg
+    marker: "# {mark} Default for {{ansible_default_ipv4['macaddress']}}"
+    block: |
+      if [ "${net_default_mac}" = "{{ansible_default_ipv4['macaddress']}}" ]; then
+          set timeout=5
+          set default='Install Debian Jessie - RP {{erp_debian_installer_environment}} - Image {{erp_latest_build}} - Automated'
+      fi
+
+- name: Update menuentry /var/lib/tftpboot/grub.cfg
+  blockinfile:
+    path: /var/lib/tftpboot/grub.cfg
+    insertafter: EOF
+    marker: "# {mark} debian-{{erp_debian_installer_environment}} {{erp_latest_build}}"
+    block: |
+      menuentry 'Install Debian Jessie - RP {{erp_debian_installer_environment}} - Image {{erp_latest_build}} - Automated' {
+          linux /debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}/linux module_blacklist=mlx5_core auto=true interface=auto priority=critical noshell BOOT_DEBUG=1 DEBIAN_FRONTEND=text url=http://people.linaro.org/~dan.rue/erp-test-automation/releases/17.08/debian/preseed.cfg ---
+          initrd /debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}/initrd.gz
+      }
+
+# XXX: Replace with automation script
+- name: Connect to console
+  pause:
+    prompt: "In a separate window, run the following to attach to the amberwing: 'ssh -t aus-colo.linaro.org ipmitool -I lanplus -A PASSWORD -U admin -P Password1 -H {{inventory_hostname}}-bmc sol activate' and then press <enter> to reboot"
+
+- name: amberwing - reboot
+  command: ipmitool -I lanplus -A PASSWORD -U admin -P Password1 -H {{inventory_hostname}}-bmc chassis power cycle
+
+# TODO: Replace with automation script
+- name: Reboot machines and install OS
+  pause:
+    prompt: "Install new OS. <enter> once complete"
diff --git a/erp-playbook/roles/erp_provision_cambridge_host/README.md b/erp-playbook/roles/erp_provision_cambridge_host/README.md
new file mode 100644
index 0000000..843e3fd
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_cambridge_host/README.md
@@ -0,0 +1,29 @@
+ERP Provision Cambridge Host
+============================
+
+Provision a host in the Cambridge lab.
+
+Requirements
+------------
+
+This role expects erp_get_build to run first.
+
+Role Variables
+--------------
+
+None
+
+Dependencies
+------------
+
+erp_get_build role.
+
+License
+-------
+
+BSD
+
+Author Information
+------------------
+
+Dan Rue <dan.rue@linaro.org>
diff --git a/erp-playbook/roles/erp_provision_cambridge_host/defaults/main.yml b/erp-playbook/roles/erp_provision_cambridge_host/defaults/main.yml
new file mode 100644
index 0000000..e2d4b45
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_cambridge_host/defaults/main.yml
@@ -0,0 +1,2 @@
+---
+# defaults file for erp_provision_cambridge_lab
\ No newline at end of file
diff --git a/erp-playbook/roles/erp_provision_cambridge_host/meta/main.yml b/erp-playbook/roles/erp_provision_cambridge_host/meta/main.yml
new file mode 100644
index 0000000..809040d
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_cambridge_host/meta/main.yml
@@ -0,0 +1,12 @@
+galaxy_info:
+  author: Dan Rue <dan.rue@linaro.org>
+  description: Provision a host in the Cambridge lab.
+  company: Linaro
+
+  license: BSD
+
+  min_ansible_version: 2.2
+
+  galaxy_tags: []
+
+dependencies: []
diff --git a/erp-playbook/roles/erp_provision_cambridge_host/tasks/main.yml b/erp-playbook/roles/erp_provision_cambridge_host/tasks/main.yml
new file mode 100644
index 0000000..9bed8d0
--- /dev/null
+++ b/erp-playbook/roles/erp_provision_cambridge_host/tasks/main.yml
@@ -0,0 +1,22 @@
+- name: Verify variables are set
+  assert:
+    that: "{{ item }} is defined"
+    msg: Run erp_get_build role first
+  with_items:
+    - erp_debian_installer_environment
+    - erp_latest_build
+
+# TODO: Replace with API call, once supported.
+- name: Upload build to Cambridge Lab
+  pause:
+    prompt: "Upload ./builds/debian-{{erp_debian_installer_environment}}/{{erp_latest_build}}/ files to http://172.27.80.1:5000/admin/images with description 'debian-installer {{erp_debian_installer_environment}} Build {{erp_latest_build}}'. Press <enter> key when complete"
+
+# TODO: Replace with API call, once supported.
+- name: Edit Machines
+  pause:
+    prompt: "In Mr. Provisioner, modify each machine in your host group to {{erp_latest_build}}. <enter> once complete"
+
+# TODO: Replace with API call, once supported.
+- name: Reboot machines and install OS
+  pause:
+    prompt: "In Mr. Provisioner, PXE Reboot each machine to install new OS. <enter> once complete"
diff --git a/erp-playbook/roles/erp_run_test_suite/README.md b/erp-playbook/roles/erp_run_test_suite/README.md
new file mode 100644
index 0000000..0ef7a31
--- /dev/null
+++ b/erp-playbook/roles/erp_run_test_suite/README.md
@@ -0,0 +1,35 @@
+ERP Run Test Suite
+==================
+
+Run the Enteprise Reference Platform (ERP) test suite against a given host.
+
+Requirements
+------------
+
+
+Role Variables
+--------------
+
+| variable | description | default
+|----------|-------------|---------
+| erp_squad_environment | [production|staging] | production
+| erp_squad_auth_token | Squad API auth token | No default - required
+
+
+Dependencies
+------------
+
+
+Example Playbook
+----------------
+
+
+License
+-------
+
+BSD
+
+Author Information
+------------------
+
+Dan Rue <dan.rue@linaro.org>
diff --git a/erp-playbook/roles/erp_run_test_suite/defaults/main.yml b/erp-playbook/roles/erp_run_test_suite/defaults/main.yml
new file mode 100644
index 0000000..3e3427c
--- /dev/null
+++ b/erp-playbook/roles/erp_run_test_suite/defaults/main.yml
@@ -0,0 +1,7 @@
+---
+
+erp_squad_environment: production
+erp_squad_url:
+  production: "https://qa-reports.linaro.org"
+  staging: "https://staging-qa-reports.linaro.org"
+
diff --git a/erp-playbook/roles/erp_run_test_suite/meta/main.yml b/erp-playbook/roles/erp_run_test_suite/meta/main.yml
new file mode 100644
index 0000000..f3ca3e0
--- /dev/null
+++ b/erp-playbook/roles/erp_run_test_suite/meta/main.yml
@@ -0,0 +1,12 @@
+galaxy_info:
+  author: Dan Rue <dan.rue@linaro.org>
+  description: Run the Enteprise Reference Platform (ERP) test suite.
+  company: Linaro
+
+  license: BSD
+
+  min_ansible_version: 2.2
+
+  galaxy_tags: []
+
+dependencies: []
diff --git a/erp-playbook/roles/erp_run_test_suite/tasks/main.yml b/erp-playbook/roles/erp_run_test_suite/tasks/main.yml
new file mode 100644
index 0000000..2d962f2
--- /dev/null
+++ b/erp-playbook/roles/erp_run_test_suite/tasks/main.yml
@@ -0,0 +1,55 @@
+---
+- name: Verify variables are set
+  assert:
+    that: "{{ item }} is defined"
+  with_items:
+    - erp_latest_build
+    - erp_squad_environment
+    - erp_squad_auth_token
+
+- name: Check for /root/test-definitions
+  stat:
+    path: /root/test-definitions
+  register: tests_run
+- name: Verify host is clean
+  fail:
+    msg: Error - It looks like tests have already run on this host.
+  when: tests_run.stat.exists
+
+- name: Package Prerequisites
+  apt:
+    update_cache: yes
+    cache_valid_time: 86400
+    pkg: "{{item}}"
+  with_items:
+    - git
+    - python-pip
+    - daemon
+    - sysstat # for troubleshooting
+    - strace # for troubleshooting
+
+- name: Clone test-definitions repository
+  git:
+    repo: 'https://git.linaro.org/qa/test-definitions.git'
+    dest: /root/test-definitions
+    version: erp-17.08
+
+- name: Install python prerequisites
+  pip:
+    requirements: /root/test-definitions/automated/utils/requirements.txt
+
+- name: Install python prerequisites
+  pip:
+    name: python-slugify
+
+- name: Copy test harness
+  template:
+    src: run_erp_suite.sh
+    dest: /root/run_erp_suite.sh
+    mode: 0755
+
+- name: Run ERP test suite
+  # daemon --name ensures only one runs at a time
+  command: daemon --name=erp_suite --errlog=/root/run_erp_suite.stderr.log --dbglog=/root/run_erp_suite.stdout.log -- /root/run_erp_suite.sh {{erp_latest_build}}
+  environment:
+    SQUAD_AUTH_TOKEN: "{{erp_squad_auth_token}}"
diff --git a/erp-playbook/templates/run_erp_suite.sh b/erp-playbook/roles/erp_run_test_suite/templates/run_erp_suite.sh
index 0580eca..6ee554b 100644
--- a/erp-playbook/templates/run_erp_suite.sh
+++ b/erp-playbook/roles/erp_run_test_suite/templates/run_erp_suite.sh
@@ -5,7 +5,7 @@ if [ -z $1 ]; then
     exit 1
 fi
 
-report_url="{{squad[squad_environment]['url']}}"
+report_url="{{erp_squad_url[erp_squad_environment]}}"
 plans="plans/rpb_ee/rpb_ee_functional.yaml plans/rpb_ee/rpb_ee_enterprise.yaml plans/rpb_ee/rpb_ee_performance.yaml plans/rpb_ee/rpb_ee_ltp.yaml"
 
 root_path=/root
diff --git a/erp-playbook/run.sh b/erp-playbook/run.sh
deleted file mode 100644
index a6052f1..0000000
--- a/erp-playbook/run.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/sh
-
-if [ -z $1 ] || [ -z $2 ]; then
-    echo "Usage: $0 <build_number> <hostname>"
-    echo "Run erp tests with a given build_number against a given hostname"
-    echo "hostname must be defined in hosts."
-    exit 1
-fi
-
-ansible-playbook -i hosts main.yml -e build_number=${1} --limit ${2}