This is an automated email from the git hooks/post-receive script.

bernie.ogden pushed a change to branch bernie/benchmarking-uinstance-stability in repository toolchain/abe.

discards 9cdcb02 Support injection of metadata into test runs discards b8916ce Replace output_param with output_{value,slot} discards 87298b2 Apply NAME=VALUE CLI overrides before Benchmark.sh discards 85acbc2 Move output_param to top of Benchmark.sh discards fb7b8fd s/bench-session-debian/host-session/ discards d70ba5c Tighten CoremarkPro gather script, add many tests discards 2bb9080 Template for use in trusted LAVA uinstance discards ba6a77c Add missing tcpdump dependency discards 84fc0a5 Permit host machine to be of any type discards c90ff41 Close all tmpdir files before deletion discards 8944fa7 Benchmark.sh: Support mustang host/target discards b53ba5c Benchmark.sh: Support deploy_linaro_kernel discards 2f5b956 Preserve stdout/stderr separation in cmd output discards 075410c Output 'Running ${cmd}' to the log discards 54968a6 tee stderr to log, when there is a log. discards 5d8ca2b Always set safe_output when in a trusted LAVA discards 4f401ff controlledrun.sh logs to /dev/console if running in LAVA discards fbf9a3e controlledrun.sh output log set by board conf discards 03697fd Increase default timeout from 0.5 to 1.5 hours. discards 3993a47 Make TIMEOUT overrideable discards 0576985 Remove redundant boot_timeout parameter discards 2140659 Update CLI-related messages discards 08b9f11 Remove redundant 'tag' option discards 6a45d76 Some more detail around data collection message discards b1d38b1 Make ssh messages depend on instance discards c7d68f1 Defer to user-specified TARGET_SESSION discards 70bfb83 Rationalize *-session YAML discards f2c8b71 Do not infer device validation from triple discards ad07755 Decouple triple from toolchain discards 102be65 host-session: shift CLI args when parsing discards 0083842 host-session invokes benchmark scripts with -x discards 00a82f5 Remove redundant ABE_BRANCH parameter discards 4ba5012 host-session: Consistent use of {} with args discards 9026f13 gather scripts: Provide mime type for attachments discards fb1a1c9 benchmark.sh preserves -x for runbenchmark.sh discards 3593108 gather: set -x in all gather scripts discards 813f02f Return gather script exit code within post-run cmd discards bd382cc Wait for post-run command to complete discards b0b0c17 Convert post-run cmd to function discards 1ac71d7 Convert post-run command into multiline string discards 65c6aea Add comments about post-run/gather.sh discards 786dc32 target-session: Tidy up kill/wait code discards 6d989ec Simplify post-run-cmd-not-set NOP discards 19eea04 Check post-run command exit code discards 5a87b52 runbenchmark.sh: Fix indentation discards adee563 Match tcpdump output even if user gave username discards 92a253a Match tcpdump output even if user gave hostnames discards f0e2734 Do not report names in tcpdump discards 4d1af73 benchmark.sh: Rename GCC to something more generic discards 7437d4b Clean up toolchain usage discards ef058e5 Ensure that unset params get value None discards fdcac12 CoremarkPro.conf: Fix native paths discards c58ff94 CoremarkPro.sh: Report sizes first discards 6c108f5 CoremarkPro.sh: Require exactly one log discards 0644c0b CoremarkPro.sh: Remove commented-out code discards 438d596 CoremarkPro.sh: Pass name to report_measured discards baec681 CoremarkPro.sh: Make function-local var local discards d3109a0 CoremarkPro.sh: Count everything that can be multiple discards 43fba5d CoremarkPro.sh: Cosmetic changes to test IDs discards 5e5e4ed CoremarkPro.sh: Eliminate spaces in TEST_CASE_ID discards 0ad73de CoremarkPro.sh: [] notation for iterations only discards 031cbfa CoremarkPro.sh: Name size measurements sanely discards 1a9110b CoremarkPro.sh: Whitespace: Old comment,blank line discards 580fe0b test_CoremarkPro.sh: Don't output errors when testing fail cases discards 0a73077 CoremarkPro.sh: Attach raw data before processing discards 5d26fe5 CoremarkPro.sh: Unique names for time and rate discards 2dd9a93 Gather script for Coremark Pro discards 1dc8f04 dispatch-benchmark.py: Support Coremark Pro discards ec48461 CoremarkPro support discards 75f79b0 Add some (commented-out) debugging output discards 5976a66 Always read configs with target context discards 96c0e67 Apply target context when looking for binaries discards 055a623 Support external sysroot discards 07ed10f dispatch-benchmark.py: More detail on keyring fail discards e2449de dispatch-benchmark.py: Option to set bundle stream discards 8cb9b1b dispatch-benchmark.py: Validate input discards 2e775c6 dispatch-benchmark.py: Message on dry-run exit discards ae68211 dispatch-benchmark.py: Move dryrun exit to more natural location discards 03751fe dispatch-benchmark.py: Script to dispatch benchmarks discards 0f4db3f template.yaml: Metadata must be a string discards eccf356 Remove old jobdefs discards 47e366e Benchmark.sh,template.yaml: s/BUNDLE_STREAM_NAME/BUNDLE_STREAM/ discards 8acd803 Benchmark.sh: Make bundle stream default paranoid discards 35abce4 Benchmark.sh: Handle enquoting in parameters discards d5cf221 Benchmark.sh: Set BUNDLE_STREAM_NAME only if unset discards 8f40cde Benchmark.sh: Set the right default branch discards fdbb3dd Benchmark.sh: Remove redundant comments discards d728aa3 Benchmark.sh,template.yaml: Map from target type to image discards ae54a7a Benchmark.sh,template.yaml: Accept public key discards ca3a821 Benchmark.sh,template.yaml: s/ABE_/TESTDEF_/ discards 9285a78 Benchmark.sh: Make LAVA_USER available to params discards b0aa59d Benchmark.sh: Get bundle server from external LAVA_SERVER def discards 3519f8f Benchmark.sh: Braces-and-spaces discards fe7bc31 Benchmark.sh: Allow optional parms to be undefined discards 9216c42 Import template.yaml from lava-jobs discards 95a05fc Add script to derive all job parameters discards 3c4b2cc Remove redundant TODO discards 4eae6e9 WIP: 'Semi-trusted' concept, for testing purposes adds b204cdd Rework CPU2006.sh to use 'raw' output adds bb7e88a test_CPU2006.sh: Give doit a sensible name adds 983f3ad Rename *CPU2006.sh to *CPU200x.sh adds 94dc098 test_CPU200x: Paramaterise bits that vary for 2k adds 9e7c2c0 CPU200x*: Support CPU2000 results. adds 3a3e9b8 Lead regexp with ^ adds d1750e6 Remove unused variables adds d1c8566 raw is global, stop passing it around everywhere. adds fc8691f Handle different spellings of benchmark names adds bdda15a Move boilerplate to generate_benchmark and rename adds 199c6b6 Test cases where only int or fp was run adds 6bbfb61 Add (commented-out) debugging printf adds 8be0d1c Add a set -o pipefail adds 01c23fe Add exit, error handlers adds dcf59b2 Separate benchset definitions from individual runs adds 399b3f1 Add tests for individual benchmark runs adds 1e8a4ca Test random mixes of subbenchmarks adds 9753b68 Fix off-by-1 in iteration counts adds b39d49f Test non-ref input sizes adds 549471a Update error message adds 48f939d Add tests for cases where the script should fail adds 48be5f3 Correct naming style of test input adds 80f7cd6 Test more realistic values for input sizes adds aeed551 Print the leading 0 if integer part is 0 adds bcdad70 Make locals in generate_subbenchmark actually local adds 047be3e Multiply ratios for CPU2000 adds f6ec775 Use spec.cpu200x.invalid rather than ...valid adds 2aa9d8a Test case where some subbenchmarks fail adds ec39941 WIP: 'Semi-trusted' concept, for testing purposes adds c54aeb1 Remove redundant TODO adds 45b7b6a Add script to derive all job parameters adds 4246b1d Import template.yaml from lava-jobs adds 31ba4d2 Benchmark.sh: Allow optional parms to be undefined adds 439ea60 Benchmark.sh: Braces-and-spaces adds 92452c0 Benchmark.sh: Get bundle server from external LAVA_SERVER def adds 35304cd Benchmark.sh: Make LAVA_USER available to params adds 81b344d Benchmark.sh,template.yaml: s/ABE_/TESTDEF_/ adds 63475a5 Benchmark.sh,template.yaml: Accept public key adds 3b6a571 Benchmark.sh,template.yaml: Map from target type to image adds 2a92992 Benchmark.sh: Remove redundant comments adds 247523b Benchmark.sh: Set the right default branch adds 637d2df Benchmark.sh: Set BUNDLE_STREAM_NAME only if unset adds b570880 Benchmark.sh: Handle enquoting in parameters adds 72bd804 Benchmark.sh: Make bundle stream default paranoid adds a10bf01 Benchmark.sh,template.yaml: s/BUNDLE_STREAM_NAME/BUNDLE_STREAM/ adds 631f66b Remove old jobdefs adds 858b801 template.yaml: Metadata must be a string adds c69cfac dispatch-benchmark.py: Script to dispatch benchmarks adds e0c65e3 dispatch-benchmark.py: Move dryrun exit to more natural location adds c7e8f86 dispatch-benchmark.py: Message on dry-run exit adds aac012d dispatch-benchmark.py: Validate input adds 686e747 dispatch-benchmark.py: Option to set bundle stream adds 3326e0e dispatch-benchmark.py: More detail on keyring fail adds 18d9438 Support external sysroot adds f0b21cc Apply target context when looking for binaries adds 2d5cb47 Always read configs with target context adds d859677 Add some (commented-out) debugging output adds 8aa4179 CoremarkPro support adds a0e931d dispatch-benchmark.py: Support Coremark Pro adds da34e4d Gather script for Coremark Pro adds cc45976 CoremarkPro.sh: Unique names for time and rate adds dc788c1 CoremarkPro.sh: Attach raw data before processing adds fb29817 test_CoremarkPro.sh: Don't output errors when testing fail cases adds 5aff508 CoremarkPro.sh: Whitespace: Old comment,blank line adds c5c4cca CoremarkPro.sh: Name size measurements sanely adds 3b9a139 CoremarkPro.sh: [] notation for iterations only adds 0983a16 CoremarkPro.sh: Eliminate spaces in TEST_CASE_ID adds cace859 CoremarkPro.sh: Cosmetic changes to test IDs adds dc11405 CoremarkPro.sh: Count everything that can be multiple adds 15471cd CoremarkPro.sh: Make function-local var local adds 400ed9f CoremarkPro.sh: Pass name to report_measured adds 1a29fef CoremarkPro.sh: Remove commented-out code adds ba76222 CoremarkPro.sh: Require exactly one log adds 54294e4 CoremarkPro.sh: Report sizes first adds 51f8968 CoremarkPro.conf: Fix native paths adds d49b1cf Ensure that unset params get value None adds f9ba865 Clean up toolchain usage adds bbc24f4 benchmark.sh: Rename GCC to something more generic adds a6460a3 Do not report names in tcpdump adds 524d996 Match tcpdump output even if user gave hostnames adds 73e6e18 Match tcpdump output even if user gave username adds 202a1fb runbenchmark.sh: Fix indentation adds e97e0c1 Check post-run command exit code adds 598dbbf Simplify post-run-cmd-not-set NOP adds f64d29b target-session: Tidy up kill/wait code adds d5fbb5c Add comments about post-run/gather.sh adds 2567eda Convert post-run command into multiline string adds fa2edb7 Convert post-run cmd to function adds e2da049 Wait for post-run command to complete adds cade9d8 Return gather script exit code within post-run cmd adds 0925d63 gather: set -x in all gather scripts adds 73804fc benchmark.sh preserves -x for runbenchmark.sh adds 733f894 gather scripts: Provide mime type for attachments adds 8ea5146 host-session: Consistent use of {} with args adds 4b43ccb Remove redundant ABE_BRANCH parameter adds d7c4e10 host-session invokes benchmark scripts with -x adds 1355fa0 host-session: shift CLI args when parsing adds 237c840 Decouple triple from toolchain adds 4dd43f1 Do not infer device validation from triple adds 1071f91 Rationalize *-session YAML adds 64bbb11 Defer to user-specified TARGET_SESSION adds bf530ad Make ssh messages depend on instance adds dd73e3a Some more detail around data collection message adds 8c7b632 Remove redundant 'tag' option adds 5cabcee Update CLI-related messages adds fd15217 Remove redundant boot_timeout parameter adds 10a6d5e Make TIMEOUT overrideable adds 7935ece Increase default timeout from 0.5 to 1.5 hours. adds a8ba392 controlledrun.sh output log set by board conf adds 254f27c controlledrun.sh logs to /dev/console if running in LAVA adds 8c9803f Always set safe_output when in a trusted LAVA adds 21dcf20 tee stderr to log, when there is a log. adds 6ae1b98 Output 'Running ${cmd}' to the log adds b9a84c8 Preserve stdout/stderr separation in cmd output adds ccbb3a3 Benchmark.sh: Support deploy_linaro_kernel adds c0972b5 Benchmark.sh: Support mustang host/target adds 58d9c03 Close all tmpdir files before deletion adds f6eaed0 Permit host machine to be of any type adds 714232f Add missing tcpdump dependency adds 7df5dad Template for use in trusted LAVA uinstance adds 0a3303f Tighten CoremarkPro gather script, add many tests adds bd469b8 s/bench-session-debian/host-session/ adds 1a3da32 Move output_param to top of Benchmark.sh adds 2e8f956 Apply NAME=VALUE CLI overrides before Benchmark.sh adds 70dcb35 Replace output_param with output_{value,slot} adds 60cf45a Support injection of metadata into test runs new 5e4b212 Centralize default-setting and input validation new 81f2084 Make JOB_NAME user-settable new 9c27017 Only look up token in keychain for non-dry-runs new b4e49d2 Confirm that one of TOOLCHAIN and PREBUILT is set

This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this:

* -- * -- B -- O -- O -- O (9cdcb02) \ N -- N -- N refs/heads/bernie/benchmarking-uinstance-stability (b4e49d2)

You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B.

Any revisions marked "omits" are not gone; other references still refer to them. Any revisions marked "discards" are gone forever.

The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "adds" were already present in the repository and have only been added to this reference.

Summary of changes: config/bench/gather/CPU2006.sh | 185 -------------------- config/bench/gather/CPU200x.sh | 207 ++++++++++++++++++++++ config/bench/gather/test_CPU2006.sh | 108 ------------ config/bench/gather/test_CPU200x.sh | 337 ++++++++++++++++++++++++++++++++++++ scripts/Benchmark.sh | 32 +++- scripts/dispatch-benchmark.py | 30 +--- 6 files changed, 582 insertions(+), 317 deletions(-) delete mode 100755 config/bench/gather/CPU2006.sh create mode 100755 config/bench/gather/CPU200x.sh delete mode 100755 config/bench/gather/test_CPU2006.sh create mode 100755 config/bench/gather/test_CPU200x.sh