On Fri, May 20, 2011 at 7:33 PM, Richard Sandiford richard.sandiford@linaro.org wrote:
Michael Hope michael.hope@linaro.org writes:
How about also 'Ensure vectorised code doesn't regress over non-vectorised code'? The goal would be for 90 % of benchmarks to not regress and 99 % to regress no more than 2 %. At the moment good 'ol CoreMark is worse with -O3 -omfpu=neon...
Well, I suppose if we're setting figures like that, then it's really "Limit regressions in vectorised code over non-vectorised code". :-) But maybe it'd be better to keep figures out of it. 99% is awkward because I don't think we even have 100 benchmarks yet. And what about benchmarks like DENbench that run the same code more than once, but with a different data set? Does each data set count as a separate benchmark?
I felt a bit silly writing the 99 % thing. How about 'Ensure vectorised code doesn't regress over non-vectorised in almost all cases. Ensure vectorised doesn't regress by more than n % in all cases.' with some type of escape clause for one benchmark which is too hard for this cycle.
-- Michael