Hi Theodore,
On 11/20/23 22:51, Theodore Ts'o wrote:
On Mon, Nov 20, 2023 at 02:30:49PM +0100, Ricardo Cañuelo wrote:
This is not trivial because tests vary a lot and we'd first need to define which artifacts to link to, and because whatever is linked (test commands, output log, results summary) would need to be stored forever. But since we're doing that already for basically all kernel mailing lists, I wonder if something like "public-inbox for test results" could be possible some day.
What we have at work is a way to upload the test results summary (e.g., just KTAP result lines, or the xfstests junit XML) along with test run metadata (e.g., what was the kernel commit on which the test was run, and the test hardware), and this would be stored permanently. Test artifacts is also preserved but for a limited amount of time (e.g., some number of months or a year).
The difference in storage lifetimes is because the junit XML file might be a few kilobytes to tens of kilobytes. but the test artifacts might be a few megabytes to tens of megabytes.
Of course once you have this data, it becomes possible to detect when a test may have regressed, or to detect flaky tests, and perhaps to figure out if certain hardware configurations or kernel configurations are more likely to trigger a particular test to fail. So having all of this data stored centrally would be really cool. The only question is who might be able to create such an infrastructure, and be able to pay for the ongoing development and operational costs....
Yes, I agree, having public result storage would be awesome. I think KCIDB is relatively-well positioned to take on some of that work. We have the POC dashboard already. Well, *had* until somebody scraped it and exhausted our cloud budget, I'm working on making it cheaper before bringing it back.
Meanwhile you can get a glimpse of it in one of my slide decks: https://static.sched.com/hosted_files/eoss2023/ef/Kernel%20CI%20%E2%80%93%20...
We also have an artifact-mirroring system (quite basic at the moment), expecting the submitter to already have the files hosted somewhere. We would need to add a file upload service to that.
Finally, I'm considering opening up submissions to (Google-authenticated) public, as opposed to just CI systems, so we could support this. That's still more work though.
Regarding analysis, I have plenty of ideas for that too, and even more ideas are explored by others lately. I just need to bring back the dashboard and find the time for all of it :D
Anyone interested in helping with any of this?
Nick