On 11/15/23 22:14, Mark Brown wrote:
On Wed, Nov 15, 2023 at 07:43:49PM +0200, Nikolai Kondrashov wrote:
Introduce a new tag, 'Tested-with:', documented in the Documentation/process/submitting-patches.rst file. The tag is expected to reference the documented test suites, similarly to the 'V:' field, and to certify that the submitter executed the test suite on the change, and that it passed.
This doesn't feel like it fits so well with git based flows - generally the tags end up in git one way or another so there'll be a strong tendency for this to end up getting added for one version and then carried forward to the next version. The way the tooling is at present it doesn't really feel like there's a good point at which to insert the tag.
I'm not sure exactly what'd be better though.
Yeah, I agree that's a bit of a problem. One that only automated tools/testing/CI could fully solve. Cough, git forges, cough.
OTOH, once you managed to run an automated suite once, it's much easier to do it again, and most of the time developers *want* their code to work and pass the tests (it's much easier than manual testing after all). So it's likely they will keep running them for new revisions, even though they might not notice they simply reused the previously-added Tested-with: tag.
Still, one way to make this better could be requiring a URL pointing at test results to follow the test suite name in the Tested-with: tag. Then the maintainer could check that they're indeed fresh.
This would be however getting way ahead of ourselves and, with the current (average) state of testing infra, hard to do. Perhaps sometime later.
For now, I think this could already go a long way towards having more (and better) testing.
Nick