Hi Wolfgang,
thanks for having taking the time to summarize the stumbling blocks you encountered. Your request is very much in line with my ambitions.
As Martin already mentioned, there is already an automated testing infrastructure in place. All the run scripts listed in tool/autopilot.list are executed on various kernels/boards every night - all in all there are about 700 test runs. So far, however, these tests were not visible for the public. From your mail, I get that there is indeed interest from the community in this "boring" topic. So it seems worthwhile to explore a way to open up the test environment.
Admittedly, the ambition to test all components goes beyond our capacities. I agree, though, that the tests should cover the fundamental components, i.e., those hosted in the base and os repositories. That said, I think that extending the coverage of our tests is unlikely to remove the stumbling blocks you encountered - simply because the tests won't cover many combinations of components.
This means at the moment if I use something and anything goes wrong there are many possible suspects:
- Documentation is outdated.
- Documentation has something missing which is written in some release
note/documentation
- Information from last release note is also already outdated
We need to know about those kind of issues. So far, however, we got almost no reports about stale documentation. If you encounter outdated documentation, please open a bug in the issue tracker or drop a note to the mailing list. I really wonder how often this is a problem for Genode users?
- I have done something wrong
- some component does not work as expected (e.g. from time to time you read
"haven't used it for some time, script no longer works")
It is a matter of fact that not all components and run scripts are considered as important by us regular developers. I.e., many run scripts are just vehicles for the respective developer to work on a particular component. The same goes for some components, which are just experimental. They are worth having at hand from time to time but they receive no attention otherwise. I think that this is perfectly fine.
What I take from your report, however, is that we should more clearly mark the line between run scripts and components that we deem as exemplary and stable, and those that are experimental/work-in-progress.
Then again, the experimental components are certainly the ones people like to play with as these are often prominently featured in the release notes. ;-)
To sum it up, with regard to the road map, we should consider:
* Opening up our automated test environment
* Finding a way to tell stable components apart from experimental ones
Cheers Norman