Why do we need the "Eclipse-Testing-Framework"

[laeubi]

In the past there was always some issues with the "Eclipse-Testing-Framework"

• https://wiki.eclipse.org/Platform-releng/Eclipse_Test_Framework
• https://wiki.eclipse.org/Platform-releng/Automated_Testing

Some of the issue include:

• Tests are not executed because they are not added to a special "AllTests"
• Dependencies are not properly fetched
• People are confused because we have a different (ant?) setup than in the verification builds
• ...

The recent discussions about JDT development should become more "modern" has bring this back into my mind that I always have wondered why "Eclipse-Testing-Framework" is required?

1. We are able to execute the test in regular maven verification builds (GitHub and Jenkins)
2. We can execute Eclipse Installations with Eclipse Run Mojo
3. We can even consume additional artifacts in Tycho builds

So what is the blocker here? Do we need any support from Tycho/Maven to maybe execute tests without compile them? Is running them "inside" the SDK zip an issue? Maybe something else?

As a very native aproach I would say we just need a Mojo that runs an Eclipse (isn't uiharness the way to do so already?) and probably select some (or all?) test-fragments and execute the precompiled ones with the current running system.

I'm specifically asking because I plan to improve testing-support for Tycho and covering this use-case would be a good addition here, but as mentioned I'm not really sure if there is actually something missing or if it is just no one wants to put effort in migration this stuff?

[akurtakov]

Being able to run test bundles' test suites on current system with predefined product on different os/java combos each bundle in clean osgi environment while skipping the compilation and packaging parts which Maven/Tycho kind of enforce now. As the tests are already terribly slow (up to 9 h on some combos) it's critical to not do the compile/package steps as that would seriously question the ability to have daily runs.
The rest are things like defining which test suite and etc. to run but this is already available on Maven side. An extra entry point/profile/whatever might be needed to define what to run but here we go into dirty details.
I'm really looking forward to retire this part of our code base but unfortunately it's last in the todo list.

[laeubi]

Being able to run test bundles' test suites on current system with predefined product on different os/java combos each bundle in clean osgi environment while skipping the compilation and packaging parts which Maven/Tycho kind of enforce now.

That's a part I really not sure about (and why I'm asking here), at least from the code perspective I don't se a reason why compile (or package) should be required at all.

So (possible to simple) assumption here is, that the only "issue" is that test-plugin is scanning the modules target dir, but instead should scan the classpath (or better the dependency path) and we should all be set? Maybe define used SDK zip anywhere?

The rest are things like defining which test suite and etc. to run but this is already available on Maven side.

Thats something I'm think will follow the BND (and PAX Exam is using that as well) to just record the tests in the package phase (e.g. as a file or header) that later is then investigated. But even I think this can be done dynamically if one knows what are the jars to scan (se idea above with having a dependency section) ...

An extra entry point/profile/whatever might be needed to define what to run but here we go into dirty details. I'm really looking forward to retire this part of our code base but unfortunately it's last in the todo list.

My first goal would be to replace one test fragment with the new approach that seems doable and then it is mostly a matter of transforming the others, but I first need to understand the quirks that might interfere here. So if you are aware of a very simply example let me know, I though Equinox might be good but I havent checked if equinox tests are actually executed that way...

[mickaelistria]

That's a part I really not sure about (and why I'm asking here), at least from the code perspective I don't se a reason why compile (or package) should be required at all.

Compiling can be seen as a 1st step of validation. It catches easily incompatibilities that would be hard to catch by using the packaged jar directory.
Or maybe there is a standard Java tool that can take 1 jar and 1 classpath and verify that all external types/methods/fields/APIs... referenced by the given jar can be resolved against a given target/classpath and report if some links are broken?
I would personally find it very useful.

[mickaelistria]

missing to add the test to the suite

This is not a particularity of the Test Framework. Similar issues would happen with Tycho/surefire and are more related to maintaining a JUnit test suite.

having missing dependencies somewhere

Do you have an example of a case where a missing dependency for te test framework wouldn't be equally missing to Tycho/surefire?

why don't we use maven/tycho to also execute these test

Again, improvements towards that would be welcome. But the improvements and Tycho support need to happen first; then we can start considering deprecating parts of the Test Framework.

To be clear, some requirement for official tests is that they need to run 1. without being rebuilt and 2. with same scripts on all OSs and @sravanlakkimsetti and @SDawley can probably tell more.
I am confident this can be made to work with Tycho on the current state because I managed to implement it for JBoss Tools some 6 or 7 years ago already; however it takes maybe more investigation for Platform to make sure we're not missing important requirement.

[sravanlakkimsetti]

As far as I remember eclipse platform started using maven from 4.0. The current build scripts are from 3.x time. The most popular build tool was ant at that time. If the testing can be moved to maven and tycho, yes you can move.

there are couple of requirements

1. Need to run tests in a clean environment. Download the zip file, extract it, install individual test bundle and its dependencies(these are specific to each of the test bundles. for example, pde has very specific dependencies to be installed). Run the tests. Restore the the eclipse instance to same status new download.
2. Need to have timeout for individual test and overall test suite
3. Need to take screenshot of the ui at the time of timeout.(screenshots are listed in the console logs page with extension png)
4. The scripts used are developed quite a long time ago. You can move to tycho if it is simpler that way
5. Need to publish the test results as html so that it can be read easily.

I hope most of these can be achieved by moving to tycho.

The main problem is that once in a while you change something, the verification builds runs fine, and then the SDK build next day gets KABOOOM ... then one starts to wonder and luckily someone step in and say: Have you maybe forgotten to adjust the (whatever) because we do not entirely use the maven build but have our famous (ant?) that works differently.

This happens other way as well. Sometimes ant will be able to run tests without any issues. but tycho/surefire fails. Since we are using tycho in our verification builds this may not occur that frequently.

[laeubi]

@sravanlakkimsetti thanks for this summary, this will help in the investigation!

This happens other way as well. Sometimes ant will be able to run tests without any issues. but tycho/surefire fails. Since we are using tycho in our verification builds this may not occur that frequently.

Yes, I don't wanted to indicate that test never fail with Tycho because of configuration problems, but it is a bit odd if the only fail for one run but not the other.

[laeubi]

@sravanlakkimsetti @mickaelistria @akurtakov I have one more questions, tests seem to need having a test.xml (see for example https://github.com/eclipse-equinox/equinox/blob/master/bundles/org.eclipse.equinox.http.servlet.tests/test.xml) is this some "magic file" the Eclipse Test Framework scans for? Is this file autogenerated or copied or is it customized for each project (or even on a test basis?)

[mickaelistria]

is this some "magic file" the Eclipse Test Framework scans for?

Yes. But it's not so magic; it basically defines whether tests are core or UI and what are the test suites to run. It's equivalent to configuration of the tycho-surefire-plugin.

Is this file autogenerated or copied or is it customized for each project (or even on a test basis?)

It is configured/customized for each project to define at least the test suite to run.

[akurtakov]

From releng perspective having POC that shows using few test bundles fetched from the p2 repo being run by maven in clean environment would be good first step for migration.

[mickaelistria]

There is https://tycho.eclipseprojects.io/doc/latest/tycho-surefire-plugin/test-mojo.html#testRuntime
One can create a .target that contains a product, create a Tycho module that declare the tests as dependencies and use this testRuntime configuration of tycho-surefire-plugin to see the configured tests (from the .target) running against the product (from the .target as well).