Updates from this week:

  • Why not run Python 3 conformance tests on Jenkins?

After some more discussion and code review, we finally got PR merged which makes sure that we run conformance tests in both Python 2 and 3 on Jenkins environment. The script looks very clean now and got to learn a little more of bash scripting by trying my hands on this.

  • Why not run conformance tests on Travis CI?

I really thought about this question and started tinkering with Travis config over the weekend. I was able to re-use the jenkins.bash script with almost no changes and was able to run it successfully on Travis CI. This required using language: bash as setup in the Travis config.

To parallelize the conformance and unit tests, I was able to use matrix command to have more elaborate setup for running complete Test Suite. Link to Travis Log and config. Micheal likes this idea and I might set it up on the original repo in the coming week.

  • Why not have classname for each test suite when running using CWL Test?

Now that we run conformance tests in both runntimes on Jenkins, we have the test result parsed as the following. You will see each tests run twice (in py2 and py3). Micheal opened an issue in cwltest repo to have command line argument to specify test class name when running a particular test suite. This will lead to better parsing of tests from the JUnit XML which is generated.