Discussion:
[gem5-dev] Finalizing new testing framework
(too old to reply)
Jason Lowe-Power
2018-07-05 20:22:31 UTC
Permalink
Hi all,

I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.

The patches are linked below:

https://gem5-review.googlesource.com/c/public/gem5/+/4880/7
https://gem5-review.googlesource.com/c/public/gem5/+/4881/7
https://gem5-review.googlesource.com/c/public/gem5/+/4421/12
https://gem5-review.googlesource.com/c/public/gem5/+/4882/13
https://gem5-review.googlesource.com/c/public/gem5/+/4883/13
https://gem5-review.googlesource.com/c/public/gem5/+/4422/13
https://gem5-review.googlesource.com/c/public/gem5/+/4423/13
https://gem5-review.googlesource.com/c/public/gem5/+/10121/4

Thanks,
Jason
Gabe Black
2018-07-09 11:14:42 UTC
Permalink
Hi Jason, thanks for making sure everybody has a chance to chime in. I
haven't looked at the changes in a while, but I'm also comfortable with
them going in without looking them over again given that other people have
+2-ed them, the high likelihood it will improve on the status quo, and that
we should be able to fix any serious emergent problems as they become
apparent. Thanks for keeping this going!

Gabe
Post by Jason Lowe-Power
Hi all,
I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.
https://gem5-review.googlesource.com/c/public/gem5/+/4880/7
https://gem5-review.googlesource.com/c/public/gem5/+/4881/7
https://gem5-review.googlesource.com/c/public/gem5/+/4421/12
https://gem5-review.googlesource.com/c/public/gem5/+/4882/13
https://gem5-review.googlesource.com/c/public/gem5/+/4883/13
https://gem5-review.googlesource.com/c/public/gem5/+/4422/13
https://gem5-review.googlesource.com/c/public/gem5/+/4423/13
https://gem5-review.googlesource.com/c/public/gem5/+/10121/4
Thanks,
Jason
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Andreas Sandberg
2018-07-13 08:37:40 UTC
Permalink
Hi Jason,

Thanks for doing this. I haven't had a chance to look at the code in details myself, but from what I have seen it looks like a real improvement. Ideally, I'd like someone from our side (probably Giacomo T) to have a look at it. However, Giacomo is currently travelling and I have other pressing matters to tend to. :(

I have had a quick look at the code and it all looks reasonable. There are three things we need for CI integration:

1. Ability to separate build and test.
2. Ability to schedule individual (long-running) tests separately in a cluster environment.
3. Ability to get a JUnit XML file with per-test results.

With the potential exception of 2, I think the new framework supports all of the above. We could, maybe, use the support for distributed test using the test server in the framework, but I'm a bit reluctant to deploy it for two reasons. 1) It requires credentials to be hard-coded. It would be much more convenient to use an auto-generated authentication file (with suitable permissions) in a shared file location. 2) Having a custom server is potentially very unreliable since the network tends to be noisy (e.g., port scanners, IDS software). The static allocation of a port will also be troublesome (e.g., when two servers run on the same node).

In the current framework, we typically list the available tests and then create one cluster job per test. Once the tests complete, the CI framework aggregates all of the JUnit files (it also stores the pickle files for debugging) and generates a report. I'd like to be able to do something with the new framework. Is that possible?

Cheers,
Andreas

On 05/07/2018 21:22, Jason Lowe-Power wrote:

Hi all,

I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.

The patches are linked below:

https://gem5-review.googlesource.com/c/public/gem5/+/4880/7
https://gem5-review.googlesource.com/c/public/gem5/+/4881/7
https://gem5-review.googlesource.com/c/public/gem5/+/4421/12
https://gem5-review.googlesource.com/c/public/gem5/+/4882/13
https://gem5-review.googlesource.com/c/public/gem5/+/4883/13
https://gem5-review.googlesource.com/c/public/gem5/+/4422/13
https://gem5-review.googlesource.com/c/public/gem5/+/4423/13
https://gem5-review.googlesource.com/c/public/gem5/+/10121/4

Thanks,
Jason
_______________________________________________
gem5-dev mailing list
gem5-***@gem5.org<mailto:gem5-***@gem5.org>
http://m5sim.org/mailman/listinfo/gem5-dev

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Jason Lowe-Power
2018-08-02 23:21:05 UTC
Permalink
Hi all,

Some more follow up. Sorry that this is going so slowly.

Sean just did a pretty major reworking of the main library. See
https://gem5-review.googlesource.com/4421. I'll let him address Andreas's
questions below. I think we're pretty close to having all three.

I would really like to get this pushed in ASAP so someone (me, a student,
or others in the community) can start migrating all of the current tests
over to the new framework.

As a quick preview, this is what the output looks like now.
cd tests
./main.py run
Running the new gem5 testing script.
For more information see TESTING.md.
To see details as the testing scripts are running, use the option -v, -vv,
or -vvv
======================================================================================================================
Loading Tests
Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py
Discovered 30 tests and 15 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py
Discovered 6 tests and 3 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py
Discovered 21 tests and 21 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py
... <a few errors while we are transitioning>
======================================================================================================================
Running Tests from 9 suites
Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results
======================================================================================================================
Building the following targets. This may take a while.
/home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt,
/home/jlp/Code/gem5/gem5/build/X86/gem5.opt,
/home/jlp/Code/gem5/gem5/build/ARM/gem5.opt
You may want to run with only a single ISA(--isa=), use --skip-build, or
use 'retest'.
Test: testhello64-static-X86-opt Passed
Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-dynamic-X86-opt Passed
Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-X86-opt Passed
Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-static-ARM-opt Passed
Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-ARM-opt Passed
Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed
Test: m5_exit_test-X86-opt Passed
Test: m5_exit_test-X86-opt-MatchRegex Passed
Test: build-X86-opt Passed
Test: build-RISCV-opt Passed
Test: build-ARM-opt Passed
====================================== Results: 15 Passed in 2.6e+01
seconds ========================================

Cheers,
Jason
Hi Jason,
Thanks for doing this. I haven't had a chance to look at the code in
details myself, but from what I have seen it looks like a real improvement.
Ideally, I'd like someone from our side (probably Giacomo T) to have a look
at it. However, Giacomo is currently travelling and I have other pressing
matters to tend to. :(
I have had a quick look at the code and it all looks reasonable. There are
1. Ability to separate build and test.
2. Ability to schedule individual (long-running) tests separately in a
cluster environment.
3. Ability to get a JUnit XML file with per-test results.
With the potential exception of 2, I think the new framework supports all
of the above. We could, maybe, use the support for distributed test using
the test server in the framework, but I'm a bit reluctant to deploy it for
two reasons. 1) It requires credentials to be hard-coded. It would be much
more convenient to use an auto-generated authentication file (with suitable
permissions) in a shared file location. 2) Having a custom server is
potentially very unreliable since the network tends to be noisy (e.g., port
scanners, IDS software). The static allocation of a port will also be
troublesome (e.g., when two servers run on the same node).
In the current framework, we typically list the available tests and then
create one cluster job per test. Once the tests complete, the CI framework
aggregates all of the JUnit files (it also stores the pickle files for
debugging) and generates a report. I'd like to be able to do something with
the new framework. Is that possible?
Cheers,
Andreas
Hi all,
I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.
https://gem5-review.googlesource.com/c/public/gem5/+/4880/7https://gem5-review.googlesource.com/c/public/gem5/+/4881/7https://gem5-review.googlesource.com/c/public/gem5/+/4421/12https://gem5-review.googlesource.com/c/public/gem5/+/4882/13https://gem5-review.googlesource.com/c/public/gem5/+/4883/13https://gem5-review.googlesource.com/c/public/gem5/+/4422/13https://gem5-review.googlesource.com/c/public/gem5/+/4423/13https://gem5-review.googlesource.com/c/public/gem5/+/10121/4
Thanks,
Jason
_______________________________________________
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
Andreas Sandberg
2018-08-03 09:49:33 UTC
Permalink
OK, that sounds reasonable. We shouldn't have any issues with that.

I'd like someone from us to have a look as well, most likely Giacomo since I won't be able to find the time myself. However, don't let that block anything. Last time I looked at the code, it looked reasonable. I'm sure we can fix any issues once we start integrating it into our test setup.

Thanks,
Andreas

On 03/08/2018 03:26, Sean Wilson wrote:
My attempt to address your concerns, Andreas..


Your issue with the baked-in test server was definitely reasonable. I've actually taken that part out of the library since it was so unreliable. A reliable test server would probably be a whole library in itself.

My suggestion for running distributed tests would be to do basically the same thing you do now: list all the suites and kick off a distributed task for each suite uid. Each test is individually labeled with a UID. When using the `./main.py run` command you can supply the test suite uid with `--uid <SUITE-UID>`. That test and that test only will run.

Your current xml aggregation solution sounds reasonable to me. The only issue you might run into is that a single result.xml is written to, so if all of gem5 is served over NFS you might get conflicts writing to the file. I can add an output path flag if you think this might be an issue for you.


Cheers,
Sean

On Thu, Aug 2, 2018 at 6:21 PM, Jason Lowe-Power <***@lowepower.com<mailto:***@lowepower.com>> wrote:
Hi all,

Some more follow up. Sorry that this is going so slowly.

Sean just did a pretty major reworking of the main library. See https://gem5-review.googlesource.com/4421. I'll let him address Andreas's questions below. I think we're pretty close to having all three.

I would really like to get this pushed in ASAP so someone (me, a student, or others in the community) can start migrating all of the current tests over to the new framework.

As a quick preview, this is what the output looks like now.
cd tests
./main.py run
Running the new gem5 testing script.
For more information see TESTING.md.
To see details as the testing scripts are running, use the option -v, -vv, or -vvv
======================================================================================================================
Loading Tests
Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py
Discovered 30 tests and 15 suites in /home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py
Discovered 6 tests and 3 suites in /home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py
Discovered 21 tests and 21 suites in /home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py
... <a few errors while we are transitioning>
======================================================================================================================
Running Tests from 9 suites
Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results
======================================================================================================================
Building the following targets. This may take a while.
/home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt, /home/jlp/Code/gem5/gem5/build/X86/gem5.opt, /home/jlp/Code/gem5/gem5/build/ARM/gem5.opt
You may want to run with only a single ISA(--isa=), use --skip-build, or use 'retest'.
Test: testhello64-static-X86-opt Passed
Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-dynamic-X86-opt Passed
Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-X86-opt Passed
Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-static-ARM-opt Passed
Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-ARM-opt Passed
Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed
Test: m5_exit_test-X86-opt Passed
Test: m5_exit_test-X86-opt-MatchRegex Passed
Test: build-X86-opt Passed
Test: build-RISCV-opt Passed
Test: build-ARM-opt Passed
====================================== Results: 15 Passed in 2.6e+01 seconds ========================================

Cheers,
Jason

On Fri, Jul 13, 2018 at 1:37 AM Andreas Sandberg <***@arm.com<mailto:***@arm.com>> wrote:

Hi Jason,

Thanks for doing this. I haven't had a chance to look at the code in details myself, but from what I have seen it looks like a real improvement. Ideally, I'd like someone from our side (probably Giacomo T) to have a look at it. However, Giacomo is currently travelling and I have other pressing matters to tend to. :(

I have had a quick look at the code and it all looks reasonable. There are three things we need for CI integration:

1. Ability to separate build and test.
2. Ability to schedule individual (long-running) tests separately in a cluster environment.
3. Ability to get a JUnit XML file with per-test results.

With the potential exception of 2, I think the new framework supports all of the above. We could, maybe, use the support for distributed test using the test server in the framework, but I'm a bit reluctant to deploy it for two reasons. 1) It requires credentials to be hard-coded. It would be much more convenient to use an auto-generated authentication file (with suitable permissions) in a shared file location. 2) Having a custom server is potentially very unreliable since the network tends to be noisy (e.g., port scanners, IDS software). The static allocation of a port will also be troublesome (e.g., when two servers run on the same node).

In the current framework, we typically list the available tests and then create one cluster job per test. Once the tests complete, the CI framework aggregates all of the JUnit files (it also stores the pickle files for debugging) and generates a report. I'd like to be able to do something with the new framework. Is that possible?

Cheers,
Andreas

On 05/07/2018 21:22, Jason Lowe-Power wrote:

Hi all,

I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.

The patches are linked below:

https://gem5-review.googlesource.com/c/public/gem5/+/4880/7
https://gem5-review.googlesource.com/c/public/gem5/+/4881/7
https://gem5-review.googlesource.com/c/public/gem5/+/4421/12
https://gem5-review.googlesource.com/c/public/gem5/+/4882/13
https://gem5-review.googlesource.com/c/public/gem5/+/4883/13
https://gem5-review.googlesource.com/c/public/gem5/+/4422/13
https://gem5-review.googlesource.com/c/public/gem5/+/4423/13
https://gem5-review.googlesource.com/c/public/gem5/+/10121/4

Thanks,
Jason
_______________________________________________
gem5-dev mailing list
gem5-***@gem5.org<mailto:gem5-***@gem5.org>
http://m5sim.org/mailman/listinfo/gem5-dev

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.


IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Jason Lowe-Power
2018-08-03 16:33:46 UTC
Permalink
Sounds good. We can give it a week or so before we push. I won't have any
time to work on much between now and then anyway.

Cheers,
Jason
Post by Andreas Sandberg
OK, that sounds reasonable. We shouldn't have any issues with that.
I'd like someone from us to have a look as well, most likely Giacomo since
I won't be able to find the time myself. However, don't let that block
anything. Last time I looked at the code, it looked reasonable. I'm sure we
can fix any issues once we start integrating it into our test setup.
Thanks,
Andreas
My attempt to address your concerns, Andreas..
Your issue with the baked-in test server was definitely reasonable. I've
actually taken that part out of the library since it was so unreliable. A
reliable test server would probably be a whole library in itself.
My suggestion for running distributed tests would be to do basically the
same thing you do now: list all the suites and kick off a distributed task
for each suite uid. Each test is individually labeled with a UID. When
using the `./main.py run` command you can supply the test suite uid with
`--uid <SUITE-UID>`. That test and that test only will run.
Your current xml aggregation solution sounds reasonable to me. The only
issue you might run into is that a single result.xml is written to, so if
all of gem5 is served over NFS you might get conflicts writing to the file.
I can add an output path flag if you think this might be an issue for you.
Cheers,
Sean
Post by Jason Lowe-Power
Hi all,
Some more follow up. Sorry that this is going so slowly.
Sean just did a pretty major reworking of the main library. See
https://gem5-review.googlesource.com/4421. I'll let him address
Andreas's questions below. I think we're pretty close to having all three.
I would really like to get this pushed in ASAP so someone (me, a student,
or others in the community) can start migrating all of the current tests
over to the new framework.
As a quick preview, this is what the output looks like now.
cd tests
./main.py run
Running the new gem5 testing script.
For more information see TESTING.md.
To see details as the testing scripts are running, use the option -v, -vv, or -vvv
======================================================================================================================
Loading Tests
Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py
Discovered 30 tests and 15 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py
Discovered 6 tests and 3 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py
Discovered 21 tests and 21 suites in
/home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py
... <a few errors while we are transitioning>
======================================================================================================================
Running Tests from 9 suites
Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results
======================================================================================================================
Building the following targets. This may take a while.
/home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt,
/home/jlp/Code/gem5/gem5/build/X86/gem5.opt,
/home/jlp/Code/gem5/gem5/build/ARM/gem5.opt
You may want to run with only a single ISA(--isa=), use --skip-build, or use 'retest'.
Test: testhello64-static-X86-opt Passed
Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-dynamic-X86-opt Passed
Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-X86-opt Passed
Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed
Test: testhello64-static-ARM-opt Passed
Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed
Test: testhello32-static-ARM-opt Passed
Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed
Test: m5_exit_test-X86-opt Passed
Test: m5_exit_test-X86-opt-MatchRegex Passed
Test: build-X86-opt Passed
Test: build-RISCV-opt Passed
Test: build-ARM-opt Passed
====================================== Results: 15 Passed in 2.6e+01
seconds ========================================
Cheers,
Jason
On Fri, Jul 13, 2018 at 1:37 AM Andreas Sandberg <
Hi Jason,
Thanks for doing this. I haven't had a chance to look at the code in
details myself, but from what I have seen it looks like a real improvement.
Ideally, I'd like someone from our side (probably Giacomo T) to have a look
at it. However, Giacomo is currently travelling and I have other pressing
matters to tend to. :(
I have had a quick look at the code and it all looks reasonable. There
1. Ability to separate build and test.
2. Ability to schedule individual (long-running) tests separately in
a cluster environment.
3. Ability to get a JUnit XML file with per-test results.
With the potential exception of 2, I think the new framework supports
all of the above. We could, maybe, use the support for distributed test
using the test server in the framework, but I'm a bit reluctant to deploy
it for two reasons. 1) It requires credentials to be hard-coded. It would
be much more convenient to use an auto-generated authentication file (with
suitable permissions) in a shared file location. 2) Having a custom server
is potentially very unreliable since the network tends to be noisy (e.g.,
port scanners, IDS software). The static allocation of a port will also be
troublesome (e.g., when two servers run on the same node).
In the current framework, we typically list the available tests and then
create one cluster job per test. Once the tests complete, the CI framework
aggregates all of the JUnit files (it also stores the pickle files for
debugging) and generates a report. I'd like to be able to do something with
the new framework. Is that possible?
Cheers,
Andreas
Hi all,
I'm hoping that we can finalize the new testing framework, and I can push
it in within the next week or so. All of the important changes have at
least one +2, but I want to give everyone one last chance to object or
suggest changes. I've been talking to Sean, and while there's more that
could be cleaned up in ext/testlib, we think it's good as it is. At a
minimum, this will give us a starting point. I'd like to get this in soon
so I can start adding more tests to it in the coming weeks.
https://gem5-review.googlesource.com/c/public/gem5/+/4880/7https://gem5-review.googlesource.com/c/public/gem5/+/4881/7https://gem5-review.googlesource.com/c/public/gem5/+/4421/12https://gem5-review.googlesource.com/c/public/gem5/+/4882/13https://gem5-review.googlesource.com/c/public/gem5/+/4883/13https://gem5-review.googlesource.com/c/public/gem5/+/4422/13https://gem5-review.googlesource.com/c/public/gem5/+/4423/13https://gem5-review.googlesource.com/c/public/gem5/+/10121/4
Thanks,
Jason
_______________________________________________
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
Loading...