Discussion:
[gem5-dev] testing framework
(too old to reply)
Gabe Black
2017-11-16 05:04:53 UTC
Permalink
Hi folks. How is the test framework integration coming along? Is there
anything I can do to help?

Also, looking at the whimsy documentation and talking to some other folks
at Google, it looks like the framework is good at running tests, but
doesn't itself really implement, for instance, unit tests which have their
own C++ main function, etc.

Does anyone have an opinion about using the google gtest library for
writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would fit
into whimsy fairly well.

Gabe
Andreas Sandberg
2017-11-17 16:43:48 UTC
Permalink
I'm generally in favour of switching to an existing unit test framework.
I haven't surveyed the C++ test framework landscape, but judging by the
users (Chromium, LLVM, ...) GTest seems pretty solid.

As far as I can tell, GTest is all BSD. Is that correct?

Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is there
anything I can do to help?
Also, looking at the whimsy documentation and talking to some other folks
at Google, it looks like the framework is good at running tests, but
doesn't itself really implement, for instance, unit tests which have their
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library for
writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would fit
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Matthias Jung
2017-11-17 18:14:48 UTC
Permalink
Hey,

It would be also nice if we have something that we could also use for test the SystemC stuff. Right now this is not possible with the current framework. Because we have to build gem5 without python as a library for the SystemC stuff... and then build again with linking of the gem5 library.

Regards
Matthias
Post by Andreas Sandberg
I'm generally in favour of switching to an existing unit test framework.
I haven't surveyed the C++ test framework landscape, but judging by the
users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is there
anything I can do to help?
Also, looking at the whimsy documentation and talking to some other folks
at Google, it looks like the framework is good at running tests, but
doesn't itself really implement, for instance, unit tests which have their
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library for
writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would fit
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Gabe Black
2017-11-17 22:50:31 UTC
Permalink
As far as a high level framework, I think whimsy should help there since
it's a bit more decoupled and will let you run arbitrary things as tests,
at least as far as I can tell. Since the systemC integration is just that,
an integration of two systems, I don't think unit tests would be as
applicable generally speaking. You could have one for some things though,
like the ports that translate between gem5's and systemC's memory protocol.

GTest claims to be BSD, so the license should be compatible.

Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use for test
the SystemC stuff. Right now this is not possible with the current
framework. Because we have to build gem5 without python as a library for
the SystemC stuff... and then build again with linking of the gem5 library.
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test framework.
I haven't surveyed the C++ test framework landscape, but judging by the
users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is there
anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests, but
doesn't itself really implement, for instance, unit tests which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library for
writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Paul Rosenfeld (prosenfeld)
2017-11-17 23:19:00 UTC
Permalink
Even without the complexities of setting up SystemC+gem5 it's pretty painful to "unit" test SystemC because the SystemC runtime has a lot of global state that comes along with it (for example: you can't "uncall" sc_start() to go back and re-do elaboration with different modules). The only (not very elegant) workaround we have discussed for unit testing SystemC modules is to just to build each set of related SystemC unit tests into their own independent GTest binaries.

On the whole, though, GTest is pretty great.

-----Original Message-----
From: gem5-dev [mailto:gem5-dev-***@gem5.org] On Behalf Of Gabe Black
Sent: Friday, November 17, 2017 5:51 PM
To: Matthias Jung <***@eit.uni-kl.de>
Cc: gem5 Developer List <gem5-***@gem5.org>
Subject: [EXT] Re: [gem5-dev] testing framework

As far as a high level framework, I think whimsy should help there since it's a bit more decoupled and will let you run arbitrary things as tests, at least as far as I can tell. Since the systemC integration is just that, an integration of two systems, I don't think unit tests would be as applicable generally speaking. You could have one for some things though, like the ports that translate between gem5's and systemC's memory protocol.

GTest claims to be BSD, so the license should be compatible.

Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use for
test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking of the gem5 library.
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test framework.
I haven't surveyed the C++ test framework landscape, but judging by
the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests which
have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
gem5-***@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
Jason Lowe-Power
2017-11-20 17:21:35 UTC
Permalink
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
syntax. Although, that shouldn't stop us from having unit tests!!

What's the state of getting whimsy pushed in? From my side, it's ready to
go. The only thing on my to do list that I would have liked to finish up
before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure to
be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable
future.

Jason

On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot of
global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules). The
only (not very elegant) workaround we have discussed for unit testing
SystemC modules is to just to build each set of related SystemC unit tests
into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there since
it's a bit more decoupled and will let you run arbitrary things as tests,
at least as far as I can tell. Since the systemC integration is just that,
an integration of two systems, I don't think unit tests would be as
applicable generally speaking. You could have one for some things though,
like the ports that translate between gem5's and systemC's memory protocol.
GTest claims to be BSD, so the license should be compatible.
Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use for
test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging by
the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests which
have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Potter, Brandon
2017-11-20 23:14:43 UTC
Permalink
I also support adding unit tests (and possibly a code coverage checker to identify weak points). Ideally, we'd add unit tests for the entire simulator, but we should probably put something into place for new code requiring some type of unit tests. Otherwise, only a subset of the submissions will contain unit tests.

Regards,
Brandon

-----Original Message-----
From: gem5-dev [mailto:gem5-dev-***@gem5.org] On Behalf Of Jason Lowe-Power
Sent: Monday, November 20, 2017 11:22 AM
To: gem5 Developer List <gem5-***@gem5.org>
Subject: Re: [gem5-dev] [EXT] Re: testing framework

I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its syntax. Although, that shouldn't stop us from having unit tests!!

What's the state of getting whimsy pushed in? From my side, it's ready to go. The only thing on my to do list that I would have liked to finish up before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure to be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable future.

Jason
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary things
as tests, at least as far as I can tell. Since the systemC integration
is just that, an integration of two systems, I don't think unit tests
would be as applicable generally speaking. You could have one for some
things though, like the ports that translate between gem5's and systemC's memory protocol.
GTest claims to be BSD, so the license should be compatible.
Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store
or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
gem5-***@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
Gabe Black
2017-11-21 01:37:48 UTC
Permalink
What I would minimally like to see is for people to run the unit tests that
already exist. Clearly that hasn't been happening since one of them hasn't
compiled since 2014 (now fixed). We need our tests to test something
specific, say whether or not it worked (not leave interpretation up to the
reader), and for people to actually run them.

It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to try
cleaning these up a bit. Also, they shouldn't be something that lives off
on the side or in their own directory, out of sight and out of mind, they
should be alongside the code they test.

It might even make sense to associate a unit test with a particular source
file and make compilation fail if the unit test doesn't pass. That might be
a bit overboard, but would at least ensure that the tests get run.

Gabe
Post by Potter, Brandon
I also support adding unit tests (and possibly a code coverage checker to
identify weak points). Ideally, we'd add unit tests for the entire
simulator, but we should probably put something into place for new code
requiring some type of unit tests. Otherwise, only a subset of the
submissions will contain unit tests.
Regards,
Brandon
-----Original Message-----
Sent: Monday, November 20, 2017 11:22 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
syntax. Although, that shouldn't stop us from having unit tests!!
What's the state of getting whimsy pushed in? From my side, it's ready to
go. The only thing on my to do list that I would have liked to finish up
before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure
to be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable future.
Jason
On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary things
as tests, at least as far as I can tell. Since the systemC integration
is just that, an integration of two systems, I don't think unit tests
would be as applicable generally speaking. You could have one for some
things though, like the ports that translate between gem5's and
systemC's memory protocol.
Post by Paul Rosenfeld (prosenfeld)
GTest claims to be BSD, so the license should be compatible.
Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store
or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Andreas Sandberg
2017-11-22 09:35:20 UTC
Permalink
One issue that is preventing me from running the unit tests is that they
require separate scons targets. One solution that I was considering at
one point was to make the unit tests available as Python exports in the
main build. This would allow us to run a magic config that calls unit
tests instead of running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with
other test unit test frameworks.

Cheers,
Andreas
Post by Gabe Black
What I would minimally like to see is for people to run the unit tests that
already exist. Clearly that hasn't been happening since one of them hasn't
compiled since 2014 (now fixed). We need our tests to test something
specific, say whether or not it worked (not leave interpretation up to the
reader), and for people to actually run them.
It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to try
cleaning these up a bit. Also, they shouldn't be something that lives off
on the side or in their own directory, out of sight and out of mind, they
should be alongside the code they test.
It might even make sense to associate a unit test with a particular source
file and make compilation fail if the unit test doesn't pass. That might be
a bit overboard, but would at least ensure that the tests get run.
Gabe
Post by Potter, Brandon
I also support adding unit tests (and possibly a code coverage checker to
identify weak points). Ideally, we'd add unit tests for the entire
simulator, but we should probably put something into place for new code
requiring some type of unit tests. Otherwise, only a subset of the
submissions will contain unit tests.
Regards,
Brandon
-----Original Message-----
Sent: Monday, November 20, 2017 11:22 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
syntax. Although, that shouldn't stop us from having unit tests!!
What's the state of getting whimsy pushed in? From my side, it's ready to
go. The only thing on my to do list that I would have liked to finish up
before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure
to be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable future.
Jason
On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary things
as tests, at least as far as I can tell. Since the systemC integration
is just that, an integration of two systems, I don't think unit tests
would be as applicable generally speaking. You could have one for some
things though, like the ports that translate between gem5's and
systemC's memory protocol.
Post by Paul Rosenfeld (prosenfeld)
GTest claims to be BSD, so the license should be compatible.
Gabe
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store
or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Potter, Brandon
2017-11-22 17:20:39 UTC
Permalink
Is there any documentation on how to run the current unit tests? I notice that there's a unit test directory in the X86 build folder, but nothing inside is executable. I do not see a way of generating output from SCons related to unittests either.

From the src/SConscript file, I see that the unittests should get created inside the makeEnv function, but it doesn't appear that this does anything (at least for X86).

-Brandon

-----Original Message-----
From: gem5-dev [mailto:gem5-dev-***@gem5.org] On Behalf Of Andreas Sandberg
Sent: Wednesday, November 22, 2017 3:35 AM
To: gem5 Developer List <gem5-***@gem5.org>; Gabe Black <***@google.com>
Subject: Re: [gem5-dev] [EXT] Re: testing framework

One issue that is preventing me from running the unit tests is that they require separate scons targets. One solution that I was considering at one point was to make the unit tests available as Python exports in the main build. This would allow us to run a magic config that calls unit tests instead of running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with other test unit test frameworks.

Cheers,
Andreas
Post by Gabe Black
What I would minimally like to see is for people to run the unit tests
that already exist. Clearly that hasn't been happening since one of
them hasn't compiled since 2014 (now fixed). We need our tests to test
something specific, say whether or not it worked (not leave
interpretation up to the reader), and for people to actually run them.
It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to
try cleaning these up a bit. Also, they shouldn't be something that
lives off on the side or in their own directory, out of sight and out
of mind, they should be alongside the code they test.
It might even make sense to associate a unit test with a particular
source file and make compilation fail if the unit test doesn't pass.
That might be a bit overboard, but would at least ensure that the tests get run.
Gabe
On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon
Post by Potter, Brandon
I also support adding unit tests (and possibly a code coverage
checker to identify weak points). Ideally, we'd add unit tests for
the entire simulator, but we should probably put something into place
for new code requiring some type of unit tests. Otherwise, only a
subset of the submissions will contain unit tests.
Regards,
Brandon
-----Original Message-----
Sent: Monday, November 20, 2017 11:22 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of
its syntax. Although, that shouldn't stop us from having unit tests!!
What's the state of getting whimsy pushed in? From my side, it's
ready to go. The only thing on my to do list that I would have liked
to finish up before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test
infrastructure to be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the
foreseeable future.
Jason
On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary
things as tests, at least as far as I can tell. Since the systemC
integration is just that, an integration of two systems, I don't
think unit tests would be as applicable generally speaking. You
could have one for some things though, like the ports that translate
between gem5's and
systemC's memory protocol.
Post by Paul Rosenfeld (prosenfeld)
GTest claims to be BSD, so the license should be compatible.
Gabe
On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as
a library for the SystemC stuff... and then build again with
linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the
intended recipient, please notify the sender immediately and do not
disclose the contents to any other person, use it for any purpose,
or store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
gem5-***@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
Gabe Black
2017-11-22 20:10:57 UTC
Permalink
Declaring unit tests in the SConscript file just tells scons they exist,
but if you don't ask it to build them then it won't since they aren't a
dependency of, for example, gem5.opt. It's annoying to tell it to just
build all the unit tests since you could do that based on directory, and as
discussed in another thread that would build all 5 versions of each.
Otherwise you have to ask scons to build each unit test binary by name
which is very cumbersome. I'm actively thinking about and working on making
the unit tests work and easy (or even mandatory) to run, so hopefully this
will get a lot better soon.

Gabe
Post by Potter, Brandon
Is there any documentation on how to run the current unit tests? I notice
that there's a unit test directory in the X86 build folder, but nothing
inside is executable. I do not see a way of generating output from SCons
related to unittests either.
From the src/SConscript file, I see that the unittests should get created
inside the makeEnv function, but it doesn't appear that this does anything
(at least for X86).
-Brandon
-----Original Message-----
Sent: Wednesday, November 22, 2017 3:35 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
One issue that is preventing me from running the unit tests is that they
require separate scons targets. One solution that I was considering at one
point was to make the unit tests available as Python exports in the main
build. This would allow us to run a magic config that calls unit tests
instead of running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with
other test unit test frameworks.
Cheers,
Andreas
Post by Gabe Black
What I would minimally like to see is for people to run the unit tests
that already exist. Clearly that hasn't been happening since one of
them hasn't compiled since 2014 (now fixed). We need our tests to test
something specific, say whether or not it worked (not leave
interpretation up to the reader), and for people to actually run them.
It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to
try cleaning these up a bit. Also, they shouldn't be something that
lives off on the side or in their own directory, out of sight and out
of mind, they should be alongside the code they test.
It might even make sense to associate a unit test with a particular
source file and make compilation fail if the unit test doesn't pass.
That might be a bit overboard, but would at least ensure that the tests
get run.
Post by Gabe Black
Gabe
On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon
Post by Potter, Brandon
I also support adding unit tests (and possibly a code coverage
checker to identify weak points). Ideally, we'd add unit tests for
the entire simulator, but we should probably put something into place
for new code requiring some type of unit tests. Otherwise, only a
subset of the submissions will contain unit tests.
Regards,
Brandon
-----Original Message-----
Sent: Monday, November 20, 2017 11:22 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of
its syntax. Although, that shouldn't stop us from having unit tests!!
What's the state of getting whimsy pushed in? From my side, it's
ready to go. The only thing on my to do list that I would have liked
to finish up before pushing is better integration with the current test
infrastructure.
Post by Gabe Black
Post by Potter, Brandon
It would be good for whoever is running with the old test
infrastructure to be able to run both the old-style and new-style tests
with one command.
Post by Gabe Black
Post by Potter, Brandon
Unfortunately, I don't have the time to dig into this for the foreseeable future.
Jason
On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't
"uncall"
Post by Gabe Black
Post by Potter, Brandon
Post by Paul Rosenfeld (prosenfeld)
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary
things as tests, at least as far as I can tell. Since the systemC
integration is just that, an integration of two systems, I don't
think unit tests would be as applicable generally speaking. You
could have one for some things though, like the ports that translate
between gem5's and
systemC's memory protocol.
Post by Paul Rosenfeld (prosenfeld)
GTest claims to be BSD, so the license should be compatible.
Gabe
On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as
a library for the SystemC stuff... and then build again with
linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the
intended recipient, please notify the sender immediately and do not
disclose the contents to any other person, use it for any purpose,
or store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Gabe Black
2017-11-22 21:07:45 UTC
Permalink
Also, it would be nice if the unit tests were more unit-ey and didn't
require linking in all of gem5 or even building all of gem5 to run. I don't
know how feasible that is in reality though. I think that's normally done
by stubbing out the external stuff they'd call into with mocks. That would
make them a lot quicker to build, and more self contained so that you'd
know a failure of xyztest meant xyz was broken, not that some other thing
it depended on wasn't behaving properly. This is a bit of a learning
experience for me too, so I'm working my way through this as a I go to some
extent.

Gabe
Post by Gabe Black
Declaring unit tests in the SConscript file just tells scons they exist,
but if you don't ask it to build them then it won't since they aren't a
dependency of, for example, gem5.opt. It's annoying to tell it to just
build all the unit tests since you could do that based on directory, and as
discussed in another thread that would build all 5 versions of each.
Otherwise you have to ask scons to build each unit test binary by name
which is very cumbersome. I'm actively thinking about and working on making
the unit tests work and easy (or even mandatory) to run, so hopefully this
will get a lot better soon.
Gabe
Post by Potter, Brandon
Is there any documentation on how to run the current unit tests? I notice
that there's a unit test directory in the X86 build folder, but nothing
inside is executable. I do not see a way of generating output from SCons
related to unittests either.
From the src/SConscript file, I see that the unittests should get created
inside the makeEnv function, but it doesn't appear that this does anything
(at least for X86).
-Brandon
-----Original Message-----
Sent: Wednesday, November 22, 2017 3:35 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
One issue that is preventing me from running the unit tests is that they
require separate scons targets. One solution that I was considering at one
point was to make the unit tests available as Python exports in the main
build. This would allow us to run a magic config that calls unit tests
instead of running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with
other test unit test frameworks.
Cheers,
Andreas
Post by Gabe Black
What I would minimally like to see is for people to run the unit tests
that already exist. Clearly that hasn't been happening since one of
them hasn't compiled since 2014 (now fixed). We need our tests to test
something specific, say whether or not it worked (not leave
interpretation up to the reader), and for people to actually run them.
It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to
try cleaning these up a bit. Also, they shouldn't be something that
lives off on the side or in their own directory, out of sight and out
of mind, they should be alongside the code they test.
It might even make sense to associate a unit test with a particular
source file and make compilation fail if the unit test doesn't pass.
That might be a bit overboard, but would at least ensure that the tests
get run.
Post by Gabe Black
Gabe
On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon
Post by Potter, Brandon
I also support adding unit tests (and possibly a code coverage
checker to identify weak points). Ideally, we'd add unit tests for
the entire simulator, but we should probably put something into place
for new code requiring some type of unit tests. Otherwise, only a
subset of the submissions will contain unit tests.
Regards,
Brandon
-----Original Message-----
Sent: Monday, November 20, 2017 11:22 AM
Subject: Re: [gem5-dev] [EXT] Re: testing framework
I'm all for unit tests. GTest seems fine, but I'm not a huge fan of
its syntax. Although, that shouldn't stop us from having unit tests!!
What's the state of getting whimsy pushed in? From my side, it's
ready to go. The only thing on my to do list that I would have liked
to finish up before pushing is better integration with the current
test infrastructure.
Post by Gabe Black
Post by Potter, Brandon
It would be good for whoever is running with the old test
infrastructure to be able to run both the old-style and new-style
tests with one command.
Post by Gabe Black
Post by Potter, Brandon
Unfortunately, I don't have the time to dig into this for the foreseeable future.
Jason
On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
Post by Paul Rosenfeld (prosenfeld)
Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't
"uncall"
Post by Gabe Black
Post by Potter, Brandon
Post by Paul Rosenfeld (prosenfeld)
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.
On the whole, though, GTest is pretty great.
-----Original Message-----
Sent: Friday, November 17, 2017 5:51 PM
Subject: [EXT] Re: [gem5-dev] testing framework
As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary
things as tests, at least as far as I can tell. Since the systemC
integration is just that, an integration of two systems, I don't
think unit tests would be as applicable generally speaking. You
could have one for some things though, like the ports that translate
between gem5's and
systemC's memory protocol.
Post by Paul Rosenfeld (prosenfeld)
GTest claims to be BSD, so the license should be compatible.
Gabe
On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung
Post by Matthias Jung
Hey,
It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as
a library for the SystemC stuff... and then build again with
linking of
the gem5 library.
Post by Matthias Jung
Regards
Matthias
Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
I'm generally in favour of switching to an existing unit test
framework.
Post by Matthias Jung
I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.
As far as I can tell, GTest is all BSD. Is that correct?
Cheers,
Andreas
Post by Gabe Black
Hi folks. How is the test framework integration coming along? Is
there anything I can do to help?
Also, looking at the whimsy documentation and talking to some other
folks
Post by Gabe Black
at Google, it looks like the framework is good at running tests,
but doesn't itself really implement, for instance, unit tests
which have
their
Post by Gabe Black
own C++ main function, etc.
Does anyone have an opinion about using the google gtest library
for writing C++ unit tests? I've only worked with it a little bit
superficially, but so far it seems pretty nice and seems like it would
fit
Post by Gabe Black
into whimsy fairly well.
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the
intended recipient, please notify the sender immediately and do not
disclose the contents to any other person, use it for any purpose,
or store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Continue reading on narkive:
Loading...