Discussion:
[gem5-dev] hooking up gem5's memory system and systemc models
(too old to reply)
Gabe Black
2018-11-01 23:32:23 UTC
Permalink
Hi folks, especially the folks who wrote the TLM/gem5 bridge in util/tlm.
As I'm sure most of you are aware, I added an implementation of the systemc
API/kernel in gem5. While it's now possible to instantiate systemc objects
in a gem5 config and hand around pointers to them using config parameters,
there isn't a way to hook those objects to gem5's memory system like native
objects which have gem5 ports.

TLM is a pretty big ball of stuff, and given the existing implementation
and my interest in not breaking that or creating something slightly
incompatible and/or confusing, I want to dump my thoughts so far and get
some feedback and even ideally assistance from some folks who know these
areas better than I do.



Connecting ports in gem5's configs:

At the topmost level, the port binding mechanism calls a python function
ccConnect on PortRef objects, the things which represent gem5 ports in
gem5's python configuration files. That function does some sanity checks,
and ultimately calls connectPorts which is a function exposed from C++.

The C++ implementation of connectPorts is in
src/python/pybind11/pyobject.cc, where it takes two sets of parameters, a
SimObject, name, and index for each side of the connection being made. This
file used to have a bunch of conditional compilation in it, but you may
remember some changes I made recently which got rid of that.

Looking at the body of this function, the first thing it does is to use
dynamic_cast to try to cast these SimObject pointers in to ethernet related
objects. If that's successful, then the assumption is that these objects
are really connecting together an Ethernet network, rather than the memory
interconnect network within a system. If so, then the lookupEthPort
function is called which extracts the appropriate interfaces, and those are
stuck together.

If that doesn't work, then the function moves on to check to see if the
SimObjects are MessageBuffers from Ruby. If so, then nothing happens since
apparently that's handled somewhere else(?).

Then finally if even that doesn't work, gem5 checks if the SimObjects are
MemObjects. If so, then they have their ports extracted with the
getMasterPort and getSlavePort functions which return port objects, and
then those are glued together.

Possible changes:

It seems slightly dubious to me that these are all hanging off the same
port interface, but lets set that aside for now. What we have are really
three different mechanisms which are all serially in the same function, and
that function checks them one at a time to figure out which one applies.

This has at least two deficiencies. First, this function has a lot of
unrelated stuff in it depending on what flavor of binding it's doing. On
average, two thirds of the function is dead weight for any particular
invocation.

Second, if we wanted to add a new type of binding, for instance between a
systemc object and a gem5 object or vice versa, then we'd have to modify
this common and fairly core file to build that in from the ground up.

I did a little research, and my understanding is that pybind11 will let you
polymorphically bind C++ functions to a single python function name. At
call time, it will check each one one at a time until it finds one that
works, and then will call that. By making connectPorts three different
functions which take pointers of the desired types and then binding all of
them to the connectPorts name in python, we make pybind11 the arbiter and
central collection point, rather than the hard coded body of this C++
function.



Connecting systemc and gem5 objects:

I'm thinking there are two different ways to do this. The first would be to
make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.

Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?

The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.

I understand this less, although I'll keep looking at it to try to grok all
the moving pieces. I haven't really learned much about TLM so that may be a
limiting factor in learning how this works. This seems like a more robust
solution and would leverage the work that's already been done, but I'm not
sure how much work it would be or if there would be significant gotchas
involved somewhere.



Thoughts/feedback would be greatly appreciated! Thanks!

Gabe
Gabe Black
2018-11-06 03:49:12 UTC
Permalink
Hey folks, any thoughts?

I've done a bit more digging on my own and found a few things:

Port relationships are pretty fundementally asymmetric, as their "role"s
are piped around with them. This is used when connecting piers so that they
only connect once (from the master to the slave), and when drawing the
configuration dot file which uses directed edges for port connections.

Ruby "ports" don't actually do anything, other than make sure ruby
connections are represented in the dot configuration drawing.

Ethernet connections are symmetric(-ish). They piggy back on Port
connections, likely because there's a lot of machinery already there that
would be expensive to duplicate. The symmetric nature of the Port machinery
does serve a purpose in this case, where it makes sure the connections
still only happen once. A minor downside of this is that the dot graph will
show asymmetric directed edges for these connections which is artificial
and potentially misleading. The current Ethernet related SimObjects will
generally have an EtherLink object in between them, so the port
relationships can point out from the normal/device objects and towards the
links and have it all work out, even though it artificially restricts the
types of connections that can be made.

Another difference between Ethernet connections and regular ports is that
the objects that get glued together are EtherInts and not Master/Slave ports

***
The CXX configuration stuff which stores config information in an
executable format does not seem to have any of the special handling for
Ethernet devices or ruby memory. That means that those types of connections
would likely fail when using that configuration mechanism.
***

I don't have a complete plan in my mind yet for how to rearrange this into
a better situation, but I'm thinking it will likely have two parts, one
that more clearly distinguishes these different types of "ports" on the
back end, perhaps with a symmetric property for dot to consume, and a way
to override how connections are made in c++, and some sort of front end
distinction so that Ethernet devices don't have MasterPorts declared like
they were going to hook up to memory or something.

The intermediate layer, the Ports and PortRefs, are complicated and big
enough that they should be reused. I don't want to try to mess with them
since I don't understand them entirely, and there's a good chance that
would blow up to combinatorial complexity if it was too specialized for all
the various situations it could be used in. Conceptually, it makes sense to
think of Ports as an arbitrary connection between two things in some
communication domain, historically just a memory/interconnect domain, but
also Ethernet which is somewhat tacked on.

Gabe
Post by Gabe Black
Hi folks, especially the folks who wrote the TLM/gem5 bridge in util/tlm.
As I'm sure most of you are aware, I added an implementation of the systemc
API/kernel in gem5. While it's now possible to instantiate systemc objects
in a gem5 config and hand around pointers to them using config parameters,
there isn't a way to hook those objects to gem5's memory system like native
objects which have gem5 ports.
TLM is a pretty big ball of stuff, and given the existing implementation
and my interest in not breaking that or creating something slightly
incompatible and/or confusing, I want to dump my thoughts so far and get
some feedback and even ideally assistance from some folks who know these
areas better than I do.
At the topmost level, the port binding mechanism calls a python function
ccConnect on PortRef objects, the things which represent gem5 ports in
gem5's python configuration files. That function does some sanity checks,
and ultimately calls connectPorts which is a function exposed from C++.
The C++ implementation of connectPorts is in
src/python/pybind11/pyobject.cc, where it takes two sets of parameters, a
SimObject, name, and index for each side of the connection being made. This
file used to have a bunch of conditional compilation in it, but you may
remember some changes I made recently which got rid of that.
Looking at the body of this function, the first thing it does is to use
dynamic_cast to try to cast these SimObject pointers in to ethernet related
objects. If that's successful, then the assumption is that these objects
are really connecting together an Ethernet network, rather than the memory
interconnect network within a system. If so, then the lookupEthPort
function is called which extracts the appropriate interfaces, and those are
stuck together.
If that doesn't work, then the function moves on to check to see if the
SimObjects are MessageBuffers from Ruby. If so, then nothing happens since
apparently that's handled somewhere else(?).
Then finally if even that doesn't work, gem5 checks if the SimObjects are
MemObjects. If so, then they have their ports extracted with the
getMasterPort and getSlavePort functions which return port objects, and
then those are glued together.
It seems slightly dubious to me that these are all hanging off the same
port interface, but lets set that aside for now. What we have are really
three different mechanisms which are all serially in the same function, and
that function checks them one at a time to figure out which one applies.
This has at least two deficiencies. First, this function has a lot of
unrelated stuff in it depending on what flavor of binding it's doing. On
average, two thirds of the function is dead weight for any particular
invocation.
Second, if we wanted to add a new type of binding, for instance between a
systemc object and a gem5 object or vice versa, then we'd have to modify
this common and fairly core file to build that in from the ground up.
I did a little research, and my understanding is that pybind11 will let
you polymorphically bind C++ functions to a single python function name. At
call time, it will check each one one at a time until it finds one that
works, and then will call that. By making connectPorts three different
functions which take pointers of the desired types and then binding all of
them to the connectPorts name in python, we make pybind11 the arbiter and
central collection point, rather than the hard coded body of this C++
function.
I'm thinking there are two different ways to do this. The first would be
to make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that may
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done, but
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
Thoughts/feedback would be greatly appreciated! Thanks!
Gabe
Gabe Black
2018-11-06 04:20:14 UTC
Permalink
How about pulling the port extraction out of c++ and into python? Then the
various python flavors of port could know what function to call on their
parent, and what type their parent is supposed to be. That would complicate
things for the C++ only config though...

Gabe
Post by Gabe Black
Hey folks, any thoughts?
Port relationships are pretty fundementally asymmetric, as their "role"s
are piped around with them. This is used when connecting piers so that they
only connect once (from the master to the slave), and when drawing the
configuration dot file which uses directed edges for port connections.
Ruby "ports" don't actually do anything, other than make sure ruby
connections are represented in the dot configuration drawing.
Ethernet connections are symmetric(-ish). They piggy back on Port
connections, likely because there's a lot of machinery already there that
would be expensive to duplicate. The symmetric nature of the Port machinery
does serve a purpose in this case, where it makes sure the connections
still only happen once. A minor downside of this is that the dot graph will
show asymmetric directed edges for these connections which is artificial
and potentially misleading. The current Ethernet related SimObjects will
generally have an EtherLink object in between them, so the port
relationships can point out from the normal/device objects and towards the
links and have it all work out, even though it artificially restricts the
types of connections that can be made.
Another difference between Ethernet connections and regular ports is that
the objects that get glued together are EtherInts and not Master/Slave ports
***
The CXX configuration stuff which stores config information in an
executable format does not seem to have any of the special handling for
Ethernet devices or ruby memory. That means that those types of connections
would likely fail when using that configuration mechanism.
***
I don't have a complete plan in my mind yet for how to rearrange this into
a better situation, but I'm thinking it will likely have two parts, one
that more clearly distinguishes these different types of "ports" on the
back end, perhaps with a symmetric property for dot to consume, and a way
to override how connections are made in c++, and some sort of front end
distinction so that Ethernet devices don't have MasterPorts declared like
they were going to hook up to memory or something.
The intermediate layer, the Ports and PortRefs, are complicated and big
enough that they should be reused. I don't want to try to mess with them
since I don't understand them entirely, and there's a good chance that
would blow up to combinatorial complexity if it was too specialized for all
the various situations it could be used in. Conceptually, it makes sense to
think of Ports as an arbitrary connection between two things in some
communication domain, historically just a memory/interconnect domain, but
also Ethernet which is somewhat tacked on.
Gabe
Post by Gabe Black
Hi folks, especially the folks who wrote the TLM/gem5 bridge in util/tlm.
As I'm sure most of you are aware, I added an implementation of the systemc
API/kernel in gem5. While it's now possible to instantiate systemc objects
in a gem5 config and hand around pointers to them using config parameters,
there isn't a way to hook those objects to gem5's memory system like native
objects which have gem5 ports.
TLM is a pretty big ball of stuff, and given the existing implementation
and my interest in not breaking that or creating something slightly
incompatible and/or confusing, I want to dump my thoughts so far and get
some feedback and even ideally assistance from some folks who know these
areas better than I do.
At the topmost level, the port binding mechanism calls a python function
ccConnect on PortRef objects, the things which represent gem5 ports in
gem5's python configuration files. That function does some sanity checks,
and ultimately calls connectPorts which is a function exposed from C++.
The C++ implementation of connectPorts is in
src/python/pybind11/pyobject.cc, where it takes two sets of parameters, a
SimObject, name, and index for each side of the connection being made. This
file used to have a bunch of conditional compilation in it, but you may
remember some changes I made recently which got rid of that.
Looking at the body of this function, the first thing it does is to use
dynamic_cast to try to cast these SimObject pointers in to ethernet related
objects. If that's successful, then the assumption is that these objects
are really connecting together an Ethernet network, rather than the memory
interconnect network within a system. If so, then the lookupEthPort
function is called which extracts the appropriate interfaces, and those are
stuck together.
If that doesn't work, then the function moves on to check to see if the
SimObjects are MessageBuffers from Ruby. If so, then nothing happens since
apparently that's handled somewhere else(?).
Then finally if even that doesn't work, gem5 checks if the SimObjects are
MemObjects. If so, then they have their ports extracted with the
getMasterPort and getSlavePort functions which return port objects, and
then those are glued together.
It seems slightly dubious to me that these are all hanging off the same
port interface, but lets set that aside for now. What we have are really
three different mechanisms which are all serially in the same function, and
that function checks them one at a time to figure out which one applies.
This has at least two deficiencies. First, this function has a lot of
unrelated stuff in it depending on what flavor of binding it's doing. On
average, two thirds of the function is dead weight for any particular
invocation.
Second, if we wanted to add a new type of binding, for instance between a
systemc object and a gem5 object or vice versa, then we'd have to modify
this common and fairly core file to build that in from the ground up.
I did a little research, and my understanding is that pybind11 will let
you polymorphically bind C++ functions to a single python function name. At
call time, it will check each one one at a time until it finds one that
works, and then will call that. By making connectPorts three different
functions which take pointers of the desired types and then binding all of
them to the connectPorts name in python, we make pybind11 the arbiter and
central collection point, rather than the hard coded body of this C++
function.
I'm thinking there are two different ways to do this. The first would be
to make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that may
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done, but
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
Thoughts/feedback would be greatly appreciated! Thanks!
Gabe
Christian Menard
2018-11-06 10:50:00 UTC
Permalink
Hi Gabe,

I don't know much about the port extraction in gem5, so I don't really
have a comment here, but I can say a few things about connecting systemc
and gem5 objects.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm thinking there are two different ways to do this. The first would be
to make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
Yes, I believe so. Passing through packages is possible with the bridges
in util/tlm. The basic idea is that for each incoming TLM packet a new
gem5 packet is created. Then the original TLM packet is placed as an
extension on top of the gem5 packet. Then the gem5 packet iterates
through the gem5 memory system and eventually returns as a response. The
bridge then retrieves the original TLM package and transforms it to an
response based on the gem5 response. Finally, it simply sends back the
response the TLM way. The same process applies for sending packages from
gem5 to TLM.

To implement you first solution, one would need to incorporate the
existing bridging code in a custom implementation of TLM taregt and
initiator sockets (e.g gem5_initiator_socket). These sockets would also
implement the MemObject interface and provide a gem5 port that can bind
to any other gem5 model. So in the end, a SystemC module would need to
explicitly use a gem5 tailored socket, but could then operate on this
socket transparently. However, I am not sure if this complies to the TLM
standard as in the end we need to have sockets that are not bound to
other TLM sockets which I believe is forbidden.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.
To me this sounds like the simpler and more reasonable
approach. Translation between gem5 and TLM would be explicit and gem5
and SystemC models could be used unmodified. Combining both sides
should not be hard, as the TLM side is only a stub that is responsible
for binding to the right gem5 port where all the transaction work
happens. Since with your solution, both parts can live within the same
entity, the TLM side does not even have to find the right gem5 port
anymore.


Are you aware of the paper I wrote with Mathias? It explains most of the
details and also contains an introduction to TLM that points out
differences to the gem5 memory system.
https://cfaed.tu-dresden.de/files/Images/people/chair-cc/publications/1707_Menard_SAMOS.pdf


Apart from connecting gem5 objects, do you plan to support connecting
TLM models to each other from a python configuration?


Regards,

Christian


As I see it, the best way for connecting TLM and gem5, depends on
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm thinking there are two different ways to do this. The first would be
to make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that may
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done, but
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
How about pulling the port extraction out of c++ and into python? Then the
various python flavors of port could know what function to call on their
parent, and what type their parent is supposed to be. That would complicate
things for the C++ only config though...
Gabe
Post by Gabe Black
Hey folks, any thoughts?
Port relationships are pretty fundementally asymmetric, as their "role"s
are piped around with them. This is used when connecting piers so that they
only connect once (from the master to the slave), and when drawing the
configuration dot file which uses directed edges for port connections.
Ruby "ports" don't actually do anything, other than make sure ruby
connections are represented in the dot configuration drawing.
Ethernet connections are symmetric(-ish). They piggy back on Port
connections, likely because there's a lot of machinery already there that
would be expensive to duplicate. The symmetric nature of the Port machinery
does serve a purpose in this case, where it makes sure the connections
still only happen once. A minor downside of this is that the dot graph will
show asymmetric directed edges for these connections which is artificial
and potentially misleading. The current Ethernet related SimObjects will
generally have an EtherLink object in between them, so the port
relationships can point out from the normal/device objects and towards the
links and have it all work out, even though it artificially restricts the
types of connections that can be made.
Another difference between Ethernet connections and regular ports is that
the objects that get glued together are EtherInts and not Master/Slave ports
***
The CXX configuration stuff which stores config information in an
executable format does not seem to have any of the special handling for
Ethernet devices or ruby memory. That means that those types of connections
would likely fail when using that configuration mechanism.
***
I don't have a complete plan in my mind yet for how to rearrange this into
a better situation, but I'm thinking it will likely have two parts, one
that more clearly distinguishes these different types of "ports" on the
back end, perhaps with a symmetric property for dot to consume, and a way
to override how connections are made in c++, and some sort of front end
distinction so that Ethernet devices don't have MasterPorts declared like
they were going to hook up to memory or something.
The intermediate layer, the Ports and PortRefs, are complicated and big
enough that they should be reused. I don't want to try to mess with them
since I don't understand them entirely, and there's a good chance that
would blow up to combinatorial complexity if it was too specialized for all
the various situations it could be used in. Conceptually, it makes sense to
think of Ports as an arbitrary connection between two things in some
communication domain, historically just a memory/interconnect domain, but
also Ethernet which is somewhat tacked on.
Gabe
Post by Gabe Black
Hi folks, especially the folks who wrote the TLM/gem5 bridge in util/tlm.
As I'm sure most of you are aware, I added an implementation of the systemc
API/kernel in gem5. While it's now possible to instantiate systemc objects
in a gem5 config and hand around pointers to them using config parameters,
there isn't a way to hook those objects to gem5's memory system like native
objects which have gem5 ports.
TLM is a pretty big ball of stuff, and given the existing implementation
and my interest in not breaking that or creating something slightly
incompatible and/or confusing, I want to dump my thoughts so far and get
some feedback and even ideally assistance from some folks who know these
areas better than I do.
At the topmost level, the port binding mechanism calls a python function
ccConnect on PortRef objects, the things which represent gem5 ports in
gem5's python configuration files. That function does some sanity checks,
and ultimately calls connectPorts which is a function exposed from C++.
The C++ implementation of connectPorts is in
src/python/pybind11/pyobject.cc, where it takes two sets of parameters, a
SimObject, name, and index for each side of the connection being made. This
file used to have a bunch of conditional compilation in it, but you may
remember some changes I made recently which got rid of that.
Looking at the body of this function, the first thing it does is to use
dynamic_cast to try to cast these SimObject pointers in to ethernet related
objects. If that's successful, then the assumption is that these objects
are really connecting together an Ethernet network, rather than the memory
interconnect network within a system. If so, then the lookupEthPort
function is called which extracts the appropriate interfaces, and those are
stuck together.
If that doesn't work, then the function moves on to check to see if the
SimObjects are MessageBuffers from Ruby. If so, then nothing happens since
apparently that's handled somewhere else(?).
Then finally if even that doesn't work, gem5 checks if the SimObjects are
MemObjects. If so, then they have their ports extracted with the
getMasterPort and getSlavePort functions which return port objects, and
then those are glued together.
It seems slightly dubious to me that these are all hanging off the same
port interface, but lets set that aside for now. What we have are really
three different mechanisms which are all serially in the same function, and
that function checks them one at a time to figure out which one applies.
This has at least two deficiencies. First, this function has a lot of
unrelated stuff in it depending on what flavor of binding it's doing. On
average, two thirds of the function is dead weight for any particular
invocation.
Second, if we wanted to add a new type of binding, for instance between a
systemc object and a gem5 object or vice versa, then we'd have to modify
this common and fairly core file to build that in from the ground up.
I did a little research, and my understanding is that pybind11 will let
you polymorphically bind C++ functions to a single python function name. At
call time, it will check each one one at a time until it finds one that
works, and then will call that. By making connectPorts three different
functions which take pointers of the desired types and then binding all of
them to the connectPorts name in python, we make pybind11 the arbiter and
central collection point, rather than the hard coded body of this C++
function.
I'm thinking there are two different ways to do this. The first would be
to make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides, a
gem5 side and a TLM side. This is a little different from how it looks like
it works in the examples since it's (by necessity) split into two halves
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that may
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done, but
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
Thoughts/feedback would be greatly appreciated! Thanks!
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Gabe Black
2018-11-07 00:42:03 UTC
Permalink
Thanks guys. I'll (re)read your paper (I read it once, but a while ago) and
look things over again and see if I can get a good idea of what's going on.
I think one thing I should do is look for some sort of online tutorial on
TLM and get the basics that way, rather than trying to digest the beefy
section of the spec dedicated to it.

Gabe

On Tue, Nov 6, 2018 at 2:50 AM Christian Menard <
Post by Matthias Jung
Hi Gabe,
I don't know much about the port extraction in gem5, so I don't really
have a comment here, but I can say a few things about connecting systemc
and gem5 objects.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm thinking there are two different ways to do this. The first would
be
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
to make a systemc interface type which is the same as or inherits from
the
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 port interface. The systemc module/model would need to speak the
gem5
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol
natively.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
Looking at the code in util/tlm, I don't 100% understand what's going
on
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
there (although the examples are great, thank you for those), but it
looks
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
like you've already done something at least sort of like this. Is any
of
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
that applicable here?
Yes, I believe so. Passing through packages is possible with the bridges
in util/tlm. The basic idea is that for each incoming TLM packet a new
gem5 packet is created. Then the original TLM packet is placed as an
extension on top of the gem5 packet. Then the gem5 packet iterates
through the gem5 memory system and eventually returns as a response. The
bridge then retrieves the original TLM package and transforms it to an
response based on the gem5 response. Finally, it simply sends back the
response the TLM way. The same process applies for sending packages from
gem5 to TLM.
To implement you first solution, one would need to incorporate the
existing bridging code in a custom implementation of TLM taregt and
initiator sockets (e.g gem5_initiator_socket). These sockets would also
implement the MemObject interface and provide a gem5 port that can bind
to any other gem5 model. So in the end, a SystemC module would need to
explicitly use a gem5 tailored socket, but could then operate on this
socket transparently. However, I am not sure if this complies to the TLM
standard as in the end we need to have sockets that are not bound to
other TLM sockets which I believe is forbidden.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides,
a
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 side and a TLM side. This is a little different from how it looks
like
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
it works in the examples since it's (by necessity) split into two
halves
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
which live on either side of the gem5/systemc dividing line.
To me this sounds like the simpler and more reasonable
approach. Translation between gem5 and TLM would be explicit and gem5
and SystemC models could be used unmodified. Combining both sides
should not be hard, as the TLM side is only a stub that is responsible
for binding to the right gem5 port where all the transaction work
happens. Since with your solution, both parts can live within the same
entity, the TLM side does not even have to find the right gem5 port
anymore.
Are you aware of the paper I wrote with Mathias? It explains most of the
details and also contains an introduction to TLM that points out
differences to the gem5 memory system.
https://cfaed.tu-dresden.de/files/Images/people/chair-cc/publications/1707_Menard_SAMOS.pdf
Apart from connecting gem5 objects, do you plan to support connecting
TLM models to each other from a python configuration?
Regards,
Christian
As I see it, the best way for connecting TLM and gem5, depends on
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm thinking there are two different ways to do this. The first would
be
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
to make a systemc interface type which is the same as or inherits from
the
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 port interface. The systemc module/model would need to speak the
gem5
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol
natively.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
Looking at the code in util/tlm, I don't 100% understand what's going
on
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
there (although the examples are great, thank you for those), but it
looks
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
like you've already done something at least sort of like this. Is any
of
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides,
a
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 side and a TLM side. This is a little different from how it looks
like
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
it works in the examples since it's (by necessity) split into two
halves
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that
may
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done,
but
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
How about pulling the port extraction out of c++ and into python? Then
the
Post by Gabe Black
various python flavors of port could know what function to call on their
parent, and what type their parent is supposed to be. That would
complicate
Post by Gabe Black
things for the C++ only config though...
Gabe
Post by Gabe Black
Hey folks, any thoughts?
Port relationships are pretty fundementally asymmetric, as their "role"s
are piped around with them. This is used when connecting piers so that
they
Post by Gabe Black
Post by Gabe Black
only connect once (from the master to the slave), and when drawing the
configuration dot file which uses directed edges for port connections.
Ruby "ports" don't actually do anything, other than make sure ruby
connections are represented in the dot configuration drawing.
Ethernet connections are symmetric(-ish). They piggy back on Port
connections, likely because there's a lot of machinery already there
that
Post by Gabe Black
Post by Gabe Black
would be expensive to duplicate. The symmetric nature of the Port
machinery
Post by Gabe Black
Post by Gabe Black
does serve a purpose in this case, where it makes sure the connections
still only happen once. A minor downside of this is that the dot graph
will
Post by Gabe Black
Post by Gabe Black
show asymmetric directed edges for these connections which is artificial
and potentially misleading. The current Ethernet related SimObjects will
generally have an EtherLink object in between them, so the port
relationships can point out from the normal/device objects and towards
the
Post by Gabe Black
Post by Gabe Black
links and have it all work out, even though it artificially restricts
the
Post by Gabe Black
Post by Gabe Black
types of connections that can be made.
Another difference between Ethernet connections and regular ports is
that
Post by Gabe Black
Post by Gabe Black
the objects that get glued together are EtherInts and not Master/Slave
ports
Post by Gabe Black
Post by Gabe Black
***
The CXX configuration stuff which stores config information in an
executable format does not seem to have any of the special handling for
Ethernet devices or ruby memory. That means that those types of
connections
Post by Gabe Black
Post by Gabe Black
would likely fail when using that configuration mechanism.
***
I don't have a complete plan in my mind yet for how to rearrange this
into
Post by Gabe Black
Post by Gabe Black
a better situation, but I'm thinking it will likely have two parts, one
that more clearly distinguishes these different types of "ports" on the
back end, perhaps with a symmetric property for dot to consume, and a
way
Post by Gabe Black
Post by Gabe Black
to override how connections are made in c++, and some sort of front end
distinction so that Ethernet devices don't have MasterPorts declared
like
Post by Gabe Black
Post by Gabe Black
they were going to hook up to memory or something.
The intermediate layer, the Ports and PortRefs, are complicated and big
enough that they should be reused. I don't want to try to mess with them
since I don't understand them entirely, and there's a good chance that
would blow up to combinatorial complexity if it was too specialized for
all
Post by Gabe Black
Post by Gabe Black
the various situations it could be used in. Conceptually, it makes
sense to
Post by Gabe Black
Post by Gabe Black
think of Ports as an arbitrary connection between two things in some
communication domain, historically just a memory/interconnect domain,
but
Post by Gabe Black
Post by Gabe Black
also Ethernet which is somewhat tacked on.
Gabe
Post by Gabe Black
Hi folks, especially the folks who wrote the TLM/gem5 bridge in
util/tlm.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
As I'm sure most of you are aware, I added an implementation of the
systemc
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
API/kernel in gem5. While it's now possible to instantiate systemc
objects
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
in a gem5 config and hand around pointers to them using config
parameters,
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
there isn't a way to hook those objects to gem5's memory system like
native
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
objects which have gem5 ports.
TLM is a pretty big ball of stuff, and given the existing
implementation
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
and my interest in not breaking that or creating something slightly
incompatible and/or confusing, I want to dump my thoughts so far and
get
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
some feedback and even ideally assistance from some folks who know
these
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
areas better than I do.
At the topmost level, the port binding mechanism calls a python
function
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
ccConnect on PortRef objects, the things which represent gem5 ports in
gem5's python configuration files. That function does some sanity
checks,
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
and ultimately calls connectPorts which is a function exposed from C++.
The C++ implementation of connectPorts is in
src/python/pybind11/pyobject.cc, where it takes two sets of
parameters, a
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
SimObject, name, and index for each side of the connection being made.
This
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
file used to have a bunch of conditional compilation in it, but you may
remember some changes I made recently which got rid of that.
Looking at the body of this function, the first thing it does is to use
dynamic_cast to try to cast these SimObject pointers in to ethernet
related
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
objects. If that's successful, then the assumption is that these
objects
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
are really connecting together an Ethernet network, rather than the
memory
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
interconnect network within a system. If so, then the lookupEthPort
function is called which extracts the appropriate interfaces, and
those are
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
stuck together.
If that doesn't work, then the function moves on to check to see if the
SimObjects are MessageBuffers from Ruby. If so, then nothing happens
since
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
apparently that's handled somewhere else(?).
Then finally if even that doesn't work, gem5 checks if the SimObjects
are
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
MemObjects. If so, then they have their ports extracted with the
getMasterPort and getSlavePort functions which return port objects, and
then those are glued together.
It seems slightly dubious to me that these are all hanging off the same
port interface, but lets set that aside for now. What we have are
really
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
three different mechanisms which are all serially in the same
function, and
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
that function checks them one at a time to figure out which one
applies.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
This has at least two deficiencies. First, this function has a lot of
unrelated stuff in it depending on what flavor of binding it's doing.
On
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
average, two thirds of the function is dead weight for any particular
invocation.
Second, if we wanted to add a new type of binding, for instance
between a
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
systemc object and a gem5 object or vice versa, then we'd have to
modify
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
this common and fairly core file to build that in from the ground up.
I did a little research, and my understanding is that pybind11 will let
you polymorphically bind C++ functions to a single python function
name. At
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
call time, it will check each one one at a time until it finds one that
works, and then will call that. By making connectPorts three different
functions which take pointers of the desired types and then binding
all of
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
them to the connectPorts name in python, we make pybind11 the arbiter
and
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
central collection point, rather than the hard coded body of this C++
function.
I'm thinking there are two different ways to do this. The first would
be
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
to make a systemc interface type which is the same as or inherits from
the
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 port interface. The systemc module/model would need to speak the
gem5
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol
natively.
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
Looking at the code in util/tlm, I don't 100% understand what's going
on
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
there (although the examples are great, thank you for those), but it
looks
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
like you've already done something at least sort of like this. Is any
of
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
that applicable here?
The second way would be to slightly repackage the bridge in util/tlm so
that it was more of a single entity in the gem5 config with two sides,
a
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
gem5 side and a TLM side. This is a little different from how it looks
like
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
it works in the examples since it's (by necessity) split into two
halves
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
which live on either side of the gem5/systemc dividing line.
I understand this less, although I'll keep looking at it to try to grok
all the moving pieces. I haven't really learned much about TLM so that
may
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
be a limiting factor in learning how this works. This seems like a more
robust solution and would leverage the work that's already been done,
but
Post by Gabe Black
Post by Gabe Black
Post by Gabe Black
I'm not sure how much work it would be or if there would be significant
gotchas involved somewhere.
Thoughts/feedback would be greatly appreciated! Thanks!
Gabe
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Matthias Jung
2018-11-06 09:43:53 UTC
Permalink
Hi Gabe,
Post by Gabe Black
I'm thinking there are two different ways to do this. The first would be to
make a systemc interface type which is the same as or inherits from the
gem5 port interface. The systemc module/model would need to speak the gem5
protocol directly, and this wouldn't be very easy to use when passing
through packets (or whatever TLM calls them) which were generated by a
module which didn't buy into using gem5's memory system protocol natively.
I think we should keep the TLM2.0 interoperability here such that we can
easily add any TLM2 model.
Post by Gabe Black
Looking at the code in util/tlm, I don't 100% understand what's going on
there (although the examples are great, thank you for those), but it looks
like you've already done something at least sort of like this. Is any of
that applicable here?
Yes we build a so called transactor which is able to translate between TLM’s
generic payload and gem5’s packets. Also the transactor handles the two
different ways of implementing backpressure of gem5 and TLM. This is the
main difference of both protocols.
Post by Gabe Black
I understand this less, although I'll keep looking at it to try to grok all
the moving pieces. I haven't really learned much about TLM so that may be a
limiting factor in learning how this works. This seems like a more robust
solution and would leverage the work that's already been done, but I'm not
sure how much work it would be or if there would be significant gotchas
involved somewhere.
Actually TLM is not so much different to the gem5 implementation. If you want
we could have a short Skype telco, where I could highlight the differneces
and explain to you how TLM works in general.
Post by Gabe Black
How about pulling the port extraction out of c++ and into python? Then the
various python flavors of port could know what function to call on their
parent, and what type their parent is supposed to be. That would complicate
things for the C++ only config though…
Yes, for example if you use gem5 just as a core model in a SystemC virtual
platform. There should be still the possibility to do that.

Best,
Matthias
Loading...