Discussion:
[gem5-dev] Debugging New Translating Ports / PortProxy
(too old to reply)
Joel Hestness
2012-03-17 03:55:11 UTC
Permalink
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.

I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.

The problem that I'm running into is in RubyPort.cc: When the functional
read gets to the cache hierarchy, Ruby finds that the AccessPermission on
the data it's looking for is in the Busy state (in
RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy value is 2
since the data is currently in the process of being transferred up the
cache hierarchy in a timing request). Because it's in the busy state,
doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the if-statement
checking if the access failed (line 445).

In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port to
indicate the access permissions on the data? Suggestions on how this might
be fixed?

Thanks!
Joel
--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
Nilay Vaish
2012-03-18 21:54:02 UTC
Permalink
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU is
trying to make functional accesses. I rolled back the repo to look if
there is any entity by the name funcport, but grep did not provide
any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the functional
read gets to the cache hierarchy, Ruby finds that the AccessPermission on
the data it's looking for is in the Busy state (in
RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy value is 2
since the data is currently in the process of being transferred up the
cache hierarchy in a timing request). Because it's in the busy state,
doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the if-statement
checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port to
indicate the access permissions on the data? Suggestions on how this might
be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two controllers
in transient states, so the data might be in the network. You might want
to look at the access permissions of the transient states and decide if it
would be safe to read data from one of the controllers. If it is, then you
can change the access permissions accordingly.

--
Nilay
Gabe Black
2012-03-19 06:42:13 UTC
Permalink
Somewhat tangential to the problem you're having, the SE/FS merge and
the change to how ports work were pretty much entirely unrelated, they
just happened to go in around the same time. What you describe sounds
like it could be from the ports change, but, at least from what you've
described, I don't think the SE/FS merge is a factor.

Gabe
Post by Nilay Vaish
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU
is trying to make functional accesses. I rolled back the repo to look
if there is any entity by the name funcport, but grep did not provide
any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the functional
read gets to the cache hierarchy, Ruby finds that the
AccessPermission on
the data it's looking for is in the Busy state (in
RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy value is 2
since the data is currently in the process of being transferred up the
cache hierarchy in a timing request). Because it's in the busy state,
doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the if-statement
checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port to
indicate the access permissions on the data? Suggestions on how this might
be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two
controllers in transient states, so the data might be in the network.
You might want to look at the access permissions of the transient
states and decide if it would be safe to read data from one of the
controllers. If it is, then you can change the access permissions
accordingly.
--
Nilay
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Andreas Hansson
2012-03-19 14:41:31 UTC
Permalink
As we already saw with one of the Ruby regressions, the changes in the port proxies did expose similar issues already (see changeset f2ac0bca75df). In short, I am fairly confident that the port proxies are not the source of the problem.

I think it boils down to the access permissions in Ruby, but wouldn't know how/what to change. Nilay?

Andreas


-----Original Message-----
From: gem5-dev-bounces-1Gs4CP2/***@public.gmane.org [mailto:gem5-dev-bounces-1Gs4CP2/***@public.gmane.org] On Behalf Of Gabe Black
Sent: 19 March 2012 06:42
To: gem5-dev-1Gs4CP2/***@public.gmane.org
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy

Somewhat tangential to the problem you're having, the SE/FS merge and
the change to how ports work were pretty much entirely unrelated, they
just happened to go in around the same time. What you describe sounds
like it could be from the ports change, but, at least from what you've
described, I don't think the SE/FS merge is a factor.

Gabe
Post by Nilay Vaish
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU
is trying to make functional accesses. I rolled back the repo to look
if there is any entity by the name funcport, but grep did not provide
any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the functional
read gets to the cache hierarchy, Ruby finds that the
AccessPermission on
the data it's looking for is in the Busy state (in
RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy value is 2
since the data is currently in the process of being transferred up the
cache hierarchy in a timing request). Because it's in the busy state,
doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the if-statement
checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port to
indicate the access permissions on the data? Suggestions on how this might
be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two
controllers in transient states, so the data might be in the network.
You might want to look at the access permissions of the transient
states and decide if it would be safe to read data from one of the
controllers. If it is, then you can change the access permissions
accordingly.
--
Nilay
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
gem5-dev-1Gs4CP2/***@public.gmane.org
http://m5sim.org/mailman/listinfo/gem5-dev


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Joel Hestness
2012-04-09 22:56:19 UTC
Permalink
Hey guys,
I've dug into this some more, and this challenge looks pretty complex.
For some detail on what's happening, I'm trying to integrate another
simulator in with gem5, and I need to use functional accesses to get data
that this other simulator needs in the functional aspects of it's
simulation.

As I mentioned previously, when using Ruby, if a cache line happens to be
in the busy state on a functional access (e.g. if it is being passed
between caches in most protocols), the RubyPort doesn't know how to grab
the data on a functional read, or which data to update on a functional
write. The current code handles read-write and read-only data in the
caches, but doesn't handle this busy state.

@Brad, Nilay:
I tracked down some conversation you've had about functional access in a
previous thread (here:
http://www.mail-archive.com/gem5-dev-***@public.gmane.org/msg12120.html
"[gem5-dev] Functional Memory
Accesses in Ruby"), in which you articulate the exact problems that I'm
dealing with here. Unfortunately, the assumption that Ruby should not
handle functional access when a cache line is in an intermediate state has
given me some major problems with the integration I'm working on (at least
5 of my 12 preliminary benchmarks).

I'm hoping you guys could clarify a couple things for me:
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they just
contain pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate state,
there is a copy of the data that is not pointed to by the caches that are
being updated. Can you clarify this?

2) Can you also clarify what the RubyPort::M5Port variable
access_phys_mem is supposed to indicate? If set, does this mean that the
physical memory actually contains the most up-to-date (possibly even dirty
cache line) data?

Thanks!
Joel



-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access code
to work with other Ruby protocols, this solution will have to be generic to
handle different types of intermediate states. I think the correct way for
this to work is the following:
- On functional reads, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and read the portion of the line
that the functional access needs. If the line does not exist in caches,
just read from memory.
- On functional writes, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and write the portion of the
line that the functional access needs. If the write switches the cache
line's state (e.g. exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to
memory (this includes if we update lines in the shared state that aren't
dirty).
Post by Andreas Hansson
As we already saw with one of the Ruby regressions, the changes in the
port proxies did expose similar issues already (see changeset
f2ac0bca75df). In short, I am fairly confident that the port proxies are
not the source of the problem.
I think it boils down to the access permissions in Ruby, but wouldn't know
how/what to change. Nilay?
Andreas
-----Original Message-----
Behalf Of Gabe Black
Sent: 19 March 2012 06:42
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy
Somewhat tangential to the problem you're having, the SE/FS merge and
the change to how ports work were pretty much entirely unrelated, they
just happened to go in around the same time. What you describe sounds
like it could be from the ports change, but, at least from what you've
described, I don't think the SE/FS merge is a factor.
Gabe
Post by Nilay Vaish
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU
is trying to make functional accesses. I rolled back the repo to look
if there is any entity by the name funcport, but grep did not provide
any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the functional
read gets to the cache hierarchy, Ruby finds that the
AccessPermission on
the data it's looking for is in the Busy state (in
RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy value is 2
since the data is currently in the process of being transferred up the
cache hierarchy in a timing request). Because it's in the busy state,
doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the if-statement
checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port to
indicate the access permissions on the data? Suggestions on how this might
be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two
controllers in transient states, so the data might be in the network.
You might want to look at the access permissions of the transient
states and decide if it would be safe to read data from one of the
controllers. If it is, then you can change the access permissions
accordingly.
--
Nilay
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the
information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
Nilay Vaish
2012-04-13 17:15:20 UTC
Permalink
Post by Joel Hestness
Hey guys,
I've dug into this some more, and this challenge looks pretty complex.
For some detail on what's happening, I'm trying to integrate another
simulator in with gem5, and I need to use functional accesses to get data
that this other simulator needs in the functional aspects of it's
simulation.
As I mentioned previously, when using Ruby, if a cache line happens to be
in the busy state on a functional access (e.g. if it is being passed
between caches in most protocols), the RubyPort doesn't know how to grab
the data on a functional read, or which data to update on a functional
write. The current code handles read-write and read-only data in the
caches, but doesn't handle this busy state.
I tracked down some conversation you've had about functional access in a
"[gem5-dev] Functional Memory
Accesses in Ruby"), in which you articulate the exact problems that I'm
dealing with here. Unfortunately, the assumption that Ruby should not
handle functional access when a cache line is in an intermediate state has
given me some major problems with the integration I'm working on (at least
5 of my 12 preliminary benchmarks).
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they just
contain pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate state,
there is a copy of the data that is not pointed to by the caches that are
being updated. Can you clarify this?
Messages do contain data, and it is conceivable that during some
particular transcation, the data would be in the interconnection network
for some time. Hence, it is not possible at all times to honour a function
read/write request.
Post by Joel Hestness
2) Can you also clarify what the RubyPort::M5Port variable
access_phys_mem is supposed to indicate? If set, does this mean that the
physical memory actually contains the most up-to-date (possibly even dirty
cache line) data?
The access_phys_mem denotes whether or not a copy of the entire memory is
being maintained. You might know that as of now, programmed IO accesses do
not flow through ruby. So it is not possible to maintain a correct copy of
all the data. Hence, is separate copy of the memory is maintained, which
gets to see all the read/write operations.
Post by Joel Hestness
Thanks!
Joel
-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access code
to work with other Ruby protocols, this solution will have to be generic to
handle different types of intermediate states. I think the correct way for
- On functional reads, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and read the portion of the line
that the functional access needs. If the line does not exist in caches,
just read from memory.
- On functional writes, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and write the portion of the
line that the functional access needs. If the write switches the cache
line's state (e.g. exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to
memory (this includes if we update lines in the shared state that aren't
dirty).
This is exactly what is being done. The problem is that at times none of
caches / memory controller may have the data.

--
Nilay
Steve Reinhardt
2012-04-13 21:48:32 UTC
Permalink
Post by Nilay Vaish
Post by Joel Hestness
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they just
contain pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate state,
there is a copy of the data that is not pointed to by the caches that are
being updated. Can you clarify this?
Messages do contain data, and it is conceivable that during some
particular transcation, the data would be in the interconnection network
for some time. Hence, it is not possible at all times to honour a function
read/write request.
The way the classic memory system works is that interconnect components
that buffer state have to implement functional access methods that check
whether data in a buffered packet should be read or written (see for
example http://repo.gem5.org/gem5/file/5de232ed3a28/src/mem/bridge.cc#l440).
Brad and I have discussed this, and my impression is that this is also
feasible for Ruby, though Brad thinks it would be a headache to graft it on
to all the existing network models (paraphrasing his opinion from my faulty
memory). Nevertheless I think it should be done eventually. Brad tends to
push back, probably because he rightly fears that he would be the one to
end up doing it ;-).

The key point is that "not possible" is an overstatement.
Post by Nilay Vaish
2) Can you also clarify what the RubyPort::M5Port variable
Post by Joel Hestness
access_phys_mem is supposed to indicate? If set, does this mean that the
physical memory actually contains the most up-to-date (possibly even dirty
cache line) data?
The access_phys_mem denotes whether or not a copy of the entire memory is
being maintained. You might know that as of now, programmed IO accesses do
not flow through ruby. So it is not possible to maintain a correct copy of
all the data. Hence, is separate copy of the memory is maintained, which
gets to see all the read/write operations.
Do we only turn on access_phys_mem for full-system simulations then?
Post by Nilay Vaish
Post by Joel Hestness
-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access code
to work with other Ruby protocols, this solution will have to be generic to
handle different types of intermediate states. I think the correct way for
- On functional reads, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and read the portion of the line
that the functional access needs. If the line does not exist in caches,
just read from memory.
- On functional writes, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and write the portion of the
line that the functional access needs. If the write switches the cache
line's state (e.g. exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to
memory (this includes if we update lines in the shared state that aren't
dirty).
This is exactly what is being done. The problem is that at times none of
caches / memory controller may have the data.
My understanding is that, in addition to cases where the only valid data is
in the network (discussed above), there may also be cases where the data is
in the caches or memory but in some intermediate protocol state such that
it's not clear if it's the latest version, and it would require manual
effort to identify all the possible protocol states and whether they should
be considered valid for the purposes of responding to a functional request.
Again, this is a potentially faulty recollection based on prior
conversations with Brad.

Overall I would love to see full support for functional accesses in Ruby,
and we may reach a point where we really need to figure out how to make it
happen (maybe we are reaching it now). It's not a simple task, but maybe
we should brainstorm on how to simplify it, or decide that we can't do it
all at once but can have a plan to get there incrementally. Brad really
needs to be part of that discussion though, and he's off at the baseball
game today...

Steve
Nilay Vaish
2012-04-14 21:54:09 UTC
Permalink
Post by Steve Reinhardt
Post by Nilay Vaish
Post by Joel Hestness
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they just
contain pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate state,
there is a copy of the data that is not pointed to by the caches that are
being updated. Can you clarify this?
Messages do contain data, and it is conceivable that during some
particular transcation, the data would be in the interconnection network
for some time. Hence, it is not possible at all times to honour a function
read/write request.
The way the classic memory system works is that interconnect components
that buffer state have to implement functional access methods that check
whether data in a buffered packet should be read or written (see for
example http://repo.gem5.org/gem5/file/5de232ed3a28/src/mem/bridge.cc#l440).
Brad and I have discussed this, and my impression is that this is also
feasible for Ruby, though Brad thinks it would be a headache to graft it on
to all the existing network models (paraphrasing his opinion from my faulty
memory). Nevertheless I think it should be done eventually. Brad tends to
push back, probably because he rightly fears that he would be the one to
end up doing it ;-).
The key point is that "not possible" is an overstatement.
When I wrote the phrase 'not possible', I only had the current setup in
mind. Last summer, Brad and I did discuss the possibility of involving
network components in satisfying functional requests, but we decided
against doing it.
Post by Steve Reinhardt
Post by Nilay Vaish
2) Can you also clarify what the RubyPort::M5Port variable
Post by Joel Hestness
access_phys_mem is supposed to indicate? If set, does this mean that the
physical memory actually contains the most up-to-date (possibly even dirty
cache line) data?
The access_phys_mem denotes whether or not a copy of the entire memory is
being maintained. You might know that as of now, programmed IO accesses do
not flow through ruby. So it is not possible to maintain a correct copy of
all the data. Hence, is separate copy of the memory is maintained, which
gets to see all the read/write operations.
Do we only turn on access_phys_mem for full-system simulations then?
Currently, it is set to true by default. It is set to false for the
various testers we have for ruby. But for se mode simulations, it is still
being set to true.
Post by Steve Reinhardt
Post by Nilay Vaish
Post by Joel Hestness
-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access code
to work with other Ruby protocols, this solution will have to be generic to
handle different types of intermediate states. I think the correct way for
- On functional reads, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and read the portion of the line
that the functional access needs. If the line does not exist in caches,
just read from memory.
- On functional writes, find the cache(s) that contains the most recent
data (e.g. modified, exclusive or shared), and write the portion of the
line that the functional access needs. If the write switches the cache
line's state (e.g. exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to
memory (this includes if we update lines in the shared state that aren't
dirty).
This is exactly what is being done. The problem is that at times none of
caches / memory controller may have the data.
My understanding is that, in addition to cases where the only valid data is
in the network (discussed above), there may also be cases where the data is
in the caches or memory but in some intermediate protocol state such that
it's not clear if it's the latest version, and it would require manual
effort to identify all the possible protocol states and whether they should
be considered valid for the purposes of responding to a functional request.
Again, this is a potentially faulty recollection based on prior
conversations with Brad.
I think most of the protocol states have been marked correctly wrt whether
or not they can satisfy functional read/write requests, though some states
may have been marked conservatively and therefore disallow functional
accesses. I think the only cases when we do not let the access succeed is
when

a) there is a read request and the latest data is in the network only,
b) there is a write request and the network holds the data.
Post by Steve Reinhardt
Overall I would love to see full support for functional accesses in Ruby,
and we may reach a point where we really need to figure out how to make it
happen (maybe we are reaching it now). It's not a simple task, but maybe
we should brainstorm on how to simplify it, or decide that we can't do it
all at once but can have a plan to get there incrementally. Brad really
needs to be part of that discussion though, and he's off at the baseball
game today...
Steve
--
Nilay
Steve Reinhardt
2012-04-15 03:49:50 UTC
Permalink
Post by Steve Reinhardt
The key point is that "not possible" is an overstatement.
When I wrote the phrase 'not possible', I only had the current setup in
mind. Last summer, Brad and I did discuss the possibility of involving
network components in satisfying functional requests, but we decided
against doing it.
It may be time to reconsider...

Steve
Beckmann, Brad
2012-04-17 05:28:37 UTC
Permalink
Hi Joel,

Sorry for the slow reply...is this still an issue? Since you and Nilay are now working on the same floor, did you resolve this locally instead of via email?

Brad
Post by Andreas Hansson
-----Original Message-----
On Behalf Of Joel Hestness
Sent: Monday, April 09, 2012 3:56 PM
To: gem5 Developer List
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy
Hey guys,
I've dug into this some more, and this challenge looks pretty complex.
For some detail on what's happening, I'm trying to integrate another simulator
in with gem5, and I need to use functional accesses to get data that this other
simulator needs in the functional aspects of it's simulation.
As I mentioned previously, when using Ruby, if a cache line happens to be in
the busy state on a functional access (e.g. if it is being passed between caches
in most protocols), the RubyPort doesn't know how to grab the data on a
functional read, or which data to update on a functional write. The current
code handles read-write and read-only data in the caches, but doesn't handle
this busy state.
I tracked down some conversation you've had about functional access in a
"[gem5-dev] Functional Memory
Accesses in Ruby"), in which you articulate the exact problems that I'm dealing
with here. Unfortunately, the assumption that Ruby should not handle
functional access when a cache line is in an intermediate state has given me
some major problems with the integration I'm working on (at least
5 of my 12 preliminary benchmarks).
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they just contain
pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate state, there is
a copy of the data that is not pointed to by the caches that are being updated.
Can you clarify this?
2) Can you also clarify what the RubyPort::M5Port variable access_phys_mem
is supposed to indicate? If set, does this mean that the physical memory
actually contains the most up-to-date (possibly even dirty cache line) data?
Thanks!
Joel
-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access code to
work with other Ruby protocols, this solution will have to be generic to handle
different types of intermediate states. I think the correct way for this to work is
- On functional reads, find the cache(s) that contains the most recent data
(e.g. modified, exclusive or shared), and read the portion of the line that the
functional access needs. If the line does not exist in caches, just read from
memory.
- On functional writes, find the cache(s) that contains the most recent data
(e.g. modified, exclusive or shared), and write the portion of the line that the
functional access needs. If the write switches the cache line's state (e.g.
exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to memory
(this includes if we update lines in the shared state that aren't dirty).
On Mon, Mar 19, 2012 at 9:41 AM, Andreas Hansson
Post by Andreas Hansson
As we already saw with one of the Ruby regressions, the changes in the
port proxies did expose similar issues already (see changeset
f2ac0bca75df). In short, I am fairly confident that the port proxies
are not the source of the problem.
I think it boils down to the access permissions in Ruby, but wouldn't
know how/what to change. Nilay?
Andreas
-----Original Message-----
On
Post by Andreas Hansson
Behalf Of Gabe Black
Sent: 19 March 2012 06:42
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy
Somewhat tangential to the problem you're having, the SE/FS merge and
the change to how ports work were pretty much entirely unrelated, they
just happened to go in around the same time. What you describe sounds
like it could be from the ports change, but, at least from what you've
described, I don't think the SE/FS merge is a factor.
Gabe
Post by Nilay Vaish
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge
my old patches. It looks like the TranslatingPort that used to be
the O3CPU funcport has changed, and I run into a fatal when trying
to run the new code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE
mode), and using a Ruby cache hierarchy. With the old code, the
O3CPU would simply use the funcport (before the FS/SE merge) to
read from the physmem directly (::aside:: this already confuses me,
since if the data is modified in a cache, shouldn't that data be
read?). With the updated gem5 code, the functional access uses the
O3CPU's data port, which is connected to a Ruby sequencer (more as
I would expect), which checks to see if the data is in a cache
before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3
CPU is trying to make functional accesses. I rolled back the repo to
look if there is any entity by the name funcport, but grep did not
provide any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the
functional read gets to the cache hierarchy, Ruby finds that the
AccessPermission on the data it's looking for is in the Busy state
(in RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy
value is 2 since the data is currently in the process of being
transferred up the cache hierarchy in a timing request). Because
it's in the busy state, doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the
if-statement checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port
to indicate the access permissions on the data? Suggestions on how
this might be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two
controllers in transient states, so the data might be in the network.
You might want to look at the access permissions of the transient
states and decide if it would be safe to read data from one of the
controllers. If it is, then you can change the access permissions
accordingly.
--
Nilay
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments
are confidential and may also be privileged. If you are not the
intended recipient, please notify the sender immediately and do not
disclose the contents to any other person, use it for any purpose, or
store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
Joel Hestness
2012-04-24 00:15:40 UTC
Permalink
Hi Brad (and others),
Sorry for my delay on this. Since running into this issue, I've decided
that a better solution for the problem I'm experiencing is to bring this
other simulator into line with the way Ruby functional accesses are meant
to work (i.e. for very infrequent, typically non-cache-resident accesses).

If I run into major issues here, I may reflect back on making some of
these changes in Ruby, but for now, I'd like to avoid the complexity
involved in making potentially major changes.

Thanks for all the input,
Joel
Post by Beckmann, Brad
Hi Joel,
Sorry for the slow reply...is this still an issue? Since you and Nilay
are now working on the same floor, did you resolve this locally instead of
via email?
Brad
Post by Andreas Hansson
-----Original Message-----
On Behalf Of Joel Hestness
Sent: Monday, April 09, 2012 3:56 PM
To: gem5 Developer List
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy
Hey guys,
I've dug into this some more, and this challenge looks pretty complex.
For some detail on what's happening, I'm trying to integrate another
simulator
Post by Andreas Hansson
in with gem5, and I need to use functional accesses to get data that
this other
Post by Andreas Hansson
simulator needs in the functional aspects of it's simulation.
As I mentioned previously, when using Ruby, if a cache line happens to
be in
Post by Andreas Hansson
the busy state on a functional access (e.g. if it is being passed
between caches
Post by Andreas Hansson
in most protocols), the RubyPort doesn't know how to grab the data on a
functional read, or which data to update on a functional write. The
current
Post by Andreas Hansson
code handles read-write and read-only data in the caches, but doesn't
handle
Post by Andreas Hansson
this busy state.
I tracked down some conversation you've had about functional access in
a
Post by Andreas Hansson
"[gem5-dev] Functional Memory
Accesses in Ruby"), in which you articulate the exact problems that I'm
dealing
Post by Andreas Hansson
with here. Unfortunately, the assumption that Ruby should not handle
functional access when a cache line is in an intermediate state has
given me
Post by Andreas Hansson
some major problems with the integration I'm working on (at least
5 of my 12 preliminary benchmarks).
1) It's not clear to me whether the messages passed between cache
controllers contain copies of the data being transferred, or if they
just contain
Post by Andreas Hansson
pointers to locations which contain the data to be transferred.
This is important if, during a functional write in an intermediate
state, there is
Post by Andreas Hansson
a copy of the data that is not pointed to by the caches that are being
updated.
Post by Andreas Hansson
Can you clarify this?
2) Can you also clarify what the RubyPort::M5Port variable
access_phys_mem
Post by Andreas Hansson
is supposed to indicate? If set, does this mean that the physical memory
actually contains the most up-to-date (possibly even dirty cache line)
data?
Post by Andreas Hansson
Thanks!
Joel
-------- Other scribblings for the record --------
Making this more complex is that in order for this functional access
code to
Post by Andreas Hansson
work with other Ruby protocols, this solution will have to be generic to
handle
Post by Andreas Hansson
different types of intermediate states. I think the correct way for
this to work is
Post by Andreas Hansson
- On functional reads, find the cache(s) that contains the most
recent data
Post by Andreas Hansson
(e.g. modified, exclusive or shared), and read the portion of the line
that the
Post by Andreas Hansson
functional access needs. If the line does not exist in caches, just
read from
Post by Andreas Hansson
memory.
- On functional writes, find the cache(s) that contains the most
recent data
Post by Andreas Hansson
(e.g. modified, exclusive or shared), and write the portion of the line
that the
Post by Andreas Hansson
functional access needs. If the write switches the cache line's state
(e.g.
Post by Andreas Hansson
exclusive to modified), update the appropriate state.
If the line does not exist in caches, simply forward the request on to
memory
Post by Andreas Hansson
(this includes if we update lines in the shared state that aren't dirty).
On Mon, Mar 19, 2012 at 9:41 AM, Andreas Hansson
Post by Andreas Hansson
As we already saw with one of the Ruby regressions, the changes in the
port proxies did expose similar issues already (see changeset
f2ac0bca75df). In short, I am fairly confident that the port proxies
are not the source of the problem.
I think it boils down to the access permissions in Ruby, but wouldn't
know how/what to change. Nilay?
Andreas
-----Original Message-----
On
Post by Andreas Hansson
Behalf Of Gabe Black
Sent: 19 March 2012 06:42
Subject: Re: [gem5-dev] Debugging New Translating Ports / PortProxy
Somewhat tangential to the problem you're having, the SE/FS merge and
the change to how ports work were pretty much entirely unrelated, they
just happened to go in around the same time. What you describe sounds
like it could be from the ports change, but, at least from what you've
described, I don't think the SE/FS merge is a factor.
Gabe
Post by Nilay Vaish
Post by Joel Hestness
Hey guys,
I just updated my repo to the gem5 head, and I'm trying to merge
my old patches. It looks like the TranslatingPort that used to be
the O3CPU funcport has changed, and I run into a fatal when trying
to run the new code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE
mode), and using a Ruby cache hierarchy. With the old code, the
O3CPU would simply use the funcport (before the FS/SE merge) to
read from the physmem directly (::aside:: this already confuses me,
since if the data is modified in a cache, shouldn't that data be
read?). With the updated gem5 code, the functional access uses the
O3CPU's data port, which is connected to a Ruby sequencer (more as
I would expect), which checks to see if the data is in a cache
before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3
CPU is trying to make functional accesses. I rolled back the repo to
look if there is any entity by the name funcport, but grep did not
provide any results. Which code are you referring to?
Post by Joel Hestness
The problem that I'm running into is in RubyPort.cc: When the
functional read gets to the cache hierarchy, Ruby finds that the
AccessPermission on the data it's looking for is in the Busy state
(in RubyPort::M5Port::doFunctionalRead(PacketPtr pkt) the num_busy
value is 2 since the data is currently in the process of being
transferred up the cache hierarchy in a timing request). Because
it's in the busy state, doFunctionalRead returns false, which
causes RubyPort::M5Port::recvFunctional to fall into the
if-statement checking if the access failed (line 445).
In this case, couldn't the functional read just grab the data from
memory? If so, shouldn't there be better signalling between the
doFunctional<type> functions and recvFunctional in RubyPort::M5Port
to indicate the access permissions on the data? Suggestions on how
this might be fixed?
It is possible that the data in the memory is not up to date, as you
mentioned before. In fact, busy = 2 means that there are two
controllers in transient states, so the data might be in the network.
You might want to look at the access permissions of the transient
states and decide if it would be safe to read data from one of the
controllers. If it is, then you can change the access permissions
accordingly.
--
Nilay
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments
are confidential and may also be privileged. If you are not the
intended recipient, please notify the sender immediately and do not
disclose the contents to any other person, use it for any purpose, or
store or copy the information in any medium. Thank you.
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Wisconsin - Madison
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
Joel Hestness
2012-04-09 23:15:26 UTC
Permalink
Hey Nilay,

Hey guys,
Post by Nilay Vaish
Post by Joel Hestness
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU is
trying to make functional accesses. I rolled back the repo to look if there
is any entity by the name funcport, but grep did not provide any results.
Which code are you referring to?
Modifications to the use of the funcPort were made in the following
changesets:
8712: http://repo.gem5.org/gem5/rev/7f762428a9f5
8852: http://repo.gem5.org/gem5/rev/c744483edfcf

--
Joel Hestness
PhD Student, Computer Architecture
Dept. of Computer Science, University of Wisconsin - Madison
Dept. of Computer Science, University of Texas - Austin
http://www.cs.utexas.edu/~hestness
Nilay Vaish
2012-04-13 17:09:07 UTC
Permalink
Post by Joel Hestness
Hey Nilay,
Hey guys,
Post by Nilay Vaish
Post by Joel Hestness
I just updated my repo to the gem5 head, and I'm trying to merge my old
patches. It looks like the TranslatingPort that used to be the O3CPU
funcport has changed, and I run into a fatal when trying to run the new
code.
I'm trying to run simulation with x86 O3CPU, full_system=False (SE mode),
and using a Ruby cache hierarchy. With the old code, the O3CPU would
simply use the funcport (before the FS/SE merge) to read from the physmem
directly (::aside:: this already confuses me, since if the data is modified
in a cache, shouldn't that data be read?). With the updated gem5 code, the
functional access uses the O3CPU's data port, which is connected to a Ruby
sequencer (more as I would expect), which checks to see if the data is in a
cache before grabbing the data for the read.
I think the first thing that you need look at is as to why the O3 CPU is
trying to make functional accesses. I rolled back the repo to look if there
is any entity by the name funcport, but grep did not provide any results.
Which code are you referring to?
Modifications to the use of the funcPort were made in the following
8712: http://repo.gem5.org/gem5/rev/7f762428a9f5
8852: http://repo.gem5.org/gem5/rev/c744483edfcf
Which fatal() are you running into? Can you post the function call trace?

--
Nilay
Loading...