Discussion:
[gem5-dev] KvmCPU Behaviour
(too old to reply)
Alexandru Duţu via gem5-dev
2014-06-11 22:05:14 UTC
Permalink
Hi all,

I am trying to understand the current behaviour of KvmCPU. Previous to the
following changeset

http://repo.gem5.org/gem5/rev/2360411a16be

after every kvmRun the status was switched to runningService, a new event
was scheduled to the event queue and so with new tick handleKvmExit will be
called.

However, with the latest changesets on kvm (
http://repo.gem5.org/gem5/rev/5c2ecad1a3c9) there are a lot of KVM
enter-exit cycles with 0 instructions executed. It seems that the vCPU gets
interrupted by the timer a lot more, which causes kvm to exit without
executing anything.

An explanation for this behaviour is highly appreciated, is there a need
for timer interrupts that are not allowing the vCPU to actually execute? Or
this might be a side effect of the differences in timers across different
x86 vendors (I am running on an AMD system)?

Best regards,
--
Alex
Alexandru Duţu via gem5-dev
2014-06-12 19:59:59 UTC
Permalink
Hi everyone,

It seems that the KVM enter-exit cycles with 0 instructions executed are
caused by the events scheduled from the DRAMCtrl object. The issues is that
these events have a very short period which is not sufficient for KvmCPU to
even start execution.

I am not familiar with DRAMCtrl memory, it seems that SimpleMemory does not
have this issue with KvmCPU.

Best regards,
Post by Alexandru Duţu via gem5-dev
Hi all,
I am trying to understand the current behaviour of KvmCPU. Previous to the
following changeset
http://repo.gem5.org/gem5/rev/2360411a16be
after every kvmRun the status was switched to runningService, a new event
was scheduled to the event queue and so with new tick handleKvmExit will be
called.
However, with the latest changesets on kvm (
http://repo.gem5.org/gem5/rev/5c2ecad1a3c9) there are a lot of KVM
enter-exit cycles with 0 instructions executed. It seems that the vCPU gets
interrupted by the timer a lot more, which causes kvm to exit without
executing anything.
An explanation for this behaviour is highly appreciated, is there a need
for timer interrupts that are not allowing the vCPU to actually execute? Or
this might be a side effect of the differences in timers across different
x86 vendors (I am running on an AMD system)?
Best regards,
--
Alex
--
Alex
Andreas Hansson via gem5-dev
2014-06-12 21:09:30 UTC
Permalink
Hi Alex,

That sounds rather odd. The only event that should happen when there are
no request going through the memory system (the KVM CPU operates directly
on the underlying backing store), is the refresh, which happens roughly
every 64 ms. Could you elaborate on what event it is you are referring to?

I’ll try and have a look as well to see what’s happening.

Thanks,

Andreas
Post by Alexandru Duţu via gem5-dev
Hi everyone,
It seems that the KVM enter-exit cycles with 0 instructions executed are
caused by the events scheduled from the DRAMCtrl object. The issues is that
these events have a very short period which is not sufficient for KvmCPU to
even start execution.
I am not familiar with DRAMCtrl memory, it seems that SimpleMemory does not
have this issue with KvmCPU.
Best regards,
Post by Alexandru Duţu via gem5-dev
Hi all,
I am trying to understand the current behaviour of KvmCPU. Previous to the
following changeset
http://repo.gem5.org/gem5/rev/2360411a16be
after every kvmRun the status was switched to runningService, a new event
was scheduled to the event queue and so with new tick handleKvmExit will be
called.
However, with the latest changesets on kvm (
http://repo.gem5.org/gem5/rev/5c2ecad1a3c9) there are a lot of KVM
enter-exit cycles with 0 instructions executed. It seems that the vCPU gets
interrupted by the timer a lot more, which causes kvm to exit without
executing anything.
An explanation for this behaviour is highly appreciated, is there a need
for timer interrupts that are not allowing the vCPU to actually execute? Or
this might be a side effect of the differences in timers across different
x86 vendors (I am running on an AMD system)?
Best regards,
--
Alex
--
Alex
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
Alex via gem5-dev
2014-06-12 22:58:34 UTC
Permalink
Post by Andreas Hansson via gem5-dev
Hi Alex,
That sounds rather odd. The only event that should happen when there are
no request going through the memory system (the KVM CPU operates directly
on the underlying backing store), is the refresh, which happens roughly
every 64 ms. Could you elaborate on what event it is you are referring to?
I’ll try and have a look as well to see what’s happening.
Thanks,
Andreas
Hi Andreas,

This is exactly right, it is the refresh event that is causing all this.

Best,
Alex
Andreas Hansson via gem5-dev
2014-06-13 07:11:06 UTC
Permalink
Hi Alex,

Thanks for the clarification. I still don¹t understand how an event every
64 ms (which should be an awful lot of CPU cycles), could cause such an
issue.

Andreas
Post by Alex via gem5-dev
Post by Andreas Hansson via gem5-dev
Hi Alex,
That sounds rather odd. The only event that should happen when there are
no request going through the memory system (the KVM CPU operates directly
on the underlying backing store), is the refresh, which happens roughly
every 64 ms. Could you elaborate on what event it is you are referring to?
I¹ll try and have a look as well to see what¹s happening.
Thanks,
Andreas
Hi Andreas,
This is exactly right, it is the refresh event that is causing all this.
Best,
Alex
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
Andreas Sandberg via gem5-dev
2014-06-13 10:19:48 UTC
Permalink
Hi Alex,

I had a quick look trying to reproduce the problem, but didn't manage to
do so. However, I remember this being a problem when I initially
developed the KVM stuff. I don't think it is related to changeset
5c2ecad1a3c9, instead I'd suspect it's something that happened before that.

In general, the KVM CPU model does not like high-frequency events. The
reason is that the CPU needs to simulate timing to make timer interrupts
happen correctly. The way this is done is by assuming that there is a
constant time conversion factor (hostFactor) between the host and the guest.

The problem here is that timing is a bit tricky. There is actually no
way (we could use performance counters, but the API is a bit broken so
it doesn't work well in practice) to setup a timer that only measures
time spent in KVM. Instead, we setup a normal POSIX timer that measures
wall-clock time.

In summary, I can think of three possible error sources:
* Your system might be taking longer than usual to enter into KVM,
which makes it more sensitive to high-frequency events.

* The time scaling factor (hostFactor) might be off.

* Weird quirkiness in the AMD virtualization driver. I haven't tested
the KVM stuff on AMD hardware, so YMMV. A more recent kernel might help.


I usually calculate the scaling factor like this:

hostFactor = c * f_host / f_guest

Where f_host is the frequency of the host, f_guest the frequency of the
simulated CPU, and c a correction factor (usually close to 1, I used
0.95 for the experiments in my thesis [1]).

If setting the host factor doesn't help, I'd suggest that you disable
the DRAM model when running in KVM as a workaround. Also, I'd highly
appreciate if you could try to bisect this issue. There should be
support for bisection in Mercurial which lets you select two revisions
(a good and a bad) and then tries to pinpoint the exact revision where
the bug was introduced.

//Andreas

[1] http://urn.kb.se/resolve?urn=urn:se:uu:diva-220652
Post by Alexandru Duţu via gem5-dev
Hi everyone,
It seems that the KVM enter-exit cycles with 0 instructions executed are
caused by the events scheduled from the DRAMCtrl object. The issues is that
these events have a very short period which is not sufficient for KvmCPU to
even start execution.
I am not familiar with DRAMCtrl memory, it seems that SimpleMemory does not
have this issue with KvmCPU.
Best regards,
Post by Alexandru Duţu via gem5-dev
Hi all,
I am trying to understand the current behaviour of KvmCPU. Previous to the
following changeset
http://repo.gem5.org/gem5/rev/2360411a16be
after every kvmRun the status was switched to runningService, a new event
was scheduled to the event queue and so with new tick handleKvmExit will be
called.
However, with the latest changesets on kvm (
http://repo.gem5.org/gem5/rev/5c2ecad1a3c9) there are a lot of KVM
enter-exit cycles with 0 instructions executed. It seems that the vCPU gets
interrupted by the timer a lot more, which causes kvm to exit without
executing anything.
An explanation for this behaviour is highly appreciated, is there a need
for timer interrupts that are not allowing the vCPU to actually execute? Or
this might be a side effect of the differences in timers across different
x86 vendors (I am running on an AMD system)?
Best regards,
--
Alex
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
Loading...