[Barrelfish-users] Questions about IDC: Poll or Interrupt

[Baumann Andrew]

Hi,

I think there are really two separate questions here: 1. is there a performance overhead to requiring a thread to call into an explicit event-handling function rather than interrupting it and/or firing off a new thread for each event, and 2. (which perhaps you were thinking of but didn’t ask explicitly) what is the overhead of polling for incoming messages if you’re using a message transport (like CC-UMP in Barrelfish) that requires it.

The answer to the first question is no. If your thread is doing something else and not calling the event-handler, then presumably it is doing that something else because it has work to do. Conversely, if you have multiple threads, and are worried that the scheduler might not run your “polling thread” (which I prefer to call the event handler thread, since it’s entirely possible that it doesn’t poll and simply sleeps until an event is signalled), then that must mean that there are other runnable threads doing work in your program. Again, if you’re doing other work, of course there is going to be a delay processing messages, but it’s hard to see why this would count as performance degradation.

The answer to the second question is maybe. In the case of the UMP transport specifically, it does actively poll to detect incoming messages, and it does that using one thread that has called into event_dispatch(). If you are running multiple threads, this will have a performance impact. We could be better about managing the polling than we are in the current version of Barrelfish, but at the end of the day this stems from a fundamental limitation of implementing message passing above shared memory. The best thing to do to avoid the overhead is to have one thread only, and structure your system as fully event driven: at the top-level you have an event-handler loop, and all the “work” your system does is driven by events. This way you only poll where there is no other work waiting to be done (i.e. no pending events).

Cheers,
Andrew

From: jhshi89 at gmail.com<mailto:jhshi89 at gmail.com> [mailto:jhshi89 at gmail.com]<mailto:[mailto:jhshi89 at gmail.com]> On Behalf Of Shi Jinghao (???)
Sent: Wednesday, 14 March, 2012 21:54
To: Baumann Andrew
Cc: barrelfish-users at lists.inf.ethz.ch<mailto:barrelfish-users at lists.inf.ethz.ch>
Subject: Re: [Barrelfish-users] Questions about IDC: Poll or Interrupt

Hi, Andrew, thanks for you reply.

Your explanation on IDC design philosophy quite make sense in that the communication becomes more controllable and predicable. However, will this on sacrifice of performance? I mean, I'll get message only when the polling thread got run, which is decided by the scheduler. So will this cause extra latency ?

I raise this question because I'm developing a library on Barrelfish that relies heavily on inter-core communication.
And I worried that polling events will caused performance degradation. But this just a guess, I plan to conduct some
benchmark to dig out more.

Also thanks for the reminder that binding operations are not thread-safe. I've noticed that and now I do all the binding stuff (export, bind, send and recv) only in the polling thread, avoiding potential race.

BTW, I also find your slides on IDC in 2010 Barrelfish workshop very helpful. Thanks for your efforts.

Regards,
Jinghao

2012/3/15 Baumann Andrew <andrewb at inf.ethz.ch<mailto:andrewb at inf.ethz.ch>>
Hi,

Barrelfish/Flounder IDC is asynchronous, but it is not concurrent. Unless you create a pool of threads that call event_dispatch() (which is what you seem to have done), there is no other thread that will execute the callbacks for you. This is by design – you shouldn’t be forced to deal with non-deterministic concurrency just to get the benefits of asynchronous messaging.

Note that if you do want the concurrency of multiple threads, the onus is on you to correctly serialise their operations when they interact with shared state such as the binding objects. The binding operations (i.e. message send, receive, etc.) are not thread-safe.

Andrew

From: Shi Jinghao (史经浩） [mailto:jhshi at cs.hku.hk<mailto:jhshi at cs.hku.hk>]
Sent: Wednesday, 14 March, 2012 7:14
To: barrelfish-users at lists.inf.ethz.ch<mailto:barrelfish-users at lists.inf.ethz.ch>
Subject: [Barrelfish-users] Questions about IDC: Poll or Interrupt

Hi,

I am recently working with IDC in Barrelfish, using the API generated by Flounder. I found that I need
to use a separate polling thread to poll the waitset, so that my call backs can get called.

For example, when sending a message:  dummy__msg_tx(binding, MKCONT(send_cb, NULL), value)

Since the send is asynchronous, I can continue to do other stuff after this call. However, I expect that
my send_cb will be called (*interrupt*) once the send is success, which is not the case:  I must manually poll the
binding's waitset using event_dispatch to get send_cb get called.

I now use a separate polling thread to do all this stuff and it works. But I was wondering:

    - Is it designed as this way? Or I don't actually need to poll, just I don't know how...

    - If this is the way as designed, then why? This seems not that "event-driven", right?

Thanks for your reading and forgive me if I asked a stupid question.

Regards,
Jinghao



-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.inf.ethz.ch/pipermail/barrelfish-users/attachments/20120316/01b59bed/attachment-0001.html