[Barrelfish-users] Porting Barrelfish to a new architecture?

[Stefan Kästle]

Hi,

indeed, we thought about notifying the receiver in the context of our 
hardware support for message passing work (however, we did not do any 
evaluation yet).

Having a hardware device dedicated to message passing buys us quite a 
bit of flexibility. Assume we had one message passing device closely 
coupled to every core in the system. This device could then receive 
messages on behalf of all applications on a core. Every channel (i.e. 
pair of sending + receiving app) has some state on this device anyway 
due to isolation between channels. It would be easy to add some more 
bits to program the notification behavior on a per-channel level (the 
programming interface could be exposed to the kernel or even the apps. 
This makes sense because the apps know what kind of message load to 
expect). The message passing device could than choose from the following 
schemes on a per-channel basis:

1) Interrupts: e.g. infrequently used high priority channels. Coalescing 
the interrupt could be done too, possibly depending on the number of 
messages received or a timeout value (see Arsenic work)

2) Do not notify the receiver in case the receiver wants to receive 
messages in a polling loop. This probably makes most sense for apps 
receiving many messages, especially if the latency is not critical.

3) Talk to the scheduler on context-switching. This is useful to wake up 
blocked threads.

In case the receiving application is already running, it might not make 
sense to issue any notification, as the app might already be in the 
polling loop.

On 03/16/2012 04:20 AM, Baumann Andrew wrote:
>
>     * What additional primitives, if any, are desirable for existing versions of
>
>         Barrelfish to run well?
>
>   
> Efficient inter-core messaging would be great! (more on this below)
>   
>
>     * What further primitives would Barrelfish developers like to see on
>
>     future hardware
>
>        platforms?
>
>   
> Message-passing is the big one. We sketched out some ideas in this HotOS paper:
> http://www.barrelfish.org/gap_hotos11.pdf
>   
> Essentially we'd like a remote write operation that pushes some data close to a remote core, and an asynchronous notification that causes a lightweight exception/control transfer on the target. Previous versions of hardware-supported message passing have provided some version of the former, but neglected the latter.
>
> I agree, that sounds extremely useful; implicit message passing by 
> piggy-backing on the coherence hardware isn't a great solution.  The 
> closest thing we have is a broadcast update instruction that 
> broadcasts a cache line from the L2 cache of one cluster of cores into 
> the L2 caches of all other cores.  We only needed broadcast when we 
> initially designed the architecture because we were focused on a 
> task-based programming model where arbitrary point-to-point 
> communication simply doesn't occur.  For being able to run more 
> general classes of program, this generalization seems very useful.  
> Being able to pass an IPI along with the data seems to give a 
> similarly large benefit.  For the use cases you have in mind, is it 
> important that the interrupt be serviced immediately, or would it be 
> sufficient to wait until the target thread finishes what it's doing 
> and does the moral equivalent of calling yield()?  If these IPIs are 
> frequent, having the servicing be cooperative rather than preemptive 
> may buy you efficiency by not having to spill the thread context to 
> memory (maybe when the target yield()s, everything in its context is 
> known to be dead).
>
> Tim and Stefan should really answer this... they have gone into much 
> more depth than I on the notification mechanisms.
>
> I think it's useful to decouple the data transfer from the 
> notification (or at least make the notification optional). I don't 
> think it's always necessary to deliver the notification immediately, 
> and coalescing multiple undelivered notifications makes sense, but I 
> don't see the value in a notification that requires an explicit action 
> on the receiver side... isn't that just polling? Also, if I may be so 
> rude, the decision of what to do with thread context and whether it 
> must be spilled is a software problem :) You can imagine a fault 
> handler that knows enough about the actual program being run to decide 
> whether to spill context and switch to a handler, or just set a flag 
> that a notification arrived and queue it for later processing.
>
>   

-- 
Stefan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.inf.ethz.ch/pipermail/barrelfish-users/attachments/20120316/9acff0a5/attachment.html