<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
indeed, we thought about notifying the receiver in the context of
our hardware support for message passing work (however, we did not
do any evaluation yet).<br>
<br>
Having a hardware device dedicated to message passing buys us quite
a bit of flexibility. Assume we had one message passing device
closely coupled to every core in the system. This device could then
receive messages on behalf of all applications on a core. Every
channel (i.e. pair of sending + receiving app) has some state on
this device anyway due to isolation between channels. It would be
easy to add some more bits to program the notification behavior on a
per-channel level (the programming interface could be exposed to the
kernel or even the apps. This makes sense because the apps know what
kind of message load to expect). The message passing device could
than choose from the following schemes on a per-channel basis:<br>
<br>
1) Interrupts: e.g. infrequently used high priority channels.
Coalescing the interrupt could be done too, possibly depending on
the number of messages received or a timeout value (see Arsenic
work)<br>
<br>
2) Do not notify the receiver in case the receiver wants to receive
messages in a polling loop. This probably makes most sense for apps
receiving many messages, especially if the latency is not critical.<br>
<br>
3) Talk to the scheduler on context-switching. This is useful to
wake up blocked threads.<br>
<br>
In case the receiving application is already running, it might not
make sense to issue any notification, as the app might already be in
the polling loop.<br>
<br>
On 03/16/2012 04:20 AM, Baumann Andrew wrote:
<blockquote
cite="mid:CB50AAD779D31B4C8B0D1D3866C291AC03D1C882@MBX11.d.ethz.ch"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas","serif";
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><br>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>* What additional primitives, if any, are desirable for existing versions of<o:p></o:p></pre>
<pre> Barrelfish to run well?<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Efficient inter-core messaging would be great! (more on this below)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>* What further primitives would Barrelfish developers like to see on <o:p></o:p></pre>
<pre>future hardware<o:p></o:p></pre>
<pre> platforms?<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Message-passing is the big one. We sketched out some ideas in this HotOS paper:<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.barrelfish.org/gap_hotos11.pdf">http://www.barrelfish.org/gap_hotos11.pdf</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Essentially we'd like a remote write operation that pushes some data close to a remote core, and an asynchronous notification that causes a lightweight exception/control transfer on the target. Previous versions of hardware-supported message passing have provided some version of the former, but neglected the latter.<o:p></o:p></pre>
<p class="MsoNormal">I agree, that sounds extremely useful;
implicit message passing by piggy-backing on the coherence
hardware isn't a great solution. The closest thing we have is
a broadcast update instruction that broadcasts a cache line
from the L2 cache of one cluster of cores into the L2 caches
of all other cores. We only needed broadcast when we
initially designed the architecture because we were focused on
a task-based programming model where arbitrary point-to-point
communication simply doesn't occur. For being able to run
more general classes of program, this generalization seems
very useful. Being able to pass an IPI along with the data
seems to give a similarly large benefit. For the use cases
you have in mind, is it important that the interrupt be
serviced immediately, or would it be sufficient to wait until
the target thread finishes what it's doing and does the moral
equivalent of calling yield()? If these IPIs are frequent,
having the servicing be cooperative rather than preemptive may
buy you efficiency by not having to spill the thread context
to memory (maybe when the target yield()s, everything in its
context is known to be dead).<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Tim
and Stefan should really answer this… they have gone into
much more depth than I on the notification mechanisms.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I
think it’s useful to decouple the data transfer from the
notification (or at least make the notification optional). I
don’t think it’s always necessary to deliver the
notification immediately, and coalescing multiple
undelivered notifications makes sense, but I don’t see the
value in a notification that requires an explicit action on
the receiver side… isn’t that just polling? Also, if I may
be so rude, the decision of what to do with thread context
and whether it must be spilled is a software problem :) You
can imagine a fault handler that knows enough about the
actual program being run to decide whether to spill context
and switch to a handler, or just set a flag that a
notification arrived and queue it for later processing.<o:p></o:p></span></p>
<pre><o:p> </o:p>
</pre>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Stefan</pre>
</body>
</html>