[Barrelfish-users] Flux control in Barrelfish

[Baumann Andrew]

Oh by the way, I should add to this that if you need to synchronize between two waitsets, you can enqueue an event rather than try to block/unblock threads directly. For example, if you need to wake up a thread that is blocked on waitset X from an event handler on waitset Y, you can trigger an event on X. There are some examples of this in lib/barrelfish/domain.c, although I'm not entirely sure you would need to do it.
Andrew
From: Baumann Andrew [mailto:andrewb at inf.ethz.ch]
Sent: Wednesday, 21 December, 2011 9:43
To: Zeus Gómez Marmolejo
Cc: barrelfish-users at lists.inf.ethz.ch
Subject: Re: [Barrelfish-users] Flux control in Barrelfish

Hi Zeus,

No, you cannot directly "put an event back" on the waitset without knowing which channel it came from, but even if you did, it's not clear to me that this would be useful - an event is just a closure, so until you execute it you really don't know what it's going to do.

I'm not 100% clear on your problem, but when/if you want to handle messages from C is just a question of when/if you service the waitset containing its channel.

By the way, there's no fundamental need to have two threads just because you have two waitsets. You could have one thread that used some application-specific policy to decide when it was going to service waitset A and when it was going to service waitset B using a combination of event_dispatch(), get_next_event() and/or check_for_event() on the two waitsets. If you're using check_for_event(), be careful not to create a busy-waiting loop!

Cheers,
Andrew

From: zeus at aluzina.org<mailto:zeus at aluzina.org> [mailto:zeus at aluzina.org]<mailto:[mailto:zeus at aluzina.org]> On Behalf Of Zeus Gómez Marmolejo
Sent: Wednesday, 21 December, 2011 9:21
To: Baumann Andrew
Cc: barrelfish-users at lists.inf.ethz.ch<mailto:barrelfish-users at lists.inf.ethz.ch>
Subject: Re: [Barrelfish-users] Flux control in Barrelfish

Hi Andrew,

Thank you for your suggestion. Two channels per connection is a very useful idea. I was thinking how to solve it, and I came up that it could be solved if the two channels are served using different waitsets. Then the receiving waitset on B can block if the transmitting channel on B is busy. This way, ACKs could be served anyway on the transmitting channel. But to do that, you need 2 threads per core. Because you need to block the handler executed on B so the ACK is not sent back to A and prevent him from sending more messages.

This has a drawback. What happens if while the receiving waitset on B is blocked, there is another node named C that wants to send a message to B. It's not being served because the receiving waitset on B is blocked to avoid the ACK back to A. So no more nodes can communicate to B.

Is there a way to send back an event to the waitset if you cannot process it in this moment?


Thanks again for your advise,

Zeus.

El 20 de diciembre de 2011 19:44, Baumann Andrew <andrewb at inf.ethz.ch<mailto:andrewb at inf.ethz.ch>> escribió:
Hi Zeus,

The first answer I would give is to make sure you handle the incoming messages on core A, but I understand that you may not have control over this.

It sounds like you've run into the problem that the two halves of a UMP channel are not independent: you cannot keep sending messages only in one direction without handling messages in the reverse direction, because the ACKs get piggy-backed on messages in the same channel. This is a nice optimisation for many use-cases, but it's unfortunate that it gets exposed to the user in this way, and it should really be fixed in the UMP implementation. The workaround THC currently uses is to have two underlying channels for each connection: one for each direction. This isolates sending and receiving and gives you true independent flow control, at the expense of some resource overhead for the extra channel.

BTW, Note that using more threads doesn't change anything: the queuing problem is on the channel structure itself.

Cheers,
Andrew

From: Zeus Gómez Marmolejo [mailto:zeus.gomez at bsc.es<mailto:zeus.gomez at bsc.es>]
Sent: Tuesday, 20 December, 2011 10:29
To: barrelfish-users at lists.inf.ethz.ch<mailto:barrelfish-users at lists.inf.ethz.ch>
Subject: [Barrelfish-users] Flux control in Barrelfish

Hi,

I'd like to ask you a question. Suppose we have core A always sending messages to core B. And core B is replying all messages back to core A. Core A never sends a new message before the previous one has not been sent (via continuation closure). But, reply message that B sends to A is much bigger than the one A sends to B. So when the 2nd reply message is to be sent from core B it always gets the FLOUNDER_ERR_TX_BUSY.

So my question is if there is a way to do a message flux control between the two cores. I understand that core B cannot send back a message if the previous reply has not been completely sent. But on the other side, the message handler cannot block and wait, as all the message processing stalls. I also tried to queue all replies using 2 threads to not block the message processing thread, sending them back when the previous one has been sent, but finally my reply queue ends up filling completely.

So what I want is that core A stop sending so many messages controlling the sending flux. Is there a way to do it with flounder? Which ways do you recommend me to do this?


Many thanks!!

--
Zeus Gómez Marmolejo
Barcelona Supercomputing Center
PhD student
http://www.bsc.es



--
Zeus Gómez Marmolejo
Barcelona Supercomputing Center
PhD student
http://www.bsc.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.inf.ethz.ch/pipermail/barrelfish-users/attachments/20111221/a58a32a3/attachment.html