[Barrelfish-users] [Barrelfish] Messages in Barrelfish

[Andrew Baumann]

Hi again,

> I would better try the multi-threaded version, so I created one extra thread
> per core just dispatching messages from the default waitset, as you
> suggested. However, in this case the problem doesn't get solved. The
> transmit queue fills and there is no more space after 32 messages. I'm
> wondering if I'm waiting in the correct waitset. You said there is an incoming
> waitset? Is it different from the default one?

Unless you're doing anything special to create your own waitset, the default waitset is the correct one. However, the support for using the low-level IDC primitives in multi-threaded domains is a bit lacking, so there are two places you can trip up:

1. The IDC binding objects are not thread-safe. You need to ensure that you do not race between the thread trying to send messages and the thread that is handling waitset events (some of which will call into the binding for message delivery). For concurrency control on bindings, you could use anything like mutexes, semaphores, etc. Note that there is also something unusual called an event_mutex  (lib/barrelfish/event_mutex.c) which is designed for mutual exclusion between both threads (with a blocking lock acquire) and event handlers (which takes a closure to be fired when the mutex is held).

2. There is no blocking send primitive: a TX_BUSY error is still possible, and must be handled by blocking the sending thread until there is space. You could do this by having it register a continuation (with register_send) and then block on a condition variable which will be signalled by the continuation.

Again, this is all very fiddly and easy to get wrong, so I'd encourage you to consider using THC instead.

Cheers,
Andrew