[Barrelfish-users] [Barrelfish] Threads, domains and their confusing API.

[Andrew Baumann]

Hi Georgios,

The current support for what we call "spanned" domains, where threads and dispatchers share an address space across multiple cores, is fairly patchy. In particular, the API to the code in domain.c results from a series of evolutions as people have put in hacks to support various benchmarks, and is far from complete. As you've noticed, it's also undocumented. Before building on top of this, my advice would be to understand what it is doing now (and feel free to ask questions if you don't), and don't be hesitant to change it if necessary.

As to what you're trying to do, I think some form of pthreads/nptl wrapper over the underlying mechanisms is a good goal. However, beware that there is nothing already present in Barrelfish which will automatically migrate threads between dispatchers: currently you need to do this explicitly by moving threads (or arranging for them to be in the right place to start with).

If you want to create a thread on another core, and get back a pointer to it, I think creating the thread structure on that core (ala domain_thread_create_on) makes sense. However, because this involves sending a message to another dispatcher, and then waiting for a reply with the thread handle, you'll either need to make remote thread creation a blocking RPC operation, or use a callback which will be called with the thread handle later when the reply message is received.

> On an other matter, are there any already-developed tools to get simple
> execution measurements like time, memory usage and core utilization?

Not yet, sorry. The benchmarks we've run have used either manually-instrumented cycle counts (rdtsc and friends) or our minimal existing support for processor performance counters.

Regards,
Andrew

> -----Original Message-----
> From: Georgios Varisteas [mailto:yorgos at kth.se]
> Sent: Thursday, 17 March, 2011 9:41
> To: barrelfish-users at lists.inf.ethz.ch
> Subject: [Barrelfish] [Barrelfish-users] Threads, domains and their confusing
> API.
> 
> Hi,
> 
> I'm doing research on shared memory user level scheduling for barrelfish and
> I'm trying to figure out the "correct way" of working with threads and
> domains in the latest release. So, I'm going to present what I have figured
> out thus far, expecting some further insight compensating for the lack of
> documentation (sorry if there actually is documentation lying around
> somewhere and I've missed it).
> 
> Taking it from the top, there is of course the threads library that replicates
> some basic functionality of pthreads. However, barrelfish has the extra
> notion of domains; hence the domain.c library contains wrappers for most
> but not all functions (call me join). Also, I have found a nice set of higher level
> wrapper functions in bomp that inspired some ideas.
> 
> What I require at this point is to replicate the exact API of Linux's nptl, as I
> have Linux code that I need to migrate. This means that I require
> pthread_create() and pthread_join() functions, with automatic scheduling of
> the created threads on the physical cores. I have written a layer that
> interfaces the pthread-based code with the bf threads library, also spanning
> the domain to use all cores. What I'm missing is a single function which will
> create a new thread on a specific core (dispatcher) and also return that
> thread structure. From what I've seen so far I can achieve this in two
> imperfect ways:
> 
> 1. Use thread_create_varstack() (threads.c) to create the thread on the
> same dispatcher and then move it with domain_thread_move_to()
> (domain.c). This method is simply one function call too much.
> 2. Use domain_thread_create_on() (domain.c) and somehow get the thread
> to return an instance of its descriptor structure with thread_self()
> (threads.c). This method is far too complicated and requires me to change
> the legacy Linux code which I desperately want to avoid.
> 
> I know that there has to be a straightforward way to achieving this with one
> simple function call and I am simply missing it. I could always alter the
> domain.c API to return the thread structure (as a return value or a void *
> parameter like in pthreads), but I thought of asking before changing
> anything. Any ideas? Thanks in advance.
> 
> On an other matter, are there any already-developed tools to get simple
> execution measurements like time, memory usage and core utilization?
> 
> Best regards,
> Georgios Varisteas
> _______________________________________________
> Barrelfish-users mailing list
> Barrelfish-users at lists.inf.ethz.ch
> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users
> 
> _______________________________________________
> Barrelfish mailing list
> Barrelfish at lists.inf.ethz.ch
> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish