[Barrelfish-users] About span domain and Vspace

[Simon Gerber]

Dear Sun,

Thank you for your interest in Barrelfish!

On 13.07.2017 08:40, 孙思杰 wrote:
> Dear all,
> I'm trying to run xmpl-span example over qemu.
> It works on x86_64. but on armv7 reports a WRONG_CAP_TYPE error which is resulted from usr/monitor/monitor_server.c:827.

Currently, our domain spanning implementation is only supported on x86_64.

> After I read all code about the span function, there is something about Vspace I don't quite understand.
> 1. After span a dispatcher on another core, the two dispatchers share the same pagetable but not the same vspace. The new dispatcher has a fresh vregion list and is not able to know which parts of virtual address space is already used. On armv7, it leads to allocating an allocated vaddr after first malloc and crash. It works on x86_64 because in function pmap_current_init, the min_mappable_va and META_DATA_RESERVED_BASE is changed according to core id. Does that mean that virtual address space is statically partitioned on x86_64?

Yes, we currently statically partition the virtual address space on x86_64 in order to support the domain spanning implementation. This is impractical on armv7, because we only have a 2GB of virtual address space available for an application to work with. Even on x86_64 it is not the preferred solution, because it does not easily allow the allocation of really large (>512GB) virtual regions.

> 2. On Barrelfish, everything between cores has to be replicated. Why new dispatch created by span function share the pagetable? Is it necessary to keep consistent between two dispatcher's vregion list. If it is, is there any mechanism to do so?

It is perfectly acceptable to share data between cores on Barrelfish. The dispatchers created by span share page tables, because they are intended to be part of a single "domain" -- a "process" in other systems -- and should be able to access shared data structures in memory. Unfortunately our implementation of span does not keep the dispatchers' replicated(!) vregion lists, or pmaps, consistent and should be treated as a incomplete proof-of-concept.

There are a few design choices to keep in mind for spanned domains. One option is to have libbarrelfish use shared memory data structures for parts that need to be consistent across cores (mainly the memory management parts).
Another option is to not share page tables by default, this however would need some 
serious rewriting of the spanning logic itself, and would make the span operation quite slow, as you would have to clone a number of page tables to get the new dispatcher up and running.
A third option, which I think is the best choice, is to keep the shared page tables, and implement synchronization of vregion lists and pmaps between libbarrelfish instances on different dispatchers of the same domain. In this case, one would have to choose an agreement protocol -- this can be as simple as choosing one dispatcher to be the master, as our message channels guarantee in-order, reliable delivery of messages -- and use this protocol to keep vregion lists and pmaps -- and probably other data structures -- of all the dispatchers of a domain in a consistent state.

In conclusion: we do have a simple proof-of-concept implementation of domain spanning on x86_64, but it has some serious issues and should not be regarded as ready to use. I'm not sure how much work it would be to make the current implementation work on armv7, especially in light of its inherent problems.

Best,
-- Simon

> 
> Regards,
> Sun SJ
> 
> 
> _______________________________________________
> Barrelfish-users mailing list
> Barrelfish-users at lists.inf.ethz.ch
> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users
>