[Barrelfish-users] Porting Barrelfish to a new architecture?

[Timothy Roscoe]

> Actually the base Barrelfish system (by design) really does not
> require any synchronisation primitives. We use spinlocks at
> user-level for applications that span a single address space across
> multiple cores, and in the current codebase these spinlocks are also
> used at user-level in the single-core case, but that’s a bug /
> missing performance optimisation to elide them. Other synchronisation
> takes place via messages.

Just to expand on this - each Barrelfish CPU driver is a 
single-threaded, non-preemptable event driven kernel which shares no 
memory or data structures with any other cores.  Hence we don't need any 
shared-memory synchronization primitives, other than the usual 
uniprocessor uses (DMA transfers, booting, etc.).

Above this, all the OS services are also written to avoid shared memory, 
with the exception of the inter-core message transport on cache-coherent 
machines, which uses a shared ring buffer.

Applications can share memory (if the hardware supports it), and as 
Andrew points out, it's common to migrate threads within a coherence 
domain or share page tables.  However, these are all application data 
structures (even hardware page tables), and are not maintained by the 
OS.  Within such an application there is, therefore, a need for 
shared-memory synchronization.

> What sources of interrupt must an architecture support? Timer?
> Off-chip? Another core on the same chip? Some combination?
>
> Preferably a per-core countdown timer for scheduling, like the x86
> APIC timer. On Beehive we used a kludge based on messages sent from a
> central “timer” core.

We require a way for a core to request an interrupt from a timer in the 
future - e.g. the APIC timer or a timer core in Beehive.  While the CPU 
driver is non-preemptable, it handles interrupts and uses the timer IRQ 
to preempt domains running above it.   The CPU driver also fields 
interrupts from peripheral devices and turns these into local messages 
to driver domains (which execute at user level).

We try to avoid IPIs since they're expensive on much hardware, but if a 
particular inter-core messaging system (an "interconnect driver" in 
Barrelfish parlance) is polled and has no natural notification 
mechanism, as is the case with cache-coherent shared memory, then we can 
also wrap IPIs up in a "notification driver".   We also do this on the SCC.

> * At a high level, is this worth doing?    There are two parts to
> that:
> - If the port is successful, is there any hope of upstreaming the
> changes so
> they can keep pace with upstream API changes?    (Remembering that
> the platform
> in question consists of a simulator, and possibly an FPGA in the
> foreseeable future)
>
> I can't give you a definitive answer (Mothy is perhaps better placed
> to do so), but there have been outside contributions to the tree. I
> think the main issue is one of maintenance bandwidth -- whether there
> are folks around who care about the port and are using/maintaining
> it. We recently removed the Beehive port from the tree, because it
> was unused, and we didn't have the cycles to keep it from
> bit-rotting.

We're very happy to take external contributions (both KTH and BSC have 
contributed valuable code and fixes, and continue to do so), and we're 
also happy to provide feedback and advice on supporting a new 
architecture.  However, the burden of keeping it up to date would have 
to fall on you guys - we run automated tests on x86, SCC, and ARM, and 
make sure Barrelfish runs fine on these systems, but it's a lot of 
effort for us to add other architectures, particularly without access to 
the hardware (whether emulated or otherwise) and particularly if it's 
radically different from more "conventional" processors.

The Beehive port was quite successful, and the OS architecture 
definitely accomodated it pretty well, despite the lack of interrupts or 
memory protection, but it was a lot of work to keep up with the latest 
toolchain and simulator from MSR (no reflection on them - they were very 
helpful), and there was little external usage of the port, and so we 
dropped support in the end (we're just a research group, after all!).

> If you're looking for reference code, the x86_64 port is the most
> complete, but the ARM port (done by Orion Hodson at MSR) is much
> cleaner.

We're about to put a lot more effort into the ARM side of things at ETHZ 
as well, so this might help with any other architectures we pick up on 
the way.

Best regards,

  -- Mothy