[Barrelfish-users] about port barrelfish on tilera platform

[Andrew Baumann]

Hi Robert,

> - I guess the CPU should be in the user mode / protection level, since 
> executing "resume" in exec.c. Is this function always called from the 
> user mode?

Yes. Everything in lib/barrelfish is user-mode code. The kernel gets to user-mode in one of three ways: return from a system call, resume() which is supposed to restore a full register context, and execute() which jumps to a user-mode entry point. For security reasons, execute() should zero out other registers (to avoid leaking state from the kernel or other domains). 

> - Should *all* registers from register_state be loaded? Is the content 
> of all registers arbitrary or can we have some assumptions (for example 
> on the THREAD_REGISTER or lr)? In the moment, we're not sure how to load 
> the program counter, after all other registers have a been restored, 
> without destroying the content of one of these registers. The possible 
> workaround for now would be to tell GCC to not use one register 
> (-ffixed-reg), and use this. TilePro has enough registers...

Yes. THREAD_REGISTER is the one possible exception to this, but it is a kludge that should go away (it already has on x86_64).

I would expect that any sane architecture would have a way to go from kernel mode to an arbitrary user-mode context (including PC), since otherwise I don't see how you can implement preemptive multitasking. If you can't figure out how to do this, then yes you'll probably need to pin a user-mode register.

> - In the x86 code you put registers into the __asm clobbered list. What 
> is the purpose of this?

Are you talking about the kernel's resume() or execute()? I don't see any clobber lists there. Perhaps we're not looking at the same thing?

> - What is the intended order of the arguments from disp_switch()? If I'm 
> not mistaken, x86 and ARM use different arguments to save to and load 
> from. Does "from" mean, the thread we are switching from or the location 
> to read the registers from?

disp_switch() is a bit special, because it relies on clobbering all the registers and getting GCC to emit the code to save/restore them. This is a little bit fragile -- in general it just needs to work like setjmp/longjmp.

from is the register storage location for the thread we're currently on, and switching away from (i.e. it's the out pointer); to is the complete register context of the thread we are switching to (it's the in pointer). I thought the header comment was pretty clear on this one:

/**
 * \brief Switch execution between two register states, and turn off
 * disabled activations.
 *
 * This function saves as much as necessary of the current register state
 * (which, when resumed will return to the caller), and switches execution
 * by resuming the given register state.  It may only be called while the
 * dispatcher is disabled.  A side effect is that activations are reenabled.
 * Note that the thread context saved is a voluntary save so only callee
 * save registers need to be saved, but we dont currently provide any
 * way to optimise the corresponding resume.
 *
 * \param disp Current dispatcher pointer
 * \param from_regs Location to save current register state
 * \param to_regs Location from which to resume new register state
 */

Cheers,
Andrew

-----Original Message-----
From: Robert Radkiewicz [mailto:rrad at kth.se] 
Sent: Friday, 30 November 2012 07:16
To: Andrew Baumann
Cc: barrelfish-users at lists.inf.ethz.ch
Subject: Re: [Barrelfish-users] about port barrelfish on tilera platform

Moin,

I have some follow-up questions for this, regarding 
lib/barrelfish/arch/<arch>/dispatch.c:

- I guess the CPU should be in the user mode / protection level, since 
executing "resume" in exec.c. Is this function always called from the 
user mode?
- Should *all* registers from register_state be loaded? Is the content 
of all registers arbitrary or can we have some assumptions (for example 
on the THREAD_REGISTER or lr)? In the moment, we're not sure how to load 
the program counter, after all other registers have a been restored, 
without destroying the content of one of these registers. The possible 
workaround for now would be to tell GCC to not use one register 
(-ffixed-reg), and use this. TilePro has enough registers...
- In the x86 code you put registers into the __asm clobbered list. What 
is the purpose of this?
- What is the intended order of the arguments from disp_switch()? If I'm 
not mistaken, x86 and ARM use different arguments to save to and load 
from. Does "from" mean, the thread we are switching from or the location 
to read the registers from?

Regards,
Robert

On 19.11.2012 20:02, Andrew Baumann wrote:
> Hi Robert,
>
> Start of day for userland is a pretty special environment on Barrelfish, and probably some of the toughest code to understand or port. The main thing you need to understand to make sense of it is the dispatch model: the crt0 code runs with the dispatcher "disabled", which means that it cannot take upcalls from the kernel. This in turn means that it cannot take a page fault, nor can it send or receive IDC messages, or use threading functionality such as mutexes. The normal printf() path uses IDC (e.g. to communicate with the serial driver), so it can't be called from this environment. Instead, the debugging facilities available boil down to a debugging system call sys_print(). You can see examples of this in lib/barrelfish/dispatcher.c for things like disp_assert_fail() etc. Also, beware that you have a very small stack at this point.
>
> Once you get onto a thread (i.e. once threads_init_disabled() completes and uses disp_resume() to start the thread) then most of these restrictions go away, and normal printf() calls should work.
>
> Hope this helps,
> Andrew
>