[Barrelfish-users] [Barrelfish] Hagfish kernel image structure and latest version

David Cock david.cock at inf.ethz.ch
Thu Apr 15 19:28:09 CEST 2021 

I don't see anything that should cause a problem if you increase the 
value.  Did you do a clean build after changing it?  I don't quite trust 
the UEFI build system.  What size did you set it to?

David

On 15/4/21 7:20 pm, David Cock wrote:
>
> That's a bit odd.  It ought to be as simple as increasing the size of 
> the table - I don't think there's anything fundamental about it.  I'm 
> pretty sure I just wanted to avoid dynamic allocation at that point in 
> the code. Let me have a look at it.
>
> David
>
> On 15/4/21 6:08 pm, sebastian.scherbel at fau.de wrote:
>> Hi,
>>
>> >The memory map is 17952B, but MEM_MAP_SIZE is 8192.
>> > This is compile-time limit in Hagfish - please report this 
>> overflow, it's a bug.
>>
>> I have stumbled across a limitation in Hagfish. 8192B is not enough 
>> on a HPE Apollo 70 with 256 cores and 256 GB RAM and 374 entries of 
>> 48B each. I tried to just increase the value, but now I crash 
>> somewhere between exiting the UEFI boot services and jumping to my 
>> ELFs entrypoint. Without any output.
>>
>> Do you happen to have an educated guess what the problem could 
>> possibly be? I don't have a hardware debugger, unfortunately, which 
>> makes it somewhat challenging. And limits me to printk debugging.
>>
>> >We'd be happy to integrate a patch if you wanted to modify it.
>> I'll do that after i have finished my project.
>>
>> Regards Sebastian
>>
>>
>> Am Freitag, 12. März 2021, 12:08:39 MEZ hat David Cock 
>> <david.cock at inf.ethz.ch> Folgendes geschrieben:
>>
>>
>> Sebastian
>>
>> It looks like what's going on is as Daniel described.  Hagfish seems 
>> to use an API call (get_file_size) that's supported by EDK2, but not 
>> by many commercial offerings.  I think it'd almost certainly be 
>> possible to update Hagfish such that it doesn't depend on it - 
>> something like preallocating a "reasonably-sized" buffer, and if it 
>> turns out it wasn't big enough trying again with a larger one. It's 
>> been a few years since I wrote this stuff, but I think you'll get 
>> back an EFI_BUFFER_TOO_SMALL if it was too small, and *BufferSize 
>> will be filled with the actual size so that you can reallocate the 
>> buffer.  We'd be happy to integrate a patch if you wanted to modify it.
>>
>> David
>>
>> On 11/3/21 2:19 pm, Schwyn Daniel wrote:
>>
>> Hi Sebastian,
>>
>> David and I are both part of the Barrelfish core team at ETH so our 
>> setup is the same😉We do load Hagfish over PXE but as far as I know 
>> always directly with the PXE implementation in EDK2. Hagfish is 
>> theoretically designed to be flexible and cover as many EFI based 
>> boot environments as possible. In practice it’s only tested on the 
>> machines we have. Those are EDK2 based and the PXE implementation in 
>> there supports the two MTFTP opcodes we use. It should be possible to 
>> either implement support in iPXE or make Hagfish not use the 
>> get_file_size e.g. by allocating a large enough buffer and make sure 
>> the read_file doesn’t overflow it.
>>
>> Regards,
>>
>> Daniel
>>
>> *Von:*sebastian.scherbel at fau.de <mailto:sebastian.scherbel at fau.de> 
>> <sebastian.scherbel at fau.de> <mailto:sebastian.scherbel at fau.de>
>> *Gesendet:* Mittwoch, 10. März 2021 16:38
>> *An:* Schwyn Daniel <daniel.schwyn at inf.ethz.ch> 
>> <mailto:daniel.schwyn at inf.ethz.ch>
>> *Cc:* barrelfish-users at lists.inf.ethz.ch 
>> <mailto:barrelfish-users at lists.inf.ethz.ch>
>> *Betreff:* Re: AW: [Barrelfish-users] [Barrelfish] Hagfish kernel 
>> image structure and latest version
>>
>> Hi Daniel,
>>
>> thank you for your reply. That's what I already suspected.
>>
>> I'm waiting to hear back from David about his setup. He seemed quite 
>> sure that chainloading works, so I was a bit surprised that neither 
>> iPXE nor Grub did work for me on EDK2.
>>
>> Otherwise, I'll fix iPXE myself.
>>
>> Regards Sebastian
>>
>> Am Mittwoch, 10. März 2021, 10:59:48 MEZ hat Schwyn Daniel 
>> <daniel.schwyn at inf.ethz.ch <mailto:daniel.schwyn at inf.ethz.ch>> 
>> Folgendes geschrieben:
>>
>> Hi Sebastian,
>>
>> I have seen this error before with iPXE when I tried chainloading 
>> Hagfish with iPXE on top of U-Boot with EFI support. U-Boot’s EFI 
>> support is rather minimalistic so I’m not sure the cause was the same 
>> in my case. Looking at the iPXE source code though it looks to me as 
>> if it only supports the EFI_PXE_BASE_CODE_MTFTP_READ_FILE opcode 
>> while Hagfish also needs the EFI_PXE_BASE_CODE_MTFTP_GET_FILE_SIZE 
>> operation to allocate appropriately sized buffers for the files it 
>> wants to load.
>>
>> Regards,
>>
>> Daniel
>>
>> *Von:*Barrelfish-users <barrelfish-users-bounces at lists.inf.ethz.ch 
>> <mailto:barrelfish-users-bounces at lists.inf.ethz.ch>> *Im Auftrag von 
>> *sebastian.scherbel at fau.de <mailto:sebastian.scherbel at fau.de>
>> *Gesendet:* Dienstag, 2. März 2021 22:56
>> *An:* Cock David <david.cock at inf.ethz.ch <mailto:david.cock at inf.ethz.ch>>
>> *Cc:* barrelfish-users at lists.inf.ethz.ch 
>> <mailto:barrelfish-users at lists.inf.ethz.ch>
>> *Betreff:* Re: [Barrelfish-users] [Barrelfish] Hagfish kernel image 
>> structure and latest version
>>
>> Hello,
>>
>> >Hagfish definitely supports chainloading over PXE - that's exactly what it was designed 
>> for.
>>
>> Which bootloaders have you tested? What does your setup look like?
>>
>> I have tried:
>>
>> EDK2 -> iPXE -> Hagfish
>>
>> /iPXE> chain tftp://x.x.x.x/Hagfish_ETHZ.efi/
>>
>> /...
>> Hagfish UEFI loader starting
>> UEFI vendor: EDK II
>> Hagfish loaded at 402D9000, size 147456B, by handle 41839B98
>> Could not connect to shell or not enough parameters, assuming PXE boot.
>> Connecting to the PXE service that loaded me.
>> PXE loader at 403ED660, revision 10000, running
>> Loading "hagfish.cfg.10.0.0.10"
>> *Mtftp: Unsupported,
>> file size: Unsupported*/
>>
>> EDK2 -> Grub2 -> Hagfish
>>
>> /grub> boot
>> Status: 0x0000000E
>> Failed to initialize ShellLib, aborting.
>> Hagfish UEFI loader starting
>> UEFI vendor: EDK II
>> Hagfish loaded at 402DA000, size 147456B, by handle 41862A18
>> Could not connect to shell or not enough parameters, assuming PXE boot.
>> Connecting to the PXE service that loaded me.
>> *OpenProtocol: Unsupported
>> Failed to initialize loader: Load Error*/
>>
>> > The error there seems to be the UEFI implementation not supporting TFTP - did you 
>> compile it in?
>>
>> When I load Hagfish directly, everything works fine. On American 
>> Megatrends Aptio V and EDK2.
>>
>> Regards Sebastian
>>
>> Am Donnerstag, 28. Januar 2021, 16:36:02 MEZ hat David Cock 
>> <david.cock at inf.ethz.ch <mailto:david.cock at inf.ethz.ch>> Folgendes 
>> geschrieben:
>>
>> What you're seeing there is Hagfish trying to load its configuration 
>> file over TFTP as it's figured out that it was itself loaded over 
>> PXE. The README might be inconsistent wrt filenames - if in doubt, 
>> trust the code. Hagfish definitely supports chainloading over PXE - 
>> that's exactly what it was designed for.
>>
>> The error there seems to be the UEFI implementation not supporting 
>> TFTP - did you compile it in?
>>
>> David
>>
>> On 28/1/21 3:53 pm, sebastian.scherbel at fau.de 
>> <mailto:sebastian.scherbel at fau.de> wrote:
>>
>> Hello,
>>
>> sounds good. I have tried to chainload Hagfish after iPXE. However, 
>> loading of the cfg fails on QEMU and on the real hardware with the 
>> same error.
>>
>> /Hagfish UEFI loader starting
>> UEFI vendor: American Megatrends
>> Hagfish loaded at F951D000, size 143360B, by handle F6EA1D18
>> Could not connect to shell or not enough parameters, assuming PXE boot.
>> Connecting to the PXE service that loaded me.
>> PXE loader at F5EE1840, revision 10000, running
>> Loading "hagfish.cfg.A.B.C.D"*(shouldn't the cfg name be 
>> hafish.A.B.C.D.cfg according to the readme?)*
>> *Mtftp: Unsupported, */
>>
>> */file size: Unsupported/*
>>
>> Is this a known bug? Is chainloading unsupported? Your setup is 
>> directly booting into Hagfish?
>>
>> Regards Sebastian
>>
>> Am Donnerstag, 28. Januar 2021, 14:03:36 MEZ hat David Cock 
>> <david.cock at inf.ethz.ch> <mailto:david.cock at inf.ethz.ch> Folgendes 
>> geschrieben:
>>
>> Sebastian,
>>
>> Hagfish is a custom loader for the Barrelfish CPU driver, which 
>> operates under a few unusual assumptions: Mainly, the CPU driver 
>> (kernel) itself is not self-relocating, but relies on the bootloader 
>> (Hagfish) to do the relocation for it.
>>
>> It would definitely be possible to modify Hagfish to load an 
>> arbitrary ELF, however. You'd have to make a few changes in the 
>> relocation code to not look for Barrelfish-specific segments and so 
>> on, but there's nothing particularly weird or non-standard about it.
>>
>> Hagfish is pretty static, and doesn't change much from version to 
>> version.  We'll update the repository shortly (which we should have 
>> done together with the release).  Feel free to ask more specific 
>> questions if you decide to use it - we're happy to help.
>>
>> David
>>
>> On 27/1/21 5:41 pm, sebastian.scherbel at fau.de 
>> <mailto:sebastian.scherbel at fau.de> wrote:
>>
>> Hello,
>>
>> I'm a CS student trying to port an research operating system from 
>> x86_64 to ARMv8 and stumbled across Hagfish.
>>
>> Hagfish should basically be able to load any ELF with its sections 
>> into memory with minor modifications? Or is there certain requirement 
>> for the kernel image structure that I didn't notice while reading 
>> your publication [1] and skimming the source code?
>>
>> The last commit in the public repository is from March 2017. But in 
>> the changelog of your last Barrelfish release from 2020 states 
>> "Update Hagfish to latest version". Have there been any bigger 
>> changes or improvements in the meantime?
>>
>> Regards Sebastian
>>
>> [0] 
>> https://lists.inf.ethz.ch/pipermail/barrelfish-users/2020-March/001597.html 
>> <https://lists.inf.ethz.ch/pipermail/barrelfish-users/2020-March/001597.html>
>>
>> [1] http://www.barrelfish.org/publications/TN-022-ARMv8.pdf 
>> <http://www.barrelfish.org/publications/TN-022-ARMv8.pdf>
>>
>> _______________________________________________
>>
>> Barrelfish-users mailing list
>>
>> Barrelfish-users at lists.inf.ethz.ch 
>> <mailto:Barrelfish-users at lists.inf.ethz.ch>
>>
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users 
>> <https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users>
>>
>> _______________________________________________
>> Barrelfish-users mailing list
>> Barrelfish-users at lists.inf.ethz.ch 
>> <mailto:Barrelfish-users at lists.inf.ethz.ch>
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users 
>> <https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users>
>>
>> _______________________________________________
>>
>> Barrelfish-users mailing list
>>
>> Barrelfish-users at lists.inf.ethz.ch 
>> <mailto:Barrelfish-users at lists.inf.ethz.ch>
>>
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users 
>> <https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users>
>>
>> _______________________________________________
>> Barrelfish-users mailing list
>> Barrelfish-users at lists.inf.ethz.ch 
>> <mailto:Barrelfish-users at lists.inf.ethz.ch>
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users 
>> <https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users>
>>
>>
>> _______________________________________________
>> Barrelfish-users mailing list
>> Barrelfish-users at lists.inf.ethz.ch  <mailto:Barrelfish-users at lists.inf.ethz.ch>
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users  <https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users>
>>
>> _______________________________________________
>> Barrelfish-users mailing list
>> Barrelfish-users at lists.inf.ethz.ch
>> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users
>
> _______________________________________________
> Barrelfish-users mailing list
> Barrelfish-users at lists.inf.ethz.ch
> https://lists.inf.ethz.ch/mailman/listinfo/barrelfish-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.inf.ethz.ch/pipermail/barrelfish-users/attachments/20210415/b5d675ef/attachment-0001.html>

More information about the Barrelfish-users mailing list