<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 17, 2017 at 8:31 AM, George Rimar <span dir="ltr"><<a href="mailto:grimar@accesssoftek.com" target="_blank">grimar@accesssoftek.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<div style="color:rgb(33,33,33)">
<div>
<div dir="ltr">
<div class="gmail_extra"><span class="gmail-">
<div class="gmail_quote"><span style="font-size:12pt;font-family:calibri,arial,helvetica,sans-serif">></span><span style="font-size:12pt;font-family:calibri,arial,helvetica,sans-serif">>That boot_params.hdr.code32_start </span><span style="font-size:12pt;font-family:calibri,arial,helvetica,sans-serif">f<wbr>ield
is probably either invalid (bad reloc or something else causing the bootloader to >>calculate the wrong address) or valid but the thing it thinks it is pointing to wasn't loaded (missing PT_LOAD etc.). </span><br>
</div>
</span><div class="gmail_quote"><span class="gmail-">
<div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span class="gmail-m_-7001478236934095985gmail-"></span>
<div style="color:rgb(33,33,33)"><span style="color:rgb(33,33,33);font-family:calibri,arial,helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)">>boot_params.hdr.code32_start <wbr>field</span> is valid :) It is 0x100000, like expected</div>
</div>
</div>
</div>
</div>
</div>
<div>> </div>
<div>>Then I suspect that that segment isn't being loaded. Is there a PT_LOAD that covers that address? Is the bootloader loading it?</div>
<div> <br>
</div>
</span><div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div style="color:rgb(33,33,33)">That issue is gone. Not sure what changed, but looks something was fixed in LLD during last week.<br>
</div>
<div style="color:rgb(33,33,33)">Latest status of booting linux kernel is next currently:<br>
</div>
<div style="color:rgb(33,33,33)"><br>
</div>
<div style="color:rgb(33,33,33)">At this location,<br>
</div>
<div style="color:rgb(33,33,33)"><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/head_64.S#L424" target="_blank">https://github.com/torvalds/<wbr>linux/blob/<wbr>5924bbecd0267d87c24110cbe2041b<wbr>5075173a25/arch/x86/boot/<wbr>compressed/head_64.S#L424</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div class="gmail_extra">kernel calls extract_kernel(). And 2 lines below it tries to jmp to the address of decompressed kernel and fails to do that for me.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><span style="color:rgb(33,33,33);font-family:calibri,arial,helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)">extract_kernel</span>() method is:<br>
</div>
<div class="gmail_extra"><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L334" target="_blank">https://github.com/torvalds/<wbr>linux/blob/<wbr>5924bbecd0267d87c24110cbe2041b<wbr>5075173a25/arch/x86/boot/<wbr>compressed/misc.c#L334</a><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">I added next lines before return:<br>
</div>
<div class="gmail_extra">
<div>__putstr("hi from extract_kernel");</div>
<div>if ((int) output == 0x1000000)<br>
</div>
<div> __putstr("== 0x1000000");<br>
</div>
<div>output[0] = 0xEB;<br>
</div>
<div>output[1] = 0xFE;<br>
</div>
<div><span class="gmail-m_-7001478236934095985pl-k" style="box-sizing:border-box;color:rgb(167,29,93);font-family:sfmono-regular,consolas,"liberation mono",menlo,courier,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">return</span><span style="color:rgb(51,51,51);font-family:sfmono-regular,consolas,"liberation mono",menlo,courier,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">
output;</span><br>
</div>
<div><br>
</div>
<div>And during boot in shows all text from above and enters infinite loop as expected. So, that means it successfully jumps to 0x1000000, but looks something is wrong in decompressed code. <span style="font-size:12pt">Next destination point should be </span><span style="color:rgb(121,93,163);font-family:sfmono-regular,consolas,"liberation mono",menlo,courier,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(248,238,199)">startup_64</span><span style="font-size:12pt">.</span></div>
<div><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/kernel/head_64.S#L48" target="_blank">https://github.com/torvalds/<wbr>linux/blob/<wbr>5924bbecd0267d87c24110cbe2041b<wbr>5075173a25/arch/x86/kernel/<wbr>head_64.S#L50</a><br>
</div>
Though as I mentioned it does not reach it for me.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Next step I probably going to do is dump/printf-trace that memory area of decompressed kernel to compare with what produced there when BFD is used. Have no better ideas now.</div></div></div></div></div></blockquote><div><br></div><div>Based on your other post and some googling, it seems like there are two separate kernel binaries. One is a "loader" binary that does the decompression and some simple dynamic loader/linker type behavior and the "real" kernel binary.</div><div><br></div><div>It seems based on your description that the "loader" binary is running fine. The bug is not likely to be corrupted data in the decompressed output (that is just calling into a gzip routine or something). You shouldn't have to dump/printf-trace from memory during boot to see that data since the "real" kernel binary that is being decompressed into that memory region is probably already somewhere in your build tree (arch/x86/boot/compressed/Makefile seems like it has the details; I think it is the file called `vmlinux`; also see this article with more info about all the pieces: <a href="https://en.wikipedia.org/wiki/Vmlinux">https://en.wikipedia.org/wiki/Vmlinux</a>). It should be possible to do some static analysis (ELF header, phdrs, and symbols/relocations) on the "real" kernel binary directly.</div><div><br></div><div><br></div><div><br></div><div>One place to start doing dynamic analysis is these two lines: <a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L407">https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L407</a></div><div><br></div><div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>parse_elf(output);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>handle_relocations(output, output_len, virt_addr);</div></div><div><br></div><div><br></div><div>parse_elf:</div><div><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L268">https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L268</a><br></div><div>This seems to implement a bare-bones loader:</div><div><br></div><div>``</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>for (i = 0; i < ehdr.e_phnum; i++) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>phdr = &phdrs[i];</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>switch (phdr->p_type) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>case PT_LOAD:</div><div>#ifdef CONFIG_RELOCATABLE</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>dest = output;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>dest += (phdr->p_paddr - LOAD_PHYSICAL_ADDR);</div><div>#else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>dest = (void *)(phdr->p_paddr);</div><div>#endif</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memmove(dest, output + phdr->p_offset, phdr->p_filesz);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>break;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>default: /* Ignore other PT_* */ break;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}`</div><div>```</div><div><br></div><div>Based on the fact that you are hitting the EB FE at the load address, this seems like it is working fine (though if there are multiple PT_LOAD's there may still be issues lurking with the other ones).</div><div><br></div><div><br></div><div>handle_relocations:</div><div><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L173">https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L173</a><br></div><div><br></div><div>Okay, this one looks likely to be where the issues are manifesting.</div><div><br></div><div>First off, it seems like they have some sort of homegrown relocation scheme:</div><div><a href="https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L211">https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/arch/x86/boot/compressed/misc.c#L211</a><br></div><div><br></div><div>```</div><div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/*</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * Process relocations: 32 bit relocations first then 64 bit after.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * Three sets of binary relocations are added to the end of the kernel</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * before compression. Each relocation table entry is the kernel</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * address of the location which needs to be updated stored as a</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 32-bit value which is sign extended to 64 bits.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> *</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * Format is:</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> *</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * kernel bits...</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 0 - zero terminator for 64 bit relocations</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 64 bit relocation repeated</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 0 - zero terminator for inverse 32 bit relocations</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 32 bit inverse relocation repeated</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 0 - zero terminator for 32 bit relocations</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * 32 bit relocation repeated</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> *</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> * So we work backwards from the end of the decompressed image.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> */</div></div><div>```</div><div><br></div><div>Grepping around, it seems like they build this list of relocations based on some sort of homegrown tooling in arch/x86/tools/. E.g. look at arch/x86/tools/relocs.c. The main exported function from there (modulo being compiled twice in 32/64 mode) is:</div><div>```</div><div><div>#if ELF_BITS == 64</div><div># define process process_64</div><div>#else</div><div># define process process_32</div><div>#endif </div></div><div><br></div><div><div><div>void process(FILE *fp, int use_real_mode, int as_text,</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> int show_absolute_syms, int show_absolute_relocs,</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span> int show_reloc_info)</div><div>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>regex_init(use_real_mode);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>read_ehdr(fp);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>read_shdrs(fp);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>read_strtabs(fp);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>read_symtabs(fp);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>read_relocs(fp);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (ELF_BITS == 64)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>percpu_init();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (show_absolute_syms) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>print_absolute_symbols();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>return;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (show_absolute_relocs) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>print_absolute_relocs();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>return;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>if (show_reloc_info) {</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>print_reloc_info();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>return;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>emit_relocs(as_text, use_real_mode);</div><div>}</div></div></div><div>```</div><div><br></div><div>There may be something that this program doesn't like about the LLD-linked "real" kernel executable. If regular static analysis of the "real" kernel binaries doesn't reveal anything wrong (I'm guessing it will though and is probably the best first thing to look at), you can try tracing the execution of this program on the LLD-linked executable and the BFD/gold executable and diff the results. arch/x86/boot/compressed/Makefile seems to have some details about the invocation of this `relocs` program.</div><div><br></div><div><br></div><div><br></div><div>Also, down the road once we have your current configuration working, it is probably worth toggling each of these kconfig options as they may tickle different LLD issues:</div><div>CONFIG_X86_NEED_RELOCS</div><div>CONFIG_RELOCATABLE<br></div><div>(this is just from a quick glance at the ifdef's in the compressed boot loader code)</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif"><div style="color:rgb(33,33,33)"><div><div dir="ltr"><div class="gmail_extra"><span class="gmail-HOEnZb"><font color="#888888"><br>
</font></span></div><span class="gmail-HOEnZb"><font color="#888888">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">George.<br>
</div>
</font></span></div>
</div>
</div>
</div>
</blockquote></div><br></div></div>