<div dir="ltr">You should probably pull in some folks who implemented/maintain the feature for Darwin.<br><br>I guess they aren't linking this info, but only communicating in the object file between tools - maybe they flag these sections (either in the object, or by the linker) as ignored/dropped during linking. That semantic could be implemented in ELF too by marking the sections SHF_IGNORED or something (same-file split DWARF uses this technique).<br><br>So maybe the goal/desire is to have a different semantic, rather than the equivalent semantic being different on ELF compared to MachO. <br><br>So if it's a different semantic - yeah, I'd guess a flag that prefixes the module metadata with a length would make sense, then it can be linked naturally on any platform. (if the "don't link these sections" support on Darwin is done by the linker hardcoding the section name - then maybe this flag would also put the data in a different section that isn't linker stripped on Darwin, so users interested in getting everything linked together can do so on any platform)<br><br>But if this data is linked, then it'd be hard to know which command line goes with which module, yes? So maybe it'd make sense then to have the command line as a header before the module, in the same section. So they're kept together.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 27, 2020 at 10:26 PM Mircea Trofin via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks, Sean, Steven,<div><br></div><div>to explore this a bit further, are there currently users for non-Darwin cases? I wonder if it would it be an issue if we inserted markers in the section (maybe as an opt-in, if there were users), such that, when concatenated, the resulting section would be self-describing, for a specialized reader, of course - basically, achieve what Sean described, but "by design". </div><div><br></div><div>For instance, each .o file could have a size, followed by the payload (maybe include in the payload the name of the module, too; maybe compress it, too). Same for the .llvmcmd case. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 27, 2020 at 6:57 PM Sean Bartell <<a href="mailto:smbarte2@illinois.edu" target="_blank">smbarte2@illinois.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u><div><div>Hi Mircea,<br></div><div><br></div><div>If you use an ordinary linker that concatenates .llvmbc sections, you can use this code to get the size of each bitcode module. As far as I know, there's no clean way to separate the .llvmcmd sections without making assumptions about what options were used.<br></div><div><br></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">// Given a bitcode file followed by garbage, get the size of the actual</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">// bitcode. This only works correctly with some kinds of garbage (in</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">// particular, it will work if the bitcode file is followed by zeros, or if</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">// it's followed by another bitcode file).</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">size_t GetBitcodeSize(MemoryBufferRef Buffer) {</span><br></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> const unsigned char *BufPtr =</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> reinterpret_cast<const unsigned char *>(Buffer.getBufferStart());</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> const unsigned char *EndBufPtr =</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> reinterpret_cast<const unsigned char *>(Buffer.getBufferEnd());</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> if (isBitcodeWrapper(BufPtr, EndBufPtr)) {</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> const unsigned char *FixedBufPtr = BufPtr;</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> if (SkipBitcodeWrapperHeader(FixedBufPtr, EndBufPtr, true))</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> report_fatal_error("Invalid bitcode wrapper");</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> return EndBufPtr - BufPtr;</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> }</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> if (!isRawBitcode(BufPtr, EndBufPtr))</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> report_fatal_error("Invalid magic bytes; not a bitcode file?");</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> BitstreamCursor Reader(Buffer);</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> Reader.Read(32); // skip signature</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> while (true) {</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> size_t EntryStart = Reader.getCurrentByteNo();</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> BitstreamEntry Entry =</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> Reader.advance(BitstreamCursor::AF_DontAutoprocessAbbrevs);</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> if (Entry.Kind == BitstreamEntry::SubBlock) {</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> if (Reader.SkipBlock())</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> report_fatal_error("Invalid bitcode file");</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> } else {</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> // We must have reached the end of the module.</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> return EntryStart;</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> }</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif"> }</span><span style="font-family:menlo,consolas,monospace,sans-serif"><br></span></div><div><span style="font-family:menlo,consolas,monospace,sans-serif">}</span><br></div><div><br></div><div>Sean<br></div><div><br></div><div>On Thu, Aug 27, 2020, at 13:17, Steven Wu via llvm-dev wrote:<br></div><blockquote type="cite" id="gmail-m_-6277529364332160225gmail-m_5901480064531742242qt"><div>Hi Mircea <br></div><div><br></div><div>From the RFC you mentioned, that is a Darwin specific implementation, which later got extended to support other targets. The main use case for the embed bitcode option is to allow compiler passing intermediate IR and command flags in the object
file it produced for later use. For Darwin, it is used for bitcode recompilation, and some might use it to achieve other goals.<br></div><div><br></div><div>In order to use this information properly, you needs to have tools that understand the layout and sections for embedded bitcode. You can't just use an ordinary linker, because like you said, an ELF linker will just append the bitcode. Depending
on what you are trying to achieve, you need to implement the downstream tools, like linker, binary analysis tools, etc. to understand this concept.<br></div><div><br></div><div>Steven<br></div><div><div><div><br></div><blockquote type="cite"><div>On Aug 24, 2020, at 7:10 PM, Mircea Trofin via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><div><br></div><div><div dir="ltr"><div>Hello, <br></div><div><br></div><div>I'm trying to understand how .llvmbc and .llvmcmd fit into an end-to-end story. From <a href="http://lists.llvm.org/pipermail/llvm-dev/2016-February/094851.html" target="_blank"> the RFC</a>, and reading through the implementation, I'm piecing together that the goal was to enable capturing IR right after clang and before passing it to LLVM's optimization passes, as well as the command line options needed for later compiling that IR
to the same native object it was compiled to originally (with the same compiler).<br></div><div><br></div><div>Here's what I don't understand: say you have a.o and b.o compiled with -fembed-bitcode=all. They are linked into a binary called my_binary. How do you re-create the corresponding IR for modules a and b (let's call them a.bc and b.bc), and their
corresponding command lines? From what I can tell, the linker just concatenates the IR for a and b in my_binary's .llvmbc, and the same for the command line in .llvmcmd. Is there a separator maybe I missed? For .llvmcmd, I could see how *maybe* -cc1 could
be that separator, what about the .llvmbc part? The magic number?<br></div><div><br></div><div>Thanks!<br></div></div><div>_______________________________________________<br></div><div> LLVM Developers mailing list<br></div><div> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br></div><div> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br></div></div></blockquote></div></div><div><br></div><div><b>Attachments:</b><br></div><ul><li>ATT00001.txt<br></li></ul></blockquote><div><br></div></div></blockquote></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>