[llvm] r189297 - Add new API lto_codegen_compile_parallel().
Shuxin Yang
shuxin.llvm at gmail.com
Mon Aug 26 17:53:01 PDT 2013
We certainly need a way to feed multiple resulting objects back to
linker. There are couple of ways
for this end:
1) 'ld -r all-resulting-obj-on-disk -o result.o" and feed the only
object file (i.e. the result.o)
back to linker
2) keep the resulting objects in memory buffer, and feedback to
buffers back to linker
(as proposed by Nick)
3) As with GNU gold, save the resulting objects on disk, and feed
the these disk files back to linker
one by one.
I'm big linker nut. I don't know which way work better. I try to
use 1) as a workaround for the time being
before 2) is available. People at Apple disagree my engineering approach.
From compiler's perspective,
o. 1) is not just workaround, 3) is certainly better than 1).
o. 2) will win if the program being compiled is small- or
medium-sized.
With huge programs, it will be difficult for compiler to
decide when and how to "spill" some stuff
from memory to disk. Folks in Apple iterate and reiterate we
only consider the case that the entire
program can be loaded in memory. So, the added difficulty for
compiler dose not seems to be a
problem for the workload we care about.
On 8/26/13 5:24 PM, Eric Christopher wrote:
> Not sure why this is a good idea? What usefulness does this have
> outside of your partitioning scheme?
>
> -eric
>
> On Mon, Aug 26, 2013 at 5:03 PM, Shuxin Yang <shuxin.llvm at gmail.com> wrote:
>> Author: shuxin_yang
>> Date: Mon Aug 26 19:03:23 2013
>> New Revision: 189297
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=189297&view=rev
>> Log:
>> Add new API lto_codegen_compile_parallel().
>>
>> This API is proposed by Nick Kledzik. The semantic is:
>>
>> --------------------------------------------------------------------------
>> Generate code for merged module into an array of native object files. On
>> success returns a pointer to an array of NativeObjectFile. The count
>> parameter returns the number of elements in the array. Each element is
>> a pointer/length for a generated mach-o/ELF buffer. The buffer is owned
>> by the lto_code_gen_t and will be freed when lto_codegen_dispose() is called,
>> or lto_codegen_compile() is called again. On failure, returns NULL
>> (check lto_get_error_message() for details).
>>
>> extern const struct NativeObjectFile*
>> lto_codegen_compile_parallel(lto_code_gen_t cg, size_t *count);
>> ---------------------------------------------------------------------------
>>
>> This API is currently only called on OSX platform. Linux or other Unixes
>> using GNU gold are not supposed to call this function, because on these systems,
>> object files are fed back to linker via disk file instead of memory buffer.
>>
>> In this commit, lto_codegen_compile_parallel() simply calls
>> lto_codegen_compile() to return a single object file. In the near future,
>> this function is the entry point for compilation with partition. Linker can
>> blindly call this function even if partition is turned off; in this case,
>> compiler will return only one object file.
>>
>> Modified:
>> llvm/trunk/include/llvm-c/lto.h
>> llvm/trunk/tools/lto/LTOCodeGenerator.cpp
>> llvm/trunk/tools/lto/LTOCodeGenerator.h
>> llvm/trunk/tools/lto/lto.cpp
>> llvm/trunk/tools/lto/lto.exports
>>
>> Modified: llvm/trunk/include/llvm-c/lto.h
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm-c/lto.h?rev=189297&r1=189296&r2=189297&view=diff
>> ==============================================================================
>> --- llvm/trunk/include/llvm-c/lto.h (original)
>> +++ llvm/trunk/include/llvm-c/lto.h Mon Aug 26 19:03:23 2013
>> @@ -27,7 +27,7 @@
>> * @{
>> */
>>
>> -#define LTO_API_VERSION 4
>> +#define LTO_API_VERSION 5
>>
>> typedef enum {
>> LTO_SYMBOL_ALIGNMENT_MASK = 0x0000001F, /* log2 of alignment */
>> @@ -284,6 +284,22 @@ lto_codegen_compile(lto_code_gen_t cg, s
>> extern bool
>> lto_codegen_compile_to_file(lto_code_gen_t cg, const char** name);
>>
>> +struct NativeObjectFile {
>> + const void *content;
>> + size_t length;
>> +};
>> +
>> +/**
>> + * Generates code for merged module into an array of native object files.
>> + * On success returns a pointer to an array of NativeObjectFile. The
>> + * count parameter returns the number of elements in the array. Each
>> + * element is a pointer/length for a generated mach-o/ELF buffer. The
>> + * buffer is owned by the lto_code_gen_t and will be freed when
>> + * lto_codegen_dispose() is called, or lto_codegen_compile() is called again.
>> + * On failure, returns NULL (check lto_get_error_message() for details).
>> + */
>> +extern const struct NativeObjectFile*
>> +lto_codegen_compile_parallel(lto_code_gen_t cg, size_t *count);
>>
>> /**
>> * Sets options to help debug codegen bugs.
>>
>> Modified: llvm/trunk/tools/lto/LTOCodeGenerator.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/lto/LTOCodeGenerator.cpp?rev=189297&r1=189296&r2=189297&view=diff
>> ==============================================================================
>> --- llvm/trunk/tools/lto/LTOCodeGenerator.cpp (original)
>> +++ llvm/trunk/tools/lto/LTOCodeGenerator.cpp Mon Aug 26 19:03:23 2013
>> @@ -74,7 +74,7 @@ LTOCodeGenerator::LTOCodeGenerator()
>> _linker(new Module("ld-temp.o", _context)), _target(NULL),
>> _emitDwarfDebugInfo(false), _scopeRestrictionsDone(false),
>> _codeModel(LTO_CODEGEN_PIC_MODEL_DYNAMIC),
>> - _nativeObjectFile(NULL) {
>> + _nativeObjectFile(NULL), ObjBufVect(0) {
>> InitializeAllTargets();
>> InitializeAllTargetMCs();
>> InitializeAllAsmPrinters();
>> @@ -85,6 +85,7 @@ LTOCodeGenerator::~LTOCodeGenerator() {
>> delete _target;
>> delete _nativeObjectFile;
>> delete _linker.getModule();
>> + delete[] ObjBufVect;
>>
>> for (std::vector<char*>::iterator I = _codegenOptions.begin(),
>> E = _codegenOptions.end(); I != E; ++I)
>> @@ -246,6 +247,23 @@ const void* LTOCodeGenerator::compile(si
>> return _nativeObjectFile->getBufferStart();
>> }
>>
>> +// Currently compile() and compile_parallel() have no difference.
>> +NativeObjectFile *LTOCodeGenerator::compile_parallel(size_t *count,
>> + std::string &ErrMsg) {
>> + assert(ObjBufVect == 0 && "Should be NULL");
>> +
>> + size_t Len;
>> + const void *Buf = compile(&Len, ErrMsg);
>> + if (!Buf)
>> + return 0;
>> +
>> + *count = 1;
>> + ObjBufVect = new NativeObjectFile[1];
>> + ObjBufVect[0].content = Buf;
>> + ObjBufVect[0].length = Len;
>> + return ObjBufVect;
>> +}
>> +
>> bool LTOCodeGenerator::determineTarget(std::string &errMsg) {
>> if (_target != NULL)
>> return true;
>>
>> Modified: llvm/trunk/tools/lto/LTOCodeGenerator.h
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/lto/LTOCodeGenerator.h?rev=189297&r1=189296&r2=189297&view=diff
>> ==============================================================================
>> --- llvm/trunk/tools/lto/LTOCodeGenerator.h (original)
>> +++ llvm/trunk/tools/lto/LTOCodeGenerator.h Mon Aug 26 19:03:23 2013
>> @@ -102,6 +102,17 @@ struct LTOCodeGenerator {
>> //
>> const void *compile(size_t *length, std::string &errMsg);
>>
>> + // Corresponding to lto_codegen_compile_parallel() API.
>> + // Generates code for merged module into an array of native object files.
>> + // On success returns a pointer to an array of NativeObjectFile. The count
>> + // parameter returns the number of elements in the array. Each element is
>> + // a pointer/length for a generated mach-o/ELF buffer. The buffer is owned
>> + // by the lto_code_gen_t and will be freed when lto_codegen_dispose() is
>> + // called, or lto_codegen_compile() is called again. On failure, returns
>> + // NULL (check lto_get_error_message() for details).
>> + //
>> + NativeObjectFile *compile_parallel(size_t *count, std::string &ErrMsg);
>> +
>> private:
>> void initializeLTOPasses();
>>
>> @@ -127,6 +138,7 @@ private:
>> std::vector<char*> _codegenOptions;
>> std::string _mCpu;
>> std::string _nativeObjectPath;
>> + NativeObjectFile *ObjBufVect;
>> };
>>
>> #endif // LTO_CODE_GENERATOR_H
>>
>> Modified: llvm/trunk/tools/lto/lto.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/lto/lto.cpp?rev=189297&r1=189296&r2=189297&view=diff
>> ==============================================================================
>> --- llvm/trunk/tools/lto/lto.cpp (original)
>> +++ llvm/trunk/tools/lto/lto.cpp Mon Aug 26 19:03:23 2013
>> @@ -207,6 +207,11 @@ bool lto_codegen_compile_to_file(lto_cod
>> return !cg->compile_to_file(name, sLastErrorString);
>> }
>>
>> +extern const struct NativeObjectFile *
>> +lto_codegen_compile_parallel(lto_code_gen_t cg, size_t *count) {
>> + return cg->compile_parallel(count, sLastErrorString);
>> +}
>> +
>> /// lto_codegen_debug_options - Used to pass extra options to the code
>> /// generator.
>> void lto_codegen_debug_options(lto_code_gen_t cg, const char *opt) {
>>
>> Modified: llvm/trunk/tools/lto/lto.exports
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/lto/lto.exports?rev=189297&r1=189296&r2=189297&view=diff
>> ==============================================================================
>> --- llvm/trunk/tools/lto/lto.exports (original)
>> +++ llvm/trunk/tools/lto/lto.exports Mon Aug 26 19:03:23 2013
>> @@ -28,6 +28,7 @@ lto_codegen_set_assembler_args
>> lto_codegen_set_assembler_path
>> lto_codegen_set_cpu
>> lto_codegen_compile_to_file
>> +lto_codegen_compile_parallel
>> LLVMCreateDisasm
>> LLVMCreateDisasmCPU
>> LLVMDisasmDispose
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list