[cfe-dev] Remove unused types from ASTContext

Brian Cain via cfe-dev cfe-dev at lists.llvm.org
Sun Apr 30 18:32:52 PDT 2017 

On Sun, Apr 30, 2017 at 6:12 PM, mateusz janek <stryku2393 at gmail.com> wrote:

> Thanks again. Unfortunately I think that even "only AST" will be to much
> for my RAM.
>
>
When I did the compile using ToT clang and libc++, it completed in seconds
and occupied a peak of 4GB of memory.  This strikes me as pretty sane
behavior and very different from your compile-takes-overnight experience.
How much physical memory do you have on the build machine and how what is
your target time budget for this job?  How much memory is consumed on your
existing clang 3.9/4, g++ 6 baselines?  If those builds use more than 4GB,
maybe you can just upgrade to a more current release.

If by some chance your build host doesn't have 4GB of physical memory,
maybe that's an easier problem to solve.


> I have one more idea, please tell me if this makes any sense. I'm thinking
> about storing class specializations in term of scopes.
> (It don't want this to work with every C++ project, I'm ok with that it'll
> work only with my project)
> (And I'd stick to couple of assumptions during ctai implementation, e.g.
> only one depth scope resolution, A::B)
> (I'm aware that probably I'd need to write my own implementation of
> ClassTemplateSpecializationDecl and couple of more classes)
>
> Example code and the flow:
>
> template <int N>
> struct N_Holder
> {
> static constexpr int N_Value = N;
> };
>
> template <int Val1, int Val2>
> struct Add_impl
> {
> using Val1_Holder = Holder<Val1>;
> using Val2_Holder = Holder<Val2>;
> static constexpr int Result_impl = Val1_Holder::N_Value +
> Val2_Holder::N_Value;
> };
>
> template <int Val1, int Val2>
> struct Add
> {
> static constexpr int Result = Add_impl<Val1, Val2>::Result_impl;
> };
>
> struct A_Struct
> {
> static constexpr int Value = Add<10, 20>::Result;
> };
>
>
>
> Flow:
>
> (...)
> -see class A_Struct
> -go into A_Struct scope
> -see a member Value =
> -create class Add<10, 20> specialization
> -go into Add<10, 20> scope
> -see parameters {non-type 10, non-type 20}
> -see member Result =
> -create Add_impl<parameters[0], parameters[1]> specialization
> -go into Add_impl<10, 20> scope
> -see parameters {non-type 10, non-type 20}
> -see alias Val1_Holder =
> -create Holder<parameters[0]> specialization
> -go into Holder<10> scope
> -see parameters {non-type 10}
> -see member N_Value = parameters[0]
> -go out of Holder<10> scope (nothing do delete)
> -assign Val1_Holder = Holder<10> specialization 'by value':
> {
> aliases {}
> members { N_Value = 10 }
> }
> -delete Holder<10> specialization
> -see alias Val2_Holder =
> -create Holder<parameters[1]> specialization
> -go into Holder<20> scope
> -see parameters {non-type 20}
> -see member N_Value = parameters[0]
> -go out of Holder<20> scope (nothing do delete)
> -assign Val2_Holder = Holder<20> specialization 'by value':
> {
> aliases {}
> members { N_Value = 20 }
> }
> -delete Holder<20> specialization
> -see member Result_impl = Val1_Holder::N_value + Val2_Holder::N_Value;
> -go out of Add_impl<10, 20> scope:
> -assign Result = Add_impl<10, 20>::Result_impl
> -delete Add_impl<10, 20> specialization
> -go out of Add<10, 20> specialization
> -assign Value = Add<10, 20>::Result
> -delete Add<10, 20> specialization
> -go out of A_Struct scope
> (...)
>
>
> Thanks,
> Stryku
>
>
>
> 2017-04-30 23:00 GMT+02:00 mats petersson <mats at planetcatfish.com>:
>
>> Like I said, you need those types for most of the compilation process.
>> It's REALLY hard to keep track of when these things don't matter any longer
>> - you certainly need types that actually generate code all the way to
>> LLVM-IR generation. That's for the AST types. IR types, which isn't quite
>> the same thing.
>>
>> The latter, you can do by doing "syntax-only" run on the compiler - it
>> won't generate any LLVM-IR, so memory usage here is "only" AST. Also,
>> prodcuing LLVM-IR output from the compiler (-S -emit-llvm) and then using
>> `llc` to compile the IR to machine code, to determine memory usage of the
>> IR types (obviously with other things).
>>
>> [In my compiler project, I decided it's so hard to track what is needed
>> again and what isn't, and decided to NOT free any memory for that reason -
>> of course, not a particularly great strategy if you want to run it as a
>> library in something else, but if I really had to, I could probably stick
>> every object in one or more container(s) and free it all at the end]
>>
>> --
>> Mats
>>
>> On 30 April 2017 at 21:12, mateusz janek <stryku2393 at gmail.com> wrote:
>>
>>> Sorry for no response, I was quite busy.
>>>
>>> Thanks Brian for help with the build (: Also I've tried with the g++-6
>>> and it has same problem with the memory usage.
>>>
>>> The problem here is caused by creating a lot of template class
>>> specializations. As Mats partially said and I found in the llvm sources
>>> here: https://github.com/stryku/llvm/blob/master/include/llvm/IR/T
>>> ype.h#L42 types are stored till the end of program life. AFAIU every
>>> class specialization is a separate type.
>>>
>>> Simpler program that shows my problem is e.g. https://gist.github.com/s
>>> tryku/986bcd6a5d5773ef0e0136a5eca97e20
>>> There is a lot of specializations of values_container, create_container
>>> , create_impl etc. Additionally every values_container have a lot of
>>> template parameters which in fact are also types (built-in types, but
>>> still).
>>>
>>> For now, I'm playing with llvm and clang sources to understand what's
>>> going on there and I'll probably try to implement my own mechanism of
>>> storing a template class specializations.
>>>
>>> Thanks for your interest,
>>> Stryku
>>>
>>>
>>>
>>> 2017-04-29 0:02 GMT+02:00 Brian Cain <brian.cain at gmail.com>:
>>>
>>>>
>>>> On Fri, Apr 28, 2017 at 6:11 AM, mateusz janek <stryku2393 at gmail.com>
>>>> wrote:
>>>>
>>>>> Did you use CMake? If no, ctai requres clang 4.0 and to be compiled
>>>>> with flags: `-std=c++1z -ftemplate-backtrace-limit=0
>>>>> -ftemplate-depth=8096`
>>>>>
>>>>>
>>>> I got it to work with some tweaks to the CMake config.  It ran quickly
>>>> for me, but it did allocate a decent amount of memory on the way.  Also I
>>>> used a nightly build of clang and libc++ (and not libstdc++).
>>>>
>>>> 15.85user 0.48system 0:16.38elapsed 99%CPU (0avgtext+0avgdata
>>>> 4053852maxresident)k
>>>> 0inputs+24outputs (0major+381214minor)pagefaults 0swaps
>>>>
>>>>
>>>>
>>>> diff:
>>>>
>>>> diff --git a/CMakeLists.txt b/CMakeLists.txt
>>>> index 0143461..109d975 100644
>>>> --- a/CMakeLists.txt
>>>> +++ b/CMakeLists.txt
>>>> @@ -1,9 +1,9 @@
>>>>  cmake_minimum_required(VERSION 3.5)
>>>>  project(ctai)
>>>>
>>>> -add_definitions(-std=c++1z -ftemplate-backtrace-limit=0
>>>> -ftemplate-depth=1024)
>>>> -
>>>> -set(CMAKE_CXX_COMPILER clang++-3.9)
>>>> +set(CMAKE_CXX_COMPILER clang++-3.9 CACHE STRING "")
>>>> +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++1z -stdlib=libc++
>>>> -ftemplate-backtrace-limit=0 -ftemplate-depth=8192" CACHE STRING "" FORCE)
>>>> +set(CMAKE_MODULE_LINKER_FLAGS ${CMAKE_MODULE_LINKER_FLAGS} "-std=c++1z
>>>> -stdlib=libc++" CACHE STRING "" FORCE)
>>>>
>>>>  set(INCLUDE_FILES
>>>>          register.hpp
>>>> diff --git a/main.cpp b/main.cpp
>>>> index b95eb4d..35e6d6f 100644
>>>> --- a/main.cpp
>>>> +++ b/main.cpp
>>>> @@ -30,6 +30,66 @@ using code = decltype(
>>>>      "mov ebx , 1 "
>>>>      "add eax , ebx "
>>>>      "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>> +    "mov DWORD PTR [ ebp + 4 ] , eax "
>>>>      "jmp .loop_label "
>>>>  ":end_label "
>>>>      "mov eax , DWORD PTR [ ebp + 16 ] "
>>>> @@ -39,4 +99,4 @@ int main()
>>>>  {
>>>>      std::cout << "15th element of fibonacci series is: " <<
>>>> cai::execute_code<code>;
>>>>      return 0;
>>>> -}
>>>> \ No newline at end of file
>>>> +}
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>


-- 
-Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170430/376a6706/attachment.html>

More information about the cfe-dev mailing list