<div dir="ltr">Unfortunately I needed to update the links. Here they are, hopefully these work correctly.<div><br></div><div><a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQNAcTDfyQvh6Jq7IRdCvK_fuluUFrzrsGL_75Ile29hX3caBSfT6_jHulxeCJ5MXIHp5SB--A_goEi/pubhtml?gid=987260531&single=true">SPEC2006</a></div><div><br></div><div><a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQNAcTDfyQvh6Jq7IRdCvK_fuluUFrzrsGL_75Ile29hX3caBSfT6_jHulxeCJ5MXIHp5SB--A_goEi/pubhtml?gid=0&single=true">Internal benchmarks, clang, opt</a></div><div><br></div><div>Thanks!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 8, 2020 at 2:23 PM Mircea Trofin <<a href="mailto:mtrofin@google.com">mtrofin@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font face="monospace">It turns out it's me, sorry. Let me see how I can sort this out. In the meantime, here is the csv:</font><div><font face="monospace"><br></font></div><div><font face="monospace">SPEC2006 data:</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">binary,base -Oz size,ML -Oz size,ML size shrink by,,perf: base -Oz scores,perf: ML -Oz scores,ML improvement by<br>400.perlbench,2054200,2086776,-1.59%,,2.9,2.9,0.00%<br>401.bzip2,1129976,1095544,3.05%,,6.4,6.2,-3.13%<br>403.gcc,4078488,4130840,-1.28%,,11.6,11.7,0.86%<br>429.mcf,1089368,1054552,3.20%,,2.3,2.4,4.35%<br>433.milc,1161336,1129592,2.73%,,0.5,0.5,0.00%<br>444.namd,1324824,1290968,2.56%,,4.4,4.3,-2.27%<br>445.gobmk,5003096,4992472,0.21%,,3.9,4.1,5.13%<br>447.dealII,1003376,975024,2.83%,,162.1,197.8,22.02%<br>450.soplex,1359416,1326008,2.46%,,0.1,0.1,0.00%<br>453.povray,1921432,1952280,-1.61%,,32.9,32.7,-0.61%<br>456.hmmer,1210744,1184632,2.16%,,1.5,1.5,0.00%<br>458.sjeng,1185976,1155320,2.58%,,812.9,840.2,3.36%<br>462.libquantum,1101144,1066712,3.13%,,12.3,12.2,-0.81%<br>464.h264ref,1557176,1593272,-2.32%,,0.3,0.3,0.00%<br>470.lbm,1091352,1056664,3.18%,,2.3,2.5,8.70%<br>471.omnetpp,1045496,1046664,-0.11%,,27.9,28.4,1.79%<br>473.astar,1106680,1071992,3.13%,,2.4,2.6,8.33%<br>482.sphinx3,1216600,1182680,2.79%,,16.5,16.3,-1.21%<br>483.xalancbmk,4666936,4669112,-0.05%,,9.7,9.6,-1.03%<br>,,,,,,,<br>SIZE SUM,34307616,34061104,0.72%,TOTAL SCORES,5.7,5.8,1.75%<br></font></div><div><br></div><div><br></div><div><font face="monospace">various benchmarks:</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">binary,base -Oz size,ML -Oz size,ML shrink over base<br>llvm:opt,66318624,62124256,6.32%<br>math_1,6181552,5884144,4.81%<br>event_management_1,4998728,4802696,3.92%<br>llvm:lcalsBRaw,1270168,1222168,3.78%<br>llvm:lcalsBLambda,1270232,1222232,3.78%<br>llvm:lcalsARaw,1276248,1228248,3.76%<br>llvm:lcalsALambda,1276440,1228504,3.76%<br>llvm:lcalsCRaw,1278936,1231000,3.75%<br>llvm:lcalsCLambda,1279064,1231256,3.74%<br>llvm:Blur,1236888,1191192,3.69%<br>image_processing_diffusion,1236120,1190488,3.69%<br>llvm:Interpolation,1237208,1191576,3.69%<br>llvm:harris,1237208,1191768,3.67%<br>llvm:Dither,1238232,1192792,3.67%<br>event_management_2,4741096,4568584,3.64%<br>hashing_1,5189360,5004176,3.57%<br>compression_2,5329392,5140496,3.54%<br>math_2,6072336,5858992,3.51%<br>crypto_1,4721576,4556008,3.51%<br>math_4,27720208,26755568,3.48%<br>math_3,27732496,26767920,3.48%<br>infra_1,30673232,29630464,3.40%<br>hashing_2,5834544,5637168,3.38%<br>image_processing_entropy,5854960,5657008,3.38%<br>hashing_3,5831088,5634288,3.38%<br>memory_management_1,4957960,4790632,3.37%<br>hasing_4,5816048,5619888,3.37%<br>data_storage_2,5852976,5655792,3.37%<br>compression_1,5971984,5771120,3.36%<br>arena_unittest,6672944,6453584,3.29%<br>data_storage_3,44113232,42710960,3.18%<br>data_storage_1,22858992,22132960,3.18%<br>datastructures_1,6362032,6161264,3.16%<br>infra_2,56828720,55053496,3.12%<br>datastructures_2,164558704,160790320,2.29%<br>llvm:clang,77027416,75672088,1.76%<br>benchmark_infra,20221424,19944432,1.37%<br>search_1,169475360,167645632,1.08%<br>protobuf_3,24071504,23895440,0.73%<br>protobuf_1,24071504,23895504,0.73%<br>protobuf_2,24071504,23895504,0.73%<br>protobuf_4,24071504,23895504,0.73%<br>protobuf_5,24071504,23895504,0.73%<br>protobuf_6,23982800,23809424,0.72%</font><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 8, 2020 at 2:11 PM Johannes Doerfert <<a href="mailto:johannesdoerfert@gmail.com" target="_blank">johannesdoerfert@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Cool!<br>
<br>
I skipped to the end and tried to access the gdoc and the spreadsheet<br>
but it did tell me I need permissions. Can you make them accessible or<br>
am I the problem?<br>
<br>
Thanks,<br>
Johannes<br>
<br>
On 4/8/20 4:04 PM, Mircea Trofin via llvm-dev wrote:<br>
> TL;DR; We can improve compiler optimizations driven by heuristics by<br>
> replacing those heuristics with machine-learned policies (ML models).<br>
> Policies are trained offline and ship as part of the compiler. <br>
Determinism<br>
> is maintained because models are fixed when the compiler is operating in<br>
> production. Fine-tuning or regressions may be handled by <br>
incorporating the<br>
> interesting cases in the ML training set, retraining the compiler, and<br>
> redeploying it.<br>
><br>
> For a first milestone, we chose inlining for size (-Oz) on X86-64. We <br>
were<br>
> able to train an ML model to produce binaries 1.5-6% smaller than -Oz of<br>
> tip-of-tree. The trained model appears to generalize well over a diverse<br>
> set of binaries. Compile time is increased marginally (under 5%). The <br>
model<br>
> also happens to produce slightly better - performing code under <br>
SPEC2006 -<br>
> total score improvement by 1.75%. As we only wanted to verify there is no<br>
> significant regression in SPEC, and given the milestone goals, we haven’t<br>
> dug any deeper into the speed results.<br>
><br>
> We see these results as promising, and as a reasonable point for<br>
> contributing our current work as a build-time opt-in to LLVM to <br>
benefit the<br>
> community, in the hope of fostering collaboration and learning from the<br>
> community’s feedback, as we try to better understand the trade-offs <br>
such an<br>
> approach entails, and as we work on expanding the depth and breadth of<br>
> applying these techniques to compiler optimization problems.<br>
><br>
> <a href="https://reviews.llvm.org/D77752" rel="noreferrer" target="_blank">https://reviews.llvm.org/D77752</a><br>
> Motivation<br>
><br>
> Optimization problems, such as inlining or register allocation, are hard:<br>
> we don’t know of algorithms that are guaranteed to produce the optimum<br>
> solution in all cases in a timely fashion. Instead, we use heuristics:<br>
> algorithms that we expect to produce some approximation of the optimum,<br>
> with some expected degree of generality, in a practical amount of time.<br>
><br>
> Heuristics have some common characteristics. Taking inlining as a case<br>
> study, it traverses the problem space in some way (bottom-up traversal of<br>
> the SCC graph), extracts some properties (let’s call them “features”) of<br>
> the program being optimized, and combine them with some weights (“tune”),<br>
> to produce a cost (InlineCost), which allows for trade-off analysis. We<br>
> validate the effectiveness of a heuristic over some accepted set of<br>
> benchmarks. Over time, we react to regressions or pathological cases<br>
> observed in the field, by manually analyzing such cases, figuring out an<br>
> enhancement to the heuristic, and then re-validating over that set of<br>
> benchmarks (maybe augmented by adding the newly found cases).<br>
><br>
> Because heuristics are code that needs to be maintained, there is <br>
pressure<br>
> to reduce complexity: adding more features means we need to reason about<br>
> the interactions between the old and new features, which scales<br>
> combinatorially. Re-tuning because of the new features adds a similar <br>
kind<br>
> of complexity. Potentially, we miss out on optimization improvements as a<br>
> result.<br>
><br>
> Because tuning is manual, there is pressure to keep the number of<br>
> benchmarks that can be studied in depth to a humanly-manageable size, <br>
which<br>
> potentially affects the generality of a heuristic or heuristic tuning.<br>
><br>
> The main advantage of manual heuristics is arguably their relatively <br>
lower<br>
> overhead: no additional dependencies and more transparent to human <br>
analysis<br>
> and improvement.<br>
><br>
> Machine learning, in particular reinforcement learning, can address the<br>
> tensions found in manual heuristics: once features are extracted from the<br>
> program, the way they are combined and tuned can easily be scaled up<br>
> through automation, improving effectiveness and generality. A major<br>
> drawback, at least at this point in time, of machine learning, is that we<br>
> don’t yet have a fully developed systematic approach for improving policy<br>
> effectiveness.<br>
> High level design<br>
><br>
> We identify two scenarios for a compiler using ML policies: <br>
development and<br>
> release.<br>
><br>
> The release scenario is equivalent to the regular compilation we have <br>
today<br>
> - the only difference is that it uses a pre-trained model (trained in the<br>
> development scenario beforehand) to make decisions instead of the<br>
> heuristics. Determinism is guaranteed since the model in the release<br>
> scenario is fixed. We imagine teams wishing to fine tune the <br>
effectiveness<br>
> of the optimization to their scenarios would train a different model.<br>
><br>
> The decision previously evaluated using a human-crafted heuristic is<br>
> optionally replaced by:<br>
><br>
> -<br>
><br>
> a compiler-specific component, extracting features from IR (i.e. a<br>
> vector of values)<br>
> -<br>
><br>
> an evaluation of an ML model using those features, to obtain a result.<br>
> In ML nomenclature, this is referred to using the model for <br>
inference (as<br>
> opposed to training it)<br>
><br>
> For example, when we replaced the decision of whether to inline a <br>
callsite,<br>
> the ML model produces a boolean (inline/don’t inline) based on a features<br>
> vector characterizing the call site and some broader module-wide context.<br>
><br>
> Training/development is more complicated, and happens offline - akin to<br>
> how, today, attempts to improve an optimizing pass also happen offline. A<br>
> description of the high level design and the specifics we used for the<br>
> current scope are given in Appendix.<br>
> Current Scope<br>
><br>
> The goal of our first milestone was to evaluate end to end an integration<br>
> of ML with LLVM, and get a first promising result. To that end, we chose<br>
> inlining for size (-Oz) as a stepping stone, as we perceived it to be <br>
more<br>
> likely to require a simpler evaluation setup than performance-oriented<br>
> optimizations might. At this point, we only train whether a call site may<br>
> be inlined or not, leaving the SCC traversal order as-is.<br>
><br>
> We are proposing an initial change demonstrating the inference mechanism<br>
> using a pre-trained model, as a build-time opt-in to llvm. The compiler<br>
> components needed to perform training are also included in this first<br>
> change. Subsequent changes would include more training-related <br>
components.<br>
><br>
> At a high level, the changes we are proposing consist of:<br>
><br>
> 1.<br>
><br>
> a new module analysis pass, InliningAdvisor. By default, its<br>
> implementation does nothing.<br>
> 2.<br>
><br>
> minimal hooks into Inliner.cpp.<br>
> 3.<br>
><br>
> the implementation of InliningAdvisor, activated when we opt-in <br>
ML. This<br>
> is available in Analysis/ML, together with:<br>
> 1.<br>
><br>
> Rel/Dev specific ML model handing, also under Analysis/ML<br>
> 2.<br>
><br>
> a pre-trained model for inlining for size<br>
> (Analysis/ML/models/inlining)<br>
> 3.<br>
><br>
> a pre-trained model for predicting native size from IR<br>
> (Analysis/ML/models/ir_2_native_x86_64), used in Dev mode only.<br>
> 4.<br>
><br>
> Some refactorings in PassBuilder, to allow opting into running <br>
mandatory<br>
> inlining first - some compilation speedup for the ML case, minimal,<br>
> noise-like size effect. Also simplifies testing (these would be <br>
introduced<br>
> as a preliminary patch).<br>
><br>
> We discuss ‘rel’ mode here, and ‘dev’ mode in the Appendix, as it is more<br>
> involved.<br>
> Inference Opt-In Mechanism<br>
><br>
> The feature is primarily controlled by the cmake flag<br>
> LLVM_USE_ML_POLICY={“Rel”|”Dev”}. Each has different dependencies. The<br>
> “Rel”ease case requires specifying the location of the pip tensorflow<br>
> package (currently, that’s tf_nightly, and it should soon be available in<br>
> tensorflow)<br>
><br>
> To opt in the ‘Rel’ case:<br>
><br>
> 1.<br>
><br>
> install tensorflow pip package<br>
><br>
> pip3 install tf_nightly --user<br>
><br>
> 1.<br>
><br>
> configure llvm build<br>
><br>
> cmake ../llvm -DLLVM_USE_ML_POLICY=Rel \<br>
><br>
> -DLLVM_TF_AOT_RUNTIME=~/.local/lib/python3.7/site-packages/tensorflow \<br>
><br>
> {-DLLVM_TF_AOT_COMPILER=<path to saved_model_cli tool, if needed - it’s<br>
> usually in the path>}<br>
><br>
><br>
><br>
> 1.<br>
><br>
> build llvm as usual.<br>
> 2.<br>
><br>
> pass -mllvm -enable-ml-inliner -mllvm -mandatory-inlinings-first to<br>
> clang.<br>
><br>
> Details<br>
><br>
> The ML model is captured as a TensorFlow ‘saved model’. When building <br>
llvm,<br>
> we use TensorFlow’s XLA native compiler (saved_model_cli) to compile the<br>
> saved model into a native static library and a header file. Insofar <br>
as LLVM<br>
> is concerned, there are minimal additional runtime requirements, packaged<br>
> as source within the pip package: C++ wrappers around the compiled model.<br>
> These will also be statically linked in the LLVM target. The compiled <br>
code<br>
> is otherwise just a complex arithmetical computation, with no special<br>
> requirements - it is single threaded and runs natively on the targeted<br>
> architecture. Together with the aforementioned runtime dependencies, it<br>
> adds ~115KB to the clang binary (0.08% increase)<br>
><br>
> Runtime-wise, we observed a ~10% increase in the time spent in the <br>
inliner,<br>
> for a large (33MB) binary IR module; inlining typically consumes <br>
~10-15% of<br>
> total compilation time, so the overall compile time overhead of the<br>
> approach is arguably negligible. This cost is almost in entirety<br>
> attributable to feature extraction.<br>
><br>
> Memory-wise, the precompiled model has a fixed size buffer for its <br>
inputs,<br>
> and performs a fixed amount of computations, so the memory overhead<br>
> inherent to our approach is independent from the program being optimized.<br>
> Using a small example to avoid effects such as memory use differences due<br>
> to different inlinings, we observed an 300KB increase in the maximum<br>
> resident size.<br>
><br>
> A table showing effect on -Oz compiled binaries’ size is given in <br>
Appendix.<br>
> Next directions<br>
><br>
> Our next milestone has two main high level goals: detailing a systematic<br>
> approach to driving policy effectiveness; and exploring in depth the type<br>
> of features and training algorithms most appropriate for compiler <br>
problems,<br>
> or at least problems like inlining. For the latter, we expect embedding<br>
> more of the call graph structure to play an important role, as well as,<br>
> potentially, delegating the problem space traversal to the ML model.<br>
><br>
> We plan to include inlining for speed as part of our work on these goals.<br>
> AppendixTraining - High Level<br>
><br>
> We use Reinforcement Learning (RL) to train the Inline-for-size <br>
model. At a<br>
> high level, it is composed of 3 parts: training data collection, model<br>
> training, and iterative data collection/model training. We use TensorFlow<br>
> as our ML framework.<br>
><br>
> Related, we also needed to learn a separate model to evaluate the native<br>
> size of a function, given its IR, in order to calculate a more precise<br>
> reward for the reinforcement learning algorithm (“IR2Native”). We <br>
evaluated<br>
> ‘just counting IR’ and TargetTransformInfo, but they appeared to provide<br>
> too noisy of a signal for the reward, insofar as the RL training <br>
algorithm<br>
> for the inlining model was concerned. This model is only used during<br>
> training.<br>
><br>
> RL - Training data collection: the training data we need to feed into a<br>
> reinforcement learning algorithm are sequences consisting of: state <br>
of the<br>
> problem (i.e. features); action (inline/not inline), and reward (native<br>
> size shrinkage after inline/not inline, using ir2native). To collect the<br>
> sequences, we hook the logging infrastructure into LLVM Inliner that is<br>
> able to produce logs after the inline optimization pass.<br>
><br>
> RL - Model training: We use DQN (Deep Q-Network) to train our<br>
> inlining-for-size ML policy. On a high level, the DQN algorithm trains a<br>
> neural network to predict the value of different actions --- the DQN <br>
policy<br>
> then chooses to take the action with the highest predicted value. In our<br>
> scenario, we have two actions: 1) inline; 2) not inline, so the neural<br>
> network predicts the size reduction of these two actions based on <br>
features,<br>
> and then decides to conduct inlining if the neural network believes doing<br>
> inlining leads to higher size reduction. After the training finishes, it<br>
> produces a TensorFlow SavedModel that takes features as input and <br>
generates<br>
> inline decisions (whether to inline or not) as output.<br>
><br>
> The choice of the features and reward are essential in model <br>
training. The<br>
> features are chosen with the consideration of being helpful in making the<br>
> decision --- the input to the cost model is a good starting point. <br>
Ideally,<br>
> the reward in the inline-for-size problem is the native size shrinkage<br>
> after inline/not inline. It is difficult to obtain precisely, so we <br>
use the<br>
> estimate as an alternative. This means that, for training, we also need a<br>
> model (not necessarily machine learned) for estimating rewards.<br>
><br>
> RL - Iterative data collection/model training: Reinforcement learning is<br>
> ideally an iterative model/policy improvement process that: 1) the <br>
trained<br>
> model is deployed to the field to collect new data; 2) newly <br>
collected data<br>
> are used to update the model. Thus, we need to do iterative data<br>
> collection/model training. To facilitate data collection (avoid complex<br>
> build dependencies and time spent before/after the pass being <br>
trained), we<br>
> isolate out IR modules captured right before the optimization we are<br>
> interested in, and iterate on them with opt. We bootstrap the <br>
training from<br>
> the current heuristic, and stop the process once we are happy with the<br>
> outcome.<br>
><br>
> IR2Native: We collect IR features (different from the ones used for<br>
> inlining) for each function at the end of inlining, right before we <br>
perform<br>
> function simplification passes, and right after. This means we have <br>
two IR<br>
> ‘shapes’ of the same function, and we know no further inlinings will be<br>
> performed, so whatever changes happen are based on that IR. We then <br>
extract<br>
> the native size at the end of compilation. Together, this data forms two<br>
> records per function that can be used in supervised learning - the <br>
features<br>
> are those extracted from IR, and the label is the native size. Training<br>
> IR2Native happens independently from the training of the inliner model.<br>
> Training support for the current scope<br>
><br>
> The initial change includes the logging mechanism, an ir2native model<br>
> trained for x86-64, and the means to rapidly iterate over development ML<br>
> models. For the components that will be included in subsequent <br>
changes, the<br>
> rest of this section describes the mechanisms we employed. These <br>
components<br>
> are detailed further below.<br>
><br>
> To build LLVM with the ML policy in ‘Dev’ mode, we need the tensorflow C<br>
> API library <<a href="https://www.tensorflow.org/install/lang_c" rel="noreferrer" target="_blank">https://www.tensorflow.org/install/lang_c</a>>. We then <br>
configure<br>
> the build:<br>
><br>
> cmake .. -DLLVM_USE_ML_POLICY=Dev \<br>
><br>
> -DLLVM_TF_C_LIB=<path to unarchived package> \<br>
><br>
> {-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=True, to copy tensorflow shared<br>
> library over, if it’s not on LD_LIBRARY_PATH}<br>
><br>
><br>
> To extract IR right before inlining, we hacked on top of the ThinLTO<br>
> infrastructure, interrupting its pre-link pipeline right before inlining.<br>
> This means clang produces binary IR files captured at that stage. We then<br>
> built a large target, obtaining a corpus of ~25K modules. We intend to<br>
> provide a clean mechanism in a subsequent change.<br>
><br>
> To obtain features/labels for training this “IR to Native Size” model, we<br>
> had to make some changes to the AsmPrinter (to get real native sizes) and<br>
> llvm-readobj, as well as add an analysis pass for extracting the IR<br>
> features for this model. We plan on upstreaming these changes <br>
subsequently.<br>
><br>
> Finally, the infrastructure driving the policy training is currently <br>
built<br>
> on proprietary APIs, as it benefits from a distributed computing<br>
> infrastructure. We are currently evaluating options for open sourcing it.<br>
> In the meantime, we are presenting the high level implementation details.<br>
><br>
> To train a new model, the infrastructure performs 2 steps: extracting the<br>
> logs, and using them in a training algorithm.<br>
><br>
> Log extraction is highly parallelizable: for each IR module in the <br>
training<br>
> corpus, we need to run opt once (or a few times, when we explore<br>
> improvements). Specifically, each run is this invocation:<br>
><br>
> opt -passes=scc-oz-module-inliner -ml-inliner-ir2native-model=<path to<br>
> ir2native> -training-log=<path to training log output> -enable-ml-inliner<br>
> -mandatory-inlinings-first -o <output> <module.o><br>
><br>
> Then collect the logs, and pass them to the next step.<br>
><br>
><br>
> Experimental results<br>
><br>
> Experimental results are available as follows:<br>
><br>
><br>
> -<br>
><br>
> SPEC2006<br>
> <br>
<<a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQv0bAsUlgnG114zMYy_zKR6x-lTjcXVNt7VEeSwq2-pDr5oTxdASCscPRRg6L7iYLu2AVJ44G2xEkp/pubhtml?gid=1870752756&single=true" rel="noreferrer" target="_blank">https://docs.google.com/spreadsheets/d/e/2PACX-1vQv0bAsUlgnG114zMYy_zKR6x-lTjcXVNt7VEeSwq2-pDr5oTxdASCscPRRg6L7iYLu2AVJ44G2xEkp/pubhtml?gid=1870752756&single=true</a>><br>
> binary sizes (-Oz) and ‘test run’ scores.<br>
> -<br>
><br>
> Size report<br>
> <br>
<<a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQv0bAsUlgnG114zMYy_zKR6x-lTjcXVNt7VEeSwq2-pDr5oTxdASCscPRRg6L7iYLu2AVJ44G2xEkp/pubhtml?gid=935959003&single=true" rel="noreferrer" target="_blank">https://docs.google.com/spreadsheets/d/e/2PACX-1vQv0bAsUlgnG114zMYy_zKR6x-lTjcXVNt7VEeSwq2-pDr5oTxdASCscPRRg6L7iYLu2AVJ44G2xEkp/pubhtml?gid=935959003&single=true</a>><br>
> from some internal benchmarks and binaries, including opt and clang<br>
><br>
><br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br>
</blockquote></div>
</blockquote></div>