[llvm] r254774 - [llc/opt] Add an option to run all passes twice
Keno Fischer via llvm-commits
llvm-commits at lists.llvm.org
Fri Dec 4 16:09:49 PST 2015
r254804
On Fri, Dec 4, 2015 at 6:58 PM, Keno Fischer <kfischer at college.harvard.edu>
wrote:
> Thanks,
>
> I think I have a handle on this. Fix coming shortly.
>
> On Fri, Dec 4, 2015 at 6:52 PM, Justin Bogner <mail at justinbogner.com>
> wrote:
>
>> Keno Fischer via llvm-commits <llvm-commits at lists.llvm.org> writes:
>> > Author: kfischer
>> > Date: Fri Dec 4 15:56:46 2015
>> > New Revision: 254774
>> >
>> > URL: http://llvm.org/viewvc/llvm-project?rev=254774&view=rev
>> > Log:
>> > [llc/opt] Add an option to run all passes twice
>> >
>> > Summary: Lately, I have submitted a number of patches to fix bugs that
>> > only occurred when using the same pass manager to compile multiple
>> > modules (generally these bugs are failure to reset some persistent
>> > state). Unfortunately I don't think there is currently a way to test
>> > that from the command line. This adds a very simple flag to both llc
>> > and opt, under which the tools will simply re-run their respective
>> > pass pipelines using the same pass manager on (a clone of the same
>> > module). Additionally, we verify that both outputs are bitwise the
>> > same.
>>
>> I'm getting a large number of Asan+Ubsan errors in opt calls from `ninja
>> check` after this change. See below.
>>
>> > Reviewers: yaron.keren
>> >
>> > Subscribers: loladiro, yaron.keren, kcc, llvm-commits
>> >
>> > Differential Revision: http://reviews.llvm.org/D14965
>> >
>> > Added:
>> > llvm/trunk/test/MC/ELF/empty-twice.ll
>> > llvm/trunk/test/Other/opt-twice.ll
>> > Modified:
>> > llvm/trunk/tools/llc/llc.cpp
>> > llvm/trunk/tools/opt/opt.cpp
>> >
>> > Added: llvm/trunk/test/MC/ELF/empty-twice.ll
>> > URL:
>> >
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/MC/ELF/empty-twice.ll?rev=254774&view=auto
>> >
>> ==============================================================================
>> > --- llvm/trunk/test/MC/ELF/empty-twice.ll (added)
>> > +++ llvm/trunk/test/MC/ELF/empty-twice.ll Fri Dec 4 15:56:46 2015
>> > @@ -0,0 +1,6 @@
>> > +; Check that there is no persistent state in the ELF emitter that
>> crashes us
>> > +; when we try to reuse the pass manager
>> > +; RUN: llc -compile-twice -filetype=obj %s -o -
>> > +
>> > +target datalayout =
>> "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:32:32-n8:16:32"
>> > +target triple = "i386-pc-linux-gnu"
>> >
>> > Added: llvm/trunk/test/Other/opt-twice.ll
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Other/opt-twice.ll?rev=254774&view=auto
>> >
>> ==============================================================================
>> > --- llvm/trunk/test/Other/opt-twice.ll (added)
>> > +++ llvm/trunk/test/Other/opt-twice.ll Fri Dec 4 15:56:46 2015
>> > @@ -0,0 +1,14 @@
>> > +; The pass here doesn't matter (we use deadargelim), but test
>> > +; that the -run-twice options exists, generates output, and
>> > +; doesn't crash
>> > +; RUN: opt -run-twice -deadargelim -S < %s | FileCheck %s
>> > +
>> > +; CHECK: define internal void @test
>> > +define internal {} @test() {
>> > + ret {} undef
>> > +}
>> > +
>> > +define void @caller() {
>> > + call {} @test()
>> > + ret void
>> > +}
>> >
>> > Modified: llvm/trunk/tools/llc/llc.cpp
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llc/llc.cpp?rev=254774&r1=254773&r2=254774&view=diff
>> >
>> ==============================================================================
>> > --- llvm/trunk/tools/llc/llc.cpp (original)
>> > +++ llvm/trunk/tools/llc/llc.cpp Fri Dec 4 15:56:46 2015
>> > @@ -45,6 +45,7 @@
>> > #include "llvm/Support/ToolOutputFile.h"
>> > #include "llvm/Target/TargetMachine.h"
>> > #include "llvm/Target/TargetSubtargetInfo.h"
>> > +#include "llvm/Transforms/Utils/Cloning.h"
>> > #include <memory>
>> > using namespace llvm;
>> >
>> > @@ -96,6 +97,12 @@ static cl::opt<bool> AsmVerbose("asm-ver
>> > cl::desc("Add comments to
>> directives."),
>> > cl::init(true));
>> >
>> > +static cl::opt<bool>
>> > + CompileTwice("compile-twice", cl::Hidden,
>> > + cl::desc("Run everything twice, re-using the same
>> pass "
>> > + "manager and verify the the result is the
>> same."),
>> > + cl::init(false));
>> > +
>> > static int compileModule(char **, LLVMContext &);
>> >
>> > static std::unique_ptr<tool_output_file>
>> > @@ -325,10 +332,15 @@ static int compileModule(char **argv, LL
>> >
>> > {
>> > raw_pwrite_stream *OS = &Out->os();
>> > - std::unique_ptr<buffer_ostream> BOS;
>> > - if (FileType != TargetMachine::CGFT_AssemblyFile &&
>> > - !Out->os().supportsSeeking()) {
>> > - BOS = make_unique<buffer_ostream>(*OS);
>> > +
>> > + // Manually do the buffering rather than using buffer_ostream,
>> > + // so we can memcmp the contents in CompileTwice mode
>> > + SmallVector<char, 0> Buffer;
>> > + std::unique_ptr<raw_svector_ostream> BOS;
>> > + if ((FileType != TargetMachine::CGFT_AssemblyFile &&
>> > + !Out->os().supportsSeeking()) ||
>> > + CompileTwice) {
>> > + BOS = make_unique<raw_svector_ostream>(Buffer);
>> > OS = BOS.get();
>> > }
>> >
>> > @@ -378,7 +390,39 @@ static int compileModule(char **argv, LL
>> > // Before executing passes, print the final values of the LLVM
>> options.
>> > cl::PrintOptionValues();
>> >
>> > + // If requested, run the pass manager over the same module again,
>> > + // to catch any bugs due to persistent state in the passes. Note
>> that
>> > + // opt has the same functionality, so it may be worth abstracting
>> this out
>> > + // in the future.
>> > + SmallVector<char, 0> CompileTwiceBuffer;
>> > + if (CompileTwice) {
>> > + std::unique_ptr<Module> M2(llvm::CloneModule(M.get()));
>> > + PM.run(*M2);
>> > + CompileTwiceBuffer = Buffer;
>> > + Buffer.clear();
>> > + }
>> > +
>> > PM.run(*M);
>> > +
>> > + // Compare the two outputs and make sure they're the same
>> > + if (CompileTwice) {
>> > + if (Buffer.size() != CompileTwiceBuffer.size() ||
>> > + (memcmp(Buffer.data(), CompileTwiceBuffer.data(),
>> Buffer.size()) !=
>> > + 0)) {
>> > + errs()
>> > + << "Running the pass manager twice changed the output.\n"
>> > + "Writing the result of the second run to the specified
>> output\n"
>> > + "To generate the one-run comparison binary, just run
>> without\n"
>> > + "the compile-twice option\n";
>> > + Out->os() << Buffer;
>> > + Out->keep();
>> > + return 1;
>> > + }
>> > + }
>> > +
>> > + if (BOS) {
>> > + Out->os() << Buffer;
>> > + }
>> > }
>> >
>> > // Declare success.
>> >
>> > Modified: llvm/trunk/tools/opt/opt.cpp
>> > URL:
>> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/opt/opt.cpp?rev=254774&r1=254773&r2=254774&view=diff
>> >
>> ==============================================================================
>> > --- llvm/trunk/tools/opt/opt.cpp (original)
>> > +++ llvm/trunk/tools/opt/opt.cpp Fri Dec 4 15:56:46 2015
>> > @@ -28,6 +28,7 @@
>> > #include "llvm/IR/DebugInfo.h"
>> > #include "llvm/IR/IRPrintingPasses.h"
>> > #include "llvm/IR/LLVMContext.h"
>> > +#include "llvm/IR/LegacyPassManager.h"
>> > #include "llvm/IR/LegacyPassNameParser.h"
>> > #include "llvm/IR/Module.h"
>> > #include "llvm/IR/Verifier.h"
>> > @@ -36,7 +37,6 @@
>> > #include "llvm/LinkAllIR.h"
>> > #include "llvm/LinkAllPasses.h"
>> > #include "llvm/MC/SubtargetFeature.h"
>> > -#include "llvm/IR/LegacyPassManager.h"
>> > #include "llvm/Support/Debug.h"
>> > #include "llvm/Support/FileSystem.h"
>> > #include "llvm/Support/Host.h"
>> > @@ -51,6 +51,7 @@
>> > #include "llvm/Support/ToolOutputFile.h"
>> > #include "llvm/Target/TargetMachine.h"
>> > #include "llvm/Transforms/IPO/PassManagerBuilder.h"
>> > +#include "llvm/Transforms/Utils/Cloning.h"
>> > #include <algorithm>
>> > #include <memory>
>> > using namespace llvm;
>> > @@ -190,6 +191,11 @@ static cl::opt<bool> PreserveAssemblyUse
>> > cl::desc("Preserve use-list order when writing LLVM assembly."),
>> > cl::init(false), cl::Hidden);
>> >
>> > +static cl::opt<bool>
>> > + RunTwice("run-twice",
>> > + cl::desc("Run all passes twice, re-using the same pass
>> manager."),
>> > + cl::init(false), cl::Hidden);
>> > +
>> > static inline void addPass(legacy::PassManagerBase &PM, Pass *P) {
>> > // Add the pass to the pass manager...
>> > PM.add(P);
>> > @@ -582,14 +588,25 @@ int main(int argc, char **argv) {
>> > if (!NoVerify && !VerifyEach)
>> > Passes.add(createVerifierPass());
>> >
>> > + // In run twice mode, we want to make sure the output is bit-by-bit
>> > + // equivalent if we run the pass manager again, so setup two buffers
>> and
>> > + // a stream to write to them. Note that llc does something similar
>> and it
>> > + // may be worth to abstract this out in the future.
>> > + SmallVector<char, 0> Buffer;
>> > + SmallVector<char, 0> CompileTwiceBuffer;
>> > + std::unique_ptr<raw_svector_ostream> BOS;
>> > + raw_ostream *OS = &Out->os();
>>
>> Apparently `Out` is sometimes (often?) null here. If you want/need to
>> investigate without having to build with sanitizers, just throw an
>> assert(Out) here.
>>
>> > + if (RunTwice) {
>> > + BOS = make_unique<raw_svector_ostream>(Buffer);
>> > + OS = BOS.get();
>> > + }
>> > +
>> > // Write bitcode or assembly to the output as the last step...
>> > if (!NoOutput && !AnalyzeOnly) {
>> > if (OutputAssembly)
>> > - Passes.add(
>> > - createPrintModulePass(Out->os(), "",
>> PreserveAssemblyUseListOrder));
>> > + Passes.add(createPrintModulePass(*OS, "",
>> PreserveAssemblyUseListOrder));
>> > else
>> > - Passes.add(
>> > - createBitcodeWriterPass(Out->os(),
>> PreserveBitcodeUseListOrder));
>> > + Passes.add(createBitcodeWriterPass(*OS,
>> PreserveBitcodeUseListOrder));
>> > }
>> >
>> > // Before executing passes, print the final values of the LLVM
>> options.
>> > @@ -598,6 +615,27 @@ int main(int argc, char **argv) {
>> > // Now that we have all of the passes ready, run them.
>> > Passes.run(*M);
>> >
>> > + // If requested, run all passes again with the same pass manager to
>> catch
>> > + // bugs caused by persistent state in the passes
>> > + if (RunTwice) {
>> > + CompileTwiceBuffer = Buffer;
>> > + Buffer.clear();
>> > + std::unique_ptr<Module> M2(CloneModule(M.get()));
>> > + Passes.run(*M2);
>> > + if (Buffer.size() != CompileTwiceBuffer.size() ||
>> > + (memcmp(Buffer.data(), CompileTwiceBuffer.data(),
>> Buffer.size()) !=
>> > + 0)) {
>> > + errs() << "Running the pass manager twice changed the output.\n"
>> > + "Writing the result of the second run to the specified
>> output."
>> > + "To generate the one-run comparison binary, just run
>> without\n"
>> > + "the compile-twice option\n";
>> > + Out->os() << BOS->str();
>> > + Out->keep();
>> > + return 1;
>> > + }
>> > + Out->os() << BOS->str();
>> > + }
>> > +
>> > // Declare success.
>> > if (!NoOutput || PrintBreakpoints)
>> > Out->keep();
>> >
>> >
>> > _______________________________________________
>> > llvm-commits mailing list
>> > llvm-commits at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151204/df249baf/attachment.html>
More information about the llvm-commits
mailing list