[llvm] r254774 - [llc/opt] Add an option to run all passes twice
Keno Fischer via llvm-commits
llvm-commits at lists.llvm.org
Fri Dec 4 15:58:58 PST 2015
Thanks,
I think I have a handle on this. Fix coming shortly.
On Fri, Dec 4, 2015 at 6:52 PM, Justin Bogner <mail at justinbogner.com> wrote:
> Keno Fischer via llvm-commits <llvm-commits at lists.llvm.org> writes:
> > Author: kfischer
> > Date: Fri Dec 4 15:56:46 2015
> > New Revision: 254774
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=254774&view=rev
> > Log:
> > [llc/opt] Add an option to run all passes twice
> >
> > Summary: Lately, I have submitted a number of patches to fix bugs that
> > only occurred when using the same pass manager to compile multiple
> > modules (generally these bugs are failure to reset some persistent
> > state). Unfortunately I don't think there is currently a way to test
> > that from the command line. This adds a very simple flag to both llc
> > and opt, under which the tools will simply re-run their respective
> > pass pipelines using the same pass manager on (a clone of the same
> > module). Additionally, we verify that both outputs are bitwise the
> > same.
>
> I'm getting a large number of Asan+Ubsan errors in opt calls from `ninja
> check` after this change. See below.
>
> > Reviewers: yaron.keren
> >
> > Subscribers: loladiro, yaron.keren, kcc, llvm-commits
> >
> > Differential Revision: http://reviews.llvm.org/D14965
> >
> > Added:
> > llvm/trunk/test/MC/ELF/empty-twice.ll
> > llvm/trunk/test/Other/opt-twice.ll
> > Modified:
> > llvm/trunk/tools/llc/llc.cpp
> > llvm/trunk/tools/opt/opt.cpp
> >
> > Added: llvm/trunk/test/MC/ELF/empty-twice.ll
> > URL:
> >
> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/MC/ELF/empty-twice.ll?rev=254774&view=auto
> >
> ==============================================================================
> > --- llvm/trunk/test/MC/ELF/empty-twice.ll (added)
> > +++ llvm/trunk/test/MC/ELF/empty-twice.ll Fri Dec 4 15:56:46 2015
> > @@ -0,0 +1,6 @@
> > +; Check that there is no persistent state in the ELF emitter that
> crashes us
> > +; when we try to reuse the pass manager
> > +; RUN: llc -compile-twice -filetype=obj %s -o -
> > +
> > +target datalayout =
> "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:32:64-f32:32:32-f64:32:64-v64:64:64-v128:128:128-a0:0:64-f80:32:32-n8:16:32"
> > +target triple = "i386-pc-linux-gnu"
> >
> > Added: llvm/trunk/test/Other/opt-twice.ll
> > URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Other/opt-twice.ll?rev=254774&view=auto
> >
> ==============================================================================
> > --- llvm/trunk/test/Other/opt-twice.ll (added)
> > +++ llvm/trunk/test/Other/opt-twice.ll Fri Dec 4 15:56:46 2015
> > @@ -0,0 +1,14 @@
> > +; The pass here doesn't matter (we use deadargelim), but test
> > +; that the -run-twice options exists, generates output, and
> > +; doesn't crash
> > +; RUN: opt -run-twice -deadargelim -S < %s | FileCheck %s
> > +
> > +; CHECK: define internal void @test
> > +define internal {} @test() {
> > + ret {} undef
> > +}
> > +
> > +define void @caller() {
> > + call {} @test()
> > + ret void
> > +}
> >
> > Modified: llvm/trunk/tools/llc/llc.cpp
> > URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/llc/llc.cpp?rev=254774&r1=254773&r2=254774&view=diff
> >
> ==============================================================================
> > --- llvm/trunk/tools/llc/llc.cpp (original)
> > +++ llvm/trunk/tools/llc/llc.cpp Fri Dec 4 15:56:46 2015
> > @@ -45,6 +45,7 @@
> > #include "llvm/Support/ToolOutputFile.h"
> > #include "llvm/Target/TargetMachine.h"
> > #include "llvm/Target/TargetSubtargetInfo.h"
> > +#include "llvm/Transforms/Utils/Cloning.h"
> > #include <memory>
> > using namespace llvm;
> >
> > @@ -96,6 +97,12 @@ static cl::opt<bool> AsmVerbose("asm-ver
> > cl::desc("Add comments to directives."),
> > cl::init(true));
> >
> > +static cl::opt<bool>
> > + CompileTwice("compile-twice", cl::Hidden,
> > + cl::desc("Run everything twice, re-using the same pass
> "
> > + "manager and verify the the result is the
> same."),
> > + cl::init(false));
> > +
> > static int compileModule(char **, LLVMContext &);
> >
> > static std::unique_ptr<tool_output_file>
> > @@ -325,10 +332,15 @@ static int compileModule(char **argv, LL
> >
> > {
> > raw_pwrite_stream *OS = &Out->os();
> > - std::unique_ptr<buffer_ostream> BOS;
> > - if (FileType != TargetMachine::CGFT_AssemblyFile &&
> > - !Out->os().supportsSeeking()) {
> > - BOS = make_unique<buffer_ostream>(*OS);
> > +
> > + // Manually do the buffering rather than using buffer_ostream,
> > + // so we can memcmp the contents in CompileTwice mode
> > + SmallVector<char, 0> Buffer;
> > + std::unique_ptr<raw_svector_ostream> BOS;
> > + if ((FileType != TargetMachine::CGFT_AssemblyFile &&
> > + !Out->os().supportsSeeking()) ||
> > + CompileTwice) {
> > + BOS = make_unique<raw_svector_ostream>(Buffer);
> > OS = BOS.get();
> > }
> >
> > @@ -378,7 +390,39 @@ static int compileModule(char **argv, LL
> > // Before executing passes, print the final values of the LLVM
> options.
> > cl::PrintOptionValues();
> >
> > + // If requested, run the pass manager over the same module again,
> > + // to catch any bugs due to persistent state in the passes. Note
> that
> > + // opt has the same functionality, so it may be worth abstracting
> this out
> > + // in the future.
> > + SmallVector<char, 0> CompileTwiceBuffer;
> > + if (CompileTwice) {
> > + std::unique_ptr<Module> M2(llvm::CloneModule(M.get()));
> > + PM.run(*M2);
> > + CompileTwiceBuffer = Buffer;
> > + Buffer.clear();
> > + }
> > +
> > PM.run(*M);
> > +
> > + // Compare the two outputs and make sure they're the same
> > + if (CompileTwice) {
> > + if (Buffer.size() != CompileTwiceBuffer.size() ||
> > + (memcmp(Buffer.data(), CompileTwiceBuffer.data(),
> Buffer.size()) !=
> > + 0)) {
> > + errs()
> > + << "Running the pass manager twice changed the output.\n"
> > + "Writing the result of the second run to the specified
> output\n"
> > + "To generate the one-run comparison binary, just run
> without\n"
> > + "the compile-twice option\n";
> > + Out->os() << Buffer;
> > + Out->keep();
> > + return 1;
> > + }
> > + }
> > +
> > + if (BOS) {
> > + Out->os() << Buffer;
> > + }
> > }
> >
> > // Declare success.
> >
> > Modified: llvm/trunk/tools/opt/opt.cpp
> > URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/opt/opt.cpp?rev=254774&r1=254773&r2=254774&view=diff
> >
> ==============================================================================
> > --- llvm/trunk/tools/opt/opt.cpp (original)
> > +++ llvm/trunk/tools/opt/opt.cpp Fri Dec 4 15:56:46 2015
> > @@ -28,6 +28,7 @@
> > #include "llvm/IR/DebugInfo.h"
> > #include "llvm/IR/IRPrintingPasses.h"
> > #include "llvm/IR/LLVMContext.h"
> > +#include "llvm/IR/LegacyPassManager.h"
> > #include "llvm/IR/LegacyPassNameParser.h"
> > #include "llvm/IR/Module.h"
> > #include "llvm/IR/Verifier.h"
> > @@ -36,7 +37,6 @@
> > #include "llvm/LinkAllIR.h"
> > #include "llvm/LinkAllPasses.h"
> > #include "llvm/MC/SubtargetFeature.h"
> > -#include "llvm/IR/LegacyPassManager.h"
> > #include "llvm/Support/Debug.h"
> > #include "llvm/Support/FileSystem.h"
> > #include "llvm/Support/Host.h"
> > @@ -51,6 +51,7 @@
> > #include "llvm/Support/ToolOutputFile.h"
> > #include "llvm/Target/TargetMachine.h"
> > #include "llvm/Transforms/IPO/PassManagerBuilder.h"
> > +#include "llvm/Transforms/Utils/Cloning.h"
> > #include <algorithm>
> > #include <memory>
> > using namespace llvm;
> > @@ -190,6 +191,11 @@ static cl::opt<bool> PreserveAssemblyUse
> > cl::desc("Preserve use-list order when writing LLVM assembly."),
> > cl::init(false), cl::Hidden);
> >
> > +static cl::opt<bool>
> > + RunTwice("run-twice",
> > + cl::desc("Run all passes twice, re-using the same pass
> manager."),
> > + cl::init(false), cl::Hidden);
> > +
> > static inline void addPass(legacy::PassManagerBase &PM, Pass *P) {
> > // Add the pass to the pass manager...
> > PM.add(P);
> > @@ -582,14 +588,25 @@ int main(int argc, char **argv) {
> > if (!NoVerify && !VerifyEach)
> > Passes.add(createVerifierPass());
> >
> > + // In run twice mode, we want to make sure the output is bit-by-bit
> > + // equivalent if we run the pass manager again, so setup two buffers
> and
> > + // a stream to write to them. Note that llc does something similar
> and it
> > + // may be worth to abstract this out in the future.
> > + SmallVector<char, 0> Buffer;
> > + SmallVector<char, 0> CompileTwiceBuffer;
> > + std::unique_ptr<raw_svector_ostream> BOS;
> > + raw_ostream *OS = &Out->os();
>
> Apparently `Out` is sometimes (often?) null here. If you want/need to
> investigate without having to build with sanitizers, just throw an
> assert(Out) here.
>
> > + if (RunTwice) {
> > + BOS = make_unique<raw_svector_ostream>(Buffer);
> > + OS = BOS.get();
> > + }
> > +
> > // Write bitcode or assembly to the output as the last step...
> > if (!NoOutput && !AnalyzeOnly) {
> > if (OutputAssembly)
> > - Passes.add(
> > - createPrintModulePass(Out->os(), "",
> PreserveAssemblyUseListOrder));
> > + Passes.add(createPrintModulePass(*OS, "",
> PreserveAssemblyUseListOrder));
> > else
> > - Passes.add(
> > - createBitcodeWriterPass(Out->os(),
> PreserveBitcodeUseListOrder));
> > + Passes.add(createBitcodeWriterPass(*OS,
> PreserveBitcodeUseListOrder));
> > }
> >
> > // Before executing passes, print the final values of the LLVM
> options.
> > @@ -598,6 +615,27 @@ int main(int argc, char **argv) {
> > // Now that we have all of the passes ready, run them.
> > Passes.run(*M);
> >
> > + // If requested, run all passes again with the same pass manager to
> catch
> > + // bugs caused by persistent state in the passes
> > + if (RunTwice) {
> > + CompileTwiceBuffer = Buffer;
> > + Buffer.clear();
> > + std::unique_ptr<Module> M2(CloneModule(M.get()));
> > + Passes.run(*M2);
> > + if (Buffer.size() != CompileTwiceBuffer.size() ||
> > + (memcmp(Buffer.data(), CompileTwiceBuffer.data(),
> Buffer.size()) !=
> > + 0)) {
> > + errs() << "Running the pass manager twice changed the output.\n"
> > + "Writing the result of the second run to the specified
> output."
> > + "To generate the one-run comparison binary, just run
> without\n"
> > + "the compile-twice option\n";
> > + Out->os() << BOS->str();
> > + Out->keep();
> > + return 1;
> > + }
> > + Out->os() << BOS->str();
> > + }
> > +
> > // Declare success.
> > if (!NoOutput || PrintBreakpoints)
> > Out->keep();
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151204/6489df24/attachment-0001.html>
More information about the llvm-commits
mailing list