[Lldb-commits] [PATCH] D89600: [lldb] Move copying of reproducer out of process

[Pavel Labath via Phabricator via lldb-commits]

labath added a comment.

In D89600#2338996 <https://reviews.llvm.org/D89600#2338996>, @JDevlieghere wrote:

> In D89600#2337868 <https://reviews.llvm.org/D89600#2337868>, @labath wrote:
>
>> Hm... Are you sure that this will help with anything? IIUC, the serialization of the VFS file map still happens in the context of the crashing process. This requires walking a bunch of data structures with liberally-asserting code, which will abort on any inconsistency it runs into. If *that* succeeds, I'd expect that the copying of the files will succeed too.
>
> I only have a handful of crash logs to base myself off. Currently the walking and the copying are intertwined, so I can't say for sure. My idea was to give this a try and see where that got us and make more invasive changes (like what you proposed) if needed. But since you sound pretty convinced that we'll need it anyway I will extend the patch.

Well.. I'd expect that the reason why saving of the reproducer fails is because we corrupt the heap/stack before we finally crash. Copying files is basically a bunch of syscalls (though some of our abstractions get in the way), so it wouldn't normally be affected by this. Our ability to _find_ the files that we're supposed to copy is a different matter...

That said, this doesn't seem significantly more invasive than the previous approach.

>> I think it would be better if we moved the vfs map construction into the other process too. Maybe via something like this:
>>
>> - The first process opens a "log" file, which will contain the list of files to be copied. For added safety the file should be opened append-only.
>> - As it works, it writes file names to this file.
>> - Upon a crash, the re-execed process reads this file, constructs the vfs map, and copies all necessary files. It should be prepared to handle/ignore any garbled entries, and the format of the file should make this easy. (Since the log file must by definition contain enough information to recreate the vfs map, it might be simpler to not write out the map, but just always recreate it in memory when replaying.)
>
> Yes, that would be similar to how we immediately flush GDB remote packets to disk. It would require adding some "immediate" mode to the FileCollector and an API to convert between the formats. I think the new format could be just host paths separated by newlines.

A nul character is better than a newline, as paths can contain newline characters.



================
Comment at: lldb/include/lldb/Utility/Reproducer.h:241-256
+class Finalizer {
+public:
+  Finalizer(Loader *loader) : m_loader(loader) {}
+  llvm::Error Finalize() const;
+
+protected:
+  Loader *m_loader;
----------------
This appears to be over-engineered. Is there a reason plain functions won't do?
```
llvm::Error FinalizeReproducer(Loader&);
llvm::Error FinalizeReproducerInProcess(FileSpec); // calls the first one
```
That said, I'm not sure "InProcess" captures the distinction very well, as both are "in process" at the point where they are used. Maybe name them according to what they do (as in, whether they generate a full reproducer, or one that needs to be fixed up afterwards)


================
Comment at: lldb/source/Utility/Reproducer.cpp:389
+
+  if (auto files = llvm::MemoryBuffer::getFile(files_path)) {
+    SmallVector<StringRef, 20> paths;
----------------
Is this deliberately ignoring a possible error?


================
Comment at: lldb/source/Utility/ReproducerProvider.cpp:53
+  std::error_code ec;
+  llvm::raw_fd_ostream os(m_files_path, ec, llvm::sys::fs::OF_Append);
+  if (ec)
----------------
So, this reopens the file every time a file is added? That is suboptimal... Why not just open it once and keep it that way? That would also enable handling of error during opening of files.


================
Comment at: lldb/source/Utility/ReproducerProvider.cpp:65-67
+  if (ec)
+    return vfs->dir_begin(dir, dir_ec);
+  os << dir << "\n";
----------------
`if(!ec) os<<dir;` ?


================
Comment at: lldb/test/Shell/Reproducer/TestCaptureEnvOverride.test:1
 # This tests the LLDB_CAPTURE_REPRODUCER override.
 
----------------
Any chance of testing the finalization code by letting it loose on an hand-generated unfinalized reproducer?


================
Comment at: lldb/tools/driver/Driver.cpp:734
   if (SBReproducer::Generate()) {
-    auto exe = static_cast<const char *>(argv0);
-    llvm::outs() << "********************\n";
-    llvm::outs() << "Crash reproducer for ";
-    llvm::outs() << lldb::SBDebugger::GetVersionString() << '\n';
-    llvm::outs() << '\n';
-    llvm::outs() << "Reproducer written to '" << SBReproducer::GetPath()
-                 << "'\n";
-    llvm::outs() << '\n';
-    llvm::outs() << "Before attaching the reproducer to a bug report:\n";
-    llvm::outs() << " - Look at the directory to ensure you're willing to "
-                    "share its content.\n";
-    llvm::outs()
-        << " - Make sure the reproducer works by replaying the reproducer.\n";
-    llvm::outs() << '\n';
-    llvm::outs() << "Replay the reproducer with the following command:\n";
-    llvm::outs() << exe << " -replay " << SBReproducer::GetPath() << "\n";
-    llvm::outs() << "********************\n";
+    std::system(static_cast<const char *>(cmd));
+    fflush(stdout);
----------------
this is really cryptic. Maybe renaming cmd to finalize_cmd would help?


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89600/new/

https://reviews.llvm.org/D89600