[PATCH] D116787: [llvm-readobj][MachO] Add option to sort the symbol table before dumping (MachO only, for now).

[James Henderson via Phabricator via llvm-commits]

jhenderson added inline comments.


================
Comment at: llvm/test/tools/llvm-readobj/sort-symbols.test:16
+# CHECK: Format: aixcoff-rs6000
+# CHECK-NOT: warning '{{.+}}_macho': --sort-symbols is not supported yet for this format.
+# CHECK: Format: Mach-O 64-bit x86-64
----------------
oontvoo wrote:
> oontvoo wrote:
> > oontvoo wrote:
> > > oontvoo wrote:
> > > > jhenderson wrote:
> > > > > oontvoo wrote:
> > > > > > jhenderson wrote:
> > > > > > > I'd consider pruning this back to just `warning '{{.+}}_macho':` to reduce the risk of false negatives due to a slight change in the warning message.
> > > > > > Wouldn't that make the test more prone to false positives? that is, if some new warning pops up somewhere else, this would trip. So i'm going to keep this.
> > > > > Yes, it would, but we probably shouldn't be invoking behaviour that causes those warnings anyway, so they're harmless (it would be easy to fix the test if it did trigger one in the future).
> > > > > 
> > > > > The test as it was before the last edit demonstrates why overly strict CHECK-NOTs are not much use, because typos can cause them to pass spuriously.
> > > > But that's not this test's job to guard against *OTHER* kinds of warnings.
> > > Furthermore, false negatives/brittle tests are just as frustrating.
> > > Yes, it would, but we probably shouldn't be invoking behaviour that causes those warnings anyway, so they're harmless (it would be easy to fix the test if it did trigger one in the future).
> > 
> > I disagree with the assertion that it's easy to fix these. Imagine there were a dozen tests similar to this one which were not expecting some  warnings, then someone added a  new warning and they would have to go update all these tests, even though it's not their fault. (it is the test's fault that it casts too wide a nest on the warning).
> > 
> s/nest/net
> 
> Do you have any other comment on this patch because it seems we've been back on forth for a very long time and it doesn't seem to get any more progress ...
@MaskRay, any thoughts on this or other aspectse of this patch?

As someone who has been stung way too many times by rotten tests caused by negative matches like this not actually catching the thing I'm expecting to be caught, I'm incredibly wary of strict -NOT patterns like this. This isn't some arbitrary concern: I've seen bugs in released products because of this exact kind of overly precise check pattern.

That being said, there is an alternative approach I think you could consider: stick the message after the colon in a FileCheck define and then use it in both the positive and negative matches. That way, if the message is changed, the positive matches will start failing, prompting the developer to update that check too, which in turn will ensure the CHECK-NOT doesn't rot (since it's testing guaranteed to be testing the exact same string).
```
# RUN: ... | FileCheck %s -DMSG="--sort-symbols is not supported yet for this format"

# CHECK: warning: '{{.+}}_coff': [[MSG]]
...
# CHECK-NOT: warning: '{{.+}}': [[MSG]]
```
(NB: I've left the file path loose in the negative match so that if somebody changes the input file name, the check pattern is still valid).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D116787/new/

https://reviews.llvm.org/D116787