[llvm-dev] [RFC] Implementing LLVM MC Protobuf Fuzzer for Assembly and Encoding for RISC-V target

[via llvm-dev]

Hi Daniel,

Thanks for the feedback.

That is correct, you can invoke the fuzzers without a golden reference 
implementation. The driver program to compare behaviors is just a 
convenient tool for those who have a reference implementation.

I am not sure I understood your suggestion about de-serializing Protobuf 
messages as MCInst objects. Can you clarify?

Thanks,
Ana.

On 2018-10-16 10:34, Daniel Sanders wrote:
>> On 16 Oct 2018, at 10:09, Daniel Sanders via llvm-dev 
>> <llvm-dev at lists.llvm.org> wrote:
>> 
>> Hi Ana,
>> 
>> I think this looks interesting although unfortunately I'm not sure I'm 
>> going to be able to make use of it for my current target as I don't 
>> have a golden reference tool available.
> 
> Thinking about it a bit more, the lack of a golden reference only
> really affects my ability to use the driver script. With a
> different/modified driver I should be able to use the underlying
> fuzzer without a reference tool available.
> 
>> One of the key weaknesses of llvm-mc-disassembler-fuzzer for most 
>> targets is that it only finds a corpus of tests that improve coverage 
>> but doesn't provide any assessment on what the correct behaviour is. A 
>> human is required to make proper test cases out of the corpus and feed 
>> it back in so the fuzzer can drop the corresponding generated tests. 
>> Having a fuzzer that can verify the behaviour as well would be very 
>> useful for targets with access to a golden reference tool.
>> 
>> One thing that occurred to me while skimming through D51144 was that 
>> something similar to proto_to_asm_main.cpp could be used to generate 
>> MCInst objects directly from the same protobuf. This would allow you 
>> to attribute bugs to the parser, instruction printer, or object 
>> emitter since you'd be able to tell, for example, that the parser 
>> emitted the an MCInst that matched the one expected by the protobuf.
>> 
>>> On 15 Oct 2018, at 12:29, via llvm-dev <llvm-dev at lists.llvm.org> 
>>> wrote:
>>> 
>>> 
>>> Hello,
>>> 
>>> We have implemented LLVM Machine Code Protobuf fuzzers for the RISC-V 
>>> target as part of a Summer internship project with our intern Jocelyn 
>>> Wei.
>>> 
>>> The fuzzers for the assembler and disassembler proved to be useful. 
>>> We uncovered bugs and detected compatibility issues with other tools, 
>>> e.g., by running a driver program that implements a round trip with a 
>>> golden (i.e., more tested) tool such as GNU AS.
>>> 
>>> We built different fuzzer versions to experiment with the level of 
>>> fuzzing for the instruction operands.
>>> The versions are labeled sample, semi-constrained, unconstrained. We 
>>> fix opcodes, and depending on the fuzzer version, allow number of 
>>> operands, operand value ranges, and operand types to vary.
>>> 
>>> The code is available for review:
>>> https://reviews.llvm.org/D51710 Implemented Protobuf fuzzer for LLVM 
>>> RISC-V MC Disassembler
>>> https://reviews.llvm.org/D51144 Implemented Protobuf fuzzer for LLVM 
>>> RISC-V MC Assembler
>>> 
>>> We would like to assess people's interest in adding this type of tool 
>>> to the LLVM code base.
>>> 
>>> It can be further improved for RISC-V target and also expanded to 
>>> other targets.
>>> 
>>> We have a Poster about the fuzzers at the LLVM Dev Conf this week.
>>> 
>>> Please visit our poster and come by with your comments and 
>>> suggestions. We appreciate your feebdack.
>>> 
>>> Thank you,
>>> Ana.
>>> 
>>> --
>>> Ana Pazos
>>> Qualcomm Innovation Center, Inc.
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>>> a Linux Foundation Collaborative Project.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
Ana Pazos
Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project.