<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Ana,<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 24, 2018, at 13:05, <a href="mailto:apazos@codeaurora.org" class="">apazos@codeaurora.org</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi Daniel,<br class=""><br class="">Thanks for the feedback.<br class=""><br class="">That is correct, you can invoke the fuzzers without a golden reference implementation. The driver program to compare behaviors is just a convenient tool for those who have a reference implementation.<br class=""><br class="">I am not sure I understood your suggestion about de-serializing Protobuf messages as MCInst objects. Can you clarify?<br class=""></div></div></blockquote><div><br class=""></div><div>I'm thinking that a single protobuf message could be used to generate the Assembly, MCInst, and Encoding for a given instruction. From that you can:</div><div><ul class="MailOutline"><li class="">Parse the generated assembly and check the intermediate MCInst matches the generated MCInst</li><li class="">Disassemble the generated encoding and check the intermediate MCInst matches the generated MCInst</li><li class="">Assemble the generated MCInst and check the encoding matches the generated encoding</li><li class="">Disassemble the generated MCInst and check the assembly matches the generated assembly</li></ul></div><div>in addition to testing the Assembly->Encoding and Encoding->Assembly. With these additional tests, the fuzzer can identify whether it's the parsing, assembling, disassembling, or instruction printing that is incorrect.</div><div><br class=""></div><div>For example, given 'addiu $1, $2, 3', 'ADD reg(1), reg(2), imm(3)', and 0x0123 the fuzzer would be able to report something like:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>0x0123 didn't match golden reference (0x0123 != 0x0126)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parsing 'addiu $1, $2, 3' produces 'ADD reg(1), reg(2), imm(3)': Parser is ok</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>Assembling 'ADD reg(1), reg(2), imm(3)' produces '0x0126': Assembler is wrong</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>Disassembling '0x0123' produces 'ADD reg(1), reg(2), imm(3)' produces '0x0124': Disassembler is ok</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>Printing 'ADD reg(1), reg(2), imm(3)' produces 'addiu $1, $2, 3': Instruction printer is ok</div><div><br class=""></div><div>I'm also thinking that fuzzing the MCInst->Assembly, and MCInst->Encoding paths in isolation would allow us to be sure that the MCInst's that are emitted by CodeGen behave correctly when given to the MC layer. At the moment, we assume that Assembly->MCInst->Encoding and Encoding->MCInst->Assembly being correct implies that CodeGen->MCInst->Assembly and CodeGen->MCInst->Encoding are correct too. However, this doesn't quite hold as there are multiple ways of expressing the same operand in an MCInst. For example:</div><div><ul class="MailOutline"><li class="">addExpr(MCConstantExpr::create(2, Ctx))</li><li class="">addImm(2)</li><li class="">addExpr(/* some REL relocation evaluating to Symbol+2 */)</li></ul></div><div><div class="">and Assembly->MCInst->Encoding/Encoding->MCInst->Assembly will only exercise one of them.</div></div><div><br class=""></div></div><div><blockquote type="cite" class=""><div class=""><div class="">Thanks,<br class="">Ana.<br class=""><br class="">On 2018-10-16 10:34, Daniel Sanders wrote:<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">On 16 Oct 2018, at 10:09, Daniel Sanders via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class="">Hi Ana,<br class="">I think this looks interesting although unfortunately I'm not sure I'm going to be able to make use of it for my current target as I don't have a golden reference tool available.<br class=""></blockquote>Thinking about it a bit more, the lack of a golden reference only<br class="">really affects my ability to use the driver script. With a<br class="">different/modified driver I should be able to use the underlying<br class="">fuzzer without a reference tool available.<br class=""><blockquote type="cite" class="">One of the key weaknesses of llvm-mc-disassembler-fuzzer for most targets is that it only finds a corpus of tests that improve coverage but doesn't provide any assessment on what the correct behaviour is. A human is required to make proper test cases out of the corpus and feed it back in so the fuzzer can drop the corresponding generated tests. Having a fuzzer that can verify the behaviour as well would be very useful for targets with access to a golden reference tool.<br class="">One thing that occurred to me while skimming through D51144 was that something similar to proto_to_asm_main.cpp could be used to generate MCInst objects directly from the same protobuf. This would allow you to attribute bugs to the parser, instruction printer, or object emitter since you'd be able to tell, for example, that the parser emitted the an MCInst that matched the one expected by the protobuf.<br class=""><blockquote type="cite" class="">On 15 Oct 2018, at 12:29, via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class="">Hello,<br class="">We have implemented LLVM Machine Code Protobuf fuzzers for the RISC-V target as part of a Summer internship project with our intern Jocelyn Wei.<br class="">The fuzzers for the assembler and disassembler proved to be useful. We uncovered bugs and detected compatibility issues with other tools, e.g., by running a driver program that implements a round trip with a golden (i.e., more tested) tool such as GNU AS.<br class="">We built different fuzzer versions to experiment with the level of fuzzing for the instruction operands.<br class="">The versions are labeled sample, semi-constrained, unconstrained. We fix opcodes, and depending on the fuzzer version, allow number of operands, operand value ranges, and operand types to vary.<br class="">The code is available for review:<br class=""><a href="https://reviews.llvm.org/D51710" class="">https://reviews.llvm.org/D51710</a> Implemented Protobuf fuzzer for LLVM RISC-V MC Disassembler<br class=""><a href="https://reviews.llvm.org/D51144" class="">https://reviews.llvm.org/D51144</a> Implemented Protobuf fuzzer for LLVM RISC-V MC Assembler<br class="">We would like to assess people's interest in adding this type of tool to the LLVM code base.<br class="">It can be further improved for RISC-V target and also expanded to other targets.<br class="">We have a Poster about the fuzzers at the LLVM Dev Conf this week.<br class="">Please visit our poster and come by with your comments and suggestions. We appreciate your feebdack.<br class="">Thank you,<br class="">Ana.<br class="">--<br class="">Ana Pazos<br class="">Qualcomm Innovation Center, Inc.<br class="">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,<br class="">a Linux Foundation Collaborative Project.<br class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></blockquote>_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></blockquote></blockquote><br class="">-- <br class="">Ana Pazos<br class="">Qualcomm Innovation Center, Inc.<br class="">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,<br class="">a Linux Foundation Collaborative Project.<br class=""></div></div></blockquote></div><br class=""></body></html>