Skip to content

proposal: cmd/go: make fuzzing a first class citizen, like tests or benchmarks #19109

Closed
@bradfitz

Description

@bradfitz
Contributor

Filing a proposal on behalf of @kcc and @dvyukov:

They request that cmd/go support fuzzing natively, just like it does tests and benchmarks and race detection today.

https://github.com/dvyukov/go-fuzz exists but it's not as easy as writing tests and benchmarks and running "go test -race" today.

Should we make this easier?

Motivation
Proposal

Activity

added this to the Proposal milestone on Feb 15, 2017
ianlancetaylor

ianlancetaylor commented on Feb 15, 2017

@ianlancetaylor
Contributor

I think it would be easier to evaluate the idea if it were slightly less abstract.

For example:

  • _test.go are permitted to contain functions of the form FuzzXxx(f *testing.F, data []byte)
  • these functions are expected to run some test based on the random bytes in data
  • errors are reported using the testing.F argument in the usual way
  • f.Useful() may be called to indicate useful data, i.e., data that parses correctly
  • f.Discard() may be called to indicate that the data should be discarded
  • go test -fuzz=. runs the fuzz functions using a regexp like -test and -bench
  • naturally go test -fuzz must also rebuild the package in fuzz mode
  • the data is cached somewhere under $GOROOT/pkg, but where?
bradfitz

bradfitz commented on Feb 15, 2017

@bradfitz
ContributorAuthor

@ianlancetaylor, yes, FuzzXxx(f *testing.F, ...) is what this is about. The exact API is probably TBD.

I think the first step before it's designed completely is to determine whether there's interest.

ianlancetaylor

ianlancetaylor commented on Feb 15, 2017

@ianlancetaylor
Contributor

As a general concept, I'm in favor.

dsnet

dsnet commented on Feb 15, 2017

@dsnet
Member

I would expect that there would be an additional required flag (when fuzzing) where you specify the corpus directory.

ianlancetaylor

ianlancetaylor commented on Feb 15, 2017

@ianlancetaylor
Contributor

Can we just cache the corpus somewhere under $GOROOT/pkg? Are there cases where a typical user would be expected to modify the corpus themselves?

dsnet

dsnet commented on Feb 15, 2017

@dsnet
Member

I think it's wrong to think of the corpus as strictly a cache. The corpus is the save state of the fuzzer and the documentation for go-fuzz even recommends committing them into a version control system. The pkg directory is treated strictly as cache and it is not uncommon for people to recommend clearing out the directory, which will unfortunately delete the fuzzer state.

A specified corpus is not so much for the user modify the corpus themselves, but for them to specify how to persist the corpus data.

jimmyfrasche

jimmyfrasche commented on Feb 15, 2017

@jimmyfrasche
Member

Could there be some default convention say a _fuzz/xxx directory (where xxx corresponds with FuzzXxx) and a method on the *testing.F object to load a different corpus from the _fuzz/ directory if necessary? It seems like it should just know where the corpus is.

minux

minux commented on Feb 15, 2017

@minux
Member
cznic

cznic commented on Feb 16, 2017

@cznic
Contributor

Quoting @dvyukov

I would appreciate if you drop a line there if you found fuzzing useful and a brief of your success story.

It was very useful for me - found bugs in several lexers.

mvdan

mvdan commented on Feb 16, 2017

@mvdan
Member

I use it regularly on a lexer/parser/formatter for Bash (https://github.com/mvdan/sh).

Having it be a first-class citizen would simplify things for me and for contributors.

dsnet

dsnet commented on Feb 16, 2017

@dsnet
Member

Found a bug in the C decoder for google/brotli by fuzzing a Go implementation of a Brotli decoder.

Also found some divergences in Go bzip2 decoders from the canonical C decoder (this and #18516). All by fuzzing.

fatih

fatih commented on Feb 16, 2017

@fatih
Member

My coworker at DigitalOcean was working on a side project to make fuzzing easier. Check his repo out here: https://github.com/tam7t/cautious-pancake Adding it here as I think it would be a valuable piece of information for this discussion.

168 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @bradfitz@webRat@josharian@flyingmutant@rsc

        Issue actions

          proposal: cmd/go: make fuzzing a first class citizen, like tests or benchmarks · Issue #19109 · golang/go