MIME-Version: 1.0
In-Reply-To: <CACh7GpHFY_KUhhtk09H_oCzBtRh66artDCqz8pXNTh_ZzkAABg@mail.gmail.com>
References: <5061F8CC.9070906@mistfpga.net>
	<1348605677.2284.2.camel@localhost.localdomain>
	<5062F4F8.6040504@mistfpga.net>
	<CA+s+GJBM4DwDoqT8RC0+SyrLYrLGZuGZSuoj7zbHunQa3kFoRA@mail.gmail.com>
	<506301AC.90101@mistfpga.net>
	<CACh7GpHFY_KUhhtk09H_oCzBtRh66artDCqz8pXNTh_ZzkAABg@mail.gmail.com>
Date: Wed, 26 Sep 2012 13:10:31 -0400
Message-ID: <CA+8xBpe_QC-U=HjvMQa-+RkFKZJvdU4Gq0NJZXrme=Uh-rns+w@mail.gmail.com>
From: Jeff Garzik <jgarzik@exmulti.com>
To: Mark Friedenbach <mark@monetize.io>
Content-Type: text/plain; charset=ISO-8859-1
Cc: Bitcoin Development List <bitcoin-development@lists.sourceforge.net>,
	Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Precedence: list

On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach <mark@monetize.io> wrote:
> Certainly developers should be responsible for making sure that regression
> tests for bugs they fix make it either into the unit tests or Matt's
> functional test repository. QA should hold them accountable for that
> (re-opening tickets for bugs that have been fixed but without regression
> tests).

As a goal or general principle, this is agreeable.

But slavish attention to this will only get ignored.  There is finite
developer resources, and regression tests for certain types of bugs,
like prickly P2P network interaction bugs or RPC API bugs, could
potentially involve many days or weeks of coding, to sufficiently
simulate the environment.  The ability to easily, automatically and
programmatically reproduce certain classes of bugs is simply out of
reach right now, and nobody is going to shut down development to fix
that problem.

We should move towards this direction, yes, but bitcoin test cases are
not always going to be as easy as writing (say) a compiler testcase.

We can always use the help of a few good QA coders:  simulating a P2P
environment and checking the RPC API are two examples of very
complicated problems that -can- be automated for testing... with a lot
of work.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com