1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
|
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <jgarzik@exmulti.com>) id 1TGv8E-0005en-H6
for bitcoin-development@lists.sourceforge.net;
Wed, 26 Sep 2012 17:10:42 +0000
X-ACL-Warn:
Received: from mail-qc0-f175.google.com ([209.85.216.175])
by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1TGv89-0006Vc-0T
for bitcoin-development@lists.sourceforge.net;
Wed, 26 Sep 2012 17:10:42 +0000
Received: by qcad10 with SMTP id d10so757037qca.34
for <bitcoin-development@lists.sourceforge.net>;
Wed, 26 Sep 2012 10:10:31 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=google.com; s=20120113;
h=mime-version:x-originating-ip:in-reply-to:references:date
:message-id:subject:from:to:cc:content-type:x-gm-message-state;
bh=gEOGK3lLDrAzobHcfIGr6i7BVOv6XQW8g5/AGgOk3/c=;
b=ACeYx3ounPO1wvtNMIb5TI7ytXrOrJ7hJy+z7qBUyZlNMB77EkcX9H1Y+w8tkLi1tP
SIg46ihnlHZdoDi9DQ98Zs1EmImMfxh52+RMLIlKHq2ol0yXtlQu7A3TrWqL3uQWsdqu
fCFtOYD8M6JF2taulP+5xr/8Vnq5degp1C+hBGNm5mZGVAxSw5ZJHl3WL+vXVTD7xBOZ
isF+57dmCu1z/Uop4suByoR726Sna89A0SudZ+ViBBS3BF8UgjmXCoXiJb1iwsGQDT7L
5YeggOVxWN2Cf8lNubXttKfRZ4rCUtRwNH7f24p892BHVS6wjclT3JuO6wBi3ZzVVTZn
Movw==
MIME-Version: 1.0
Received: by 10.224.70.138 with SMTP id d10mr3538694qaj.12.1348679431459; Wed,
26 Sep 2012 10:10:31 -0700 (PDT)
Received: by 10.49.97.6 with HTTP; Wed, 26 Sep 2012 10:10:31 -0700 (PDT)
X-Originating-IP: [2001:4830:1603:2:21c:c0ff:fe79:c8c2]
In-Reply-To: <CACh7GpHFY_KUhhtk09H_oCzBtRh66artDCqz8pXNTh_ZzkAABg@mail.gmail.com>
References: <5061F8CC.9070906@mistfpga.net>
<1348605677.2284.2.camel@localhost.localdomain>
<5062F4F8.6040504@mistfpga.net>
<CA+s+GJBM4DwDoqT8RC0+SyrLYrLGZuGZSuoj7zbHunQa3kFoRA@mail.gmail.com>
<506301AC.90101@mistfpga.net>
<CACh7GpHFY_KUhhtk09H_oCzBtRh66artDCqz8pXNTh_ZzkAABg@mail.gmail.com>
Date: Wed, 26 Sep 2012 13:10:31 -0400
Message-ID: <CA+8xBpe_QC-U=HjvMQa-+RkFKZJvdU4Gq0NJZXrme=Uh-rns+w@mail.gmail.com>
From: Jeff Garzik <jgarzik@exmulti.com>
To: Mark Friedenbach <mark@monetize.io>
Content-Type: text/plain; charset=ISO-8859-1
X-Gm-Message-State: ALoCoQlh3RoymHYHf9YQsU45+X0UJA1XGoaUjky6KQjIMlwNa6Ilq7+pmH9jYgCemomob4AeF29u
X-Spam-Score: 0.0 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
X-Headers-End: 1TGv89-0006Vc-0T
Cc: Bitcoin Development List <bitcoin-development@lists.sourceforge.net>,
Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Wed, 26 Sep 2012 17:10:42 -0000
On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach <mark@monetize.io> wrote:
> Certainly developers should be responsible for making sure that regression
> tests for bugs they fix make it either into the unit tests or Matt's
> functional test repository. QA should hold them accountable for that
> (re-opening tickets for bugs that have been fixed but without regression
> tests).
As a goal or general principle, this is agreeable.
But slavish attention to this will only get ignored. There is finite
developer resources, and regression tests for certain types of bugs,
like prickly P2P network interaction bugs or RPC API bugs, could
potentially involve many days or weeks of coding, to sufficiently
simulate the environment. The ability to easily, automatically and
programmatically reproduce certain classes of bugs is simply out of
reach right now, and nobody is going to shut down development to fix
that problem.
We should move towards this direction, yes, but bitcoin test cases are
not always going to be as easy as writing (say) a compiler testcase.
We can always use the help of a few good QA coders: simulating a P2P
environment and checking the RPC API are two examples of very
complicated problems that -can- be automated for testing... with a lot
of work.
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com
|
