1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
helo=mx.sourceforge.net)
by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <aritter@gmail.com>) id 1UOCKk-0001eX-DB
for bitcoin-development@lists.sourceforge.net;
Fri, 05 Apr 2013 19:29:58 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.216.175 as permitted sender)
client-ip=209.85.216.175; envelope-from=aritter@gmail.com;
helo=mail-qc0-f175.google.com;
Received: from mail-qc0-f175.google.com ([209.85.216.175])
by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1UOCKi-0003d1-Vn
for bitcoin-development@lists.sourceforge.net;
Fri, 05 Apr 2013 19:29:58 +0000
Received: by mail-qc0-f175.google.com with SMTP id j3so1375847qcs.34
for <bitcoin-development@lists.sourceforge.net>;
Fri, 05 Apr 2013 12:29:51 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.229.14.130 with SMTP id g2mr3772175qca.90.1365190191405;
Fri, 05 Apr 2013 12:29:51 -0700 (PDT)
Received: by 10.49.26.41 with HTTP; Fri, 5 Apr 2013 12:29:51 -0700 (PDT)
In-Reply-To: <CAAS2fgT71s0EgF055wyqcCdXmMtY-=q0pQDh9P8pRgKJcELOQg@mail.gmail.com>
References: <CAKuKjyUFBuPMPRV6R-u0iTa=8DWMN9vqdOxnr8o8kxg9rJtVBA@mail.gmail.com>
<CAAS2fgT71s0EgF055wyqcCdXmMtY-=q0pQDh9P8pRgKJcELOQg@mail.gmail.com>
Date: Fri, 5 Apr 2013 14:29:51 -0500
Message-ID: <CAKuKjyWF_su0c7USd6-vzeFtJEx-YJoQZOc_zJq1U=av50CajA@mail.gmail.com>
From: Adam Ritter <aritter@gmail.com>
To: Gregory Maxwell <gmaxwell@gmail.com>,
Matt Corallo <bitcoin-list@bluematt.me>
Content-Type: text/plain; charset=ISO-8859-1
X-Spam-Score: -1.6 (-)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(aritter[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UOCKi-0003d1-Vn
Cc: Bitcoin Development <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Integration testing for BitCoin
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Fri, 05 Apr 2013 19:29:58 -0000
Thanks guys, it sounds great.
Testing the JSON-RPC is/was not the main goal, just an interface for testing.
I didn't know that the bitcoinj implementation is getting close to a
full implementation..it sounds interesting, as it's much easier to
understand and work with. I'll look at the test cases.
Thanks very much,
Adam
On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
>> Hey guys,
>>
>> I just bought some BitCoins after being lazy to do it for the last few
>> years, but also looked at the client code and the messages that are
>> going on this mailing list.
>> I saw that there are quite some unit tests, but I didn't find
>> integration test for BitCoin, and I believe that it's quite important
>> for the future of BitCoin (making the current code more stable,
>> testing attack scenarios, refactoring and extending code).
> [...]
>> Tests that simulate multiple bitcoin users and can verify that the
>> whole network of bitcoin clients work together
>> to achieve the goals of Bitcoin. Also maybe [System
>> testing](http://en.wikipedia.org/wiki/System_testing)
>> would be a better name for the tests, but I'm not sure.
>
> I prefer to call them system tests.
>
> We use a system called blocktester that Matt Corallo wrote,
> https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
>
> It's based on BitcoinJ and works by simulating a peer against a
> slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
> computationally expensive mining). The tests simulates many
> complicated network scenarios and tests the boundaries of many
> (hopefully all) the particular rules of the blockchain validation
> protocol. We can use these tests to compare different versions of the
> reference software to each other and to bitcoinj (or other full node
> implementations) as well as comparing them to our abstract
> understanding of what we believe the rules of the protocol to be.
>
> These tests are run as part of the automated tests on every proposed
> patch to the reference software. Via a robot called pulltester which
> comments on github requests and produces logs like this:
> http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/.
> Pulltester also performs automatic code coverage measurements.
>
> Additionally, we run a public secondary test bitcoin network called
> 'testnet', which can be accessed by anyone by starting the reference
> software with testnet=1. Testnet operates the same as the production
> network except it allows mining low difficulty blocks to prevent it
> going for long times without blocks, and some of the protective
> relaying rules against "non standard" transaction types are disabled.
>
> Most of this testing work has been centered around validating the
> blockchain behavior because thats what has serious systemic risk.
> Measuring the json rpc behavior is strictly less interesting, though
> interesting too.
|
