1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
|
Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
helo=mx.sourceforge.net)
by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <mh.in.england@gmail.com>) id 1UOS7u-0008Bc-52
for bitcoin-development@lists.sourceforge.net;
Sat, 06 Apr 2013 12:21:46 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.214.178 as permitted sender)
client-ip=209.85.214.178; envelope-from=mh.in.england@gmail.com;
helo=mail-ob0-f178.google.com;
Received: from mail-ob0-f178.google.com ([209.85.214.178])
by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1UOS7o-00017A-5H
for bitcoin-development@lists.sourceforge.net;
Sat, 06 Apr 2013 12:21:46 +0000
Received: by mail-ob0-f178.google.com with SMTP id wd20so4514488obb.9
for <bitcoin-development@lists.sourceforge.net>;
Sat, 06 Apr 2013 05:21:34 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.60.56.36 with SMTP id x4mr10522629oep.25.1365250894666; Sat,
06 Apr 2013 05:21:34 -0700 (PDT)
Sender: mh.in.england@gmail.com
Received: by 10.76.162.198 with HTTP; Sat, 6 Apr 2013 05:21:34 -0700 (PDT)
In-Reply-To: <CAKuKjyWF_su0c7USd6-vzeFtJEx-YJoQZOc_zJq1U=av50CajA@mail.gmail.com>
References: <CAKuKjyUFBuPMPRV6R-u0iTa=8DWMN9vqdOxnr8o8kxg9rJtVBA@mail.gmail.com>
<CAAS2fgT71s0EgF055wyqcCdXmMtY-=q0pQDh9P8pRgKJcELOQg@mail.gmail.com>
<CAKuKjyWF_su0c7USd6-vzeFtJEx-YJoQZOc_zJq1U=av50CajA@mail.gmail.com>
Date: Sat, 6 Apr 2013 13:21:34 +0100
X-Google-Sender-Auth: NLSTB40hsDI2iP1xKPVc2SsXGyo
Message-ID: <CANEZrP35kFE0mkJjFWOkKMorqFg+p0BuhDpJs2Ne=MtnZwb=DA@mail.gmail.com>
From: Mike Hearn <mike@plan99.net>
To: Adam Ritter <aritter@gmail.com>
Content-Type: multipart/alternative; boundary=001a11c249f69b226204d9b03e09
X-Spam-Score: -0.5 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(mh.in.england[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UOS7o-00017A-5H
Cc: Bitcoin Development <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Integration testing for BitCoin
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Sat, 06 Apr 2013 12:21:46 -0000
--001a11c249f69b226204d9b03e09
Content-Type: text/plain; charset=UTF-8
In bitcoinj we desperately need integration tests to exercise the wallet
code, and I think if it was done well the tests would be applicable to
bitcoind as well. There have been a series of bugs in bitcoinj that boiled
down to "the unit tests were not realistic enough", either because they
stopped simulating too early or they weren't combining multiple different
things together in the same ways as happens on the real network. Sometimes
timing was an issue too.
Examples of what I mean - ensure that re-orgs are handled correctly and
update the wallet properly in every case, etc.
Something else that would be really useful, a standalone tool that
stress-tests the system. If we had a tool that randomly generated chains of
transactions we might have caught the bdb lock limit bug earlier. You could
write such a tool using bitcoinj easily, or the raw transaction APIs on
bitcoind.
On Fri, Apr 5, 2013 at 8:29 PM, Adam Ritter <aritter@gmail.com> wrote:
> Thanks guys, it sounds great.
> Testing the JSON-RPC is/was not the main goal, just an interface for
> testing.
> I didn't know that the bitcoinj implementation is getting close to a
> full implementation..it sounds interesting, as it's much easier to
> understand and work with. I'll look at the test cases.
>
> Thanks very much,
> Adam
>
>
> On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com>
> wrote:
> > On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
> >> Hey guys,
> >>
> >> I just bought some BitCoins after being lazy to do it for the last few
> >> years, but also looked at the client code and the messages that are
> >> going on this mailing list.
> >> I saw that there are quite some unit tests, but I didn't find
> >> integration test for BitCoin, and I believe that it's quite important
> >> for the future of BitCoin (making the current code more stable,
> >> testing attack scenarios, refactoring and extending code).
> > [...]
> >> Tests that simulate multiple bitcoin users and can verify that the
> >> whole network of bitcoin clients work together
> >> to achieve the goals of Bitcoin. Also maybe [System
> >> testing](http://en.wikipedia.org/wiki/System_testing)
> >> would be a better name for the tests, but I'm not sure.
> >
> > I prefer to call them system tests.
> >
> > We use a system called blocktester that Matt Corallo wrote,
> >
> https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
> >
> > It's based on BitcoinJ and works by simulating a peer against a
> > slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
> > computationally expensive mining). The tests simulates many
> > complicated network scenarios and tests the boundaries of many
> > (hopefully all) the particular rules of the blockchain validation
> > protocol. We can use these tests to compare different versions of the
> > reference software to each other and to bitcoinj (or other full node
> > implementations) as well as comparing them to our abstract
> > understanding of what we believe the rules of the protocol to be.
> >
> > These tests are run as part of the automated tests on every proposed
> > patch to the reference software. Via a robot called pulltester which
> > comments on github requests and produces logs like this:
> >
> http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/
> .
> > Pulltester also performs automatic code coverage measurements.
> >
> > Additionally, we run a public secondary test bitcoin network called
> > 'testnet', which can be accessed by anyone by starting the reference
> > software with testnet=1. Testnet operates the same as the production
> > network except it allows mining low difficulty blocks to prevent it
> > going for long times without blocks, and some of the protective
> > relaying rules against "non standard" transaction types are disabled.
> >
> > Most of this testing work has been centered around validating the
> > blockchain behavior because thats what has serious systemic risk.
> > Measuring the json rpc behavior is strictly less interesting, though
> > interesting too.
>
>
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire
> the most talented Cisco Certified professionals. Visit the
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--001a11c249f69b226204d9b03e09
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">In bitcoinj we desperately need integration tests to exerc=
ise the wallet code, and I think if it was done well the tests would be app=
licable to bitcoind as well. There have been a series of bugs in bitcoinj t=
hat boiled down to "the unit tests were not realistic enough", ei=
ther because they stopped simulating too early or they weren't combinin=
g multiple different things together in the same ways as happens on the rea=
l network. Sometimes timing was an issue too.<div>
<br></div><div style>Examples of what I mean - ensure that re-orgs are hand=
led correctly and update the wallet properly in every case, etc.</div><div =
style><br></div><div style>Something else that would be really useful, a st=
andalone tool that stress-tests the system. If we had a tool that randomly =
generated chains of transactions we might have caught the bdb lock limit bu=
g earlier. You could write such a tool using bitcoinj easily, or the raw tr=
ansaction APIs on bitcoind.=C2=A0</div>
<div style><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D=
"gmail_quote">On Fri, Apr 5, 2013 at 8:29 PM, Adam Ritter <span dir=3D"ltr"=
><<a href=3D"mailto:aritter@gmail.com" target=3D"_blank">aritter@gmail.c=
om</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Thanks guys, it sounds great.<br>
Testing the JSON-RPC is/was not the main goal, just an interface for testin=
g.<br>
I didn't know that the bitcoinj implementation is getting close to a<br=
>
full implementation..it sounds interesting, as it's much easier to<br>
understand and work with. I'll look at the test cases.<br>
<br>
Thanks very much,<br>
Adam<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <<a href=3D"mailto:gmax=
well@gmail.com">gmaxwell@gmail.com</a>> wrote:<br>
> On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <<a href=3D"mailto:ari=
tter@gmail.com">aritter@gmail.com</a>> wrote:<br>
>> Hey guys,<br>
>><br>
>> I just bought some BitCoins after being lazy to do it for the last=
few<br>
>> years, but also looked at the client code and the messages that ar=
e<br>
>> going on this mailing list.<br>
>> I saw that there are quite some unit tests, but I didn't find<=
br>
>> integration test for BitCoin, and I believe that it's quite im=
portant<br>
>> for the future of BitCoin (making the current code more stable,<br=
>
>> testing attack scenarios, refactoring and extending code).<br>
> [...]<br>
>> Tests that simulate multiple bitcoin users and can verify that the=
<br>
>> whole network of bitcoin clients work together<br>
>> to achieve the goals of Bitcoin. Also maybe [System<br>
>> testing](<a href=3D"http://en.wikipedia.org/wiki/System_testing" t=
arget=3D"_blank">http://en.wikipedia.org/wiki/System_testing</a>)<br>
>> would be a better name for the tests, but I'm not sure.<br>
><br>
> I prefer to call them system tests.<br>
><br>
> We use a system called blocktester that Matt Corallo wrote,<br>
> <a href=3D"https://code.google.com/r/bluemattme-bitcoinj/source/browse=
/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?nam=
e=3Dfullverif&r=3D874c5904b12d1fcec5b556429cf208f63cd4e1bc" target=3D"_=
blank">https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src=
/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=3Dfullv=
erif&r=3D874c5904b12d1fcec5b556429cf208f63cd4e1bc</a><br>
><br>
> It's based on BitcoinJ and works by simulating a peer against a<br=
>
> slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid<br>
> computationally expensive mining). =C2=A0The tests simulates many<br>
> complicated network scenarios and tests the boundaries of many<br>
> (hopefully all) the particular rules of the blockchain validation<br>
> protocol. =C2=A0We can use these tests to compare different versions o=
f the<br>
> reference software to each other and to bitcoinj (or other full node<b=
r>
> implementations) as well as comparing them to our abstract<br>
> understanding of what we believe the rules of the protocol to be.<br>
><br>
> These tests are run as part of the automated tests on every proposed<b=
r>
> patch to the reference software. Via a robot called pulltester which<b=
r>
> comments on github requests and produces logs like this:<br>
> <a href=3D"http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c=
7f876aa8adb405c88e17/" target=3D"_blank">http://jenkins.bluematt.me/pull-te=
ster/92a129980fb9b506da6c7f876aa8adb405c88e17/</a>.<br>
> Pulltester also performs automatic code coverage measurements.<br>
><br>
> Additionally, we run a public secondary test bitcoin network called<br=
>
> 'testnet', which can be accessed by anyone by starting the ref=
erence<br>
> software with testnet=3D1. =C2=A0Testnet operates the same as the prod=
uction<br>
> network except it allows mining low difficulty blocks to prevent it<br=
>
> going for long times without blocks, and some of the protective<br>
> relaying rules against "non standard" transaction types are =
disabled.<br>
><br>
> Most of this testing work has been centered around validating the<br>
> blockchain behavior because thats what has serious systemic risk.<br>
> Measuring the json rpc behavior is strictly less interesting, though<b=
r>
> interesting too.<br>
<br>
---------------------------------------------------------------------------=
---<br>
Minimize network downtime and maximize team effectiveness.<br>
Reduce network management and security costs.Learn how to hire<br>
the most talented Cisco Certified professionals. Visit the<br>
Employer Resources Portal<br>
<a href=3D"http://www.cisco.com/web/learning/employer_resources/index.html"=
target=3D"_blank">http://www.cisco.com/web/learning/employer_resources/ind=
ex.html</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</div></div></blockquote></div><br></div>
--001a11c249f69b226204d9b03e09--
|
