1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
|
Return-Path: <eric@voskuil.org>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 8655E408
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 28 Jul 2015 06:40:26 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-pa0-f46.google.com (mail-pa0-f46.google.com
[209.85.220.46])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 894678D
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 28 Jul 2015 06:40:25 +0000 (UTC)
Received: by padck2 with SMTP id ck2so64740731pad.0
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 27 Jul 2015 23:40:25 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to
:cc:subject:references:in-reply-to:content-type;
bh=aabFVj6kATt7aOeAP1E40FvqxWuZtpvgB2J3qaRHcUY=;
b=mLH7IGRtDbpxHPfxl2MsIton55AJnfZLc+g7X9mO2WbQ/+i15oFowMPLUB7Oa7tk6V
RdzE7ImdzQ6jZluB6wG0cG2MHO+xYHonPK5jz2b1l620jkeFYQjtfP/vTxu8sYfn8I5u
e1I6EyPJOv49I3YLI8FS6kFiE707lFi1P2ngDB8IBR2FSSRaN4lXNPnZFbylCLhVI14E
P0RIC36AohE3BVHVdCJ0+F9b7iNVa2TrNs3hm9E29JCNmD5D1UyHjXDyd6lCR9g282Uh
0E1fTq0YFlkO4I8JigdbeBRTLsTbKNCy7p7HJoSMzmIRzUbTXv+bO8HwTJ/LvKuumD6w
3BRg==
X-Gm-Message-State: ALoCoQk9sEQpdEjFWc/AeC7yJrtKBl8RO/BibWVBM2Rkj8GgkA63vbwwt8cqF6MJFWEEE3cr75EM
X-Received: by 10.66.132.98 with SMTP id ot2mr54972089pab.31.1438065625277;
Mon, 27 Jul 2015 23:40:25 -0700 (PDT)
Received: from [10.0.1.14] (c-67-161-88-20.hsd1.wa.comcast.net. [67.161.88.20])
by smtp.googlemail.com with ESMTPSA id
ym6sm33226179pac.32.2015.07.27.23.40.24
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Mon, 27 Jul 2015 23:40:24 -0700 (PDT)
Message-ID: <55B723EA.7010700@voskuil.org>
Date: Mon, 27 Jul 2015 23:40:42 -0700
From: Eric Voskuil <eric@voskuil.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
rv:31.0) Gecko/20100101 Thunderbird/31.2.0
MIME-Version: 1.0
To: =?UTF-8?B?Sm9yZ2UgVGltw7Nu?= <jtimon@jtimon.cc>
References: <CABm2gDrApVuxF8DFf32V=pQhDKvvVfcDK=LeCXJ9h9o8CY+wNQ@mail.gmail.com>
In-Reply-To: <CABm2gDrApVuxF8DFf32V=pQhDKvvVfcDK=LeCXJ9h9o8CY+wNQ@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature";
boundary="uNOa5RRXaWrqG2OJBmmI6WR7q0NXqmaKV"
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW
autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Cc: bitcoin-dev@lists.linuxfoundation.org
Subject: Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin
Core and hard forks)
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 28 Jul 2015 06:40:26 -0000
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--uNOa5RRXaWrqG2OJBmmI6WR7q0NXqmaKV
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 07/23/2015 07:30 AM, Jorge Tim=C3=B3n wrote:
> On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev wrote:
>> On 07/22/2015 05:13 PM, Eric Lombrozo via bitcoin-dev wrote:
>>> Only being partly serious - I strongly am in favor of a sufficiently
>>> modularized codebase that swapping out consensus rules is fairly
>>> straightforward and easy to test...
>>
>> We (libbitcoin) have taken the time to publish and maintain bitcoind's=
>> "libbitcoinconsensus" source files as an independent C++ library...
> I think there were some misunderstandings in our previous conversation
> about this topic.
> I completely agree with having a separated repository for libconsensus
> (that's the whole point, alternative implementations can be
> consensus-safe by using it, and in the event of a schism fork[1], they
> can fork just that smaller project without having to relay on Bitcoin
> Core [satoshi] at all).
> But I thought you also wanted Bitcoin Core to use libconsensus instead
> of just having a subtree/subrepository like it currently does with
> libsecp256k1.
libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A
separate repository was what I considered as a requirement for us to use =
it.
> I'm not sure if that would ever be accepted, but in any case we're
> certainly far away from that goal.
If it's not certain whether this would even be accepted, the commitment
to a community consensus library is too weak to take a strong dependency
on. But for us it's moot, as we have made the already accomplished that
goal.
> Here are some things that need to
> happen first:
>=20
> 1) Finish encapsulating consensus code so that it can be built without
> any (we've done it only with script-related code so far). Here are
> some related PRs (other people have done other things that help with
> this as well):
=2E..
> 2) Finish libconsensus's API: expose more things than VerifyScript, at
> the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
> Feedback from alternative implementations like libbitcoin is extremely
> valuable here. Some related closed-for-now PRs:
In our earlier discussion I believe you said that the library would not
be undergoing significant change or feature creep. If this is the very
least that's projected it would seem that constraint will not hold.
> 3) Move libconsensus to a separate repository as a
> subtree/subrepository of Bitcoin Core.
>=20
> Only after all that we can discuss whether Bitcoin Core itself should
> include libconsensus' code or just use its API directly.
I don't think it's a question of whether it *should* use its own library
as it is published for others - this is a practically self-evident
conclusion.
> I hope that after all this, libbitcoin also reconsiders whether to
> reimplement its own libconsensus or use the "official" one directly
> instead.
We use a fork of libsecp256k1 and would probably do the same with the
consensus library if it was cleanly isolated.
>> In any case I agree with your stated need for this isolation (if not t=
he
>> means) for the reasons you state. The community needs to move beyond a=
>> largely singular and monolithic codebase that is holding that position=
>> in part due to fear about consensus bug forks.
>=20
> I completely agree. That's the goal of libconsensus (and an
> alternative implementation like libbitcoin being able to use it
> without sacrificing any of its current or future design differences
> from Bitcoin Core would be a sign of success in this reward).
It's a performance sacrifice, and then there's the OpenSSL dependency,
but these are both optional within our stack - so the application
developer has the option. So the only downside is that we are
maintaining the conditional compilation.
> Unfortunately any changes that touch consensus code are risky and
> therefore slow. And when consensus encapsulation changes conflict with
> other changes (not because the other changes need to change consensus
> but because consensus code is still coupled with policy and other
> bitcoind-specific code), refactors are never prioritized. Ironically,
> you need to encapsulate the consensus code to avoid such conflicts,
> which would make all non-consensus changes far less risky (reducing
> the consensus-critical review development bottleneck).
>=20
> Unfortunately and ironically again, safer, small and incremental
> changes are less interesting for reviewers.
> For example, I've been trying to move consensus code to the consensus
> folder for a long time. The correctness of a MOVEONLY change is
> trivial to review for anyone who knows how to copy/paste in its
> favorite editor and how to use git diff, but will I ever get answers
> to my questions in [1]?
I think it's worthwhile work, especially if you are passionate about the
longer term objectives. I haven't been involved in these reviews as I
spend very little time with the satoshi client
> I know there's many people who really care about this, Cory Fields,
> Wladimir and Pieter Wuille to name a few have reviewed many of this
> changes (I've just got used to publicly whine about lack of review on
> this front and policy encapsulation [very related fronts] as an
> attempt to get some attention: not always, but begging for review
> actually works some times).
Well a cynic might observe that fear of consensus bugs is what keeps
people on the satoshi client, and therefore accelerating the development
of a clean and independent consensus library would be a very low priority=
=2E
> Another unfortunate fact is that although a script-only libconsensus
> allows you to avoid a big part of all possible consensus fork bugs,
> there cannot be users of a finished libconsensus to ask things to until=
> a finished libconsensus actually exists.
Software is never finished, but this exists and we are using it.
> At the same time, the future
> users (alternative implementations, since bitcoin core is already
> "using libconsensus")=20
It is using the same source files, but AFAICT not the library.
> are the most relevant people to listen when it
> comes to the C API. That's why I beg you to comment on [2], even if
> #5995 is currently closed. Your input on [1] would be very appreciated
> as well (maybe you think it's better to expose verifyTx before
> exposing verifyHeader, even if exposing verifyHeader is something that
> could be done faster).
I haven't looked at any of these commits, but I'll make some time to at
least give a cursory review.
> > To make choice regarding consensus an actual choice (and thereby act=
ual
>> consensus) the modularity you suggest is essential. One must be able t=
o
>> take new developments without having to take consensus changes. The
>> option to fork the codebase is not reasonable for most people. At this=
>> point there is no defensible reason for coupling consensus checks with=
>> other features.
>=20
> Would you agree that asking people to fork an independent libconsensus
> project instead of having to fork the full Bitcoin-qt is much more
> reasonable?
Yes, of course. We've already done it. For each release of the satoshi
client since we made libbitcoin-consensus I've copied the sources. It's
pretty much automated and easy to visually verify that the sources
match. That would be quite a bit more difficult if there wasn't an
independent build.
> I mean, I agree with your points. If "the specification of the
> consensus rules is an implementation", then that implementation
> shouldn't be coupled with a bunch of policy and non-consensus
> technical choices (storage, dependencies, p2p protocol...). But I
> still think that "the specification of the consensus rules should be a
> concrete implementation" rather than based purely on a natural
> language like English.
Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.
> I believe that's the only point where we fundamentally disagree, but
> it shouldn't be a barrier in our common goal of taking "power" away
> from Bitcoin Core development. If we're successful Bitcoin Core won't
> have any privileged position with regards to, say, libbitcoin when it
> comes to deciding consensus rules changes.
I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.
=2E.
> 1) libconsensus us finished and used by libbitcoin
> 2) Bitcoin Core was unanimously in favor of Gavin's 32 GB initial
> proposal and the changes are applied to bitcoin/bitcoin and
> bitcoin/libconsensus (or Bitcoin Core has a dictator like Mike
> wants[4] and he accepts it, it doesn't really matter for this
> example).
>=20
> But let's also assume that X% of the users and 10% of the miners are
> against that Schism hardfork, and they don't want to be forced to
> change the rules by any influential group, mining, economic or user
> majority.
> Libbitcoin cannot be forced to accept the next, controversial version
> of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
> of the last ok version.
This is already done.
> Centralized-bitcoin and old-bitcoin would become 2 separated
> currencies and some people would likely lose money in the transition
> from one currency to 2 of them, but the users of old-bitcoin have the
> right of keeping the rules they signed up for and the only responsible
> people for this likely-catastrophic schism would be the Bitcoin core
> developers for trying to impose consensus changes into others against
> their will. Trying to impose consensus changes against the will of
> some users is wrong, and it is irrelevant if that happens in Bitcoin
> Core or Bitcoin Tx (if it is uncontroversial, it's also irrelevant
> where it gets implemented first).
I don't disagree, but you previously argued that *everyone* had to agree
on consensus. Above you are making the argument that people can choose
to disagree. Yes, this is important. Yet it's unrealistic for any
alternative consensus to overcome inertia unless there are widely
deployed independent implementations.
> I really believe bitcoin needs competitive alternative implementations
> and I believe libconsensus is a tool to help that happen and reduce
> the "gatekeeping" friction that there's (unfortunately) around Bitcoin
> Core. I look forward to any potential collaboration on this front.
> Even if you still want to maintain a reimplementation of libconsensus
> (which I humbly think it's a mistake, but I don't think there's any
> point on keep discussing that, since we know we disagree) we can
> collaborate on the future common API of a complete libconsensus (with
> verifyBlock and all). I really hope we can do that.
I'm maintain our fork of the consensus sources until they are (1)
properly isolated from the satoshi client and (2) the satoshi client
uses the actual library.
I don't see it as important or even productive to try and wedge the
consensus library into the repository/build for bitcoind and Bitcoin-QT.
It's straightforward to maintain the consensus library independently.
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.
e
--uNOa5RRXaWrqG2OJBmmI6WR7q0NXqmaKV
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJVtyPqAAoJEDzYwH8LXOFOQPsH/3HZrA7Mzi/zoYYs3+8r4ISe
+BtnXaK5CtYXmOAK4lgZflFlAJpzCBYCC2tUhBftPqm3Z9Ta3OUf+L7St74cZb/y
NXzUgaajGoxw1nWJyECtKKgXHmvRt2ALWfoiVlrT6NgH7ZQDrUuoOw4OW/D7oZzn
0YE5VsOMFrwO+H4QbOLIn/Hkp5OeNWkszJdrodWHqpRfbVX2Qcwx91Iojk53z4UI
XoK8d7G+7EfGAKYYcwxl5mMZvTijYNdJSga8sB2v4RWYxjp6fd14VtWGJNqBZnko
bfpomau/mdYiMSyHQyr5ORRu2mzRRTOjTLDam00Nq06a6j1O3Pp/S+d8OOagSiw=
=fIeR
-----END PGP SIGNATURE-----
--uNOa5RRXaWrqG2OJBmmI6WR7q0NXqmaKV--
|