summaryrefslogtreecommitdiff
path: root/04/d4e394a9be3c43c3e32d097c8c4751d740bd65
blob: eb98a0038d602e82595e0f9c6aa85434e765608e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
Return-Path: <eric@voskuil.org>
Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137])
 by lists.linuxfoundation.org (Postfix) with ESMTP id 557F5C0051
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Tue, 18 Aug 2020 18:56:19 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by fraxinus.osuosl.org (Postfix) with ESMTP id 3AE648525D
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Tue, 18 Aug 2020 18:56:19 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
Received: from fraxinus.osuosl.org ([127.0.0.1])
 by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id xBUIZRDJv3-k
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Tue, 18 Aug 2020 18:56:18 +0000 (UTC)
X-Greylist: delayed 00:45:00 by SQLgrey-1.7.6
Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com
 [209.85.216.65])
 by fraxinus.osuosl.org (Postfix) with ESMTPS id EF38C851D6
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Tue, 18 Aug 2020 18:56:17 +0000 (UTC)
Received: by mail-pj1-f65.google.com with SMTP id d4so9633401pjx.5
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Tue, 18 Aug 2020 11:56:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=voskuil-org.20150623.gappssmtp.com; s=20150623;
 h=content-transfer-encoding:from:mime-version:subject:date:message-id
 :references:cc:in-reply-to:to;
 bh=Ws5FdAk8wuXIL9zyeOB9G35pZvi14IGIz+4A9/WyHkg=;
 b=KrTWsd9m4yRpNP8Jj5rqqtvnOSDdktpyR1qJls7zjtlAdrI1CX3EvJlHeIjuFhHrU7
 R0diQy5MEajVhen9daek+C5x7bxrQGacNSyCZPvZGsSOdJ/L5zgaogTye+of6+tjbkY3
 tg/WfHuAso0TiOHI4577IuqHeXMOrGExa4ZPWVjpAW6pzNU3bsqMHb+FoLrkqLAavWtI
 95Qbtdq15gZouYlKvIJozYq1xlD+qxamkfxVsk6NsR4dnrhWByEWfNSZf0hgAnh/+qDk
 Z7i5Lq4nrGh/HieSuHSzxha4FCovDyRTzwbJypt0nSuYjWqdsarNbWtYj5GQaoNDCVMl
 FwOg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:content-transfer-encoding:from:mime-version
 :subject:date:message-id:references:cc:in-reply-to:to;
 bh=Ws5FdAk8wuXIL9zyeOB9G35pZvi14IGIz+4A9/WyHkg=;
 b=CzzFqnjKYEQdJfJ/n6xRVSTJq5As+QJz9V60YjVmOQg2EvBqz08zNIdhroZ4uaLQ2q
 RjeP4CxFGjUDMplPS8Sb0fi4iV10Gwz/qwbk9l8k8nibirAm3IHZMBzOWY+tkCbQ38WL
 tldXk5iYyWDVTjOaT8PL4LpToH/Hyb6dTWfcW8FmUFeNBk2x8V7jV1XJgT8J6TwljXtW
 jN0r7M4sKwAOhXlwON6E1eyQLuR8HSvMOmWbF3gJOXKQ/5McKF05IGcQw5zNdfcVNwvM
 Xtxu3ERzLXro4KppxLBP2O5YV08VHwmGXMwPQrwcpFgRfhaXRg2uUdgs6KLokr9rON+1
 LH4g==
X-Gm-Message-State: AOAM533xCiWo4FvN1Q6Yy34xap949NPzM0jTJdzoh1e6z2cH1N5LpbKK
 o4UyEy0BuxAoneJ5OBfh3VIEmw==
X-Google-Smtp-Source: ABdhPJy2kBDU4KE9xER56SEAeOEjzO0UaIuAKKnxcd5gvtVJWApJEb/58W+VjftUydXb2noOjAWArg==
X-Received: by 2002:a17:902:7605:: with SMTP id
 k5mr16661556pll.122.1597776977241; 
 Tue, 18 Aug 2020 11:56:17 -0700 (PDT)
Received: from ?IPv6:2600:380:7066:ce6e:35fb:26cc:838e:c70d?
 ([2600:380:7066:ce6e:35fb:26cc:838e:c70d])
 by smtp.gmail.com with ESMTPSA id q10sm24920124pfs.75.2020.08.18.11.56.15
 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
 Tue, 18 Aug 2020 11:56:16 -0700 (PDT)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Eric Voskuil <eric@voskuil.org>
Mime-Version: 1.0 (1.0)
Date: Tue, 18 Aug 2020 11:56:13 -0700
Message-Id: <CE6A0401-8503-4E0C-8857-32E624CA1D28@voskuil.org>
References: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com>
In-Reply-To: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com>
To: Matt Corallo <lf-lists@mattcorallo.com>
X-Mailer: iPhone Mail (17G68)
X-Mailman-Approved-At: Tue, 18 Aug 2020 19:03:38 +0000
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Generalizing feature negotiation when new p2p
	connections are setup
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Aug 2020 18:56:19 -0000


> On Aug 18, 2020, at 11:25, Matt Corallo <lf-lists@mattcorallo.com> wrote:
>=20
> On 8/18/20 2:11 PM, Eric Voskuil wrote:
> - snip -
>>> Still, I think we're talking pedantics here, and not in a useful way.
>> Not to be pedantic, but I don=E2=80=99t know what that means.
>=20
> It means that part of the discussion is not useful, and not worth botherin=
g to go back and figure out what was shipped before the version increase and=
 what wasn't, lets talk about what makes sense for the future :).

When the discussion centers on backward compatibility, and there is confusio=
n over what that actually implies, this is a central question. You snipped t=
he bit about what actually constitutes the existing protocol. I have impleme=
nted every aspect of the protocol that is widely deployed, and I can say wit=
hout question that the protocol does not require a peer to accept arbitrary m=
essages. In other words, your statement on the subject was a very relevant f=
actual error. Furthermore no reason has been demonstrated here to accept arb=
itrary traffic.

>>> Ultimately we need some kind of negotiation which is flexible in allowin=
g different software to negotiate different features without a global lock-s=
tep version number increase.
>> I have shown below how that already works.
>>> Or, to put it another way, if a feature is fully optional, why should th=
ere be a version number increase for it
>> For the reasons previously given.
>>> - the negotiation of it is independent and a version number only increas=
es confusion over which change "owns" a given version number.
>> Presumably this is why we have a standards process. Any new message impli=
es ownership. Deconflicting that is required, which implies it can easily be=
 version associated (as it has been).
>=20
> I think the point is, this doesn't work today, bumping the protocol versio=
n requires everyone agreeing on which features make sense,

As I have shown, this is not the case. While I have given my support to simp=
lifying the process, we should not proceed based on an incorrect understandi=
ng of actual behavior.

> and as we can see from this email thread alone, that isn't a common result=
 in this community. People happily ignore BIPs that make no sense, of which t=
here are a lot, and they should totally be able to do that!
>=20
> You can say that the current world works, but there's a reason over time w=
e've shifted away from the original "shove another bit on the end of the ver=
sion message, and everyone agrees on the order of those bits for new feature=
 negotiation." Version bumping is an extension of that, really.

Actually the protocol has not done this. It has used the version to signal a=
 new sub-protocol, and then in some cases that sub-protocol has been made op=
tional through subsequent negotiation. What is being proposed here is to sim=
plify that process by collapsing the secondary negotiation into the handshak=
e.

In fact I argued against this secondary ad-hoc negotiation when it began. No=
w we are coming around to recognizing that it=E2=80=99s a handshake issue, a=
s I said at the time. I=E2=80=99m glad to see that.

>>> I presume you'd support a single message that lists the set of features w=
hich a node (optionally) wishes to support on the connection. This proposal i=
s fully equivalent to that, instead opting to list them as individual messag=
es instead of one message, which is a bit nicer in that they can be handled m=
ore independently or by different subsystems including even the message hash=
ing.
>> This presumes an implementation. As part of the handshake, collection of a=
n arbitrary set of messages is a significant and unnecessary complication *o=
f the protocol*. Extension of the verack is not. It is the simplest change p=
ossible to implement the desired behavior. Each peer simply supplies the mat=
rix of sub-protocols it supports and only those supported by both are allowe=
d. There is no reason for the protocol to split that matrix up into multiple=
 messages, requiring termination. Independent messages exist because of timi=
ng or ordering requirements. Implementing dependent messages as if they were=
 independent is wasteful and complicating.
>=20
> Some things may need further negotiation. eg compact blocks sends multiple=
 redundant messages with different versions and then deduces the correct ver=
sion based on the message ordering and version set supported. Doing this via=
 verack locks you into a very specific possible negotiation protocols.

This is a moot point. Whether a list of supported optional sub-protocols is l=
isted in one or multiple messages in the handshake would not change this.

> You could extend it further and suggest a verack K-V list which allows for=
 more flexible negotiation, but I'm not sure that it isn't more complicated t=
han just shoving more messages on the wire.

There is no difference between a constrained set of key-value pairs and a di=
stinct set of options, so there is no additional complexity here.

>> I=E2=80=99m well aware of the inefficiency produced by version linearity i=
n the face of optional sub-protocols. The protocol must negotiate to the ver=
sion where it can then negotiate support, which has been done. I support cre=
ating a simpler system, eliminating these extra messages. The existing numer=
ic version can be reserved exclusively for =E2=80=9Cmust=E2=80=9D implement,=
 and can be used to signal an extension to the verack. The verack can then c=
arry a list of =E2=80=9Cmay=E2=80=9D or =E2=80=9Cshould=E2=80=9D sub-protoco=
ls for final negotiation.
>=20
> I think we agree here - the current method of protocol version bumping isn=
't scalable and something more flexible is definitely a better world.

To be clear this does not increase flexibility, it reduces communication and=
 therefore complexity, and allows peers to lock in allowed message semantics=
 by the end of the handshake, as opposed to allowing them to change at any t=
ime.

>> The format of the matrix is arbitrary, but the requirement is to list a s=
et of optional sub-protocols. This implies a namespace. This implies =E2=80=9C=
ownership=E2=80=9C of names. In other words, that coordination requirement i=
s not eliminated.
>=20
> This is true, there is some ownership requirement, we could switch to hash=
es or something of the like, but human-readable names have shown to be relat=
ively non-colliding in past Bitcoin protocol changes.

Hashes don=E2=80=99t prevent collisions. Someone can just use the same hash.=
 Bitcoin uses names (message names) and numbers (version, service, relay...)=
. It=E2=80=99s a protocol, coordination is the whole point.

e

>> e
>>> Matt
>>>=20
>>>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
>>>> =E2=80=9CBitcoin protocol has always expected clients to ignore unknown=
 messages=E2=80=9D
>>>> This is not true. Bitcoin has long implemented version negotiation, whi=
ch is the opposite expectation. Libbitcoin=E2=80=99s p2p protocol implementa=
tion immediately drops a peer that sends an invalid message according to the=
 negotiated version. The fact that a given client does not validate the prot=
ocol does not make it an expectation that the protocol not be validated.
>>>> Features can clearly be optional within an actual protocol. There have b=
een post-handshake negotiations implemented for optional messages which are v=
alid at the negotiated version. The protocol may be flexible while remaining=
 validateable. There is no reason to force a client to accept unknown messag=
e traffic.
>>>> A generalized versioning change can be implemented in or after the hand=
shake. The latter is already done on an ad-hoc basis. The former is possible=
 as long as the peer=E2=80=99s version is sufficient to be aware of the beha=
vior. This does not imply any need to send invalid messages. The verack itse=
lf can simply be extended with a matrix of feature support. There is no reas=
on to complicate negotiation with an additional message(s).
>>>> FWIW, bip37 did this poorly, adding a feature field to the version mess=
age, resulting in bip60. Due to this design, older protocol-validating clien=
ts were broken. In this case it was message length that was presumed to not b=
e validated.
>>>> e
>>>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@=
lists.linuxfoundation.org> wrote:
>>>>>=20
>>>>> =EF=BB=BFThis sounds like a great idea!
>>>>>=20
>>>>> Bitcoin is no longer a homogeneous network of one client - it is many,=
 with different features implemented in each. The Bitcoin protocol hasn't (f=
ully) evolved to capture that reality. Initially the Bitcoin protocol had a s=
imple numerical version field, but that is wholly impractical for any divers=
e network - some clients may not wish to implement every possible new relay m=
echanic, and why should they have to in order to use other new features?
>>>>>=20
>>>>> Bitcoin protocol changes have, many times in recent history, been made=
 via new dummy "negotiation" messages, which take advantage of the fact that=
 the Bitcoin protocol has always expected clients to ignore unknown messages=
. Given that pattern, it makes sense to have an explicit negotiation phase -=
 after version and before verack, just send the list of features that you su=
pport to negotiate what the connection will be capable of. The exact way we d=
o that doesn't matter much, and sending it as a stream of messages which eac=
h indicate support for a given protocol feature perfectly captures the patte=
rn that has been used in several recent network upgrades, keeping consistenc=
y.
>>>>>=20
>>>>> Matt
>>>>>=20
>>>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
>>>>>> Hi,
>>>>>> Back in February I posted a proposal for WTXID-based transaction rela=
y[1] (now known as BIP 339), which included a proposal for feature negotiati=
on to take place prior to the VERACK message being received by each side.  I=
n my email to this list, I had asked for feedback as to whether that proposa=
l was problematic, and didn't receive any responses.
>>>>>> Since then, the implementation of BIP 339 has been merged into Bitcoi=
n Core, though it has not yet been released.
>>>>>> In thinking about the mechanism used there, I thought it would be hel=
pful to codify in a BIP the idea that Bitcoin network clients should ignore u=
nknown messages received before a VERACK.  A draft of my proposal is availab=
le here[2].
>>>>>> I presume that software upgrading past protocol version 70016 was alr=
eady planning to either implement BIP 339, or ignore the wtxidrelay message p=
roposed in BIP 339 (if not, then this would create network split concerns in=
 the future -- so I hope that someone would speak up if this were a problem)=
.  When we propose future protocol upgrades that would benefit from feature n=
egotiation at the time of connection, I think it would be nice to be able to=
 use the same method as proposed in BIP 339, without even needing to bump th=
e protocol version.  So having an understanding that this is the standard of=
 how other network clients operate would be helpful.
>>>>>> If, on the other hand, this is problematic for some reason, I look fo=
rward to hearing that as well, so that we can be careful about how we deploy=
 future p2p changes to avoid disruption.
>>>>>> Thanks,
>>>>>> Suhas Daftuar
>>>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-Febr=
uary/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/20=
20-February/017648.html>
>>>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature=
-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaft=
uar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negoti=
ation.mediawiki>
>>>>>> _______________________________________________
>>>>>> bitcoin-dev mailing list
>>>>>> bitcoin-dev@lists.linuxfoundation.org
>>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>>> _______________________________________________
>>>>> bitcoin-dev mailing list
>>>>> bitcoin-dev@lists.linuxfoundation.org
>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev