Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Eric Voskuil <eric@voskuil.org>
Mime-Version: 1.0 (1.0)
Date: Tue, 18 Aug 2020 11:11:12 -0700
Message-Id: <95DD247F-028E-432A-9C26-6337F5819A31@voskuil.org>
References: <6ccf6ff8-6b4f-c8f4-0dbb-36c5d076528f@mattcorallo.com>
In-Reply-To: <6ccf6ff8-6b4f-c8f4-0dbb-36c5d076528f@mattcorallo.com>
To: Matt Corallo <lf-lists@mattcorallo.com>
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Generalizing feature negotiation when new p2p
	connections are setup
Precedence: list


> On Aug 18, 2020, at 10:26, Matt Corallo <lf-lists@mattcorallo.com> wrote:
>=20
> =EF=BB=BFThere are several cases where a new message has been sent as a pa=
rt of a negotiation without changing the protocol version.

Such as?

> You may chose to ignore that, but that doesn't mean that it isn't an under=
stood and even relied upon feature of the Bitcoin P2P protocol. If you wish t=
o fail connections to new nodes (and risk network splits, as Suhas points ou=
t), then you may do so, but that doesn't make it a part of the Bitcoin P2P p=
rotocol that you must do so. Of course there is no "official document" by wh=
ich we can make a formal appeal, but historical precedent suggests otherwise=
.

You are misrepresenting =E2=80=9Chistorical precedent=E2=80=9D. I=E2=80=99ve=
 seen several attempts to require arbitrary traffic over the years and none h=
ave been realized.

> Still, I think we're talking pedantics here, and not in a useful way.

Not to be pedantic, but I don=E2=80=99t know what that means.

> Ultimately we need some kind of negotiation which is flexible in allowing d=
ifferent software to negotiate different features without a global lock-step=
 version number increase.

I have shown below how that already works.

> Or, to put it another way, if a feature is fully optional, why should ther=
e be a version number increase for it

For the reasons previously given.

> - the negotiation of it is independent and a version number only increases=
 confusion over which change "owns" a given version number.

Presumably this is why we have a standards process. Any new message implies o=
wnership. Deconflicting that is required, which implies it can easily be ver=
sion associated (as it has been).

> I presume you'd support a single message that lists the set of features wh=
ich a node (optionally) wishes to support on the connection. This proposal i=
s fully equivalent to that, instead opting to list them as individual messag=
es instead of one message, which is a bit nicer in that they can be handled m=
ore independently or by different subsystems including even the message hash=
ing.

This presumes an implementation. As part of the handshake, collection of an a=
rbitrary set of messages is a significant and unnecessary complication *of t=
he protocol*. Extension of the verack is not. It is the simplest change poss=
ible to implement the desired behavior. Each peer simply supplies the matrix=
 of sub-protocols it supports and only those supported by both are allowed. T=
here is no reason for the protocol to split that matrix up into multiple mes=
sages, requiring termination. Independent messages exist because of timing o=
r ordering requirements. Implementing dependent messages as if they were ind=
ependent is wasteful and complicating.

I=E2=80=99m well aware of the inefficiency produced by version linearity in t=
he face of optional sub-protocols. The protocol must negotiate to the versio=
n where it can then negotiate support, which has been done. I support creati=
ng a simpler system, eliminating these extra messages. The existing numeric v=
ersion can be reserved exclusively for =E2=80=9Cmust=E2=80=9D implement, and=
 can be used to signal an extension to the verack. The verack can then carry=
 a list of =E2=80=9Cmay=E2=80=9D or =E2=80=9Cshould=E2=80=9D sub-protocols f=
or final negotiation.

The format of the matrix is arbitrary, but the requirement is to list a set o=
f optional sub-protocols. This implies a namespace. This implies =E2=80=9Cow=
nership=E2=80=9C of names. In other words, that coordination requirement is n=
ot eliminated.

e

> Matt
>=20
>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
>> =E2=80=9CBitcoin protocol has always expected clients to ignore unknown m=
essages=E2=80=9D
>> This is not true. Bitcoin has long implemented version negotiation, which=
 is the opposite expectation. Libbitcoin=E2=80=99s p2p protocol implementati=
on immediately drops a peer that sends an invalid message according to the n=
egotiated version. The fact that a given client does not validate the protoc=
ol does not make it an expectation that the protocol not be validated.
>> Features can clearly be optional within an actual protocol. There have be=
en post-handshake negotiations implemented for optional messages which are v=
alid at the negotiated version. The protocol may be flexible while remaining=
 validateable. There is no reason to force a client to accept unknown messag=
e traffic.
>> A generalized versioning change can be implemented in or after the handsh=
ake. The latter is already done on an ad-hoc basis. The former is possible a=
s long as the peer=E2=80=99s version is sufficient to be aware of the behavi=
or. This does not imply any need to send invalid messages. The verack itself=
 can simply be extended with a matrix of feature support. There is no reason=
 to complicate negotiation with an additional message(s).
>> FWIW, bip37 did this poorly, adding a feature field to the version messag=
e, resulting in bip60. Due to this design, older protocol-validating clients=
 were broken. In this case it was message length that was presumed to not be=
 validated.
>> e
>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@li=
sts.linuxfoundation.org> wrote:
>>>=20
>>> =EF=BB=BFThis sounds like a great idea!
>>>=20
>>> Bitcoin is no longer a homogeneous network of one client - it is many, w=
ith different features implemented in each. The Bitcoin protocol hasn't (ful=
ly) evolved to capture that reality. Initially the Bitcoin protocol had a si=
mple numerical version field, but that is wholly impractical for any diverse=
 network - some clients may not wish to implement every possible new relay m=
echanic, and why should they have to in order to use other new features?
>>>=20
>>> Bitcoin protocol changes have, many times in recent history, been made v=
ia new dummy "negotiation" messages, which take advantage of the fact that t=
he Bitcoin protocol has always expected clients to ignore unknown messages. G=
iven that pattern, it makes sense to have an explicit negotiation phase - af=
ter version and before verack, just send the list of features that you suppo=
rt to negotiate what the connection will be capable of. The exact way we do t=
hat doesn't matter much, and sending it as a stream of messages which each i=
ndicate support for a given protocol feature perfectly captures the pattern t=
hat has been used in several recent network upgrades, keeping consistency.
>>>=20
>>> Matt
>>>=20
>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
>>>> Hi,
>>>> Back in February I posted a proposal for WTXID-based transaction relay[=
1] (now known as BIP 339), which included a proposal for feature negotiation=
 to take place prior to the VERACK message being received by each side.  In m=
y email to this list, I had asked for feedback as to whether that proposal w=
as problematic, and didn't receive any responses.
>>>> Since then, the implementation of BIP 339 has been merged into Bitcoin C=
ore, though it has not yet been released.
>>>> In thinking about the mechanism used there, I thought it would be helpf=
ul to codify in a BIP the idea that Bitcoin network clients should ignore un=
known messages received before a VERACK.  A draft of my proposal is availabl=
e here[2].
>>>> I presume that software upgrading past protocol version 70016 was alrea=
dy planning to either implement BIP 339, or ignore the wtxidrelay message pr=
oposed in BIP 339 (if not, then this would create network split concerns in t=
he future -- so I hope that someone would speak up if this were a problem). =
 When we propose future protocol upgrades that would benefit from feature ne=
gotiation at the time of connection, I think it would be nice to be able to u=
se the same method as proposed in BIP 339, without even needing to bump the p=
rotocol version.  So having an understanding that this is the standard of ho=
w other network clients operate would be helpful.
>>>> If, on the other hand, this is problematic for some reason, I look forw=
ard to hearing that as well, so that we can be careful about how we deploy f=
uture p2p changes to avoid disruption.
>>>> Thanks,
>>>> Suhas Daftuar
>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-Februa=
ry/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020=
-February/017648.html>
>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-n=
egotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftua=
r/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiat=
ion.mediawiki>
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev