Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: Eric Voskuil <eric@voskuil.org>
Mime-Version: 1.0 (1.0)
Date: Tue, 18 Aug 2020 11:56:13 -0700
Message-Id: <CE6A0401-8503-4E0C-8857-32E624CA1D28@voskuil.org>
References: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com>
In-Reply-To: <995919b9-0845-3a70-4b24-9b2c1ca5fd3d@mattcorallo.com>
To: Matt Corallo <lf-lists@mattcorallo.com>
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Generalizing feature negotiation when new p2p
	connections are setup
Precedence: list


> On Aug 18, 2020, at 11:25, Matt Corallo <lf-lists@mattcorallo.com> wrote:
>=20
> On 8/18/20 2:11 PM, Eric Voskuil wrote:
> - snip -
>>> Still, I think we're talking pedantics here, and not in a useful way.
>> Not to be pedantic, but I don=E2=80=99t know what that means.
>=20
> It means that part of the discussion is not useful, and not worth botherin=
g to go back and figure out what was shipped before the version increase and=
 what wasn't, lets talk about what makes sense for the future :).

When the discussion centers on backward compatibility, and there is confusio=
n over what that actually implies, this is a central question. You snipped t=
he bit about what actually constitutes the existing protocol. I have impleme=
nted every aspect of the protocol that is widely deployed, and I can say wit=
hout question that the protocol does not require a peer to accept arbitrary m=
essages. In other words, your statement on the subject was a very relevant f=
actual error. Furthermore no reason has been demonstrated here to accept arb=
itrary traffic.

>>> Ultimately we need some kind of negotiation which is flexible in allowin=
g different software to negotiate different features without a global lock-s=
tep version number increase.
>> I have shown below how that already works.
>>> Or, to put it another way, if a feature is fully optional, why should th=
ere be a version number increase for it
>> For the reasons previously given.
>>> - the negotiation of it is independent and a version number only increas=
es confusion over which change "owns" a given version number.
>> Presumably this is why we have a standards process. Any new message impli=
es ownership. Deconflicting that is required, which implies it can easily be=
 version associated (as it has been).
>=20
> I think the point is, this doesn't work today, bumping the protocol versio=
n requires everyone agreeing on which features make sense,

As I have shown, this is not the case. While I have given my support to simp=
lifying the process, we should not proceed based on an incorrect understandi=
ng of actual behavior.

> and as we can see from this email thread alone, that isn't a common result=
 in this community. People happily ignore BIPs that make no sense, of which t=
here are a lot, and they should totally be able to do that!
>=20
> You can say that the current world works, but there's a reason over time w=
e've shifted away from the original "shove another bit on the end of the ver=
sion message, and everyone agrees on the order of those bits for new feature=
 negotiation." Version bumping is an extension of that, really.

Actually the protocol has not done this. It has used the version to signal a=
 new sub-protocol, and then in some cases that sub-protocol has been made op=
tional through subsequent negotiation. What is being proposed here is to sim=
plify that process by collapsing the secondary negotiation into the handshak=
e.

In fact I argued against this secondary ad-hoc negotiation when it began. No=
w we are coming around to recognizing that it=E2=80=99s a handshake issue, a=
s I said at the time. I=E2=80=99m glad to see that.

>>> I presume you'd support a single message that lists the set of features w=
hich a node (optionally) wishes to support on the connection. This proposal i=
s fully equivalent to that, instead opting to list them as individual messag=
es instead of one message, which is a bit nicer in that they can be handled m=
ore independently or by different subsystems including even the message hash=
ing.
>> This presumes an implementation. As part of the handshake, collection of a=
n arbitrary set of messages is a significant and unnecessary complication *o=
f the protocol*. Extension of the verack is not. It is the simplest change p=
ossible to implement the desired behavior. Each peer simply supplies the mat=
rix of sub-protocols it supports and only those supported by both are allowe=
d. There is no reason for the protocol to split that matrix up into multiple=
 messages, requiring termination. Independent messages exist because of timi=
ng or ordering requirements. Implementing dependent messages as if they were=
 independent is wasteful and complicating.
>=20
> Some things may need further negotiation. eg compact blocks sends multiple=
 redundant messages with different versions and then deduces the correct ver=
sion based on the message ordering and version set supported. Doing this via=
 verack locks you into a very specific possible negotiation protocols.

This is a moot point. Whether a list of supported optional sub-protocols is l=
isted in one or multiple messages in the handshake would not change this.

> You could extend it further and suggest a verack K-V list which allows for=
 more flexible negotiation, but I'm not sure that it isn't more complicated t=
han just shoving more messages on the wire.

There is no difference between a constrained set of key-value pairs and a di=
stinct set of options, so there is no additional complexity here.

>> I=E2=80=99m well aware of the inefficiency produced by version linearity i=
n the face of optional sub-protocols. The protocol must negotiate to the ver=
sion where it can then negotiate support, which has been done. I support cre=
ating a simpler system, eliminating these extra messages. The existing numer=
ic version can be reserved exclusively for =E2=80=9Cmust=E2=80=9D implement,=
 and can be used to signal an extension to the verack. The verack can then c=
arry a list of =E2=80=9Cmay=E2=80=9D or =E2=80=9Cshould=E2=80=9D sub-protoco=
ls for final negotiation.
>=20
> I think we agree here - the current method of protocol version bumping isn=
't scalable and something more flexible is definitely a better world.

To be clear this does not increase flexibility, it reduces communication and=
 therefore complexity, and allows peers to lock in allowed message semantics=
 by the end of the handshake, as opposed to allowing them to change at any t=
ime.

>> The format of the matrix is arbitrary, but the requirement is to list a s=
et of optional sub-protocols. This implies a namespace. This implies =E2=80=9C=
ownership=E2=80=9C of names. In other words, that coordination requirement i=
s not eliminated.
>=20
> This is true, there is some ownership requirement, we could switch to hash=
es or something of the like, but human-readable names have shown to be relat=
ively non-colliding in past Bitcoin protocol changes.

Hashes don=E2=80=99t prevent collisions. Someone can just use the same hash.=
 Bitcoin uses names (message names) and numbers (version, service, relay...)=
. It=E2=80=99s a protocol, coordination is the whole point.

e

>> e
>>> Matt
>>>=20
>>>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
>>>> =E2=80=9CBitcoin protocol has always expected clients to ignore unknown=
 messages=E2=80=9D
>>>> This is not true. Bitcoin has long implemented version negotiation, whi=
ch is the opposite expectation. Libbitcoin=E2=80=99s p2p protocol implementa=
tion immediately drops a peer that sends an invalid message according to the=
 negotiated version. The fact that a given client does not validate the prot=
ocol does not make it an expectation that the protocol not be validated.
>>>> Features can clearly be optional within an actual protocol. There have b=
een post-handshake negotiations implemented for optional messages which are v=
alid at the negotiated version. The protocol may be flexible while remaining=
 validateable. There is no reason to force a client to accept unknown messag=
e traffic.
>>>> A generalized versioning change can be implemented in or after the hand=
shake. The latter is already done on an ad-hoc basis. The former is possible=
 as long as the peer=E2=80=99s version is sufficient to be aware of the beha=
vior. This does not imply any need to send invalid messages. The verack itse=
lf can simply be extended with a matrix of feature support. There is no reas=
on to complicate negotiation with an additional message(s).
>>>> FWIW, bip37 did this poorly, adding a feature field to the version mess=
age, resulting in bip60. Due to this design, older protocol-validating clien=
ts were broken. In this case it was message length that was presumed to not b=
e validated.
>>>> e
>>>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@=
lists.linuxfoundation.org> wrote:
>>>>>=20
>>>>> =EF=BB=BFThis sounds like a great idea!
>>>>>=20
>>>>> Bitcoin is no longer a homogeneous network of one client - it is many,=
 with different features implemented in each. The Bitcoin protocol hasn't (f=
ully) evolved to capture that reality. Initially the Bitcoin protocol had a s=
imple numerical version field, but that is wholly impractical for any divers=
e network - some clients may not wish to implement every possible new relay m=
echanic, and why should they have to in order to use other new features?
>>>>>=20
>>>>> Bitcoin protocol changes have, many times in recent history, been made=
 via new dummy "negotiation" messages, which take advantage of the fact that=
 the Bitcoin protocol has always expected clients to ignore unknown messages=
. Given that pattern, it makes sense to have an explicit negotiation phase -=
 after version and before verack, just send the list of features that you su=
pport to negotiate what the connection will be capable of. The exact way we d=
o that doesn't matter much, and sending it as a stream of messages which eac=
h indicate support for a given protocol feature perfectly captures the patte=
rn that has been used in several recent network upgrades, keeping consistenc=
y.
>>>>>=20
>>>>> Matt
>>>>>=20
>>>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
>>>>>> Hi,
>>>>>> Back in February I posted a proposal for WTXID-based transaction rela=
y[1] (now known as BIP 339), which included a proposal for feature negotiati=
on to take place prior to the VERACK message being received by each side.  I=
n my email to this list, I had asked for feedback as to whether that proposa=
l was problematic, and didn't receive any responses.
>>>>>> Since then, the implementation of BIP 339 has been merged into Bitcoi=
n Core, though it has not yet been released.
>>>>>> In thinking about the mechanism used there, I thought it would be hel=
pful to codify in a BIP the idea that Bitcoin network clients should ignore u=
nknown messages received before a VERACK.  A draft of my proposal is availab=
le here[2].
>>>>>> I presume that software upgrading past protocol version 70016 was alr=
eady planning to either implement BIP 339, or ignore the wtxidrelay message p=
roposed in BIP 339 (if not, then this would create network split concerns in=
 the future -- so I hope that someone would speak up if this were a problem)=
.  When we propose future protocol upgrades that would benefit from feature n=
egotiation at the time of connection, I think it would be nice to be able to=
 use the same method as proposed in BIP 339, without even needing to bump th=
e protocol version.  So having an understanding that this is the standard of=
 how other network clients operate would be helpful.
>>>>>> If, on the other hand, this is problematic for some reason, I look fo=
rward to hearing that as well, so that we can be careful about how we deploy=
 future p2p changes to avoid disruption.
>>>>>> Thanks,
>>>>>> Suhas Daftuar
>>>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-Febr=
uary/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/20=
20-February/017648.html>
>>>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature=
-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaft=
uar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negoti=
ation.mediawiki>
>>>>>> _______________________________________________
>>>>>> bitcoin-dev mailing list
>>>>>> bitcoin-dev@lists.linuxfoundation.org
>>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>>> _______________________________________________
>>>>> bitcoin-dev mailing list
>>>>> bitcoin-dev@lists.linuxfoundation.org
>>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev