Content-Type: multipart/alternative;
 boundary=Apple-Mail-601D1338-A5BA-40E5-8410-A242E48E8B36
Content-Transfer-Encoding: 7bit
From: Eric Voskuil <eric@voskuil.org>
Mime-Version: 1.0 (1.0)
Date: Tue, 18 Aug 2020 09:54:58 -0700
Message-Id: <84F1FD44-0482-4F77-8FFA-65492E445D23@voskuil.org>
References: <8a7031f2-a598-ac09-f2df-1672cd82980b@mattcorallo.com>
In-Reply-To: <8a7031f2-a598-ac09-f2df-1672cd82980b@mattcorallo.com>
To: Matt Corallo <lf-lists@mattcorallo.com>,
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Generalizing feature negotiation when new p2p
	connections are setup
Precedence: list


--Apple-Mail-601D1338-A5BA-40E5-8410-A242E48E8B36
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

=E2=80=9CBitcoin protocol has always expected clients to ignore unknown mess=
ages=E2=80=9D

This is not true. Bitcoin has long implemented version negotiation, which is=
 the opposite expectation. Libbitcoin=E2=80=99s p2p protocol implementation i=
mmediately drops a peer that sends an invalid message according to the negot=
iated version. The fact that a given client does not validate the protocol d=
oes not make it an expectation that the protocol not be validated.

Features can clearly be optional within an actual protocol. There have been p=
ost-handshake negotiations implemented for optional messages which are valid=
 at the negotiated version. The protocol may be flexible while remaining val=
idateable. There is no reason to force a client to accept unknown message tr=
affic.

A generalized versioning change can be implemented in or after the handshake=
. The latter is already done on an ad-hoc basis. The former is possible as l=
ong as the peer=E2=80=99s version is sufficient to be aware of the behavior.=
 This does not imply any need to send invalid messages. The verack itself ca=
n simply be extended with a matrix of feature support. There is no reason to=
 complicate negotiation with an additional message(s).

FWIW, bip37 did this poorly, adding a feature field to the version message, r=
esulting in bip60. Due to this design, older protocol-validating clients wer=
e broken. In this case it was message length that was presumed to not be val=
idated.

e

> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev@lists=
.linuxfoundation.org> wrote:
>=20
> =EF=BB=BFThis sounds like a great idea!
>=20
> Bitcoin is no longer a homogeneous network of one client - it is many, wit=
h different features implemented in each. The Bitcoin protocol hasn't (fully=
) evolved to capture that reality. Initially the Bitcoin protocol had a simp=
le numerical version field, but that is wholly impractical for any diverse n=
etwork - some clients may not wish to implement every possible new relay mec=
hanic, and why should they have to in order to use other new features?
>=20
> Bitcoin protocol changes have, many times in recent history, been made via=
 new dummy "negotiation" messages, which take advantage of the fact that the=
 Bitcoin protocol has always expected clients to ignore unknown messages. Gi=
ven that pattern, it makes sense to have an explicit negotiation phase - aft=
er version and before verack, just send the list of features that you suppor=
t to negotiate what the connection will be capable of. The exact way we do t=
hat doesn't matter much, and sending it as a stream of messages which each i=
ndicate support for a given protocol feature perfectly captures the pattern t=
hat has been used in several recent network upgrades, keeping consistency.
>=20
> Matt
>=20
>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
>> Hi,
>> Back in February I posted a proposal for WTXID-based transaction relay[1]=
 (now known as BIP 339), which included a proposal for feature negotiation t=
o take place prior to the VERACK message being received by each side.  In my=
 email to this list, I had asked for feedback as to whether that proposal wa=
s problematic, and didn't receive any responses.
>> Since then, the implementation of BIP 339 has been merged into Bitcoin Co=
re, though it has not yet been released.
>> In thinking about the mechanism used there, I thought it would be helpful=
 to codify in a BIP the idea that Bitcoin network clients should ignore unkn=
own messages received before a VERACK.  A draft of my proposal is available h=
ere[2].
>> I presume that software upgrading past protocol version 70016 was already=
 planning to either implement BIP 339, or ignore the wtxidrelay message prop=
osed in BIP 339 (if not, then this would create network split concerns in th=
e future -- so I hope that someone would speak up if this were a problem).  W=
hen we propose future protocol upgrades that would benefit from feature nego=
tiation at the time of connection, I think it would be nice to be able to us=
e the same method as proposed in BIP 339, without even needing to bump the p=
rotocol version.  So having an understanding that this is the standard of ho=
w other network clients operate would be helpful.
>> If, on the other hand, this is problematic for some reason, I look forwar=
d to hearing that as well, so that we can be careful about how we deploy fut=
ure p2p changes to avoid disruption.
>> Thanks,
>> Suhas Daftuar
>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February=
/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-Fe=
bruary/017648.html>
>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-neg=
otiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/=
bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiatio=
n.mediawiki>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

--Apple-Mail-601D1338-A5BA-40E5-8410-A242E48E8B36
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div dir=3D"ltr">=E2=80=9CBitcoin protocol h=
as always expected clients to ignore unknown messages=E2=80=9D</div><div dir=
=3D"ltr"><br></div><div dir=3D"ltr">This is not true. Bitcoin has long imple=
mented version negotiation, which is the opposite expectation. Libbitcoin=E2=
=80=99s p2p protocol implementation immediately drops a peer that sends an i=
nvalid message according to the negotiated version.&nbsp;<span style=3D"care=
t-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">The fact that a given client do=
es not validate the protocol does not make it an expectation that the protoc=
ol not be validated.</span></div><div dir=3D"ltr"><br></div><div dir=3D"ltr"=
><span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Features ca=
n clearly be optional within an actual protocol.&nbsp;</span><span style=3D"=
caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">There have been post-handsh=
ake negotiations implemented for optional messages which are valid at the ne=
gotiated version. The protocol may be flexible while remaining validateable.=
 There is no reason to force a client to accept unknown message traffic.</sp=
an></div><div dir=3D"ltr"><span style=3D"caret-color: rgb(0, 0, 0); color: r=
gb(0, 0, 0);"><br></span></div><div dir=3D"ltr"><font color=3D"#000000"><spa=
n style=3D"caret-color: rgb(0, 0, 0);">A generalized versioning change can b=
e implemented in or after the handshake. The latter is already done on an ad=
-hoc basis. The former is possible as long as the peer=E2=80=99s version is s=
ufficient to be aware of the behavior. This does not imply any need to send i=
nvalid messages. The verack itself can simply be extended with a matrix of f=
eature support. There is no reason to complicate negotiation with an additio=
nal message(s).</span></font></div><div dir=3D"ltr"><font color=3D"#000000">=
<span style=3D"caret-color: rgb(0, 0, 0);"><br></span></font></div><div dir=3D=
"ltr"><font color=3D"#000000"><span style=3D"caret-color: rgb(0, 0, 0);">FWI=
W, bip37 did this poorly, adding a feature field to the version message, res=
ulting in bip60. Due to this design, older protocol-validating clients were b=
roken. In this case it was message length that was presumed to not be valida=
ted.</span></font></div><div dir=3D"ltr"><br></div><div dir=3D"ltr">e</div><=
div dir=3D"ltr"><br><blockquote type=3D"cite">On Aug 18, 2020, at 07:59, Mat=
t Corallo via bitcoin-dev &lt;bitcoin-dev@lists.linuxfoundation.org&gt; wrot=
e:<br><br></blockquote></div><blockquote type=3D"cite"><div dir=3D"ltr">=EF=BB=
=BF<span>This sounds like a great idea!</span><br><span></span><br><span>Bit=
coin is no longer a homogeneous network of one client - it is many, with dif=
ferent features implemented in each. The Bitcoin protocol hasn't (fully) evo=
lved to capture that reality. Initially the Bitcoin protocol had a simple nu=
merical version field, but that is wholly impractical for any diverse networ=
k - some clients may not wish to implement every possible new relay mechanic=
, and why should they have to in order to use other new features?</span><br>=
<span></span><br><span>Bitcoin protocol changes have, many times in recent h=
istory, been made via new dummy "negotiation" messages, which take advantage=
 of the fact that the Bitcoin protocol has always expected clients to ignore=
 unknown messages. Given that pattern, it makes sense to have an explicit ne=
gotiation phase - after version and before verack, just send the list of fea=
tures that you support to negotiate what the connection will be capable of. T=
he exact way we do that doesn't matter much, and sending it as a stream of m=
essages which each indicate support for a given protocol feature perfectly c=
aptures the pattern that has been used in several recent network upgrades, k=
eeping consistency.</span><br><span></span><br><span>Matt</span><br><span></=
span><br><span>On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:</spa=
n><br><blockquote type=3D"cite"><span>Hi,</span><br></blockquote><blockquote=
 type=3D"cite"><span>Back in February I posted a proposal for WTXID-based tr=
ansaction relay[1] (now known as BIP 339), which included a proposal for fea=
ture negotiation to take place prior to the VERACK message being received by=
 each side.&nbsp; In my email to this list, I had asked for feedback as to w=
hether that proposal was problematic, and didn't receive any responses.</spa=
n><br></blockquote><blockquote type=3D"cite"><span>Since then, the implement=
ation&nbsp;of BIP 339 has been merged into Bitcoin Core, though it has not y=
et been released.</span><br></blockquote><blockquote type=3D"cite"><span>In t=
hinking about the mechanism used there, I thought it would&nbsp;be helpful t=
o codify in a BIP the idea that Bitcoin network clients should ignore unknow=
n messages received before a VERACK.&nbsp; A draft of my proposal is availab=
le here[2].</span><br></blockquote><blockquote type=3D"cite"><span>I presume=
 that software upgrading past protocol version 70016 was already planning to=
 either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 3=
39 (if not, then this would create network split concerns in the future -- s=
o I hope that someone would speak up if this were a problem).&nbsp; When we p=
ropose future protocol upgrades that would benefit from feature negotiation a=
t the time of connection, I think it would be nice to be able to use the sam=
e method as proposed in BIP 339, without even needing to bump the protocol v=
ersion.&nbsp; So having an understanding that this is the standard of how ot=
her network clients operate would be helpful.</span><br></blockquote><blockq=
uote type=3D"cite"><span>If, on the other hand, this is problematic for some=
 reason, I look forward to hearing that as well, so that we can be careful a=
bout how we deploy future p2p changes to avoid disruption.</span><br></block=
quote><blockquote type=3D"cite"><span>Thanks,</span><br></blockquote><blockq=
uote type=3D"cite"><span>Suhas Daftuar</span><br></blockquote><blockquote ty=
pe=3D"cite"><span>[1] https://lists.linuxfoundation.org/pipermail/bitcoin-de=
v/2020-February/017648.html &lt;https://lists.linuxfoundation.org/pipermail/=
bitcoin-dev/2020-February/017648.html&gt;</span><br></blockquote><blockquote=
 type=3D"cite"><span>[2] https://github.com/sdaftuar/bips/blob/2020-08-gener=
alized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki &lt;https:/=
/github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p=
2p-feature-negotiation.mediawiki&gt;</span><br></blockquote><blockquote type=
=3D"cite"><span>_______________________________________________</span><br></=
blockquote><blockquote type=3D"cite"><span>bitcoin-dev mailing list</span><b=
r></blockquote><blockquote type=3D"cite"><span>bitcoin-dev@lists.linuxfounda=
tion.org</span><br></blockquote><blockquote type=3D"cite"><span>https://list=
s.linuxfoundation.org/mailman/listinfo/bitcoin-dev</span><br></blockquote><s=
pan>_______________________________________________</span><br><span>bitcoin-=
dev mailing list</span><br><span>bitcoin-dev@lists.linuxfoundation.org</span=
><br><span>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</s=
pan><br></div></blockquote></body></html>=

--Apple-Mail-601D1338-A5BA-40E5-8410-A242E48E8B36--