path: root/22/266b42ef70befdb547e8b8b006f3821223aed4

Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
	helo=mx.sourceforge.net)
	by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <pieter.wuille@gmail.com>) id 1UpcU6-0000WH-1t
	for bitcoin-development@lists.sourceforge.net;
	Thu, 20 Jun 2013 10:52:58 +0000
Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.223.173 as permitted sender)
	client-ip=209.85.223.173; envelope-from=pieter.wuille@gmail.com;
	helo=mail-ie0-f173.google.com; 
Received: from mail-ie0-f173.google.com ([209.85.223.173])
	by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1UpcU4-0002uv-KH
	for bitcoin-development@lists.sourceforge.net;
	Thu, 20 Jun 2013 10:52:58 +0000
Received: by mail-ie0-f173.google.com with SMTP id k13so16094116iea.32
	for <bitcoin-development@lists.sourceforge.net>;
	Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.42.47.69 with SMTP id n5mr2975623icf.24.1371725571261; Thu,
	20 Jun 2013 03:52:51 -0700 (PDT)
Received: by 10.50.149.233 with HTTP; Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
Received: by 10.50.149.233 with HTTP; Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
In-Reply-To: <1371724625.50978.YahooMailNeo@web162706.mail.bf1.yahoo.com>
References: <4DE0E45E-BB48-4DFF-9C86-ACBE312B3049@bitsofproof.com>
	<CANEZrP1MVKSNU92UrQ36sPCN0LNLKrVRXnfVwEZKhgoBFXatBA@mail.gmail.com>
	<20130620090649.GA17765@vps7135.xlshosting.net>
	<CANEZrP2cLR9sYZ59O-S6HNi9bKPtaY5yOkvVx+O8XAC792acZg@mail.gmail.com>
	<1371724625.50978.YahooMailNeo@web162706.mail.bf1.yahoo.com>
Date: Thu, 20 Jun 2013 12:52:51 +0200
Message-ID: <CAPg+sBiJvWckJYa-iw=qtjfj1PoY_bKhFD3sx6LqfUrFbWrGHA@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Turkey Breast <turkeybreast@yahoo.com>
Content-Type: multipart/alternative; boundary=90e6ba6149c667c07804df93bf35
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(pieter.wuille[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UpcU4-0002uv-KH
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 20 Jun 2013 10:52:58 -0000

--90e6ba6149c667c07804df93bf35
Content-Type: text/plain; charset=ISO-8859-1

Let's just increase the version number and be done with this discussion.
It's a small benefit, but it simplifies things and it's trivial to do.

I don't understand how a policy of requiring version increases could limit
future extensions: after the version/verack exchange, the protocol version
is negotiated between peers, and there is no need for anything optional
anymore.

Note thay this is just about parsing, not about relaying - you should still
relay parts of a message you haven't parsed. But that doesn't apply to the
version message anyway, which is the only place where this matters.

-- 
Pieter
 On 20 Jun 2013 12:38, "Turkey Breast" <turkeybreast@yahoo.com> wrote:

> I don't get why this is such a contentious change?
>
> Before I was able to use asserts to check the expected length of length of
> messages per protocol version, I could pass in dumb iterators that just
> parse the byte stream and I could serialize and deserialize a message to
> check the parser is correct (in debug mode).
>
> This 'simple' change causes all that behaviour to be lost. You can no
> longer just use iterators but must know the remaining length (or if you use
> std::distance, you can only use specific std containers - not just anything
> with an iterator and an operator++). You cannot check the deserialization
> process by serializing the deserialized message and comparing it to the
> original data (because the bool is always present in the serializer).
>
> It's a bit stupid you call it buggy code when this behaviour has never
> been present in Bitcoin. The BIP doesn't introduce any unwanted
> side-effects and is a trivial reasonable change.
>
> If you want optional fields then the proper way to do it, is to either set
> a flag in the Services field of the "version" message to indicate different
> formats for messages (i.e use this template structure for a message, not
> that one), introduce a new message (if the changes are big), to
> approve/improve Stefan's BIP 32 for custom services or to have a value in
> the byte stream indicating which fields are present (maybe a bitfield or
> so).
>
> Using a quirk of an implementation is just bad form and sloppy coding.
> Optional fields should have their own mechanism that allows them to remain
> as optional fields between protocol version upgrades.
>
> The bitcoind software can probably be improved too, by checking that the
> length of the version message is consistent for the protocol version given
> by the connected node. Right now it makes no assumptions based on that
> which is a mistake (new clients can broadcast older version messages that
> don't have all the fields required). Probably the software should penalise
> hosts which do that.
>
> What's the big deal to update the protocol version number from 70001 to
> 70002? It's not like we'll run out of integers. The field has now gone from
> optional to required now anyway - that's a behaviour change. It'd be good
> to enforce that. I see this as a bug.
>
>   ------------------------------
>  *From:* Mike Hearn <mike@plan99.net>
> *To:* Pieter Wuille <pieter.wuille@gmail.com>
> *Cc:* Bitcoin Dev <bitcoin-development@lists.sourceforge.net>; Tamas
> Blummer <tamas@bitsofproof.com>
> *Sent:* Thursday, June 20, 2013 11:17 AM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>
> There's no problem, but there's no benefit either. It also locks us in to
> a potentially problematic guarantee - what if in future we want to have,
> say, two optional new pieces of data in two different messages. We don't
> want to require that if version > X then you have to implement all features
> up to and including that point.
>
> Essentially the number of fields in a message is like a little version
> number, just for that message. It adds flexibility to keep it that way, and
> there's no downside, seeing as that bridge was already crossed and people
> with parsers that can't handle it need to fix their code anyway.
>
> So I have a slight preference for keeping things the way they are, it
> keeps things flexible for future and costs nothing.
>
>
>
> On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille <pieter.wuille@gmail.com>wrote:
>
> On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> > Sure but why not do that when there's an actual new field to add? Does
> > anyone have a proposal for a feature that needs a new version field at
> the
> > moment? There's no point changing the protocol now unless there's
> actually
> > a new field to add.
> >
> > Anyway I still don't see why anyone cares about this issue. The Bitcoin
> > protocol does not and never has required that all messages have a fixed
> > number of fields per version. Any parser written on the assumption it did
> > was just buggy. Look at how tx messages are relayed for the most obvious
> > example of that pattern in action - it's actually the raw byte stream
> > that's stored and relayed to ensure that fields added in new versions
> > aren't dropped during round-tripping. Old versions are supposed to
> preserve
> > fields from the future.
>
> Actually, that is not the same issue. What is being argued for here is that
> the version in the version message itself should indicate which fields are
> present, so a parser doesn't need to look at the length of the message.
> That
> seems like a minor but very reasonable request to me, and it's trivial to
> do.
> That doesn't mean that you may receive versions higher than what you know
> of,
> and thus messages with fields you don't know about. That doesn't matter,
> you
> can just ignore them.
>
> I see no problem with raising the protocol version number to indicate
> "all fields up to fRelayTxes are required, if the announced nVersion is
> above N".
> In fact, I believe (though haven't checked) all previous additions to the
> version
> message were accompanied with a protocol version (then: client version)
> increase
> as well.
>
> --
> Pieter
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

--90e6ba6149c667c07804df93bf35
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Let&#39;s just increase the version number and be done with =
this discussion. It&#39;s a small benefit, but it simplifies things and it&=
#39;s trivial to do.</p>
<p dir=3D"ltr">I don&#39;t understand how a policy of requiring version inc=
reases could limit future extensions: after the version/verack exchange, th=
e protocol version is negotiated between peers, and there is no need for an=
ything optional anymore.</p>

<p dir=3D"ltr">Note thay this is just about parsing, not about relaying - y=
ou should still relay parts of a message you haven&#39;t parsed. But that d=
oesn&#39;t apply to the version message anyway, which is the only place whe=
re this matters.</p>

<p dir=3D"ltr">-- <br>
Pieter<br>
</p>
<div class=3D"gmail_quote">On 20 Jun 2013 12:38, &quot;Turkey Breast&quot; =
&lt;<a href=3D"mailto:turkeybreast@yahoo.com">turkeybreast@yahoo.com</a>&gt=
; wrote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style=3D"font-size:12pt;font-family:times new roman,new york,time=
s,serif"><div><span>I don&#39;t get why this is such a contentious change?<=
/span></div><div style=3D"font-style:normal;font-size:16px;background-color=
:transparent;font-family:times new roman,new york,times,serif">
<br><span></span></div><div style=3D"font-style:normal;font-size:16px;backg=
round-color:transparent;font-family:times new roman,new york,times,serif"><=
span>Before I was able to use asserts to check the expected length of lengt=
h of messages per protocol version, I could pass in dumb iterators that jus=
t parse the byte stream and I could serialize and deserialize a message to =
check the parser is correct (in debug mode).</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>This &#39;simple&#39; change causes all that behaviour to be lost. Yo=
u can no longer just use iterators but must know the remaining length (or i=
f you use std::distance, you can only use specific std containers - not jus=
t anything with an iterator and an operator++). You cannot check the deseri=
alization process by serializing the deserialized message and comparing it =
to the original data (because the bool is always present in the serializer)=
.</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>It&#39;s a bit stupid you call it buggy code when this behaviour has =
never been present in Bitcoin. The BIP doesn&#39;t introduce any unwanted s=
ide-effects and is a trivial reasonable change.</span></div><div style=3D"f=
ont-style:normal;font-size:16px;background-color:transparent;font-family:ti=
mes new roman,new york,times,serif">
<br><span></span></div><div style=3D"font-style:normal;font-size:16px;backg=
round-color:transparent;font-family:times new roman,new york,times,serif"><=
span>If you want optional fields then the proper way to do it, is to either=
 set a flag in the Services field of the &quot;version&quot; message to ind=
icate different formats for messages (i.e use this template structure for a=
 message, not that one), introduce a new message (if the changes are big), =
to approve/improve Stefan&#39;s BIP 32 for custom services or to have a val=
ue in the byte stream indicating which fields are present (maybe
 a bitfield or so).</span></div><div style=3D"font-style:normal;font-size:1=
6px;background-color:transparent;font-family:times new roman,new york,times=
,serif"><br><span></span></div><div style=3D"font-style:normal;font-size:16=
px;background-color:transparent;font-family:times new roman,new york,times,=
serif">
<span>Using a quirk of an implementation is just bad form and sloppy coding=
. Optional fields should have their own mechanism that allows them to remai=
n as optional fields between protocol version upgrades.<br></span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>The bitcoind software can probably be improved
 too, by checking that the length of the version message is consistent for =
the protocol version given by the connected node. Right now it makes no ass=
umptions based on that which is a mistake (new clients can broadcast older =
version messages that don&#39;t have all the fields required). Probably the=
 software should penalise hosts which do that.</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>What&#39;s the big deal to update the protocol version number from 70=
001 to 70002? It&#39;s not like we&#39;ll run out of integers. The field ha=
s now gone from optional to required now anyway - that&#39;s a behaviour ch=
ange. It&#39;d be good to enforce that. I see this as a
 bug.<br></span></div><div><br></div>  <div style=3D"font-family:times new =
roman,new york,times,serif;font-size:12pt"> <div style=3D"font-family:times=
 new roman,new york,times,serif;font-size:12pt"> <div dir=3D"ltr"> <hr size=
=3D"1">
  <font face=3D"Arial"> <b><span style=3D"font-weight:bold">From:</span></b=
> Mike Hearn &lt;<a href=3D"mailto:mike@plan99.net" target=3D"_blank">mike@=
plan99.net</a>&gt;<br> <b><span style=3D"font-weight:bold">To:</span></b> P=
ieter Wuille &lt;<a href=3D"mailto:pieter.wuille@gmail.com" target=3D"_blan=
k">pieter.wuille@gmail.com</a>&gt; <br>
<b><span style=3D"font-weight:bold">Cc:</span></b> Bitcoin Dev &lt;<a href=
=3D"mailto:bitcoin-development@lists.sourceforge.net" target=3D"_blank">bit=
coin-development@lists.sourceforge.net</a>&gt;; Tamas Blummer &lt;<a href=
=3D"mailto:tamas@bitsofproof.com" target=3D"_blank">tamas@bitsofproof.com</=
a>&gt; <br>
 <b><span style=3D"font-weight:bold">Sent:</span></b> Thursday, June 20, 20=
13 11:17 AM<br> <b><span style=3D"font-weight:bold">Subject:</span></b> Re:=
 [Bitcoin-development] Missing fRelayTxes in version<br> </font> </div> <di=
v>
<br>
<div><div dir=3D"ltr">There&#39;s no problem, but there&#39;s no benefit ei=
ther. It also locks us in to a potentially problematic guarantee - what if =
in future we want to have, say, two optional new pieces of data in two diff=
erent messages. We don&#39;t want to require that if version &gt; X then yo=
u have to implement all features up to and including that point.<div>

<br></div><div>Essentially the number of fields in a message is like a litt=
le version number, just for that message. It adds flexibility to keep it th=
at way, and there&#39;s no downside, seeing as that bridge was already cros=
sed and people with parsers that can&#39;t handle it need to fix their code=
 anyway.</div>

<div><br></div><div>So I have a slight preference for keeping things the wa=
y they are, it keeps things flexible for future and costs nothing.</div><di=
v><br></div></div><div><br><br><div>
On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille <span dir=3D"ltr">&lt;<a re=
l=3D"nofollow" href=3D"mailto:pieter.wuille@gmail.com" target=3D"_blank">pi=
eter.wuille@gmail.com</a>&gt;</span> wrote:<br><blockquote style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div>On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:<br>
&gt; Sure but why not do that when there&#39;s an actual new field to add? =
Does<br>
&gt; anyone have a proposal for a feature that needs a new version field at=
 the<br>
&gt; moment? There&#39;s no point changing the protocol now unless there&#3=
9;s actually<br>
&gt; a new field to add.<br>
&gt;<br>
&gt; Anyway I still don&#39;t see why anyone cares about this issue. The Bi=
tcoin<br>
&gt; protocol does not and never has required that all messages have a fixe=
d<br>
&gt; number of fields per version. Any parser written on the assumption it =
did<br>
&gt; was just buggy. Look at how tx messages are relayed for the most obvio=
us<br>
&gt; example of that pattern in action - it&#39;s actually the raw byte str=
eam<br>
&gt; that&#39;s stored and relayed to ensure that fields added in new versi=
ons<br>
&gt; aren&#39;t dropped during round-tripping. Old versions are supposed to=
 preserve<br>
&gt; fields from the future.<br>
<br>
</div>Actually, that is not the same issue. What is being argued for here i=
s that<br>
the version in the version message itself should indicate which fields are<=
br>
present, so a parser doesn&#39;t need to look at the length of the message.=
 That<br>
seems like a minor but very reasonable request to me, and it&#39;s trivial =
to do.<br>
That doesn&#39;t mean that you may receive versions higher than what you kn=
ow of,<br>
and thus messages with fields you don&#39;t know about. That doesn&#39;t ma=
tter, you<br>
can just ignore them.<br>
<br>
I see no problem with raising the protocol version number to indicate<br>
&quot;all fields up to fRelayTxes are required, if the announced nVersion i=
s above N&quot;.<br>
In fact, I believe (though haven&#39;t checked) all previous additions to t=
he version<br>
message were accompanied with a protocol version (then: client version) inc=
rease<br>
as well.<br>
<span><font color=3D"#888888"><br>
--<br>
Pieter<br>
<br>
</font></span></blockquote></div><br></div>
</div><br>-----------------------------------------------------------------=
-------------<br>This SF.net email is sponsored by Windows:<br><br>Build fo=
r Windows Store.<br><br><a href=3D"http://p.sf.net/sfu/windows-dev2dev" tar=
get=3D"_blank">http://p.sf.net/sfu/windows-dev2dev</a><br>
_______________________________________________<br>Bitcoin-development mail=
ing list<br><a href=3D"mailto:Bitcoin-development@lists.sourceforge.net" ta=
rget=3D"_blank">Bitcoin-development@lists.sourceforge.net</a><br><a href=3D=
"https://lists.sourceforge.net/lists/listinfo/bitcoin-development" target=
=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-developmen=
t</a><br>
<br><br></div> </div> </div>  </div></div><br>-----------------------------=
-------------------------------------------------<br>
This SF.net email is sponsored by Windows:<br>
<br>
Build for Windows Store.<br>
<br>
<a href=3D"http://p.sf.net/sfu/windows-dev2dev" target=3D"_blank">http://p.=
sf.net/sfu/windows-dev2dev</a><br>_________________________________________=
______<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
<br></blockquote></div>

--90e6ba6149c667c07804df93bf35--