Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1UpZ0W-0003Vs-F3 for bitcoin-development@lists.sourceforge.net; Thu, 20 Jun 2013 07:10:12 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.219.53 as permitted sender) client-ip=209.85.219.53; envelope-from=mh.in.england@gmail.com; helo=mail-oa0-f53.google.com; Received: from mail-oa0-f53.google.com ([209.85.219.53]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1UpZ0U-0001xW-Md for bitcoin-development@lists.sourceforge.net; Thu, 20 Jun 2013 07:10:12 +0000 Received: by mail-oa0-f53.google.com with SMTP id k14so7522816oag.26 for ; Thu, 20 Jun 2013 00:10:05 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.124.18 with SMTP id me18mr3819578oeb.100.1371712205259; Thu, 20 Jun 2013 00:10:05 -0700 (PDT) Sender: mh.in.england@gmail.com Received: by 10.76.23.36 with HTTP; Thu, 20 Jun 2013 00:10:05 -0700 (PDT) In-Reply-To: <1371709237.57104.YahooMailNeo@web162701.mail.bf1.yahoo.com> References: <1371709237.57104.YahooMailNeo@web162701.mail.bf1.yahoo.com> Date: Thu, 20 Jun 2013 09:10:05 +0200 X-Google-Sender-Auth: tMjnP5WFWtIotIujyidsPNAFOyU Message-ID: From: Mike Hearn To: Turkey Breast Content-Type: multipart/alternative; boundary=047d7b450304ba8b0504df90a2f9 X-Spam-Score: -0.2 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (mh.in.england[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message 0.3 HTML_FONT_FACE_BAD BODY: HTML font face is not a word 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1UpZ0U-0001xW-Md Cc: "bitcoin-development@lists.sourceforge.net" Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2013 07:10:12 -0000 --047d7b450304ba8b0504df90a2f9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The protocol version was bumped when Bloom filtering was added so there's not much point bumping it again - you have to handle the old clients no matter what. Nobody brought this up as an issue when the BIP or code was first written and as you can see from main.cpp, it was done this way to be consistent with how other version fields are handled: if (!vRecv.empty()) vRecv >> addrFrom >> nNonce; if (!vRecv.empty()) vRecv >> pfrom->strSubVer; if (!vRecv.empty()) vRecv >> pfrom->nStartingHeight; if (!vRecv.empty()) vRecv >> pfrom->fRelayTxes; // set to true after we get the first filter* message The existence of the nStartingHeight field for instance depends on the message length and not anything else. Anyway, are you really asking for the protocol to be changed to work around an issue specific to how you wrote your parsing code? This is the first time anyone has suggested this minor detail is a problem. It doesn't present any issues for the C++ code or bitcoinj where message objects know their own length at parse time. On Thu, Jun 20, 2013 at 8:20 AM, Turkey Breast wrot= e: > I never said that Bitcoin message field lengths should always be the same= . > But before this change they certainly were constant per protocol version. > All I'm saying is that optional lengths shouldn't be used (a field exists > or not) and for every field change, the protocol version should be upgrad= ed. > > Now that fRelayTxes is part of the protocol, the version number should be > upgraded to reflect this fact. > > ------------------------------ > *From:* Mike Hearn > *To:* Paul Lyon > *Cc:* Turkey Breast ; " > bitcoin-development@lists.sourceforge.net" < > bitcoin-development@lists.sourceforge.net> > *Sent:* Wednesday, June 19, 2013 3:20 PM > > *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version messag= e > > If you want to criticise the Bitcoin protocol for sloppyness, the variabl= e > length of some messages isn't where I'd start. > > Note that ping has the same issue, its length has changed over time to > include the nonce. > > If your parser can't handle that kind of thing, you need to fix it. The > protocol has always worked that way. > > > > On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon wrote: > > I=E2=80=99m also running into this exact same issue with my parser, now = I > understand why the relay field behavior I was seeing doesn=E2=80=99t matc= h the wiki. > > So to parse a version message, you can=E2=80=99t rely on the protocol ver= sion? You > have to know how long the payload is, and then parse the message > accordingly? I agree with Turkey Breast, this seems a bit sloppy to me. > > Paul > > P.S. I=E2=80=99ve never used a dev mailing list before and I want to get = involved > with the Bitcoin dev community, so let me know if I=E2=80=99m horribly vi= olating > any mailing list etiquette. =F0=9F=98=8A > > *From:* Mike Hearn > *Sent:* =E2=80=8EWednesday=E2=80=8E, =E2=80=8EJune=E2=80=8E =E2=80=8E19= =E2=80=8E, =E2=80=8E2013 =E2=80=8E7=E2=80=8E:=E2=80=8E43=E2=80=8E =E2=80=8E= AM > *To:* Turkey Breast > *Cc:* bitcoin-development@lists.sourceforge.net > > Bitcoin-Qt on master does send it now although it doesn't affect anything= , > but as old pre-filtering versions will continue to exist, you'll always > have to be able to deserialize version messages without it. > > Bitcoin version messages have always had variable length, look at how the > code is written in main.cpp. If you didn't experience issues until now al= l > it means is that no sufficiently old nodes were talking to yours. > > The standard does not say it should appear. Read it again - BIP 37 says > about the new version message field: > If false then broadcast transactions will not be announced until a > filter{load,add,clear} command is received. *If missing or true*, no > change in protocol behaviour occurs. > > > On Wed, Jun 19, 2013 at 12:33 PM, Turkey Breast w= rote: > > It's a problem if you work with iterators to deserialize the byte stream. > Even failing that, it's just sloppy programming. What happens in the futu= re > when new fields are added to the version message? It's not a big deal to > say that this protocol version has X number of fields, that (higher) > protocol version message has X + N number of fields. Deterministic number > of fields per protocol version is sensical and how Bitcoin has been for a > long time. > > And yes, it was a problem for me that caused a lot of confusion why this > byte didn't exist in many version messages despite the standard saying it > should and the code in bitcoind indicating it should. Nowhere was this > written. It doesn't help other implementations to have an unclear behavio= ur > that depends on some magic from one implementation. > > ------------------------------ > *From:* Mike Hearn > *To:* Turkey Breast > *Cc:* "bitcoin-development@lists.sourceforge.net" < > bitcoin-development@lists.sourceforge.net> > *Sent:* Wednesday, June 19, 2013 11:39 AM > > *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version messag= e > > It has to be optional because old clients don't send it, obviously. > > Why is this even an issue? There's no problem with variable length > messages in any codebase that I'm aware of. Is this solving some actual > problem? > > > On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast w= rote: > > That's me. I never said to make all messages fixed length. I said to make > a fixed number of fields per protocol. So given a protocol version number= , > you know the number of fields in a message. This is not only easier for > parsing messages, but just good practice. I don't see why a 1 byte flag > needs to be optional anyway. > > ------------------------------ > *From:* Mike Hearn > *To:* Turkey Breast > *Cc:* Bitcoin Dev > *Sent:* Tuesday, June 18, 2013 9:48 PM > *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version messag= e > > It's not a bug (although there was recently a change to make bitcoind/qt > always send this field anyway). > > I don't know where Amir is going with BIP 60. Version messages have alway= s > been variable length. There's nothing inherent in the Bitcoin protocol th= at > says all messages are fixed length, indeed, tx messages are allowed to ha= ve > arbitrary data appended after them that gets relayed. > > > On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast wr= ote: > > See this BIP. I'm not sure if this is a bug or what, but it would be good > if messages always had a fixed number of fields per protocol version. > > https://en.bitcoin.it/wiki/BIP_0060#Code_Updates > > This BIP details everything that needs to be done and proposes a protocol > upgrade. > > > -------------------------------------------------------------------------= ----- > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > > > -------------------------------------------------------------------------= ----- > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > > > -------------------------------------------------------------------------= ----- > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > > > > -------------------------------------------------------------------------= ----- > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > --047d7b450304ba8b0504df90a2f9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The protocol version was bumped when Bloom filtering was a= dded so there's not much point bumping it again - you have to handle th= e old clients no matter what. Nobody brought this up as an issue when the B= IP or code was first written and as you can see from main.cpp, it was done = this way to be consistent with how other version fields are handled:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (!vRecv.empty())
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 vRecv >> addrFrom >&g= t; nNonce;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (!vRecv.empty())
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 vRecv >> pfrom->strS= ubVer;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (!vRecv.empty())
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 vRecv >> pfrom->nSt= artingHeight;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (!vRecv.empty())
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 vRecv >> pfrom->f= RelayTxes; // set to true after we get the first filter* message

The existence of the nStartingHeight field for in= stance depends on the message length and not anything else.

Anyway, are you really asking for the protocol to be = changed to work around an issue specific to how you wrote your parsing code= ? This is the first time anyone has suggested this minor detail is a proble= m. It doesn't present any issues for the C++ code or bitcoinj where mes= sage objects know their own length at parse time.



On Thu, Jun 20, 2013 at 8:20 AM, Turkey Breast <turkeyb= reast@yahoo.com> wrote:
I never said that Bitcoin message f= ield lengths should always be the same. But before this change they certain= ly were constant per protocol version. All I'm saying is that optional = lengths shouldn't be used (a field exists or not) and for every field c= hange, the protocol version should be upgraded.

Now that fRelayTxes is part of the protocol, the version number should = be upgraded to reflect this fact.

From: Mike Hearn <mike@plan99.net>
= To: Paul Lyon <pmlyon@hotmail.ca>
Cc: Turkey Breast <<= a href=3D"mailto:turkeybreast@yahoo.com" target=3D"_blank">turkeybreast@yah= oo.com>; "bitcoin-development@lists.sourceforge.net&quo= t; <bitcoin-development@lists.sourceforge.net>
Sent: Wednesday, June 19, 2= 013 3:20 PM

Subject: Re: [Bitcoin-development] Missing fRelayTxes in v= ersion message

If you want to criticise the Bitcoin protocol for sloppyness, the = variable length of some messages isn't where I'd start.

Note that ping has the same issue, its length has changed over time to= include the nonce.

If your parser can't handle that kind of thing, you= need to fix it. The protocol has always worked that way.



On Wed, Jun 19, 2013 at 3:03 PM, Paul Lyon <pmlyon@hotmai= l.ca> wrote:
I=E2=80=99m also running into this exact same issue with my parser, no= w I understand why the relay field behavior I was seeing=C2=A0doesn=E2=80= =99t match the wiki.
=C2=A0
So to parse a version messa= ge, you can=E2=80=99t rely on the protocol version? You have to know how lo= ng the payload is, and then parse the message accordingly? I agree with Tur= key Breast, this seems a bit sloppy to me.
=C2=A0
Paul
=C2=A0
P.S. I=E2=80=99ve nev= er used a dev mailing list before and I want to get involved with the Bitco= in dev community, so let me know if I=E2=80=99m horribly violating any=C2= =A0mailing list etiquette. =F0=9F=98=8A
=C2=A0
From:=C2=A0= Mike Hearn
Sent:=C2=A0=E2=80=8EWednesday=E2=80=8E, =E2=80=8EJune=E2=80=8E =E2= =80=8E19=E2=80=8E, =E2=80=8E2013 =E2=80=8E7=E2=80=8E:=E2=80=8E43=E2=80=8E = =E2=80=8EAM
To:=C2=A0Turkey Breast
Cc:=C2=A0bitcoin-development@lists.sourceforge.net
=C2=A0
Bitcoin-Qt on master does= send it now although it doesn't affect anything, but as old pre-filter= ing versions will continue to exist, you'll always have to be able to d= eserialize version messages without it.

Bitcoin version messages have always had variable length, lo= ok at how the code is written in main.cpp. If you didn't experience iss= ues until now all it means is that no sufficiently old nodes were talking t= o yours.

The standard does not say it should appear. Read it aga= in - BIP 37 says about the new version message field:
If= false then broadcast transactions will not be announced until a filter{loa= d,add,clear} command is received. If missing or true, no change in p= rotocol behaviour occurs.


On Wed, Jun 19, 201= 3 at 12:33 PM, Turkey Breast <turkeybreast@yahoo.com> wrote:
=
It's a problem if you work with iterators to deserialize the= byte stream. Even failing that, it's just sloppy programming. What hap= pens in the future when new fields are added to the version message? It'= ;s not a big deal to say that this protocol version has X number of fields,= that (higher) protocol version message has X + N number of fields. Determi= nistic number of fields per protocol version is sensical and how Bitcoin ha= s been for a long time.

<= div style=3D"font-family:times new roman,new york,times,serif;font-size:16p= x;font-style:normal;background-color:transparent"> And yes, it was a problem for me that caused a lot of confusion why this byte didn't exist in many vers= ion messages despite the standard saying it should and the code in bitcoind= indicating it should. Nowhere was this written. It doesn't help other = implementations to have an unclear behaviour that depends on some magic fro= m one implementation.

=
From: Mike Hearn <mike@plan99.net>
To: Turkey Breast <turkeybreast@yahoo.com>
Cc: "bitcoin= -development@lists.sourceforge.net" <bitcoin-development@= lists.sourceforge.net>
Sent: Wednesday, June 19, 2= 013 11:39 AM

Subject: Re: [Bitcoin-development] Missing fRelayTxes in version message

It has to be optional because old clients don't s= end it, obviously.

Why is this even an issue? There'= s no problem with variable length messages in any codebase that I'm awa= re of. Is this solving some actual problem?


On Wed, Jun 19, 2013 at 12:30 AM, Turkey Breast <turkeybreast@y= ahoo.com> wrote:
=
That's me. I never said to make all messages fixed length. I= said to make a fixed number of fields per protocol. So given a protocol ve= rsion number, you know the number of fields in a message. This is not only = easier for parsing messages, but just good practice. I don't see why a = 1 byte flag needs to be optional anyway.

From: Mike Hearn <mike@plan99.net>
To: Turkey Breast <turkeybreast@yahoo.com>
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net> Sent: Tuesday, June 18, 20= 13 9:48 PM
Subject: Re: [Bitcoin-devel= opment] Missing fRelayTxes in version message
=

It's not a bug (although there was recently a cha= nge to make bitcoind/qt always send this field anyway).=C2=A0

I don't know where Amir is going with BIP 60. Version messages ha= ve always been variable length. There's nothing inherent in the Bitcoin= protocol that says all messages are fixed length, indeed, tx messages are = allowed to have arbitrary data appended after them that gets relayed.


On Tue, Jun 18, 2013 at 7:45 PM, Turkey Breast <turkeybreast@ya= hoo.com> wrote:
=
See this BIP. I'm not sure if this is a bug or what, but it would = be good if messages always had a fixed number of fields per protocol versio= n.


This BIP details everything that needs to be done and proposes a protocol u= pgrade.

------------------------------------------= ------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/w= indows-dev2dev
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/li= stinfo/bitcoin-development





-----------= -------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/w= indows-dev2dev
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/li= stinfo/bitcoin-development





-----------= -------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/w= indows-dev2dev
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/li= stinfo/bitcoin-development





<= /div>
---------------------------------= ---------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.= sf.net/sfu/windows-dev2dev
_________________________________________= ______
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment


--047d7b450304ba8b0504df90a2f9--