summaryrefslogtreecommitdiff
path: root/22/266b42ef70befdb547e8b8b006f3821223aed4
blob: 8e39d9eb0b9f2cb4d2d1ad02f972cab34c387d79 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
	helo=mx.sourceforge.net)
	by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <pieter.wuille@gmail.com>) id 1UpcU6-0000WH-1t
	for bitcoin-development@lists.sourceforge.net;
	Thu, 20 Jun 2013 10:52:58 +0000
Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.223.173 as permitted sender)
	client-ip=209.85.223.173; envelope-from=pieter.wuille@gmail.com;
	helo=mail-ie0-f173.google.com; 
Received: from mail-ie0-f173.google.com ([209.85.223.173])
	by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1UpcU4-0002uv-KH
	for bitcoin-development@lists.sourceforge.net;
	Thu, 20 Jun 2013 10:52:58 +0000
Received: by mail-ie0-f173.google.com with SMTP id k13so16094116iea.32
	for <bitcoin-development@lists.sourceforge.net>;
	Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.42.47.69 with SMTP id n5mr2975623icf.24.1371725571261; Thu,
	20 Jun 2013 03:52:51 -0700 (PDT)
Received: by 10.50.149.233 with HTTP; Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
Received: by 10.50.149.233 with HTTP; Thu, 20 Jun 2013 03:52:51 -0700 (PDT)
In-Reply-To: <1371724625.50978.YahooMailNeo@web162706.mail.bf1.yahoo.com>
References: <4DE0E45E-BB48-4DFF-9C86-ACBE312B3049@bitsofproof.com>
	<CANEZrP1MVKSNU92UrQ36sPCN0LNLKrVRXnfVwEZKhgoBFXatBA@mail.gmail.com>
	<20130620090649.GA17765@vps7135.xlshosting.net>
	<CANEZrP2cLR9sYZ59O-S6HNi9bKPtaY5yOkvVx+O8XAC792acZg@mail.gmail.com>
	<1371724625.50978.YahooMailNeo@web162706.mail.bf1.yahoo.com>
Date: Thu, 20 Jun 2013 12:52:51 +0200
Message-ID: <CAPg+sBiJvWckJYa-iw=qtjfj1PoY_bKhFD3sx6LqfUrFbWrGHA@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Turkey Breast <turkeybreast@yahoo.com>
Content-Type: multipart/alternative; boundary=90e6ba6149c667c07804df93bf35
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(pieter.wuille[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UpcU4-0002uv-KH
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Missing fRelayTxes in version
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 20 Jun 2013 10:52:58 -0000

--90e6ba6149c667c07804df93bf35
Content-Type: text/plain; charset=ISO-8859-1

Let's just increase the version number and be done with this discussion.
It's a small benefit, but it simplifies things and it's trivial to do.

I don't understand how a policy of requiring version increases could limit
future extensions: after the version/verack exchange, the protocol version
is negotiated between peers, and there is no need for anything optional
anymore.

Note thay this is just about parsing, not about relaying - you should still
relay parts of a message you haven't parsed. But that doesn't apply to the
version message anyway, which is the only place where this matters.

-- 
Pieter
 On 20 Jun 2013 12:38, "Turkey Breast" <turkeybreast@yahoo.com> wrote:

> I don't get why this is such a contentious change?
>
> Before I was able to use asserts to check the expected length of length of
> messages per protocol version, I could pass in dumb iterators that just
> parse the byte stream and I could serialize and deserialize a message to
> check the parser is correct (in debug mode).
>
> This 'simple' change causes all that behaviour to be lost. You can no
> longer just use iterators but must know the remaining length (or if you use
> std::distance, you can only use specific std containers - not just anything
> with an iterator and an operator++). You cannot check the deserialization
> process by serializing the deserialized message and comparing it to the
> original data (because the bool is always present in the serializer).
>
> It's a bit stupid you call it buggy code when this behaviour has never
> been present in Bitcoin. The BIP doesn't introduce any unwanted
> side-effects and is a trivial reasonable change.
>
> If you want optional fields then the proper way to do it, is to either set
> a flag in the Services field of the "version" message to indicate different
> formats for messages (i.e use this template structure for a message, not
> that one), introduce a new message (if the changes are big), to
> approve/improve Stefan's BIP 32 for custom services or to have a value in
> the byte stream indicating which fields are present (maybe a bitfield or
> so).
>
> Using a quirk of an implementation is just bad form and sloppy coding.
> Optional fields should have their own mechanism that allows them to remain
> as optional fields between protocol version upgrades.
>
> The bitcoind software can probably be improved too, by checking that the
> length of the version message is consistent for the protocol version given
> by the connected node. Right now it makes no assumptions based on that
> which is a mistake (new clients can broadcast older version messages that
> don't have all the fields required). Probably the software should penalise
> hosts which do that.
>
> What's the big deal to update the protocol version number from 70001 to
> 70002? It's not like we'll run out of integers. The field has now gone from
> optional to required now anyway - that's a behaviour change. It'd be good
> to enforce that. I see this as a bug.
>
>   ------------------------------
>  *From:* Mike Hearn <mike@plan99.net>
> *To:* Pieter Wuille <pieter.wuille@gmail.com>
> *Cc:* Bitcoin Dev <bitcoin-development@lists.sourceforge.net>; Tamas
> Blummer <tamas@bitsofproof.com>
> *Sent:* Thursday, June 20, 2013 11:17 AM
> *Subject:* Re: [Bitcoin-development] Missing fRelayTxes in version
>
> There's no problem, but there's no benefit either. It also locks us in to
> a potentially problematic guarantee - what if in future we want to have,
> say, two optional new pieces of data in two different messages. We don't
> want to require that if version > X then you have to implement all features
> up to and including that point.
>
> Essentially the number of fields in a message is like a little version
> number, just for that message. It adds flexibility to keep it that way, and
> there's no downside, seeing as that bridge was already crossed and people
> with parsers that can't handle it need to fix their code anyway.
>
> So I have a slight preference for keeping things the way they are, it
> keeps things flexible for future and costs nothing.
>
>
>
> On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille <pieter.wuille@gmail.com>wrote:
>
> On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:
> > Sure but why not do that when there's an actual new field to add? Does
> > anyone have a proposal for a feature that needs a new version field at
> the
> > moment? There's no point changing the protocol now unless there's
> actually
> > a new field to add.
> >
> > Anyway I still don't see why anyone cares about this issue. The Bitcoin
> > protocol does not and never has required that all messages have a fixed
> > number of fields per version. Any parser written on the assumption it did
> > was just buggy. Look at how tx messages are relayed for the most obvious
> > example of that pattern in action - it's actually the raw byte stream
> > that's stored and relayed to ensure that fields added in new versions
> > aren't dropped during round-tripping. Old versions are supposed to
> preserve
> > fields from the future.
>
> Actually, that is not the same issue. What is being argued for here is that
> the version in the version message itself should indicate which fields are
> present, so a parser doesn't need to look at the length of the message.
> That
> seems like a minor but very reasonable request to me, and it's trivial to
> do.
> That doesn't mean that you may receive versions higher than what you know
> of,
> and thus messages with fields you don't know about. That doesn't matter,
> you
> can just ignore them.
>
> I see no problem with raising the protocol version number to indicate
> "all fields up to fRelayTxes are required, if the announced nVersion is
> above N".
> In fact, I believe (though haven't checked) all previous additions to the
> version
> message were accompanied with a protocol version (then: client version)
> increase
> as well.
>
> --
> Pieter
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

--90e6ba6149c667c07804df93bf35
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Let&#39;s just increase the version number and be done with =
this discussion. It&#39;s a small benefit, but it simplifies things and it&=
#39;s trivial to do.</p>
<p dir=3D"ltr">I don&#39;t understand how a policy of requiring version inc=
reases could limit future extensions: after the version/verack exchange, th=
e protocol version is negotiated between peers, and there is no need for an=
ything optional anymore.</p>

<p dir=3D"ltr">Note thay this is just about parsing, not about relaying - y=
ou should still relay parts of a message you haven&#39;t parsed. But that d=
oesn&#39;t apply to the version message anyway, which is the only place whe=
re this matters.</p>

<p dir=3D"ltr">-- <br>
Pieter<br>
</p>
<div class=3D"gmail_quote">On 20 Jun 2013 12:38, &quot;Turkey Breast&quot; =
&lt;<a href=3D"mailto:turkeybreast@yahoo.com">turkeybreast@yahoo.com</a>&gt=
; wrote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style=3D"font-size:12pt;font-family:times new roman,new york,time=
s,serif"><div><span>I don&#39;t get why this is such a contentious change?<=
/span></div><div style=3D"font-style:normal;font-size:16px;background-color=
:transparent;font-family:times new roman,new york,times,serif">
<br><span></span></div><div style=3D"font-style:normal;font-size:16px;backg=
round-color:transparent;font-family:times new roman,new york,times,serif"><=
span>Before I was able to use asserts to check the expected length of lengt=
h of messages per protocol version, I could pass in dumb iterators that jus=
t parse the byte stream and I could serialize and deserialize a message to =
check the parser is correct (in debug mode).</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>This &#39;simple&#39; change causes all that behaviour to be lost. Yo=
u can no longer just use iterators but must know the remaining length (or i=
f you use std::distance, you can only use specific std containers - not jus=
t anything with an iterator and an operator++). You cannot check the deseri=
alization process by serializing the deserialized message and comparing it =
to the original data (because the bool is always present in the serializer)=
.</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>It&#39;s a bit stupid you call it buggy code when this behaviour has =
never been present in Bitcoin. The BIP doesn&#39;t introduce any unwanted s=
ide-effects and is a trivial reasonable change.</span></div><div style=3D"f=
ont-style:normal;font-size:16px;background-color:transparent;font-family:ti=
mes new roman,new york,times,serif">
<br><span></span></div><div style=3D"font-style:normal;font-size:16px;backg=
round-color:transparent;font-family:times new roman,new york,times,serif"><=
span>If you want optional fields then the proper way to do it, is to either=
 set a flag in the Services field of the &quot;version&quot; message to ind=
icate different formats for messages (i.e use this template structure for a=
 message, not that one), introduce a new message (if the changes are big), =
to approve/improve Stefan&#39;s BIP 32 for custom services or to have a val=
ue in the byte stream indicating which fields are present (maybe
 a bitfield or so).</span></div><div style=3D"font-style:normal;font-size:1=
6px;background-color:transparent;font-family:times new roman,new york,times=
,serif"><br><span></span></div><div style=3D"font-style:normal;font-size:16=
px;background-color:transparent;font-family:times new roman,new york,times,=
serif">
<span>Using a quirk of an implementation is just bad form and sloppy coding=
. Optional fields should have their own mechanism that allows them to remai=
n as optional fields between protocol version upgrades.<br></span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>The bitcoind software can probably be improved
 too, by checking that the length of the version message is consistent for =
the protocol version given by the connected node. Right now it makes no ass=
umptions based on that which is a mistake (new clients can broadcast older =
version messages that don&#39;t have all the fields required). Probably the=
 software should penalise hosts which do that.</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:times new roman,new york,times,serif"><br><span></span></div><=
div style=3D"font-style:normal;font-size:16px;background-color:transparent;=
font-family:times new roman,new york,times,serif">
<span>What&#39;s the big deal to update the protocol version number from 70=
001 to 70002? It&#39;s not like we&#39;ll run out of integers. The field ha=
s now gone from optional to required now anyway - that&#39;s a behaviour ch=
ange. It&#39;d be good to enforce that. I see this as a
 bug.<br></span></div><div><br></div>  <div style=3D"font-family:times new =
roman,new york,times,serif;font-size:12pt"> <div style=3D"font-family:times=
 new roman,new york,times,serif;font-size:12pt"> <div dir=3D"ltr"> <hr size=
=3D"1">
  <font face=3D"Arial"> <b><span style=3D"font-weight:bold">From:</span></b=
> Mike Hearn &lt;<a href=3D"mailto:mike@plan99.net" target=3D"_blank">mike@=
plan99.net</a>&gt;<br> <b><span style=3D"font-weight:bold">To:</span></b> P=
ieter Wuille &lt;<a href=3D"mailto:pieter.wuille@gmail.com" target=3D"_blan=
k">pieter.wuille@gmail.com</a>&gt; <br>
<b><span style=3D"font-weight:bold">Cc:</span></b> Bitcoin Dev &lt;<a href=
=3D"mailto:bitcoin-development@lists.sourceforge.net" target=3D"_blank">bit=
coin-development@lists.sourceforge.net</a>&gt;; Tamas Blummer &lt;<a href=
=3D"mailto:tamas@bitsofproof.com" target=3D"_blank">tamas@bitsofproof.com</=
a>&gt; <br>
 <b><span style=3D"font-weight:bold">Sent:</span></b> Thursday, June 20, 20=
13 11:17 AM<br> <b><span style=3D"font-weight:bold">Subject:</span></b> Re:=
 [Bitcoin-development] Missing fRelayTxes in version<br> </font> </div> <di=
v>
<br>
<div><div dir=3D"ltr">There&#39;s no problem, but there&#39;s no benefit ei=
ther. It also locks us in to a potentially problematic guarantee - what if =
in future we want to have, say, two optional new pieces of data in two diff=
erent messages. We don&#39;t want to require that if version &gt; X then yo=
u have to implement all features up to and including that point.<div>

<br></div><div>Essentially the number of fields in a message is like a litt=
le version number, just for that message. It adds flexibility to keep it th=
at way, and there&#39;s no downside, seeing as that bridge was already cros=
sed and people with parsers that can&#39;t handle it need to fix their code=
 anyway.</div>

<div><br></div><div>So I have a slight preference for keeping things the wa=
y they are, it keeps things flexible for future and costs nothing.</div><di=
v><br></div></div><div><br><br><div>
On Thu, Jun 20, 2013 at 11:06 AM, Pieter Wuille <span dir=3D"ltr">&lt;<a re=
l=3D"nofollow" href=3D"mailto:pieter.wuille@gmail.com" target=3D"_blank">pi=
eter.wuille@gmail.com</a>&gt;</span> wrote:<br><blockquote style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div>On Thu, Jun 20, 2013 at 09:36:40AM +0200, Mike Hearn wrote:<br>
&gt; Sure but why not do that when there&#39;s an actual new field to add? =
Does<br>
&gt; anyone have a proposal for a feature that needs a new version field at=
 the<br>
&gt; moment? There&#39;s no point changing the protocol now unless there&#3=
9;s actually<br>
&gt; a new field to add.<br>
&gt;<br>
&gt; Anyway I still don&#39;t see why anyone cares about this issue. The Bi=
tcoin<br>
&gt; protocol does not and never has required that all messages have a fixe=
d<br>
&gt; number of fields per version. Any parser written on the assumption it =
did<br>
&gt; was just buggy. Look at how tx messages are relayed for the most obvio=
us<br>
&gt; example of that pattern in action - it&#39;s actually the raw byte str=
eam<br>
&gt; that&#39;s stored and relayed to ensure that fields added in new versi=
ons<br>
&gt; aren&#39;t dropped during round-tripping. Old versions are supposed to=
 preserve<br>
&gt; fields from the future.<br>
<br>
</div>Actually, that is not the same issue. What is being argued for here i=
s that<br>
the version in the version message itself should indicate which fields are<=
br>
present, so a parser doesn&#39;t need to look at the length of the message.=
 That<br>
seems like a minor but very reasonable request to me, and it&#39;s trivial =
to do.<br>
That doesn&#39;t mean that you may receive versions higher than what you kn=
ow of,<br>
and thus messages with fields you don&#39;t know about. That doesn&#39;t ma=
tter, you<br>
can just ignore them.<br>
<br>
I see no problem with raising the protocol version number to indicate<br>
&quot;all fields up to fRelayTxes are required, if the announced nVersion i=
s above N&quot;.<br>
In fact, I believe (though haven&#39;t checked) all previous additions to t=
he version<br>
message were accompanied with a protocol version (then: client version) inc=
rease<br>
as well.<br>
<span><font color=3D"#888888"><br>
--<br>
Pieter<br>
<br>
</font></span></blockquote></div><br></div>
</div><br>-----------------------------------------------------------------=
-------------<br>This SF.net email is sponsored by Windows:<br><br>Build fo=
r Windows Store.<br><br><a href=3D"http://p.sf.net/sfu/windows-dev2dev" tar=
get=3D"_blank">http://p.sf.net/sfu/windows-dev2dev</a><br>
_______________________________________________<br>Bitcoin-development mail=
ing list<br><a href=3D"mailto:Bitcoin-development@lists.sourceforge.net" ta=
rget=3D"_blank">Bitcoin-development@lists.sourceforge.net</a><br><a href=3D=
"https://lists.sourceforge.net/lists/listinfo/bitcoin-development" target=
=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-developmen=
t</a><br>
<br><br></div> </div> </div>  </div></div><br>-----------------------------=
-------------------------------------------------<br>
This SF.net email is sponsored by Windows:<br>
<br>
Build for Windows Store.<br>
<br>
<a href=3D"http://p.sf.net/sfu/windows-dev2dev" target=3D"_blank">http://p.=
sf.net/sfu/windows-dev2dev</a><br>_________________________________________=
______<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
<br></blockquote></div>

--90e6ba6149c667c07804df93bf35--