1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
|
Return-Path: <peter.tschipper@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id A48968F5
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 16:17:43 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com
[209.85.220.47])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id A9819150
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 16:17:42 +0000 (UTC)
Received: by pabfh17 with SMTP id fh17so1218324pab.0
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 08:17:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=subject:to:references:from:message-id:date:user-agent:mime-version
:in-reply-to:content-type;
bh=qbWD9KlDptARj3tea74YXm2kTeocPta1CNfq0UMGnLQ=;
b=f2aXIKN7tqMbOV5UnIm+bXAJyQdaQbdigJMX05YdRTZ47xPxdqf9ZrgAqx3y9I1A55
olLBXpsdC1IlY0bqnGuvFTr9mK78EWWhF0D+u6EVIC/XwwJIp4mD3ae+IW48sZV6L9sk
RwtXwgcz4kEQKxSyFAtT9kG9Jp9mfRyewNRYw+QhZohQ7C7xxh1101VcqEkYVnVJW43E
mSICqoFZl5mzl728sDBmQqRGGGbABKlyHO/MQfoCyy3zq+c1G0amS8Z+K+hfQdc5VKz3
KguLeBXgb5N2goV8F3XuECAno27yySApEuo6hlUDS5p9LPhKqUS4SaIxQat2N46RkksZ
ZT8w==
X-Received: by 10.68.192.8 with SMTP id hc8mr6716276pbc.117.1447172262206;
Tue, 10 Nov 2015 08:17:42 -0800 (PST)
Received: from [192.168.0.132] (S0106bcd165303d84.cc.shawcable.net.
[96.54.102.88]) by smtp.googlemail.com with ESMTPSA id
c5sm4924306pbu.18.2015.11.10.08.17.41
for <bitcoin-dev@lists.linuxfoundation.org>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Tue, 10 Nov 2015 08:17:41 -0800 (PST)
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
References: <5640F172.3010004@gmail.com> <20151109210449.GE5886@mcelrath.org>
<CAL7-sS0Apm4O_Qi0FmY7=H580rEVD6DYjk2y+ACpZmKqUJTQwA@mail.gmail.com>
<CALOxbZtTUrZwDfy_jTbs60n=K8RKDGg5X0gkLsh-OX3ikLf1FQ@mail.gmail.com>
<CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com>
<5642172C.701@gmail.com>
From: Peter Tschipper <peter.tschipper@gmail.com>
Message-ID: <564218A4.8070102@gmail.com>
Date: Tue, 10 Nov 2015 08:17:40 -0800
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <5642172C.701@gmail.com>
Content-Type: multipart/alternative;
boundary="------------030702010507070103060804"
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW
autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream
Compression"
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2015 16:17:43 -0000
This is a multi-part message in MIME format.
--------------030702010507070103060804
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
On 10/11/2015 8:11 AM, Peter Tschipper wrote:
> On 10/11/2015 1:44 AM, Tier Nolan via bitcoin-dev wrote:
>> The network protocol is not quite consensus critical, but it is
>> important.
>>
>> Two implementations of the decompressor might not be bug for bug
>> compatible. This (potentially) means that a block could be designed
>> that won't decode properly for some version of the client but would
>> work for another. This would fork the network.
>>
>> A "raw" network library is unlikely to have the same problem.
>>
>> Rather than just compress the stream, you could compress only block
>> messages only. A new "cblock" message could be created that is a
>> compressed block. This shouldn't reduce efficiency by much.
>>
> I chose the more generic datastream compression so we could in the
> future apply to possibly to transactions but currently all that is
> planned, is to compress blocks, and that was really my only original
> intent until I saw that there might be some bandwidth savings for
> transactions as well.
>
> The compression however could be applied to any datastream but is not
> *forced* . Basically it would just be a method call in CDatastream so
> we could do ss.compress and ss.decompress and apply that to blocks and
> possibly transactions if worthwhile and only IF compression is turned
> on. But there is no intend to apply this to every type of message
> since most would be too small to benefit from compression.
>
> Here are some results of using the code in the PR to
> compress/decompress blocks using zlib compression level = 6. This
> data was taken from the first 275K blocks in the mainnet blockchain.
> Clearly once we get past 10KB we get pretty decent compression but
> even below that there is some benefit. I'm still collecting data and
> will get the same for the whole blockchain.
>
> range = block size range
> ubytes = average size of uncompressed blocks
> cbytes = average size of compressed blocks
> ctime = average time to compress
> dtime = average time to decompress
> cmp_ratio% = compression ratio
> datapoints = number of datapoints taken
>
> range ubytes cbytes ctime dtime cmp_ratio% datapoints
> 0-250b 215 189 0.001 0.000 12.41 79498
> 250-500b 440 405 0.001 0.000 7.82 11903
> 500-1KB 762 702 0.001 0.000 7.83 10448
> 1KB-10KB 4166 3561 0.001 0.000 14.51 50572
> 10KB-100KB 40820 31597 0.005 0.001 22.59 75555
> 100KB-200KB 146238 106320 0.015 0.001 27.30 25024
> 200KB-300KB 242913 175482 0.025 0.002 27.76 20450
> 300KB-400KB 343430 251760 0.034 0.003 26.69 2069
> 400KB-500KB 457448 343495 0.045 0.004 24.91 1889
> 500KB-600KB 540736 424255 0.056 0.007 21.54 90
> 600KB-700KB 647851 506888 0.063 0.007 21.76 59
> 700KB-800KB 749513 586551 0.073 0.007 21.74 48
> 800KB-900KB 859439 652166 0.086 0.008 24.12 39
> 900KB-1MB 952333 725191 0.089 0.009 23.85 78
>
>> If a client fails to decode a cblock, then it can ask for the block
>> to be re-sent as a standard "block" message.
> interesting idea.
>>
>> This means that it is a pure performance improvement. If problems
>> occur, then the client can just switch back to uncompressed mode for
>> that block.
>>
>> You should look into the block relay system. This gives a larger
>> improvement than simply compressing the stream. The main benefit is
>> latency but it means that actual blocks don't have to be sent, so
>> gives a potential 50% compression ratio. Normally, a node receives
>> all the transactions and then those transactions are included later
>> in the block.
>>
> There are better ways of sending new blocks, that's certainly true but
> for sending historical blocks and seding transactions I don't think
> so. This PR is really designed to save bandwidth and not intended to
> be a huge performance improvement in terms of time spent sending.
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>
>>
>> I think 25% bandwidth savings is certainly considerable,
>> especially for people running full nodes in countries like
>> Australia where internet bandwidth is lower and there are
>> data caps.
>>
>>
>> This reinforces the idea that such trade-off decisions should be
>> be local and negotiated between peers, not a required feature of
>> the network P2P.
>>
>>
>> --
>> Johnathan Corgan
>> Corgan Labs - SDR Training and Development Services
>> http://corganlabs.com
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
--------------030702010507070103060804
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 10/11/2015 8:11 AM, Peter Tschipper
wrote:<br>
</div>
<blockquote cite="mid:5642172C.701@gmail.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<div class="moz-cite-prefix">On 10/11/2015 1:44 AM, Tier Nolan via
bitcoin-dev wrote:<br>
</div>
<blockquote
cite="mid:CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>The network protocol is not quite consensus
critical, but it is important.<br>
<br>
</div>
Two implementations of the decompressor might not be
bug for bug compatible. This (potentially) means that
a block could be designed that won't decode properly
for some version of the client but would work for
another. This would fork the network.<br>
<br>
</div>
<div>A "raw" network library is unlikely to have the
same problem.<br>
</div>
<div><br>
</div>
Rather than just compress the stream, you could compress
only block messages only. A new "cblock" message could
be created that is a compressed block. This shouldn't
reduce efficiency by much.<br>
<br>
</div>
</div>
</div>
</div>
</blockquote>
I chose the more generic datastream compression so we could in the
future apply to possibly to transactions but currently all that is
planned, is to compress blocks, and that was really my only
original intent until I saw that there might be some bandwidth
savings for transactions as well. <br>
<br>
The compression however could be applied to any datastream but is
not *forced* . Basically it would just be a method call in
CDatastream so we could do ss.compress and ss.decompress and apply
that to blocks and possibly transactions if worthwhile and only IF
compression is turned on. But there is no intend to apply this to
every type of message since most would be too small to benefit
from compression.<br>
<br>
Here are some results of using the code in the PR to
compress/decompress blocks using zlib compression level = 6. This
data was taken from the first 275K blocks in the mainnet
blockchain. Clearly once we get past 10KB we get pretty decent
compression but even below that there is some benefit. I'm still
collecting data and will get the same for the whole blockchain.<br>
<br>
range = block size range<br>
ubytes = average size of uncompressed blocks<br>
cbytes = average size of compressed blocks<br>
ctime = average time to compress<br>
dtime = average time to decompress<br>
cmp_ratio% = compression ratio<br>
datapoints = number of datapoints taken<br>
<br>
range ubytes cbytes ctime dtime cmp_ratio%
datapoints<br>
0-250b 215 189 0.001 0.000 12.41
79498<br>
250-500b 440 405 0.001 0.000 7.82
11903<br>
500-1KB 762 702 0.001 0.000 7.83
10448<br>
1KB-10KB 4166 3561 0.001 0.000 14.51
50572<br>
10KB-100KB 40820 31597 0.005 0.001 22.59
75555<br>
100KB-200KB 146238 106320 0.015 0.001 27.30
25024<br>
200KB-300KB 242913 175482 0.025 0.002 27.76
20450<br>
300KB-400KB 343430 251760 0.034 0.003 26.69
2069<br>
400KB-500KB 457448 343495 0.045 0.004 24.91
1889<br>
500KB-600KB 540736 424255 0.056 0.007 21.54
90<br>
600KB-700KB 647851 506888 0.063 0.007 21.76
59<br>
700KB-800KB 749513 586551 0.073 0.007 21.74
48<br>
800KB-900KB 859439 652166 0.086 0.008 24.12
39<br>
900KB-1MB 952333 725191 0.089 0.009 23.85
78<br>
<br>
<blockquote
cite="mid:CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>If a client fails to decode a cblock, then it can ask
for the block to be re-sent as a standard "block"
message. <br>
</div>
</div>
</div>
</blockquote>
interesting idea.<br>
<blockquote
cite="mid:CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
This means that it is a pure performance improvement. If
problems occur, then the client can just switch back to
uncompressed mode for that block.<br>
<br>
</div>
You should look into the block relay system. This gives a
larger improvement than simply compressing the stream. The
main benefit is latency but it means that actual blocks don't
have to be sent, so gives a potential 50% compression ratio.
Normally, a node receives all the transactions and then those
transactions are included later in the block.<br>
<div>
<div><br>
</div>
</div>
</div>
</blockquote>
There are better ways of sending new blocks, that's certainly true
but for sending historical blocks and seding transactions I don't
think so. This PR is really designed to save bandwidth and not
intended to be a huge performance improvement in terms of time
spent sending.<br>
<blockquote
cite="mid:CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com"
type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10, 2015 at 5:40 AM,
Johnathan Corgan via bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><span class="">
<div class="gmail_default" style="font-size:small">On
Mon, Nov 9, 2015 at 5:58 PM, gladoscc via
bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
</div>
</span>
<div class="gmail_extra">
<div class="gmail_quote"><span class="">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">I think 25% bandwidth savings is
certainly considerable, especially for people
running full nodes in countries like Australia
where internet bandwidth is lower and there
are data caps.</div>
</blockquote>
<div><br>
</div>
</span>
<div>
<div class="gmail_default"
style="font-size:small;display:inline">This
reinforces the idea that such trade-off
decisions should be be local and negotiated
between peers, not a required feature of the
network P2P.</div>
</div>
</div>
<span class="">
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Johnathan Corgan<br>
Corgan Labs - SDR Training and
Development Services</div>
<div><a moz-do-not-send="true"
href="http://corganlabs.com"
style="font-size:12.8px"
target="_blank">http://corganlabs.com</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
--------------030702010507070103060804--
|
