Return-Path: <peter.tschipper@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 9865B8F5
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 17:09:09 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-pa0-f49.google.com (mail-pa0-f49.google.com
[209.85.220.49])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DDED918C
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 17:09:08 +0000 (UTC)
Received: by pacdm15 with SMTP id dm15so2288391pac.3
for <bitcoin-dev@lists.linuxfoundation.org>;
Tue, 10 Nov 2015 09:09:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=subject:to:references:from:message-id:date:user-agent:mime-version
:in-reply-to:content-type;
bh=zlFur4n/0B6eLomAxYY62X+7zd8ptgD3U5vyHe+LmjY=;
b=fONFnjQ4veTAgUvjwBZ7/OK4Y3Q/QjmPPCueE8ljcwSP0mL0UbN+MBAi27W8j2y2zL
NWIwhhEBJ23nYgQ5yg5RPaQxgjAZ1/7gn0W4DY7KFWAMV2dJJiMfPj7a4QB4+sx4Swgd
zDolZn2vPKtuvhJMh/K7DR4c32GENt2qIICEUFJXMwwDYK0hg8KsvUBGopVJYeMDDAG/
w7yvsYnc5tAGGtlqWjXM4UsSQ+4BRFKV/8p+MAnBzb4Tqj9NXdYmxC/kUj6d1imTICZT
NZlDCDOVILRHh+jT5wEoVL8kQH5CHXMXXmuU+EziYhICeEmY1FB2z7QWyVgmlcs3I64s
+cjQ==
X-Received: by 10.68.223.226 with SMTP id qx2mr7125087pbc.157.1447175348558;
Tue, 10 Nov 2015 09:09:08 -0800 (PST)
Received: from [192.168.0.132] (S0106bcd165303d84.cc.shawcable.net.
[96.54.102.88]) by smtp.googlemail.com with ESMTPSA id
zk3sm5105417pbb.41.2015.11.10.09.09.07
for <bitcoin-dev@lists.linuxfoundation.org>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Tue, 10 Nov 2015 09:09:07 -0800 (PST)
To: bitcoin-dev@lists.linuxfoundation.org
References: <5640F172.3010004@gmail.com> <20151109210449.GE5886@mcelrath.org>
<CAL7-sS0Apm4O_Qi0FmY7=H580rEVD6DYjk2y+ACpZmKqUJTQwA@mail.gmail.com>
<CALOxbZtTUrZwDfy_jTbs60n=K8RKDGg5X0gkLsh-OX3ikLf1FQ@mail.gmail.com>
<CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com>
<5642172C.701@gmail.com>
<CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com>
<CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com>
From: Peter Tschipper <peter.tschipper@gmail.com>
X-Enigmail-Draft-Status: N1110
Message-ID: <564224B2.9090903@gmail.com>
Date: Tue, 10 Nov 2015 09:09:06 -0800
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com>
Content-Type: multipart/alternative;
boundary="------------070208000204040609060805"
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW
autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream
Compression"
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2015 17:09:09 -0000
This is a multi-part message in MIME format.
--------------070208000204040609060805
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
On 10/11/2015 8:46 AM, Jeff Garzik via bitcoin-dev wrote:
> Comments:
>
> 1) cblock seems a reasonable way to extend the protocol. Further
> wrapping should probably be done at the stream level.
agreed.
>
> 2) zlib has crappy security track record.
>
Zlib had a bad buffer overflow bug but that was in 2005 and it got a lot
of press at the time. It's was fixed in version 1.2.3...we're on 1.2.8
now. I'm not aware of any other current issues with zlib. Do you have a
citation?
> 3) A fallback path to non-compressed is required, should compression
> fail or crash.
agreed.
>
> 4) Most blocks and transactions have runs of zeroes and/or highly
> common bit-patterns, which contributes to useful compression even at
> smaller sizes. Peter Ts's most recent numbers bear this out. zlib
> has a dictionary (32K?) which works well with repeated patterns such
> as those you see with concatenated runs of transactions.
>
> 5) LZO should provide much better compression, at a cost of CPU
> performance and using a less-reviewed, less-field-tested library.
I don't think LZO will give as good compression here but I will do some
benchmarking when I can.
>
>
>
>
>
> On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>
>
>
> On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper
> <peter.tschipper@gmail.com <mailto:peter.tschipper@gmail.com>> wrote:
>
> There are better ways of sending new blocks, that's certainly
> true but for sending historical blocks and seding transactions
> I don't think so. This PR is really designed to save
> bandwidth and not intended to be a huge performance
> improvement in terms of time spent sending.
>
>
> If the main point is for historical data, then sticking to just
> blocks is the best plan.
>
> Since small blocks don't compress well, you could define a
> "cblocks" message that handles multiple blocks (just concatenate
> the block messages as payload before compression).
>
> The sending peer could combine blocks so that each cblock is
> compressing at least 10kB of block data (or whatever is optimal).
> It is probably worth specifying a maximum size for network buffer
> reasons (either 1MB or 1 block maximum).
>
> Similarly, transactions could be combined together and compressed
> "ctxs". The inv messages could be modified so that you can
> request groups of 10-20 transactions. That would depend on how
> much of an improvement compressed transactions would represent.
>
> More generally, you could define a message which is a compressed
> message holder. That is probably to complex to be worth the
> effort though.
>
>
>
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via
>> bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>
>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>>
>>
>> I think 25% bandwidth savings is certainly
>> considerable, especially for people running full
>> nodes in countries like Australia where internet
>> bandwidth is lower and there are data caps.
>>
>>
>> This reinforces the idea that such trade-off decisions
>> should be be local and negotiated between peers, not a
>> required feature of the network P2P.
>>
>>
>> --
>> Johnathan Corgan
>> Corgan Labs - SDR Training and Development Services
>> http://corganlabs.com
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> <mailto:bitcoin-dev@lists.linuxfoundation.org>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--------------070208000204040609060805
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 10/11/2015 8:46 AM, Jeff Garzik via
bitcoin-dev wrote:<br>
</div>
<blockquote
cite="mid:CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com"
type="cite">
<div dir="ltr">Comments:
<div><br>
</div>
<div>1) cblock seems a reasonable way to extend the protocol.
Further wrapping should probably be done at the stream level.</div>
</div>
</blockquote>
agreed.<br>
<blockquote
cite="mid:CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>2) zlib has crappy security track record.</div>
<div><br>
</div>
</div>
</blockquote>
Zlib had a bad buffer overflow bug but that was in 2005 and it got a
lot of press at the time. It's was fixed in version 1.2.3...we're
on 1.2.8 now. I'm not aware of any other current issues with zlib.
Do you have a citation?<br>
<br>
<blockquote
cite="mid:CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>3) A fallback path to non-compressed is required, should
compression fail or crash.</div>
</div>
</blockquote>
agreed.<br>
<blockquote
cite="mid:CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>4) Most blocks and transactions have runs of zeroes and/or
highly common bit-patterns, which contributes to useful
compression even at smaller sizes. Peter Ts's most recent
numbers bear this out. zlib has a dictionary (32K?) which
works well with repeated patterns such as those you see with
concatenated runs of transactions.</div>
<div><br>
</div>
<div>5) LZO should provide much better compression, at a cost of
CPU performance and using a less-reviewed, less-field-tested
library.</div>
</div>
</blockquote>
I don't think LZO will give as good compression here but I will do
some benchmarking when I can.<br>
<br>
<blockquote
cite="mid:CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10, 2015 at 11:30 AM, Tier
Nolan via bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span class="">On Tue, Nov 10,
2015 at 4:11 PM, Peter Tschipper <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:peter.tschipper@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:peter.tschipper@gmail.com">peter.tschipper@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<span></span><span></span>There are better ways of
sending new blocks, that's certainly true but for
sending historical blocks and seding transactions
I don't think so. This PR is really designed to
save bandwidth and not intended to be a huge
performance improvement in terms of time spent
sending.<span><br>
</span></blockquote>
<div><br>
</div>
</span>
<div>If the main point is for historical data, then
sticking to just blocks is the best plan.<br>
<br>
</div>
<div>Since small blocks don't compress well, you could
define a "cblocks" message that handles multiple
blocks (just concatenate the block messages as
payload before compression). <br>
<br>
The sending peer could combine blocks so that each
cblock is compressing at least 10kB of block data
(or whatever is optimal). It is probably worth
specifying a maximum size for network buffer reasons
(either 1MB or 1 block maximum).<br>
<br>
</div>
<div>Similarly, transactions could be combined
together and compressed "ctxs". The inv messages
could be modified so that you can request groups of
10-20 transactions. That would depend on how much
of an improvement compressed transactions would
represent. <br>
<br>
</div>
<div>More generally, you could define a message which
is a compressed message holder. That is probably to
complex to be worth the effort though.<br>
</div>
<span class="">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10,
2015 at 5:40 AM, Johnathan Corgan via
bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr"><span>
<div style="font-size:small">On
Mon, Nov 9, 2015 at 5:58 PM,
gladoscc via bitcoin-dev <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
</div>
</span>
<div class="gmail_extra">
<div class="gmail_quote"><span>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">I think 25%
bandwidth savings is
certainly considerable,
especially for people
running full nodes in
countries like Australia
where internet bandwidth
is lower and there are
data caps.</div>
</blockquote>
<div><br>
</div>
</span>
<div>
<div
style="font-size:small;display:inline">
This reinforces the idea
that such trade-off
decisions should be be local
and negotiated between
peers, not a required
feature of the network P2P.</div>
</div>
</div>
<span>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Johnathan
Corgan<br>
Corgan Labs -
SDR Training
and
Development
Services</div>
<div><a
moz-do-not-send="true"
href="http://corganlabs.com" style="font-size:12.8px" target="_blank"><a class="moz-txt-link-freetext" href="http://corganlabs.com">http://corganlabs.com</a></a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</span></div>
</blockquote>
</span></div>
<br>
</div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
bitcoin-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>
<a class="moz-txt-link-freetext" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>
--------------070208000204040609060805--