path: root/5b/1dc6078e268d550817e791d859c38a4958cfe6

Return-Path: <jgarzik@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 16BA5941
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 10 Nov 2015 16:46:18 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C958B1A7
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 10 Nov 2015 16:46:16 +0000 (UTC)
Received: by wmww144 with SMTP id w144so125925226wmw.1
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 10 Nov 2015 08:46:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	bh=inA/tJCHoldemqC0wCF+7yIcm48vnaGcNX0IZ/GPa6I=;
	b=imcq8Enqj++b/dcoMPQJVYsVNJBEUSOnR8DMlG+JS/cHlt2dofUhRlJ9eaPdpLuYk1
	B8q7ZPUkkF13GeMdyGRsKbi4vnX+OkwY8LyqjxQPZ84UA4ZP83QAhanZbLCqozNU55Go
	vQIHbnCz07CdwHxGk58m9CQOF+hivg4mWvdRKmOAIlNIB40KuSP19B8nmXTODxqiO1rl
	g8AyZUeFjvWC2ZgTJob5ovwnu8Dg+DsgX/cJaMPFTSfzYO2YCXTmxr9M5+aBTFLKKttg
	wZVZ9Ek9904XjwueBSVpchJbsa2E1ivVQYFbWJOdDuTRUDfNIzTvvTEWrbMxCQ24SSAw
	CYiQ==
MIME-Version: 1.0
X-Received: by 10.28.215.205 with SMTP id o196mr6136561wmg.95.1447173975557;
	Tue, 10 Nov 2015 08:46:15 -0800 (PST)
Received: by 10.28.212.79 with HTTP; Tue, 10 Nov 2015 08:46:15 -0800 (PST)
In-Reply-To: <CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com>
References: <5640F172.3010004@gmail.com> <20151109210449.GE5886@mcelrath.org>
	<CAL7-sS0Apm4O_Qi0FmY7=H580rEVD6DYjk2y+ACpZmKqUJTQwA@mail.gmail.com>
	<CALOxbZtTUrZwDfy_jTbs60n=K8RKDGg5X0gkLsh-OX3ikLf1FQ@mail.gmail.com>
	<CAE-z3OUB-se_HUvW2NLjWt=0d5sgMiPEciu0hLzr_HQN0m9fqQ@mail.gmail.com>
	<5642172C.701@gmail.com>
	<CAE-z3OXgWCHL_3CDR-ACc7ojbLi7EavyObNa3s7hPUMGj_V2+A@mail.gmail.com>
Date: Tue, 10 Nov 2015 11:46:15 -0500
Message-ID: <CADm_WcYAj9_r6tu8Be-U81LDwWvnv04PZJMmc-S4cY7+jxfzGw@mail.gmail.com>
From: Jeff Garzik <jgarzik@gmail.com>
To: Tier Nolan <tier.nolan@gmail.com>
Content-Type: multipart/alternative; boundary=001a11467f5cbdcd88052432722c
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE autolearn=ham
	version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream
	Compression"
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2015 16:46:18 -0000

--001a11467f5cbdcd88052432722c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Comments:

1) cblock seems a reasonable way to extend the protocol.  Further wrapping
should probably be done at the stream level.

2) zlib has crappy security track record.

3) A fallback path to non-compressed is required, should compression fail
or crash.

4) Most blocks and transactions have runs of zeroes and/or highly common
bit-patterns, which contributes to useful compression even at smaller
sizes.  Peter Ts's most recent numbers bear this out.  zlib has a
dictionary (32K?) which works well with repeated patterns such as those you
see with concatenated runs of transactions.

5) LZO should provide much better compression, at a cost of CPU performance
and using a less-reviewed, less-field-tested library.





On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
>
> On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper <
> peter.tschipper@gmail.com> wrote:
>
>> There are better ways of sending new blocks, that's certainly true but
>> for sending historical blocks and seding transactions I don't think so.
>> This PR is really designed to save bandwidth and not intended to be a hu=
ge
>> performance improvement in terms of time spent sending.
>>
>
> If the main point is for historical data, then sticking to just blocks is
> the best plan.
>
> Since small blocks don't compress well, you could define a "cblocks"
> message that handles multiple blocks (just concatenate the block messages
> as payload before compression).
>
> The sending peer could combine blocks so that each cblock is compressing
> at least 10kB of block data (or whatever is optimal).  It is probably wor=
th
> specifying a maximum size for network buffer reasons (either 1MB or 1 blo=
ck
> maximum).
>
> Similarly, transactions could be combined together and compressed "ctxs".
> The inv messages could be modified so that you can request groups of 10-2=
0
> transactions.  That would depend on how much of an improvement compressed
> transactions would represent.
>
> More generally, you could define a message which is a compressed message
> holder.  That is probably to complex to be worth the effort though.
>
>
>
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev <
>> <bitcoin-dev@lists.linuxfoundation.org>
>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev <
>>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>>
>>>
>>>> I think 25% bandwidth savings is certainly considerable, especially fo=
r
>>>> people running full nodes in countries like Australia where internet
>>>> bandwidth is lower and there are data caps.
>>>>
>>>
>>> =E2=80=8BThis reinforces the idea that such trade-off decisions should =
be be
>>> local and negotiated between peers, not a required feature of the netwo=
rk
>>> P2P.=E2=80=8B
>>>
>>>
>>> --
>>> Johnathan Corgan
>>> Corgan Labs - SDR Training and Development Services
>>> http://corganlabs.com
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing listbitcoin-dev@lists.linuxfoundation.orghttps://lis=
ts.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

--001a11467f5cbdcd88052432722c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Comments:<div><br></div><div>1) cblock seems a reasonable =
way to extend the protocol.=C2=A0 Further wrapping should probably be done =
at the stream level.</div><div><br></div><div>2) zlib has crappy security t=
rack record.</div><div><br></div><div>3) A fallback path to non-compressed =
is required, should compression fail or crash.</div><div><br></div><div>4) =
Most blocks and transactions have runs of zeroes and/or highly common bit-p=
atterns, which contributes to useful compression even at smaller sizes.=C2=
=A0 Peter Ts&#39;s most recent numbers bear this out. =C2=A0zlib has a dict=
ionary (32K?) which works well with repeated patterns such as those you see=
 with concatenated runs of transactions.</div><div><br></div><div>5) LZO sh=
ould provide much better compression, at a cost of CPU performance and usin=
g a less-reviewed, less-field-tested library.</div><div><br></div><div><br>=
</div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br><d=
iv class=3D"gmail_quote">On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via b=
itcoin-dev <span dir=3D"ltr">&lt;<a href=3D"mailto:bitcoin-dev@lists.linuxf=
oundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&=
gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><br><d=
iv class=3D"gmail_extra"><br><div class=3D"gmail_quote"><span class=3D"">On=
 Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:peter.tschipper@gmail.com" target=3D"_blank">peter.tschipper@gm=
ail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <span></span><span></span>There are better ways of sending new blocks, th=
at&#39;s certainly true
    but for sending historical blocks and seding transactions I don&#39;t
    think so.=C2=A0 This PR is really designed to save bandwidth and not
    intended to be a huge performance improvement in terms of time spent
    sending.<span><br></span></blockquote><div><br></div></span><div>If the=
 main point is for historical data, then sticking to just blocks is the bes=
t plan.<br><br></div><div>Since small blocks don&#39;t compress well, you c=
ould define a &quot;cblocks&quot; message that handles multiple blocks (jus=
t concatenate the block messages as payload before compression).=C2=A0 <br>=
<br>The sending peer could combine blocks so that each cblock is compressin=
g at least 10kB of block data (or whatever is optimal).=C2=A0 It is probabl=
y worth specifying a maximum size for network buffer reasons (either 1MB or=
 1 block maximum).<br><br></div><div>Similarly, transactions could be combi=
ned together and compressed &quot;ctxs&quot;.=C2=A0 The inv messages could =
be modified so that you can request groups of 10-20 transactions.=C2=A0 Tha=
t would depend on how much of an improvement compressed transactions would =
represent. <br><br></div><div>More generally, you could define a message wh=
ich is a compressed message holder.=C2=A0 That is probably to complex to be=
 worth the effort though.<br></div><span class=3D""><div><br>=C2=A0</div><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex"><div bgcolor=3D"#FFFFFF" text=3D"#000000"><spa=
n>
    <blockquote type=3D"cite">
      <div class=3D"gmail_extra"><br>
        <div class=3D"gmail_quote">On Tue, Nov 10, 2015 at 5:40 AM,
          Johnathan Corgan via bitcoin-dev <span dir=3D"ltr">&lt;<a href=3D=
"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank"></a><a hre=
f=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">bitcoi=
n-dev@lists.linuxfoundation.org</a>&gt;</span>
          wrote:<br>
          <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex">
            <div dir=3D"ltr"><span>
                <div style=3D"font-size:small">On
                  Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
                  <span dir=3D"ltr">&lt;<a href=3D"mailto:bitcoin-dev@lists=
.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.o=
rg</a>&gt;</span>
                  wrote:<br>
                </div>
              </span>
              <div class=3D"gmail_extra">
                <div class=3D"gmail_quote"><span>
                    <div>=C2=A0</div>
                    <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0=
 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir=3D"ltr">I think 25% bandwidth savings is
                        certainly considerable, especially for people
                        running full nodes in countries like Australia
                        where internet bandwidth is lower and there are
                        data caps.</div>
                    </blockquote>
                    <div><br>
                    </div>
                  </span>
                  <div>
                    <div style=3D"font-size:small;display:inline">=E2=80=8B=
This
                      reinforces the idea that such trade-off decisions
                      should be be local and negotiated between peers,
                      not a required feature of the network P2P.=E2=80=8B</=
div>
                    =C2=A0</div>
                </div>
                <span>
                  <div><br>
                  </div>
                  -- <br>
                  <div>
                    <div dir=3D"ltr">
                      <div>
                        <div dir=3D"ltr">
                          <div dir=3D"ltr">
                            <div dir=3D"ltr">
                              <div dir=3D"ltr">
                                <div dir=3D"ltr">
                                  <div>Johnathan Corgan<br>
                                    Corgan Labs - SDR Training and
                                    Development Services</div>
                                  <div><a href=3D"http://corganlabs.com" st=
yle=3D"font-size:12.8px" target=3D"_blank">http://corganlabs.com</a><br>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </span></div>
            </div>
            <br>
            _______________________________________________<br>
            bitcoin-dev mailing list<br>
            <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=
=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
            <a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/b=
itcoin-dev" rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundat=
ion.org/mailman/listinfo/bitcoin-dev</a><br>
            <br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
bitcoin-dev mailing list
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
target=3D"_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoi=
n-dev</a>
</pre>
    </blockquote>
    <br>
  </span></div>

</blockquote></span></div><br></div></div>
<br>_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.=
linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></blockquote></div><br></div>

--001a11467f5cbdcd88052432722c--