Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 16BA5941 for ; Tue, 10 Nov 2015 16:46:18 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C958B1A7 for ; Tue, 10 Nov 2015 16:46:16 +0000 (UTC) Received: by wmww144 with SMTP id w144so125925226wmw.1 for ; Tue, 10 Nov 2015 08:46:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=inA/tJCHoldemqC0wCF+7yIcm48vnaGcNX0IZ/GPa6I=; b=imcq8Enqj++b/dcoMPQJVYsVNJBEUSOnR8DMlG+JS/cHlt2dofUhRlJ9eaPdpLuYk1 B8q7ZPUkkF13GeMdyGRsKbi4vnX+OkwY8LyqjxQPZ84UA4ZP83QAhanZbLCqozNU55Go vQIHbnCz07CdwHxGk58m9CQOF+hivg4mWvdRKmOAIlNIB40KuSP19B8nmXTODxqiO1rl g8AyZUeFjvWC2ZgTJob5ovwnu8Dg+DsgX/cJaMPFTSfzYO2YCXTmxr9M5+aBTFLKKttg wZVZ9Ek9904XjwueBSVpchJbsa2E1ivVQYFbWJOdDuTRUDfNIzTvvTEWrbMxCQ24SSAw CYiQ== MIME-Version: 1.0 X-Received: by 10.28.215.205 with SMTP id o196mr6136561wmg.95.1447173975557; Tue, 10 Nov 2015 08:46:15 -0800 (PST) Received: by 10.28.212.79 with HTTP; Tue, 10 Nov 2015 08:46:15 -0800 (PST) In-Reply-To: References: <5640F172.3010004@gmail.com> <20151109210449.GE5886@mcelrath.org> <5642172C.701@gmail.com> Date: Tue, 10 Nov 2015 11:46:15 -0500 Message-ID: From: Jeff Garzik To: Tier Nolan Content-Type: multipart/alternative; boundary=001a11467f5cbdcd88052432722c X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] request BIP number for: "Support for Datastream Compression" X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Nov 2015 16:46:18 -0000 --001a11467f5cbdcd88052432722c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Comments: 1) cblock seems a reasonable way to extend the protocol. Further wrapping should probably be done at the stream level. 2) zlib has crappy security track record. 3) A fallback path to non-compressed is required, should compression fail or crash. 4) Most blocks and transactions have runs of zeroes and/or highly common bit-patterns, which contributes to useful compression even at smaller sizes. Peter Ts's most recent numbers bear this out. zlib has a dictionary (32K?) which works well with repeated patterns such as those you see with concatenated runs of transactions. 5) LZO should provide much better compression, at a cost of CPU performance and using a less-reviewed, less-field-tested library. On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > > > On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper < > peter.tschipper@gmail.com> wrote: > >> There are better ways of sending new blocks, that's certainly true but >> for sending historical blocks and seding transactions I don't think so. >> This PR is really designed to save bandwidth and not intended to be a hu= ge >> performance improvement in terms of time spent sending. >> > > If the main point is for historical data, then sticking to just blocks is > the best plan. > > Since small blocks don't compress well, you could define a "cblocks" > message that handles multiple blocks (just concatenate the block messages > as payload before compression). > > The sending peer could combine blocks so that each cblock is compressing > at least 10kB of block data (or whatever is optimal). It is probably wor= th > specifying a maximum size for network buffer reasons (either 1MB or 1 blo= ck > maximum). > > Similarly, transactions could be combined together and compressed "ctxs". > The inv messages could be modified so that you can request groups of 10-2= 0 > transactions. That would depend on how much of an improvement compressed > transactions would represent. > > More generally, you could define a message which is a compressed message > holder. That is probably to complex to be worth the effort though. > > > >> >> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev < >> >> bitcoin-dev@lists.linuxfoundation.org> wrote: >> >>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev < >>> bitcoin-dev@lists.linuxfoundation.org> wrote: >>> >>> >>>> I think 25% bandwidth savings is certainly considerable, especially fo= r >>>> people running full nodes in countries like Australia where internet >>>> bandwidth is lower and there are data caps. >>>> >>> >>> =E2=80=8BThis reinforces the idea that such trade-off decisions should = be be >>> local and negotiated between peers, not a required feature of the netwo= rk >>> P2P.=E2=80=8B >>> >>> >>> -- >>> Johnathan Corgan >>> Corgan Labs - SDR Training and Development Services >>> http://corganlabs.com >>> >>> _______________________________________________ >>> bitcoin-dev mailing list >>> bitcoin-dev@lists.linuxfoundation.org >>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>> >>> >> >> >> _______________________________________________ >> bitcoin-dev mailing listbitcoin-dev@lists.linuxfoundation.orghttps://lis= ts.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> >> > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > --001a11467f5cbdcd88052432722c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Comments:

1) cblock seems a reasonable = way to extend the protocol.=C2=A0 Further wrapping should probably be done = at the stream level.

2) zlib has crappy security t= rack record.

3) A fallback path to non-compressed = is required, should compression fail or crash.

4) = Most blocks and transactions have runs of zeroes and/or highly common bit-p= atterns, which contributes to useful compression even at smaller sizes.=C2= =A0 Peter Ts's most recent numbers bear this out. =C2=A0zlib has a dict= ionary (32K?) which works well with repeated patterns such as those you see= with concatenated runs of transactions.

5) LZO sh= ould provide much better compression, at a cost of CPU performance and usin= g a less-reviewed, less-field-tested library.


=



On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via b= itcoin-dev <bitcoin-dev@lists.linuxfoundation.org&= gt; wrote:


On= Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper <peter.tschipper@gm= ail.com> wrote:
=20 =20 =20 There are better ways of sending new blocks, th= at's certainly true but for sending historical blocks and seding transactions I don't think so.=C2=A0 This PR is really designed to save bandwidth and not intended to be a huge performance improvement in terms of time spent sending.

If the= main point is for historical data, then sticking to just blocks is the bes= t plan.

Since small blocks don't compress well, you c= ould define a "cblocks" message that handles multiple blocks (jus= t concatenate the block messages as payload before compression).=C2=A0
=
The sending peer could combine blocks so that each cblock is compressin= g at least 10kB of block data (or whatever is optimal).=C2=A0 It is probabl= y worth specifying a maximum size for network buffer reasons (either 1MB or= 1 block maximum).

Similarly, transactions could be combi= ned together and compressed "ctxs".=C2=A0 The inv messages could = be modified so that you can request groups of 10-20 transactions.=C2=A0 Tha= t would depend on how much of an improvement compressed transactions would = represent.

More generally, you could define a message wh= ich is a compressed message holder.=C2=A0 That is probably to complex to be= worth the effort though.

=C2=A0

On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev <bitcoi= n-dev@lists.linuxfoundation.org> wrote:
On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.o= rg> wrote:
=C2=A0
I think 25% bandwidth savings is certainly considerable, especially for people running full nodes in countries like Australia where internet bandwidth is lower and there are data caps.
=E2=80=8B= This reinforces the idea that such trade-off decisions should be be local and negotiated between peers, not a required feature of the network P2P.=E2=80=8B =C2=A0

--
Johnathan Corgan
Corgan Labs - SDR Training and Development Services

_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundat= ion.org/mailman/listinfo/bitcoin-dev




_______________________________________________
bitcoin-dev mailing list
=
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoi=
n-dev



_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.= linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev


--001a11467f5cbdcd88052432722c--