path: root/c7/e0cbb6fbc88d545e6d666ec80ef13a31d189ef

Return-Path: <dev@jonasschnelli.ch>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 33C316C
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 27 Nov 2017 20:49:16 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from bitcoin.jonasschnelli.ch (bitcoinsrv.jonasschnelli.ch
	[138.201.55.219])
	by smtp1.linuxfoundation.org (Postfix) with ESMTP id E71FB1AE
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 27 Nov 2017 20:49:14 +0000 (UTC)
Received: by bitcoin.jonasschnelli.ch (Postfix, from userid 1002)
	id DE4B115E201C; Mon, 27 Nov 2017 21:49:13 +0100 (CET)
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE,
	T_RP_MATCHES_RCVD autolearn=ham version=3.3.1
Received: from [192.168.1.3] (cpe-66-91-52-81.hawaii.res.rr.com [66.91.52.81])
	by bitcoin.jonasschnelli.ch (Postfix) with ESMTPSA id 29C8A15E082F; 
	Mon, 27 Nov 2017 21:49:11 +0100 (CET)
From: Jonas Schnelli <dev@jonasschnelli.ch>
Content-Type: multipart/signed;
	boundary="Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907";
	protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Date: Mon, 27 Nov 2017 10:49:07 -1000
References: <CAHAcUENRbrz6pv_HSe4KDAY-onyMKnMJk8H6yUhqx4LoK0NeEw@mail.gmail.com>
To: Jeff Johnson <jjxtra@gmail.com>,
	Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
In-Reply-To: <CAHAcUENRbrz6pv_HSe4KDAY-onyMKnMJk8H6yUhqx4LoK0NeEw@mail.gmail.com>
Message-Id: <A30B6CDA-21D7-4AA2-8BBB-9D2E9458B9F0@jonasschnelli.ch>
X-Mailer: Apple Mail (2.3273)
X-Virus-Scanned: clamav-milter 0.99.2 at bitcoinsrv.jonasschnelli.ch
X-Virus-Status: Clean
Subject: Re: [bitcoin-dev] Block compression
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Nov 2017 20:49:16 -0000


--Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265"


--Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Jeff

There where previous discussions about similar approaches [1] [2].

I=E2=80=99m not sure if compression should be built into the protocol.
My humble understanding of it, is, that it should be built into =
different layers.

If bandwidth is a concern, then on the fly gzip compression like apaches =
mod_deflate could be something. But I expect fast propagation is often =
more important then a ~30% bandwidth reduction.
Bandwidth may be a concern for historical blocks transmission. If you =
continue the proposal, I think you should focus on historical blocks.

If disk space is a concern, then the database layer should handle the =
compression.

Thanks
=E2=80=94
</jonas>


[1] =
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/0116=
92.html =
<https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011=
692.html>
[2] https://github.com/bitcoin/bitcoin/pull/6973 =
<https://github.com/bitcoin/bitcoin/pull/6973>



> Am 26.11.2017 um 16:11 schrieb Jeff Johnson via bitcoin-dev =
<bitcoin-dev@lists.linuxfoundation.org>:
>=20
> I'm new to this mailing list and apologize if this has been suggested =
before. I was directed from the Bitcoin core github to this mailing list =
for suggestions.
>=20
> I'd just like to post a possible solution that increases the amount of =
data in a block without actually increasing the size on disk or the size =
in memory or the size transmitted over the Internet. Simply applying =
various compression algorithms, I was able to achieve about a 50% =
compression ratio. Here are my findings on a recent Bitcoin block using =
max compression for all methods:
>=20
> Raw block
> 998,198 bytes
>=20
> Gzip
> 521,212 bytes (52% ratio)
> (needs 2MB to decompress).
>=20
> LZMA
> 415,308 bytes (41% ratio)
> (1MB dictionary, needs 3MB to decompress)
>=20
> - ZStandard: 469,179 bytes (47% ratio)
> (1MB memory to decompress)
>=20
> - LZ4: 641,063 bytes (64% ratio)
> (32-64K to decompress)
>=20
> The compression time on my modest laptop (2 years old) was "instant". =
I ran all from the command line and did not notice any lag as I pressed =
enter to do the compression, so easily less than a second. But =
compression time doesn't matter, decompression time is what matters as =
blocks will be decompressed billions of times more than they will be =
compressed. Decompression speed for LZ4 is the fastest of the above =
methods, at 3.3GB / second, slightly less than half the speed of memcpy, =
see char at (https://github.com/lz4/lz4 <https://github.com/lz4/lz4>).
>=20
> If decompression speed, CPU and memory usage is a concern, LZ4 is a no =
brainer. You basically get a 33% larger block size for "free". But =
ZStandard, in my opinion, makes the most sense as it offers greater than =
50% compression ratio with a very good decompression ratio of 900MB / =
second.
>=20
> If this were implemented in the Bitcoin protocol, there would need to =
be a place to specify the compression type in a set of bits somewhere, =
so that future compression algorithms could potentially be added.
>=20
> Miners could do nothing and keep sending blocks as is, and these =
blocks would have "no compression" as the type of compression, just as =
today. Or they could opt in to compress blocks and choose how many =
transactions they want to stuff into the block, keeping the compressed =
size under the limit.
>=20
> The bitcoin client code would also need to be able to handle the =
appropriate compression bits, and limits of signature data, etc. =
modified to deal with the compression.
>=20
> I understand schnorr signatures are on the roadmap as a 25% =
compression gain which is great, I suspect that schnorr signatures would =
compress even further when compressed with the above compression =
methods.
>=20
> Here is a link to the block that I compressed: =
https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98 =
<https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98>
>=20
> Thanks for reading, best wishes to all.
>=20
> -- Jeff Johnson
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


--Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hi Jeff<div class=3D""><br class=3D""></div><div =
class=3D"">There where previous discussions about similar approaches [1] =
[2].</div><div class=3D""><br class=3D""></div><div class=3D"">I=E2=80=99m=
 not sure if compression should be built into the protocol.</div><div =
class=3D"">My humble understanding of it, is, that it should be built =
into different layers.</div><div class=3D""><br class=3D""></div><div =
class=3D"">If bandwidth is a concern, then on the fly gzip compression =
like apaches mod_deflate could be something. But I expect fast =
propagation is often more important then a ~30% bandwidth =
reduction.</div><div class=3D"">Bandwidth may be a concern for =
historical blocks transmission. If you continue the proposal, I think =
you should focus on historical blocks.</div><div class=3D""><br =
class=3D""></div><div class=3D"">If disk space is a concern, then the =
database layer should handle the compression.</div><div class=3D""><br =
class=3D""></div><div class=3D"">Thanks</div><div class=3D"">=E2=80=94</di=
v><div class=3D"">&lt;/jonas&gt;</div><div class=3D""><br =
class=3D""></div><div class=3D""><br class=3D""></div><div =
class=3D"">[1]&nbsp;<a =
href=3D"https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-Novem=
ber/011692.html" =
class=3D"">https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-No=
vember/011692.html</a></div><div class=3D"">[2]&nbsp;<a =
href=3D"https://github.com/bitcoin/bitcoin/pull/6973" =
class=3D"">https://github.com/bitcoin/bitcoin/pull/6973</a></div><div =
class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div =
class=3D""><br class=3D""></div><div class=3D""><div><blockquote =
type=3D"cite" class=3D""><div class=3D"">Am 26.11.2017 um 16:11 schrieb =
Jeff Johnson via bitcoin-dev &lt;<a =
href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
class=3D"">bitcoin-dev@lists.linuxfoundation.org</a>&gt;:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" =
class=3D""><div class=3D"">I'm new to this mailing list and apologize if =
this has been suggested before. I was directed from the Bitcoin core =
github to this mailing list for suggestions.</div><div class=3D""><br =
class=3D""></div>I'd just like to post a possible solution that =
increases the amount of data in a block without actually increasing the =
size on disk or the size in memory or the size transmitted over the =
Internet. Simply applying various compression algorithms, I was able to =
achieve about a 50% compression ratio. Here are my findings on a recent =
Bitcoin block using max compression for all methods:<div class=3D""><br =
class=3D""></div><div class=3D"">Raw block</div><div class=3D"">998,198 =
bytes</div><div class=3D""><br class=3D""></div><div =
class=3D"">Gzip</div><div class=3D"">521,212 bytes (52% ratio)</div><div =
class=3D"">(needs 2MB to decompress).</div><div class=3D""><br =
class=3D""></div><div class=3D"">LZMA</div><div class=3D"">415,308 bytes =
(41% ratio)</div><div class=3D""> (1MB dictionary, needs 3MB to =
decompress)</div><div class=3D""><br class=3D""></div><div class=3D"">- =
ZStandard:&nbsp;469,179 bytes (47% ratio)</div><div class=3D"">(1MB =
memory to decompress)</div><div class=3D""><br class=3D""></div><div =
class=3D"">- LZ4:&nbsp;641,063 bytes (64% ratio)</div><div =
class=3D"">(32-64K to decompress)</div><div class=3D""><br =
class=3D""></div><div class=3D"">The compression time on my modest =
laptop (2 years old) was "instant". I ran all from the command line and =
did not notice any lag as I pressed enter to do the compression, so =
easily less than a second. But compression time doesn't matter, =
decompression time is what matters as blocks will be decompressed =
billions of times more than they will be compressed. Decompression speed =
for LZ4 is the fastest of the above methods, at 3.3GB / second, slightly =
less than half the speed of memcpy, see char at (<a =
href=3D"https://github.com/lz4/lz4" =
class=3D"">https://github.com/lz4/lz4</a>).</div><div class=3D""><br =
class=3D""></div><div class=3D"">If decompression speed, CPU and memory =
usage is a concern, LZ4 is a no brainer. You basically get a 33% larger =
block size for "free". But ZStandard, in my opinion, makes the most =
sense as it offers greater than 50% compression ratio with a very good =
decompression ratio of 900MB / second.</div><div class=3D""><br =
class=3D""></div><div class=3D"">If this were implemented in the Bitcoin =
protocol, there would need to be a place to specify the compression type =
in a set of bits somewhere, so that future compression algorithms could =
potentially be added.<br class=3D""></div><div class=3D""><br =
class=3D""></div><div class=3D"">Miners could do nothing and keep =
sending blocks as is, and these blocks would have "no compression" as =
the type of compression, just as today. Or they could opt in to compress =
blocks and choose how many transactions they want to stuff into the =
block, keeping the compressed size under the limit.</div><div =
class=3D""><br class=3D""></div><div class=3D"">The bitcoin client code =
would also need to be able to handle the appropriate compression bits, =
and limits of signature data, etc. modified to deal with the =
compression.</div><div class=3D""><br class=3D""></div><div class=3D"">I =
understand&nbsp;schnorr signatures are on the roadmap as a 25% =
compression gain which is great, I suspect that schnorr =
signatures&nbsp;would compress even further when compressed with the =
above compression methods.</div><div class=3D""><br class=3D""></div><div =
class=3D"">Here is a link to the block that I compressed:&nbsp;<a =
href=3D"https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolE=
tq98" =
class=3D"">https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNU=
olEtq98</a><br class=3D""></div><div class=3D""><span class=3D"gmail-key =
gmail-file-link-info" =
style=3D"font-size:13px;outline:none;color:rgb(217,0,7);font-family:source=
_sans_proregular,Arial;white-space:pre-wrap"><br =
class=3D""></span></div><div class=3D"">Thanks for reading, best wishes =
to all.</div><div class=3D""><br class=3D""></div><div class=3D"">-- =
Jeff Johnson<br class=3D""></div></div>
_______________________________________________<br class=3D"">bitcoin-dev =
mailing list<br class=3D""><a =
href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
class=3D"">bitcoin-dev@lists.linuxfoundation.org</a><br =
class=3D"">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev<=
br class=3D""></div></blockquote></div><br class=3D""></div></body></html>=

--Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265--

--Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEyhopCNzi8TB0xizeHrd2uwPHki0FAlocekMACgkQHrd2uwPH
ki26qBAAu8JF95UrzGX2fX4e9CzAOq9nh4C7sDLAlo1iQZv4U18QqyHW1m1jwcDx
+R+PCUe/mnc5aBnPJIz+54hI6KkKT5aFak8iQKDrxUz4sJ3ENJ+jrpf1oBb+eUyc
8fBZ2mia/18OAUBKxYKBvPsmAFEnOeF0eOdYoQMva/A9If6c7WHY4BfGd55K8kJY
2uAUIKhgTKeJoTBJe3yDwDXC3J/k6Rkrybn7vQDVCKyeivdpTxJNnMJ6qkZcydpt
bhUrdXtQlgGXngolXWCAYoBmVzsfb/KZ45f2w+u1Mf271IkPKqK6vhWbV+eKySmj
7MrjcqgYoS7MBwrvXgUSJx0zZ/0UD4ytkMEzqrapIiQR8etUjbcdCaftOVEhRpws
vTFBIE7JG/ZU9szFZTZWChnLPuXM3NtmcCk68g/Tnn0CgQXctKfi/jWyXI3P2mV7
PylD/e62KmZ73nSwcoTQqod1wNIiuj7l4Ae3HxjCn7XaI/DFMdk7FCruV+N5v7O3
YbOM+RMTMiFGUmPmsh1v/WGaGIYJhdYxPKUAX24IYEvzdvPO7k5Lv6YYfVpRz2Rg
84WjZuHUET4KBAhZ5E25daydG8v/63F65lbUk4/KJBIwN1vZSGkypXtK9S/lIVG/
daa0M3DIcYsESh6XeBd1atK7suXFm6Sxt/rEyCPKlURKt2iTLEA=
=dNX8
-----END PGP SIGNATURE-----

--Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907--