Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 33C316C for ; Mon, 27 Nov 2017 20:49:16 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from bitcoin.jonasschnelli.ch (bitcoinsrv.jonasschnelli.ch [138.201.55.219]) by smtp1.linuxfoundation.org (Postfix) with ESMTP id E71FB1AE for ; Mon, 27 Nov 2017 20:49:14 +0000 (UTC) Received: by bitcoin.jonasschnelli.ch (Postfix, from userid 1002) id DE4B115E201C; Mon, 27 Nov 2017 21:49:13 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 Received: from [192.168.1.3] (cpe-66-91-52-81.hawaii.res.rr.com [66.91.52.81]) by bitcoin.jonasschnelli.ch (Postfix) with ESMTPSA id 29C8A15E082F; Mon, 27 Nov 2017 21:49:11 +0100 (CET) From: Jonas Schnelli Content-Type: multipart/signed; boundary="Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Date: Mon, 27 Nov 2017 10:49:07 -1000 References: To: Jeff Johnson , Bitcoin Protocol Discussion In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3273) X-Virus-Scanned: clamav-milter 0.99.2 at bitcoinsrv.jonasschnelli.ch X-Virus-Status: Clean Subject: Re: [bitcoin-dev] Block compression X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 20:49:16 -0000 --Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907 Content-Type: multipart/alternative; boundary="Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265" --Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Jeff There where previous discussions about similar approaches [1] [2]. I=E2=80=99m not sure if compression should be built into the protocol. My humble understanding of it, is, that it should be built into = different layers. If bandwidth is a concern, then on the fly gzip compression like apaches = mod_deflate could be something. But I expect fast propagation is often = more important then a ~30% bandwidth reduction. Bandwidth may be a concern for historical blocks transmission. If you = continue the proposal, I think you should focus on historical blocks. If disk space is a concern, then the database layer should handle the = compression. Thanks =E2=80=94 [1] = https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/0116= 92.html = [2] https://github.com/bitcoin/bitcoin/pull/6973 = > Am 26.11.2017 um 16:11 schrieb Jeff Johnson via bitcoin-dev = : >=20 > I'm new to this mailing list and apologize if this has been suggested = before. I was directed from the Bitcoin core github to this mailing list = for suggestions. >=20 > I'd just like to post a possible solution that increases the amount of = data in a block without actually increasing the size on disk or the size = in memory or the size transmitted over the Internet. Simply applying = various compression algorithms, I was able to achieve about a 50% = compression ratio. Here are my findings on a recent Bitcoin block using = max compression for all methods: >=20 > Raw block > 998,198 bytes >=20 > Gzip > 521,212 bytes (52% ratio) > (needs 2MB to decompress). >=20 > LZMA > 415,308 bytes (41% ratio) > (1MB dictionary, needs 3MB to decompress) >=20 > - ZStandard: 469,179 bytes (47% ratio) > (1MB memory to decompress) >=20 > - LZ4: 641,063 bytes (64% ratio) > (32-64K to decompress) >=20 > The compression time on my modest laptop (2 years old) was "instant". = I ran all from the command line and did not notice any lag as I pressed = enter to do the compression, so easily less than a second. But = compression time doesn't matter, decompression time is what matters as = blocks will be decompressed billions of times more than they will be = compressed. Decompression speed for LZ4 is the fastest of the above = methods, at 3.3GB / second, slightly less than half the speed of memcpy, = see char at (https://github.com/lz4/lz4 ). >=20 > If decompression speed, CPU and memory usage is a concern, LZ4 is a no = brainer. You basically get a 33% larger block size for "free". But = ZStandard, in my opinion, makes the most sense as it offers greater than = 50% compression ratio with a very good decompression ratio of 900MB / = second. >=20 > If this were implemented in the Bitcoin protocol, there would need to = be a place to specify the compression type in a set of bits somewhere, = so that future compression algorithms could potentially be added. >=20 > Miners could do nothing and keep sending blocks as is, and these = blocks would have "no compression" as the type of compression, just as = today. Or they could opt in to compress blocks and choose how many = transactions they want to stuff into the block, keeping the compressed = size under the limit. >=20 > The bitcoin client code would also need to be able to handle the = appropriate compression bits, and limits of signature data, etc. = modified to deal with the compression. >=20 > I understand schnorr signatures are on the roadmap as a 25% = compression gain which is great, I suspect that schnorr signatures would = compress even further when compressed with the above compression = methods. >=20 > Here is a link to the block that I compressed: = https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98 = >=20 > Thanks for reading, best wishes to all. >=20 > -- Jeff Johnson > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev --Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi Jeff

There where previous discussions about similar approaches [1] = [2].

I=E2=80=99m= not sure if compression should be built into the protocol.
My humble understanding of it, is, that it should be built = into different layers.

If bandwidth is a concern, then on the fly gzip compression = like apaches mod_deflate could be something. But I expect fast = propagation is often more important then a ~30% bandwidth = reduction.
Bandwidth may be a concern for = historical blocks transmission. If you continue the proposal, I think = you should focus on historical blocks.

If disk space is a concern, then the = database layer should handle the compression.

Thanks
=E2=80=94
</jonas>





Am 26.11.2017 um 16:11 schrieb = Jeff Johnson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>:

I'm new to this mailing list and apologize if = this has been suggested before. I was directed from the Bitcoin core = github to this mailing list for suggestions.

I'd just like to post a possible solution that = increases the amount of data in a block without actually increasing the = size on disk or the size in memory or the size transmitted over the = Internet. Simply applying various compression algorithms, I was able to = achieve about a 50% compression ratio. Here are my findings on a recent = Bitcoin block using max compression for all methods:

Raw block
998,198 = bytes

Gzip
521,212 bytes (52% ratio)
(needs 2MB to decompress).

LZMA
415,308 bytes = (41% ratio)
(1MB dictionary, needs 3MB to = decompress)

- = ZStandard: 469,179 bytes (47% ratio)
(1MB = memory to decompress)

- LZ4: 641,063 bytes (64% ratio)
(32-64K to decompress)

The compression time on my modest = laptop (2 years old) was "instant". I ran all from the command line and = did not notice any lag as I pressed enter to do the compression, so = easily less than a second. But compression time doesn't matter, = decompression time is what matters as blocks will be decompressed = billions of times more than they will be compressed. Decompression speed = for LZ4 is the fastest of the above methods, at 3.3GB / second, slightly = less than half the speed of memcpy, see char at (https://github.com/lz4/lz4).

If decompression speed, CPU and memory = usage is a concern, LZ4 is a no brainer. You basically get a 33% larger = block size for "free". But ZStandard, in my opinion, makes the most = sense as it offers greater than 50% compression ratio with a very good = decompression ratio of 900MB / second.

If this were implemented in the Bitcoin = protocol, there would need to be a place to specify the compression type = in a set of bits somewhere, so that future compression algorithms could = potentially be added.

Miners could do nothing and keep = sending blocks as is, and these blocks would have "no compression" as = the type of compression, just as today. Or they could opt in to compress = blocks and choose how many transactions they want to stuff into the = block, keeping the compressed size under the limit.

The bitcoin client code = would also need to be able to handle the appropriate compression bits, = and limits of signature data, etc. modified to deal with the = compression.

I = understand schnorr signatures are on the roadmap as a 25% = compression gain which is great, I suspect that schnorr = signatures would compress even further when compressed with the = above compression methods.

Here is a link to the block that I compressed: https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNU= olEtq98

Thanks for reading, best wishes = to all.

-- = Jeff Johnson
_______________________________________________
bitcoin-dev = mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev<= br class=3D"">

= --Apple-Mail=_387BD3B3-49D5-46C7-B22E-6193339FD265-- --Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEyhopCNzi8TB0xizeHrd2uwPHki0FAlocekMACgkQHrd2uwPH ki26qBAAu8JF95UrzGX2fX4e9CzAOq9nh4C7sDLAlo1iQZv4U18QqyHW1m1jwcDx +R+PCUe/mnc5aBnPJIz+54hI6KkKT5aFak8iQKDrxUz4sJ3ENJ+jrpf1oBb+eUyc 8fBZ2mia/18OAUBKxYKBvPsmAFEnOeF0eOdYoQMva/A9If6c7WHY4BfGd55K8kJY 2uAUIKhgTKeJoTBJe3yDwDXC3J/k6Rkrybn7vQDVCKyeivdpTxJNnMJ6qkZcydpt bhUrdXtQlgGXngolXWCAYoBmVzsfb/KZ45f2w+u1Mf271IkPKqK6vhWbV+eKySmj 7MrjcqgYoS7MBwrvXgUSJx0zZ/0UD4ytkMEzqrapIiQR8etUjbcdCaftOVEhRpws vTFBIE7JG/ZU9szFZTZWChnLPuXM3NtmcCk68g/Tnn0CgQXctKfi/jWyXI3P2mV7 PylD/e62KmZ73nSwcoTQqod1wNIiuj7l4Ae3HxjCn7XaI/DFMdk7FCruV+N5v7O3 YbOM+RMTMiFGUmPmsh1v/WGaGIYJhdYxPKUAX24IYEvzdvPO7k5Lv6YYfVpRz2Rg 84WjZuHUET4KBAhZ5E25daydG8v/63F65lbUk4/KJBIwN1vZSGkypXtK9S/lIVG/ daa0M3DIcYsESh6XeBd1atK7suXFm6Sxt/rEyCPKlURKt2iTLEA= =dNX8 -----END PGP SIGNATURE----- --Apple-Mail=_61E1E831-54FC-4DBA-95C6-5F3666D58907--