Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 19C4A1BB for ; Fri, 4 Dec 2015 01:28:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ozlabs.org (ozlabs.org [103.22.144.67]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 7865315C for ; Fri, 4 Dec 2015 01:28:01 +0000 (UTC) Received: by ozlabs.org (Postfix, from userid 1011) id 9D62F1409C2; Fri, 4 Dec 2015 12:27:58 +1100 (AEDT) From: Rusty Russell To: Gavin Andresen , Emin =?utf-8?Q?G=C3=BCn?= Sirer In-Reply-To: References: <565CD7D8.3070102@gmail.com> <90EF4E6C-9A71-4A35-A938-EAFC1A24DD24@mattcorallo.com> User-Agent: Notmuch/0.20.2 (http://notmuchmail.org) Emacs/24.5.1 (x86_64-pc-linux-gnu) Date: Fri, 04 Dec 2015 09:37:56 +1030 Message-ID: <87vb8f5dwz.fsf@rustcorp.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks and Transactions X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2015 01:28:02 -0000 Gavin Andresen via bitcoin-dev writes: > On Wed, Dec 2, 2015 at 1:57 PM, Emin G=C3=BCn Sirer < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> How to Do It >> >> If we want to compress Bitcoin, a programming challenge/contest would be >> one of the best ways to find the best possible, Bitcoin-specific >> compressor. This is the kind of self-contained exercise that bright young >> hackers love to tackle. It'd bring in new programmers into the ecosystem, >> and many of us would love to discover the limits of compressibility for >> Bitcoin bits on a wire. And the results would be interesting even if the >> final compression engine is not enabled by default, or not even merged. >> > > I love this idea. Lets build a standardized data set to test against using > real data from the network (has anybody done this yet?). https://github.com/rustyrussell/bitcoin-corpus It includes mempool contents and tx receipt logs for 1 week across 4 nodes. I vaguely plan to update it every year. A more ambitious version would add some topology information, but we need to figure out some anonymization strategy for the data. Cheers, Rusty.