From: Rusty Russell <rusty@rustcorp.com.au>
To: Gavin Andresen <gavinandresen@gmail.com>, Emin =?utf-8?Q?G=C3=BCn?=
	Sirer <el33th4x0r@gmail.com>
In-Reply-To: <CABsx9T3+y6ip3cUW8jP-0n+1FS8t_FdX8fA42JaTO7qW5pMhzA@mail.gmail.com>
References: <565CD7D8.3070102@gmail.com>
	<90EF4E6C-9A71-4A35-A938-EAFC1A24DD24@mattcorallo.com>
	<CAPkFh0t9SwVOLrPnL7z80s-Rriezhqxn_3vXKYRxr6JVGNiUZQ@mail.gmail.com>
	<CABsx9T3+y6ip3cUW8jP-0n+1FS8t_FdX8fA42JaTO7qW5pMhzA@mail.gmail.com>
User-Agent: Notmuch/0.20.2 (http://notmuchmail.org) Emacs/24.5.1
	(x86_64-pc-linux-gnu)
Date: Fri, 04 Dec 2015 09:37:56 +1030
Message-ID: <87vb8f5dwz.fsf@rustcorp.com.au>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks
	and	Transactions
Precedence: list

Gavin Andresen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
writes:
> On Wed, Dec 2, 2015 at 1:57 PM, Emin G=C3=BCn Sirer <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> How to Do It
>>
>> If we want to compress Bitcoin, a programming challenge/contest would be
>> one of the best ways to find the best possible, Bitcoin-specific
>> compressor. This is the kind of self-contained exercise that bright young
>> hackers love to tackle. It'd bring in new programmers into the ecosystem,
>> and many of us would love to discover the limits of compressibility for
>> Bitcoin bits on a wire. And the results would be interesting even if the
>> final compression engine is not enabled by default, or not even merged.
>>
>
> I love this idea. Lets build a standardized data set to test against using
> real data from the network (has anybody done this yet?).

https://github.com/rustyrussell/bitcoin-corpus

It includes mempool contents and tx receipt logs for 1 week across 4
nodes.  I vaguely plan to update it every year.

A more ambitious version would add some topology information, but we
need to figure out some anonymization strategy for the data.

Cheers,
Rusty.