Return-Path: <rusty@ozlabs.org>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 19C4A1BB
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Fri,  4 Dec 2015 01:28:02 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from ozlabs.org (ozlabs.org [103.22.144.67])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 7865315C
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Fri,  4 Dec 2015 01:28:01 +0000 (UTC)
Received: by ozlabs.org (Postfix, from userid 1011)
	id 9D62F1409C2; Fri,  4 Dec 2015 12:27:58 +1100 (AEDT)
From: Rusty Russell <rusty@rustcorp.com.au>
To: Gavin Andresen <gavinandresen@gmail.com>, Emin =?utf-8?Q?G=C3=BCn?=
	Sirer <el33th4x0r@gmail.com>
In-Reply-To: <CABsx9T3+y6ip3cUW8jP-0n+1FS8t_FdX8fA42JaTO7qW5pMhzA@mail.gmail.com>
References: <565CD7D8.3070102@gmail.com>
	<90EF4E6C-9A71-4A35-A938-EAFC1A24DD24@mattcorallo.com>
	<CAPkFh0t9SwVOLrPnL7z80s-Rriezhqxn_3vXKYRxr6JVGNiUZQ@mail.gmail.com>
	<CABsx9T3+y6ip3cUW8jP-0n+1FS8t_FdX8fA42JaTO7qW5pMhzA@mail.gmail.com>
User-Agent: Notmuch/0.20.2 (http://notmuchmail.org) Emacs/24.5.1
	(x86_64-pc-linux-gnu)
Date: Fri, 04 Dec 2015 09:37:56 +1030
Message-ID: <87vb8f5dwz.fsf@rustcorp.com.au>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED,
	T_RP_MATCHES_RCVD autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] [BIP Draft] Datastream compression of Blocks
	and	Transactions
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 04 Dec 2015 01:28:02 -0000

Gavin Andresen via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
writes:
> On Wed, Dec 2, 2015 at 1:57 PM, Emin G=C3=BCn Sirer <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> How to Do It
>>
>> If we want to compress Bitcoin, a programming challenge/contest would be
>> one of the best ways to find the best possible, Bitcoin-specific
>> compressor. This is the kind of self-contained exercise that bright young
>> hackers love to tackle. It'd bring in new programmers into the ecosystem,
>> and many of us would love to discover the limits of compressibility for
>> Bitcoin bits on a wire. And the results would be interesting even if the
>> final compression engine is not enabled by default, or not even merged.
>>
>
> I love this idea. Lets build a standardized data set to test against using
> real data from the network (has anybody done this yet?).

https://github.com/rustyrussell/bitcoin-corpus

It includes mempool contents and tx receipt logs for 1 week across 4
nodes.  I vaguely plan to update it every year.

A more ambitious version would add some topology information, but we
need to figure out some anonymization strategy for the data.

Cheers,
Rusty.