summaryrefslogtreecommitdiff
path: root/f1/3a1b05323c9a67c86fe87eeb9e19802ff70123
blob: 5ccf237188905abf914bfdfc86f7a129ee6b5477 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
Return-Path: <rusty@ozlabs.org>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 1716325A
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 10 May 2016 05:39:12 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from ozlabs.org (ozlabs.org [103.22.144.67])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4CF35134
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 10 May 2016 05:39:11 +0000 (UTC)
Received: by ozlabs.org (Postfix, from userid 1011)
	id 3r3p2z6k6Fz9sdm; Tue, 10 May 2016 15:39:07 +1000 (AEST)
From: Rusty Russell <rusty@rustcorp.com.au>
To: Pieter Wuille <pieter.wuille@gmail.com>,
	Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
	bitcoin-dev@lists.linuxfoundation.org
In-Reply-To: <5730C37E.2000004@gmail.com>
References: <5727D102.1020807@mattcorallo.com> <5730C37E.2000004@gmail.com>
User-Agent: Notmuch/0.21 (http://notmuchmail.org) Emacs/24.5.1
	(x86_64-pc-linux-gnu)
Date: Tue, 10 May 2016 14:58:19 +0930
Message-ID: <87twi6zdl8.fsf@rustcorp.com.au>
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Status: No, score=-6.3 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED,
	RP_MATCHES_RCVD autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] Compact Block Relay BIP
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 10 May 2016 05:39:12 -0000

Pieter Wuille via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> writes:
> On 05/03/2016 12:13 AM, lf-lists at mattcorallo.com (Matt Corallo) wrote:
>> Hi all,
>> 
>> The following is a BIP-formatted design spec for compact block relay
>> designed to limit on wire bytes during block relay. You can find the
>> latest version of this document at
>> https://github.com/TheBlueMatt/bips/blob/master/bip-TODO.mediawiki.
>
> Hi Matt,
>
> thank you for working on this!

Indeed!  Sorry for the delayed feedback.

>> |shortids||List of uint64_ts||8*shortids_length bytes||Little
>> Endian||The short transaction IDs calculated from the transactions which
>> were not provided explicitly in prefilledtxn
>
> I tried to derive what length of short ids is actually necessary (some
> write-up is on
> https://gist.github.com/sipa/b2eb2e486156b5509ac711edd16153ed but it's
> incomplete).

I did this for IBLT testing.

I used variable-length bit encodings, and used the shortest encoding
which is unique to you (including mempool).  It's a little more work,
but for an average node transmitting a block with 1300 txs and another
~3000 in the mempool, you expect about 12 bits per transaction.  IOW,
about 1/5 of your current size.  Critically, we might be able to fit in
two or three TCP packets.

The wire encoding of all those bit arrays was:
  [varint-min-numbits] - Shortest bit array length
  [varint-array-size]  - Number of bit arrays.
          [varint-num].... - Number of entries in array N (x varint-array-size)
  [packed-bit-arrays...]

  Last byte was padded with zeros.
  See: https://github.com/rustyrussell/bitcoin-iblt/blob/master/wire_encode.cpp#L12

I would also avoid the nonce to save recalculating for each node, and
instead define an id as:

        [<64-bit-short-id>][txid]

Since you only ever send as many bits as needed to distinguish, this only
makes a difference if there actually are collisions.

As Peter R points out, we could later enhance receiver to brute force
collisions (you could speed that by sending a XOR of all the txids, but
really if there are more than a few collisions, give up).

And a prototype could just always send 64-bit ids to start.

Cheers,
Rusty.