1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
|
Return-Path: <tomz@freedommail.ch>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 3CEB2411
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 9 May 2016 09:35:43 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from mx-out02.mykolab.com (mx01.mykolab.com [95.128.36.1])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id E9AD317F
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 9 May 2016 09:35:41 +0000 (UTC)
X-Virus-Scanned: amavisd-new at kolabnow.com
X-Spam-Score: -2.9
X-Spam-Level:
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW
autolearn=ham version=3.3.1
Received: from mx03.mykolab.com (mx03.mykolab.com [10.20.7.101])
by mx-out02.mykolab.com (Postfix) with ESMTPS id 19E8160306;
Mon, 9 May 2016 11:35:34 +0200 (CEST)
From: Tom Zander <tomz@freedommail.ch>
To: Matt Corallo <lf-lists@mattcorallo.com>
Date: Mon, 09 May 2016 10:35:32 +0100
Message-ID: <86058327.pdmfHP132A@kiwi>
In-Reply-To: <572EB166.5070305@mattcorallo.com>
References: <5727D102.1020807@mattcorallo.com>
<CALOxbZv5GL5br=Z5idR-ACVzkBxS6JP_KgSr3JBuYVLgsej3eA@mail.gmail.com>
<572EB166.5070305@mattcorallo.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
X-Mailman-Approved-At: Mon, 09 May 2016 09:38:46 +0000
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Compact Block Relay BIP
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 09 May 2016 09:35:43 -0000
On Sunday, May 08, 2016 03:24:22 AM Matt Corallo wrote:
> >> ===Intended Protocol Flow===
> >
> > I'm not a fan of the solution that a CNode should keep state and talk to
> > its remote nodes differently while announcing new blocks.
> > Its too complicated and ultimately counter-productive.
> >
> > The problem is that an individual node needs to predict network behaviour
> > in advance. With the downside that if it guesses wrong that both nodes
> > end up paying for the wrong guess.
> > This is not a good way to design a p2p layer.
>
> Nodes don't need to predict much in advance, and the cost for predicting
> wrong is 0 if your peers receive blocks with a few hundred ms between
> them (as we should expect) and you haven't set the announce bit on more
> than a few peers (as the spec requires for this reason).
You misunderstand the networking effects.
The fact that your node is required to choose which one to set the announce
bit on implies that it needs to predict which node will have the best data in
the future.
It needs to predict which nodes will not start being incommunicado and it
requires them to predict all the things that are not possible to predict in a
network.
In networking it is even more true than in stocks; results of the past are no
guarantee for the future.
This means you are creating a fragile system. Your system will only work in
laboratory situations. It will fail spectacularly when the network or the
internet is under stress or some parts fall away.
Another problem with your solution is that nodes send a much larger amount of
unsolicited data to peers in the form of the thin-block compared to the normal
inv or header-first data.
Saying this is mitigated by only subscribing on this data from a small
subsection of nodes means you position yourself in a situation that I
displayed above. A tradeoff of fragile and fast. With no possible way to make
a node automatically decide on a good equilibrium.
> It seems I forgot to add a suggested peer-preforwarding-selection
> algorithm in the text, but the intended use-case is to set the bit on
> peers which recently provided you blocks faster than other peers, up to
> only one or three peers. This is both simple and should be incredibly
> effective.
Network autorepair systems have been researched for decades, no real solution
has as of yet appeared.
PHDs are written on the subject and you want to make this a design for Bitcoin
based on "[it] should be incredibly effective", I think you are underestimating
the subject matter you are dealing with.
> > I would suggest that a new block is announced to all nodes equally and
> > then
> > individual nodes can respond with a request of either a 'compact' or a
> > normal block.
> > This is much more in line with the current design as well.
> >
> > Detection if remote nodes support compact blocks, for the purpose of
> > requesting a compact-block, can be done either via a network-bit or just a
> > protocol version. Or something else entirely, if you have better
> > suggestions.
>
> In line with recent trends, neither service bits nor protocol versions
> are particularly well-suited for this purpose.
Am I to understand that you choose the solution based on the fact that service
bits are too expensive to extend? (if not, please respond to my previous
question actually answering the suggestion)
That sounds like a rather bad way of doing design. Maybe you can add a second
service bits field of message instead and then do the compact blocks correctly.
> >> Variable-length integers: bytes are a MSB base-128 encoding of the
> >> number.
> >> The high bit in each byte signifies whether another digit follows.
> >> [snip bitwise spec]
> >
> > I suggest just referring to UTF-8 which describes this just fine.
> > it is good practice to refer to existing specs when possible and not copy
> > the details.
>
> Hmm? There is no UTF anywhere in this protocol. Indeed this section
> needs to be rewritten, as indicated. I'd recommend you read the code
> until I update the section with better text if you're confused.
Wait, you didn't steal the variable length encoding from an existing standard
and you programmed a new one?
I strongly suggest you don't reinvent this kind of protocol level encodings
but instead steal from something like UTF8. Which has been around for decades.
Please base your standard on other standards where possible.
Look at UTF-8 on wikipedia, you may have "invented" the same encoding that IBM
published in 1992.
> >> ====Short transaction IDs====
> >> Short transaction IDs are used to represent a transaction without
> >> sending a full 256-bit hash. They are calculated by:
> >> # single-SHA256 hashing the block header with the nonce appended (in
> >> little-endian)
> >> # XORing each 8-byte chunk of the double-SHA256 transaction hash with
> >> each corresponding 8-byte chunk of the hash from the previous step
> >> # Adding each of the XORed 8-byte chunks together (in little-endian)
> >> iteratively to find the short transaction ID
> >
> > I don't think this is needed. Just use the first 8 bytes.
> > The reason to do xor-ing doesn't hold up and extra complexity is unneeded.
> > Especially since you mention some lines down;
> >
> >> The short transaction ID calculation is designed to take absolutely
> >> minimal processing time during block compaction to avoid introducing
> >> serious DoS vulnerabilities
>
> I'm confused as to what, specifically, you're proposing this be changed
> to.
Just the first (highest) 8 bytes of a sha256 hash.
The amount of collisions will not be less if you start xoring the rest.
The whole reason for doing this extra work is also irrelevant as a spam
protection.
--
Tom Zander
|
