1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
|
Return-Path: <Peter_R@gmx.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 35BAB8EE
for <bitcoin-dev@lists.linuxfoundation.org>;
Sun, 15 Nov 2015 01:02:46 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from mout.gmx.net (mout.gmx.net [212.227.17.22])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8D12390
for <bitcoin-dev@lists.linuxfoundation.org>;
Sun, 15 Nov 2015 01:02:44 +0000 (UTC)
Received: from [192.168.50.29] ([69.50.179.106]) by mail.gmx.com (mrgmx103)
with ESMTPSA (Nemesis) id 0LqQTl-1aaw1N3P5F-00e3Sr;
Sun, 15 Nov 2015 02:02:37 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
From: Peter R <peter_r@gmx.com>
In-Reply-To: <CAAS2fgRdK4bDr3x_y9UpdH234PQSfD7U539HBLA==+hLQJ_7Fw@mail.gmail.com>
Date: Sat, 14 Nov 2015 17:02:33 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <571D9B7F-077D-4B80-B577-1C18FF2ECF31@gmx.com>
References: <5631C363.5060705@neomailbox.net>
<201510290803.52734.luke@dashjr.org>
<5632DE33.7030600@bitcartel.com>
<CAAS2fgTga_vTfOKrFu_hEzXSfTfg9FRfJ6aL6ginuGFqnbm7=w@mail.gmail.com>
<3CB90C47-293E-4C18-A381-E5203483D68F@gmx.com>
<CAAS2fgRdK4bDr3x_y9UpdH234PQSfD7U539HBLA==+hLQJ_7Fw@mail.gmail.com>
To: Gregory Maxwell <gmaxwell@gmail.com>
X-Mailer: Apple Mail (2.2104)
Sender: Peter_R@gmx.com
X-Provags-ID: V03:K0:hJOP4A6k42YNTcRTX5IZ6HaS2pH4pUtL7PvuQOYXJ5bYf41cauH
3cslW3aN40fJ1WoNxrA1C4NqcTr35xZfxvPfLHZxtm3hI7hPYjvh2fprk6TGOXGvFlalQ6V
x58hKz0Bjg9Q37n+ZEhVPTxXqwpW044vrZAiUshw6HQYzF669+zvq4Pn+16WhoqVhZdvDvF
0bKxupFj0yLNpGS8ZZyxw==
X-UI-Out-Filterresults: notjunk:1;V01:K0:m5oPBZ0c7f0=:4gwraTYQLajA2hJDobBcFt
jLc3g2CnW9uBNW11Ie/7+cu9IcVuBF3tIGcNVyd6y7M/1PIPQOKUIerdnIkuuZZx+yPHBH57I
1m6XTJd7Zfnw/0lUigS+XI4CVcrxNlKPXVwGzb7FmkqM8A9NlWDT3p4UpiKGLMvvcZxia7baa
pZB7c5lQ5d1vN/hTHysOaHsoh1lBDLCw77Egkjlc/VP5eVyqG9KASN+XyNjNIRgWkziBJfNfE
CwZUmD7+8iFZlTFjC71furR7mcGvRciYGSqOeIFO1yh3BKDGZZTnBhuEWuz4D32UaAvd7iQLI
4BfmwpBqzI6qHRk2xyEdx2p691p9fexZPUzzyuM2Eg4C2qxTN1iVdoApBWttCtJ6bJ9Ni5UAy
/jk71wWk8lR6ZK9s9c0T5lOyt38ZxVWFm7y0WGprzxF02Pck/9erhlCg3vLiD+evwtLOir49i
y9U1eGwTLaM7FuB6hDoO+2OOYZUwbz7UOqd8SW6lDNs6RJla8dMt4im9Y1QzuAESMa4JFULaX
EFeEajmweZ6wCrqedduRVCdV2pvQJ12/bdpuGUNNvpWFJoTPpuVxLHsb/kryD8l5LClulzXt5
ouaq0VPUFeUnytN9StdKSR/JtT2leY51q5p3xdMahq2k1X6251MuPe2Vyrysu3sbMBKQQM8zV
NLytqbpTxTRydZ0XH5/foTvofbESVY4Ir6ihviNxjnB/ZnHABV9J+X0oxvitLxvwp63QtWWEW
h/dXyYLv/Sj6vGR4gIycgtyh3NMz7zEdK+FpfRDa8IZBJwkCz8GztXOuDO3jg5do6auYvXJXh
GbaFlMg
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM,
RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
telemaco <telemaco@neomailbox.net>
Subject: Re: [bitcoin-dev] [patch] Switching Bitcoin Core to sqlite db
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sun, 15 Nov 2015 01:02:46 -0000
Hi Greg,
Like you said, the issue with using more than one database technology is =
not that one node would prove that Block X is valid while the another =
node proves that Block X is NOT valid. Instead, the problem is that one =
node might say =E2=80=9Cvalid=E2=80=9D while the other node says =E2=80=9C=
I don=E2=80=99t know.=E2=80=9D
It is often said that what caused the Level DB fork was that the old =
version determined that the triggering block was invalid while the new =
version determined the opposite. This interpretation of the fork event =
has lead to the =E2=80=9Cbug-for-bug=E2=80=9D-compatibility fear =
regarding multiple implementations of the protocol (or multiple database =
technologies). In reality, this fear is largely unfounded. =20
The old version ran out of LDB locks and couldn=E2=80=99t execute =
properly. If the software was written with the philosophy that tracking =
consensus was more important than bug-for-bug compatibility, it would =
have returned an exception rather than =E2=80=9Cinvalid.=E2=80=9D At =
that point, the software would have checked what decision the rest of =
the network came to about the block in question. My node would have =
forked to the longest chain, automatically assuming that the =
questionable block was valid; your node may have made a different =
decision (it=E2=80=99s a question of ideology at that point). =20
A group of us have been exploring this =E2=80=9Cmeta-cognition=E2=80=9D =
idea with Bitcoin Unlimited. For example, Bitcoin Unlimited can be =
(optionally) made to automatically fork to the longest chain if it =
=E2=80=9Cgets stuck=E2=80=9D and can neither prove that a block is valid =
nor that the block is invalid. Similarly, Bitcoin Unlimited can be =
(optionally) made to automatically fork to a chain that contains a block =
marginally bigger than its block size limit=E2=80=94once that block is =
buried sufficiently in what has emerged as the longest persistent chain.=20=
Thinking from this perspective might go along way towards decentralizing =
development, the emergence of multiple competing implementations of the =
protocol, and true =E2=80=9Cbottom up=E2=80=9D governance. =20
Best regards,
Peter
> On Oct 29, 2015, at 9:28 PM, Gregory Maxwell <gmaxwell@gmail.com> =
wrote:
>=20
> On Fri, Oct 30, 2015 at 4:04 AM, Peter R <peter_r@gmx.com> wrote:
>> Can you give a specific example of how nodes that used different =
database technologies might determine different answers to whether a =
given transaction is valid or invalid? I=E2=80=99m not a database =
expert, but to me it would seem that if all the unspent outputs can be =
found in the database, and if the relevant information about each output =
can be retrieved without corruption, then that=E2=80=99s all that really =
matters as far as the database is concerned.
>=20
> If you add to those set of assumptions the handling of write ordering
> is the same (e.g. multiple updates in an change end up with the same
> entry surviving) and read/write interleave returning the same results
> then it wouldn't.
>=20
> But databases sometimes have errors which cause them to fail to return
> records, or to return stale data. And if those exist consistency must
> be maintained; and "fixing" the bug can cause a divergence in
> consensus state that could open users up to theft.
>=20
> Case in point, prior to leveldb's use in Bitcoin Core it had a bug
> that, under rare conditions, could cause it to consistently return not
> found on records that were really there (I'm running from memory so I
> don't recall the specific cause). Leveldb fixed this serious bug in a
> minor update. But deploying a fix like this in an uncontrolled manner
> in the bitcoin network would potentially cause a fork in the consensus
> state; so any such fix would need to be rolled out in an orderly
> manner.
>=20
>> I=E2=80=99d like a concrete example to help me understand why more =
than one implementation of something like the UTXO database would be =
unreasonable.
>=20
> It's not unreasonable, but great care is required around the =
specifics.
>=20
> Bitcoin consensus implements a mathematical function that defines the
> operation of the system and above all else all systems must agree (or
> else the state can diverge and permit double-spends); if you could
> prove that a component behaves identically under all inputs to another
> function then it can be replaced without concern but this is something
> that cannot be done generally for all software, and proving
> equivalence even in special cases it is an open area of research. The
> case where the software itself is identical or nearly so is much
> easier to gain confidence in the equivalence of a change through
> testing and review.
>=20
> With that cost in mind one must then consider the other side of the
> equation-- utxo database is an opaque compressed representation,
> several of the posts here have been about desirability of blockchain
> analysis interfaces, and I agree they're sometimes desirable but
> access to the consensus utxo database is not helpful for that.
> Similarly, other things suggested are so phenomenally slow that it's
> unlikely that a node would catch up and stay synced even on powerful
> hardware. Regardless, in Bitcoin core the storage engine for this is
> fully internally abstracted and so it is relatively straight forward
> for someone to drop something else in to experiment with; whatever the
> motivation.
>=20
> I think people are falling into a trap of thinking "It's a <database>,
> I know a <black box> for that!"; but the application and needs are
> very specialized here; no less than, say-- the table of pre-computed
> EC points used for signing in the ECDSA application. It just so
> happens that on the back of the very bitcoin specific cryptographic
> consensus algorithim there was a slot where a pre-existing high
> performance key-value store fit; and so we're using one and saving
> ourselves some effort. If, in the future, Bitcoin Core adopts a
> merkelized commitment for the UTXO it would probably need to stop
> using any off-the-shelf key value store entirely, in order to avoid a
> 20+ fold write inflation from updating hash tree paths (And Bram Cohen
> has been working on just such a thing, in fact).
|
