1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
|
Return-Path: <fjahr@protonmail.com>
Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133])
by lists.linuxfoundation.org (Postfix) with ESMTP id 2B2BAC0032
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 20 Oct 2023 22:02:07 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
by smtp2.osuosl.org (Postfix) with ESMTP id 026D34344B
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 20 Oct 2023 22:02:07 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 026D34344B
Authentication-Results: smtp2.osuosl.org;
dkim=pass (2048-bit key) header.d=protonmail.com header.i=@protonmail.com
header.a=rsa-sha256 header.s=protonmail3 header.b=Ws2VTzOH
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: -2.799
X-Spam-Level:
X-Spam-Status: No, score=-2.799 tagged_above=-999 required=5
tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001]
autolearn=ham autolearn_force=no
Received: from smtp2.osuosl.org ([127.0.0.1])
by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id OF3XBNSnxqvV
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 20 Oct 2023 22:02:02 +0000 (UTC)
Received: from mail-0301.mail-europe.com (mail-0301.mail-europe.com
[188.165.51.139])
by smtp2.osuosl.org (Postfix) with ESMTPS id 367F1404DA
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 20 Oct 2023 22:02:02 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 367F1404DA
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com;
s=protonmail3; t=1697839314; x=1698098514;
bh=yGpQoqq+qzDfv7S3hlVi41niXPvgNL9i+5e32xlxctQ=;
h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References:
Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID:
Message-ID:BIMI-Selector;
b=Ws2VTzOHAyyTONimWYGSAG/j4Hpdv4vRGeAw39t6w5w9TJ3ccePxBn0snbFD7R6TY
VKBgRnhWpFcR0qObs0hcTr9koJ9gfcfP6tslh9Xeynfh4na2Vj2DX1pY6m2SgRrGkS
lYUlx5DgpmHVX27vQoDX1fBXJQ2eY93K5n6QAyXZovSgWDtMzarv9HwjkuAam7sl0Y
aRlC6g1nH5i0W7G8GhHUjz9Zb/NQdueGKBSKBDNh9q0Xagii0MfVHrqaMcmP+4EuxS
q7OLWuSzDBDJJhhPOSoYRCDrVIsDeIX3nQOxlXFTujV/WeVrsvJ7ecUlF5nER4SFn2
eCZPLJGLdRKyw==
Date: Fri, 20 Oct 2023 22:01:40 +0000
To: Peter Todd <pete@petertodd.org>
From: Fabian <fjahr@protonmail.com>
Message-ID: <aAzprT3_Jlpgb6Z_sFOiC1q9KMmB9mCTaCk19hO8oe0vA1Z__kQhzprdDZblXGvR2_xTaUYk67RNFGxJcnm5QkAmi_PE8d51E80z077FpoM=@protonmail.com>
In-Reply-To: <ZTK6JINSo6WyvJL0@petertodd.org>
References: <kxXtwQMByYbMavS5P9a2tAUd8wz0yTUifost_txwTiQfNKTBtgdepLmAyV4XN6m4wY74cdZLX4EtsiEJ-jpZsnSxPIrCAN5wK8eK8xx1WGw=@protonmail.com>
<ZTK6JINSo6WyvJL0@petertodd.org>
Feedback-ID: 5067558:user:proton
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Mailman-Approved-At: Sat, 21 Oct 2023 12:04:52 +0000
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Breaking change in calculation of
hash_serialized_2
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Oct 2023 22:02:07 -0000
Hi Peter,
to my knowledge, this was never considered as an option previously (James c=
orrect me if I am wrong). At least I couldn't find any reference to that in=
the original proposal [1] and I can not remember it being discussed since =
I have followed the project more closely (ca. 2020).
Here are the reasons that I can think of why that might be the case:
- If the serialization and hashing of the UTXO set works as intended, that =
hash should be working just as well as the flat file hash and hash_serializ=
ed_2 certainly was assumed to be robust since it has been around for a very=
long time. So it may simply have been viewed as additional overhead.
- We may want to optimize the serialization of data to file further, adding=
compression, etc. to have smaller files that result in the same UTXO set w=
ithout having to change the chainparams committing to that UTXO set or pote=
ntially having multiple file hashes for the same block.
- We may want to introduce other file hashing strategies instead that are m=
ore optimized for P2P sharing of the UTXO snapshots. P2P sharing the UTXO s=
et has always been part of the idea of assumeutxo but so far it hasn't been=
explored very deeply. For more on this see the conversation on IRC that st=
arted in the meeting yesterday between sipa, aj et al [2][3].
Cheers,
Fabian
[1] https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/propos=
al
[2] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-19#976439;
[3] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-20#976636;
------- Original Message -------
On Friday, October 20th, 2023 at 7:34 PM, Peter Todd <pete@petertodd.org> w=
rote:
> On Fri, Oct 20, 2023 at 05:19:19PM +0000, Fabian via bitcoin-dev wrote:
>=20
> > Hello list,
> >=20
> > on Wednesday I found a potential malleability issue in the UTXO set dum=
p files
> > generated for and used by assumeutxo [1]. On Thursday morning theStack =
had
> > found the cause of the issue [2]: A bug in the serialization of UTXOs f=
or the
> > calculation of hash_serialized_2. This is the value used by Bitcoin Cor=
e to
> > check if the UTXO set loaded from a dump file matches what is expected.=
The
> > value of hash_serialized_2 expected for a particular block is hardcoded=
into
> > the chainparams of each chain.
>=20
>=20
> <snip>
>=20
> > [1] https://github.com/bitcoin/bitcoin/issues/28675
> > [2] https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-177038=
9468[3] https://github.com/bitcoin/bitcoin/pull/28685
>=20
>=20
> James made the following comment on the above issue:
>=20
> > Wow, good find @fjahr et al. I wonder if there's any value in committin=
g to a
> > sha256sum of the snapshot file itself in the source code as a
> > belt-and-suspenders remediation for issues like this.
>=20
>=20
> Why isn't the sha256 hash of the snapshot file itself the canonical hash?
> That would obviously eliminate any malleability issues. gettxoutsetinfo a=
lready
> has to walk the entire UTXO set to calculate the hash. Making it simply
> generate the actual contents of the dump file and calculate the hash of i=
t is
> the obvious way to implement this.
>=20
> --
> https://petertodd.org 'peter'[:-1]@petertodd.org
|
