Date: Fri, 20 Oct 2023 22:01:40 +0000
To: Peter Todd <pete@petertodd.org>
From: Fabian <fjahr@protonmail.com>
Message-ID: <aAzprT3_Jlpgb6Z_sFOiC1q9KMmB9mCTaCk19hO8oe0vA1Z__kQhzprdDZblXGvR2_xTaUYk67RNFGxJcnm5QkAmi_PE8d51E80z077FpoM=@protonmail.com>
In-Reply-To: <ZTK6JINSo6WyvJL0@petertodd.org>
References: <kxXtwQMByYbMavS5P9a2tAUd8wz0yTUifost_txwTiQfNKTBtgdepLmAyV4XN6m4wY74cdZLX4EtsiEJ-jpZsnSxPIrCAN5wK8eK8xx1WGw=@protonmail.com>
 <ZTK6JINSo6WyvJL0@petertodd.org>
Feedback-ID: 5067558:user:proton
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Breaking change in calculation of
	hash_serialized_2
Precedence: list

Hi Peter,

to my knowledge, this was never considered as an option previously (James c=
orrect me if I am wrong). At least I couldn't find any reference to that in=
 the original proposal [1] and I can not remember it being discussed since =
I have followed the project more closely (ca. 2020).

Here are the reasons that I can think of why that might be the case:

- If the serialization and hashing of the UTXO set works as intended, that =
hash should be working just as well as the flat file hash and hash_serializ=
ed_2 certainly was assumed to be robust since it has been around for a very=
 long time. So it may simply have been viewed as additional overhead.
- We may want to optimize the serialization of data to file further, adding=
 compression, etc. to have smaller files that result in the same UTXO set w=
ithout having to change the chainparams committing to that UTXO set or pote=
ntially having multiple file hashes for the same block.
- We may want to introduce other file hashing strategies instead that are m=
ore optimized for P2P sharing of the UTXO snapshots. P2P sharing the UTXO s=
et has always been part of the idea of assumeutxo but so far it hasn't been=
 explored very deeply. For more on this see the conversation on IRC that st=
arted in the meeting yesterday between sipa, aj et al [2][3].

Cheers,
Fabian

[1] https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/propos=
al
[2] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-19#976439;
[3] https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-10-20#976636;

------- Original Message -------
On Friday, October 20th, 2023 at 7:34 PM, Peter Todd <pete@petertodd.org> w=
rote:


> On Fri, Oct 20, 2023 at 05:19:19PM +0000, Fabian via bitcoin-dev wrote:
>=20
> > Hello list,
> >=20
> > on Wednesday I found a potential malleability issue in the UTXO set dum=
p files
> > generated for and used by assumeutxo [1]. On Thursday morning theStack =
had
> > found the cause of the issue [2]: A bug in the serialization of UTXOs f=
or the
> > calculation of hash_serialized_2. This is the value used by Bitcoin Cor=
e to
> > check if the UTXO set loaded from a dump file matches what is expected.=
 The
> > value of hash_serialized_2 expected for a particular block is hardcoded=
 into
> > the chainparams of each chain.
>=20
>=20
> <snip>
>=20
> > [1] https://github.com/bitcoin/bitcoin/issues/28675
> > [2] https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-177038=
9468[3] https://github.com/bitcoin/bitcoin/pull/28685
>=20
>=20
> James made the following comment on the above issue:
>=20
> > Wow, good find @fjahr et al. I wonder if there's any value in committin=
g to a
> > sha256sum of the snapshot file itself in the source code as a
> > belt-and-suspenders remediation for issues like this.
>=20
>=20
> Why isn't the sha256 hash of the snapshot file itself the canonical hash?
> That would obviously eliminate any malleability issues. gettxoutsetinfo a=
lready
> has to walk the entire UTXO set to calculate the hash. Making it simply
> generate the actual contents of the dump file and calculate the hash of i=
t is
> the obvious way to implement this.
>=20
> --
> https://petertodd.org 'peter'[:-1]@petertodd.org