Delivery-date: Fri, 28 Jun 2024 11:49:47 -0700 Received: from mail-yw1-f189.google.com ([209.85.128.189]) by mail.fairlystable.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sNGfK-0006Xg-Gk for bitcoindev@gnusha.org; Fri, 28 Jun 2024 11:49:47 -0700 Received: by mail-yw1-f189.google.com with SMTP id 00721157ae682-6452f8f2942sf17296727b3.1 for ; Fri, 28 Jun 2024 11:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1719600580; x=1720205380; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:sender:from :to:cc:subject:date:message-id:reply-to; bh=CCiDRfxhGTHmOkpko15ZOpnCAVP6G+aYD2PdN7iio/c=; b=AvDLLlP376m29PEM4Hf6IoEq9tXUT3qER2556T7fa8DOdxXDQIvyGIYGeSV6dF2Q92 YhZk1DrxgO95wqn+OdG2EWRoanDgYQLrqhWddjnkPrcLQYQ46o857L7e4H9qzrVqG6sg Num0TsQ9LvAPTtA8uWUMOnl2XlXdeJZ2hWq8FG06XlEW0NwKpIK1cXJ2YtTvUmxrzZT3 6nvBhYBZ9Px79lAdHPgXc76MUEgAziDcwuBfdHVHZYRzehqzziU6hTKuWdPxwpbb4t75 dXKpIdjJjpV6cJoe+DWuA+aH9+bcWaRJkOJcrRrrn2wfNLM4+moY8ZprSJC3t4q2NovU wUrg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups-com.20230601.gappssmtp.com; s=20230601; t=1719600580; x=1720205380; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=CCiDRfxhGTHmOkpko15ZOpnCAVP6G+aYD2PdN7iio/c=; b=edbep6Wc69PEidLa6Hm90pcDIIm3Ct2Rb3PtHv9ZVqn17KNSCiGotzwlEjUhLhpoRJ KuFQVrM57Sj8UoSF96dfOHWPrSF0+A32Tck/yA+bKgbTCFK5w20q0DYq/A4j5TUdtdbW bLg+Y9SYDO+2EE2ABcBK/kFpnOHEYctAgnWY+i0QkQJWWYnwD9zFOvp+zetyumOTQ62i 9xC+tB52p4uIez66sRerGCN55Y+yjv9ZeiFo9pDabopc45jw4eEoclI0h0a/PjTx/WKJ 3ZtZAnRIu+8gk7j9klGm55aFtCHoJVXhiOD728ZEC4eK+5exr+cDHU/662lJJWceN2Tf H64Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719600580; x=1720205380; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=CCiDRfxhGTHmOkpko15ZOpnCAVP6G+aYD2PdN7iio/c=; b=KD5aBd2bweh2LEMvhnkvmKBZYWxZTSowMUA3MAjUo2DTLVjG8QO2UwN/IuDxayyhTY I+Qgzvd6CSB2UN8VMDOr2GRigIttt0M3Hg81LhoLSpmq0JWzK3/UcrIjn+JCOxgQgOjz wxDLK3aE9OpP+jKcb8cwVlzwFmTj0esRc0trn8ZHaX2DpBtOLMiroWz0ydPwOdPedadk 5m2kFHAoyFz6+x+hX61Nuk5gTiBrn30vJ6i4wYE3YtMifrZ/amaTeVxSeZDR54GgUeqi nw2YOxC+C9JfUr1XtbhccNfIx2vLUbs05vSGK6C3MS8TOtICv9n9ff4ZlbMsa09pK1mN FnPA== Sender: bitcoindev@googlegroups.com X-Forwarded-Encrypted: i=1; AJvYcCVh+8D5G1ih+sNB4Oc/mXxOKLknCvxu31b6fPHa0pUPM2ll28l34ZbrECwb6PFddw2szzjKSOoMYWTqMeTSTmG8MDwwvu4= X-Gm-Message-State: AOJu0YzQojYwgbnwhI8AYbZu8nUw3oOoqO6l4G2NkDtF9zhNf447MOcI 6Fn6Qs40kk0Fu0SgfQ/TfwGnE/S2/GpxGiRlLhJDAX4BQUHilAKx X-Google-Smtp-Source: AGHT+IGushFeZy08e+f2shaNUZW0+VsjyQ1V96L+9DnIxVFBHMQ3AbLmUktBkS/ByoFCSRRvA2qbOA== X-Received: by 2002:a25:ad98:0:b0:e03:5bf4:d508 with SMTP id 3f1490d57ef6-e035bf4d6c3mr2669928276.27.1719600579748; Fri, 28 Jun 2024 11:49:39 -0700 (PDT) X-BeenThere: bitcoindev@googlegroups.com Received: by 2002:a05:6902:150d:b0:dff:aa9:b9ad with SMTP id 3f1490d57ef6-e035a580299ls1374913276.2.-pod-prod-08-us; Fri, 28 Jun 2024 11:49:38 -0700 (PDT) X-Received: by 2002:a05:6902:2b84:b0:e02:c739:367 with SMTP id 3f1490d57ef6-e0303fe4816mr965886276.13.1719600578361; Fri, 28 Jun 2024 11:49:38 -0700 (PDT) Received: by 2002:a0d:e6cd:0:b0:64a:6fb4:b878 with SMTP id 00721157ae682-64a84019cd1ms7b3; Fri, 28 Jun 2024 10:14:05 -0700 (PDT) X-Received: by 2002:a05:6902:18c6:b0:e03:510d:3b6e with SMTP id 3f1490d57ef6-e03510d3f6amr312250276.3.1719594844324; Fri, 28 Jun 2024 10:14:04 -0700 (PDT) Date: Fri, 28 Jun 2024 10:14:04 -0700 (PDT) From: Eric Voskuil To: Bitcoin Development Mailing List Message-Id: <9a4c4151-36ed-425a-a535-aa2837919a04n@googlegroups.com> In-Reply-To: References: <72e83c31-408f-4c13-bff5-bf0789302e23n@googlegroups.com> <5b0331a5-4e94-465d-a51d-02166e2c1937n@googlegroups.com> Subject: Re: [bitcoindev] Re: Great Consensus Cleanup Revival MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_317950_1433319534.1719594844096" X-Original-Sender: eric@voskuil.org Precedence: list Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com List-ID: X-Google-Group-Id: 786775582512 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Score: -0.7 (/) ------=_Part_317950_1433319534.1719594844096 Content-Type: multipart/alternative; boundary="----=_Part_317951_1488383171.1719594844096" ------=_Part_317951_1488383171.1719594844096 Content-Type: text/plain; charset="UTF-8" >> It is not clear to me how determining the coinbase size can be done at an earlier stage of validation than detection of the non-null coinbase. > My point wasn't about checking the coinbase size, it was about being able to cache the hash of a (non-malleated) invalid block as permanently invalid to avoid re-downloading and re-validating it. This I understood, but I think you misunderstood me. Your point was specifically that, "it would let node implementations cache block failures at an earlier stage of validation." Since you have not addressed that aspect I assume you agree with my assertion above that the proposed rule does not actually achieve this. Regarding the question of checking coinbase size, the issue is of detecting (or preventing) hashes mallied via the 64 byte tx technique. A rule against 64 byte txs would allow this determination by checking the coinbase alone. If the coinbase is 64 bytes the block is invalid, if it is not the block hash cannot have been mallied (all txs must have been 64 bytes, see previous reference). In that case if the block is invalid the invalidity can be cached. But block invalidity cannot actually be cached until the block is fully validated. A rule to prohibit *all* 64 byte txs is counterproductive as it only adds additional checks on typically thousands of txs per block, serving no purpose. >> It seems to me that introducing an arbitrary tx size validity may create more potential implementation bugs than it resolves. > The potential for implementation bugs is a fair point to raise, but in this case i don't think it's a big concern. Verifying no transaction in a block is 64 bytes is as simple a check as you can get. You appear to be making the assumption that the check is performed after the block is fully parsed (contrary to your "earlier" criterion above). The only way to determine the tx sizes is to parse each tx for witness marker, input count, output count, input script sizes, output script sizes, witness sizes, and skipping over the header, several constants, and associated buffers. Doing this "early" to detect malleation is an extraordinarily complex and costly process. On the other hand, as I pointed out, a rational implementation would only do this early check for the coinbase. Yet even determining the size of the coinbase is significantly more complex and costly than checking its first input point against null. That check (which is already necessary for validation) resolves the malleation question, can be performed on the raw unparsed block buffer by simply skipping header, version, reading input count and witness marker as necessary, offsetting to the 36 byte point buffer, and performing a byte comparison against [0000000000000000000000000000000000000000000000000000000000000000ffffffff]. This is: (1) earlier (2) faster (3) simpler (4) already consensus >> And certainly anyone implementing such a verifier must know many intricacies of the protocol. > They need to know some, but i don't think it's reasonable to expect them to realize the merkle tree construction is such that an inner node may be confused with a 64 bytes transaction. A protocol developer needs to understand that the hash of an invalid block cannot be cached unless at least the coinbase has been restricted in size (under the proposal) -or- that the coinbase is a null point (presently or under the proposal). In the latter case the check is already performed in validation, so there is no way a block would presently be cached as invalid without checking it. The proposal adds a redundant check, even if limited to just the coinbase. [He must also understand the second type of malleability, discussed below.] If this proposed rule was to activate we would implement it in a late stage tx.check, after txs/blocks had been fully deserialized. We would not check it an all in the case where the block is under checkpoint or milestone ("assume valid"). In this case we would retain the early null point malleation check (along with the hash duplication malleation check) that we presently have, would validate tx commitments, and commit the block. In other words, the proposal adds unnecessary late stage checks only. Implementing it otherwise would just add complexity and hurt performance. >> I do not see this. I see a very ugly perpetual seam which will likely result in unexpected complexities over time. > What makes you think making 64 bytes transactions invalid could result in unexpected complexities? And why do you think it's likely? As described above, it's later, slower, more complex, unnecessarily broad, and a consensus change. Beyond that it creates an arbitrary size limit - not a lower or upper bound, but a slice out of the domain. Discontinuities are inherent complexities in computing. The "unexpected" part speaks for itself. >> This does not produce unmalleable block hashes. Duplicate tx hash malleation remains in either case, to the same effect. Without a resolution to both issues this is an empty promise. > Duplicate txids have been invalid since 2012 (CVE-2012-2459). I think again here you may have misunderstood me. I was not making a point pertaining to BIP30. I was referring to the other form of block hash malleability, which results from duplicating sets of trailing txs in a single block (see previous reference). This malleation vector remains, even with invalid 64 byte txs. As I pointed out, this has the "same effect" as the 64 byte tx issue. Merkle hashing the set of txs is insufficient to determine identity. In one case the coinbase must be checked (null point or size) and in the other case the set of tx hashes must be checked for trailing duplicated sets. [Core performs this second check within the Merkle hashing algorithm (with far more comparisons than necessary), though this can be performed earlier and independently to avoid any hashing in the malleation case.] I would also point out in the interest of correctness that Core reverted its BIP30 soft fork implementation as a consequence of the BIP90 hard fork, following and requiring the BIP34 soft fork that presumably precluded it but didn't, so it is no longer the case that duplicate tx hashes are invalid in implementation. As you have proposed in this rollup, this requires fixing again. > If 64 bytes transactions are also made invalid, this would make it impossible for two valid blocks to have the same hash. Aside from the BIP30/34/90 issue addressed above, it is already "impossible" (cannot be stronger than computationally infeasible) for two *valid* blocks to have the same hash. The proposal does not enable that objective, it is already the case. No malleated block is a valid block. The proposal aims only to make it earlier or easier or faster to check for block hash malleation. And as I've pointed out above, it doesn't achieve those objectives. Possibly the perception that this would be the case is a consequence of implementation details, but as I have shown above, it is not in fact the case. Given either type of malleation, the malleated block can be determined to be invalid by a context free check. But this knowledge cannot ever be cached against the block hash, since the same hash may be valid. Invalidity can only be cached once a non-mallied block is validated and determined to be invalid. Block hash malleations are and will remain invalid blocks with or without the proposal, and it will continue to be necessary to avoid caching invalid against the malleation. As you said: > it was about being able to cache the hash of a (non-malleated) invalid block as permanently invalid to avoid re-downloading and re-validating it. This is already the case, and requires validating the full non-malleated block. Adding a redundant invalidity check doesn't improve this in any way. Best, Eric -- You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bitcoindev/9a4c4151-36ed-425a-a535-aa2837919a04n%40googlegroups.com. ------=_Part_317951_1488383171.1719594844096 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable >> It is not clear to me how determining the coinbase size can be don= e at an earlier stage of validation than detection of the non-null coinbase= .
> My point wasn't about checking the coinbase size, it was about = being able to cache the hash of a (non-malleated) invalid block as permanen= tly invalid to avoid re-downloading and re-validating it.

This I= understood, but I think you misunderstood me. Your point was specifically = that, "it would let node implementations cache block failures at an earlier= stage of validation." Since you have not addressed that aspect I assume yo= u agree with my assertion above that the proposed rule does not actually ac= hieve this.

Regarding the question of checking coinbase size, th= e issue is of detecting (or preventing) hashes mallied via the 64 byte tx t= echnique. A rule against 64 byte txs would allow this determination by chec= king the coinbase alone. If the coinbase is 64 bytes the block is invalid, = if it is not the block hash cannot have been mallied (all txs must have bee= n 64 bytes, see previous reference).

In that case if the block i= s invalid the invalidity can be cached. But block invalidity cannot actuall= y be cached until the block is fully validated. A rule to prohibit *all* 64= byte txs is counterproductive as it only adds additional checks on typical= ly thousands of txs per block, serving no purpose.

>> It s= eems to me that introducing an arbitrary tx size validity may create more p= otential implementation bugs than it resolves.
> The potential for = implementation bugs is a fair point to raise, but in this case i don't thin= k it's a big concern. Verifying no transaction in a block is 64 bytes is as= simple a check as you can get.

You appear to be making the assu= mption that the check is performed after the block is fully parsed (contrar= y to your "earlier" criterion above). The only way to determine the tx size= s is to parse each tx for witness marker, input count, output count, input = script sizes, output script sizes, witness sizes, and skipping over the hea= der, several constants, and associated buffers. Doing this "early" to detec= t malleation is an extraordinarily complex and costly process. On the other= hand, as I pointed out, a rational implementation would only do this early= check for the coinbase.

Yet even determining the size of the co= inbase is significantly more complex and costly than checking its first inp= ut point against null. That check (which is already necessary for validatio= n) resolves the malleation question, can be performed on the raw unparsed b= lock buffer by simply skipping header, version, reading input count and wit= ness marker as necessary, offsetting to the 36 byte point buffer, and perfo= rming a byte comparison against [000000000000000000000000000000000000000000= 0000000000000000000000ffffffff].

This is:

(1) earlier=
(2) faster
(3) simpler
(4) already consensus

>= ;> And certainly anyone implementing such a verifier must know many intr= icacies of the protocol.
> They need to know some, but i don't thin= k it's reasonable to expect them to realize the merkle tree construction is= such that an inner node may be confused with a 64 bytes transaction.
=
A protocol developer needs to understand that the hash of an invalid = block cannot be cached unless at least the coinbase has been restricted in = size (under the proposal) -or- that the coinbase is a null point (presently= or under the proposal). In the latter case the check is already performed = in validation, so there is no way a block would presently be cached as inva= lid without checking it. The proposal adds a redundant check, even if limit= ed to just the coinbase. [He must also understand the second type of mallea= bility, discussed below.]

If this proposed rule was to activate = we would implement it in a late stage tx.check, after txs/blocks had been f= ully deserialized. We would not check it an all in the case where the block= is under checkpoint or milestone ("assume valid"). In this case we would r= etain the early null point malleation check (along with the hash duplicatio= n malleation check) that we presently have, would validate tx commitments, = and commit the block. In other words, the proposal adds unnecessary late st= age checks only. Implementing it otherwise would just add complexity and hu= rt performance.

>> I do not see this. I see a very ugly pe= rpetual seam which will likely result in unexpected complexities over time.=
> What makes you think making 64 bytes transactions invalid could = result in unexpected complexities? And why do you think it's likely?
<= br />As described above, it's later, slower, more complex, unnecessarily br= oad, and a consensus change. Beyond that it creates an arbitrary size limit= - not a lower or upper bound, but a slice out of the domain. Discontinuiti= es are inherent complexities in computing. The "unexpected" part speaks for= itself.

>> This does not produce unmalleable block hashes= . Duplicate tx hash malleation remains in either case, to the same effect. = Without a resolution to both issues this is an empty promise.
> Dup= licate txids have been invalid since 2012 (CVE-2012-2459).

I thi= nk again here you may have misunderstood me. I was not making a point perta= ining to BIP30. I was referring to the other form of block hash malleabilit= y, which results from duplicating sets of trailing txs in a single block (s= ee previous reference). This malleation vector remains, even with invalid 6= 4 byte txs. As I pointed out, this has the "same effect" as the 64 byte tx = issue. Merkle hashing the set of txs is insufficient to determine identity.= In one case the coinbase must be checked (null point or size) and in the o= ther case the set of tx hashes must be checked for trailing duplicated sets= . [Core performs this second check within the Merkle hashing algorithm (wit= h far more comparisons than necessary), though this can be performed earlie= r and independently to avoid any hashing in the malleation case.]

I would also point out in the interest of correctness that Core reverted = its BIP30 soft fork implementation as a consequence of the BIP90 hard fork,= following and requiring the BIP34 soft fork that presumably precluded it b= ut didn't, so it is no longer the case that duplicate tx hashes are invalid= in implementation. As you have proposed in this rollup, this requires fixi= ng again.

> If 64 bytes transactions are also made invalid, t= his would make it impossible for two valid blocks to have the same hash.
Aside from the BIP30/34/90 issue addressed above, it is already "i= mpossible" (cannot be stronger than computationally infeasible) for two *va= lid* blocks to have the same hash. The proposal does not enable that object= ive, it is already the case. No malleated block is a valid block.

The proposal aims only to make it earlier or easier or faster to check fo= r block hash malleation. And as I've pointed out above, it doesn't achieve = those objectives. Possibly the perception that this would be the case is a = consequence of implementation details, but as I have shown above, it is not= in fact the case.

Given either type of malleation, the malleate= d block can be determined to be invalid by a context free check. But this k= nowledge cannot ever be cached against the block hash, since the same hash = may be valid. Invalidity can only be cached once a non-mallied block is val= idated and determined to be invalid. Block hash malleations are and will re= main invalid blocks with or without the proposal, and it will continue to b= e necessary to avoid caching invalid against the malleation. As you said:
> it was about being able to cache the hash of a (non-malleate= d) invalid block as permanently invalid to avoid re-downloading and re-vali= dating it.

This is already the case, and requires validating the= full non-malleated block. Adding a redundant invalidity check doesn't impr= ove this in any way.

Best,
Eric

--
You received this message because you are subscribed to the Google Groups &= quot;Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoind= ev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg= id/bitcoindev/9a4c4151-36ed-425a-a535-aa2837919a04n%40googlegroups.com.=
------=_Part_317951_1488383171.1719594844096-- ------=_Part_317950_1433319534.1719594844096--