path: root/91/efa7e7cca3fb4bcbff5ab5bca6282677f4cff7

Return-Path: <laolu32@gmail.com>
Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133])
 by lists.linuxfoundation.org (Postfix) with ESMTP id 93D61C000E
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Thu,  4 Nov 2021 22:01:27 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by smtp2.osuosl.org (Postfix) with ESMTP id 7BC8F4014B
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Thu,  4 Nov 2021 22:01:27 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: 1.527
X-Spam-Level: *
X-Spam-Status: No, score=1.527 tagged_above=-999 required=5
 tests=[BAYES_05=-0.5, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
 DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.975,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001]
 autolearn=no autolearn_force=no
Authentication-Results: smtp2.osuosl.org (amavisd-new);
 dkim=pass (2048-bit key) header.d=gmail.com
Received: from smtp2.osuosl.org ([127.0.0.1])
 by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id m2fmfla-0Qce
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Thu,  4 Nov 2021 22:01:26 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.8.0
Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com
 [IPv6:2607:f8b0:4864:20::b2f])
 by smtp2.osuosl.org (Postfix) with ESMTPS id DF65540114
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Thu,  4 Nov 2021 22:01:25 +0000 (UTC)
Received: by mail-yb1-xb2f.google.com with SMTP id q74so17890264ybq.11
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Thu, 04 Nov 2021 15:01:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:from:date:message-id:subject:to;
 bh=WUN/q9027srP+Vv1BvPwzWs4wm1VZ7aY3X40i+c/rGY=;
 b=hIA1bVEhWw0QPPci4h7KYlhpp+cjk+BSUAxaZG2Vo6aXz09x2XTRV+ogJ2GBZ6Fayo
 OZqu4FoOZRYrbKcs3klsoCOio76bXUMTeTBrvaxl/FKsN1ArkipxSo1dIVSPCfilweHf
 PyjNaQgGe7StPvJ9RxDWyhnyF0zKeSdXp4RsiJKR6CVBrJn1V5k8kHDcICjUq28aRo0J
 xOQoyiNeU42bRO0AdZwiA46IO8tz/wTpN5PSzOYPgfYGuUbNYaFHeC9BP/GtNjIcj0Wc
 uvC16ycOCHh8Ns5+ySrUl8jWqCp3GEE4yHZLgSNWlzM3rn74TLe1pwYGTL5l2UMxJDtY
 g1lg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=WUN/q9027srP+Vv1BvPwzWs4wm1VZ7aY3X40i+c/rGY=;
 b=mBAGQGQ5dlpJACFnqbGzVp0N/JCJA/q+CnXigsF48Ktz6u12bQOgtHA4dNw2Iywsp4
 SL9yVCM+kMfE5cSSCUQjDhp8RFbJMrdT6Rkqtz5Jln0QaHha1E1cUtlaL2D5RLl6vAUQ
 hH/HlCv7q1GYo0C9J4ySA5yixL2NBms5F9YG54zjfT7T02OTMLWhSIelMp9UT8/wULNi
 bmacrrn7fiOueot77soyJvqUys3+0B0zYO40M4FDv+YOPTaUbX17WwrUJ0zga+13GujU
 p3NsbbKnbrH+gHBITOt4Uah0drpnimPQt/++5S0Y6GQKYxcETfH+L7lI+ELeoW6zDSmL
 drew==
X-Gm-Message-State: AOAM531o1Meth6jdQ8rnr/Mnh4bMKRDZSHkiMuvYtPO0+BSlpeBBaP6N
 4P97N8n6XRfSaAmPPp4qDTplSRQNLeuIM7BQs5VqyO5wGvA=
X-Google-Smtp-Source: ABdhPJzdru6LeOiQGf+55Ix0kpPCT7ddwOnMsXqp4qGxwIk6LvGbSETD05iD+WuETdMu9DKOOdegpqgpOM8qgrdLECk=
X-Received: by 2002:a25:bc83:: with SMTP id e3mr42810637ybk.255.1636063284525; 
 Thu, 04 Nov 2021 15:01:24 -0700 (PDT)
MIME-Version: 1.0
From: Olaoluwa Osuntokun <laolu32@gmail.com>
Date: Thu, 4 Nov 2021 15:01:13 -0700
Message-ID: <CAO3Pvs-Jazo27Vmi3Rforke++T5rkboS=2PK9CSSTtCk4enF2w@mail.gmail.com>
To: lnd@lightning.engineering, 
 Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev
 <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000e77fd905cffdaa33"
Subject: [bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Nov 2021 22:01:27 -0000

--000000000000e77fd905cffdaa33
Content-Type: text/plain; charset="UTF-8"

Hi y'all,

If you're an active user of neutrino [8], then you probably heard about what
went down over the past week or so on testnet as related to Taproot. First,
i just wanted to reassure everyone that nothing is fundamentally broken with
BIP 157/158 as it relates to taproot, and we already have a mitigation patch
in place for the issue we encountered.

The rest of this mail is structured in a FAQ style to make it easy to skim
and extract the information that may be relevant to the reader.

## What happened on testnet?

Neutrino nodes on testnet rejected a filter (thinking it was invalid) due
to this transaction spending a taproot input [1]. This was due to a faulty
heuristics in the neutrino _client code_ (not part of the protocol) that
attempted to verify the contents of a filter more completely.

In retrospect, the heuristic in question wasn't full proof, as it attempted
to derive the _pk script_ of a transaction based on its input
witness/sigScript. This worked pretty well in the context of segwit v0, but
it isn't possible to exhaustively do as we don't know what future spends
will look like.

## Is neutrino broken?

No, the client side is fine, and the protocol is fine.

The problematic heuristic has been removed in this PR [2], which will be
included in lnd 0.14, and has been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted to use such a heuristic, we'll need to
revisit how BIP 158 works briefly. For each block, we insert the `pkScript`s
of all the outputs, and also the prev out's pkScript (the script being
spent) as well. This lets the filter compress script re-use in both inputs
and outputs, and also makes it possible to implement some protocols in a
more light-client friendly manner (for example Loop uses this to has the
client watch HTLC _scripts_ being spent, as it doesn't necessarily know the
txid/outpoint).

The one issue with this, is that while clients can ensure that all the
`pkScripts` of outputs have been inserted, they can't do the same for the
inputs (which is why we added that heuristic in the client code). Luckily we
know how to properly fix this at the protocol level, more on that below.

## How can I make sure my neutrino clients handle the Taproot upgrade on
mainnet smoothly?

Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4].
The patch just demotes an error case to a warning message, so anyone running
a fork should be able to easily apply the fix.

Alongside, optionally extend these filter header guides [7].

We're looking into some intermediate ground where we can verify the scripts
that we know are relevant to the node.

## How will BIP 158/157 evolve post taproot?

In terms of adding more taproot specific functionality, I've had a number of
items in my laundry list such as:

  * creating new segwit-only filters with re-parameterized fp rates (also
    examine other filter types such as pure outpoints, etc)

  * creating filters that include witness data to allow matching on
    internal/external keys, the control block, merkle root, annex, etc

  * add a new protocol extension to btcd (with a corresponding BIP) to
    allow notes to fetch block undo data (as described here [5]) to fully
    verify fetched filters or a node needs to reconcile conflicting filters

  * new filters that span across multiple blocks as previously worked on by
    Kalle Alm (couldn't find a link to his PR when typing this...)

Make further progress towards a proposal that allows filters to be committed
either as a soft-fork, or a "velvet fork" [6] where miners optionally
commit to
the past filter header chain.


-- Laolu

[1]:
https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0
[4]: https://github.com/lightninglabs/neutrino/pull/234/files
[5]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html
[6]: https://eprint.iacr.org/2018/087
[7]:
https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15
[8]: https://github.com/lightninglabs/neutrino

--000000000000e77fd905cffdaa33
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi y&#39;all,<br><br>If you&#39;re an active user of neutr=
ino [8], then you probably heard about what<br>went down over the past week=
 or so on testnet as related to Taproot. First,<br>i just wanted to reassur=
e everyone that nothing is fundamentally broken with<br>BIP 157/158 as it r=
elates to taproot, and we already have a mitigation patch<br>in place for t=
he issue we encountered.<br><br>The rest of this mail is structured in a FA=
Q style to make it easy to skim<br>and extract the information that may be =
relevant to the reader.<br><br>## What happened on testnet?<br><br>Neutrino=
 nodes on testnet rejected a filter (thinking it was invalid) due<br>to thi=
s transaction spending a taproot input [1]. This was due to a faulty<br>heu=
ristics in the neutrino _client code_ (not part of the protocol) that<br>at=
tempted to verify the contents of a filter more completely. <br><br>In retr=
ospect, the heuristic in question wasn&#39;t full proof, as it attempted<br=
>to derive the _pk script_ of a transaction based on its input<br>witness/s=
igScript. This worked pretty well in the context of segwit v0, but<br>it is=
n&#39;t possible to exhaustively do as we don&#39;t know what future spends=
<br>will look like.<br><br>## Is neutrino broken?<br><br>No, the client sid=
e is fine, and the protocol is fine.<br><br>The problematic heuristic has b=
een removed in this PR [2], which will be<br>included in lnd 0.14, and has =
been tagged with neutrino 0.13 [3].<br><br>To dig into _why_ we attempted t=
o use such a heuristic, we&#39;ll need to<br>revisit how BIP 158 works brie=
fly. For each block, we insert the `pkScript`s<br>of all the outputs, and a=
lso the prev out&#39;s pkScript (the script being<br>spent) as well. This l=
ets the filter compress script re-use in both inputs<br>and outputs, and al=
so makes it possible to implement some protocols in a<br>more light-client =
friendly manner (for example Loop uses this to has the<br>client watch HTLC=
 _scripts_ being spent, as it doesn&#39;t necessarily know the<br>txid/outp=
oint).<br><br>The one issue with this, is that while clients can ensure tha=
t all the<br>`pkScripts` of outputs have been inserted, they can&#39;t do t=
he same for the<br>inputs (which is why we added that heuristic in the clie=
nt code). Luckily we<br>know how to properly fix this at the protocol level=
, more on that below.<br><br>## How can I make sure my neutrino clients han=
dle the Taproot upgrade on mainnet smoothly?<br><br>Upgrade to 0.14 (assumi=
ng it&#39;s out in time), or apply this small patch [4].<br>The patch just =
demotes an error case to a warning message, so anyone running<br>a fork sho=
uld be able to easily apply the fix.<br><br>Alongside, optionally extend th=
ese filter header guides [7].<br><br>We&#39;re looking into some intermedia=
te ground where we can verify the scripts<br>that we know are relevant to t=
he node.<br><br>## How will BIP 158/157 evolve post taproot?<br><br>In term=
s of adding more taproot specific functionality, I&#39;ve had a number of<b=
r>items in my laundry list such as:<br><br>=C2=A0 * creating new segwit-onl=
y filters with re-parameterized fp rates (also<br>=C2=A0 =C2=A0 examine oth=
er filter types such as pure outpoints, etc)<br><br>=C2=A0 * creating filte=
rs that include witness data to allow matching on<br>=C2=A0 =C2=A0 internal=
/external keys, the control block, merkle root, annex, etc<br><br>=C2=A0 * =
add a new protocol extension to btcd (with a corresponding BIP) to<br>=C2=
=A0 =C2=A0 allow notes to fetch block undo data (as described here [5]) to =
fully<br>=C2=A0 =C2=A0 verify fetched filters or a node needs to reconcile =
conflicting filters<br><br>=C2=A0 * new filters that span across multiple b=
locks as previously worked on by<br>=C2=A0 =C2=A0 Kalle Alm (couldn&#39;t f=
ind a link to his PR when typing this...)<br><br>Make further progress towa=
rds a proposal that allows filters to be committed<br>either as a soft-fork=
, or a &quot;velvet fork&quot; [6] where miners optionally commit to<br>the=
 past filter header chain.<br><br><br>-- Laolu<br><br>[1]: <a href=3D"https=
://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be13=
98e3f561cc3f19fb8">https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b8=
10c53078773fb768acd2be1398e3f561cc3f19fb8</a><br>[2]: <a href=3D"https://gi=
thub.com/lightninglabs/neutrino/pull/234">https://github.com/lightninglabs/=
neutrino/pull/234</a><br>[3]: <a href=3D"https://github.com/lightninglabs/n=
eutrino/releases/tag/v0.13.0">https://github.com/lightninglabs/neutrino/rel=
eases/tag/v0.13.0</a><br>[4]: <a href=3D"https://github.com/lightninglabs/n=
eutrino/pull/234/files">https://github.com/lightninglabs/neutrino/pull/234/=
files</a><br>[5]: <a href=3D"https://lists.linuxfoundation.org/pipermail/bi=
tcoin-dev/2019-February/016649.html">https://lists.linuxfoundation.org/pipe=
rmail/bitcoin-dev/2019-February/016649.html</a><br>[6]: <a href=3D"https://=
eprint.iacr.org/2018/087">https://eprint.iacr.org/2018/087</a><br>[7]: <a h=
ref=3D"https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07=
aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15">https://github.com/ligh=
tninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/=
filtercontrol.go#L15</a><br>[8]: <a href=3D"https://github.com/lightninglab=
s/neutrino">https://github.com/lightninglabs/neutrino</a><br></div>

--000000000000e77fd905cffdaa33--