Return-Path: Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 455DFC000E for ; Thu, 4 Nov 2021 22:08:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3595E81A50 for ; Thu, 4 Nov 2021 22:08:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: 0.149 X-Spam-Level: X-Spam-Status: No, score=0.149 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.997, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=no autolearn_force=no Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OtKIe2_eW2sv for ; Thu, 4 Nov 2021 22:08:05 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-yb1-xb29.google.com (mail-yb1-xb29.google.com [IPv6:2607:f8b0:4864:20::b29]) by smtp1.osuosl.org (Postfix) with ESMTPS id DAD1D81A3E for ; Thu, 4 Nov 2021 22:08:04 +0000 (UTC) Received: by mail-yb1-xb29.google.com with SMTP id d10so17994295ybe.3 for ; Thu, 04 Nov 2021 15:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=2LKZJqbRCfivMlgrJ7cimVgGhfZpMCbfshzZ9vQIwgU=; b=Mn6tyqxCmjMEhi4+1loL6mRdihtykvuTgFQ+GFc0Je7YPP3KFtcXDn2vMqtYjsmEPi lHYl3lX6LsxQwlTOdwOUWf9pycUxer6WJyqIVpDU3wPAGhdTZjIzgPX7tIHFscN+1Fmc TtvMN2EiORVFZ0jbNhx+ELput8e3OwEOkptVt/ZXALAwZtkZ2AIfpBCtfY7XKWpnHJ9t O3I6s7SDQUQHU/NCMs5VyJhNbTix8CcaF/bWnzjBPtUDi+mSRm2nwdAFKb7Yc2yJu79m IwgBUIFlZek7/cnk1AH4tcT8XAv1qfCu5OSxY0Tf70VcCBBvICOezLOxB2mIJk3h0pMl uOpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=2LKZJqbRCfivMlgrJ7cimVgGhfZpMCbfshzZ9vQIwgU=; b=KM+1SS80EadzFlspuT0PErcHLNlWq0nyptoO0F0CwJ9TzX8C2w3f9hrGWsCbLQRoi+ dFlqrDM4ubAQ4TF/5Ch5rK2yjCU+0ZeBSn8wJK71zi/fkby94HJozmjSaDw6xBlIxb3x O+B9eTBGZFbVLMhV0eegcxlywm2vNWeu3jhMIENcetJWfNUjVZ85T9XlqQ6jn9wtyvvk jQCEWHj5YOCZRoDBHFEwmNK0D98W2Zydjoc0XOgtm5SHi4dEeGjq0Q2+EJweUaiettuv O0965Fi/UDiiln233KiLENBCsNloAWmNgkAC6MF6tYRoo7my+ogS3tuBzmhSCpn7DIrc roeg== X-Gm-Message-State: AOAM5300ri9hq6zkMpiFn/Nq5NNYbBLgaebggdt5izONXMI7f517CFdE Qcb5o9pRZ58V6gSBlNFvedKpSoBmvf4oCc3TyseE7uTZno4= X-Google-Smtp-Source: ABdhPJxEyr4t3fVw3QZNcLTegz6lpvQrHWX9VEoGewuK7OPJ4bIp06u38+okHofGyUe31laSC7ty5vssI8122M1y2b4= X-Received: by 2002:a25:aa67:: with SMTP id s94mr47649860ybi.343.1636063683646; Thu, 04 Nov 2021 15:08:03 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Olaoluwa Osuntokun Date: Thu, 4 Nov 2021 15:07:52 -0700 Message-ID: To: lnd@lightning.engineering, Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev Content-Type: multipart/alternative; boundary="000000000000b199ca05cffdc228" Subject: Re: [bitcoin-dev] Neutrino, Taproot, and The Evolution of BiPs 157/158 X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Nov 2021 22:08:06 -0000 --000000000000b199ca05cffdc228 Content-Type: text/plain; charset="UTF-8" > In terms of adding more taproot specific functionality, I've had a number of > items in my laundry list such as: Forgot to add this other item (also the list wasn't meant to be only tapoot stuff): * reviving old projects to include a micropayment-for-data layer to incentivize nodes to serve the filters and other data On Thu, Nov 4, 2021 at 3:01 PM Olaoluwa Osuntokun wrote: > Hi y'all, > > If you're an active user of neutrino [8], then you probably heard about > what > went down over the past week or so on testnet as related to Taproot. First, > i just wanted to reassure everyone that nothing is fundamentally broken > with > BIP 157/158 as it relates to taproot, and we already have a mitigation > patch > in place for the issue we encountered. > > The rest of this mail is structured in a FAQ style to make it easy to skim > and extract the information that may be relevant to the reader. > > ## What happened on testnet? > > Neutrino nodes on testnet rejected a filter (thinking it was invalid) due > to this transaction spending a taproot input [1]. This was due to a faulty > heuristics in the neutrino _client code_ (not part of the protocol) that > attempted to verify the contents of a filter more completely. > > In retrospect, the heuristic in question wasn't full proof, as it attempted > to derive the _pk script_ of a transaction based on its input > witness/sigScript. This worked pretty well in the context of segwit v0, but > it isn't possible to exhaustively do as we don't know what future spends > will look like. > > ## Is neutrino broken? > > No, the client side is fine, and the protocol is fine. > > The problematic heuristic has been removed in this PR [2], which will be > included in lnd 0.14, and has been tagged with neutrino 0.13 [3]. > > To dig into _why_ we attempted to use such a heuristic, we'll need to > revisit how BIP 158 works briefly. For each block, we insert the > `pkScript`s > of all the outputs, and also the prev out's pkScript (the script being > spent) as well. This lets the filter compress script re-use in both inputs > and outputs, and also makes it possible to implement some protocols in a > more light-client friendly manner (for example Loop uses this to has the > client watch HTLC _scripts_ being spent, as it doesn't necessarily know the > txid/outpoint). > > The one issue with this, is that while clients can ensure that all the > `pkScripts` of outputs have been inserted, they can't do the same for the > inputs (which is why we added that heuristic in the client code). Luckily > we > know how to properly fix this at the protocol level, more on that below. > > ## How can I make sure my neutrino clients handle the Taproot upgrade on > mainnet smoothly? > > Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4]. > The patch just demotes an error case to a warning message, so anyone > running > a fork should be able to easily apply the fix. > > Alongside, optionally extend these filter header guides [7]. > > We're looking into some intermediate ground where we can verify the scripts > that we know are relevant to the node. > > ## How will BIP 158/157 evolve post taproot? > > In terms of adding more taproot specific functionality, I've had a number > of > items in my laundry list such as: > > * creating new segwit-only filters with re-parameterized fp rates (also > examine other filter types such as pure outpoints, etc) > > * creating filters that include witness data to allow matching on > internal/external keys, the control block, merkle root, annex, etc > > * add a new protocol extension to btcd (with a corresponding BIP) to > allow notes to fetch block undo data (as described here [5]) to fully > verify fetched filters or a node needs to reconcile conflicting filters > > * new filters that span across multiple blocks as previously worked on by > Kalle Alm (couldn't find a link to his PR when typing this...) > > Make further progress towards a proposal that allows filters to be > committed > either as a soft-fork, or a "velvet fork" [6] where miners optionally > commit to > the past filter header chain. > > > -- Laolu > > [1]: > https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8 > [2]: https://github.com/lightninglabs/neutrino/pull/234 > [3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0 > [4]: https://github.com/lightninglabs/neutrino/pull/234/files > [5]: > https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html > [6]: https://eprint.iacr.org/2018/087 > [7]: > https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15 > [8]: https://github.com/lightninglabs/neutrino > --000000000000b199ca05cffdc228 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> In terms of adding more taproot specific functionalit= y, I've had a number of
> items in my laundry list such as:
= =C2=A0=C2=A0
Forgot to add this other item (also the list wasn= 9;t meant to be only tapoot stuff):
=C2=A0 * reviving old projects to in= clude a micropayment-for-data layer to
=C2=A0 =C2=A0 incentivize nodes t= o serve the filters and other data

On Thu, Nov 4, 2021 at 3:01 PM = Olaoluwa Osuntokun <laolu32@gmail.c= om> wrote:
Hi y'all,

If you're an active user of neu= trino [8], then you probably heard about what
went down over the past we= ek or so on testnet as related to Taproot. First,
i just wanted to reass= ure everyone that nothing is fundamentally broken with
BIP 157/158 as it= relates to taproot, and we already have a mitigation patch
in place for= the issue we encountered.

The rest of this mail is structured in a = FAQ style to make it easy to skim
and extract the information that may b= e relevant to the reader.

## What happened on testnet?

Neutri= no nodes on testnet rejected a filter (thinking it was invalid) due
to t= his transaction spending a taproot input [1]. This was due to a faulty
h= euristics in the neutrino _client code_ (not part of the protocol) that
= attempted to verify the contents of a filter more completely.

In re= trospect, the heuristic in question wasn't full proof, as it attempted<= br>to derive the _pk script_ of a transaction based on its input
witness= /sigScript. This worked pretty well in the context of segwit v0, but
it = isn't possible to exhaustively do as we don't know what future spen= ds
will look like.

## Is neutrino broken?

No, the client s= ide is fine, and the protocol is fine.

The problematic heuristic has= been removed in this PR [2], which will be
included in lnd 0.14, and ha= s been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted= to use such a heuristic, we'll need to
revisit how BIP 158 works br= iefly. For each block, we insert the `pkScript`s
of all the outputs, and= also the prev out's pkScript (the script being
spent) as well. This= lets the filter compress script re-use in both inputs
and outputs, and = also makes it possible to implement some protocols in a
more light-clien= t friendly manner (for example Loop uses this to has the
client watch HT= LC _scripts_ being spent, as it doesn't necessarily know the
txid/ou= tpoint).

The one issue with this, is that while clients can ensure t= hat all the
`pkScripts` of outputs have been inserted, they can't do= the same for the
inputs (which is why we added that heuristic in the cl= ient code). Luckily we
know how to properly fix this at the protocol lev= el, more on that below.

## How can I make sure my neutrino clients h= andle the Taproot upgrade on mainnet smoothly?

Upgrade to 0.14 (assu= ming it's out in time), or apply this small patch [4].
The patch jus= t demotes an error case to a warning message, so anyone running
a fork s= hould be able to easily apply the fix.

Alongside, optionally extend = these filter header guides [7].

We're looking into some intermed= iate ground where we can verify the scripts
that we know are relevant to= the node.

## How will BIP 158/157 evolve post taproot?

In te= rms of adding more taproot specific functionality, I've had a number of=
items in my laundry list such as:

=C2=A0 * creating new segwit-o= nly filters with re-parameterized fp rates (also
=C2=A0 =C2=A0 examine o= ther filter types such as pure outpoints, etc)

=C2=A0 * creating fil= ters that include witness data to allow matching on
=C2=A0 =C2=A0 intern= al/external keys, the control block, merkle root, annex, etc

=C2=A0 = * add a new protocol extension to btcd (with a corresponding BIP) to
=C2= =A0 =C2=A0 allow notes to fetch block undo data (as described here [5]) to = fully
=C2=A0 =C2=A0 verify fetched filters or a node needs to reconcile = conflicting filters

=C2=A0 * new filters that span across multiple b= locks as previously worked on by
=C2=A0 =C2=A0 Kalle Alm (couldn't f= ind a link to his PR when typing this...)

Make further progress towa= rds a proposal that allows filters to be committed
either as a soft-fork= , or a "velvet fork" [6] where miners optionally commit to
the= past filter header chain.


-- Laolu

[1]: https://mempool.space/testnet/tx/4b425= a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0<= /a>
[4]: https://github.com/lightninglabs/neutrino/pull/234/= files
[5]: https://lists.linuxf= oundation.org/pipermail/bitcoin-dev/2019-February/016649.html
[6]: <= a href=3D"https://eprint.iacr.org/2018/087" target=3D"_blank">https://eprin= t.iacr.org/2018/087
[7]: https://github.com/lightninglabs/neutrino/bl= ob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15<= /a>
[8]: https://github.com/lightninglabs/neutrino
--000000000000b199ca05cffdc228--