MIME-Version: 1.0
References: <d43c6082-1b2c-c95b-5144-99ad0021ea6c@mattcorallo.com>
	<CAAS2fgRF-MhOvpFY6c_qAPzNMo3GQ28RExdSbOV6Q6Oy2iWn1A@mail.gmail.com>
	<CAO3Pvs8DaphZjZUp8_Og+SMmYrrgFi3HyWTZb5J1mGVEcmkn8A@mail.gmail.com>
In-Reply-To: <CAO3Pvs8DaphZjZUp8_Og+SMmYrrgFi3HyWTZb5J1mGVEcmkn8A@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
Date: Fri, 18 May 2018 20:06:10 -0700
Message-ID: <CAPg+sBhL8ZV+kswgyQfQyhd0Qv5Mkt1cYxrfFV4H32s9QYLo0A@mail.gmail.com>
To: Olaoluwa Osuntokun <laolu32@gmail.com>,
	Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="00000000000077a485056c865a77"
Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
Precedence: list

--00000000000077a485056c865a77
Content-Type: text/plain; charset="UTF-8"

On Fri, May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Greg wrote:
> > What about also making input prevouts filter based on the scriptpubkey
> being
> > _spent_?  Layering wise in the processing it's a bit ugly, but if you
> > validated the block you have the data needed.
>
> AFAICT, this would mean that in order for a new node to catch up the filter
> index (index all historical blocks), they'd either need to: build up a
> utxo-set in memory during indexing, or would require a txindex in order to
> look up the prev out's script. The first option increases the memory load
> during indexing, and the second requires nodes to have a transaction index
> (and would also add considerable I/O load). When proceeding from tip, this
> doesn't add any additional load assuming that your synchronously index the
> block as you validate it, otherwise the utxo set will already have been
> updated (the spent scripts removed).
>

I was wondering about that too, but it turns out that isn't necessary. At
least in Bitcoin Core, all the data needed for such a filter is in the
block + undo files (the latter contain the scriptPubKeys of the outputs
being spent).

I have a script running to compare the filter sizes assuming the regular
> filter switches to include the prev out's script rather than the prev
> outpoint itself. The script hasn't yet finished (due to the increased I/O
> load to look up the scripts when indexing), but I'll report back once it's
> finished.
>

That's very helpful, thank you.

Cheers,

-- 
Pieter

--00000000000077a485056c865a77
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, =
May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev &lt;<a href=3D"mailt=
o:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.=
org</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"=
><div>Greg wrote:</div><div>&gt; What about also making input prevouts filt=
er based on the scriptpubkey being</div><div>&gt; _spent_?=C2=A0 Layering w=
ise in the processing it&#39;s a bit ugly, but if you</div><div>&gt; valida=
ted the block you have the data needed.</div><div><br></div><div>AFAICT, th=
is would mean that in order for a new node to catch up the filter</div><div=
>index (index all historical blocks), they&#39;d either need to: build up a=
</div><div>utxo-set in memory during indexing, or would require a txindex i=
n order to</div><div>look up the prev out&#39;s script. The first option in=
creases the memory load</div><div>during indexing, and the second requires =
nodes to have a transaction index</div><div>(and would also add considerabl=
e I/O load). When proceeding from tip, this</div><div>doesn&#39;t add any a=
dditional load assuming that your synchronously index the</div><div>block a=
s you validate it, otherwise the utxo set will already have been</div><div>=
updated (the spent scripts removed).</div></div></blockquote></div></div><d=
iv dir=3D"auto"><br></div><div dir=3D"auto">I was wondering about that too,=
 but it turns out that isn&#39;t necessary. At least in Bitcoin Core, all t=
he data needed for such a filter is in the block + undo files (the latter c=
ontain the scriptPubKeys of the outputs being spent).</div><div dir=3D"auto=
"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr"><div>I have a script running to compare the =
filter sizes assuming the regular</div><div>filter switches to include the =
prev out&#39;s script rather than the prev</div><div>outpoint itself. The s=
cript hasn&#39;t yet finished (due to the increased I/O</div><div>load to l=
ook up the scripts when indexing), but I&#39;ll report back once it&#39;s</=
div><div>finished.</div></div></blockquote></div></div><div dir=3D"auto"><b=
r></div><div dir=3D"auto">That&#39;s very helpful, thank you.</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">Cheers,</div><div dir=3D"auto"><br></=
div><div dir=3D"auto">--=C2=A0</div><div dir=3D"auto">Pieter</div><div dir=
=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex">
</blockquote></div></div></div>

--00000000000077a485056c865a77--