1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
|
Return-Path: <pieter.wuille@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 40C697A8
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 19 May 2018 03:06:24 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-oi0-f45.google.com (mail-oi0-f45.google.com
[209.85.218.45])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9A736B0
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 19 May 2018 03:06:23 +0000 (UTC)
Received: by mail-oi0-f45.google.com with SMTP id b130-v6so8774566oif.12
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 18 May 2018 20:06:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=FR6J0tAYy2F3YucOEtAkMRzrocede2dBsefHwIFMjl8=;
b=RGCKVwu7xYXUhXmkPn3VXtO1Hlwt88tXl0Vi31GwTFUoNnV9mVDcGzNLRJxlqjs621
hGWtRPyC1hJ3yEA1T4eWsvL17FELApp5gbxh4WqtLs8SeiN4+CYjdnTA/eRsBNr33jrl
D7nNj/1PS8R4QejwjUcF+HV691atxiksLKieEMOqGRMGPmyaTATOP5tDzX5CE/NI9wLz
0PHpR3XJa+u/Ugxry47QxUQlzBCP1d1LtNFA5ahHtIdTNVydVjLULNBlT40lGLEIV+eI
ZgehYhziUum/hEq0+JFBfkbL39gntl6ZMFFa9qRBoqGiIC0hI8wY/PRNLGk5Q5cClo4K
irgg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=FR6J0tAYy2F3YucOEtAkMRzrocede2dBsefHwIFMjl8=;
b=UdB0H7NP41dFcC27DeQlbo7ZI+C4xuLvW8FKzM5bKJewO35RWj9V9eZA/IUiiD8db5
Z9FMVIubI9s3BTaEf29Xzyt1k2maMZjs53xD5307Rb2yRPoiFy3u9gSpP8XIgYVRHEvk
kFzCXq9zey9pbG5Oli7Tv5HA84BwmFXQsTCvIvSErKytp1IKKRLsp0F8GQadqc10eLQy
85tO31UXI4mN+AuFZSnWQsr2BzNpXG09/zZ5GoCEt8UHRrcLwzJoGDsCG9TT2o1jOtk+
UFqSU/SW4oXcTjUJ0Q62oIw7VUhzr/LPx2RMM79h7ep4/zzRsbh3vOQ00OhJrbkncSgJ
rwhQ==
X-Gm-Message-State: ALKqPwdBPBGJx/wCHbuu4+MyY18cBU3qO6G0pyq0tejlyPMs0+qIo+xf
m3qjFm1HJxt3JJDMsgQI9k3f9zH8sW727N6ozRk=
X-Google-Smtp-Source: AB8JxZoAZSHEIPBFfmK8nXsJsG3qAczJlJedRLr1MeVZa4X/5SN64J1PkOiYbOk9JfQR4jiZTcHraj5+VKw+xvVpZeU=
X-Received: by 2002:aca:f12:: with SMTP id 18-v6mr7377507oip.46.1526699182757;
Fri, 18 May 2018 20:06:22 -0700 (PDT)
MIME-Version: 1.0
References: <d43c6082-1b2c-c95b-5144-99ad0021ea6c@mattcorallo.com>
<CAAS2fgRF-MhOvpFY6c_qAPzNMo3GQ28RExdSbOV6Q6Oy2iWn1A@mail.gmail.com>
<CAO3Pvs8DaphZjZUp8_Og+SMmYrrgFi3HyWTZb5J1mGVEcmkn8A@mail.gmail.com>
In-Reply-To: <CAO3Pvs8DaphZjZUp8_Og+SMmYrrgFi3HyWTZb5J1mGVEcmkn8A@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
Date: Fri, 18 May 2018 20:06:10 -0700
Message-ID: <CAPg+sBhL8ZV+kswgyQfQyhd0Qv5Mkt1cYxrfFV4H32s9QYLo0A@mail.gmail.com>
To: Olaoluwa Osuntokun <laolu32@gmail.com>,
Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="00000000000077a485056c865a77"
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE,
RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sat, 19 May 2018 03:06:24 -0000
--00000000000077a485056c865a77
Content-Type: text/plain; charset="UTF-8"
On Fri, May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:
> Greg wrote:
> > What about also making input prevouts filter based on the scriptpubkey
> being
> > _spent_? Layering wise in the processing it's a bit ugly, but if you
> > validated the block you have the data needed.
>
> AFAICT, this would mean that in order for a new node to catch up the filter
> index (index all historical blocks), they'd either need to: build up a
> utxo-set in memory during indexing, or would require a txindex in order to
> look up the prev out's script. The first option increases the memory load
> during indexing, and the second requires nodes to have a transaction index
> (and would also add considerable I/O load). When proceeding from tip, this
> doesn't add any additional load assuming that your synchronously index the
> block as you validate it, otherwise the utxo set will already have been
> updated (the spent scripts removed).
>
I was wondering about that too, but it turns out that isn't necessary. At
least in Bitcoin Core, all the data needed for such a filter is in the
block + undo files (the latter contain the scriptPubKeys of the outputs
being spent).
I have a script running to compare the filter sizes assuming the regular
> filter switches to include the prev out's script rather than the prev
> outpoint itself. The script hasn't yet finished (due to the increased I/O
> load to look up the scripts when indexing), but I'll report back once it's
> finished.
>
That's very helpful, thank you.
Cheers,
--
Pieter
--00000000000077a485056c865a77
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"auto"><div><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, =
May 18, 2018, 19:57 Olaoluwa Osuntokun via bitcoin-dev <<a href=3D"mailt=
o:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.=
org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"=
><div>Greg wrote:</div><div>> What about also making input prevouts filt=
er based on the scriptpubkey being</div><div>> _spent_?=C2=A0 Layering w=
ise in the processing it's a bit ugly, but if you</div><div>> valida=
ted the block you have the data needed.</div><div><br></div><div>AFAICT, th=
is would mean that in order for a new node to catch up the filter</div><div=
>index (index all historical blocks), they'd either need to: build up a=
</div><div>utxo-set in memory during indexing, or would require a txindex i=
n order to</div><div>look up the prev out's script. The first option in=
creases the memory load</div><div>during indexing, and the second requires =
nodes to have a transaction index</div><div>(and would also add considerabl=
e I/O load). When proceeding from tip, this</div><div>doesn't add any a=
dditional load assuming that your synchronously index the</div><div>block a=
s you validate it, otherwise the utxo set will already have been</div><div>=
updated (the spent scripts removed).</div></div></blockquote></div></div><d=
iv dir=3D"auto"><br></div><div dir=3D"auto">I was wondering about that too,=
but it turns out that isn't necessary. At least in Bitcoin Core, all t=
he data needed for such a filter is in the block + undo files (the latter c=
ontain the scriptPubKeys of the outputs being spent).</div><div dir=3D"auto=
"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr"><div>I have a script running to compare the =
filter sizes assuming the regular</div><div>filter switches to include the =
prev out's script rather than the prev</div><div>outpoint itself. The s=
cript hasn't yet finished (due to the increased I/O</div><div>load to l=
ook up the scripts when indexing), but I'll report back once it's</=
div><div>finished.</div></div></blockquote></div></div><div dir=3D"auto"><b=
r></div><div dir=3D"auto">That's very helpful, thank you.</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">Cheers,</div><div dir=3D"auto"><br></=
div><div dir=3D"auto">--=C2=A0</div><div dir=3D"auto">Pieter</div><div dir=
=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex">
</blockquote></div></div></div>
--00000000000077a485056c865a77--
|
