1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
Return-Path: <igor.cota.hr@gmail.com>
Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138])
by lists.linuxfoundation.org (Postfix) with ESMTP id 192BAC0001
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 27 Feb 2021 07:10:54 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
by smtp1.osuosl.org (Postfix) with ESMTP id E859D842FC
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 27 Feb 2021 07:10:53 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: 1.3
X-Spam-Level: *
X-Spam-Status: No, score=1.3 tagged_above=-999 required=5 tests=[BAYES_50=0.8,
DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
FREEMAIL_FORGED_FROMDOMAIN=0.249,
FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001,
RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001]
autolearn=no autolearn_force=no
Authentication-Results: smtp1.osuosl.org (amavisd-new);
dkim=pass (2048-bit key)
header.d=codexapertus-com.20150623.gappssmtp.com
Received: from smtp1.osuosl.org ([127.0.0.1])
by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id WCTb3dtYA3iz
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 27 Feb 2021 07:10:52 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0
Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com
[209.85.219.170])
by smtp1.osuosl.org (Postfix) with ESMTPS id 8892B842FB
for <bitcoin-dev@lists.linuxfoundation.org>;
Sat, 27 Feb 2021 07:10:52 +0000 (UTC)
Received: by mail-yb1-f170.google.com with SMTP id c131so11284867ybf.7
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 26 Feb 2021 23:10:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=codexapertus-com.20150623.gappssmtp.com; s=20150623;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
bh=n/ht1PcMwBb+RstIxCx6iVJxZKk2ayvn7+TU1GPbDp4=;
b=e+kVkufVBFt8I+zIRJmadJg5Onr1FcY6MySa4FHNQ8KdTlnsg6H9aD5yJhu7aciy74
KC3orlHpezl968qu1nn0AAw7P4+pllwUSZ2glFyazCKDdrsxSBDBQGiERwsqVJFepecl
BJBxswzN8h2RSqiyteSKkS6ehLW30TsJhFKlezxteNa1i8AZ7Ovbp88bOIzMbJ4NqAoM
LfxfEdJeHpClZ40EAjZuUDf2h9RUBSUt55jFNWAU/offVfPai0zJ2pX5d6i9ruOC3mcy
m8frybNohImoS5aPwNrp1Euz3ram3MxKPFdx3/UrjHSiA1L+VYfM3+/ju+l+H6BYI/W/
zELg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to;
bh=n/ht1PcMwBb+RstIxCx6iVJxZKk2ayvn7+TU1GPbDp4=;
b=oK41sWwF67mzlsxJ+qH5Tr2/nsKGo5xTbaSNhcb3pqQDy+KmcEIm38aUSfOUE2cxqm
8tMW2AUtDCsj1+kLGHi2PEju/KUUbouAOJsMsRfU1ZiJtCZRS3N2TJOIPasbg7QbmydA
HDHeQO2xxUW8k9/fiT26yKUwAD4tlfn/KeMLrHYcvhs/ab1Ui726pyyhx5WTVuqj3GYm
QcIMAJbF2Fq0E5UKUbxhAM+psyK1GLzlpGcRO2fqAJBz9lWxDx92mRr8xhojQozh9HzR
O5CwGpxQJIWl9LcKFMOwCmvoVgJ4CxAsx/Wa8pxPlbSAKgNc5Hrh5W2NyZ4YLRvcHoPT
++SQ==
X-Gm-Message-State: AOAM531LFiES1Z10mAt6/ce1g5lNIzSWW28PY1QupRVhtI8vT72l9jtO
4NbhNr/J5kV/yDFceZkq/fu73GR8nonl0YUjA0U=
X-Google-Smtp-Source: ABdhPJxQf93Hmq8RbilEXj+BZDmspFJkcuIo891kYDurzipk/CiIvOBHsabFyZ3wfkbddSExUK9cq4D5ctWdeuyMYeo=
X-Received: by 2002:a5b:847:: with SMTP id v7mr9526758ybq.354.1614409849835;
Fri, 26 Feb 2021 23:10:49 -0800 (PST)
MIME-Version: 1.0
References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
In-Reply-To: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
From: Igor Cota <igor@codexapertus.com>
Date: Sat, 27 Feb 2021 08:10:39 +0100
Message-ID: <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
To: Keagan McClelland <keagan.mcclelland@gmail.com>,
Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
chike@sfc.wide.ad.jp, Artur Brugeman <brugeman.artur@gmail.com>
Content-Type: multipart/alternative; boundary="0000000000009f13fa05bc4c1565"
X-Mailman-Approved-At: Sat, 27 Feb 2021 11:49:53 +0000
Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sat, 27 Feb 2021 07:10:54 -0000
--0000000000009f13fa05bc4c1565
Content-Type: text/plain; charset="UTF-8"
Hi Keagan,
I had a very similar idea. The only difference being for the node to decide
on a range of blocks to keep beforehand, rather than making the decision
block-by-block like you suggest.
I felt the other nodes would be better served by ranges due to the
sequential nature of IBD. Perhaps this would be computationally lighter as
well.
I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
scheme to solve this same problem.
Cheers,
Igor
[1] https://arxiv.org/abs/1902.02174
On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:
> Hi all,
>
> I've been thinking for quite some time about the problem of pruned nodes
> and ongoing storage costs for full nodes. One of the things that strikes me
> as odd is that we only really have two settings.
>
> A. Prune everything except the most recent blocks, down to the cache size
> B. Keep everything since genesis
>
> From my observations and conversations with various folks in the
> community, they would like to be able to run a "partially" pruned node to
> help bear the load of bootstrapping other nodes and helping with data
> redundancy in the network, but would prefer to not dedicate hundreds of
> Gigabytes of storage space to the cause.
>
> This led me to the idea that a node could randomly prune some of the
> blocks from history if it passed some predicate. A rough sketch of this
> would look as follows.
>
> 1. At node startup, it would generate a random seed, this would be unique
> to the node but not necessary that it be cryptographically secure.
> 2. In the node configuration it would also carry a "threshold" expressed
> as some percentage of blocks it wanted to keep.
> 3. As IBD occurs, based off of the threshold, the block hash, and the
> node's unique seed, the node would either decide to prune the data or keep
> it. The uniqueness of the node's hash should ensure that no block is
> systematically overrepresented in the set of nodes choosing this storage
> scheme.
> 4. Once the node's IBD is complete it would advertise this as a peer
> service, advertising its seed and threshold, so that nodes could
> deterministically deduce which of its peers had which blocks.
>
> The goals are to increase data redundancy in a way that more uniformly
> shares the load across nodes, alleviating some of the pressure of full
> archive nodes on the IBD problem. I am working on a draft BIP for this
> proposal but figured I would submit it as a high level idea in case anyone
> had any feedback on the initial design before I go into specification
> levels of detail.
>
> If you have thoughts on
>
> A. The protocol design itself
> B. The barriers to put this kind of functionality into Core
>
> I would love to hear from you,
>
> Cheers,
> Keagan
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
--
*Igor Cota*
Codex Apertus d.o.o.
--0000000000009f13fa05bc4c1565
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Keagan,<div><br></div><div>I had a very similar idea. T=
he only difference being for the node to decide on a range of blocks to kee=
p beforehand, rather than making the decision block-by-block like you sugge=
st.</div><div><br></div><div>I felt the other nodes would be better served =
by ranges due to the sequential nature of IBD. Perhaps this would be comput=
ationally lighter as well.</div><div><br></div><div>I also encourage=C2=A0y=
ou to read=C2=A0Ryosuke Abe's paper [1] that proposes a=C2=A0DHT scheme=
to solve this same problem.</div><div><br></div><div>Cheers,</div><div>Igo=
r</div><div><br></div><div>[1]=C2=A0<a href=3D"https://arxiv.org/abs/1902.0=
2174">https://arxiv.org/abs/1902.02174</a></div></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 26 Feb 2021 at 21:5=
7, Keagan McClelland via bitcoin-dev <<a href=3D"mailto:bitcoin-dev@list=
s.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.=
org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div dir=3D"ltr">Hi all,<div><br></div><div>I've been thinking for q=
uite some time about the problem of pruned nodes and ongoing storage costs =
for full nodes. One of the things that strikes me as odd is that we only re=
ally have two settings.</div><div><br></div><div>A. Prune everything except=
the most recent blocks, down to the cache size</div><div>B. Keep everythin=
g since genesis</div><div><br></div><div>From my observations and conversat=
ions with various folks in the community, they would like to be able to run=
a "partially" pruned node to help bear the load of bootstrapping=
other nodes and helping with data redundancy in the network, but would pre=
fer to not dedicate hundreds of Gigabytes of storage space to the cause.</d=
iv><div><br></div><div>This led me to the idea that a node could randomly p=
rune some of the blocks from history if it passed some predicate. A rough s=
ketch of this would look as follows.</div><div><br></div><div>1. At node st=
artup, it would generate a random seed, this would be unique to the node bu=
t not necessary that it be cryptographically secure.</div><div>2. In the no=
de configuration it would also carry a "threshold" expressed as s=
ome percentage of blocks it wanted to keep.</div><div>3. As IBD occurs, bas=
ed off of the threshold, the block hash, and the node's unique seed, th=
e node would either decide to prune the data or keep it. The uniqueness of =
the node's hash should ensure that no block is systematically overrepre=
sented in the set of nodes choosing this storage scheme.</div><div>4. Once =
the node's IBD is complete it would advertise this as a peer service, a=
dvertising its seed and threshold, so that nodes could deterministically de=
duce which of its peers had which blocks.</div><div><br></div><div>The goal=
s are to increase data redundancy in a way that more uniformly shares the l=
oad across nodes, alleviating some of the pressure of full archive nodes on=
the IBD problem. I am working on a draft BIP for this proposal but figured=
I would submit it as a high level idea in case anyone had any feedback on =
the initial design before I go into specification levels of detail.</div><d=
iv><br></div><div>If you have thoughts on</div><div><br></div><div>A. The p=
rotocol design itself</div><div>B. The barriers to put this kind of functio=
nality into Core</div><div><br></div><div>I would love to hear from you,</d=
iv><div><br></div><div>Cheers,</div><div>Keagan</div></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
><div dir=3D"ltr"><div><div dir=3D"ltr"><b>Igor Cota</b><div>Codex Apertus =
d.o.o.<br></div></div></div></div></div>
--0000000000009f13fa05bc4c1565--
|