path: root/9e/1ec16fede91e0c90010a16c3e7cf4674a83f7d

Return-Path: <keagan.mcclelland@gmail.com>
Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137])
 by lists.linuxfoundation.org (Postfix) with ESMTP id B05BFC0001
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Mar 2021 20:55:24 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by smtp4.osuosl.org (Postfix) with ESMTP id 9E2984EC14
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Mar 2021 20:55:24 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: -0.2
X-Spam-Level: 
X-Spam-Status: No, score=-0.2 tagged_above=-999 required=5
 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
 DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001]
 autolearn=ham autolearn_force=no
Authentication-Results: smtp4.osuosl.org (amavisd-new);
 dkim=pass (2048-bit key) header.d=gmail.com
Received: from smtp4.osuosl.org ([127.0.0.1])
 by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id MB09ykEDiqyt
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Mar 2021 20:55:23 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0
Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com
 [209.85.221.52])
 by smtp4.osuosl.org (Postfix) with ESMTPS id 114414EC0D
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon,  1 Mar 2021 20:55:22 +0000 (UTC)
Received: by mail-wr1-f52.google.com with SMTP id h98so17578446wrh.11
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 01 Mar 2021 12:55:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
 bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=;
 b=fX3NDwT1/WqhBXGH/kwz7v+XmuezAwgousr5Z/AHmXJKOV+ADij30stGQ1THvUTKhh
 dBnn/M4RfQD64WfWPjdNUG0yt3LrJKJgT2pcMGPUh9bbusCIbu5pFdxVxz2vB/qZM7Nr
 P1/oCM+BX/eJ6ZtSkeRh6vrITReukGbol5e0k6mbxS/ddd92wJxo4VJ8IxxytukaZwJj
 X+QobceAR6kUgbngvbLwyPU25znTOUgb+plqnMigxti3vSRe+hgN7XYSMQEfXlCdGWpt
 1vCWuYma/b1Cy5pkMAC3zlC3ZLldcrPW2rNWFYn3G5N83ItaPhmfeDET3PrBmFVbzmxR
 jguQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to;
 bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=;
 b=ZDoXWoKywMS1ruweQAhHCnVicdDclRnRjwOHQCkbwPQq111BrpNO+ZQB26FR99TlPA
 fpIVw1EtbCJWpf7KI7BOXzQwSPFkYdz16yikIytV3nnob8czAwJe2M+EY0sWHHptymxE
 axklR4v+pTDS3JY0mAbG3VNLwH5MAP95HJwXGL7tVefKk5UA+mkzxtjAjDK5Sz/QrLqv
 HxUWl9dMPrLf9cfpZKHL7PewYWC0x83lfxON1dia754f08TofOrYoRx6Z9AnJbICcpaA
 G+zlUYIffPHG4rV0gDmrO88A9+dptX2dXS3EBK197QfJbNi64W3akLJpxFaXJxgPFThx
 SD0A==
X-Gm-Message-State: AOAM530tAp8KWhFz9P7B73wmg2K3ydszLOM73HOPQPS7zVK/c+GB50yG
 Q2xv/bTvb2S7Yn7rmP2fPRYW9vTu67OfzHUD9vUPMHonUnNgJQ==
X-Google-Smtp-Source: ABdhPJw660+u4kMrJZAQEDx6v52aZeTBSdoR/sYxWp3xhjpBvxneu9biH4EKb6VmmChZ1agDSOmDv05d0NbV3j5baQI=
X-Received: by 2002:a5d:4443:: with SMTP id x3mr3220394wrr.49.1614632121231;
 Mon, 01 Mar 2021 12:55:21 -0800 (PST)
MIME-Version: 1.0
References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
 <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
 <b895f2e4-513f-0c0d-91ac-52af055f332c@LeoWandersleb.de>
 <CAPR5oBND9DS1Ea=RMqrhnMBjFqU6wpCpv-sMeX9mxqgyLJGYiQ@mail.gmail.com>
 <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org>
In-Reply-To: <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org>
From: Keagan McClelland <keagan.mcclelland@gmail.com>
Date: Mon, 1 Mar 2021 13:55:10 -0700
Message-ID: <CALeFGL2=JLo4SZ4eeDWLbRRMhNV7T58-Pe2jYa6aC6XtPWmC9A@mail.gmail.com>
To: eric@voskuil.org, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000076db405bc7fd653"
X-Mailman-Approved-At: Mon, 01 Mar 2021 21:06:49 +0000
Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Mar 2021 20:55:24 -0000

--000000000000076db405bc7fd653
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

> Personally I consider this counterproductive. Apart from the complexity,
it=E2=80=99s not healthy. And the chain grows linearly with storage cost fa=
lling
exponentially, leading to a straightforward conclusion.

The motivation for this change is not to encourage full archival nodes to
prune, but to make it possible for pruned nodes to beef up what kind of
archive they retain. Personally I think using the falling storage costs as
a means of providing access to more users is more important than using it
to justify requiring higher node requirements.

> Something to consider adding to this proposal is to keep the idea of
pruning - i.e. retain a sequentially uninterrupted number of the most
recent blocks.
>
> Many users do not run a node for entirely altruistic reasons - they do
so, at least in part, because it allows them to use their wallets
privately. Without this ability, I think the number of users willing to run
their node in this configuration might be reduced.
>
> Another related thought is to have a decreasing density over blocks over
time as you go backwards towards genesis, in order for the data density of
the storage to match the actual usage of the network, in which (I would
imagine) more recent blocks are more heavily requested than early ones.

Per my above comments, this change is actually capitalizing primarily upon
those who wish to do it for more altruistic reasons. Furthermore, doing
linear block scans when you need to query blocks that you don't keep does
not leak privacy details in the same way that bloom filters do. You are not
signaling to the peer that there is something specific in that block that
you care about, because you don't actually know. You are signalling only
that you do not have that block right now, which from the other parts of
the design you are already leaking. In light of this, I don't think that it
is necessary for the blocks to be in sequential sets at all. If there is no
requirement on them being sequential, uniform randomness will take care of
the density problem automatically.

Keagan


On Mon, Mar 1, 2021 at 4:20 AM Eric Voskuil via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>
>
> > Only headers need to be downloaded sequentially so downloading relevant
> blocks from one node is totally possible with gaps in between.
>
>
>
> In fact this is exactly how libbitcoin v4 works. We download and store
> blocks in parallel. In the case of a restart block gaps are repopulated.
> Given that headers are validated, we go after the most responsive nodes.
> Based on standard deviation, we drop the slowest peers and rebalance load
> to new/empty channels. We make ordered but not necessarily sequential
> requests. There is no distinction between =E2=80=9Cinitial=E2=80=9D block=
 download, a
> restart, or a single or few blocks at the top. So it=E2=80=99s referred t=
o as
> continuous parallel block download.
>
>
>
> But we don=E2=80=99t prune. Personally I consider this counterproductive.=
 Apart
> from the complexity, it=E2=80=99s not healthy. And the chain grows linear=
ly with
> storage cost falling exponentially, leading to a straightforward conclusi=
on.
>
>
>
> e
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--000000000000076db405bc7fd653
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&gt; Personally I consider this counterproductive. Apart f=
rom the complexity, it=E2=80=99s not healthy. And the chain grows linearly =
with storage cost falling exponentially, leading to a straightforward concl=
usion.<div><br></div><div>The motivation for this change is not to encourag=
e full archival nodes to prune, but to make it possible for pruned nodes to=
 beef up what kind of archive they retain. Personally I think using the fal=
ling storage costs as a means of providing access to more users is more imp=
ortant than using it to justify requiring higher node requirements.</div><d=
iv><br></div><div>&gt; Something to consider adding to this proposal is to =
keep the idea of pruning - i.e. retain a sequentially uninterrupted number =
of the most recent blocks.<div>&gt;</div><div>&gt; Many users do not run a =
node for entirely altruistic reasons - they do so, at least in part, becaus=
e it allows them to use their wallets privately. Without this ability, I th=
ink the number of users willing to run their node in this configuration mig=
ht be reduced.</div><div>&gt;</div><div>&gt; Another related thought is to =
have a decreasing density over blocks over time as you go backwards towards=
 genesis, in order for the data density of the storage to match the actual =
usage of the network, in which (I would imagine) more recent blocks are mor=
e heavily requested than early ones.</div><div style=3D"color:rgb(136,136,1=
36)"><br></div><div style=3D""><font color=3D"#000000">Per my above comment=
s, this change is actually capitalizing primarily upon those who wish to do=
 it for more altruistic reasons. Furthermore, doing linear block scans when=
 you need to query blocks that you don&#39;t keep does not leak privacy det=
ails in the same way that bloom filters do. You are not signaling to the pe=
er that there is something specific in that block that you care about, beca=
use you don&#39;t actually know. You are signalling only that you do not ha=
ve that block right now, which from the other parts of the design you are a=
lready leaking. In light of this, I don&#39;t think that it is necessary fo=
r the blocks to be in sequential sets at all. If there is no requirement on=
 them being sequential, uniform randomness will take care of the density pr=
oblem automatically.</font></div><div style=3D""><font color=3D"#000000"><b=
r></font></div><div style=3D""><font color=3D"#000000">Keagan</font></div><=
div style=3D"color:rgb(136,136,136)"><br></div></div></div><br><div class=
=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 1, 2021 =
at 4:20 AM Eric Voskuil via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@l=
ists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt; wro=
te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang=3D=
"EN-US" style=3D"overflow-wrap: break-word;"><div class=3D"gmail-m_17503728=
01160849895WordSection1"><div><div><p class=3D"MsoNormal">On Sun, Feb 28, 2=
021 at 10:18 AM Leo Wandersleb via bitcoin-dev &lt;<a href=3D"mailto:bitcoi=
n-dev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxf=
oundation.org</a>&gt; wrote:<u></u><u></u></p><p class=3D"MsoNormal"><u></u=
>=C2=A0<u></u></p><p class=3D"MsoNormal">&gt; Only headers need to be downl=
oaded sequentially so downloading relevant blocks from one node is totally =
possible with gaps in between.<u></u><u></u></p><p class=3D"MsoNormal"><u><=
/u>=C2=A0<u></u></p><p class=3D"MsoNormal">In fact this is exactly how libb=
itcoin v4 works. We download and store blocks in parallel. In the case of a=
 restart block gaps are repopulated. Given that headers are validated, we g=
o after the most responsive nodes. Based on standard deviation, we drop the=
 slowest peers and rebalance load to new/empty channels. We make ordered bu=
t not necessarily sequential requests. There is no distinction between =E2=
=80=9Cinitial=E2=80=9D block download, a restart, or a single or few blocks=
 at the top. So it=E2=80=99s referred to as continuous parallel block downl=
oad.<u></u><u></u></p><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><p cla=
ss=3D"MsoNormal">But we don=E2=80=99t prune. Personally I consider this cou=
nterproductive. Apart from the complexity, it=E2=80=99s not healthy. And th=
e chain grows linearly with storage cost falling exponentially, leading to =
a straightforward conclusion.<u></u><u></u></p><p class=3D"MsoNormal"><u></=
u>=C2=A0<u></u></p><p class=3D"MsoNormal">e<u></u><u></u></p></div></div></=
div></div>_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>

--000000000000076db405bc7fd653--