Return-Path: Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2D42BC0001 for ; Sun, 28 Feb 2021 03:48:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1AC326F4B8 for ; Sun, 28 Feb 2021 03:48:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -0.148 X-Spam-Level: X-Spam-Status: No, score=-0.148 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KHOP_HELO_FCRDNS=0.001, NICE_REPLY_A=-0.35, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=ham autolearn_force=no Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=neutral reason="invalid (public key: not available)" header.d=leowandersleb.de Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DgSr1-rxpfrE for ; Sun, 28 Feb 2021 03:48:41 +0000 (UTC) X-Greylist: delayed 00:07:27 by SQLgrey-1.8.0 Received: from geekbox.info (mx01.geekbox.info [95.216.118.16]) by smtp3.osuosl.org (Postfix) with ESMTPS id AE1CF6F4A1 for ; Sun, 28 Feb 2021 03:48:40 +0000 (UTC) Received: from [192.168.0.6] (pc-180-175-104-200.cm.vtr.net [200.104.175.180]) (Authenticated sender: leo) by geekbox.info (Postfix) with ESMTPSA id EB3CC7000E8 for ; Sun, 28 Feb 2021 03:41:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=leowandersleb.de; s=mail; t=1614483671; bh=siAMY4VuDYminTEQrvZyiTluxbJnsHdIV3mudl7uTqk=; h=Subject:To:References:From:Date:In-Reply-To; b=pf3Xal3lxu57EfFlM8p71uYTv9N9YRYL6D8zsZNpuvYjsQ/k2nfqKua2u+gJP6uG3 fZKW3Mvh+ejPrfh4UVJJqi6WdZTHJXKmT9nwGNA97Dt4+7Zr1cxbsaE6C/R7BouwM+ GCE10dvDVcjpPRKNktRKX3c+oAYjXNbhMv8rGKj8= To: bitcoin-dev@lists.linuxfoundation.org References: From: Leo Wandersleb Autocrypt: addr=leo@LeoWandersleb.de; prefer-encrypt=mutual; keydata= mQENBE3ePdoBCADZmTvPyledh9wmNRE+i5k7dUmegSmBpe1kMgzlt3nxwzyE6l3wzT9wen8E clmA4ZCLn77DjxF3Dg9X0sRio9No5r4u6luI0CzM7axvTi+5DWU25b+JGGdhKTMkyfyKBZdT 4xvjLzW9tT7upUuAh9nTcz9MBZS3vhKB1SqiBUdYqj33+2erIQzaYMIZ/crR6FbgDT5dI3gg 2YKJn4rdFy9saA0qngoDPkyU9u5TYtFk8zOreQrFXmXpLiptoxRolF9tqHyn51H4uV8ggIGc xJWRW/i2vBFHSM6ZD7PceWCRU5ehf3h/wat1xtn/H7AB+wcywkYrdg1TLifftwcKYBlzABEB AAG0JUxlbyBXYW5kZXJzbGViIDxMZW9ATGVvV2FuZGVyc2xlYi5kZT6JATgEEwECACIFAk3e PdoCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEGuaHwy3wggSrqQH/Amh6+kYnIUM 8X5yicpXd37h/Nqnrs6KIE0nYCZe03tkQytndbWXmf2pBth6CEzqrS+wivU7L8dPdiFQtdEW dDe8bjJ993TsVSyYrPHv549vD4DbjtB8WS7KEkMhhl+uVaC07YXNwOTAkshv4l3G2ll6dmBn LDeF8kSU+DZ1eNzUYHjuUcQYl0XAK5wE7rVpzZUlR0Z5bKNm8QmUPgrZPu6cVRZ6tOD0EMR6 3dcGI5x0fB35gEUasgNYxQ/g2arh66Pg45olKT1bTcrTJYrE8IePElSCIFDrzrZMhStY3TbQ qI3T0vsFGzEeU9M6LmKV8aii4g8o7af0kr4vYS3Nsp65AQ0ETd492gEIAMnahwe+MNliWgpF B+UfrwzfgiLjtizH307HYB67THS4h2+MoBJUAqrV7CQkZ8kMYLBl4D+hvxrXDYte6wTb0kcY X4NTGof8EDBPxUERYJlWbTbLccbA8d3ia++aTtqd9yRHobK8AO4R22cslYV/nQChHc+1/NFV 8V1O67U7XpnxQMUzYIhhvVcUwj+8bw/F3zPzS03PWnDv+gmwHt63w72Dth+P0F93/rI+OiEA kC/7//1ngQ7auqOeWVdqR8Z1s7tI4kxsVkJfqj3wVu+qaso8FpjNGs888xH8xhgXlll8APWj V9Rl3GtpRTdobzNh58jUfGmHh5bcRrYTrtldDwkAEQEAAYkBHwQYAQIACQUCTd492gIbDAAK CRBrmh8Mt8IIEi3MB/0UlpEORi7qBOH9egn/8E9fXj3QGpmQ7IAvzvLjkp3cArerBhObhGJ+ lZ679/9KZ4FxWLDStzGJH3thQstZjcUZzLwTP1sNdY+uPkvOqrzclfPK13z5hJhORD8W0brB UptTvyZthIxNIbY4uSbrvaAJmcPFi3cdKOjZz9XyH8AAxXcWGUjQuqxEc0GuM8h8asaUJesB bE4J+HPZXcAOC3D+qIqvM438Nbb8Pu7yYWrdq5kmbDMg2nPHXUOOovEsXWBGjWEiXWVUei75 5Fumqg5mz4P8/Ry3DwZBuHN3Op77gXogvyeE/B5O6dZzWWpSqBIS6MQcLD1divwi/2vNS4c4 Message-ID: Date: Sun, 28 Feb 2021 00:41:06 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Mailman-Approved-At: Sun, 28 Feb 2021 08:18:08 +0000 Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2021 03:48:43 -0000 Only headers need to be downloaded sequentially so downloading relevant blocks from one node is totally possible with gaps in between. On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote: > Hi Keagan, > > I had a very similar idea. The only difference being for the node to decide on > a range of blocks to keep beforehand, rather than making the decision > block-by-block like you suggest. > > I felt the other nodes would be better served by ranges due to the sequential > nature of IBD. Perhaps this would be computationally lighter as well. > > I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT > scheme to solve this same problem. > > Cheers, > Igor > > [1] https://arxiv.org/abs/1902.02174 > > On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev > > wrote: > > Hi all, > > I've been thinking for quite some time about the problem of pruned nodes > and ongoing storage costs for full nodes. One of the things that strikes > me as odd is that we only really have two settings. > > A. Prune everything except the most recent blocks, down to the cache size > B. Keep everything since genesis > > From my observations and conversations with various folks in the > community, they would like to be able to run a "partially" pruned node to > help bear the load of bootstrapping other nodes and helping with data > redundancy in the network, but would prefer to not dedicate hundreds of > Gigabytes of storage space to the cause. > > This led me to the idea that a node could randomly prune some of the > blocks from history if it passed some predicate. A rough sketch of this > would look as follows. > > 1. At node startup, it would generate a random seed, this would be unique > to the node but not necessary that it be cryptographically secure. > 2. In the node configuration it would also carry a "threshold" expressed > as some percentage of blocks it wanted to keep. > 3. As IBD occurs, based off of the threshold, the block hash, and the > node's unique seed, the node would either decide to prune the data or keep > it. The uniqueness of the node's hash should ensure that no block is > systematically overrepresented in the set of nodes choosing this storage > scheme. > 4. Once the node's IBD is complete it would advertise this as a peer > service, advertising its seed and threshold, so that nodes could > deterministically deduce which of its peers had which blocks. > > The goals are to increase data redundancy in a way that more uniformly > shares the load across nodes, alleviating some of the pressure of full > archive nodes on the IBD problem. I am working on a draft BIP for this > proposal but figured I would submit it as a high level idea in case anyone > had any feedback on the initial design before I go into specification > levels of detail. > > If you have thoughts on > > A. The protocol design itself > B. The barriers to put this kind of functionality into Core > > I would love to hear from you, > > Cheers, > Keagan > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > > -- > *Igor Cota* > Codex Apertus d.o.o. > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev