Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1WYHlT-0005Ra-R2 for bitcoin-development@lists.sourceforge.net; Thu, 10 Apr 2014 16:23:47 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.192.175 as permitted sender) client-ip=209.85.192.175; envelope-from=brianchoffman@gmail.com; helo=mail-pd0-f175.google.com; Received: from mail-pd0-f175.google.com ([209.85.192.175]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1WYHlS-000671-NG for bitcoin-development@lists.sourceforge.net; Thu, 10 Apr 2014 16:23:47 +0000 Received: by mail-pd0-f175.google.com with SMTP id x10so4070254pdj.20 for ; Thu, 10 Apr 2014 09:23:40 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.68.136.226 with SMTP id qd2mr5987684pbb.72.1397147020788; Thu, 10 Apr 2014 09:23:40 -0700 (PDT) Received: by 10.70.89.237 with HTTP; Thu, 10 Apr 2014 09:23:40 -0700 (PDT) In-Reply-To: References: Date: Thu, 10 Apr 2014 12:23:40 -0400 Message-ID: From: Brian Hoffman To: Wladimir Content-Type: multipart/alternative; boundary=e89a8f234a39df8cc204f6b2a33e X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (brianchoffman[at]gmail.com) -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [209.85.192.175 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1WYHlS-000671-NG Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Chain pruning X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2014 16:23:48 -0000 --e89a8f234a39df8cc204f6b2a33e Content-Type: text/plain; charset=UTF-8 This is probably just noise, but what if nodes could compress and store earlier transaction sets (archive sets) and serve them up conditionally. So if there were let's say 100 archive sets of (10,000 blocks) you might have 5 open at any time when you're an active archive node while the others sit on your disk compressed and unavailable to the network. This would allow nodes to have all full transactions but conserve disk space and network activity since they wouldn't ever respond about every possible transaction. This could be based on a rotational request period, based on request count or done periodically. Once their considered active they would be expected to uncompress a set and make it available to the network. Clients would have to piece together archive sets from different nodes, but if there weren't enough archive nodes to cover the chain they could ratchet up the amount of required open archive sets when your node was active. I fully expect to have my idea trashed, but I'm dipping toes in the waters of contribution. On Thu, Apr 10, 2014 at 10:19 AM, Wladimir wrote: > > On Thu, Apr 10, 2014 at 2:10 PM, Gregory Maxwell wrote: > >> But sure I could see a fixed range as also being a useful contribution >> though I'm struggling to figure out what set of constraints would >> leave a node without following the consensus? Obviously it has >> bandwidth if you're expecting to contribute much in serving those >> historic blocks... and verifying is reasonably cpu cheap with fast >> ecdsa code. Maybe it has a lot of read only storage? >> > > The use case is that you could burn the node implementation + block data + > a live operating system on a read-only medium. This could be set in stone > for a long time. > > There would be no consensus code to keep up to date with protocol > developments, because it doesn't take active part in it. > > I don't think it would be terribly useful right now, but it could be > useful when nodes that host all history become rare. It'd allow > distributing 'pieces of history' in a self-contained form. > > >> I think it should be possible to express and use such a thing in the >> protocol even if I'm currently unsure as to why you wouldn't do 100000 >> - 200000 _plus_ the most recent 144 that you were already keeping >> around for reorgs. >> > > Yes, it would be nice to at least be able to express it, if it doesn't > make the protocol too finicky. > > In terms of peer selection, if the blocks you need aren't covered by >> the nodes you're currently connected to I think you'd prefer to seek >> node nodes which have the least rare-ness in the ranges they offer. >> E.g. if you're looking for a block 50 from the tip, you're should >> probably not prefer to fetch it from someone with blocks 100000-150000 >> if its one of only 100 nodes that has that range. >> > > That makes sense. > > In general, if you want a block 50 from the tip, it would be best to > request it from a node that only serves the last N (N>~50) blocks, and not > a history node that could use the same bandwidth to serve earlier, rarer > blocks to others. > > Wladimir > > > > ------------------------------------------------------------------------------ > Put Bad Developers to Shame > Dominate Development with Jenkins Continuous Integration > Continuously Automate Build, Test & Deployment > Start a new project now. Try Jenkins in the cloud. > http://p.sf.net/sfu/13600_Cloudbees > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > --e89a8f234a39df8cc204f6b2a33e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This is probably just noise, but what if nodes could compr= ess and store earlier transaction sets (archive sets) and serve them up con= ditionally. So if there were let's say 100 archive sets of (10,000 bloc= ks) you might have 5 open at any time when you're an active archive nod= e while the others sit on your disk compressed and unavailable to the netwo= rk. This would allow nodes to have all full transactions but conserve disk = space and network activity since they wouldn't ever respond about every= possible transaction.

This could be based on a rotational request period, based on= request count or done periodically. Once their considered active they woul= d be expected to uncompress a set and make it available to the network. Cli= ents would have to piece together archive sets from different nodes, but if= there weren't enough archive nodes to cover the chain they could ratch= et up the amount of required open archive sets when your node was active.

I fully expect to have my idea trashed, but I'm dip= ping toes in the waters of contribution.




On Thu= , Apr 10, 2014 at 10:19 AM, Wladimir <laanwj@gmail.com> wrote= :
=
On Thu, Apr 10, 2014 at 2:10 PM, Gregory Max= well <gmaxwell@gmail.com> wrote:
But sure I could see a fixed range as also b= eing a useful contribution
though I'm struggling to figure out what set of constraints would
leave a node without following the consensus? =C2=A0 Obviously it has
bandwidth if you're expecting to contribute much in serving those
historic blocks... and verifying is reasonably cpu cheap with fast
ecdsa code. =C2=A0 Maybe it has a lot of read only storage?

The use case is that you could burn the node implementation + blo= ck data + a live operating system on a read-only medium. This could be set = in stone for a long time.

There would be no consensus code to keep up to date with pro= tocol developments, because it doesn't take active part in it.

I don't think it would be terribly useful right no= w, but it could be useful when nodes that host all history become rare. It&= #39;d allow distributing 'pieces of history' in a self-contained fo= rm.
=C2=A0
I think it should be possible to express and use such a thing in the
protocol even if I'm currently unsure as to why you wouldn't do 100= 000
- 200000 =C2=A0_plus_ the most recent 144 that you were already keeping
around for reorgs.

Yes, it would be nic= e to at least be able to express it, if it doesn't make the protocol to= o finicky.

In terms of peer selection, if the blocks you need aren't covered by the nodes you're currently connected to I think you'd prefer to see= k
node nodes which have the least rare-ness in the ranges they offer.
E.g. if you're looking for a block 50 from the tip, =C2=A0you're sh= ould
probably not prefer to fetch it from someone with blocks 100000-150000
if its one of only 100 nodes that has that range.

=
That makes sense.

In general, if you want a block 50 fro= m the tip, it would be best to request it from a node that only serves the = last N (N>~50) blocks, and not a history node that could use the same ba= ndwidth to serve earlier, rarer blocks to others.
Wladimir


-----------------------------------------------------------------------= -------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment
Start a new project now. Try Jenkins in the cloud.
http://p.= sf.net/sfu/13600_Cloudbees
_________________________________________= ______
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment


--e89a8f234a39df8cc204f6b2a33e--