Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.212.47 as permitted sender)
	client-ip=209.85.212.47; envelope-from=etotheipi@gmail.com;
	helo=mail-vb0-f47.google.com; 
Message-ID: <4FCA33EB.5030706@gmail.com>
Date: Sat, 02 Jun 2012 11:40:27 -0400
From: Alan Reiner <etotheipi@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64;
	rv:12.0) Gecko/20120428 Thunderbird/12.0.1
MIME-Version: 1.0
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative;
	boundary="------------010405000806020803010308"
Subject: [Bitcoin-development] Full Clients in the future - Blockchain
	management
Precedence: list

This is a multi-part message in MIME format.
--------------010405000806020803010308
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Devs,

I have decided to upgrade Armory's blockchain utilities, partly out of 
necessity due to a poor code decision I made before I even decided I was 
making a client.  In an effort to avoid such mistakes again, I want to 
do it "right" this time around, and realize that this is a good 
discussion for all the devs that will have to deal with this eventually...

The part I'm having difficulty with, is the idea that in a few years 
from now, it just may not be feasible to hold transactions 
file-/pointers/ in RAM, because even that would overwhelm standard RAM 
sizes.  Without any degree of blockchain compression, I see that the 
most general, scalable solution is probably a complicated one.

On the other hand, where this fails may be where we have already 
predicted that the network will have to split into "super-nodes" and 
"lite nodes."  In which case, this discussion is still a good one, but 
just directed more towards the super-nodes.  But, there may still be a 
point at which super-nodes don't have enough RAM to hold this data...

(1)  As for how small you can get the data:  my original idea was that 
the entire blockchain is stored on disk as blkXXXX.dat files.  I store 
all transactions as 10-byte "file-references."  10 bytes would be

     -- X in blkX.dat (2 bytes)
     -- Tx start byte (4 bytes)
     -- Tx size bytes (4 bytes)

The file-refs would be stored in a multimap indexed by the first 6 bytes 
of the tx-hash.  In this way, when I search the multimap, I potentially 
get a list of file-refs, and I might have to retrieve a couple of tx 
from disk before finding the right one, but it would be a good trade-off 
compared to storing all 32 bytes (that's assuming that multimap nodes 
don't have too much overhead).

But even with this, if there are 1,000,000,000 transactions in the 
blockchain, each node is probably 48 bytes  (16 bytes + map/container 
overhead), then you're talking about 48 GB to track all the data in 
RAM.  mmap() may help here, but I'm not sure it's the right solution

(2) What other ways are there, besides some kind of blockchain 
compression, to maintain a multi-terabyte blockchain, assuming that 
storing references to each tx would overwhelm available RAM?   Maybe 
that assumption isn't necessary, but I think it prepares for the worst.

Or maybe I'm too narrow in my focus.  How do other people envision this 
will be handled in the future.  I've heard so many vague notions of 
"well we could do /this/ or /that/, or it wouldn't be hard to do /that/" 
but I haven't heard any serious proposals for it.  And while I believe 
that blockchain compression will become ubiquitous in the future, not 
everyone believes that, and there will undoubtedly be users/devs that 
/want/ to maintain everything under all circumstances.

-Alan

--------------010405000806020803010308
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Devs,<br>
    <br>
    I have decided to upgrade Armory's blockchain utilities, partly out
    of necessity due to a poor code decision I made before I even
    decided I was making a client.&nbsp; In an effort to avoid such mistakes
    again, I want to do it "right" this time around, and realize that
    this is a good discussion for all the devs that will have to deal
    with this eventually...<br>
    <br>
    The part I'm having difficulty with, is the idea that in a few years
    from now, it just may not be feasible to hold transactions file-<i>pointers</i>
    in RAM, because even that would overwhelm standard RAM sizes.&nbsp;
    Without any degree of blockchain compression, I see that the most
    general, scalable solution is probably a complicated one.<br>
    <br>
    On the other hand, where this fails may be where we have already
    predicted that the network will have to split into "super-nodes" and
    "lite nodes."&nbsp; In which case, this discussion is still a good one,
    but just directed more towards the super-nodes.&nbsp; But, there may
    still be a point at which super-nodes don't have enough RAM to hold
    this data...<br>
    <br>
    (1)&nbsp; As for how small you can get the data:&nbsp; my original idea was
    that the entire blockchain is stored on disk as blkXXXX.dat files.&nbsp;
    I store all transactions as 10-byte "file-references."&nbsp; 10 bytes
    would be <br>
    <br>
    &nbsp;&nbsp;&nbsp; -- X in blkX.dat (2 bytes) <br>
    &nbsp;&nbsp;&nbsp; -- Tx start byte (4 bytes) <br>
    &nbsp;&nbsp;&nbsp; -- Tx size bytes (4 bytes)<br>
    <br>
    The file-refs would be stored in a multimap indexed by the first 6
    bytes of the tx-hash.&nbsp; In this way, when I search the multimap, I
    potentially get a list of file-refs, and I might have to retrieve a
    couple of tx from disk before finding the right one, but it would be
    a good trade-off compared to storing all 32 bytes (that's assuming
    that multimap nodes don't have too much overhead). <br>
    <br>
    But even with this, if there are 1,000,000,000 transactions in the
    blockchain, each node is probably 48 bytes&nbsp; (16 bytes +
    map/container overhead), then you're talking about 48 GB to track
    all the data in RAM.&nbsp; mmap() may help here, but I'm not sure it's
    the right solution<br>
    <br>
    (2) What other ways are there, besides some kind of blockchain
    compression, to maintain a multi-terabyte blockchain, assuming that
    storing references to each tx would overwhelm available RAM? &nbsp; Maybe
    that assumption isn't necessary, but I think it prepares for the
    worst.<br>
    <br>
    Or maybe I'm too narrow in my focus.&nbsp; How do other people envision
    this will be handled in the future.&nbsp; I've heard so many vague
    notions of "well we could do <i>this</i> or <i>that</i>, or it
    wouldn't be hard to do <i>that</i>" but I haven't heard any serious
    proposals for it.&nbsp; And while I believe that blockchain compression
    will become ubiquitous in the future, not everyone believes that,
    and there will undoubtedly be users/devs that <i>want</i> to
    maintain everything under all circumstances.<br>
    <br>
    -Alan<br>
  </body>
</html>

--------------010405000806020803010308--