path: root/eb/8fecbfc20a15bf6f66270066975685ee0e5221

Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
	helo=mx.sourceforge.net)
	by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <etotheipi@gmail.com>) id 1SarvP-0001Jj-3T
	for bitcoin-development@lists.sourceforge.net;
	Sat, 02 Jun 2012 17:15:39 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.212.47 as permitted sender)
	client-ip=209.85.212.47; envelope-from=etotheipi@gmail.com;
	helo=mail-vb0-f47.google.com; 
Received: from mail-vb0-f47.google.com ([209.85.212.47])
	by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1SarvL-0006Ux-7s
	for bitcoin-development@lists.sourceforge.net;
	Sat, 02 Jun 2012 17:15:39 +0000
Received: by vbbfr13 with SMTP id fr13so2196504vbb.34
	for <bitcoin-development@lists.sourceforge.net>;
	Sat, 02 Jun 2012 10:15:29 -0700 (PDT)
Received: by 10.52.22.38 with SMTP id a6mr6213436vdf.37.1338657329640;
	Sat, 02 Jun 2012 10:15:29 -0700 (PDT)
Received: from [192.168.1.85] (c-76-111-96-126.hsd1.md.comcast.net.
	[76.111.96.126])
	by mx.google.com with ESMTPS id k4sm8770325vdi.6.2012.06.02.10.15.28
	(version=SSLv3 cipher=OTHER); Sat, 02 Jun 2012 10:15:28 -0700 (PDT)
Message-ID: <4FCA4A2E.3000900@gmail.com>
Date: Sat, 02 Jun 2012 13:15:26 -0400
From: Alan Reiner <etotheipi@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64;
	rv:12.0) Gecko/20120428 Thunderbird/12.0.1
MIME-Version: 1.0
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
References: <89639546-2608-44B2-AF84-E600A33CB514@jrbobdobbs.org>
In-Reply-To: <89639546-2608-44B2-AF84-E600A33CB514@jrbobdobbs.org>
X-Forwarded-Message-Id: <89639546-2608-44B2-AF84-E600A33CB514@jrbobdobbs.org>
Content-Type: multipart/alternative;
	boundary="------------040300020902000005080508"
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(etotheipi[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1SarvL-0006Ux-7s
Subject: [Bitcoin-development] Fwd: Re: Full Clients in the future -
 Blockchain management
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Sat, 02 Jun 2012 17:15:39 -0000

This is a multi-part message in MIME format.
--------------040300020902000005080508
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

(response from Doug forwarded below)

It's a very good point.  I have no experience with database engines.  I 
had assumed that in most cases, data could always be indexed in RAM, and 
wanted to know where to go when that's not the case.  I will look into 
that solution, further.

I am very interested to solve the blockchain compression problem, and 
think I've got a great way that will not just compress the blockchain, 
but improve the network for lightweight clients.  But the idea is not 
fully formed yet, so I was holding off...



-------- Original Message --------
Subject: 	Re: [Bitcoin-development] Full Clients in the future - 
Blockchain management
Date: 	Sat, 2 Jun 2012 12:07:44 -0500
From: 	Douglas Huff <mith@jrbobdobbs.org>
To: 	Alan Reiner <etotheipi@gmail.com>



I think you're trying to solve something a little out of scope, really. 
Most of the issues aren't really issues for other clients using 
established storage mechanisms (bdb,SQLite,etc) and they're using them 
precisely because this is a problem that people have been working on for 
decades and a poor candidate for reinvention.

If you really look at what you're proposing it's fundamentally how bdb 
operates except your indexing format is usage domain specific and you're 
in charge of all the resource management semantics. While at the same 
time you'll be missing many of the newer features that make working 
with/recovering/diagnosing issues in the storage layer easier.

If you're really wanting to talk about pruning methods to prevent the 
massive amount of duplicated; but no longer pertinent, data that's a 
different story and please continue. :)

-- 
Douglas Huff

On Jun 2, 2012, at 10:40, Alan Reiner <etotheipi@gmail.com 
<mailto:etotheipi@gmail.com>> wrote:

> Devs,
>
> I have decided to upgrade Armory's blockchain utilities, partly out of 
> necessity due to a poor code decision I made before I even decided I 
> was making a client.  In an effort to avoid such mistakes again, I 
> want to do it "right" this time around, and realize that this is a 
> good discussion for all the devs that will have to deal with this 
> eventually...
>
> The part I'm having difficulty with, is the idea that in a few years 
> from now, it just may not be feasible to hold transactions 
> file-/pointers/ in RAM, because even that would overwhelm standard RAM 
> sizes.  Without any degree of blockchain compression, I see that the 
> most general, scalable solution is probably a complicated one.
>
> On the other hand, where this fails may be where we have already 
> predicted that the network will have to split into "super-nodes" and 
> "lite nodes."  In which case, this discussion is still a good one, but 
> just directed more towards the super-nodes.  But, there may still be a 
> point at which super-nodes don't have enough RAM to hold this data...
>
> (1)  As for how small you can get the data:  my original idea was that 
> the entire blockchain is stored on disk as blkXXXX.dat files.  I store 
> all transactions as 10-byte "file-references."  10 bytes would be
>
>     -- X in blkX.dat (2 bytes)
>     -- Tx start byte (4 bytes)
>     -- Tx size bytes (4 bytes)
>
> The file-refs would be stored in a multimap indexed by the first 6 
> bytes of the tx-hash.  In this way, when I search the multimap, I 
> potentially get a list of file-refs, and I might have to retrieve a 
> couple of tx from disk before finding the right one, but it would be a 
> good trade-off compared to storing all 32 bytes (that's assuming that 
> multimap nodes don't have too much overhead).
>
> But even with this, if there are 1,000,000,000 transactions in the 
> blockchain, each node is probably 48 bytes  (16 bytes + map/container 
> overhead), then you're talking about 48 GB to track all the data in 
> RAM.  mmap() may help here, but I'm not sure it's the right solution
>
> (2) What other ways are there, besides some kind of blockchain 
> compression, to maintain a multi-terabyte blockchain, assuming that 
> storing references to each tx would overwhelm available RAM?   Maybe 
> that assumption isn't necessary, but I think it prepares for the worst.
>
> Or maybe I'm too narrow in my focus.  How do other people envision 
> this will be handled in the future.  I've heard so many vague notions 
> of "well we could do /this/ or /that/, or it wouldn't be hard to do 
> /that/" but I haven't heard any serious proposals for it.  And while I 
> believe that blockchain compression will become ubiquitous in the 
> future, not everyone believes that, and there will undoubtedly be 
> users/devs that /want/ to maintain everything under all circumstances.
>
> -Alan
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net 
> <mailto:Bitcoin-development@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

--------------040300020902000005080508
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    (response from Doug forwarded below)<br>
    <br>
    It's a very good point.  I have no experience with database
    engines.  I had assumed that in most cases, data could always be
    indexed in RAM, and wanted to know where to go when that's not the
    case.  I will look into that solution, further.  <br>
    <br>
    I am very interested to solve the blockchain compression problem,
    and think I've got a great way that will not just compress the
    blockchain, but improve the network for lightweight clients.  But
    the idea is not fully formed yet, so I was holding off...<br>
    <br>
    <br>
    <br>
    -------- Original Message --------
    <table class="moz-email-headers-table" border="0" cellpadding="0"
      cellspacing="0">
      <tbody>
        <tr>
          <th valign="BASELINE" nowrap="nowrap" align="RIGHT">Subject: </th>
          <td>Re: [Bitcoin-development] Full Clients in the future -
            Blockchain management</td>
        </tr>
        <tr>
          <th valign="BASELINE" nowrap="nowrap" align="RIGHT">Date: </th>
          <td>Sat, 2 Jun 2012 12:07:44 -0500</td>
        </tr>
        <tr>
          <th valign="BASELINE" nowrap="nowrap" align="RIGHT">From: </th>
          <td>Douglas Huff <a class="moz-txt-link-rfc2396E" href="mailto:mith@jrbobdobbs.org">&lt;mith@jrbobdobbs.org&gt;</a></td>
        </tr>
        <tr>
          <th valign="BASELINE" nowrap="nowrap" align="RIGHT">To: </th>
          <td>Alan Reiner <a class="moz-txt-link-rfc2396E" href="mailto:etotheipi@gmail.com">&lt;etotheipi@gmail.com&gt;</a></td>
        </tr>
      </tbody>
    </table>
    <br>
    <br>
    <div>I think you're trying to solve something a little out of scope,
      really. Most of the issues aren't really issues for other clients
      using established storage mechanisms (bdb,SQLite,etc) and they're
      using them precisely because this is a problem that people have
      been working on for decades and a poor candidate for reinvention.</div>
    <div><br>
    </div>
    <div>If you really look at what you're proposing it's fundamentally
      how bdb operates except your indexing format is usage domain
      specific and you're in charge of all the resource management
      semantics. While at the same time you'll be missing many of the
      newer features that make working with/recovering/diagnosing issues
      in the storage layer easier. </div>
    <div><br>
    </div>
    <div>If you're really wanting to talk about pruning methods to
      prevent the massive amount of duplicated; but no longer pertinent,
      data that's a different story and please continue. :)<br>
      <br>
      -- 
      <div>Douglas Huff</div>
    </div>
    <div><br>
      On Jun 2, 2012, at 10:40, Alan Reiner &lt;<a
        moz-do-not-send="true" href="mailto:etotheipi@gmail.com">etotheipi@gmail.com</a>&gt;
      wrote:<br>
      <br>
    </div>
    <div><span></span></div>
    <blockquote type="cite">
      <div>
        <meta http-equiv="content-type" content="text/html;
          charset=UTF-8">
        Devs,<br>
        <br>
        I have decided to upgrade Armory's blockchain utilities, partly
        out of necessity due to a poor code decision I made before I
        even decided I was making a client.  In an effort to avoid such
        mistakes again, I want to do it "right" this time around, and
        realize that this is a good discussion for all the devs that
        will have to deal with this eventually...<br>
        <br>
        The part I'm having difficulty with, is the idea that in a few
        years from now, it just may not be feasible to hold transactions
        file-<i>pointers</i> in RAM, because even that would overwhelm
        standard RAM sizes.  Without any degree of blockchain
        compression, I see that the most general, scalable solution is
        probably a complicated one.<br>
        <br>
        On the other hand, where this fails may be where we have already
        predicted that the network will have to split into "super-nodes"
        and "lite nodes."  In which case, this discussion is still a
        good one, but just directed more towards the super-nodes.  But,
        there may still be a point at which super-nodes don't have
        enough RAM to hold this data...<br>
        <br>
        (1)  As for how small you can get the data:  my original idea
        was that the entire blockchain is stored on disk as blkXXXX.dat
        files.  I store all transactions as 10-byte "file-references." 
        10 bytes would be <br>
        <br>
            -- X in blkX.dat (2 bytes) <br>
            -- Tx start byte (4 bytes) <br>
            -- Tx size bytes (4 bytes)<br>
        <br>
        The file-refs would be stored in a multimap indexed by the first
        6 bytes of the tx-hash.  In this way, when I search the
        multimap, I potentially get a list of file-refs, and I might
        have to retrieve a couple of tx from disk before finding the
        right one, but it would be a good trade-off compared to storing
        all 32 bytes (that's assuming that multimap nodes don't have too
        much overhead). <br>
        <br>
        But even with this, if there are 1,000,000,000 transactions in
        the blockchain, each node is probably 48 bytes  (16 bytes +
        map/container overhead), then you're talking about 48 GB to
        track all the data in RAM.  mmap() may help here, but I'm not
        sure it's the right solution<br>
        <br>
        (2) What other ways are there, besides some kind of blockchain
        compression, to maintain a multi-terabyte blockchain, assuming
        that storing references to each tx would overwhelm available
        RAM?   Maybe that assumption isn't necessary, but I think it
        prepares for the worst.<br>
        <br>
        Or maybe I'm too narrow in my focus.  How do other people
        envision this will be handled in the future.  I've heard so many
        vague notions of "well we could do <i>this</i> or <i>that</i>,
        or it wouldn't be hard to do <i>that</i>" but I haven't heard
        any serious proposals for it.  And while I believe that
        blockchain compression will become ubiquitous in the future, not
        everyone believes that, and there will undoubtedly be users/devs
        that <i>want</i> to maintain everything under all
        circumstances.<br>
        <br>
        -Alan<br>
      </div>
    </blockquote>
    <blockquote type="cite">
      <div><span>------------------------------------------------------------------------------</span><br>
        <span>Live Security Virtual Conference</span><br>
        <span>Exclusive live event will cover all the ways today's
          security and </span><br>
        <span>threat landscape has changed and how IT managers can
          respond. Discussions </span><br>
        <span>will include endpoint security, mobile security and the
          latest in malware </span><br>
        <span>threats. <a moz-do-not-send="true"
            href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a></span></div>
    </blockquote>
    <blockquote type="cite">
      <div><span>_______________________________________________</span><br>
        <span>Bitcoin-development mailing list</span><br>
        <span><a moz-do-not-send="true"
            href="mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-development@lists.sourceforge.net</a></span><br>
        <span><a moz-do-not-send="true"
            href="https://lists.sourceforge.net/lists/listinfo/bitcoin-development">https://lists.sourceforge.net/lists/listinfo/bitcoin-development</a></span><br>
      </div>
    </blockquote>
  </body>
</html>

--------------040300020902000005080508--