Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1TPhba-0007id-ON for bitcoin-development@lists.sourceforge.net; Sat, 20 Oct 2012 22:33:18 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.219.47 as permitted sender) client-ip=209.85.219.47; envelope-from=pieter.wuille@gmail.com; helo=mail-oa0-f47.google.com; Received: from mail-oa0-f47.google.com ([209.85.219.47]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1TPhba-0003iF-2M for bitcoin-development@lists.sourceforge.net; Sat, 20 Oct 2012 22:33:18 +0000 Received: by mail-oa0-f47.google.com with SMTP id h1so1562398oag.34 for ; Sat, 20 Oct 2012 15:33:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.216.71 with SMTP id oo7mr3593651obc.70.1350772392454; Sat, 20 Oct 2012 15:33:12 -0700 (PDT) Received: by 10.76.143.38 with HTTP; Sat, 20 Oct 2012 15:33:12 -0700 (PDT) Date: Sun, 21 Oct 2012 00:33:12 +0200 Message-ID: From: Pieter Wuille To: Bitcoin Dev Content-Type: text/plain; charset=ISO-8859-1 X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (pieter.wuille[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1TPhba-0003iF-2M Subject: [Bitcoin-development] Ultraprune merged in mainline X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 22:33:18 -0000 Hello everyone, I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work). This is a very significant change, and all testing is certainly welcome. As a result of this, many pull requests probably don't apply cleanly anymore. If you need help rebasing them on the new structure, ask me. The idea behind ultraprune is to use an ultra-pruned copy (only unspent transaction outputs in a custom compact format) of the block chain for validation (as opposed to a transaction index into the block chain). It still keeps all blocks around for serving them to other nodes, for rescanning, and for reorganisations. As such, it is still a full node. So, despite the name, it does not implement any actual pruning yet, though pruning would be trivial to implement now. This would have profound effects on the network though, so may still need some discussion first. A small summary of the changes: * Instead of blk000?.dat, we have blocks/blk000??.dat files of max 128 MiB, pre-allocated per 16 MiB * Instead of a Berklely DB blkindex.dat, we have a LevelDB directory blktree/. This only contains a block index, no transaction index. * A new LevelDB directory coins/, which contains data about the current unspent transaction output set. * New files blocks/rev000??.dat contain undo data for blocks (necessary for reorganisation). * More information is kept about blocks and block files, so facilitate pruning in the future, and to prepare for a headers-first mode. * Two new RPC calls are added: gettxout and gettxoutsetinfo. The most noticeable change should be performance: LevelDB deals much better with slow I/O than BDB does, and the working set size for validation is an order of magnitude smaller. In the longer run, I think it is an evolution towards separation between validation nodes and archive nodes, which is needed in my opinion. -- Pieter