Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1XdC4K-0006dd-Tw for bitcoin-development@lists.sourceforge.net; Sun, 12 Oct 2014 05:51:49 +0000 Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.192.46 as permitted sender) client-ip=209.85.192.46; envelope-from=voisine@gmail.com; helo=mail-qg0-f46.google.com; Received: from mail-qg0-f46.google.com ([209.85.192.46]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1XdC4J-0006NJ-By for bitcoin-development@lists.sourceforge.net; Sun, 12 Oct 2014 05:51:48 +0000 Received: by mail-qg0-f46.google.com with SMTP id z60so5570089qgd.5 for ; Sat, 11 Oct 2014 22:51:41 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.229.178.129 with SMTP id bm1mr26238345qcb.20.1413093101892; Sat, 11 Oct 2014 22:51:41 -0700 (PDT) Received: by 10.140.251.86 with HTTP; Sat, 11 Oct 2014 22:51:41 -0700 (PDT) In-Reply-To: References: Date: Sat, 11 Oct 2014 22:51:41 -0700 Message-ID: From: Aaron Voisine To: Pieter Wuille Content-Type: text/plain; charset=UTF-8 X-Spam-Score: 0.1 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (voisine[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.7 URIBL_WS_SURBL Contains an URL listed in the WS SURBL blocklist [URIs: sipa.be] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1XdC4J-0006NJ-By Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Request for review/testing: headers-first synchronization in Bitcoin Core X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 05:51:49 -0000 This is great Pieter. I was able to sync the entire blockchain from scratch in a little over 4 hours on a laptop over cable modem. :) No issues to report. Even my family photos are intact! This makes it practical to run a full node, part time on a laptop again. Aaron Voisine breadwallet.com On Sat, Oct 11, 2014 at 4:34 PM, Pieter Wuille wrote: > Hi all, > > I believe that a large change that I've been working on for Bitcoin > Core is ready for review and testing: headers-first synchronization. > In short, it changes the way the best chain is discovered, downloaded > and verified, with several advantages: > * Parallel block downloading (much faster sync on typical network connections). > * No more stalled downloads. > * Much more robust against unresponsive or slow peers. > * Removes a class of DoS attacks related to peers feeding you > low-difficulty valid large blocks on a side branch. > * Reduces the need for checkpoints in the code. > * No orphan blocks stored in memory anymore (reducing memory usage during sync). > * A major step step towards an SPV mode using the reference codebase. > > Historically, this mode of operation has been known for years (Greg > Maxwell wrote up a description of a very similar method in > https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync > in early 2012, but it was known before that), but it took a long time > to refactor these code enough to support it. > > Technically, it works by replacing the single-peer blocks download by > a single-peer headers download (which typically takes seconds/minutes) > and verification, and simultaneously fetching blocks along the best > known headers chain from all peers that are known to have the relevant > blocks. Downloading is constrained to a moving window to avoid > unbounded unordering of blocks on disk (which would interfere with > pruning later). > > At the protocol level, it increases the minimally supported version > for peers to 31800 (corresponding to bitcoin v3.18, released in > december 2010), as earlier versions did not support the getheaders P2P > message. > > So, the code is available as a github pull request > (https://github.com/bitcoin/bitcoin/pull/4468), or packaged on > http://bitcoin.sipa.be/builds/headersfirst, where you can also find > binaries to test with. > > Known issues: > * At the very start of the sync, especially before all headers are > processed, downloading is very slow due to a limited number of blocks > that are requested per peer simultaneously. The policies around this > will need some experimentation can certainly be improved. > * Blocks will be stored on disk out of order (in the order they are > received, really), which makes it incompatible with some tools or > other programs. Reindexing using earlier versions will also not work > anymore as a result of this. > * The block index database will now hold headers for which no block is > stored on disk, which earlier versions won't support. If you are fully > synced, it may still be possible to go back to an earlier version. > > Unknown issues: > * Who knows, maybe it will replace your familiy pictures with Nyan > Cat? Use at your own risk. > > TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or > http://bitcoin.sipa.be/builds/headersfirst. > > -- > Pieter > > ------------------------------------------------------------------------------ > Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer > Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports > Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper > Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer > http://p.sf.net/sfu/Zoho > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development