Date: Fri, 18 Sep 2015 02:07:20 +0200
From: "Wladimir J. van der Laan" <laanwj@gmail.com>
To: Peter Todd <pete@petertodd.org>
Message-ID: <20150918000718.GA5251@amethyst.visucore.com>
References: <CADm_WcY8Vy+k+5BaBS+jV6D6tmSXrok8rAxoPxxKOzUhyPWgMg@mail.gmail.com>
	<20150916222928.GC32688@muck>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20150916222928.GC32688@muck>
Cc: Bitcoin development mailing list <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] libconsensus and bitcoin development process
Precedence: list

On Wed, Sep 16, 2015 at 06:29:28PM -0400, Peter Todd via bitcoin-dev wrote:

> I've run into a number of cases where companies were maintaining forks
> of Bitcoin Core unnecessarily, where a different, loosely coupled,
> architecture could do what they needed to do without including the new
> logic in the codebase itself.

This is the same point I have been making to Jeff privately.

Refactors are a means to an end: a more modular, reusable and maintainable codebase. This goal is that new functionality can be plugged in more easily, and rebase work for e.g. functionality built on top can go down, not up, if it just hooks into well-defined interfaces here and there.

Although there has been a lot of progress, bitcoind's design is still too monolithic. To add a more involved feature, like say a new index over the block chain data, code needs to be touched all over the place. This change interacts with all other functionality, potentially breaking the base node functionality - risk for users that do NOT use the functionality. This increases risk and review time.

- *If possible* functionality should be built without changing bitcoind's code at all. An external process should be able to keep up to date with the chain, notice reorgs, and process block data accordingly. If bitcoind's interface does not allow that, or it is too difficult, that is what should be fixed. 
- *if not possible* then a change should at least touch the code in as few places as possible, and integrate with e.g. signal notification.

To name an example of it done right, IMO: Monero's 'simplewallet'. It is a command-line utility wallet that communicates with the node software, and remembers where it was in the chain, and processes changes to the chain state since its last invocation when it 'refreshes'. 
What is nice is that one can run an arbitary number of simplewallets against one node daemon, and unlike bitcoind's wallet it doesn't need to run as always-on daemon itself. It can be invoked when the user wants to do something with the wallet, or see if there are new transactions.

An index could be implemented entirely externally in a similar way, while still fully handling reorgs.

What one needs for that, I think, is a library that communicate with the node, and which offers functionality abstractly be similar to 'git pull': give me the tree path from my current known tip to the best tip, and supply the block hashes (and block data) along the way. 

My long-term vision of bitcoind is a P2P node with validation and blockchain store, with a couple of data sources that can be subscribed to or pulled from.

Wladimir