Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 33BE874 for ; Sun, 23 Aug 2015 06:42:26 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from wp059.webpack.hosteurope.de (wp059.webpack.hosteurope.de [80.237.132.66]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 7D8D0E5 for ; Sun, 23 Aug 2015 06:42:24 +0000 (UTC) Received: from [37.143.74.116] (helo=[192.168.2.11]); authenticated by wp059.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) id 1ZTOyw-0002b4-SK; Sun, 23 Aug 2015 08:42:18 +0200 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Content-Type: multipart/signed; boundary="Apple-Mail=_17BEBBF6-E660-4298-A3A0-1309321DE788"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5.1 From: Tamas Blummer In-Reply-To: Date: Sun, 23 Aug 2015 08:42:17 +0200 Message-Id: <22CF03CB-7A4B-44D9-8989-22798795BB16@bitsofproof.com> References: <55B723EA.7010700@voskuil.org> <55B939CF.1080903@voskuil.org> <3390F712-879A-46E9-ABCD-D35B51190304@bitsofproof.com> To: Eric Lombrozo X-Mailer: Apple Mail (2.2104) X-bounce-key: webpack.hosteurope.de; tamas@bitsofproof.com; 1440312144; 7eb00295; X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW,URIBL_BLACK autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , Libbitcoin Subject: Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks) X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 06:42:26 -0000 --Apple-Mail=_17BEBBF6-E660-4298-A3A0-1309321DE788 Content-Type: multipart/alternative; boundary="Apple-Mail=_B72C70AC-BB6F-472F-83AF-BB1FEAC4B96B" --Apple-Mail=_B72C70AC-BB6F-472F-83AF-BB1FEAC4B96B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I see the huge amount of sweat and love that went into core and it = actually hurts to see that most is expended in friction and lack of a = vision for the software architecture. To be concrete, this was my plan if dealing with the Core code base: 1) I'd consider the separation of networking and storage as suggested = for a future extended libconsensus low priority, as their design should = be (are) dominated by the need of the consensus logic only. 2) create an API to the consensus+networking+storage service that is not = at the C++ language level but some scaleable cross-platform remoting, = like eg. ZeroMQ. This API should be minimal and simple, assuming that one fully trusts = the node answering it. This API would unlock user land development by = distinct teams with diverse technologies. 3) move the wallet, QT and RPC and other backward compatibility stuff = (if e.g. there is some mining support) in-top of the new API and into = distinct source code repositories. Tamas Blummer > On Aug 23, 2015, at 03:23, Eric Lombrozo wrote: >=20 > I've been pushing for greater modularization since I first got into = bitcoin. I got quickly frustrated when I was only able to get through = very few things (i.e. moving core structure serialization classes to a = separate unit not called main). Working on Bitcoin has an added layer of = frustration that goes beyond most open source projects: even though = we're clearly in userland working at the application layer, a good = layered protocol design is still lacking. We have no standards process = separate from what basically amount to updates to one specific reference = implementation. And we all need to agree on any major change, since a = blockchain that is easily forked in contentious ways pretty much defeats = its own purpose. >=20 > I went off to develop my own stack, where I could more easily avoid = politics and focus on engineering. But I now understand the politics are = inevitable. Bitcoin is inherently a cooperative project. Several people = have poured themselves passionately into the reference codebase, most of = whom did it (at least initially) purely as unpaid volunteers. There's a = lot of love that's gone into this. But it's become pretty clear that the = modularization is no longer merely a matter of good engineering - it is = essential to resolving serious political challenges. >=20 > Perhaps the most frustrating thing of all is watching people pushing = for relatively superficial yet highly controversial changes while we = still lack the proper infrastructure to handle these kinds of = divergences of opinion without either stagnating or becoming polarized. >=20 > I could continue working to reimplement an entire stack from scratch, = as several others have also done - but besides the serious effort = duplication this entails, it doesn't really seem like it will ultimately = be a convergent process. It's too easy to let ego and habit dictate = one's preferences rather than rational engineering considerations. >=20 > I know that some might feel I'm just preaching to the choir, but we = should probably take a step back from implementation hackery and try to = specify some core protocol layers, focusing on interfaces. Specifically, = we need a consensus layer that doesn't try to specify networking, = storage, wallets, UI, etc. Let different people improve upon these = things independently in their own implementations. What matters is that = we all converge on a common history and state. At the same time, let's = open up more competition on all these other things that are separate = from the consensus layer. >=20 > If only we were to dedicate a fraction of the effort we've put into = this whole block size circus into what's actually important...and I = blame myself as well... >=20 >=20 > On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev = > wrote: >> On Aug 21, 2015, at 21:46, Jorge Tim=C3=B3n > wrote: >>=20 >> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer = > wrote: >>> Every re-implementation, re-factoring even copy-paste introduces a = risk of disagreement, >>> but also open the chance of doing the work better, in the sense of = software engineering. >>=20 >> But you don't want something better, you want something functionally = identical. >> You may want to watch sipa's explanation on why "the implementation = is >> the specification" and the reasons to separate libconsensus: >> https://youtu.be/l3O4nh79CUU?t=3D764 = >=20 > I do want something better, but not for the focus you have. >=20 > Not because what you produce was not high quality, but because quality = is achieved at a very > high cost and is hard to uphold over generations of developer. You = focus on a single use case > while there are many out there for distributed ledgers. >=20 > I think in an infrastructure for enterprise applications, building = consensus on the ledger is a > cornerstone there, but is only a piece of the solution. I built = several commercially successful > deployments where I delegated the consensus building to a border = router, a Bitcoin Core, > then interfaced that trusted peer with my implementation that = accepted Core=E2=80=99s decisions > in an SPV manner. One might think of this setup as wasteful and = unsuitable for =E2=80=9Csmall devices=E2=80=9D > therefore an example of centralization people here try to avoid. >=20 > Enterprises have sufficient resources. Solving the business problem is = valuable to them even at > magnitudes higher cost than a hobbyist would bear. >=20 > For mainstream adoption you need to get enterprises on board too, and = that is what I care of. > Enterprises want code that is not only high quality, but is easy to = maintain with a development > team with high attrition. One has to take whatever help is offered for = that, and one is modern > languages and runtimes. >=20 > Bits of Proof=E2=80=99s own implementation of the scripts was not = practically relevant in my commercially > successful deployments, because of the use of a border router, but it = helped development, > enabling easier debug and precise error feedback esp. end even after = Core had a reject message. >=20 > I integrated libconsensus only for the hope that is significantly = fastens application side tx verification, > which it has turned out it does not, until secp265k1 is integrated. >=20 > I would likely use an other extended libconsensus too, but do not = think there was a dependency on > that for enterprise development. >=20 > It would help there more to have a slim protocol server, no wallet, no = rpc, no qt but a high > performance remoting API. >=20 >> Since you already depend on libconsensus for VerifyScript, wouldn't = it >> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock? >> You would still have complete control over storage, concurrency, >> networking, policy... >> My plan is for the C API to interface with the external storage by >> passing a function pointer to it. >=20 >=20 > Storage and validation is non-trivially interconnected, but I now the = separation can be done, > since I did it. >=20 > Excuse me, but function pointers is a pattern I used in the 80=E2=80=99s= . I know that they are behind > the curtain of modern abstractions with similar use, I still prefer = not to see them again. >=20 > Tamas Blummer >=20 > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org = > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev = --Apple-Mail=_B72C70AC-BB6F-472F-83AF-BB1FEAC4B96B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 I see the huge amount of sweat and love that went into core = and it actually hurts to see that most is expended in friction and lack = of a vision for the software architecture.

To be concrete, this was my plan if = dealing with the Core code base:

1) I'd consider the separation of = networking and storage as suggested for a future extended libconsensus = low priority, as their design should be (are) dominated by the need of = the consensus logic only.

2) create an API to the consensus+networking+storage service = that is not at the C++ language level but some scaleable cross-platform = remoting, like eg. ZeroMQ. 
This API should be = minimal and simple, assuming that one fully trusts the node answering = it. This API would unlock user land development by distinct teams with = diverse technologies.

3) move the wallet, QT and RPC and other backward = compatibility stuff (if e.g. there is some mining support) in-top of the = new API and into distinct source code repositories. 


Tamas Blummer

On Aug 23, 2015, at 03:23, Eric Lombrozo = <elombrozo@gmail.com> wrote:

I've been pushing for greater modularization since I first = got into bitcoin. I got quickly frustrated when I was only able to get = through very few things (i.e. moving core structure serialization = classes to a separate unit not called main). Working on Bitcoin has an = added layer of frustration that goes beyond most open source projects: = even though we're clearly in userland working at the application layer, = a good layered protocol design is still lacking. We have no standards = process separate from what basically amount to updates to one specific = reference implementation. And we all need to agree on any major change, = since a blockchain that is easily forked in contentious ways pretty much = defeats its own purpose.

I went off to = develop my own stack, where I could more easily avoid politics and focus = on engineering. But I now understand the politics are inevitable. = Bitcoin is inherently a cooperative project. Several people have poured = themselves passionately into the reference codebase, most of whom did it = (at least initially) purely as unpaid volunteers. There's a lot of love = that's gone into this. But it's become pretty clear that the = modularization is no longer merely a matter of good engineering - it is = essential to resolving serious political challenges.

Perhaps the most frustrating thing of all is watching people = pushing for relatively superficial yet highly controversial changes = while we still lack the proper infrastructure to handle these kinds of = divergences of opinion without either stagnating or becoming = polarized.

I could continue working to = reimplement an entire stack from scratch, as several others have also = done - but besides the serious effort duplication this entails, it = doesn't really seem like it will ultimately be a convergent process. = It's too easy to let ego and habit dictate one's preferences rather than = rational engineering considerations.

I = know that some might feel I'm just preaching to the choir, but we should = probably take a step back from implementation hackery and try to specify = some core protocol layers, focusing on interfaces. Specifically, we need = a consensus layer that doesn't try to specify networking, storage, = wallets, UI, etc. Let different people improve upon these things = independently in their own implementations. What matters is that we all = converge on a common history and state. At the same time, let's open up = more competition on all these other things that are separate from the = consensus layer.

If only we were to = dedicate a fraction of the effort we've put into this whole block size = circus into what's actually important...and I blame myself as well...


On = Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
On Aug 21, 2015, at 21:46, = Jorge Tim=C3=B3n <jtimon@jtimon.cc> wrote:

On Thu, Aug = 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com> = wrote:
Every re-implementation, re-factoring even = copy-paste introduces a risk of disagreement,
but also = open the chance of doing the work better, in the sense of software = engineering.

But you don't = want something better, you want something functionally = identical.
You may want = to watch sipa's explanation on why "the implementation is
the = specification" and the reasons to separate libconsensus:
https://youtu.be/l3O4nh79CUU?t=3D764

I do want something better, = but not for the focus you have. 

Not because what you produce was not = high quality, but because quality is achieved at a very
high cost and is hard to uphold over generations of = developer. You focus on a single use case
while = there are many out there for distributed ledgers.
I think in an infrastructure for = enterprise applications, building consensus on the ledger is = a 
cornerstone there, but is only a piece of = the solution. I built several commercially successful
deployments where I delegated the consensus building to a = border router, a Bitcoin Core, 
then = interfaced that trusted peer with my  implementation that accepted = Core=E2=80=99s decisions 
in an SPV manner. = One might think of this setup as wasteful and unsuitable for =E2=80=9Csmal= l devices=E2=80=9D
therefore an example of = centralization people here try to avoid.

Enterprises have sufficient resources. = Solving the business problem is valuable to them even at 
magnitudes higher cost than a hobbyist would bear.

For mainstream adoption = you need to get enterprises on board too, and  that is what I care = of. 
Enterprises want code that is not only = high quality, but is easy to maintain with a development 
team with high attrition. One has to take whatever help is = offered for that, and one is modern 
languages = and runtimes.

Bits of Proof=E2=80=99s own implementation of = the scripts was not practically relevant in my commercially
successful deployments, because of the use of a border = router, but it helped development, 
enabling = easier debug and precise error feedback esp. end even after Core had a = reject message. 

I integrated libconsensus only for the hope that is = significantly fastens application side tx verification,
 which it has turned out it does not, until secp265k1 is = integrated.

I = would likely use an other extended libconsensus too, but do not think = there was a dependency on 
that for enterprise = development. 

It would help there more to have a slim protocol server, no = wallet, no rpc, no qt but a high 
performance = remoting API.

Since you = already depend on libconsensus for VerifyScript, wouldn't it
be nice that = it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would = still have complete control over storage, concurrency,
networking, = policy...
My plan is = for the C API to interface with the external storage by
passing a = function pointer to it.

Storage and validation = is non-trivially interconnected, but I now the separation can be = done,
since I did it. 

Excuse me, but function pointers is a = pattern I used in the 80=E2=80=99s. I know that they are = behind 
the curtain of modern abstractions = with similar use, I still prefer not to see them again.

Tamas = Blummer

________________________________= _______________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev<= /a>

= --Apple-Mail=_B72C70AC-BB6F-472F-83AF-BB1FEAC4B96B-- --Apple-Mail=_17BEBBF6-E660-4298-A3A0-1309321DE788 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJV2WtKAAoJEPZykcUXcTkc5WcIAJCySmxF3G2DiiZ4OegkX0Fs ONIkmP9+l/+jPcAeZ1CdbZvXajAnMpAIlNdM+B/qK44vU8T4PCIFZNGgNOX0F1Fx d2opKN+K+ILfGVhsbE4Tb55C9ExqYdlIinEXAaYVrZRda4zZfHqeO+oQ378zFYwX c2PXOHVqbK76Qzmua5BmN63mq+dpm+DALpSi+aTOP2f5Y4xCViQPZjO0E6UL1WPd qR+s9AIWYDLBcGAYPymC7zIWjFbwuNnRgdxhKLoZnfHOdWI4YnKosEj4ey/N0mJg 24oYb48zG3p4jemY0qh7CewZpFz1qS0PDJry1hr2cEEOIemdgK4YC8coi4T0D1w= =3sUq -----END PGP SIGNATURE----- --Apple-Mail=_17BEBBF6-E660-4298-A3A0-1309321DE788--