Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 82F9E847 for ; Sun, 23 Aug 2015 02:19:37 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4B9A7255 for ; Sun, 23 Aug 2015 02:19:36 +0000 (UTC) Received: by iods203 with SMTP id s203so117101270iod.0 for ; Sat, 22 Aug 2015 19:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=gZzWMi8MNGbLMhrW89gINxXWLRBlf+L9D5vxR73Nmnw=; b=GCGv8x4tn22YrwAfcoVdtpOQQd5CVP1JRbc7mpomibRpUm8o+ZHMmzf5J0K28Pe90+ c/+M7Sj67N8nB6nXes68QZSTkXA4LBkNe5+GWtb809gD+7F0XM8+e4WqxjJUtuDo9naq KBz3iRTJwv8ruU9KRAV7eESCZ0t02/t6hrNG0DL9fS2D9kkwPcJbsDTSxsTgW3NWDQGo I7CbYA9giVzwvLph1pwGjzvVeYiXukgmWQk+9j3E+pbO4mHi46DmLuSAP3/AVTrQnBkO /8vnhBhT6Xhq8+IqUtf/PACOvFBu/SFCw+xtw+M59qcVL+ZvvdEVFhw5yBMm7FcNEhBa Sncg== X-Received: by 10.107.4.85 with SMTP id 82mr12727249ioe.180.1440296375444; Sat, 22 Aug 2015 19:19:35 -0700 (PDT) MIME-Version: 1.0 References: <55B723EA.7010700@voskuil.org> <55B939CF.1080903@voskuil.org> <3390F712-879A-46E9-ABCD-D35B51190304@bitsofproof.com> In-Reply-To: From: Eric Lombrozo Date: Sun, 23 Aug 2015 02:19:26 +0000 Message-ID: To: Tamas Blummer , =?UTF-8?B?Sm9yZ2UgVGltw7Nu?= , Matt Corallo Content-Type: multipart/alternative; boundary=001a113d54eed46ad6051df12118 X-Spam-Status: No, score=-1.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_LOW, URIBL_BLACK autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , Libbitcoin Subject: Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin Core and hard forks) X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 02:19:37 -0000 --001a113d54eed46ad6051df12118 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable One thing it occurs to me (and I don't know if this has been suggested before) we could do is separate the BIP process into at several distinct areas: 1) Commit structure changes/consensus rule change proposals - Consensus-building process (how are proposals debated, improved, vetted, and selected) - Update/deployment mechanisms for rule changes - Specific hard fork proposals - Specific soft fork proposals 2) Peer policies - Seeding and discovery mechanisms - Relay policies - p2p message support 3) RPC 4) Everything else On Sat, Aug 22, 2015, 6:28 PM Eric Lombrozo wrote: > I've been pushing for greater modularization since I first got into > bitcoin. I got quickly frustrated when I was only able to get through ver= y > few things (i.e. moving core structure serialization classes to a separat= e > unit not called main). Working on Bitcoin has an added layer of frustrati= on > that goes beyond most open source projects: even though we're clearly in > userland working at the application layer, a good layered protocol design > is still lacking. We have no standards process separate from what basical= ly > amount to updates to one specific reference implementation. And we all ne= ed > to agree on any major change, since a blockchain that is easily forked in > contentious ways pretty much defeats its own purpose. > > I went off to develop my own stack, where I could more easily avoid > politics and focus on engineering. But I now understand the politics are > inevitable. Bitcoin is inherently a cooperative project. Several people > have poured themselves passionately into the reference codebase, most of > whom did it (at least initially) purely as unpaid volunteers. There's a l= ot > of love that's gone into this. But it's become pretty clear that the > modularization is no longer merely a matter of good engineering - it is > essential to resolving serious political challenges. > > Perhaps the most frustrating thing of all is watching people pushing for > relatively superficial yet highly controversial changes while we still la= ck > the proper infrastructure to handle these kinds of divergences of opinion > without either stagnating or becoming polarized. > > I could continue working to reimplement an entire stack from scratch, as > several others have also done - but besides the serious effort duplicatio= n > this entails, it doesn't really seem like it will ultimately be a > convergent process. It's too easy to let ego and habit dictate one's > preferences rather than rational engineering considerations. > > I know that some might feel I'm just preaching to the choir, but we shoul= d > probably take a step back from implementation hackery and try to specify > some core protocol layers, focusing on interfaces. Specifically, we need = a > consensus layer that doesn't try to specify networking, storage, wallets, > UI, etc. Let different people improve upon these things independently in > their own implementations. What matters is that we all converge on a comm= on > history and state. At the same time, let's open up more competition on al= l > these other things that are separate from the consensus layer. > > If only we were to dedicate a fraction of the effort we've put into this > whole block size circus into what's actually important...and I blame myse= lf > as well... > > On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> On Aug 21, 2015, at 21:46, Jorge Tim=C3=B3n wrote: >> >> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer >> wrote: >> >> Every re-implementation, re-factoring even copy-paste introduces a risk >> of disagreement, >> but also open the chance of doing the work better, in the sense of >> software engineering. >> >> >> But you don't want something better, you want something functionally >> identical. >> You may want to watch sipa's explanation on why "the implementation is >> the specification" and the reasons to separate libconsensus: >> https://youtu.be/l3O4nh79CUU?t=3D764 >> >> >> I do want something better, but not for the focus you have. >> >> Not because what you produce was not high quality, but because quality i= s >> achieved at a very >> high cost and is hard to uphold over generations of developer. You focus >> on a single use case >> while there are many out there for distributed ledgers. >> >> I think in an infrastructure for enterprise applications, building >> consensus on the ledger is a >> cornerstone there, but is only a piece of the solution. I built several >> commercially successful >> deployments where I delegated the consensus building to a border router, >> a Bitcoin Core, >> then interfaced that trusted peer with my implementation that accepted >> Core=E2=80=99s decisions >> in an SPV manner. One might think of this setup as wasteful and >> unsuitable for =E2=80=9Csmall devices=E2=80=9D >> therefore an example of centralization people here try to avoid. >> >> Enterprises have sufficient resources. Solving the business problem is >> valuable to them even at >> magnitudes higher cost than a hobbyist would bear. >> >> For mainstream adoption you need to get enterprises on board too, and >> that is what I care of. >> Enterprises want code that is not only high quality, but is easy to >> maintain with a development >> team with high attrition. One has to take whatever help is offered for >> that, and one is modern >> languages and runtimes. >> >> Bits of Proof=E2=80=99s own implementation of the scripts was not practi= cally >> relevant in my commercially >> successful deployments, because of the use of a border router, but it >> helped development, >> enabling easier debug and precise error feedback esp. end even after Cor= e >> had a reject message. >> >> I integrated libconsensus only for the hope that is significantly fasten= s >> application side tx verification, >> which it has turned out it does not, until secp265k1 is integrated. >> >> I would likely use an other extended libconsensus too, but do not think >> there was a dependency on >> that for enterprise development. >> >> It would help there more to have a slim protocol server, no wallet, no >> rpc, no qt but a high >> performance remoting API. >> >> Since you already depend on libconsensus for VerifyScript, wouldn't it >> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock? >> You would still have complete control over storage, concurrency, >> networking, policy... >> My plan is for the C API to interface with the external storage by >> passing a function pointer to it. >> >> >> Storage and validation is non-trivially interconnected, but I now the >> separation can be done, >> since I did it. >> >> Excuse me, but function pointers is a pattern I used in the 80=E2=80=99s= . I know >> that they are behind >> the curtain of modern abstractions with similar use, I still prefer not >> to see them again. >> >> Tamas Blummer >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > --001a113d54eed46ad6051df12118 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

One thing it occurs to me (and I don't know if this has = been suggested before) we could do is separate the BIP process into at seve= ral distinct areas:

1) Commit structure changes/consensus rule change proposals<= br> - Consensus-building process (how are proposals debated, improved, vetted, = and selected)
- Update/deployment mechanisms for rule changes
- Specific hard fork proposals
- Specific soft fork proposals

2) Peer policies
- Seeding and discovery mechanisms
- Relay policies
- p2p message support

3) RPC

4) Everything else


On Sat, Aug 22, 2015, 6:28 = PM=C2=A0Eric Lombrozo <elombrozo@= gmail.com> wrote:

I've been pushing for greater modularization since I first got= into bitcoin. I got quickly frustrated when I was only able to get through= very few things (i.e. moving core structure serialization classes to a sep= arate unit not called main). Working on Bitcoin has an added layer of frust= ration that goes beyond most open source projects: even though we're cl= early in userland working at the application layer, a good layered protocol= design is still lacking. We have no standards process separate from what b= asically amount to updates to one specific reference implementation. And we= all need to agree on any major change, since a blockchain that is easily f= orked in contentious ways pretty much defeats its own purpose.

I went off to develop my own stack, where I could more easil= y avoid politics and focus on engineering. But I now understand the politic= s are inevitable. Bitcoin is inherently a cooperative project. Several peop= le have poured themselves passionately into the reference codebase, most of= whom did it (at least initially) purely as unpaid volunteers. There's = a lot of love that's gone into this. But it's become pretty clear t= hat the modularization is no longer merely a matter of good engineering - i= t is essential to resolving serious political challenges.

Perhaps the most frustrating thing of all is watching people= pushing for relatively superficial yet highly controversial changes while = we still lack the proper infrastructure to handle these kinds of divergence= s of opinion without either stagnating or becoming polarized.

I could continue working to reimplement an entire stack from= scratch, as several others have also done - but besides the serious effort= duplication this entails, it doesn't really seem like it will ultimate= ly be a convergent process. It's too easy to let ego and habit dictate = one's preferences rather than rational engineering considerations.

I know that some might feel I'm just preaching to the ch= oir, but we should probably take a step back from implementation hackery an= d try to specify some core protocol layers, focusing on interfaces. Specifi= cally, we need a consensus layer that doesn't try to specify networking= , storage, wallets, UI, etc. Let different people improve upon these things= independently in their own implementations. What matters is that we all co= nverge on a common history and state. At the same time, let's open up m= ore competition on all these other things that are separate from the consen= sus layer.

If only we were to dedicate a fraction of the effort we'= ve put into this whole block size circus into what's actually important= ...and I blame myself as well...


On Sat, Aug 22, 2015, 4:05 = AM=C2=A0Tamas Blummer via bitcoin-dev <bitcoin-dev@lists.linuxfoundation= .org> wrote:
On Aug 21, 2015, a= t 21:46, Jorge Tim=C3=B3n <jtimon@jtimon.cc> wrote:

On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blum= mer <tamas@bitsofproof.com> wrote:
= Every re-implementation, re-factoring even copy-paste introduces a risk of = disagreement,
but also open the chance of doing the work better, in the = sense of software engineering.

But you don't want something better= , you want something functionally identical.
You may want to watch sipa's ex= planation on why "the implementation is
the specification" and the rea= sons to separate libconsensus:
https://youtu.be/l3O4nh79CUU?t=3D= 764

I do want something better, but not fo= r the focus you have.=C2=A0

Not because what you p= roduce was not high quality, but because quality is achieved at a very
high cost and is hard to uphold over generations of developer. You fo= cus on a single use case
while there are many out there for distr= ibuted ledgers.

I think in an infrastructure for e= nterprise applications, building consensus on the ledger is a=C2=A0
cornerstone there, but is only a piece of the solution. I built several = commercially successful
deployments where I delegated the consens= us building to a border router, a Bitcoin Core,=C2=A0
then interf= aced that trusted peer with my =C2=A0implementation that accepted Core=E2= =80=99s decisions=C2=A0
in an SPV manner. One might think of this= setup as wasteful and unsuitable for =E2=80=9Csmall devices=E2=80=9D
=
therefore an example of centralization people here try to avoid.
=

Enterprises have sufficient resources. Solving the busi= ness problem is valuable to them even at=C2=A0
magnitudes higher = cost than a hobbyist would bear.

For mainstream ad= option you need to get enterprises on board too, and =C2=A0that is what I c= are of.=C2=A0
Enterprises want code that is not only high quality= , but is easy to maintain with a development=C2=A0
team with high= attrition. One has to take whatever help is offered for that, and one is m= odern=C2=A0
languages and runtimes.

Bits of Proof=E2=80=99s own implementation of the scripts was not practica= lly relevant in my commercially
successful deployments, because o= f the use of a border router, but it helped development,=C2=A0
en= abling easier debug and precise error feedback esp. end even after Core had= a reject message.=C2=A0

I integrated libcon= sensus only for the hope that is significantly fastens application side tx = verification,
=C2=A0which it has turned out it does not, until se= cp265k1 is integrated.

I would likely use an other= extended libconsensus too, but do not think there was a dependency on=C2= =A0
that for enterprise development.=C2=A0

It would help there more to have a slim protocol server, no wallet, no r= pc, no qt but a high=C2=A0
performance remoting API.
<= /div>

Since y= ou already depend on libconsensus for VerifyScript, wouldn't it<= br style=3D"font-family:Helvetica;font-size:12px;font-style:normal;font-var= iant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;tex= t-align:start;text-indent:0px;text-transform:none;white-space:normal;word-s= pacing:0px">be nice that= it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have co= mplete control over storage, concurrency,
networking, policy...
My plan is for the C= API to interface with the external storage by
passing a function pointer to it.=
<= div>

Storage and validation is non-trivially inter= connected, but I now the separation can be done,
since I did it.= =C2=A0

Excuse me, but function pointers is a patte= rn I used in the 80=E2=80=99s. I know that they are behind=C2=A0
= the curtain of modern abstractions with similar use, I still prefer not to = see them again.

Tamas Blummer

______________________________= _________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--001a113d54eed46ad6051df12118--