MIME-Version: 1.0
References: <CABm2gDrApVuxF8DFf32V=pQhDKvvVfcDK=LeCXJ9h9o8CY+wNQ@mail.gmail.com>
	<55B723EA.7010700@voskuil.org>
	<CABm2gDok2gH2R8=x3a8PmPiM56WFg3TKzCum_WS=uV9-T1Ss3g@mail.gmail.com>
	<55B939CF.1080903@voskuil.org>
	<CABm2gDq1wHP01Tncw3hu=SCWQHaAOMjWOVYQWdQsOZ+E7zp9Yw@mail.gmail.com>
	<CABm2gDr_ePfPeWB8pEO8QNHDjUZpybVuCAdDmMxJxMaC8+2bGA@mail.gmail.com>
	<C4EA4A39-1920-4F33-822C-FBD12DF81004@bitsofproof.com>
	<CABm2gDqkF20ZoexQSV8iORb3ukxxZr5RasTLxJqQfSTsTqHvog@mail.gmail.com>
	<3390F712-879A-46E9-ABCD-D35B51190304@bitsofproof.com>
	<CABm2gDpcEmiPNQWeUk5aTjuTSRAJSPYfgAKc7B_qrqw0w04xoA@mail.gmail.com>
	<C486E9D9-D799-48B9-B80F-1A165DFD6536@bitsofproof.com>
In-Reply-To: <C486E9D9-D799-48B9-B80F-1A165DFD6536@bitsofproof.com>
From: Eric Lombrozo <elombrozo@gmail.com>
Date: Sun, 23 Aug 2015 01:23:33 +0000
Message-ID: <CABr1YTcx4V0Q4-ZQBEiNG-z1NKeFzxzhMekRN3YRRLz+bme2iw@mail.gmail.com>
To: Tamas Blummer <tamas@bitsofproof.com>,
	=?UTF-8?B?Sm9yZ2UgVGltw7Nu?= <jtimon@jtimon.cc>, 
	Matt Corallo <lf-lists@mattcorallo.com>
Content-Type: multipart/alternative; boundary=047d7bd771220b3d48051df05aeb
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
	Libbitcoin <libbitcoin@lists.dyne.org>
Subject: Re: [bitcoin-dev] Libconsensus separated repository (was Bitcoin
 Core and hard forks)
Precedence: list

--047d7bd771220b3d48051df05aeb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I've been pushing for greater modularization since I first got into
bitcoin. I got quickly frustrated when I was only able to get through very
few things (i.e. moving core structure serialization classes to a separate
unit not called main). Working on Bitcoin has an added layer of frustration
that goes beyond most open source projects: even though we're clearly in
userland working at the application layer, a good layered protocol design
is still lacking. We have no standards process separate from what basically
amount to updates to one specific reference implementation. And we all need
to agree on any major change, since a blockchain that is easily forked in
contentious ways pretty much defeats its own purpose.

I went off to develop my own stack, where I could more easily avoid
politics and focus on engineering. But I now understand the politics are
inevitable. Bitcoin is inherently a cooperative project. Several people
have poured themselves passionately into the reference codebase, most of
whom did it (at least initially) purely as unpaid volunteers. There's a lot
of love that's gone into this. But it's become pretty clear that the
modularization is no longer merely a matter of good engineering - it is
essential to resolving serious political challenges.

Perhaps the most frustrating thing of all is watching people pushing for
relatively superficial yet highly controversial changes while we still lack
the proper infrastructure to handle these kinds of divergences of opinion
without either stagnating or becoming polarized.

I could continue working to reimplement an entire stack from scratch, as
several others have also done - but besides the serious effort duplication
this entails, it doesn't really seem like it will ultimately be a
convergent process. It's too easy to let ego and habit dictate one's
preferences rather than rational engineering considerations.

I know that some might feel I'm just preaching to the choir, but we should
probably take a step back from implementation hackery and try to specify
some core protocol layers, focusing on interfaces. Specifically, we need a
consensus layer that doesn't try to specify networking, storage, wallets,
UI, etc. Let different people improve upon these things independently in
their own implementations. What matters is that we all converge on a common
history and state. At the same time, let's open up more competition on all
these other things that are separate from the consensus layer.

If only we were to dedicate a fraction of the effort we've put into this
whole block size circus into what's actually important...and I blame myself
as well...

On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Aug 21, 2015, at 21:46, Jorge Tim=C3=B3n <jtimon@jtimon.cc> wrote:
>
> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas@bitsofproof.com>
> wrote:
>
> Every re-implementation, re-factoring even copy-paste introduces a risk o=
f
> disagreement,
> but also open the chance of doing the work better, in the sense of
> software engineering.
>
>
> But you don't want something better, you want something functionally
> identical.
> You may want to watch sipa's explanation on why "the implementation is
> the specification" and the reasons to separate libconsensus:
> https://youtu.be/l3O4nh79CUU?t=3D764
>
>
> I do want something better, but not for the focus you have.
>
> Not because what you produce was not high quality, but because quality is
> achieved at a very
> high cost and is hard to uphold over generations of developer. You focus
> on a single use case
> while there are many out there for distributed ledgers.
>
> I think in an infrastructure for enterprise applications, building
> consensus on the ledger is a
> cornerstone there, but is only a piece of the solution. I built several
> commercially successful
> deployments where I delegated the consensus building to a border router, =
a
> Bitcoin Core,
> then interfaced that trusted peer with my  implementation that accepted
> Core=E2=80=99s decisions
> in an SPV manner. One might think of this setup as wasteful and unsuitabl=
e
> for =E2=80=9Csmall devices=E2=80=9D
> therefore an example of centralization people here try to avoid.
>
> Enterprises have sufficient resources. Solving the business problem is
> valuable to them even at
> magnitudes higher cost than a hobbyist would bear.
>
> For mainstream adoption you need to get enterprises on board too, and
>  that is what I care of.
> Enterprises want code that is not only high quality, but is easy to
> maintain with a development
> team with high attrition. One has to take whatever help is offered for
> that, and one is modern
> languages and runtimes.
>
> Bits of Proof=E2=80=99s own implementation of the scripts was not practic=
ally
> relevant in my commercially
> successful deployments, because of the use of a border router, but it
> helped development,
> enabling easier debug and precise error feedback esp. end even after Core
> had a reject message.
>
> I integrated libconsensus only for the hope that is significantly fastens
> application side tx verification,
>  which it has turned out it does not, until secp265k1 is integrated.
>
> I would likely use an other extended libconsensus too, but do not think
> there was a dependency on
> that for enterprise development.
>
> It would help there more to have a slim protocol server, no wallet, no
> rpc, no qt but a high
> performance remoting API.
>
> Since you already depend on libconsensus for VerifyScript, wouldn't it
> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
> You would still have complete control over storage, concurrency,
> networking, policy...
> My plan is for the C API to interface with the external storage by
> passing a function pointer to it.
>
>
> Storage and validation is non-trivially interconnected, but I now the
> separation can be done,
> since I did it.
>
> Excuse me, but function pointers is a pattern I used in the 80=E2=80=99s.=
 I know
> that they are behind
> the curtain of modern abstractions with similar use, I still prefer not t=
o
> see them again.
>
> Tamas Blummer
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--047d7bd771220b3d48051df05aeb
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">I&#39;ve been pushing for greater modularization since I fir=
st got into bitcoin. I got quickly frustrated when I was only able to get t=
hrough very few things (i.e. moving core structure serialization classes to=
 a separate unit not called main). Working on Bitcoin has an added layer of=
 frustration that goes beyond most open source projects: even though we&#39=
;re clearly in userland working at the application layer, a good layered pr=
otocol design is still lacking. We have no standards process separate from =
what basically amount to updates to one specific reference implementation. =
And we all need to agree on any major change, since a blockchain that is ea=
sily forked in contentious ways pretty much defeats its own purpose.</p>
<p dir=3D"ltr">I went off to develop my own stack, where I could more easil=
y avoid politics and focus on engineering. But I now understand the politic=
s are inevitable. Bitcoin is inherently a cooperative project. Several peop=
le have poured themselves passionately into the reference codebase, most of=
 whom did it (at least initially) purely as unpaid volunteers. There&#39;s =
a lot of love that&#39;s gone into this. But it&#39;s become pretty clear t=
hat the modularization is no longer merely a matter of good engineering - i=
t is essential to resolving serious political challenges.</p>
<p dir=3D"ltr">Perhaps the most frustrating thing of all is watching people=
 pushing for relatively superficial yet highly controversial changes while =
we still lack the proper infrastructure to handle these kinds of divergence=
s of opinion without either stagnating or becoming polarized.</p>
<p dir=3D"ltr">I could continue working to reimplement an entire stack from=
 scratch, as several others have also done - but besides the serious effort=
 duplication this entails, it doesn&#39;t really seem like it will ultimate=
ly be a convergent process. It&#39;s too easy to let ego and habit dictate =
one&#39;s preferences rather than rational engineering considerations. </p>
<p dir=3D"ltr">I know that some might feel I&#39;m just preaching to the ch=
oir, but we should probably take a step back from implementation hackery an=
d try to specify some core protocol layers, focusing on interfaces. Specifi=
cally, we need a consensus layer that doesn&#39;t try to specify networking=
, storage, wallets, UI, etc. Let different people improve upon these things=
 independently in their own implementations. What matters is that we all co=
nverge on a common history and state. At the same time, let&#39;s open up m=
ore competition on all these other things that are separate from the consen=
sus layer.</p>
<p dir=3D"ltr">If only we were to dedicate a fraction of the effort we&#39;=
ve put into this whole block size circus into what&#39;s actually important=
...and I blame myself as well...<br></p>
<br><div class=3D"gmail_quote"><div dir=3D"ltr">On Sat, Aug 22, 2015, 4:05 =
AM=C2=A0Tamas Blummer via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@lis=
ts.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt; wrote=
:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-wor=
d"><div><blockquote type=3D"cite"><div>On Aug 21, 2015, at 21:46, Jorge Tim=
=C3=B3n &lt;<a href=3D"mailto:jtimon@jtimon.cc" target=3D"_blank">jtimon@jt=
imon.cc</a>&gt; wrote:</div><br><div><span style=3D"font-family:Helvetica;f=
ont-size:12px;font-style:normal;font-variant:normal;font-weight:normal;lett=
er-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-=
transform:none;white-space:normal;word-spacing:0px;float:none;display:inlin=
e!important">On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer &lt;</span><a =
href=3D"mailto:tamas@bitsofproof.com" style=3D"font-family:Helvetica;font-s=
ize:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-sp=
acing:normal;line-height:normal;text-align:start;text-indent:0px;text-trans=
form:none;white-space:normal;word-spacing:0px" target=3D"_blank">tamas@bits=
ofproof.com</a><span style=3D"font-family:Helvetica;font-size:12px;font-sty=
le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line=
-height:normal;text-align:start;text-indent:0px;text-transform:none;white-s=
pace:normal;word-spacing:0px;float:none;display:inline!important">&gt; wrot=
e:</span><br style=3D"font-family:Helvetica;font-size:12px;font-style:norma=
l;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:=
normal;text-align:start;text-indent:0px;text-transform:none;white-space:nor=
mal;word-spacing:0px"><blockquote type=3D"cite" style=3D"font-family:Helvet=
ica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal=
;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;=
text-transform:none;white-space:normal;word-spacing:0px">Every re-implement=
ation, re-factoring even copy-paste introduces a risk of disagreement,<br>b=
ut also open the chance of doing the work better, in the sense of software =
engineering.<br></blockquote><br style=3D"font-family:Helvetica;font-size:1=
2px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing=
:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:=
none;white-space:normal;word-spacing:0px"><span style=3D"font-family:Helvet=
ica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal=
;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;=
text-transform:none;white-space:normal;word-spacing:0px;float:none;display:=
inline!important">But you don&#39;t want something better, you want somethi=
ng functionally identical.</span><br style=3D"font-family:Helvetica;font-si=
ze:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spa=
cing:normal;line-height:normal;text-align:start;text-indent:0px;text-transf=
orm:none;white-space:normal;word-spacing:0px"><span style=3D"font-family:He=
lvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:no=
rmal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:=
0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;disp=
lay:inline!important">You may want to watch sipa&#39;s explanation on why &=
quot;the implementation is</span><br style=3D"font-family:Helvetica;font-si=
ze:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spa=
cing:normal;line-height:normal;text-align:start;text-indent:0px;text-transf=
orm:none;white-space:normal;word-spacing:0px"><span style=3D"font-family:He=
lvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:no=
rmal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:=
0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;disp=
lay:inline!important">the specification&quot; and the reasons to separate l=
ibconsensus:</span><br style=3D"font-family:Helvetica;font-size:12px;font-s=
tyle:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;li=
ne-height:normal;text-align:start;text-indent:0px;text-transform:none;white=
-space:normal;word-spacing:0px"><a href=3D"https://youtu.be/l3O4nh79CUU?t=
=3D764" target=3D"_blank">https://youtu.be/l3O4nh79CUU?t=3D764</a><br style=
=3D"font-family:Helvetica;font-size:12px;font-style:normal;font-variant:nor=
mal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:=
start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0=
px"></div></blockquote><div><br></div></div></div><div style=3D"word-wrap:b=
reak-word"><div><div>I do want something better, but not for the focus you =
have.=C2=A0</div><div><br></div><div>Not because what you produce was not h=
igh quality, but because quality is achieved at a very</div><div>high cost =
and is hard to uphold over generations of developer. You focus on a single =
use case</div><div>while there are many out there for distributed ledgers.<=
/div><div><br></div><div>I think in an infrastructure for enterprise applic=
ations, building consensus on the ledger is a=C2=A0</div><div>cornerstone t=
here, but is only a piece of the solution. I built several commercially suc=
cessful</div><div>deployments where I delegated the consensus building to a=
 border router, a Bitcoin Core,=C2=A0</div><div>then interfaced that truste=
d peer with my =C2=A0implementation that accepted Core=E2=80=99s decisions=
=C2=A0</div><div>in an SPV manner. One might think of this setup as wastefu=
l and unsuitable for =E2=80=9Csmall devices=E2=80=9D</div><div>therefore an=
 example of centralization people here try to avoid.</div><div><br></div><d=
iv>Enterprises have sufficient resources. Solving the business problem is v=
aluable to them even at=C2=A0</div><div>magnitudes higher cost than a hobby=
ist would bear.</div><div><br></div><div>For mainstream adoption you need t=
o get enterprises on board too, and =C2=A0that is what I care of.=C2=A0</di=
v><div>Enterprises want code that is not only high quality, but is easy to =
maintain with a development=C2=A0</div><div>team with high attrition. One h=
as to take whatever help is offered for that, and one is modern=C2=A0</div>=
<div>languages and runtimes.</div><div><br></div><div><div>Bits of Proof=E2=
=80=99s own implementation of the scripts was not practically relevant in m=
y commercially</div><div>successful deployments, because of the use of a bo=
rder router, but it helped development,=C2=A0</div><div>enabling easier deb=
ug and precise error feedback esp. end even after Core had a reject message=
.=C2=A0</div></div><div><br></div><div>I integrated libconsensus only for t=
he hope that is significantly fastens application side tx verification,</di=
v><div>=C2=A0which it has turned out it does not, until secp265k1 is integr=
ated.</div><div><br></div><div>I would likely use an other extended libcons=
ensus too, but do not think there was a dependency on=C2=A0</div><div>that =
for enterprise development.=C2=A0</div><div><br></div><div>It would help th=
ere more to have a slim protocol server, no wallet, no rpc, no qt but a hig=
h=C2=A0</div><div>performance remoting API.</div></div></div><div style=3D"=
word-wrap:break-word"><div><div><br></div><blockquote type=3D"cite"><div><s=
pan style=3D"font-family:Helvetica;font-size:12px;font-style:normal;font-va=
riant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;te=
xt-align:start;text-indent:0px;text-transform:none;white-space:normal;word-=
spacing:0px;float:none;display:inline!important">Since you already depend o=
n libconsensus for VerifyScript, wouldn&#39;t it</span><br style=3D"font-fa=
mily:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-we=
ight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-=
indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span s=
tyle=3D"font-family:Helvetica;font-size:12px;font-style:normal;font-variant=
:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-al=
ign:start;text-indent:0px;text-transform:none;white-space:normal;word-spaci=
ng:0px;float:none;display:inline!important">be nice that it also offered Ve=
rifyTx, VerifyHeader and VerifyBlock?</span><br style=3D"font-family:Helvet=
ica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal=
;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;=
text-transform:none;white-space:normal;word-spacing:0px"><span style=3D"fon=
t-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;fon=
t-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;t=
ext-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;floa=
t:none;display:inline!important">You would still have complete control over=
 storage, concurrency,</span><br style=3D"font-family:Helvetica;font-size:1=
2px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing=
:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:=
none;white-space:normal;word-spacing:0px"><span style=3D"font-family:Helvet=
ica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal=
;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;=
text-transform:none;white-space:normal;word-spacing:0px;float:none;display:=
inline!important">networking, policy...</span><br style=3D"font-family:Helv=
etica;font-size:12px;font-style:normal;font-variant:normal;font-weight:norm=
al;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0p=
x;text-transform:none;white-space:normal;word-spacing:0px"><span style=3D"f=
ont-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;f=
ont-weight:normal;letter-spacing:normal;line-height:normal;text-align:start=
;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;fl=
oat:none;display:inline!important">My plan is for the C API to interface wi=
th the external storage by</span><br style=3D"font-family:Helvetica;font-si=
ze:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spa=
cing:normal;line-height:normal;text-align:start;text-indent:0px;text-transf=
orm:none;white-space:normal;word-spacing:0px"><span style=3D"font-family:He=
lvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:no=
rmal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:=
0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;disp=
lay:inline!important">passing a function pointer to it.</span></div></block=
quote></div></div><div style=3D"word-wrap:break-word"><div></div><div><br><=
/div><div>Storage and validation is non-trivially interconnected, but I now=
 the separation can be done,</div><div>since I did it.=C2=A0</div><div><br>=
</div><div>Excuse me, but function pointers is a pattern I used in the 80=
=E2=80=99s. I know that they are behind=C2=A0</div><div>the curtain of mode=
rn abstractions with similar use, I still prefer not to see them again.</di=
v></div><div style=3D"word-wrap:break-word"><div><br></div><div><div><div>T=
amas Blummer</div><div><br></div></div><div><blockquote type=3D"cite"></blo=
ckquote></div></div></div>_______________________________________________<b=
r>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>

--047d7bd771220b3d48051df05aeb--