Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of bitpay.com
	designates 209.85.213.174 as permitted sender)
	client-ip=209.85.213.174; envelope-from=jgarzik@bitpay.com;
	helo=mail-ig0-f174.google.com; 
MIME-Version: 1.0
From: Jeff Garzik <jgarzik@bitpay.com>
Date: Tue, 16 Dec 2014 12:59:06 -0500
Message-ID: <CAJHLa0M0YeEWWaP4gNpwnaDq56w=AHW_kmLr=f3AVvHDB89SNA@mail.gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=047d7bdca2a8a90905050a591ed4
Subject: [Bitcoin-development] Open development processes and reddit charms
Precedence: list

--047d7bdca2a8a90905050a591ed4
Content-Type: text/plain; charset=UTF-8

It can be useful to review open source development processes from time to
time.  This reddit thread[1] serves use both as a case study, and also a
moment of OSS process introduction for newbies.
[1]
http://www.reddit.com/r/Bitcoin/comments/2pd0zy/peter_todd_is_saying_shoddy_development_on_v010/


*Dirty Laundry*
When building businesses or commercial software projects, outsiders
typically hear little about the internals of project development.  The
public only hears what the companies release, which is prepped and
polished. Internal disagreements, schedule slips, engineer fistfights are
all unseen.

Open source development is the opposite.  The goal is radical
transparency.  Inevitably there is private chatter (0day bugs etc.), but
the default is openness.  This means that is it normal practice to "air
dirty laundry in public."  Engineers will disagree, sometimes quietly,
sometimes loudly, sometimes rudely and with ad hominem attacks.  On the
Internet, there is a pile-on effect, where informed and uninformed
supporters add their 0.02 BTC.

Competing interests cloud the issues further.  Engineers are typically
employed by an organization, as a technology matures.  Those organizations
have different strategies and motivations.  These organizations will
sponsor work they find beneficial.  Sometimes those orgs are non-profit
foundations, sometimes for-profit corporations.  Sometimes that work is
maintenance ("keep it running"), sometimes that work is developing new,
competitive features that company feels will give it a better market
position.  In a transparent development environment, all parties are
hyperaware of these competing interests.  Internet natterers painstakingly
document and repeat every conspiracy theory about Bitcoin Foundation,
Blockstream, BitPay, various altcoin developers, and more as a result of
these competing interests.

Bitcoin and altcoin development adds an interesting new dimension.
Sometimes engineers have a more direct conflict of interest, in that the
technology they are developing is also potentially their road to instant
$millions.  Investors, amateur and professional, have direct stakes in a
certain coin or coin technology.  Engineers also have an emotional stake in
technology they design and nurture.  This results in incentives where
supporters of a non-bitcoin technology work very hard to thump bitcoin.
And vice versa.  Even inside bitcoin, you see "tree chains vs. side chains"
threads of a similar stripe.  This can lead to a very skewed debate.

That should not distract from the engineering discussion.  Starting from
first principles, Assume Good Faith[2].  Most engineers in open source tend
to mean what they say.  Typically they speak for themselves first, and
their employers value that engineer's freedom of opinion.  Pay attention to
the engineers actually working on the technology, and less attention to the
noise bubbling around the Internet like the kindergarten game of grapevine.
[2] http://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith

Being open and transparent means engineering disagreements happen in
public.  This is normal.  Open source engineers live an aquarium life[3].
[3] https://www.youtube.com/watch?v=QKe-aO44R7k


*What the fork?*
In this case, a tweet suggests consensus bug risks, which reddit account
"treeorsidechains" hyperbolizes into a dramatic headline[1].  However, the
headline would seem to be the opposite of the truth.  Several changes were
merged during 0.10 development which move snippets of source code into new
files and new sub-directories.  The general direction of this work is
creating a "libconsensus" library that carefully encapsulates consensus
code in a manner usable by external projects.  This is a good thing.

The development was performed quite responsible:  Multiple developers would
verify each cosmetic change, ensuring no behavior changes had been
accidentally (or maliciously!) introduced.  Each pull request receives a
full multi-platform build + automated testing, over and above individual
dev testing.  Comparisons at the assembly language level were sometimes
made in critical areas, to ensure zero before-and-after change.  Each
transformation gets the Bitcoin Core codebase to a more sustainable, more
reusable state.

Certainly zero-change is the most conservative approach. Strictly speaking,
that has the lowest consensus risk.  But that is a short term mentality.
Both Bitcoin Core and the larger ecosystem will benefit when the "hairball"
pile of source code is cleaned up.  Progress has been made on that front in
the past 2 years, and continues.   *Long term*, combined with the
"libconsensus" work, that leads to less community-wide risk.

The key is balance.  Continue software engineering practices -- like those
just mentioned above -- that enable change with least consensus risk.  Part
of those practices is review at each step of the development process:
social media thought bubble, mailing list post, pull request, git merge,
pre-release & release.  It probably seems chaotic at times.  In effect,
git[hub] and the Internet enable a dynamic system of review and feedback,
where each stage provides a check-and-balance for bad ideas and bad
software changes.  It's a human process, designed to acknowledge and handle
that human engineers are fallible and might make mistakes (or be
coerced/under duress!).  History and field experience will be the ultimate
judge, but I think Bitcoin Core is doing good on this score, all things
considered.

At the end of the day, while no change is without risk, version 0.10 work
was done with attention to consensus risk at multiple levels (not just
short term).


*Technical and social debt*
Working on the Linux kernel was an interesting experience that combined
git-driven parallel development and a similar source code hairball.  One of
the things that quickly became apparent is that cosmetic patches,
especially code movement, was hugely disruptive.  Some even termed it
anti-social.  To understand why, it is important to consider how modern
software changes are developed:

Developers work in parallel on their personal computers to develop XYZ
change, then submit their change "upstream" as a github pull request.  Then
time passes.  If code movement and refactoring changes are accepted
upstream before XYZ, then the developer is forced update XYZ -- typically
trivial fixes, re-review XYZ, and re-test XYZ to ensure it remains in a
known-working state.

Seemingly cosmetic changes such as code movement have a ripple effect on
participating developers, and wider developer community.  Every developer
who is *not* immediately merged upstream must bear the costs of updating
their unmerged work.

Normally, this is expected.  Encouraging developers to build on top of
"upstream" produces virtuous cycles.

However, a constant stream of code movement and cosmetic changes may
produce a constant stream of disruption to developers working on
non-trivial features that take a bit longer to develop before going
upstream.  Trivial changes are encouraged, and non-trivial changes face a
binary choice of (a) be merged immediately or (b) bear added re-base,
re-view, re-test costs.

Taken over a timescale of months, I argue that a steady stream of cosmetic
code movement changes serves as a disincentive to developers working with
upstream.  Each upstream breakage has a ripple effect to all developers
downstream, and imposes some added chance of newly introduced bugs on
downstream developers.  I'll call this "social debt", a sort of technical
debt[4] for developers.
[4] http://en.wikipedia.org/wiki/Technical_debt

As mentioned above, the libconsensus and code movement work is a net gain.
The codebase needs cleaning up.  Each change however incurs a little bit of
social debt.  Life is a little bit harder on people trying to get work into
the tree.  Developers are a little bit more discouraged at the busy-work
they must perform.  Non-trivial pull requests take a little bit longer to
approve, because they take a little bit more work to rebase (again).

A steady flow of code movement and cosmetic breakage into the tree may be a
net gain, but it also incurs a *lot* of social debt.  In such situations,
developers find that tested, working out-of-tree code repeatedly stops
working *during the process of trying to get that work in-tree*.  Taken
over time, it discourages working on the tree.  It is rational to sit back,
*not* work on the tree, let the breakage stop, and then pick up the pieces.


*Paradox Unwound*
Bitcoin Core, then, is pulled in opposite directions by a familiar
problem.  It is generally agreed that the codebase needs further
refactoring.  That's not just isolated engineer nit-picking.  However, for
non-trivial projects, refactoring is always anti-social in the short term.
It impacts projects other than your own, projects you don't even know
about. One change causes work for N developers.  Given these twin opposing
goals, the key, as ever, is finding the right balance.

Much like "feature freeze" in other software projects, developing a policy
that opens and closes windows for code movement and major disruptive
changes seems prudent.  One week of code movement & cosmetics followed by 3
weeks without, for example.  Part of open source parallel development
is *social
signalling*:  Signal to developers when certain changes are favored or not,
then trust they can handle the rest from there.

While recent code movement commits themselves are individually ACK-worthy,
professionally executed and moving towards a positive goal, I think the
project could strike a better balance when it comes to disruptive cosmetic
changes, a balance that better encourages developers to work on more
involved Bitcoin Core projects.


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

--047d7bdca2a8a90905050a591ed4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><br></div>It can be useful t=
o review open source development processes from time to time.=C2=A0 This re=
ddit thread[1] serves use both as a case study, and also a moment of OSS pr=
ocess introduction for newbies.<br>[1] <a href=3D"http://www.reddit.com/r/B=
itcoin/comments/2pd0zy/peter_todd_is_saying_shoddy_development_on_v010/">ht=
tp://www.reddit.com/r/Bitcoin/comments/2pd0zy/peter_todd_is_saying_shoddy_d=
evelopment_on_v010/</a><br><br><br></div><div><b>Dirty Laundry<br><br></b><=
/div>When building businesses or commercial software projects, outsiders ty=
pically hear little about the internals of project development.=C2=A0 The p=
ublic only hears what the companies release, which is prepped and polished.=
 Internal disagreements, schedule slips, engineer fistfights are all unseen=
.<br><br></div>Open source development is the opposite.=C2=A0 The goal is r=
adical transparency.=C2=A0 Inevitably there is private chatter (0day bugs e=
tc.), but the default is openness.=C2=A0 This means that is it normal pract=
ice to &quot;air dirty laundry in public.&quot;=C2=A0 Engineers will disagr=
ee, sometimes quietly, sometimes loudly, sometimes rudely and with ad homin=
em attacks.=C2=A0 On the Internet, there is a pile-on effect, where informe=
d and uninformed supporters add their 0.02 BTC.<br><br></div>Competing inte=
rests cloud the issues further.=C2=A0 Engineers are typically employed by a=
n organization, as a technology matures.=C2=A0 Those organizations have dif=
ferent strategies and motivations.=C2=A0 These organizations will sponsor w=
ork they find beneficial.=C2=A0 Sometimes those orgs are non-profit foundat=
ions, sometimes for-profit corporations.=C2=A0 Sometimes that work is maint=
enance (&quot;keep it running&quot;), sometimes that work is developing new=
, competitive features that company feels will give it a better market posi=
tion.=C2=A0 In a transparent development environment, all parties are hyper=
aware of these competing interests.=C2=A0 Internet natterers painstakingly =
document and repeat every conspiracy theory about Bitcoin Foundation, Block=
stream, BitPay, various altcoin developers, and more as a result of these c=
ompeting interests.<br><br></div>Bitcoin and altcoin development adds an in=
teresting new dimension.=C2=A0 Sometimes engineers have a more direct confl=
ict of interest, in that the technology they are developing is also potenti=
ally their road to instant $millions.=C2=A0 Investors, amateur and professi=
onal, have direct stakes in a certain coin or coin technology.=C2=A0 Engine=
ers also have an emotional stake in technology they design and nurture.=C2=
=A0 This results in incentives where supporters of a non-bitcoin technology=
 work very hard to thump bitcoin.=C2=A0 And vice versa.=C2=A0 Even inside b=
itcoin, you see &quot;tree chains vs. side chains&quot; threads of a simila=
r stripe.=C2=A0 This can lead to a very skewed debate.<br><br>That should n=
ot distract from the engineering discussion.=C2=A0 Starting from first prin=
ciples, Assume Good Faith[2].=C2=A0 Most engineers in open source tend to m=
ean what they say.=C2=A0 Typically they speak for themselves first, and the=
ir employers value that engineer&#39;s freedom of opinion.=C2=A0 Pay attent=
ion to the engineers actually working on the technology, and less attention=
 to the noise bubbling around the Internet like the kindergarten game of gr=
apevine.<br>[2] <a href=3D"http://en.wikipedia.org/wiki/Wikipedia:Assume_go=
od_faith">http://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith</a><br><=
br></div><div>Being open and transparent means engineering disagreements ha=
ppen in public.=C2=A0 This is normal.=C2=A0 Open source engineers live an a=
quarium life[3].<br>[3] <a href=3D"https://www.youtube.com/watch?v=3DQKe-aO=
44R7k">https://www.youtube.com/watch?v=3DQKe-aO44R7k</a><br><br><br></div><=
div><b>What the fork?<br><br></b></div><div>In this case, a tweet suggests =
consensus bug risks, which reddit account &quot;treeorsidechains&quot; hype=
rbolizes into a dramatic headline[1].=C2=A0 However, the headline would see=
m to be the opposite of the truth.=C2=A0 Several changes were merged during=
 0.10 development which move snippets of source code into new files and new=
 sub-directories.=C2=A0 The general direction of this work is creating a &q=
uot;libconsensus&quot; library that carefully encapsulates consensus code i=
n a manner usable by external projects.=C2=A0 This is a good thing.<br><br>=
</div><div>The development was performed quite responsible:=C2=A0 Multiple =
developers would verify each cosmetic change, ensuring no behavior changes =
had been accidentally (or maliciously!) introduced.=C2=A0 Each pull request=
 receives a full multi-platform build + automated testing, over and above i=
ndividual dev testing.=C2=A0 Comparisons at the assembly language level wer=
e sometimes made in critical areas, to ensure zero before-and-after change.=
=C2=A0 Each transformation gets the Bitcoin Core codebase to a more sustain=
able, more reusable state.<br><br></div><div>Certainly zero-change is the m=
ost conservative approach. Strictly speaking, that has the lowest consensus=
 risk.=C2=A0 But that is a short term mentality.=C2=A0 Both Bitcoin Core an=
d the larger ecosystem will benefit when the &quot;hairball&quot; pile of s=
ource code is cleaned up.=C2=A0 Progress has been made on that front in the=
 past 2 years, and continues. =C2=A0 <i>Long term</i>, combined with the &q=
uot;libconsensus&quot; work, that leads to less community-wide risk.<br><br=
>The key is balance.=C2=A0 Continue software engineering practices -- like =
those just mentioned above -- that enable change with least consensus risk.=
=C2=A0 Part of those practices is review at each step of the development pr=
ocess:=C2=A0 social media thought bubble, mailing list post, pull request, =
git merge, pre-release &amp; release.=C2=A0 It probably seems chaotic at ti=
mes.=C2=A0 In effect, git[hub] and the Internet enable a dynamic system of =
review and feedback, where each stage provides a check-and-balance for bad =
ideas and bad software changes.=C2=A0 It&#39;s a human process, designed to=
 acknowledge and handle that human engineers are fallible and might make mi=
stakes (or be coerced/under duress!).=C2=A0 History and field experience wi=
ll be the ultimate judge, but I think Bitcoin Core is doing good on this sc=
ore, all things considered.<br><br></div><div>At the end of the day, while =
no change is without risk, version 0.10 work was done with attention to con=
sensus risk at multiple levels (not just short term).<br><br><br></div><div=
><b>Technical and social debt<br><br></b></div><div>Working on the Linux ke=
rnel was an interesting experience that combined git-driven parallel develo=
pment and a similar source code hairball.=C2=A0 One of the things that quic=
kly became apparent is that cosmetic patches, especially code movement, was=
 hugely disruptive.=C2=A0 Some even termed it anti-social.=C2=A0 To underst=
and why, it is important to consider how modern software changes are develo=
ped:<br><br></div><div>Developers work in parallel on their personal comput=
ers to develop XYZ change, then submit their change &quot;upstream&quot; as=
 a github pull request.=C2=A0 Then time passes.=C2=A0 If code movement and =
refactoring changes are accepted upstream before XYZ, then the developer is=
 forced update XYZ -- typically trivial fixes, re-review XYZ, and re-test X=
YZ to ensure it remains in a known-working state.<br><br></div><div>Seeming=
ly cosmetic changes such as code movement have a ripple effect on participa=
ting developers, and wider developer community.=C2=A0 Every developer who i=
s <i>not</i> immediately merged upstream must bear the costs of updating th=
eir unmerged work.</div><div><br></div><div>Normally, this is expected.=C2=
=A0 Encouraging developers to build on top of &quot;upstream&quot; produces=
 virtuous cycles.<br><br></div><div>However, a constant stream of code move=
ment and cosmetic changes may produce a constant stream of disruption to de=
velopers working on non-trivial features that take a bit longer to develop =
before going upstream.=C2=A0 Trivial changes are encouraged, and non-trivia=
l changes face a binary choice of (a) be merged immediately or (b) bear add=
ed re-base, re-view, re-test costs.<br><br></div><div>Taken over a timescal=
e of months, I argue that a steady stream of cosmetic code movement changes=
 serves as a disincentive to developers working with upstream.=C2=A0 Each u=
pstream breakage has a ripple effect to all developers downstream, and impo=
ses some added chance of newly introduced bugs on downstream developers.=C2=
=A0 I&#39;ll call this &quot;social debt&quot;, a sort of technical debt[4]=
 for developers.<br>[4] <a href=3D"http://en.wikipedia.org/wiki/Technical_d=
ebt">http://en.wikipedia.org/wiki/Technical_debt</a><br></div><div><br></di=
v><div>As mentioned above, the libconsensus and code movement work is a net=
 gain.=C2=A0 The codebase needs cleaning up.=C2=A0 Each change however incu=
rs a little bit of social debt.=C2=A0 Life is a little bit harder on people=
 trying to get work into the tree.=C2=A0 Developers are a little bit more d=
iscouraged at the busy-work they must perform.=C2=A0 Non-trivial pull reque=
sts take a little bit longer to approve, because they take a little bit mor=
e work to rebase (again).<br><br></div><div>A steady flow of code movement =
and cosmetic breakage into the tree may be a net gain, but it also incurs a=
 <i>lot</i> of social debt.=C2=A0 In such situations, developers find that =
tested, working out-of-tree code repeatedly stops working <i>during the pro=
cess of trying to get that work in-tree</i>.=C2=A0 Taken over time, it disc=
ourages working on the tree.=C2=A0 It is rational to sit back, <i>not</i> w=
ork on the tree, let the breakage stop, and then pick up the pieces.<br></d=
iv><div><b><br><br></b></div><div><b>Paradox Unwound<br><br></b></div><div>=
Bitcoin Core, then, is pulled in opposite directions by a familiar problem.=
=C2=A0 It is generally agreed that the codebase needs further refactoring.=
=C2=A0 That&#39;s not just isolated engineer nit-picking.=C2=A0 However, fo=
r non-trivial projects, refactoring is always anti-social in the short term=
.=C2=A0 It impacts projects other than your own, projects you don&#39;t eve=
n know about. One change causes work for N developers.=C2=A0 Given these tw=
in opposing goals, the key, as ever, is finding the right balance.<br><br><=
/div><div>Much like &quot;feature freeze&quot; in other software projects, =
developing a policy that opens and closes windows for code movement and maj=
or disruptive changes seems prudent.=C2=A0 One week of code movement &amp; =
cosmetics followed by 3 weeks without, for example.=C2=A0 Part of open sour=
ce parallel development is <i>social signalling</i>:=C2=A0 Signal to develo=
pers when certain changes are favored or not, then trust they can handle th=
e rest from there.<br><br></div><div>While recent code movement commits the=
mselves are individually ACK-worthy, professionally executed and moving tow=
ards a positive goal, I think the project could strike a better balance whe=
n it comes to disruptive cosmetic changes, a balance that better encourages=
 developers to work on more involved Bitcoin Core projects.<br><br></div><d=
iv><b><br></b></div><div>-- <br></div><div><div><div><div><div><div><div><d=
iv><div class=3D"gmail_signature">Jeff Garzik<br>Bitcoin core developer and=
 open source evangelist<br>BitPay, Inc. =C2=A0 =C2=A0 =C2=A0<a href=3D"http=
s://bitpay.com/" target=3D"_blank">https://bitpay.com/</a></div>
</div></div></div></div></div></div></div></div></div>

--047d7bdca2a8a90905050a591ed4--