Date: Tue, 16 Dec 2014 13:24:15 -0600
From: Troy Benjegerdes <hozer@hozed.org>
To: Jeff Garzik <jgarzik@bitpay.com>
Message-ID: <20141216192415.GQ29130@nl.grid.coop>
References: <CAJHLa0M0YeEWWaP4gNpwnaDq56w=AHW_kmLr=f3AVvHDB89SNA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
In-Reply-To: <CAJHLa0M0YeEWWaP4gNpwnaDq56w=AHW_kmLr=f3AVvHDB89SNA@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Open development processes and reddit
 charms
Precedence: list

Thank you Jeff.

Having looked at a lot of linux code, and now a lot of bitcoin code, the
biggest long-term systemic risk I see is that Bitcoin has is the lack of 
code janitors.

The problem is that janitoring was *disruptive* for non-x86 linux architectures
when it first got going, and it's going to be very disruptive for bitcoin as
well, but it **needs** to happen. 

The code is too complex and hard to follow as it is now. (now, I could just
be speaking because I haven't paid the social debt of looking at the latest
bitcoin code, including libconsensus), but there really needs to be a focus
on readability, maintainability, and (as much as I hate to say it) a rather
hard-line policy on coding standards.

I don't care which tabbing style or column width you pick, but **pick one**,
and enforce it across the entire codebase.

Maybe this should be bitcoin-stable, and bitcoin-devel, with a 6-9 month
social expectation of minimal cosmetic changes in -stable, with a 1 month
'merge window' where -devel turns into -stable.


On Tue, Dec 16, 2014 at 12:59:06PM -0500, Jeff Garzik wrote:
> It can be useful to review open source development processes from time to
> time.  This reddit thread[1] serves use both as a case study, and also a
> moment of OSS process introduction for newbies.
> [1]
> http://www.reddit.com/r/Bitcoin/comments/2pd0zy/peter_todd_is_saying_shoddy_development_on_v010/
> 
> 
> 
> 
> *Dirty Laundry*
> When building businesses or commercial software projects, outsiders
> typically hear little about the internals of project development.  The
> public only hears what the companies release, which is prepped and
> polished. Internal disagreements, schedule slips, engineer fistfights are
> all unseen.
> 
> Open source development is the opposite.  The goal is radical
> transparency.  Inevitably there is private chatter (0day bugs etc.), but
> the default is openness.  This means that is it normal practice to "air
> dirty laundry in public."  Engineers will disagree, sometimes quietly,
> sometimes loudly, sometimes rudely and with ad hominem attacks.  On the
> Internet, there is a pile-on effect, where informed and uninformed
> supporters add their 0.02 BTC.
> 
> Competing interests cloud the issues further.  Engineers are typically
> employed by an organization, as a technology matures.  Those organizations
> have different strategies and motivations.  These organizations will
> sponsor work they find beneficial.  Sometimes those orgs are non-profit
> foundations, sometimes for-profit corporations.  Sometimes that work is
> maintenance ("keep it running"), sometimes that work is developing new,
> competitive features that company feels will give it a better market
> position.  In a transparent development environment, all parties are
> hyperaware of these competing interests.  Internet natterers painstakingly
> document and repeat every conspiracy theory about Bitcoin Foundation,
> Blockstream, BitPay, various altcoin developers, and more as a result of
> these competing interests.
> 
> Bitcoin and altcoin development adds an interesting new dimension.
> Sometimes engineers have a more direct conflict of interest, in that the
> technology they are developing is also potentially their road to instant
> $millions.  Investors, amateur and professional, have direct stakes in a
> certain coin or coin technology.  Engineers also have an emotional stake in
> technology they design and nurture.  This results in incentives where
> supporters of a non-bitcoin technology work very hard to thump bitcoin.
> And vice versa.  Even inside bitcoin, you see "tree chains vs. side chains"
> threads of a similar stripe.  This can lead to a very skewed debate.
> 
> That should not distract from the engineering discussion.  Starting from
> first principles, Assume Good Faith[2].  Most engineers in open source tend
> to mean what they say.  Typically they speak for themselves first, and
> their employers value that engineer's freedom of opinion.  Pay attention to
> the engineers actually working on the technology, and less attention to the
> noise bubbling around the Internet like the kindergarten game of grapevine.
> [2] http://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith
> 
> Being open and transparent means engineering disagreements happen in
> public.  This is normal.  Open source engineers live an aquarium life[3].
> [3] https://www.youtube.com/watch?v=QKe-aO44R7k
> 
> 
> 
> 
> *What the fork?*
> In this case, a tweet suggests consensus bug risks, which reddit account
> "treeorsidechains" hyperbolizes into a dramatic headline[1].  However, the
> headline would seem to be the opposite of the truth.  Several changes were
> merged during 0.10 development which move snippets of source code into new
> files and new sub-directories.  The general direction of this work is
> creating a "libconsensus" library that carefully encapsulates consensus
> code in a manner usable by external projects.  This is a good thing.
> 
> The development was performed quite responsible:  Multiple developers would
> verify each cosmetic change, ensuring no behavior changes had been
> accidentally (or maliciously!) introduced.  Each pull request receives a
> full multi-platform build + automated testing, over and above individual
> dev testing.  Comparisons at the assembly language level were sometimes
> made in critical areas, to ensure zero before-and-after change.  Each
> transformation gets the Bitcoin Core codebase to a more sustainable, more
> reusable state.
> 
> Certainly zero-change is the most conservative approach. Strictly speaking,
> that has the lowest consensus risk.  But that is a short term mentality.
> Both Bitcoin Core and the larger ecosystem will benefit when the "hairball"
> pile of source code is cleaned up.  Progress has been made on that front in
> the past 2 years, and continues.   *Long term*, combined with the
> "libconsensus" work, that leads to less community-wide risk.
> 
> The key is balance.  Continue software engineering practices -- like those
> just mentioned above -- that enable change with least consensus risk.  Part
> of those practices is review at each step of the development process:
> social media thought bubble, mailing list post, pull request, git merge,
> pre-release & release.  It probably seems chaotic at times.  In effect,
> git[hub] and the Internet enable a dynamic system of review and feedback,
> where each stage provides a check-and-balance for bad ideas and bad
> software changes.  It's a human process, designed to acknowledge and handle
> that human engineers are fallible and might make mistakes (or be
> coerced/under duress!).  History and field experience will be the ultimate
> judge, but I think Bitcoin Core is doing good on this score, all things
> considered.
> 
> At the end of the day, while no change is without risk, version 0.10 work
> was done with attention to consensus risk at multiple levels (not just
> short term).
> 
> 
> 
> 
> *Technical and social debt*
> Working on the Linux kernel was an interesting experience that combined
> git-driven parallel development and a similar source code hairball.  One of
> the things that quickly became apparent is that cosmetic patches,
> especially code movement, was hugely disruptive.  Some even termed it
> anti-social.  To understand why, it is important to consider how modern
> software changes are developed:
> 
> Developers work in parallel on their personal computers to develop XYZ
> change, then submit their change "upstream" as a github pull request.  Then
> time passes.  If code movement and refactoring changes are accepted
> upstream before XYZ, then the developer is forced update XYZ -- typically
> trivial fixes, re-review XYZ, and re-test XYZ to ensure it remains in a
> known-working state.
> 
> Seemingly cosmetic changes such as code movement have a ripple effect on
> participating developers, and wider developer community.  Every developer
> who is *not* immediately merged upstream must bear the costs of updating
> their unmerged work.
> 
> Normally, this is expected.  Encouraging developers to build on top of
> "upstream" produces virtuous cycles.
> 
> However, a constant stream of code movement and cosmetic changes may
> produce a constant stream of disruption to developers working on
> non-trivial features that take a bit longer to develop before going
> upstream.  Trivial changes are encouraged, and non-trivial changes face a
> binary choice of (a) be merged immediately or (b) bear added re-base,
> re-view, re-test costs.
> 
> Taken over a timescale of months, I argue that a steady stream of cosmetic
> code movement changes serves as a disincentive to developers working with
> upstream.  Each upstream breakage has a ripple effect to all developers
> downstream, and imposes some added chance of newly introduced bugs on
> downstream developers.  I'll call this "social debt", a sort of technical
> debt[4] for developers.
> [4] http://en.wikipedia.org/wiki/Technical_debt
> 
> As mentioned above, the libconsensus and code movement work is a net gain.
> The codebase needs cleaning up.  Each change however incurs a little bit of
> social debt.  Life is a little bit harder on people trying to get work into
> the tree.  Developers are a little bit more discouraged at the busy-work
> they must perform.  Non-trivial pull requests take a little bit longer to
> approve, because they take a little bit more work to rebase (again).
> 
> A steady flow of code movement and cosmetic breakage into the tree may be a
> net gain, but it also incurs a *lot* of social debt.  In such situations,
> developers find that tested, working out-of-tree code repeatedly stops
> working *during the process of trying to get that work in-tree*.  Taken
> over time, it discourages working on the tree.  It is rational to sit back,
> *not* work on the tree, let the breakage stop, and then pick up the pieces.
> 
> 
> 
> 
> *Paradox Unwound*
> Bitcoin Core, then, is pulled in opposite directions by a familiar
> problem.  It is generally agreed that the codebase needs further
> refactoring.  That's not just isolated engineer nit-picking.  However, for
> non-trivial projects, refactoring is always anti-social in the short term.
> It impacts projects other than your own, projects you don't even know
> about. One change causes work for N developers.  Given these twin opposing
> goals, the key, as ever, is finding the right balance.
> 
> Much like "feature freeze" in other software projects, developing a policy
> that opens and closes windows for code movement and major disruptive
> changes seems prudent.  One week of code movement & cosmetics followed by 3
> weeks without, for example.  Part of open source parallel development
> is *social
> signalling*:  Signal to developers when certain changes are favored or not,
> then trust they can handle the rest from there.
> 
> While recent code movement commits themselves are individually ACK-worthy,
> professionally executed and moving towards a positive goal, I think the
> project could strike a better balance when it comes to disruptive cosmetic
> changes, a balance that better encourages developers to work on more
> involved Bitcoin Core projects.