MIME-Version: 1.0
Sender: gmaxwell@gmail.com
In-Reply-To: <0119661e-a11a-6d4b-c9ec-fd510bd4f144@gmail.com>
References: <0119661e-a11a-6d4b-c9ec-fd510bd4f144@gmail.com>
From: Gregory Maxwell <greg@xiph.org>
Date: Tue, 11 Jul 2017 21:11:38 +0000
Message-ID: <CAAS2fgRDVgdMYZo776iLwbm23aGNDWL85YgD=yF=M-0_vqJ5nQ@mail.gmail.com>
To: Paul Sztorc <truthcoin@gmail.com>, 
	Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: [bitcoin-dev] Updating the Scaling Roadmap
Precedence: list

I think it's great that people want to experiment with things like
drivechains/sidechains and what not, but their security model is very
distinct from Bitcoin's and, given the current highly centralized
mining ecosystem, arguably not very good.  So positioning them as a
major solution for the Bitcoin project is the wrong way to go. Instead
we should support people trying cool stuff, at their own risk.

So, given that although the vast majority of the things in the document
are things I've been supporting for months (Please see Note 1 way down
at the bottom) I cannot support your document.

I think you may have have missed how much work I put into what I published
before talking with people who actually work on the project to find out
what they wouldn't object to before publishing the prior document--
and how much I left out that I would have loved to have in; and why
I specifically held back from describing it as a roadmap or prompting
people to sign onto it (though they did of their own accord).

On a more meta-subject, I think grandly stated "top down" roadmaps
in highly collaborative development are of minimal utility at best and
actively misleading at worst. Fundamentally, it misunderstands the nature
of peer collaboration. It's kind of like asking for a roadmap for the
development of fusion power; individual practitioners have their own
roadmaps, but the collaboration of science does not.

Consider an example,

The Linux kernel is one of the largest and best funded open source
projects, which produces the most widely used operating system kernel
in the world and one of the most widely used pieces of software of all
time, and it produces _no_ roadmaps.

Quoting Andrew Morton, "Instead of a roadmap, there are technical
guidelines. Instead of a central resource allocation, there are
persons and companies who all have a stake in the further development
of the Linux kernel, quite independently from one another: People like
Linus Torvalds and I don=E2=80=99t plan the kernel evolution. We don=E2=80=
=99t sit
there and think up the roadmap for the next two years, then assign
resources to the various new features. That's because we don=E2=80=99t have
any resources. The resources are all owned by the various corporations
who use and contribute to Linux, as well as by the various independent
contributors out there. It's those people who own the resources who
decide."

Linus remarked, "I look at the current release and the next one, as I
don't think planning 10 years ahead is sane."

Yet the Linux kernel still has every advantage over us:  They have far
more contributing resources from far more sources, they have a fairly
centralized model and control over their own destiny because they have
a much more functional pathway to disagreement than we have in Bitcoin
for consensus rules.

IMO the way to do "roadmaps" in Bitcoin is to roadmap the finalization
and release process once the basic technology is done; because it's
only past that point that guarantees can really start being made.

But that isn't what your document does-- it names a lot of things which
are still various shades of research at this point (much of it research
I'm working on and strongly support, in fact--)

We can also talk in a visionary sense about the sorts of things we're
exploring, but it just isn't possible to nail down when they'll be
ready until they are: If it's not something the linux kernel can do,
it's not something we can do.  These are neat and existing ideas,
but not a roadmap.

At least not as a group. Individually contributing parties have a lot
more visibility and control into their own activities, and there is
virtue in forecasting at that level. Redhat, for example, has
roadmaps. They primarily deal with stabilization and support of
already existing technology that you could get in the raw from the
various upstream sources (fedora, kernel, etc.). (see for an example,
http://www.slideshare.net/albertspijkers/red-hat-enterprise-linux-roadmap
)

Separately, what we can do stably in Core is have timely reliable
releases.  Juniper made it 10 years with regular timed releases that
did not slip by more than IIRC three days and which were _all_ production
deployable (things changed later, but thats another story).

This was an incredible benefit to our customers, but the only way it was
possible was because _features_ were not guaranteed in a release.
If a feature failed during the final testing and it needed more than the
most trivial of fixes, it was _removed_ or disabled.  As soon as there
are multiple in-flight deliverables it becomes more important to be
timely over-all even that that significantly delays single deliverables.
This is effectively at odds with hard deadlines on functionality, even
before getting into the fact that for consensus features delivery in
software is irrelevant until activation which is a public election.
(E.g. we shipped segwit almost a year ago, but it still hasn't arrived
for users).

From the perspective of Bitcoin I think what people are actually
asking for us to do is to (help) drive the Bitcoin Story-- the actual
technology and its timelines is usually not that relevant: if it were,
they'd be stepping up with resources to contribute to it. There are
many ways to do that, -- we don't have to use highly authoritarian
methods that wouldn't even work for the Linux kernel.

[And all these projects you listed, your help would be more than welcome!]

This can be done by a mixture of talking about research _as research_
not a done deal, and by moving discussions of non-research things to
where they're actually more forecastable.  98% of the public
discussion about segwit was before the pull request, -- there were
solid political reasons for this-- but it was the wrong timing.  On
the case of CSV and CLTV, the general public didn't hear about them
until they were merged -- pretty much-- and the timing then was much
more compatible with 'roadmapping' +/- activation uncertainty.

Some of this is driven by competitive pressure with things like
Ethereum or other altcoins (e.g. dash, god save us) that have a lot
lower commitment to engineering integrity or even logical consistency.
We can't compete with technobabble bullshit, and we shouldn't try and
as a side effect back ourselves into a corner where we're making
remarkable accomplishments but regarded as failures in payment
(because we either forecast it taking N years, which is the best we
could promise, or because we forecast the best we might achieve and it
was both still too long and the timeframe got missed).

One of the big screwups with segwit handling was people sticking
some random unrealistic date on it it which was done AFAIK,
by well meaning folks who took some random numbers from people
working on it for when they'd be done with the development-- not the
testing, not the integration, and certainly not deployment and published
it as The Date.  Then even though the development was actually done
by them, segwit was portrayed as massively delayed, because what
matters to the users is deployment-- which we can't control.

I see you've done this yourself with signature aggregation, sticking Q4 201=
6
right on it, which as far as I can tell is a figure that comes from mars.
(Well not quite mars, see Note 1)
Probably we'll have the research and an implementation done by then, but
with so much uncertainty around segwit activation, I doubt anyone can be
about when users will be able to use it (which is what they care about!)

It's also not really appropriate to ask people to sign onto stuff when they
can't even review it.  Perhaps the signature aggregation stuff is insecure,
patent encumbered, or practically useless... (It won't be but no one could
tell that from your post; because it doesn't even exist as a concrete propo=
sal)

I think people would rightly protest about a number of these things-- espec=
ially
things like the the agg sigs and tx compaction, "wtf, I've not heard
of this. Who
are you to insist this goes into Bitcoin?"

In any case; I can repeat the same story for major RFCs, FWIW.  Collaborati=
ve
innovation is _very_ hard to stick to a tight schedule.  And road-maps
of totally prospective technology that no one has the actual authority to m=
ake
happen aren't a productive thing for the industry.

In a few weeks you'll start seeing articles on the major new things
coming in Bitcoin Core 0.15,
-- now that we can do, because the work is done, and the outcome is
very predictable. There are some awesome improvements around it-- ones
we can rally around; and know will be delivered for sure.


[  Note 1: I think it is important to disclose that several of the
items in this list appear to be more or less quoted out of my own
blockstream-internal descriptions of things we've been working on in
Bitcoin.

A while back Adam Back asked me to publish something which contained
significant chunks of this document more or less verbatim, and I
declined saying that I personally disagree with some of his points and
didn't think that Blockstream attempting to redirect the Bitcoin
project (esp towards drivechains) was appropriate-- along with my
(above) views on roadmaps (which I have included here a private email
thread on the subject). I feel it's important to disclose this, and
that the document was not otherwise created with the input of project
contributors (except Luke-Jr, apparently). I wasn't previously aware
that Adam had been working with Paul on this, had I been I would have
also encouraged people to be a little more transparent about it. ]