Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
	designates 74.125.82.53 as permitted sender)
	client-ip=74.125.82.53; envelope-from=pieter.wuille@gmail.com;
	helo=mail-wg0-f53.google.com; 
MIME-Version: 1.0
In-Reply-To: <20120610090357.GA25567@vps7135.xlshosting.net>
References: <20120610090357.GA25567@vps7135.xlshosting.net>
Date: Sun, 10 Jun 2012 16:18:47 +0200
Message-ID: <CAPg+sBiqi_NDGOfhzrFy1V3o24kPnxvGtYzsSsk6iy9dyyhF8w@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=0016e6db2e7061274e04c21ee9be
Subject: [Bitcoin-development] BIP22/getmemorypool
Precedence: list

--0016e6db2e7061274e04c21ee9be
Content-Type: text/plain; charset=ISO-8859-1

Hello everyone,

Luke's getmemorypool/BIP22 pull request has been open for a long time, and
didn't receive too much discussion.

I think that having a stable and flexible API for negotiating block
generation is important to be standardized. The fact that it allows moving
block generation to specialized programs is a step in the right direction.
However, it seems to me that too few people (myself included) understand
all the details of BIP22 (or don't care enough) to judge their necessity. I
gave up trying to follow all design decisions some time ago, and as it
seems I'm not alone, nobody liked merging support for it in the Satoshi
client. This is a pity, and I hope the situation can be improved soon.

I'm sorry for being this late with these comments, but I think it's
essential that the standard is not more complex than necessary (making it
as easy as possible to write either servers or clients for it), and perhaps
even more important, that its purpose and intended use cases are clear.

From what I understand, the three subrequests are template, proposal and
submit. The general idea is that
 1) a miner requests a block template
 2) builds/modifies a block based on this, and optionally uses propose to
check whether the server is willing to accept it before doing work
 3) submits when valid proof-of-work is found
I'd like to see this process described in the BIP at least, it too me way
too long to extract this.

Regarding the block template: is there a particular reason for sending the
full transactions (serialized in hex) both in templates and submissions?
The server will always need to have access to the transaction database
anyway, and clients will (afaics) rarely care about the actual
transactions. My suggestion would be to just transfer txids - if the client
is interested in the actual transactions, we have the gettransaction RPC
call already. This seems to be captured by the several "submit/*" and
"share/*" variations, but making it optional seems way more complex than
just limiting the API to that way of working.

That's another thing that bothers me in the standard: too many optional
features. In particular, I understand the usefulness of having some
flexibility in what miner clients are allowed to modify, but I'm
unconvinced we need 10 individually selectable variations. In particular:
* coinbase outputs: can we just add a list of required coinbase outputs
(amount + scriptPubKey) to the template? If no generation+fee amount
remains, nothing can be added.
* coinbase input: put the required part in the template; miners can always
add whatever they like. Is there any known use case where a server would
not allow a client to add stuff to the coinbase?
* noncerange limiting: if coinbase input variation is not limited, there is
certainly no reason to limit nonceranges. This adds unnecessary complexity
to clients, in my option.
* time/*: put a minimum and maximum timestamp in the template (i believe
those are already there anyway). Anything in between is valid.
* transactions/add: what is the use case?
* transactions/remove: i'd just standarize on having all transactions be
removable (perhaps except those marked 'required').
* prevblock: one getmemorypool per new block shouldn't be a problem imho,
so do a longpoll instead of having the client able to modify prevblock
themselves.

One more thing that I do not like is often several ways for specifying the
same behaviour. For example, txrequires specifies that the first N
transactions are mandatory, a 'required' field in the transaction list
itself specifies that that transaction is mandatory, and the lack of
transactions as variation means that they must not be touched at all. Pick
one way that is flexible enough, and discard the others.

In summary, I'd like to see the standard simplified. I have no problem
merging code that makes getmemorypool compliant to a standard that is
agreed upon, but like to understand it first.

In my opinion - but I'm certainly open to discussion here - the standard
could be simplified to:
* getblocktemplate: create a new block template, and return it. The result
contains:
 * bits, previousblockhash, version: as to be used in block
 * curtime, maxtimeoff, maxtimeoff: client chooses a timestamp between
(curtime - local_time_at_receipt + local_time), decreased by mintimeoff and
increased maxtimeoff
 * expires, sigoplimit, sizelimit: unchanged
 * subsidy: amount generated (5000000000 for now)
 * coinbaseaux: what generated coinbase's scriptSig must start with
 * coinbaseoutputs: list of objects, each specifying a required coinbase
output. Each has fields:
   * amount: sent amount
   * scriptPubKey: hex serialized of the output script
 * transactions: list of object, each specifying a suggested transaction
(except for the coinbase) in the generated block. Each has fields:
   * txid: transaction id
   * depends: list of dependencies (txids of earlier objects in this same
transactions list).
   * fee: fee generated by this transaction, which increases the max output
of the coinbase.
   * required: if present, transaction may not be dropped.
* submitblocktemplate: submit an object containing a hex serialized block
header, hex serialized coinbase transaction, and a list of txids. Returns
true or string describing the problem. Proof of work is checked last, so
that error is only returned if there is no other problem with the suggested
block (this allows it to replace both propose and submit).

Are there important use cases I'm missing?

--
Pieter

--0016e6db2e7061274e04c21ee9be
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<p><br>
Hello everyone,</p>
<p>Luke&#39;s getmemorypool/BIP22 pull request has been open for a long tim=
e, and didn&#39;t receive too much discussion.</p>
<p>I think that having a stable and flexible API for negotiating block gene=
ration is important to be standardized. The fact that it allows moving bloc=
k generation to specialized programs is a step in the right direction. Howe=
ver, it seems to me that too few people (myself included) understand all th=
e details of BIP22 (or don&#39;t care enough) to judge their necessity. I g=
ave up trying to follow all design decisions some time ago, and as it seems=
 I&#39;m not alone, nobody liked merging support for it in the Satoshi clie=
nt. This is a pity, and I hope the situation can be improved soon.</p>

<p>I&#39;m sorry for being this late with these comments, but I think it=
9;s essential that the standard is not more complex than necessary (making =
it as easy as possible to write either servers or clients for it), and perh=
aps even more important, that its purpose and intended use cases are clear.=
</p>

<p>From what I understand, the three subrequests are template, proposal and=
 submit. The general idea is that<br>
=A01) a miner requests a block template<br>
=A02) builds/modifies a block based on this, and optionally uses propose to=
 check whether the server is willing to accept it before doing work<br>
=A03) submits when valid proof-of-work is found<br>
I&#39;d like to see this process described in the BIP at least, it too me w=
ay too long to extract this.</p>
<p>Regarding the block template: is there a particular reason for sending t=
he full transactions (serialized in hex) both in templates and submissions?=
 The server will always need to have access to the transaction database any=
way, and clients will (afaics) rarely care about the actual transactions. M=
y suggestion would be to just transfer txids - if the client is interested =
in the actual transactions, we have the gettransaction RPC call already. Th=
is seems to be captured by the several &quot;submit/*&quot; and &quot;share=
/*&quot; variations, but making it optional seems way more complex than jus=
t limiting the API to that way of working.</p>

<p>That&#39;s another thing that bothers me in the standard: too many optio=
nal features. In particular, I understand the usefulness of having some fle=
xibility in what miner clients are allowed to modify, but I&#39;m unconvinc=
ed we need 10 individually selectable variations. In particular:<br>

* coinbase outputs: can we just add a list of required coinbase outputs (am=
ount + scriptPubKey) to the template? If no generation+fee amount remains, =
nothing can be added.<br>
* coinbase input: put the required part in the template; miners can always =
add whatever they like. Is there any known use case where a server would no=
t allow a client to add stuff to the coinbase?<br>
* noncerange limiting: if coinbase input variation is not limited, there is=
 certainly no reason to limit nonceranges. This adds unnecessary complexity=
 to clients, in my option.<br>
* time/*: put a minimum and maximum timestamp in the template (i believe th=
ose are already there anyway). Anything in between is valid.<br>
* transactions/add: what is the use case?<br>
* transactions/remove: i&#39;d just standarize on having all transactions b=
e removable (perhaps except those marked &#39;required&#39;).<br>
* prevblock: one getmemorypool per new block shouldn&#39;t be a problem imh=
o, so do a longpoll instead of having the client able to modify prevblock t=
hemselves.</p>
<p>One more thing that I do not like is often several ways for specifying t=
he same behaviour. For example, txrequires specifies that the first N trans=
actions are mandatory, a &#39;required&#39; field in the transaction list i=
tself specifies that that transaction is mandatory, and the lack of transac=
tions as variation means that they must not be touched at all. Pick one way=
 that is flexible enough, and discard the others.</p>

<p>In summary, I&#39;d like to see the standard simplified. I have no probl=
em merging code that makes getmemorypool compliant to a standard that is ag=
reed upon, but like to understand it first.</p>
<p>In my opinion - but I&#39;m certainly open to discussion here - the stan=
dard could be simplified to:<br>
* getblocktemplate: create a new block template, and return it. The result =
contains:<br>
=A0* bits, previousblockhash, version: as to be used in block<br>
=A0* curtime, maxtimeoff, maxtimeoff: client chooses a timestamp between (c=
urtime - local_time_at_receipt + local_time), decreased by mintimeoff and i=
ncreased maxtimeoff<br>
=A0* expires, sigoplimit, sizelimit: unchanged<br>
=A0* subsidy: amount generated (5000000000 for now)<br>
=A0* coinbaseaux: what generated coinbase&#39;s scriptSig must start with<b=
r>
=A0* coinbaseoutputs: list of objects, each specifying a required coinbase =
output. Each has fields:<br>
=A0 =A0* amount: sent amount<br>
=A0 =A0* scriptPubKey: hex serialized of the output script<br>
=A0* transactions: list of object, each specifying a suggested transaction =
(except for the coinbase) in the generated block. Each has fields:<br>
=A0 =A0* txid: transaction id<br>
=A0 =A0* depends: list of dependencies (txids of earlier objects in this sa=
me transactions list).<br>
=A0 =A0* fee: fee generated by this transaction, which increases the max ou=
tput of the coinbase.<br>
=A0 =A0* required: if present, transaction may not be dropped.<br>
* submitblocktemplate: submit an object containing a hex serialized block h=
eader, hex serialized coinbase transaction, and a list of txids. Returns tr=
ue or string describing the problem. Proof of work is checked last, so that=
 error is only returned if there is no other problem with the suggested blo=
ck (this allows it to replace both propose and submit).</p>

<p>Are there important use cases I&#39;m missing?</p>
<p>--<br>
Pieter<br>
</p>

--0016e6db2e7061274e04c21ee9be--