Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of petertodd.org
	designates 62.13.148.96 as permitted sender)
	client-ip=62.13.148.96; envelope-from=pete@petertodd.org;
	helo=outmail148096.authsmtp.net; 
Date: Fri, 14 Jun 2013 16:06:54 -0400
From: Peter Todd <pete@petertodd.org>
To: Luke-Jr <luke@dashjr.org>
Message-ID: <20130614200654.GB11509@petertodd.org>
References: <20130527111149.GB8955@tilt> <20130531165758.GA29135@petertodd.org>
	<20130610210913.GA17242@petertodd.org>
	<201306102123.15732.luke@dashjr.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="QTprm0S8XgL7H0Dt"
Content-Disposition: inline
In-Reply-To: <201306102123.15732.luke@dashjr.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: bitcoin-development@lists.sourceforge.net
Subject: Re: [Bitcoin-development] Decentralizing mining
Precedence: list


--QTprm0S8XgL7H0Dt
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 10, 2013 at 09:23:14PM +0000, Luke-Jr wrote:
> > They always (usually?) work on the subset of transactions common to both
> > the pool's getblocktemplate and their local one. When they find a share
> > if it doesn't meet difficulty they just hand it to the pool. Currently
> > that is done by handing the whole block over, correct? I know the BIP
> > says otherwise, but we should optimize this to just hand over tx hashes
> > where possible.
>=20
> The plan was to tell the pool it doesn't need to send transactions at all=
, and=20
> only work on the ones from bitcoind. Currently, share submissions are jus=
t the=20
> block header and coinbase transaction; in this case, however, the miner w=
ill=20
> need to send merkle links also, possibly just once via a block proposal i=
n=20
> advance.

It strikes me that this would work best if the pool has a mempool with
child-pays-for-parent support where conflicts *are* allowed.

IE you record whatever transactions you know about, conflicting or not,
calculate which ones gives you the most fees/kb, and then figure out
which set of non-conflicting ones are optimal. Of course, "optimal" is
the knapsack problem...

Now you can easilly tell the miners working on shares for you which tx's
would be optimal if they wish to know, and at the same time whatever
shares they send you are most likely to include transactions in your
mempool inventory, and thus they can send just tx hashes to reduce
bandwidth.


Part of the broader issue that when we send peers INV advertisements we
should be telling them what the fee/kb is so our peers can prioritize
properly. That'd also help for the case where you want to broadcast two
transactions in a row, but the pair is only profitable because the
second is paying the fee for the first.

Speaking of, the way we tell peers about new blocks is really
suboptimal: we tell every peer, in no particular order, about a new
block via a block INV message, and then we give them the new block in
parallel. I was looking through comp-sci papers on optimal
flood-fill/gossip algorithms for random graph networks and it appears
that optimal is to spend all your bandwidth to send the message to your
fastest peer first, followed by your next fastest and so on. This works
best because you get the exponential growth scaling faster by
propagating the message as "deep" as possible in the network, and it
then can flood outwards from there. Just sorting the peer list by
#inv-recevied/time when doing INV pushes and when attending to incoming
messages would probably be a big improvement.

> > If the share does meet difficulty, hand it to both the pool and the
> > local bitcoind. Should hand it to the pool first though, because the
> > pool likely has the fastest block propagation, then hand it to local
> > bitcoind. An optimized version may want to have some record of measured
> > bandwidth - this applies Bitcoin in general too, although also has other
> > issues.
>=20
> Currently, BFGMiner is doing submission to the pool, waiting for a respon=
se,=20
> then submitting to a local bitcoind. This is because the pool might need =
to=20
> receive/record the share before it processes the block on bitcoind, or yo=
u=20
> could lose credit for it. The response from the pool is rather small (a s=
ingle=20
> TCP packet), so this shouldn't delay much longer.

Right, I guess the pool wants to be sure you were actually the one who
found the share, rather than just someone who was lucky enough to see it
on the network and submitted it as your own.

> > ## Reducing bandwidth
> >=20
> > How about for normal shares we just pass the block header, and have the
> > pool randomly pick a subset of transactions to audit? Any fraud cancels
> > the users shares. This will work best in conjunction with a UTXO proof
> > tree to prove fees, or by just picking whole shares randomly to audit.
>=20
> Might as well just use higher difficulty shares (each one audited) for th=
e=20
> same effect. Block proposals allow the miner to tell the pool its transac=
tion=20
> set once (per txset change) for any number of shares.

That's a good point - the current practice most pools seem to follow of
about a share per second seems very excessive to me. On the other hand,
it does have good user optics. The right solution might be something
akin to P2Pool where the UI level is telling the user shares are being
found so it's clear "stuff is happening", but under the hood only a
small subset are ever sent to the pool.

> > # Pool work
> >=20
> > So does eliopool already accept arbitrary shares like this and do the
> > correct accounting already? (IE adjust share amount based on fees?) What
> > happens when the pool doesn't get the share directly, but does see the
> > new block?
> >=20
> > + possible protocol extensions
>=20
> I don't follow.

What part don't you follow?

> > # Miner work
> >=20
> > Basically we need code to merge the two block templates together to find
> > commonality. I guess you probably want to implement this in bfgminer
> > first, so add the code to libblkmaker first, then maybe python-blkmaker.
>=20
> libblkmaker's API was designed for this from the start, so it should be f=
airly=20
> easily implemented.

Good.

> > We also want an automatic fallback to local solo mining if the pool
> > can't be contacted.
> >=20
> > + possible protocol extensions
>=20
> Failover already functions, but probably could use a rewrite...

Sounds good.

--=20
'peter'[:-1]@petertodd.org
000000000000006ed83d97e7266a66dcb057cd53750795bd209076fa0a34d4ec

--QTprm0S8XgL7H0Dt
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlG7d94ACgkQpEFN739thozcBwCffMU//scgRaKgy69KipZgmMnU
+lEAoI4YI6dnZ3LanuuIP0e2Q+R4yRLP
=T0RA
-----END PGP SIGNATURE-----

--QTprm0S8XgL7H0Dt--