Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1SjIax-00058G-4D for bitcoin-development@lists.sourceforge.net; Mon, 25 Jun 2012 23:21:23 +0000 Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.210.47 as permitted sender) client-ip=209.85.210.47; envelope-from=lidstrom83@gmail.com; helo=mail-pz0-f47.google.com; Received: from mail-pz0-f47.google.com ([209.85.210.47]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-MD5:128) (Exim 4.76) id 1SjIav-0000Ww-P5 for bitcoin-development@lists.sourceforge.net; Mon, 25 Jun 2012 23:21:23 +0000 Received: by dalh21 with SMTP id h21so5527392dal.34 for ; Mon, 25 Jun 2012 16:21:16 -0700 (PDT) Received: by 10.68.138.166 with SMTP id qr6mr45638674pbb.43.1340666475806; Mon, 25 Jun 2012 16:21:15 -0700 (PDT) Received: from [192.168.0.112] (70-36-48-157.dyn.novuscom.net. [70.36.48.157]) by mx.google.com with ESMTPS id hz10sm9867981pbc.32.2012.06.25.16.21.14 (version=SSLv3 cipher=OTHER); Mon, 25 Jun 2012 16:21:15 -0700 (PDT) Message-ID: <4FE8F26A.6050103@gmail.com> Date: Mon, 25 Jun 2012 16:21:14 -0700 From: Daniel Lidstrom User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: bitcoin-development@lists.sourceforge.net Content-Type: multipart/alternative; boundary="------------040205050107020903070900" X-Spam-Score: -0.3 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (lidstrom83[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends in digit (lidstrom83[at]gmail.com) 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1SjIav-0000Ww-P5 Subject: Re: [Bitcoin-development] Enforcing inflation rules for SPV clients X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jun 2012 23:21:23 -0000 This is a multi-part message in MIME format. --------------040205050107020903070900 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Here's the conversation I had with Mike that Gregory requested a link to: Thanks! Bad or hacked client devs is indeed a huge, worrying problem. The official client is addressing this with a system called gitian, where multiple developers all compile the same source to the same binary and then sign the results. Multi-signatures raise the bar for releasing hacked clients a lot. We're starting to investigate this with bitcoinj too, but it's a lot of work. Generally, the more people you have to involve in a conspiracy, the less likely it is to succeed. If a few miners started to dominate the system they have strong financial incentives to cheat, alternatively, they may be subjected to government pressure. Having to get the client developers involved too makes it much harder, especially as users have to actually upgrade. I started a thread on the development mailing list with your suggestion, by the way. On Mon, Jun 25, 2012 at 1:00 AM, Daniel Lidstrom > wrote: Hey Mike, I put our conversation in the email for easy reference. In the unlikely event of a miner conspiracy to print money, is it really so much of a further stretch to think the developers of a widely used client could also be involved? (Well, maybe, since miners are unaccountable and developers are not. OTOH if most users are apathetic...) Also, isn't the advantage for lightweight clients of SPV over the server-client model that you don't have to trust any operator? Maybe I'm being too much of a purist here... Regarding errors being cheap to send and expensive to verify, compartmentalizing them the way I suggested before would make them individually cheaper to verify. Just throwing around ideas: requiring the error message be received by a quorum of peers before checking, and dropping misbehaving or unreliable peers could help. Also, not verifying error messages unless the peers relaying them are willing to send all the data necessary to do so would help. Hashcash could also be used to balance the costs to send and to verify a given type of error message. I like your idea to only check errors in blocks that are split points, and the length of the split could also be a consideration. > Can we move further conversations to email please? SMF kind of > sucks as an inbox. > > Anyway, yes, your proposal makes a lot of sense, although I think > in practice this is unlikely to be an issue. If a majority of > miners did start mining on a chain with new rules, even if SPV > clients couldn't detect the switch automatically it's very likely > the developers of those clients would notify the users out of band > in some way. For example, by pushing an update to users that > explains the new rules to them and tells them how they can cash > out of the Bitcoin economy if they disagree with the new consensus. > > If users are on the losing side of a rule change and want to stay > there (eg, maybe most non-miners want to stay on the slower > chain), then the client can just checkpoint the first block after > the rule change occurred. Now even though there's a harder chain > with the new rules, the client will stay with the old rules > despite being blind to them. There's nothing that says checkpoints > have to be hard coded - clients could poll the client developers > every day to get new ones. So as long as the SPV devs are on the > ball, most users would stay on the old rules even if the software > can't do it by itself. > > All that said, broadcasting messages proving a block broke the > rules is a nice backstop if it can be done without excessive > complexity. There are some details to think about. These messages > would be cheap to create and expensive to verify. There has to be > something that stops me claiming to SPV clients that every single > block is invalid and forcing them to do tons of useless work. > Perhaps only blocks that are split points would be eligible. Also, > currently, SPV clients do not form their own P2P network. They are > always leaves of full nodes. So propagation of the messages might > prove difficult unless that was changed. > Hi Mike, > > Thanks for your reply. It was actually an old post of yours on > the forum that made me understand the importance of lightweight > clients being able to audit the coinbase tx. > > Re: input tx download, I think that splitting the "invalid > coinbase" error notification into separate "input n in tx m is > invalid" and "invalid fee arithmetic" errors would mean mobile > clients would only ever have to download at most one input tx to > verify an invalid coinbase. > > The "invalid fee arithmetic" error could also be compartmentalized > into "invalid fee arithmetic in tx batch n", where the fee > subtotals are recorded in the block, so as to be separately > verifiable. Then the necessary tx downloads would be completely > capped. > > Anyway, I think most of my fears about lifting the block size > limit are put to rest by these lines of thinking Smiley > Hey Daniel, > > I think you're thinking along the right lines here. We should be > looking at cheap and backwards compatible ways to upgrade the SPV > trust model to more than just going along with the majority consensus. > > For the coinbase issue, it's actually really important because if > you can't calculate fees, you can't check the size of the coinbase > value and therefore a conspiracy of miners could try and change > the inflation schedule. The worst that can happen today if miners > decide to try and fork the ruleset against the best interests of > other users, is that they produce chains that are not accepted by > regular merchant/exchange/end-user nodes and tx confirmation slows > down. But if most users are on SPV nodes they will happily accept > the new blocks and the miner conspiracy will have successfully > seized control of the money supply for the majority of end users. > > The problem with calculating fees is you actually need not only > every tx in a block, but all the input txns too! You can't know > the fee of a transaction without the input transactions being > available too. > > So there's good news and bad news. The good news is you don't have > to actually store every transaction on disk or check signatures, > which is the expensive part, if you're an SPV node - you can still > check the coinbase value asynchronously after receiving a block by > requesting the entire contents and all the input transactions > (+branches, of course). If you are in an environment that is > always-on and not bandwidth constrained this can make a lot of > sense. For example, if you're running an SPV client on a regular > PC that is sitting in a shop somewhere, just downloading a lot of > data is still very cheap compared to indexing it all and verifying > all the signatures. I'm implementing support for this kind of > async verification in bitcoinj at the moment. > > Where it doesn't really make sense is mobile clients, and your > ideas of notification can help a lot there. The first type of > error broadcast I'd add is actually for detected double spends > because there's a paper from researchers at ETH which show this is > needed to address a feasible attack on todays infrastructure. But > adding more later to detect invalid blocks is not a bad idea. > Hi Mike, > > I wanted to post this on the forum and figured that since, based > on the subject, you'd probably end up reading it anyway, I thought > I'd run it by you first to see if there's any merit to it. > > Quote > I think similar ideas are currently being tossed around. > > The motivation here is to keep rule enforcement maximally > decentralized while scaling Bitcoin. Basically, SPV clients could > subscribe to peers' announcements of /invalid/ blocks + their > specific failure mode. AFAICT, most of the possible failure modes > are easily verifiable by smart phone SPV clients, even at scale. > For example, inclusion of double spends or otherwise invalid txs > are easy to verify /given the necessary data/, and Merkle tree > nodes can be deemed invalid /if no peer can produce the data that > hashes to it/ (gotta be careful to detect and deal with false > positives here). The only failure mode I can think of that isn't > easily verifiable at scale is an invalid quantity of fees spent in > the coinbase tx, since that currently requires the download of the > whole block. (I think I have an idea to fix this below.) > > This trust model relies upon having a single fully validating peer > that will reliably announce invalid blocks. To help this, one or > more peers could be periodically cycled, and new ones asked if the > main chain contains any invalid blocks. Worst case scenario, an > SPV client reverts to the current trust model, so this idea can > only improve the situation. Best case, it takes advantage of the > fact that public information is hard to suppress. > > *Verifying coinbase tx invalidity at scale with SPV clients* > > While this is currently not prohibitive for a smart phone if done > only occasionally, this will not be the case if the block size > limit is lifted. The idea is to split up the summing of the > spendable fees into verifiable pieces. A new merkle tree is > calculated where the i'th leaf's data is the total fees in the > i'th chunk of 1024 txs, along with these txs' Merkle root (to > prevent collisions). The Merkle root of this new tree is then > included somewhere in the block. > > To verify a claim of invalidity of one of these leaves requires > the download of ~1MB of tx data (along with a small amount of > Merkle tree data to verify inclusion). If no claims are made > against it, the leaf data is assumed to be valid. If this is the > case, but shenanigans are called on the coinbase tx, downloading > all of the leaf data, verifying inclusion, and calculating the > total spendable fees is easily doable for a smart phone, even at > very large tx volumes. > > Thanks for any thoughts/suggestions! > > Cheers, > Daniel --------------040205050107020903070900 Content-Type: multipart/related; boundary="------------010803050202070505030404" --------------010803050202070505030404 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Here's the conversation I had with Mike that Gregory requested a link to:


Thanks!

Bad or hacked client devs is indeed a huge, worrying problem. The official client is addressing this with a system called gitian, where multiple developers all compile the same source to the same binary and then sign the results. Multi-signatures raise the bar for releasing hacked clients a lot. We're starting to investigate this with bitcoinj too, but it's a lot of work.

Generally, the more people you have to involve in a conspiracy, the less likely it is to succeed. If a few miners started to dominate the system they have strong financial incentives to cheat, alternatively, they may be subjected to government pressure. Having to get the client developers involved too makes it much harder, especially as users have to actually upgrade.

I started a thread on the development mailing list with your suggestion, by the way.

On Mon, Jun 25, 2012 at 1:00 AM, Daniel Lidstrom <lidstrom83@gmail.com> wrote:
Hey Mike,

I put our conversation in the email for easy reference.

In the unlikely event of a miner conspiracy to print money, is it really so much of a further stretch to think the developers of a widely used client could also be involved?  (Well, maybe, since miners are unaccountable and developers are not.  OTOH if most users are apathetic...)  Also, isn't the advantage for lightweight clients of SPV over the server-client model that you don't have to trust any operator?  Maybe I'm being too much of a purist here...

Regarding errors being cheap to send and expensive to verify, compartmentalizing them the way I suggested before would make them individually cheaper to verify.  Just throwing around ideas: requiring the error message be received by a quorum of peers before checking, and dropping misbehaving or unreliable peers could help.  Also, not verifying error messages unless the peers relaying them are willing to send all the data necessary to do so would help.  Hashcash could also be used to balance the costs to send and to verify a given type of error message.  I like your idea to only check errors in blocks that are split points, and the length of the split could also be a consideration.

Can we move further conversations to email please? SMF kind of sucks as an inbox.

Anyway, yes, your proposal makes a lot of sense, although I think in practice this is unlikely to be an issue. If a majority of miners did start mining on a chain with new rules, even if SPV clients couldn't detect the switch automatically it's very likely the developers of those clients would notify the users out of band in some way. For example, by pushing an update to users that explains the new rules to them and tells them how they can cash out of the Bitcoin economy if they disagree with the new consensus.

If users are on the losing side of a rule change and want to stay there (eg, maybe most non-miners want to stay on the slower chain), then the client can just checkpoint the first block after the rule change occurred. Now even though there's a harder chain with the new rules, the client will stay with the old rules despite being blind to them. There's nothing that says checkpoints have to be hard coded - clients could poll the client developers every day to get new ones. So as long as the SPV devs are on the ball, most users would stay on the old rules even if the software can't do it by itself.

All that said, broadcasting messages proving a block broke the rules is a nice backstop if it can be done without excessive complexity. There are some details to think about. These messages would be cheap to create and expensive to verify. There has to be something that stops me claiming to SPV clients that every single block is invalid and forcing them to do tons of useless work. Perhaps only blocks that are split points would be eligible. Also, currently, SPV clients do not form their own P2P network. They are always leaves of full nodes. So propagation of the messages might prove difficult unless that was changed.

Hi Mike,

Thanks for your reply.  It was actually an old post of yours on the forum that made me understand the importance of lightweight clients being able to audit the coinbase tx.

Re: input tx download, I think that splitting the "invalid coinbase" error notification into separate "input n in tx m is invalid" and "invalid fee arithmetic" errors would mean mobile clients would only ever have to download at most one input tx to verify an invalid coinbase.

The "invalid fee arithmetic" error could also be compartmentalized into "invalid fee arithmetic in tx batch n", where the fee subtotals are recorded in the block, so as to be separately verifiable.  Then the necessary tx downloads would be completely capped.

Anyway, I think most of my fears about lifting the block size limit are put to rest by these lines of thinking Smiley

Hey Daniel,

I think you're thinking along the right lines here. We should be looking at cheap and backwards compatible ways to upgrade the SPV trust model to more than just going along with the majority consensus.

For the coinbase issue, it's actually really important because if you can't calculate fees, you can't check the size of the coinbase value and therefore a conspiracy of miners could try and change the inflation schedule. The worst that can happen today if miners decide to try and fork the ruleset against the best interests of other users, is that they produce chains that are not accepted by regular merchant/exchange/end-user nodes and tx confirmation slows down. But if most users are on SPV nodes they will happily accept the new blocks and the miner conspiracy will have successfully seized control of the money supply for the majority of end users.

The problem with calculating fees is you actually need not only every tx in a block, but all the input txns too! You can't know the fee of a transaction without the input transactions being available too.

So there's good news and bad news. The good news is you don't have to actually store every transaction on disk or check signatures, which is the expensive part, if you're an SPV node - you can still check the coinbase value asynchronously after receiving a block by requesting the entire contents and all the input transactions (+branches, of course). If you are in an environment that is always-on and not bandwidth constrained this can make a lot of sense. For example, if you're running an SPV client on a regular PC that is sitting in a shop somewhere, just downloading a lot of data is still very cheap compared to indexing it all and verifying all the signatures. I'm implementing support for this kind of async verification in bitcoinj at the moment.

Where it doesn't really make sense is mobile clients, and your ideas of notification can help a lot there. The first type of error broadcast I'd add is actually for detected double spends because there's a paper from researchers at ETH which show this is needed to address a feasible attack on todays infrastructure. But adding more later to detect invalid blocks is not a bad idea.

Hi Mike,

I wanted to post this on the forum and figured that since, based on the subject, you'd probably end up reading it anyway, I thought I'd run it by you first to see if there's any merit to it.

Quote
I think similar ideas are currently being tossed around.

The motivation here is to keep rule enforcement maximally decentralized while scaling Bitcoin.  Basically, SPV clients could subscribe to peers' announcements of invalid blocks + their specific failure mode.  AFAICT, most of the possible failure modes are easily verifiable by smart phone SPV clients, even at scale.  For example, inclusion of double spends or otherwise invalid txs are easy to verify given the necessary data, and Merkle tree nodes can be deemed invalid if no peer can produce the data that hashes to it (gotta be careful to detect and deal with false positives here).  The only failure mode I can think of that isn't easily verifiable at scale is an invalid quantity of fees spent in the coinbase tx, since that currently requires the download of the whole block.  (I think I have an idea to fix this below.)

This trust model relies upon having a single fully validating peer that will reliably announce invalid blocks.  To help this, one or more peers could be periodically cycled, and new ones asked if the main chain contains any invalid blocks.  Worst case scenario, an SPV client reverts to the current trust model, so this idea can only improve the situation.  Best case, it takes advantage of the fact that public information is hard to suppress.

Verifying coinbase tx invalidity at scale with SPV clients

While this is currently not prohibitive for a smart phone if done only occasionally, this will not be the case if the block size limit is lifted.  The idea is to split up the summing of the spendable fees into verifiable pieces.  A new merkle tree is calculated where the i'th leaf's data is the total fees in the i'th chunk of 1024 txs, along with these txs' Merkle root (to prevent collisions).  The Merkle root of this new tree is then included somewhere in the block.

To verify a claim of invalidity of one of these leaves requires the download of ~1MB of tx data (along with a small amount of Merkle tree data to verify inclusion).  If no claims are made against it, the leaf data is assumed to be valid.  If this is the case, but shenanigans are called on the coinbase tx, downloading all of the leaf data, verifying inclusion, and calculating the total spendable fees is easily doable for a smart phone, even at very large tx volumes.

Thanks for any thoughts/suggestions!

Cheers,
Daniel




--------------010803050202070505030404 Content-Type: image/gif Content-Transfer-Encoding: base64 Content-ID: R0lGODlhDwAPANUAAAAAAOG3AHlhAFFIAIaGhjIoAL+bAGZmZu7hABsWAFJCAOzRAP/wABEQ AGhUAL+mAOy9AP/dAH9xAP/WAMezAFZRAHNcAP/mAP/4AP/MAGJaAIVrAMWeAOLIAM+mACYj APXEAIBzADMzAOi6AFlHAHBmAP//ACMcAH5qAG1XAH98ANC0APjOAMyZAJmZmQAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAUUAC4ALAAA AAAPAA8AAAabQJfQdVA0AI3BYTgkFBKo1YJS+ogIzYaDFYlcGAyEpoF1FSyZifeLaVdExBMo 7QVjTBhEoyjIZFAAEhcqACoYDBoDDR5+AI4XjgBgFEcQfoAoFyEAIRcXHUiMfhNqaxcRDw0K aBkGBqRdsRUKB3KtqxsoCaQQe2YOcx4bwwF+JAVCBKp+zBkjJGRNBQ0pHBAtKQkFZUxFR6pL Q0EAOw== --------------010803050202070505030404-- --------------040205050107020903070900--