16:35 -!- won9 [~won@60.45.178.42] has quit [Quit: won9]
16:37 -!- hashtag_ [~hashtag@CPE-69-23-213-3.wi.res.rr.com] has quit [Read error: Connection reset by peer]
16:39 -!- hearn [~mike@84-75-198-85.dclient.hispeed.ch] has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
16:41 < kanzure> what is this about libbitcoincompleteconsensus versus libbitcoinconsensus?
16:42 <@gmaxwell> gibbberish what?
16:43 <@gmaxwell> oh luke's comment? I don't know that I agree.
16:43 < kanzure> yes that comment
16:43 <@gmaxwell> The distinction there is stateless vs stateful consensus.
16:43 < kanzure> ah
16:43 < kanzure> i have not thought about that one much. hmm.
16:43 < kanzure> i suppose the distinction might be interesting, but in practice i don't care how stuff is persisted as long as the format is correct.
16:44 <@gmaxwell> If there is comitted state of some kind then the two basically become one and the same. (if storage fails the failure will eventually be cleanly detectable)
16:44 < kanzure> or as long as various guarantees are kept by whatever implementation you pick
16:45 < phantomcircuit> gmaxwell, how do you cleanly detect a storage failure without replaying?
16:45 -!- baldur [~baldur@ool-45705038.dyn.optonline.net] has quit [Ping timeout: 265 seconds]
16:46 <@gmaxwell> kanzure: they usually aren't, of course.  though I think there are basically two ways around it (other than also stuffing the storage into the consensus code): one is using a comitted state which basically means that no undetectable divergence can happen; the other is by using oblivious storage with per host randomization; so that any storage failure is non-systemic with high probablity.
16:46 < kanzure> if storage code is stuffed into consensus code, then what does that actually look like?
16:46 < kanzure> in practice you are not going to have ssd firmware in consensus code, right? 
16:46 -!- hashtag_ [~hashtag@CPE-69-23-213-3.wi.res.rr.com] has joined #bitcoin-dev
16:47 < kanzure> oh you mean network-level storage. hrm.
16:47 <@gmaxwell> phantomcircuit: if your state is comitted then any corruption eventually results in a failed hash in a case where such a failure is impossible;   (oh and by commited I don't necessarily mean in the blockchain, there I mean 'comitted to in a small amount trusted storage which is assumed correct')
16:47 <@gmaxwell> no I don't mean network level storage.
16:47  * kanzure consults his philosophy book about storage
16:48 -!- felipelalli [~Thunderbi@unaffiliated/felipelalli] has joined #bitcoin-dev
16:48 <@gmaxwell> If you have your state (the utxo set in the case of bitcoin) in a hash tree. And the consensus magic box keeps track of the root (perhaps its in the blockchain, perhaps not)... then there is no way the storage can 'fail' which is results in a false acceptance or rejection of a block.
16:48 -!- jtimon [~quassel@172.56.38.110] has quit [Ping timeout: 250 seconds]
16:48 <@gmaxwell> Because to do so the storage would have to present tree fragments which are consistent with the commited state.
16:49 < phantomcircuit> gmaxwell, that's cheating but ok
16:49 <@gmaxwell> It's not cheating, it's completely pratical, it allows you a strong guarentee that no matter how screwed up your key value store is you'll never falsely accept or reject a block, at worst you'll detect that you're screwed and shut off.
16:50 <@gmaxwell> The only down side is that having authenticated state has overhead.
16:50 < phantomcircuit> no i mean the assuming you have some reliable storage that isn't the kv storage
16:51 <@gmaxwell> phantomcircuit: well if it's in the blockchain then you don't have to have it. And if it's not; well it's only storing 32 bytes, and any corruption there will _also_ be detected, unless it's some kind of malicious adjustment to be consistent with the broken storage; in which case all bets are off.  The attack model here is faulty hardware/software.
16:51 < phantomcircuit> i suspect building a kv store that provides much stronger guarantees than leveldb does for example would give you virtually identical results
16:53 <@gmaxwell> I'm quite confident that it's not possible to build such a thing without basically doing exactly the same. It is _insanely_ hard to write correct software. Leveldb is many many thousands of lines of code, and I'm quite sure that it's already not durable even absent flaky hardware. Mix in flaky hardware, and I doubt anything short of all the data being hashed is sufficient to have zero undetectable fa
16:53 <@gmaxwell> ilures.
16:54 <@gmaxwell> Keep in mind this is a discussion which is not about "don't be busted shit", it's about "be structurally impossible for a block to be falsely accepted or rejected due to storage corruption"
16:54 < kanzure> yeah i had the wrong scope in mind
16:55 <@gmaxwell> kanzure: the other path I mentioned, ... w/ oblivious storage would be to use the storage in such a way that you could have strong confidence that even if the storage has a common design flaw no two systems will fault the same. Basically by having the consensus black box hide its data and access patterns from the storage. ... but the overhead for this is somewhat higher than a comitted utxo set, sooo
16:56 <@gmaxwell> .
16:56 < phantomcircuit> gmaxwell, i guess this is relatively easy to get right but the issue im seeing with something like that is a valid but older than we think it is hash
16:56 -!- felipelalli [~Thunderbi@unaffiliated/felipelalli] has quit [Ping timeout: 240 seconds]
16:57 <@gmaxwell> Hm?,  storage_root = H(disk_data_root || block_hash)
16:58 <@gmaxwell> then when a new block comes in, a non-consensus disk manager looks up all the required utxo for the block and presents the proofs relative to the storage_root. Consensus code verifies them, and continues on.
16:58 < phantomcircuit> gmaxwell, and then you have an ssd that miraculously reverts the disk data root and block hash AFTER you've run the check
16:58 < phantomcircuit> it's unlikely but hardly impossible
16:59 < phantomcircuit> either way this would be pretty clearly better than what exists today
16:59 < phantomcircuit> :P
16:59 <@gmaxwell> if the block hash is reverted then youre just stuck on a particular block.
16:59 < phantomcircuit> i will stop being ridiculous now
16:59 <@gmaxwell> (or not even stuck)