Lessons learned with segwit in Bitcoin
Greg Sanders (instagibbs)
https://twitter.com/kanzure/status/785034301272420353
https://www.youtube.com/watch?v=_Z0ID-0DOnc
Hi. My name is Greg Sanders. I will be giving a talk about segwit in bitcoin lessons learned. I will also give some takeaways that I think are important. A little bit about myself. I work on Elements Project at Blockstream. I was a reviewer on segwit for Bitcoin Core. How do we scale protocol development? How do we scale review and keep things safe?
Segwit started as an element in elements alpha. It allows for safe chaining of pre-signed transactions, which is important for payment channels and lightning network. The version in elements just changed how we defined txids, which we can't do in bitcoin. It's hard to imagine doing hard-forks in bitcoin just for this. Non-security hard-forks are going to be very hard to get consensus on.
As deployed, there's this key insight that you deploy a type of extension block and you can enforce inside of this extension some new rules. Some things that look like hard-forks turn into soft-forks. So we can sneak in confidential transactions as a soft-fork. So you have the base block and then you commit to something, and in this other block you reveal the real data that shows you own this bitcoin. As a highly desired extension, it kind of fits this model.
There is a new special ttransaction type where to spend the bitcoin there is an empty scriptsig or empty proof. So the scriptpubkey is a commitment to the witness script. Upgraded nodes will see this new script and evaluate that and whether it is true or false. This allows the UTXO calculation to look the same for old and new nodes. However, changing the signature does not change the txid. This is backwards compatible, and this is old news.
So if segwit is already done, then why am I talking about? Well, it's activated on testnet and it's close to deployment on mainnet. The activation parameters on mainnet has not been set. It's one of the largest changes to bitcoin ever. It has touched nearly all areas of the bitcoin code base, like p2p, wallet, consensus, every layer. It's informational to talk about this for those who were not in the weeds doing the testing and development. And it's informational also to give takeaways. All of these opinions are just my own.
A lot of things didn't make it. It's important to think why it didn't make it. So here's a list of things that didn't make it in segwit, such as a new address format in bip142, it's the same type of address, but it's slightly tweaked to signal that you're segwit. Hmm I'm missing something there. The addresses might be handy; but for now we're doing nested P2SH so that it looks like a regular P2SH address (bip16). So if they can send to P2SH then they can send to P2SH segwit. New addresses might come along later for segwit.
There's also additional tweaks to the commitment structure. In the coinbase data structure, in one of the outputs, you commit to all of this witness data. You have to commit to it in the block. The commitment itself, there was discussion about putting in other commitments like previous block witness commitment to make sure that miners know what the witness data is. Fraud proofs, validation cost metric. There was an idea for a holistic validation cost metric-- that size does matter, but not just size, but also how much sighashing you're doing, and putting this into one holistic cost metric. A version of it made it into segwit where we're balancing UTXO space with witness data. I'll talk about that more later. Fraud proofs hasn't been developed yet, so it was sidelined. And also, new scripting for the most part.
I think that for every-- for Minimum Viable Products-- for things that didn't make it, some proposals were simply written and coded up and then not championed. It didn't get enough discussion. There weren't enough concept ACKs and they languished. Each proposal can increase the demand for review-- it adds complexity not removing. The biggest bottleneck in Bitcoin development today is code review. There's also a too many BIP numbres problem. I know one developer that didn't know about the new sighashing because he thought there were N - 1 bips active, but there was another bip. Some people were starting to implemet bip142 but then it got sidelined, so some people got confused and spent time on things that might not be used in the future, so that was confusing. For those of you who are into bitcoin scripting;.... things that did change, CHECKMULTISIG, we did not get rid of the weird dummy argument required on the stack. There is a merged soft-fork that once segwit activates, the dummy argument must be an empty string (pull request #8636) so that it cannot be malleated. Also bip143 is sighash overhaul which takes O(n2) hashing in the worst case to basically linear, which is useful for attack defense scenarios. Also value under hashed, so that wallets don't sign all their fee away.... "Minimal if" policy-- it should not be some big blob that evaluates to true (#8526). There's no uncompressed pubkeys as policy (#8499) still being discussed. Then there's nullfail as policy (#8634), it doesn't cost any time and space for it to be an empty string. Fixes SIGHASH_SINGLE "one" bug. Originally if there is no output, but there is an input, then SIGHASH returns 1, but you don't want to sign that. There's also a "low s" soft-fork so that if the signature is high it's invalid, but there's a corner case with further testing-- this was proposed much later-- additional testing showed this. I included a link directly to the github issue about that. Also, script versioning-- we now have a script versioning number so you can increment that and arbitrarily change the script.
Segwit is developed in initial development. It was developed on the mailing list and a lot of work on IRC. There was some contention about where to discuss it. Some people wanted the discussion in #bitcoin-core-dev but there was hesitancy because of a lot of back-and-forth about initial design work. So they moved into #segwit-dev, which has since died down now that segwit has been completed. So there was a concern here about a loss of communal memory. Having it in one place makes it much easier to understand what's going on. Bulk of the segwit design was done in pull requests of course. There were four segnet testnet iterations. There were 4 total over the lifespan. The first one started in December 2015. These testnets were actively used by downstream developers. They were spitting out 4 megabyte spam blocks which are not usable in real life, but testing the limits and seeing how it works.
As a pull request, the first pull request had 32 participants. It had a lot of conversation. It had a lot of changes. Most of these were structured as the primary pull request, and then fixups and changes. As a reviewer, I would come back and find half a dozen new bugfixes and these were kept as a history there. It's 1400 lines of code, and about 3300 lines of test. This is a ratio that I like to see, or sometimes even more. Simultaneously, near the end, there was another pull request for segwit (#8149)-- same number of lines of code added and subtracted but it was rebased and compacted into logical git commits. A new reviewer probably doesn't want to go through 128 commits and figure out what's going on, but instead go through 27 commits with more specific structure. Simultaneously at this time we were getting near the 0.13 release, and we wanted to get segwit merged into master and also compact blocks. There was some question about which changes to merge first, whether segwit or compact blocks. This caused some contention because of different priorities among developers. It was merged and has been active on testnet. Mainnet parameters are not set yet. You can play with segwit in Bitcoin Core today.
So it's merged into master, are we done yet? Well, not really. For mainnet, we want to be really sure that this is quality. For 0.13, there are a number of issues related to segwit already closed. There are security bugfixes, networking, additional policy, and some people say that 4 megabytes is too much relay abuse... so here's some of the backlog, for backporting. Since p2p relay is generally done by txid, that if someone messes with it and makes it false that even just the relayer makes it false-- so you won't see a bunch of transactions at least until another block or two when it gets reset, so this is sort of annoying p2p stuff, since oyu have this new wtxid logic. Same thing with compact blocks-- based on txid, and now we need something that works with segwit. I think this is one of the last blockers for segwit. We need compact blocks for this increased throughput because we expect something like 2 megabyte blocks. Wallet cleanups and soft-fork policy columns like nulldummy and all those things. Those were kind of added last minute. I have added a link in the slides to the milestone on github so you can see the history and how it has evolved. People have been trying to get on the same page about what's important; someone will bring it up but it will keep coming up over and over again, so it makes it hard to do action on the small things that we all agree are critical, like compact blocks with segwit.
I will make a few proposals out of this. Any non-trivial consensus changes in bitcoin should in the future should have an actively-used separate testnet spun up. If supposed downstream users aren't actively testing, is the change even desired in the first place? Ride-along changes meaning like the extra policy and soft-fork changes should be discussed and tested much earlier in the process because critical distracting issues will pop-up, like finding out that there's a corner case and is it really a risk and you spend a lot of time discussing this when you should be focusing on other issues. Testing should take up a large fraction of the LoC changes. Any additional technical channels should be carefully spun up, logged, and spun down at appropriate times, because design history and communal knowledge can get lost otherwise.
If a PR spans a number of layers, then keep the git commits in logical partitions. This makes it easier for someone who only cares about say, the wallet, to just take a look and only review specific sections and then can give an untested/tested ACK for those changes. Let's stop talking about the block size. In segwit the terminology is "weight" because there's a little confusion about what "cost" means. How big is it? How much computation does it take? How much externality does it put on the system? Also we can talk about throughput-- like aggregation with lightning and so on. We should only spend time backporting changes when there is demand. With these changes, review is strapped. If people aren't going to test it, then it's probably not safe to backport it anyway. Also, don't do large changes like segwit often. With large changes, there is a higher amount of risk for consensus problems. This slows down other work like libconsensus, network refactoring, and other long-term projects that keep getting held off. These large changes are difficult to review because you can't ust piecemeal these pieces. It also collides with the release schedule and it makes Wladimir's job much harder, and that's no good. Thank you.
Q&A
Q: Did you ever consider a bug bounty for testing program?
A: They have some value. But where would the money come from? Bitcoin Core is not a foundation. It's not a corporation. It's just a group of people that work together for likeminded goals. Blockstream is a company that pays Pieter Wuille to work on Bitcoin Core and I spent time on Blockstream time to review segwit so that might be more scalable long-term, to have people paid primarily or in part with the consensus process and iterative development process.
References
segwit next steps (2016-06-24)