Stephan Livera Podcast with Jon Atack - November 13th 2019

Bitcoin Core contribution

Podcast: https://stephanlivera.com/episode/124/

https://twitter.com/kanzure/status/1195848417186078720

Stephan Livera: Jon, welcome to the show.

Jon Atack: Hi, Stephan. Thanks for having me on. I’m pleased to be here.

Stephan Livera: So, Jon, I had the pleasure of meeting you at The Lightning Conference recently and I know you are working on Bitcoin Core in a part time capacity to do some review and development work. I was keen to discuss with you and help break that down a little bit for the listeners but let’s start with some background on yourself. Where did you come from before all this Bitcoin stuff?

Jon Atack: Right. So I was born in United States, which one can hear I think with my accent but I’ve spent most of my life traveling and living around Europe. Germany, the former Soviet Union, now France, and I currently spend most of my time between Biarritz in the southwest of France, Paris and sometimes San Francisco and New York.

Stephan Livera: Awesome. Tell us a little bit about your career before Bitcoin and software development, review etc

Jon Atack: Sure, let’s see. For the past decade I’ve been a freelance software developer doing software engineering and CTO roles for large multinational companies, often in Europe. I studied computer science and physics at Harvey Mudd college in Claremont, California and business at INSEAD in France, which is an international MBA. I speak English, French, German and some Russian. Computer languages wise, I began with Basic a very long time ago when I was nine years old. Fortunately the teacher in my math class took me over to the admin offices of our grade school and there was a Terminal printer, there was no screen back then, and I learned Basic. I taught myself assembly language for the 6502 microprocessor and from age 12 to 17, alongside school and athletic activities, I was a professional freelance games creator and programmer. During my teenage years I wrote three games that were released for Atari and Commodore computers back in the 1980s. So that gives you some idea. I first heard of Bitcoin back in 2013.

Stephan Livera: I know you were doing some work in the Ruby ecosystem. Tell us a little bit about that.

Jon Atack: Correct. After my studies I did freelancing as I mentioned for large companies and it was very often using Ruby as well as various databases. So that exposed me to open source. We were using entirely open source software, which I think is a very good thing and the theme of this podcast clearly. Around 2013 I began contributing to open source. I saw some issues with a search engine in Ruby and I began contributing to it and very quickly the lead maintainer gave me the commit bit and I became the lead maintainer rather quickly. That became the leading database search engine in the Ruby ecosystem, it’s called Ransack. Following which I got my feet wet and I started getting a bit more confidence in how the whole open source process works because it is a social process. I began contributing to the Ruby on Rails web framework. I became a top 100 contributor out of around 6 or 7,000 people now who have contributed and as well began co-editing the newsletter “This Week in Rails”. So that really was about three or four years of quite heavy open source contributing. It was a very interesting way to begin.

Stephan Livera: Yeah, that’s great. I think what we can do for this episode is really break that down in terms of what does open source development look like for those people who came to Bitcoin more from the economics or finance angle. Maybe they’re curious to understand a little bit more about how open source development works. What are the conventions, if you will, how does a new proposal come up? How does it get debated? How does that flow work? So maybe we could just start with a little bit around how do you get started working on Bitcoin Core? What are some good resources? What are some good ways to attack that problem?

Jon Atack: Sure. If you don’t mind, I’ll backtrack a bit and relate my own journey to getting started and then I’ll start providing resources for other people. Clearly as a developer what I think about day in and day out is probably of more interest to developers looking to get involved but I originally was interested in Bitcoin as well from an investor point of view and the econ, finance point of view. It’s been a whole journey. I first heard of Bitcoin in 2013. I was very skeptical and that was a foolish idea but I didn’t have bandwidth for it let’s say. I was busy with freelance missions. I started following Bitcoin more closely in 2016 and fell down the rabbit hole, the story we all know. I began saving money from my freelance missions to begin spending time, hopefully in the future learning to contribute. So I saved money for about two and a half years. In March of this year, in 2019, I decided to stop accepting missions and try to apply myself a bit towards Bitcoin as a developer. At the same time I applied to Chaincode Labs in late February for the residency this summer. I hadn’t done anything yet and so no surprise, I was told I needed to begin showing some proof of work in this space. At least I got to the first interview though, which I was pleased to have some feedback on that. So within two weeks I began applying myself to Bitcoin Core to learn how to contribute, which involved basically spending a lot of time on the Github repository where the development work happens, reading the conversations around issues and pull requests. For your viewers, I don’t know if pull requests is a familiar term?

Stephan Livera: Yeah so let’s break some of that down. So what do we use GitHub for? What are some of the common terms that they might need to learn?

Jon Atack: So Github is an online version of the Git software with a bit of a social aspect to it to facilitate interaction around code development online in an open source fashion. GitHub can be used in a closed source way but usually it’s open source which means anyone can view, anyone can follow along, anyone can participate within reason if they don’t get themselves banned for bad behavior. Like I said it’s a social process. Generally discussion takes place on GitHub where the codebase is stored, around what are called issues and pull requests. Issues are issues or problems that people have found and are reporting on, topics to discuss and pull requests are proposals to contribute some changes to the codebase or to the documentation. These are online, anyone can see them and participate.

Stephan Livera: I think to an outsider this might look really crazy, right? Because they might be coming from a more top down, authoritarian view where they think “There’s this guy in charge” and he says “Everyone just work on this.” But open source is a different beast altogether. Can you explain a little bit around why it’s different and how do different things get prioritized?

Jon Atack: That’s a really great point and often a source of misunderstanding. People come from corporate backgrounds or job backgrounds where as you said it does happen that way. Whereas open source is more of an adhoc scratch your own itch sort of process where people come in, they have issues, they try to fix them. People can work on what they want, when they want. They can fade in, they can fade out, everything is possible. Like I said, it’s ad hoc and I think that’s the way it should be. I’m fairly confident that almost any open source developer would agree with that. Nobody wants it to be structured and waterfalled and scrum managers come in and start ordering things around and following issues up on issue trackers like Trello or whatever. It’s a very free willing and anyone can participate thing. I love it, I think that’s the way it should be.

Stephan Livera: Again, just playing devil’s advocate. I’m a fan of open source development myself. Somebody who’s an outsider might look at that and think “But what if something urgent comes up and we really need to work on this particular thing? Everyone’s just putting their hands up and saying I don’t report to anyone. This is open source.”

Jon Atack: That’s true and it’s good to steelman the process with questions like that. In fact there are a fair number, a very small number, but there are people who work on it pretty much constantly who are day in, day out present working on the Bitcoin Core codebase and handling issues and discussing, reviewing pull requests. There are not nearly enough of them and even fewer who are experienced but there is a small number who are working full time and who are more or less mostly supported by sponsors and funding or even have a full time position to do that. And so these people are very reactive.

Stephan Livera: Awesome, so let’s break that down a little bit in terms of, as you were mentioning, there’s this codebase and part of it is a matter of coming to terms with it or coming to grips with it. What are some good ways for a developer to learn about that and then start contributing changes or potential changes such as a pull request to it?

Jon Atack: That highlights a common error that many new people do make in that they come and they want to change things. That would be probably be the worst step to take, to arrive and start proposing to change things. What is best is to first take some time to observe the process, to see how review takes place, to see how people interact, what kinds of things are desired, what kind of review at what kind of time. Some kinds of review are desired at certain points in the process and not in others. Watch how people interact in a positive fashion, giving positive feedback to help construct things in the right direction. The worst possible thing would be to arrive and start saying “I think we should change this and we should change that and here’s my pull requests”. Unfortunately there are quite a few people who do do that and don’t take the time to learn the social process behind contributing. I’ve been compiling since last March, I’ve been a part time Bitcoin Core developer for about eight months now. I’ve been compiling what I’ve been learning and observing into a document, which we can put in the shownotes if you’re keen to do so. One is how to make Bitcoin Core PRs and another one is how to review Bitcoin Core PRs. The second one is the first one people should begin with. There is a list of other articles that people could begin with by Jimmy Song, John Newbery, Pierre Rochard, Jeremy Rubin and my articles build on those and go deeply into the process as I’ve been observing it since eight months.

Stephan Livera: So you mentioned taking the time to understand the community and the social aspect of the development. I’ve noticed there are different communications channels, if you will. So you’ve got IRC, you’ve got the GitHub comments itself. There is the Bitcoin and Lightning, they’ve each got a mailing list as well and sometimes just from private emails and conferences and so on. Can you outline a little bit about the communication there and how a person can learn a bit more from monitoring those?

Jon Atack: Absolutely. I would say the main three vectors of communication are IRC, which is internet relay chat, which we can define for your listeners if needed because it’s an old school communications channel that many people aren’t aware of nowadays. It’s very heavily used for Bitcoin development as well as Lightning. There is the GitHub where the main conversation happens around specific issues and pull requests. And then there are the mailing lists which people can subscribe to freely. That is where more in depth discussion takes place. Finally, there are the BIPs, Bitcoin Improvement Proposals, which are proposals that can be adopted, may or may not be on an ad hoc basis in terms of if people believe that they are valid.

Stephan Livera: In my own experience, I’ve noticed it’s quite difficult to stay on top of all of these different channels. How do you do it?

Jon Atack: It’s crazy, it’s a full time job. To be honest, I’m a part time Bitcoin Core developer who aspires to hopefully one day work on it full time, depending on if I find the support, funding grants or a position doing so because there’s not nearly enough people who are working on it full time. But really following things along is a bit of a full time job. If you’re not on it pretty much night and day, it’s pretty difficult to keep up. There’s a lot happening.

Stephan Livera: I’ve found one good resource and I know you have contributed to that as well, Bitcoin Optech, which is a newsletter which spells out the digest version, the weekly digest of what was important over the last week or so.

Jon Atack: Yes Bitcoin Optech is a fabulous initiative. I believe it was founded and created by John Newbery at Chaincode Labs. John is someone who has really, really been helpful to me. I could go back and relate how instrumental he’s been in what I’ve been doing this year. Every week I review the newsletter before it is released to make sure that everything is as best as it can be. Bitcoin Optech, I don’t know if your viewers are familiar, they provide educational resources about scaling and efficiency and how to best work with Bitcoin for industry who sponsors their work. They have the newsletter but they also have workshops. They have a Bitcoin scaling book and other resources online for people to learn. It’s a really good project and I believe they’re open to ideas from people for taking it to the next level.

Stephan Livera: That’s great. I’d definitely recommend listeners if you’re interested to subscribe to Bitcoin Optech. Let’s jump into a little bit more detail on the process around what gets merged in. It’s not just anything that gets merged in. Maybe if we spoke from a life cycle point of view. Let’s say somebody has been lurking the communications channels for a little bit of time and now they’re a little bit more comfortable with how things are going. Now they’ve come up with an idea to improve Bitcoin. They might typically float that idea and get feedback on IRC or on GitHub. Can you talk through that process a little bit, what typically happens there?

Jon Atack: Sure. Would you mind if I mentioned though, I think a very important, completely under-appreciated point is that way before getting to that point, it’s better to spend time reviewing and testing. The main thing that Bitcoin is lacking is not proposals for improvements. By far the greatest resource constraint and bottleneck is experienced review and testing. Long term Bitcoin Core developers cite these as not only the most needed resource, but also the best and most helpful way to begin contributing and earning reputation in the community of developers. The developer community is a very different space from the Crypto Twitter community or the finance, econ, investment community around Bitcoin, which was one of the things that really surprised me when I started making the shift towards the start of this year, certain topics that everyone else talks about are absolutely not discussed at all. For example, price. You’ll almost never hear price discussed in Bitcoin development circles. It’s in fact maybe a good way to not be appreciated. It’s not that it isn’t important. I think this is an interesting point for people to understand because it’s not that the developers don’t care about price. I’m sure they do, like everyone else does. But I think that if you allow price to take up the bandwidth that it takes up on Twitter and in the rest of people’s mind space, I think it represents an obstacle to contributing and following the very in depth discussions. There’s so much context, so much history, so many things to learn and follow in terms of Bitcoin Core development and Lightning development, that there’s just no bandwidth to usefully give to price. Sorry for the deviation, I thought that’s something that might interest your listeners.

Stephan Livera: Let’s jump into the review component of that as well. So let’s say you want to join Bitcoin and you want to be a contributor and you want to start by reviewing. What does that look like? Are you commenting on different proposals? Are you testing them? Are you providing a concept ACK, are you testing, can you spell out what that looks like?

Jon Atack: Sure, I think at a high level the goal as a newcomer is to try to add value while being friendly and humble and learning as much as possible. Because no matter what, you’re going to make missteps, you’re going to say stupid things. I still do and I don’t think there’s an end to that. But a good approach is to not make it about yourself or as many people do, arrive hoping to get a commit in and, and wow all their friends and local community. But the better approach I think is how can I best serve? It’s really challenging because the codebase has a great deal of breadth and complexities and all the technology surrounding it. It’s not enough to be a good coder, it’s not enough to be a good cryptographer, it’s not enough without having all the history and the context. And so it’s important I think as a new person to be aware of everything that you don’t know. The long term people have years of experience and there’s a deep collective wealth of knowledge that has been built up by the community. So you have to keep in mind that some of your new ideas have most likely already been proposed or considered several times in the past. That’s why it’s important to not just show up like a bull in a china shop. At least that’s my opinion. Also you have to keep in mind that contributor and maintainer resources are limited. There’s not many people, so you have to ask for their help carefully and respectfully and to try to give more than you take ideally, and help more than you hinder as you’re getting up to speed. So it’s important to try to figure things out on your own and respect other people’s time. This begins with following the IRC, Bitcoin Core dev IRC channel, subscribing to the Bitcoin dev mailing lists and following everything on GitHub. And before you jump in, yes, you need to observe the process and the guidelines, read all the documentation, which I’m sure we’ll get to later. It’s critical to read the documentation in the codebase, watch the interactions, get to know who is doing what and how they’re giving feedback, and then we can get into the details of providing the review. And review, the big picture is more important than nits, code style and spelling, which I still fall into. I still am tempted to say “This word is spelt wrong, this code style isn’t nice. Oh, I see this little nitpicky thing.” That actually isn’t very helpful compared to understanding the big picture, which is much harder and longer to do.

Stephan Livera: So let’s take that a little bit further then. So an example might be you’re providing some feedback on a PR. What would that look like? What would you normally do as part of that? Would you say, pull that branch, test it and then provide feedback on it? What are some ideas there?

Jon Atack: Concretely what people do when they review, and that’s getting a bit more into the weeds where I was going to stay a bit high level. The technical specifics are you need to get really good and comfortable with compiling Bitcoin Core from source because you’re going to do it for every single PR nearly, that you review unless it’s documentation only. You might want to set up an automated system for compiling and running the unit tests and the functional tests, making sure that all passes and reproducing everything. You’ll want to be reproducing and testing issues, and you’ll be wanting to do the same for every PR that you review. So you launched that and that can take some time. While it’s building and running all the tests, you’ll want to start skimming through the code, reading the comments so far and getting an idea of where in the review process we currently are. Is it the beginning, are we in the middle or are we getting towards the close? For example, ACKs depend on where we are. If it’s a brand new PR and a fairly critical or large one, this is a bit more into the weeds but we’re talking now about concept ACKs and approach ACKs. ACKs are basically comments that approve a change and ACK means more or less I’ve read this, I’ve reviewed it, I’ve tested it and I think it should be merged. There are different kinds of ACKs. Concept ACKs, Approach ACKs and then full stop ACKs which basically means, okay it’s ready to go but you should give reasoning behind it. You could also disagree with the change and you could NACK it which is N-A-C-K. NACking is something as a new contributor that I would strongly avoid doing because it probably means you’re lacking context. I would first question if I understand everything. I wouldn’t NACK lightly. In fact, I don’t think I have without giving very good justification and reasoning. You don’t just say NACK, not if you’re a new person at least. Should we go into what concept ACKs and approach ACKs are?

Stephan Livera: Yeah. Do you want to just outline what they are?

Jon Atack: Sure. Concept ACKs are things that are given towards the beginning of review, which means that you agree with the concept of the change but you haven’t confirmed that you’ve really looked at the code or tested it or maybe the code isn’t ready to be tested. Perhaps the contributor might have said “I’m just looking for Concept ACKs and so you would provide one or not. It is a valuable signal, it can be a valuable signal to an author of the PR to let them know that it has merit and is heading in the right direction. Approach ACK is quite new. I think that was added in June at the last Core dev meeting discussions. Approach ACK basically means “I agree with the concept and the direction of how it’s being done in the codebase.”

Stephan Livera: Could you just outline a little bit around this idea of the structure of a branch and the master because I think that’s also something that an outsider might not understand or appreciate as well. So you might have the master branch, which is let’s say the main Bitcoin Core code and then somebody might be creating their own offshoot if you will. Could you explain what that is? What is that branch that people are testing when they want to do this review work?

Jon Atack: As you said, there’s master and then there are branches. There can be branches on the main repository and each individual contributor can create their own branches locally and then push them up when they’re proposing changes. When these changes are merged they become part of the master branch. But the GitHub repository also contains release branches which is most of what people download and install actually, released branches for 0.18, 0.19 which is coming up now and being released. So on the repository you have those master branches that people install, the release branches. You have the master branch which once every release is merged into a release branch and which contains all the latest changes. Then you have the PR branches. When someone proposes code, they push up their local branch, they’ve made a branch of master, they’ve changed it, they’ve pushed it up in a pull request, and then reviewers will pull that pull request branch down into their local repository of Bitcoin Core and test it locally. It’s very important to test things locally and not trust GitHub, don’t trust verify. So review takes place locally. GitHub tries to encourage review on its web interface but it’s generally discouraged, at least in Bitcoin Core to do that. You pull it down and you view it locally.

Stephan Livera: Another thing, just to clarify for listeners who are not as familiar, can you spell out what are some of the different roles here? Because we’ve got developers, we’ve got people who are doing review, we’ve got people who might be doing testing and then we’ve got the maintainer role and then maybe documentation as well. So could you just outline at a high level, what are these different pieces and what do they each do?

Jon Atack: At a high level you basically have a large number of contributors. You have some people who just report issues and then you have a core of four and now five maintainers. It used to be four and then in June there was a fifth one that was added. Maintainers are the people who have the commit bit. They’re the only people who have the right to actually merge changes in, after peer review, into the master branch which at one point will be merged into a release. So you have five maintainers, you have a fairly core group of I would say 10 to 15 daily contributors who are contributing, reviewing, testing on an everyday basis. It’s a very small group, surprisingly, and then you have other people who file issues as they come across them.

Stephan Livera: I think something that might be very interesting as well is around conflict when it comes to people not agreeing on what should be going into Bitcoin Core. Can you talk a bit about how that is resolved and how that’s then dealt with by a maintainer for example?

Jon Atack: Well, I can’t speak for the maintainers because I’m personally not one but conflict happens. There hasn’t been a whole lot of conflict this year. I would say if I had to point out one thing where I was surprised to see a tiny bit, certainly nothing compared to the conflicts we saw in 2017. There was a bit of discussion about introducing Rust components into Bitcoin Core, which is a new thing, injecting Rust. Bitcoin Core being in C++ with functional testing in Python at the moment. Basically discussion happens on IRC during the Bitcoin Core dev meetings, there’s a weekly meeting. There’s also a biweekly Bitcoin Core wallet meeting as well as the weekly Bitcoin Core PR review club meeting, which we should talk about for anyone interested in getting involved. I lost my train of thought.

Stephan Livera: That’s fine. Basically what I was getting at is that there’s a certain conservatism about how Bitcoin Core is developed and that things aren’t necessarily just merged in willy nilly. It sort of has to have that rough consensus idea. Can you articulate what that is?

Jon Atack: To be honest, as a developer, I’m a bit surprised how quickly things actually move. From the outside, Bitcoin has a reputation of being very slow and glacial in terms of progress. Maybe that’s the case for very large changes like Taproot and Schnorr signatures and things like that, which do take time and rightly so. They should. But at a codebase level, things are happening actually rather quickly. If you’re not a full time reviewer, sometimes things get merged in before as a part timer you’ve even had a chance to really go through it completely. It is very hard to stay on top and review everything. There’s a lot happening but basically the process is very collegial and there’s not that much conflict. Maybe at a very small micro level one person will say “I don’t agree with this change”. Another person might say that. But more or less consensus is come to. I think it reminds me of the internet standards committee process of consensus. Are you familiar with that?

Stephan Livera: I’m not. Can you outline?

Jon Atack: So back in the 90s, there was the internet engineering task force and they came together. They brought together the idea, I believe it was Dave Clark back in 92, of rough consensus. That’s where you see the familiar quote, which you’ve probably seen. Quote: “We reject kings, presidents and voting. We believe in rough consensus and running code.” There’s the whole basis of a lack of disagreement is more important than agreement and that rough consensus is achieved when all issues are addressed but not necessarily accommodated. So consensus is a path, not necessarily a destination. You won’t necessarily achieve it and in fact it states that 100 people could be for and five people against and it might not be rough consensus. Or inversely five people could be for and 100 against, but it might still be rough consensus. There’s a document that I think people interested in knowing more about could read. I could send you a link if you like. It’s the internet engineering task force, rough consensus guidelines. I suspect that that’s more or less how things are happening.

Stephan Livera: My understanding, as I’ve heard explained via some other people is that essentially for a change to come through in Bitcoin Core, it sort of has to pretty much be a net win with very few losses in terms of compatibility or security or otherwise people won’t be in favor of that change. What’s your perspective on that?

Jon Atack: I think that I completely omitted to mention that the Bitcoin Core repository actually describes the process of coming to consensus on peer review. It’s in the root of the repository, a document called contributing.md and there’s a section entitled peer review. It describes the whole process of review. For example, I quote from it: “Maintainers reserve the right to weigh the opinions of peer reviewers using a common sense judgment and also may weight based on meritocracy. Those who have demonstrated a deeper commitment and understanding towards the project over time or have clear domain expertise may naturally have more weight as one would expect in all walks of life.” I think that’s a pretty good summary. I apologize if I didn’t answer your question.

Stephan Livera: No, I think that’s a totally fair enough answer there. What about the testing component of that? So as I understand there’s different types of testing. Can you outline what they are and what’s the importance of that?

Jon Atack: Testing is vital and there are never enough people who are testing Bitcoin Core releases during the release candidate process, which it has just been in in the last month. Or testing issues and pull requests as they come in. When a change or an issue affects consensus critical code, keep in mind that Bitcoin Core’s overarching mission is to maintain consensus. Nothing is as important as maintaining that and the robustness and censorship resistance. So the bar on those sorts of issues is much higher in terms of discussion and peer review requirements because it could be very costly to the whole community. So those things tend to see many more eyes and need more ACKs during the review process than a documentation change which can be merged quite quickly and easily.

Stephan Livera: What about the role of automated testing? Can you tell us a little bit about what is automated testing and what’s the importance of that?

Jon Atack: Automated testing is writing tests that test a certain area of the codebase. It could be a unit test which is at a very micro level or it could be a functional test which tests more on an end to end basis or a user view basis. They are both in Bitcoin Core as well as other newer kinds of testing that are being worked on like fuzzing, fuzz testing is another way of saying fuzzing. Or property based testing which I’ve been working on along with Chris Stewart. Marco Falke, the maintainer and practicalswift have been adding a bit of fuzzing lately and improvements to the testing framework, making the coverage wider or better. Adding missing coverage or improving the speed of these tests are very welcome in general, in terms of changes from the community. A good way to start with contributing when one does so is to improve or add documentation or tests. Test speed is really important because how long it takes to run all the tests matters in terms of how many people are actually going to run them. It can be quite long. For example, my last commit to Bitcoin Core was in fact speeding up a functional test which took 30 seconds to run. It was one of 120, 130 tests in the functional test suite. I think it sped it up from around 30 seconds to around 8. Those are the sorts of things that we need more of because faster tests means more tests being run by people.

Stephan Livera: Great. Can you tell us a little bit about what property based testing is?

Jon Atack: That’s getting a bit into the weeds. I haven’t got it top of mind even though I did contribute to it. People who are interested in looking at it, I encourage them to look into the repository doc folder where documentation is contained doc/rapidcheck.md, rapidcheck because that’s the name of the property based testing library that is being slowly introduced into Bitcoin Core. There I added some documentation about that.

Stephan Livera: What is an example of a test that you might run in Bitcoin Core and what sort of error might it be catching?

Jon Atack: A unit test could be as small as testing one single function or one change in the code base at a very small level whereas a functional test I think is more easy to grok. Functional tests will test things like if you call an RPC or wallet function, like say if you call getaddressinfo, it might test that the information corresponds to what should be returned and what that information should be. Functional testing can be a user viewpoint of the codebase. So for example, you might test, if I call getaddressinfo, what kind of info am I receiving and is it the correct information?

Stephan Livera: Awesome. Let’s talk about review again. You mentioned the weekly PR review. I think John Newbery has been leading this one and running a weekly IRC meeting. Can you tell us a little bit about this process? What is it?

Jon Atack: Yes indeed. I’ve actually been quite involved in the review club. John started it in on May 1st I believe, this year. So the PR review club just celebrated six months of exciting existence. Basically the Bitcoin Core PR review club proposes weekly meetings on IRC, it has its own channel. The club provides questions and study notes to review and help you understand the PR. That will be the object of next week’s meeting. Ideally you should spend time going through it, reviewing it, going through the notes, questions and logs of the previous meetings as well. Take a deep dive. It can take hours or even days to really get the most from the weekly meeting. It is every Wednesday at 5pm Greenwich time which depends on your timezone when that is for you. I really encourage anyone who’s interested in getting involved in the club to participate. There are typically 5 to 10 participants per week so it’s quite an exclusive little affair, but it’s open to anyone and the website we can put in the show notes if your listeners would be interested,

Stephan Livera: Tell us what that meeting looks like. Do you open it up and say “here’s the pull request”? What’s the process then?

Jon Atack: It is adhoc. People can jump in at any time and there can be several conversations happening simultaneously on IRC which is one of the wonderful things about it. That’s all good, that’s welcomed. Generally the host of the week, it is very often John, sometimes it’s other people who guest host, I hosted two of them last month, will open up the discussion by saying “Ok here we are. What did you think of this PR? Did you test it? Did you review? What did you think? Who made a review on Github of the actual pull request which we encourage people to do? The discussion goes forward and we follow the questions. Sometimes the conversation deviates and that’s all good, it’s adhoc. It’s an hour long of adhoc, simultaneous multithreaded discussion.

Stephan Livera: Okay great. I’d also like to talk about the process of documenting and documentation within Bitcoin Core. Can you give us a high level understanding of what does documentation in Bitcoin Core look like?

Jon Atack: There are two main types of documentation. There is pure documentation which in the Bitcoin Core repository is in the folder called doc. Most of it is in that folder there, it is a bit spread out and that’s because there’s so much of it. There is so many different kinds of information that it tends to be quite spread out over at least 30, 40 different files if I recall correctly. You also have documentation in the test folder and in other places. You really have to explore. In terms of getting started contributing to Bitcoin Core, I would say there are three. The first three that one should look at are contributing.md which is in the root, the top level, and then in doc you have the developer notes and productivity. Those are the first three they need and then you should look at the testing documentation. Again anyone who adds helpful documentation or improves existing documentation, that tends to be well received in terms of a new contribution.

Stephan Livera: Let’s talk a little bit about what directions you would like to see in terms of the Bitcoin Core codebase. Are there any things that are particularly interesting to you?

Jon Atack: There’s just so much to do. I would say that my personal priorities are in order: the robustness of the codebase and Bitcoin Core functioning and maintaining consensus followed very closely by maintaining and improving privacy and censorship resistance. Those would be my one and two. After that features which are more fun to work on, they’re a bit easier, and scalability and then UI, UX, usability for average users. I put them in that order, not for what everyone should do, but that is what my personal priorities are. There are many, many projects going on at any one given time right now in Bitcoin Core. One of the ones that interests me the most is improving the robustness and privacy of the P2P networking code and the testability. It’s a really hard area to understand and to simulate and to test. I think there’s a lot more work that could be done if we really want to get into it. There’s things like better testing tools and frameworks for evaluating how the behavior changes in complex state when people are making changes. There are ideas like multiple P2P networks instead of just one. Currently there is a separation ongoing between transaction relay and block relay. That’s being done by Suhas and AJ and others are working on it. Some of these ideas are from fellow Chaincode residents like James Chiang who I really enjoyed getting to know last summer. I forgot to mention that I ended up being invited to go to Chaincode for the seminars and met a really great bunch of people. Chaincode is an amazing place and I was actually mildly depressed after coming home from it because I missed the people and the excitement of being around those ideas and that stimulation. In general we could use a better framework for thinking about privacy systematically, simulations for evaluating things like dandelion, P2P improvements, better test coverage. But also things like improving the testing like property based testing, fuzz testing, improving the modularity of the codebase, separating for example the node from the wallet. There is multiprocess work that’s ongoing. That’s very slow and difficult to slowly untangle the various components that were all in one file originally called main.cpp and to separate them out into modular things. Russ Yanofsky at Chaincode Labs has been working extensively on that and that’s a long term project. I think he has been doing it for a year or two already, at least. There’s the assumeutxo work. There’s benchmarking work. Those are both being done by James O’Beirne at Chaincode. There’s Carl and Cory Fields who have been working on improving builds and reproducibility with Guix. There’s constant interference from companies like Apple who are trying to make it harder to do that. For example, in its new release Apple is forcing users to right click and say they accept. Apple is always trying to close their garden down. There is Matt who’s working on adding Rust components into Bitcoin Core along with others. So much going on. Of course, Taproot and Schnorr, we have the large review, the BIP review that’s just begun. That’s a two month process with amazingly over 160 people participating, which is fabulous.

Stephan Livera: Let’s jump into that a little bit. Can you give us a high level overview? What is the purpose of the Schnorr, Taproot review project?

Jon Atack: We’ve often heard on crypto Twitter, people often outside of Bitcoin, complain about how slow progress is in things and they often like to cite Taproot and Schnorr as examples. Whereas of course these need review extensively and maybe there was just a need to organize it more systematically and extensively. That’s a fabulous initiative that’s been taken up by, I’m sure I’m going to miss a name, but people like Anthony Towns, his nickname is AJ on IRC, Mike Schmidt Steve Lee and others who I’m certainly forgetting and I’m sorry about that. They’ve basically organized this whole eight weeks of review by topic. There’s a curriculum and people signed up for it. We’ve split off into around 10 study groups, maybe even more because there’s 160 people involved now. We meet once a week on a per group basis. Then there’s two one-hour question and answer sessions on IRC with people like Pieter Wuille who I’m mispronouncing, other experts like Andrew Poelstra and Anthony Towns. It’s really good to be able to ask questions, then go back home to individually review and to propose changes if needed. I completely forgot Gregory Maxwell. He’s also involved. He’s been present at the question and answer sessions and there’s nothing more amazing than seeing people, when you get together in the same IRC room, Gregory Maxwell, Pieter Wuille, Andrew Poelstra and Anthony Towns riffing away on Taproot and Schnorr. It’s an amazing thing.

Stephan Livera: That’s awesome. Also I’m keen to ask you more at a high level, what are some things that you see? Obviously you’ve got some open source experience even before Bitcoin. What are some unusual or exceptional things about Bitcoin Core compared to other open source projects?

Jon Atack: That is actually a very good question to my mind. One of the most impressive differences in my experience, which was working with web frameworks, databases and search engines mostly, is that here we have a codebase that is very critical and bugs can be so costly. There’s not only the criticality but there is also the lack of experienced reviewers and we really could use more. This leads to an interesting situation where maintaining ACKs has much greater importance than in other projects I’ve seen. In other projects, often things will not be merged until everything’s just right, test coverage is complete, all the syntax nits have been handled and the linter smashes everything down to be just as it should be. In Bitcoin Core things are often merged in an incomplete state simply because if there is enough review behind something someone could find some things that still should be changed but then we don’t want to lose the ACKs, the review time that has already been invested in it. It might well be merged with a follow up “Ok, as a follow up we’ll handle the missing test coverage or “we’ll handle the missing suggestions to improve the codebase style” or this and that to make it more modular or more readable for developers or to add documentation. There are lots of little bits and pieces that are often not finished but review is so critical and so difficult to come by. Once there’s enough review things get merged even if they’re not complete. That is quite new to me and I’ve had to overcome a lot of instincts on that.

Stephan Livera: Another thing I’ve seen is this acronym BDFL. What is a BDFL?

Jon Atack: A benevolent dictator for life, yes. I’ve actually got turned off from open source because so many projects out there are led by a BDFL as well as programming languages. Everything’s fine for the best of everything until you find yourself in stark disagreement with the BDFL and you realize that the BDFL always wins. For example, in the Ruby on Rails framework there’s DHH who is very well known at least in that ecosystem and he can basically override with an iron fist the entire Ruby on Rails core team if they disagree with him if he feels strongly about something. One thing I found really nice about Bitcoin Core and that personally was very important to me becoming involved again in open source was seeing the lack of centralized BDFL attitude, at least from what I can tell in the Bitcoin Core maintainers. Yes, there are only five who have a commit bit, but the consensual approach really does seem to be respected and there seems to be, the closer you get into Bitcoin Core, I find the more humble people actually are. I think that’s amazing. I find it inspiring that sort of open source humility, “how can I serve?” attitude rather than “this is the way it’s going to be”. I really like that, I’m really drawn to that adhoc approach. That’s a really important reason along with the activism and principles behind Bitcoin. That’s why I’m here. I would say that many, many people, if there were to be a BDFL or if the adhoc approach were to be lost, I think a lot of people would leave. At least I would.

Stephan Livera: I love the explanation there. There’s a certain ethos around it and it is fair to say that Bitcoin Core has some history and specialized knowledge required and that leads me into the next topic I was keen to discuss, which is around funding of developer time and developers who are working on this. Now we have some organizations in the space that are doing this. Obviously Chaincode Labs, Blockstream, I think Xapo is funding AJ for example, there’s MIT DCI, there is now Square Crypto as well. What’s your view around funding for these developers and reviewers and contributors? What’s your take?

Jon Atack: To their credit Xapo also hired one of the Chaincode residents last summer, Amiti who I think is working with AJ. So that’s two for Xapo, congrats. You also have grants. You have sources of grant funding and sponsorship. For example, some of the exchanges like BitMEX Research, even Bitmain has been sponsoring people in the past. I think OKCoin has done a bit of funding, granting and I hope that they will be involved more and more in the future. There’s Kraken who are perhaps coming into this. They’ve hired Pierre Rochard to hopefully develop their open source program among other things. I think he’s working on the strategic side of things for Kraken, which is a great exchange as you would agree with. Some say that there has never been a better time to be funded and sponsored. I would say that, from what I can gather, most of the daily present Bitcoin Core people are either working full time on it, paid to do so or are funded in adhoc ways, receiving grants or sponsorships. Jeremy Rubin, I believe he received something from BitMEX Research or OKCoin, I’m not sure. I’m certainly mixing people and grants up. Square Crypto is hopefully looking into funding some individual developers but they’ve got a lot going on with the team they’ve hired and they did of course support BTCPay Server. There’s different things afoot. I personally would be interested in finding something like that for myself. I’m relatively new but this is something I’m aspiring to, to move sustainably from part time, a few hours a week to full time.

Stephan Livera: That’s great. And do you believe more is needed? What would you like to see in this space?

Jon Atack: Yes, definitely there is more needed. There are not enough people who are present every day reviewing and testing issues. Keep in mind that at any given point of time, Bitcoin Core alone has roughly 700 open issues to test and roughly 300 open pull requests. They need to be tested and reviewed and all of these take time. Some of them are old and sitting there rotting or need to be picked up by someone. But that’s still a great number of things that need help. So yes, I very much encourage companies, industry who are not yet supporting open source to get behind open source and do their part. As for myself, I’ve been saying to myself that if I do not find funding before I run out of runway, although I do do some freelance work, I do some lecturing at universities. I’ve been turning down full time hardcore missions, which is what I was doing for a decade and just doing little things in order to preserve time for Bitcoin Core, which has a very great opportunity cost. But I said to myself, if I run out of runway or just don’t find any support, perhaps maybe what should be done is to make an industry organization that goes around and develops it, builds a developer fund from people in the space: angels, organizations, companies and perhaps bring to bear social norms to encourage each other to up the ante and participate and be involved. Because if Bitcoin isn’t solid and robust how is that going to work out for exchanges and different people in the space? I know that Steve Lee is interested and Miles Suter from Square Crypto, John Newbery, Adam Jonas, many others are interested and Pierre Rochard as well in bringing some social norms to this so it becomes the norm to help and not the exception. Of course there are some angels who have been heavily solicited like the two Chaincode Labs founders and I believe John Pfeffer, who I have not yet personally interacted with, but these people have been supporting Bitcoin individually for years and congratulations to them.

Stephan Livera: That’s great to see. Hopefully as Bitcoin grows we’ll start to see more and more companies realize that they want to be building on a very solid foundation. Part of that involves funding development and education around Bitcoin and particularly Bitcoin Core development and contribution, potentially even other Bitcoin whales right? They should be thinking of it like “Hey I want to protect my investment. I want to help ensure that Bitcoin stays resilient and stays strong.”

Jon Atack: Absolutely. There are really good reasons to help Bitcoin. Protect your investments absolutely, visibility for your company, public relations, having a more credible voice. Not necessarily influencing developers directly because it’s important to let them work on what they feel is important. But developers will probably be more amenable to listening to you and your concerns if you’re helping the open source community. It just all makes sense.

Stephan Livera: Excellent. I think we’re coming close to time but Jon if you have any closing comments for the listeners, make sure you just leave them with some parting thoughts on Bitcoin Core and development.

Jon Atack: Personally, I’d like to really personally thank John Newbery and Adam Jonas and Chaincode Labs for all the help they’ve given me in the last eight months helping me ramp up to become a Bitcoin Core developer. I want to thank the maintainers and long term contributors on Bitcoin Core for all that they do. I think that people underestimate how difficult it can be to be a maintainer. It takes a great deal of patience, generosity and time. Don’t underestimate the difficulty of it, it’s not a power trip at all. As people might imagine from the outside, it’s really humble service, gardening, pulling weeds and trying to make the Bitcoin Core garden as healthy as it can be. I encourage everyone to help Bitcoin Core, Lightning open source, get involved and do what you can.

Stephan Livera: Fantastic. So before we let you go Jon, make sure you tell the listeners where they can find you if they want to get in touch or potentially if one of the listeners would like to fund you, how can they get in touch with you and keep up with what you’re doing?

Jon Atack: Thank you. Well, I have a Github, a Twitter account and also a Mastodon account. All three are jonatack. That’s my first name and last name. That’s my Twitter and my Github. I’ll be putting a website up soon online as a sort of blog and what I am working on, proposals for funding and grants and perhaps a position working full time on Bitcoin Core.

Stephan Livera: Excellent. Well, that’s going to do it for today. Thank you very much for joining me, Jon.

Jon Atack: Thank you Stephan.