Alex Bosworth

Channel Management

Video: https://www.youtube.com/watch?v=HlPIB6jt6ww

Slides: https://lightningresidency.com/assets/presentations/Bosworth_Channel_Management.pdf

Intro

Today I’m going to talk about something a little bit different to building apps. I’m going to talk about becoming a great router. Managing your node so that your node can be routing payments and be a good citizen if you choose to have public channels.

Uptime

So basically the big message overall is and I think I’ve said this before, if you have public channels you’re kind of like waving to everybody “please use these channels to do routing”. If you don’t want to do routing you should set your channels to private. Those could technically route but you’re not waving to everybody “please use me for routing”. So what do you want to do if you have public channels? The first thing I think that’s the most critical for public channels is that you want to maintain high uptime. If somebody is on their wallet, on their mobile wallet on their phone, they have this graph, this blob of people, of connections. They want to make a payment. How are they doing that? They don’t know if you’re online at the current moment so they’ll try different paths through the network. So what’s a common way that one of those paths could fail? It could be just that that person closed their laptop and now his public channels are inaccessible. The number one rule if you have public channels is just be online all the time, minimal maintenance downtime. And also try to connect to other people. If your connection gets interrupted try to make sure you can continue to connect. Also you should be thinking about other people more generally. You should try to avoid connecting public channels to people who themselves are not maintaining good uptime. You can even carry this forward across jumps. So even if you connect to somebody who’s online a bunch, what if you look at their channels and their channels are not connected very well so they’re going offline all the time. That’s not a great experience, that’s still going to interrupt. You should look at your own uptime, you should look at all of your peers’ uptime and you should even carry that forward to the peers of peers.

Balance

The second super important aspect of maintaining good public channels is you should really try to make sure that your channels are balanced. The second cause of failure when somebody is on his mobile phone trying to make a payment is that he will send out a payment and it will go on hop 1, it will go on hop 2 and hop 3 doesn’t have enough liquidity and he has no idea that that’s the case and it will then fail back and we will have to try another route. In order to minimize this error you want to try to make sure that your channels have a reasonable balance in them so that he has a reasonable guess that if he tries this channel with something that’s a lot smaller than the total capacity that it will have a good chance of success. One thing this means is that you…. I’ll repeat this message a bunch because people have been doing this too much, is do not open a huge number of channels to everybody in the network and just have tonnes and tonnes of outbound capacity. What will happen is that you will not be able to route at all because nobody will have any capacity to you, all of your channels will only be able to send and you will not be able to receive from anybody. You’ll be the source of a lot of problems. People will know that and they’ll hit your channel and they’ll have to go back and try again. But on the other hand I would also say don’t allow the reverse to happen. One reason we would like to reduce the amount of merchants who are accepting inbound channels is because we see this problem happening where a merchant has so many inbound channels that he can’t send out. So everybody is making just channels from themselves to the merchant and they’re all sending tiny amounts but they’re using huge channels so they don’t have to reopen channels and pay chain fees a lot. It kind of creates this relationship where if somebody opens a channel to you, you should either limit new channels or you should make new channels of your own outbound so you can keep that in balance.

Q - Would it make sense for a merchant to do what you said and open channels all across the network?

A - If somebody opens a big channel to you, you can now open a big channel to somebody else so you can keep it in balance. It doesn’t have to be one-to-one just somewhat reasonable. This will get better when we have better technologies for multiplexing as well and splicing, dual funding. You could even change that channel so that if somebody opens up a channel to you, you could create a corresponding amount of capacity back to them. Right now, the easy way to do this would be just to “ok I’ve got a channel coming in, I’ll make a channel coming out.” On Y’alls I have a one-to-one balance so people pay Y’alls and people make channels to Y’alls and when people make channels to Y’alls and assign capacity to Y’alls I make capacity out to the network and I have about a 1:1 ratio.

Re-balance

Another way in addition to opening new channels is that if channels become unbalanced, you can rebalance in network, you don’t have to make new channels. We don’t have great tooling for this yet but you can actually rebalance within the network so that if your balance is lopsided on one channel and lopsided in a different way on another channel, you can smooth those out just by routing back to yourself. That’s one way you can help fix balance problems. Ideally, you have this channel that’s out there in the wilderness and somebody on a mobile phone is watching this channel and he has no idea which side the balance is on. The best way for success is that you create within 25-75% of the balance is in one way or the other. Another way you can fix balance issues is you can use an exchange. Let’s say I open up a channel outbound. How do I balance? I can’t ask other people to start opening up channels to me. I could go on the forums and say “please open channels to me” but they have to lock their capital to me and they might not know me from Adam. One way you can do this is you can open up channels in the direction of exchanges and if they are an exchange that accepts Lightning, you can pay the exchange with Lightning and then withdraw from the exchange either totally offchain or you can do an onchain withdrawal. That will help you create a balanced channel where you can make it 50:50 pretty easily. You can also use things like submarine swaps to accomplish the same thing if you don’t have a trusted exchange or you want to automate this. Another way to balance is to spend. If you are a merchant and racking up tonnes of money and you’re doing great on Lightning, it would be nice if you would spend some of that. So you would pay your employees in Lightning, you would try to bring other merchants onto Lightning. You can think you’re actually getting a discount in this case. If you already want to rebalance and you also have this spending need you can accomplish both of them at one time.

Activity

Even just spending. If you think about it, if you’re a routing node and you’re spending and in the future you might get fees, you’re kind of getting a future discount. If I spend $100 and then later somebody else uses that $100 back through my channel and I charge them 1% I just got a dollar off my past spend. Ok, so this is kind of a controversial point. I believe that fees should be charged so that it’s very meaningful what activity is. When somebody uses my node, I really know that they needed my node. It wasn’t just something they were playing around with on the network, running rebalance scripts or running probes, something like that. They actually had to use my resources and that gives me a strong signal. It turns a costless signal into a cost signal. One thing you should do as a routing node is that you should look at that activity. Once you have fees that are being paid you can start to say “Where does money need to be flowing?”. You can make intelligent decisions about your own capital. You can say “There’s a lot of money that needs to be sent over in this direction and I’ve got all my money on this other side. I need to reassigning more to another side.” If you didn’t charge fees you would really have no incentive to do that. You could just let your box sit around and it would just be harder and harder for people to send to their destinations because you wouldn’t be actively reassigning capital in the directions it needs to go.

Q - What you do is maintain your viewpoint of the network right? So if a lot of money flows in a certain direction you would… routing node…. so my route is available but in the meantime you rebalance your channels. If you don’t do that won’t people be able to just use a different route anyway?

A - If nobody is doing this then overall it is going to become unbalanced and then there will be no routes. If nobody is adapting to how the money is flowing then all the capacity in one direction will just become used up. Even an attacker could take advantage of this. Let’s say I want to exhaust the capacity to send a Bitrefill because I’m a Bitrefill competitor. I’ll just send all the money for free through those channels that people are going to use and now none of the real customers are going to be able to spend to them. You want to make it so that there’s some incentive for people to pay those chain fees or pay those rebalance fees so that they can move the capital over to where it should be. That’s my feeling.

Q - But the flow property of the network does not change does it? If you rebalance your node by sending money to yourself on some route, some channels change. The only meaningful way is to create new channels and then you can only add outbound capacity. You have to ask others to add inbound capacity somewhere or you have to spend…

A - The flow is changing. Even as money is moving around the network, it is also leaving the network. So it’s leaving and coming back on. We want more to come back on and we want more to come back on in the right places. As a corollary to this, I feel like you should be using your activity to signal to people that you want to keep their capacity assigned to you since you can assume that people will start to watch where activity is going and assign capital accordingly. If you want capital to be assigned to you on a long term basis you should be pinging and saying “There’s some money coming this way” so that if that person is choosing between two different options, this channel has zero activity and this channel has a tiny bit of activity, they will first close the channels and reassign the channels that have zero activity. That doesn’t have to be a lot of money, it could just be paying 1 satoshi. “Hey here I am I’m still around.”

Simple Balance Guidelines

So just to sum up my balance views. You should not make too many channels, you should not make too large channels. You’re just negatively impacting the network by making it a lopsided network. I would also say don’t make too small channels. A lot of people on the network currently are making tiny channels. They’re making channels like $1. We don’t have atomic multiplexing yet. Those $1 channels, the max that they could ever send would be an article payment on Y’alls.

Q - What’s too large and what’s too small?

A - Too large I would say would be more than twice max size of a payment. What are you looking for when you’re looking at activity? If I start to get some payments flowing through a channel the thing I’m going to do is assign more capacity to that destination. If the channel is so large, it just gives me way too much time. I don’t need that time because there will be tonnes and tonnes of signals that I should be assigning more capital there. Double the largest send. If you see one huge send come through you can react quickly and if you’re charging fees it’s justified because you made the fees that can pay for the new channel or more capacity in that direction.

Q - For the autopilot integrated with c-lightning I make an estimation of how big the channel should be. The way I’m doing this is I’m looking at the candidate node that I want to connect to and I look at the average capacity that they currently have. That’s basically the ball park of the capacity that I assign. What’s your opinion on that?

A - The problem is we don’t know the balances. I’m not a fan of the existing autopilot in lnd because it uses this idea that if you have more channels you’re better. You can game this by just creating tonnes of channels and then you’ll attract even more channels. The autopilot we have in lnd is like a stub and we have the same problems of how do we choose which are good channels.

Q - I was not asking about autopilot. I was asking about the idea of looking at the average capacity. Would that be a good heuristic in your opinion or would you rather look at something else?

A - I’m working on this problem as well. The thing I would look more at is median size. What we’re trying to figure out is whether this channel…trying to do a good job of curating his channels? Are they avoiding those too huge channels? Are they avoiding too small channels? What’s the general attitude they have?

Q - Do you think that channel management should be… at the application level or on the core level?

A - I think there’s a combination of both. Definitely there should be automated software to take care of this. It’s more of a question of should there just be one implementation of the automated software or should everybody be working on their own automated software and making it better and better and smarter and smarter. I think we can have a mix of both. Let’s say we had autopilots for real with cars that are on the road. Should there just be one autopilot that drives all cars and maybe autopilots should talk to each other on the roads so that they can work together. You have tradeoffs. If you have an autopilot which is standardized then you can’t really improve it because you need to get everybody else to improve it. But if you don’t have a standard then it’s harder to coordinate people. It’s kind of a work in progress.

Q - ….

A - But the hub manager should also have autopilot. Maybe it can be called something else. They should be following these guidelines, they should be paying attention to activity. Right now it is a manual process but in the future it should be more and more automated. Maybe eventually completely automated because a machine can pay attention to these signals far better than you could.

So don’t make too tiny channels. Don’t make too huge channels. If you see usage in a channel assign more capacity that way. If you don’t see usage in a direction reduce the capacity.

Node Centrality

We do need to think about how we can coordinate between ourselves some kind of standard, some kind of policy for how we’re making channels. One thing I think we should avoid, this also goes against what people say, I don’t think you should go to 1ML and just pick the top ten channels even though one of the top ten channels is my own channel. Don’t pick those channels and make channels to them because we’re creating this mass of channels that are surrounding these top nodes that just so happen to be the top ten results on 1ML. I’m not saying this network has to be completely egalitarian and you get your share of capacity. There’s negatives to creating these central things that even impact you. If you tie up all your funds with one node and if a huge amount on the network is tied up with one node and then that one node goes down or that one node starts to sell information about routing to other people, that’s not great. Even from a selfish perspective, if you are a routing node there’s no point to connect to one of these 1ML nodes because they’re so well connected to everybody else, you’ll never make fees on it. There will always be somebody to undercut you. As a routing node you have a big incentive to spread your channels out to underserved parts of the network so that you can have a better chance of competing. We can also do some things at the policy level to say if I see a node out there and all of his channels are just to the top ten. We can say at a policy level that that this guy is not doing a good job at making public channels so I’m not going to make channels with him until he starts to branch out his channels to more nodes. You even have an incentive for that.

Reputation

Another thing to think about with routing is that once you’re running a node you’re generating reputation for yourself as a routing node. You can also start to get information about reputation on the other nodes. One amazing tool we have is that we have this channel age. Even in the protocol we have a useful tool because channel IDs actually encode the height of the funding transaction right in the ID. Very easily you can look at a channel and be like “How long has that channel been around?”. You can use that as a proxy for how good is this channel because somebody out there is assigning capacity to this channel and he kept it open for a certain amount of time. If people are charging fees and if we have a healthy market for that, you can start to make some assumptions that this channel is being useful. Reputation is closely tied to age. More age on the existing channel sets so that these relationships become firmer and firmer. That will help routing because we’ll be able to say “That route is great. Those guys’ uptime is great, they keep their balance balanced.” Age also has other advantages. If you have a channel with tonnes and tonnes of confirmations you don’t have to worry about reorgs as much. Even if we have a monster reorg it could be a disaster level attack where people say “If this ever happens Bitcoin is dead.” You can even start to rely on the social consensus over the proof of work. The other thing to think about when you’re running a routing node is that everything you do that other people can see is a signal that they can later track. Maybe they’re not tracking it now but they can start collecting that history of what are you doing. If you try to do a good job in any way that’s visible, you can start to build up this history and build up this reputation for yourself that this guy has good uptime. This guy keeps his channel balanced. I see few errors coming from his node. He’s not connecting to all these weird nodes. He’s doing a good job. I’m making a bunch of money from him. Even if you don’t necessarily go for any specific signal, there’s an incentive just to do purely a good job.

Q - Do you have any comment on trust needed for if for example, I’m a new node and need to get some reputation regarding uptime…

A - I think as a new node, by design, it will start to become more difficult to be chosen because you just won’t have that history. You’ll have to put in the time. You’ll have to work your way up and develop a reputation because people don’t know you. People are making this free choice. They are like “where should I assign my balance?”. I guess the easy answer will go back to fees. The people who have reputations, they can start to take advantage of that by increasing their fees. The people who have no reputation are forced to take a lower fee. People then will make a choice. Should I go for the super steady, super reliable payments from the nodes that have been around forever? Or should I go for a little more shaky nodes that have just been around for a month or so? I’ll give it a shot because he’s charging basically no fees, it may be negative fees.

Predictions

Another important thing to think about with these channels is that what you’re basically doing is predicting the future. That’s what people are doing when they’re routing. They’re predicting what’s a path that’s going to succeed in the future. I don’t know. You are also doing the same thing. What’s a channel that’s going to be useful in the future? One easy tactic as a routing node is to think about doing it yourself. Think about the future. This is actually one element that will be difficult for an autopilot to ever do. Let’s say I hear Bitrefill just created a new node because their old node is becoming unstable. I think that Bitrefill is going to get lots of payments through because I know Bitrefill is pretty cool. I’ll assign a bunch of capacity to Bitrefill and I’ll get rewarded with fees. This actually creates an interesting overall concept for the network where the information that helps you predict is also directly monetizable. You could actually pay somebody to tell you ahead of time that they’re going to be creating a new Lightning service because you can start to assign capacity in that direction and develop yourself as a good router to that destination. You could see that as a negative but that’s a way to incentivize people so that once a new service launches that there’s tonnes of capacity right away to send to it. The other thing that you should be predicting is you should be using that reputation to predict where you’ll get into bad situations. One bad situation is the node will go offline and he’ll log up your capital and it’ll be unusable because you need his cooperation in order to route through the network. It will start to make you look bad. That node is connected to lots of offline people so the chances are he’ll be connected to more offline people. You should try to be predicting which of your peers are going to be reliable peers. They’re going to be sticking around. You even have another incentive which is if the peer goes offline and you’ve created a channel to this peer and you have to force close. The force close cost will be much higher than if you can do a cooperative close with that peer. It’s better if you think that this node might be going vacation for a long time or start to be unstable that you cooperatively close with them before you’re forced to pay 10x higher chain fees.

Termination

I would still say that terminating channels is the ultimate punishment that you can do to a peer. There’s some softer measures you can take to warn them off or warn the network at large that these channels are not the best channels. One thing we’ve added that people don’t really see is this idea of the disable flag. On my node I can say to another node or to the graph “I’m not going to send you this guy.” Right now we’ve started to do this if the peer goes offline, we tell the rest of the network “my peer is offline so don’t send in this direction.” Obviously he’s offline so he can’t tell other people to send the other way. It’s also starting to build up something we can look at to develop a presence sense of the network. We can see who’s online and who’s offline. We can do the same thing with fees as well. Disable is more of the policy side of things. People are not implementing this disable themselves. lnd and c-lightning are doing this behind the scenes invisibly. That’s probably why you don’t know that it is happening. We can start to make expectations about what implementations are doing automatically. Another way we can softly signal to the network that this channel is not a reliable channel is that we can start to increase fees on this channel. We can tell the pathfinding algorithms that you should probably avoid that channel because that channel is a risky choice to choose to send along. The other side can also see that you’re sending these policies so they can try to make judgements in the future. “Ok he’s increased fees on me because I went offline for two hours so I’ll try to do better about staying online. Or I’ll try to do better at fulfilling other metrics to increase success”.

Q - Do you think in the future… message sending.. unreliable node…

A - Yeah I’d say the messages are part of the graph so we can look at the channels, we can look at how they’re closing. We can look at lots of different signals and we’re already sending tonnes of signals out there about who’s reliable. Let’s say I notice on the graph that everybody has disabled their channels to this one node. Why have all of his peers disabled him? It’s probably because he’s not that reliable or he’s offline or something. We can just look at those types of signals that other people are saying about them.

Who should route?

So who should be the router? I think people who have existing needs for nodes, they are great routers. They are already paying that cost so they can do a better job of competing because they have that incentive already. It is kind of like if you have free electricity maybe you should mine with it. People who are good at markets. Somebody who is already good at trading. Capacity itself is a good. Capacity itself is something you can swap for. If you’re a great trader, you can predict the future then you should be a router. And people who already have to sit on a lot of Bitcoin. That’s one thing that’s implied here is that you are assigning capacity, you’re sitting on Bitcoin. If you already have to pay that cost, you are a holder, then this is a free operation to you.

Tradeoffs

What are the negative aspects? I’ll leave with that. Why wouldn’t you want to be a router. You sacrifice some privacy. You’re putting up a big banner next to your UTXOs. This is Yalls.org. We’re trying to improve privacy but we also have some things that are tradeoffs for privacy. You now have exposed yourself to certain types of attacks. Let’s say that there was that inflation attack in Bitcoin Core. It happened, maybe it happened in a limited way where people accepted it a little bit. You could start to accept inflated coins and then send out good coins. You could start to lose money. You expose yourself to all these weird classes of attacks that if you had just kept your money in a cold wallet you wouldn’t have had to worry about. You do have to sit on timelocked funds. Let’s say you’re accepting private channels. Those private channels are going to be offline a lot of the time. You might have to think about having some coins you can’t directly access at any moment. That’s a negative of being a router. If you can’t afford that I wouldn’t be a router.

Other Costs

There’s other costs to think about. Channels are not a free operation. If you open a channel you pay the fees. In the future whoever closes it will pay the fees. There will be fees there. Right now fees are low so people aren’t really thinking about this. If fees start to get to $1 or $10 you might really be upset that somebody paid me for Y’alls for 5 cents and they closed the channel and they left me with something I have no hope of claiming. As a routing node you also have this same thing. You’ve got some routing fees but you really can’t access them. You have standard hot wallet risks. At lnd we’re trying to reduce that but that’s a real problem. Somebody gets access to your machine and maybe they can take all your money. Also I think this is kind of a distracting thing. I spend way more time trying to optimize my routing than it actually is justified in terms of fees that I’m generating. I’m watching the routing all the time, I’m balancing my channels all the time. It’s just a fun thing to do, it’s really interesting. This money isn’t coming from nowhere, somebody is paying this. It’s not boundless money. That also means it will be limited money. There’s so much competition right now. Everybody is setting their fees at zero. We need to do a lot of work to build up this ecosystem before you really have a good justification to spend a lot of your time improving your channels and becoming an amazing router.

Q&A

Q - Regarding rebalancing. What are you using to… channels… capacity one-to-one… Is that a manual process?

A - It has actually been pretty organic. People have opened channels to me and I’ve opened channels outside. I kind of take a look. There’s other reasons why I wouldn’t want to add too much to a channel. I don’t want to put too much at risk. The more you’re adding to a channel, the greater risk you’re potentially taking on with a breach attack. lnd will reflect this in the CSV time. That creates more cost for you. If you create a huge max size channel, lnd will force you to wait under a two week timelock to get that money back because there’s such a big risk with $1500 on the line. If you make smaller, more medium size channels you might only have to wait a day or a few days.