From fabrice.drouin at acinq.fr Mon Feb 5 15:08:22 2018 From: fabrice.drouin at acinq.fr (Fabrice Drouin) Date: Mon, 5 Feb 2018 16:08:22 +0100 Subject: [Lightning-dev] Improving the initial gossip sync In-Reply-To: <874lmvy4gh.fsf@gmail.com> References: <874lmvy4gh.fsf@gmail.com> Message-ID: Hi, On 5 February 2018 at 14:02, Christian Decker wrote: > Hi everyone > > The feature bit is even, meaning that it is required from the peer, > since we extend the `init` message itself, and a peer that does not > support this feature would be unable to parse any future extensions to > the `init` message. Alternatively we could create a new > `set_gossip_timestamp` message that is only sent if both endpoints > support this proposal, but that could result in duplicate messages being > delivered between the `init` and the `set_gossip_timestamp` message and > it'd require additional messages. We chose the other aproach and propose to use an optional feature > The reason I'm using timestamp and not the blockheight in the short > channel ID is that we already use the timestamp for pruning. In the > blockheight based timestamp we might ignore channels that were created, > then not announced or forgotten, and then later came back and are now > stable. Just to be clear, you propose to use the timestamp of the most recent channel updates to filter the associated channel announcements ? > I hope this rather simple proposal is sufficient to fix the short-term > issues we are facing with the initial sync, while we wait for a real > sync protocol. It is definitely not meant to allow perfect > synchronization of the topology between peers, but then again I don't > believe that is strictly necessary to make the routing successful. > > Please let me know what you think, and I'd love to discuss Pierre's > proposal as well. > > Cheers, > Christian Our idea is to group channel announcements by "buckets", create a filter for each bucket, exchange and use them to filter out channel announcements. We would add a new `use_channel_announcement_filters` optional feature bit (7 for example), and a new `channel_announcement_filters` message. When a node that supports channel announcement filters receives an `init` message with the `use_channel_announcement_filters` bit set, it sends back its channel filters. When a node that supports channel announcement filters receives a`channel_announcement_filters` message, it uses it to filter channel announcements (and, implicitly ,channel updates) before sending them. The filters we have in mind are simple: - Sort announcements by short channel id - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)` (we round to multiples of 144 to make sync easier) - Group channel announcements that were created before this marker by groups of 144 blocks - Group channel announcements that were created after this marker by groups of 1 block - For each group, sort and concatenate all channel announcements short channel ids and hash the result (we could use sha256, or the first 16 bytes of the sha256 hash) The new `channel_announcement_filters` would then be a list of (height, hash) pairs ordered by increasing heights. This implies that implementation can easily sort announcements by short channel id, which should not be very difficult. An additional step could be to send all short channel ids for all groups for which the group hash did not match. Alternatively we could use smarter filters The use case we have in mind is mobile nodes, or more generally nodes which are often offline and need to resync very often. Cheers, Fabrice