From rusty at rustcorp.com.au Fri Nov 16 00:25:34 2018 From: rusty at rustcorp.com.au (Rusty Russell) Date: Fri, 16 Nov 2018 10:55:34 +1030 Subject: [Lightning-dev] type,len,value standard In-Reply-To: References: Message-ID: <87r2flamwx.fsf@rustcorp.com.au> Conner Fromknecht writes: > Hi ZmnSCPxj, > > Thanks for writing this up! I had started an email, but you beat me to it :) > >> 1. For a sequence of `type,len,value`, each `type` must be unique. -- >> accepted. > > To add to this, it seemed that there was some agreement that repeated fields > should be serialized under a single root key, since a receiver can't know if a > field is allowed to have duplicates if they don't understand the field. If a receiver doesn't understand it, it doesn't have to enforce any rules on it *at all*. So it depends on individual cases. >> For a sequence of `type,len,value`, the `type`s must be in ascending order >> -- not explicitly accepted or rejected. It would be easier to check >> uniqueness > (the previous rule we accepted) here for a naive parser (keep >> track of some "minimum allowed type" that initializes at zero, check current >> type >= this, update to current type + 1) if `type`s are in ascending order. > > Yep ascending makes sense to me, for the reasons you stated. Again, writer MUST, with no constraint on reader. >> 1, `type` - one byte or two? > > I'd lean towards one, if a message has 256 optional fields, it might be time to > consider a new message type altogether. Exactly. >> 3. `type` - does "it's OK to be odd" apply? i.e. if an even `type` that is >> not known is found, crash and burn. But intent of this system is for future >> expansion for optional fields, so...? > > Perhaps this depends on context: > - for gossip messages, I think the primary concern is not breaking signature > validation, and that these would need to remain optional for backwards > compatibility. Yes, "not OK" is message dependent. It doesn't break parsing (if you want that, use a new message type), but it does break *usage*; "you can't use this information". TLV makes sense when there are many many options, at cost of a couple of bytes per option. We agreed to consider it for new additions, but sometimes a simple option bitmap will be a win. > - for link-level messages, we have a little more control. I imagined the fields > would be gated by feature bit negotiation, and deviating from > unsupported/required would result in being disconnected. It's nice if you can simplify your code by always sending it, even if the remote doesn't understand it. >> 5. `len` - one byte or two? I believe we tend to use two bytes for various >> lengths. > > Maybe varint? One byte is not enough for all lengths, but two seems excessive > for uint8 or even uint32. Yes, it's a nice compromise IMHO. >> 6. BOLT - I propose making a separate BOLT for `type,len,value`, which other >> messages and so on simply refer to. > > Indeed, are you thinking we'd use this to add new fields proposed in 1.1? Well, it's a very individual message thing. We would specify it in general, then each place where it's used (eg. gossip) would have their own types. > In addition to the above, do we also want to flesh out what sub-TLV structures > would look like? Or perhaps that isn't necessary, if we can continue adding more > root-level keys. I think that the general TLV definition just belongs in BOLT #1, since it's so short. Cheers, Rusty.