Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id DE991CCC for ; Tue, 26 Jun 2018 15:33:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.sldev.cz (mail.sldev.cz [88.208.115.66]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id EC37A17E for ; Tue, 26 Jun 2018 15:33:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.sldev.cz (Postfix) with ESMTP id D247FE1046; Tue, 26 Jun 2018 15:33:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.sldev.cz Received: from mail.sldev.cz ([127.0.0.1]) by localhost (mail.sldev.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OyN6STrVwhTC; Tue, 26 Jun 2018 15:33:15 +0000 (UTC) Received: from [10.8.0.37] (unknown [10.8.0.37]) by mail.sldev.cz (Postfix) with ESMTPSA id 22E3AE1040; Tue, 26 Jun 2018 15:33:15 +0000 (UTC) To: bitcoin-dev@lists.linuxfoundation.org References: <21a616f5-7a17-35b9-85ea-f779f20a6a2d@satoshilabs.com> <20180621195654.GC99379@coinkite.com> From: matejcik Openpgp: preference=signencrypt Autocrypt: addr=jan.matejek@satoshilabs.com; prefer-encrypt=mutual; keydata= xsFNBFqFmMgBEADPJ8NULpuu0nwox/tIfo+slGfcXZLUEZstNoaY9QgNuILJRtoJ6xZy8rQf S7iQlkaZcrpMJYdZtkRHvndkceBxesCG8io6tsU+t2SK6AvaW0FG95a9shFM/U9/JVO/QmBi IuQzbiE2XTZ/JStyEp4zpuyJqX1o9gzS/4MBXwj7Rzk8u+fHI28h96HILC2a0mC+c2gJ7f5t o/w+vxFZmk06COK08W5+odb9I8mjs0uf7jgTUEFrfwi6oCoTFmSon7cOy/WTieClwF/vUKuJ DBAtsMh2rxh8IHyH8xpR+Ay/K6jUWqeb3P2csQqMXmquYG/qdaHjQgxyuoJFbn+nT6jNGVQZ MjpZkMrGnjLccecaXlgx/rZK6ElCZ1PDHKOTW7A1YY1/eG7TWYnVv1ehQLueAoqyyfiEutsK E5jGbR0AmNjCahpeK7dxj+8g8TXpVsH207xJ+mqOm5RYqlX4OzfVvcnoHhlRIOu85i4I9rWm 1u/pP6uJFnBCKtuhhbmXCxM6wF7W5U6EVW3yymsPmSoVoaR024vffE3L5jZSsDMRxY6fDXNm ljRnOpT3l3d+kMVdAM3CdDCgmV87fdo4PAaGDfnmufGue/Gp0RiLCe/Wsm4DgIIa5UK6DmzD q0B6i9y/GJSPUChzZ8y7fYzuyXdpk/13gV2NRsskg9oXJVd1vQARAQABzSZtYXRlamNpayA8 amFuLm1hdGVqZWtAc2F0b3NoaWxhYnMuY29tPsLBfQQTAQgAJwUCWoWYyAIbIwUJCWYBgAUL CQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRDGf7EG5O0XHoU0D/4+fTbt4KELEtnpkirDH4mQ Vt3KtKJrI/gp/3u+r6jUWMv2V9iRFMs09GAVBmE2DkXXIlfaT1P0QfwVSpHC4k5lwKwSCSyS MUgBbQGPOiYMCgMQ+in4vjlqWWcx6jjlgxQctQHRrVG5jyi7BSb0jwG8rcYtx8SAYkN4joG/ oy2zMbq6qu+Vsl+xR5WwWF2mcUUyiVo7dSwNy+1PaeygOR9xAWkM8J42ckLfJgvyLSviBKnU 9rgg94ryEDAMNUL5yJUygQmUM/jdpyBpBycRbWMB+zIYDPVGnFj4vN8Hs9DyGUHVb2OqSW+q VPxD7U9m9z6J3NnY9HpaFX1DD8leK3TebpyYaeODY5jyk7retuLrMq+W4kJU0290xzlWa9sU wa7lTWw63pelfPUKZ+mjhSFQSZBqiuNv67CBd/UmoqMWSDrCWj+3IFQxReFbh47Wl4MUX2cK cLocYkBzDck7hH4YfK6jJ++teN6RKXr7P1y6EI25WEfJxWK9say7x/FRkNW0s98MxtOuwEsm /vHqHQQanAT4R5l+Rr7XfU7fpmH0As98qD81lc3RHbrxEXgA0ks2VuCxBWsPpzaHUFPOcE9H hsg1jSEDi/Mo6D4e2ap7FYXDgZiKye9WnSdPlVBqJxqinDDgSBv5wzKaEGQS0MKrF9myS7d0 pBSy1Dr6IWOegM7BTQRahZjIARAAwwT6h4IFvs/hmY9KHiX/GIbvybQUU71ZWYRE2KKo5E2c ZXBJj7SiDtU80bS+NCSeF2c0i4xOYgZlIYMqlgS8k1zfdBt/JHmG3tm1JgohVj+pm42RfBAF d0y05zz5wysQOw1M4WlWKZH0ameM+0/AGqspeZushWay8Q4yx1dO/6MeyPy/NwE/MKEsCOPV aN28DndN3iKOyriCQt/IhG/n6ORPRGyei3JYqxsnpW36BOmSPWJ7Qj2pFw53p5coPOEDL8mN Ique0LJZ3zVFVMa4i7HtqIEnYO+ZnKx2G8aLsHEir2pzBv6tMwlgETcUTVfK1ePN7OzhYy4q a38hMWzk0db2V+gOlAu6SuAi1ANkcPhCPUWxPIvXiNdd9iwe5gOzFy0FoZxj22rFwgUX8wcc cfWStgoE1MGE9G5zrqc01R0x7by8BOFkImAwTyJ9vq4jG+w7Npky3PhoHPgCT5knV7Q91U2I TqPOQBcMda0B+4LOaElb1sXqe44dHwcg4dMVngaea5xL7winSqU2Gtm6pqFAGut5F7JiYhPb dGUHJPMS67ONkKe5ARu/Z/r9XoFe2TxpkvNJ/+QJQ3PCiJ6ya31ij6HOIfFbZr3xlTyU/DvG SejIvDK/SnJMw+/x60bYAshYBp0uQgih1ugtoZh7cnKj3KfhlpXT0mL8rsl1QHsAEQEAAcLB ZQQYAQgADwUCWoWYyAIbDAUJCWYBgAAKCRDGf7EG5O0XHs2xD/92sa5L6gafP/rRKfo9u3/w s+7E/kKPgG4VGDeirLo8hbinCjPr0cfZ7OgDDvp0zy6lTdZc2tcHsEbiPqblzaSZimV5Y3EQ eIzz0UhY6YdDELr8pvdnB8qnOJHXgWmZTRYkRgxFOWI3v4STmOYZQ7MFv0kHBfV3htCjYTHS Qx2jQO4CTbcSEbkVwNv56OiZroabrHRf0WUSyzElf13P/MRFjUJFYYZDqc0iOWUh4QeXbFiY fLYpOCtm0nqaDdG1VD4jMpKq1FKBvTw4id1i7pONENd4BB7ytnDvKGdVI6oDnGUBsc5VUrEa h1PbbshNMbRtFigeMe8998jWhK4jQzeuDr0FSBlhxbluGfyMUgk7s6aBC9BOsdDkgtJk1Fd/ j9sWOj8Pxzc4lMQRfygm+QxxLdqa36Qh3oK+jfK7362CXlqBfb9ryerjfFGY4VqMBzQ+BFtj lYZSdVzGWlmLD9D88wzeByIZMScQPvrXSFwPO2/TuOQNCo0VHcgHpNFzeMRK2eT8bhry+dlq U+0Kxy2gQijw9j/EZlqR3w053EwUrfAAmHHeYPimXK4pc8oSw0s1A6hQO7Vc0SgblF8taFTM UhRR7xZg+l5vybAgrDYVL75b9CDscZqd7WVmZx+xU23sUG6SaxXI7PV6bPuMug0fD3SAsieu +vypQ3jCcUKGrA== Message-ID: Date: Tue, 26 Jun 2018 17:33:14 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP" X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Tue, 26 Jun 2018 15:36:54 +0000 Subject: Re: [bitcoin-dev] BIP 174 thoughts X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2018 15:33:20 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP Content-Type: multipart/mixed; boundary="HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm"; protected-headers="v1" From: matejcik To: bitcoin-dev@lists.linuxfoundation.org Cc: tomas.susanka@satoshilabs.com Message-ID: Subject: Re: [bitcoin-dev] BIP 174 thoughts References: <21a616f5-7a17-35b9-85ea-f779f20a6a2d@satoshilabs.com> <20180621195654.GC99379@coinkite.com> In-Reply-To: --HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable hello, in general, I agree with my colleague Tomas, the proposed changes are good and achieve the most important things that we wanted. We'll review the proposal in more detail later. For now a couple minor things I have run into: - valid test vector 2 ("one P2PKH input and one P2SH-P2WPKH input") seems broken, at least its hex version; a delimiter seems to be missing, misplaced or corrupted - at least the first signing vector is not updated, but you probably know that - BIP32 derivation fields don't specify size of the "master public key", which would make it un-parsable :) - "Transaction format is specified as follows" and its table need to be updated. I'm still going to argue against the key-value model though. It's true that this is not significant in terms of space. But I'm more concerned about human readability, i.e., confusing future implementers. At this point, the key-value model is there "for historical reasons", except these aren't valid even before finalizing the format. The original rationale for using key-values seems to be gone (no key-based lookups are necessary). As for combining and deduplication, whether key data is present or not is now purely a stand-in for a "repeatable" flag. We could just as easily say, e.g., that the high bit of "type" specifies whether this record can be repeated. (Moreover, as I wrote previously, the Combiner seems like a weirdly placed role. I still don't see its significance and why is it important to correctly combine PSBTs by agents that don't understand them. If you have a usecase in mind, please explain. ISTM a Combiner could just as well combine based on whole-record uniqueness, and leave the duplicate detection to the Finalizer. In case the incoming PSBTs have incompatible unique fields, the Combiner would have to fail anyway, so the Finalizer might as well do it. Perhaps it would be good to leave out the Combiner role entirely?) There's two remaining types where key data is used: BIP32 derivations and partial signatures. In case of BIP32 derivation, the key data is redundant ( pubkey =3D derive(value) ), so I'd argue we should leave that= out and save space. In case of partial signatures, it's simple enough to make the pubkey part of the value. Re breaking change, we are proposing breaking changes anyway, existing code *will* need to be touched, and given that this is a hand-parsed format, changing `parse_keyvalue` to `parse_record` seems like a small matter? --- At this point I'm obliged to again argue for using protobuf. Thing is: BIP174 *is basically protobuf* (v2) as it stands. If I'm succesful in convincing you to switch to a record set model, it's going to be "protobuf with different varint". I mean this very seriously: (the relevant subset of) protobuf is a set of records in the following format: Record types can repeat, the schema specifies whether a field is repeatable or not - if it's not, the last parsed value is used. BIP174 is a ad-hoc format, simple to parse by hand; but that results in _having to_ parse it by hand. In contrast, protobuf has a huge collection of implementations that will do the job of sorting record types into relevant struct fields, proper delimiting of records, etc. =2E..while at the same time, implementing "protobuf-based-BIP174" by hand= is roughly equally difficult as implementing the current BIP174. N.B., it's possible to write a parser for protobuf-BIP174 without needing a general protobuf library. Protobuf formats are designed with forwards- and backwards- compatibility in mind, so having a hand-written implementation should not lead to incompatibilities. I did an experiment with this and other variants of the BIP174 format. You can see them here: [1] https://github.com/matejcik/bip174-playground see in particular: [2] https://github.com/matejcik/bip174-playground/blob/master/bip174.prot= o The tool at [1] does size comparisons. On the test vectors, protobuf is slightly smaller than key-value, and roughly equal to record-set, losing out a little when BIP32 fields are used. (I'm also leaving out key-data for BIP32 fields.) There's some technical points to consider about protobuf, too: - I decided to structure the message as a single "PSBT" type, where "InputType" and "OutputType" are repeatable embedded fields. This seemed better in terms of extensibility - we could add more sections in the future. But the drawback is that you need to know the size of Input/OutputType record in advance. The other option is sending the messages separated by NUL bytes, same as now, in which case you don't need to specify size. - in InputType, i'm using "uint32" for sighash. This type is not length-delimited, so non-protobuf consumers would have to understand it specially. We could also declare that all fields must be length-delimited[1] and implement sighash as a separate message type with one field. - non-protobuf consumers will also need to understand both protobuf varint and bitcoin compact uint, which is a little ugly best regards matejcik [1] https://developers.google.com/protocol-buffers/docs/encoding#structur= e --HN3AdUEsOchTlvdQeDk1FNoAuphj1qawm-- --74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJbMly6AAoJEMZ/sQbk7RcePQoQAIRHSplzTDWvd8xmeOn7iUKc YJhXQ9bZaKXa9Gq51QFXfzL0Q2IdSeMwcBYxqxX73dQNy4keWZ/JHCu2gz8ULpcD KF9bfzKM4yq/vAcc+xwCJPtJADa5na5XrG3jcqR5YpfjaH3YTpmzKchFnGNShu61 aco8VygvOwaT6NKuJQ37q3XbWrNga1EBSc8KEfob1xbzxoQ1v4Y8obZ4PzUMmtyn 6KXAcOSGTJLJRHqquUhp7Cw6l9zwKa+ueJ1vSA7EHwxoj+RncAjUBcKskJzW2Aa1 IAcrZK/zn2vc69Y9pkotiNvvYDL1Vr+R4OqCxLaV/eJN6cr1zfRjvtyxVAm36aFC DHageXHofqI3XcD1o8C2UsbYPGJJZxSdUMo0thPDN59SLMoxJmyYSirO5Eam50eC rbHk21ToF8NsRNtFzGX39h/HQ40pLpqQoaLwXyn/ReWbgDBKFwTTu5OTy8BHMjGN iScIvVpoMBS/JHbals39e05sVZUzfjpSJdCN52gSPu/lrW7I/ku4OMeiKp+fZy1u qVPvxJ9pKLq6rcUvc54S5XEnpzDSBGrLzSZxx/VhkAAyBgsKnvac/YZPVtC2sYrT 4pp+AyLSmp392qA8aM3MBeEFzrMctri7PhBDNcXx76WZkyztlyPFw4uc6ZV4c6gZ 1D0fNidkTWcQh49Ih3C0 =bzDw -----END PGP SIGNATURE----- --74aBNEZiIFs9NuoqOguzkKgVgn9KqMPKP--