Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1X72Sw-0005PY-5w for bitcoin-development@lists.sourceforge.net; Tue, 15 Jul 2014 13:08:18 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.219.41 as permitted sender) client-ip=209.85.219.41; envelope-from=enwiner@gmail.com; helo=mail-oa0-f41.google.com; Received: from mail-oa0-f41.google.com ([209.85.219.41]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1X72Su-0005Tc-Cp for bitcoin-development@lists.sourceforge.net; Tue, 15 Jul 2014 13:08:18 +0000 Received: by mail-oa0-f41.google.com with SMTP id j17so3043195oag.28 for ; Tue, 15 Jul 2014 06:08:07 -0700 (PDT) X-Received: by 10.182.121.170 with SMTP id ll10mr26000848obb.58.1405429687819; Tue, 15 Jul 2014 06:08:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.60.116.105 with HTTP; Tue, 15 Jul 2014 06:07:47 -0700 (PDT) In-Reply-To: References: From: Eric Winer Date: Tue, 15 Jul 2014 09:07:47 -0400 Message-ID: To: Mike Hearn Content-Type: multipart/alternative; boundary=089e01536a5a4ca13704fe3b1907 X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (enwiner[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1X72Su-0005Tc-Cp Cc: Bitcoin Dev , Andreas Schildbach Subject: Re: [Bitcoin-development] BIP 38 NFC normalisation issue X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jul 2014 13:08:18 -0000 --089e01536a5a4ca13704fe3b1907 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I don't know for sure if the test vector is correct NFC form. But for what it's worth, the Pile of Poo character is pretty easily accessible on the iPhone and Android keyboards, and in this string it's already in NFC form (f09f92a9 in the test result). I've certainly seen it in usernames around the internet, and wouldn't be surprised to see it in passphrases entered on smartphones, especially if the author of a BIP38-compatible app includes a (possibly ill-advised) suggestion to have your passphrase "include special characters". I haven't seen the NULL character on any smartphone keyboards, though - I assume the iOS and Android developers had the foresight to know how much havoc that would wreak on systems assuming null-terminated strings. It seems unlikely that NULL would be in a real-world passphrase entered by a sane user. On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn wrote: > [+cc aaron] > > We recently added an implementation of BIP 38 (password protected private > keys) to bitcoinj. It came to my attention that the third test vector may > be broken. It gives a hex version of what the NFC normalised version of t= he > input string should be, but this does not match the results of the Java > unicode normaliser, and in fact I can't even get Python to print the name= s > of the characters past the embedded null. I'm curious where this normalis= ed > version came from. > > Given that "pile of poo" is not a character I think any sane user would > put into a passphrase, I question the value of this test vector. NFC form > is intended to collapse things like umlaut control characters onto their > prior code point, but here we're feeding the algorithm what is basically > garbage so I'm not totally surprised that different implementations appea= r > to disagree on the outcome. > > Proposed action: we remove this test vector as it does not represent any > real world usage of the spec, or if we desperately need to verify NFC > normalisation I suggest using a different, more realistic test string, li= ke > Z=C3=BCrich, or something written in Thai. > > > > Test 3: > > - Passphrase =CF=92=CC=81=E2=90=80=F0=90=90=80=F0=9F=92=A9 (\u03D2\u03= 01\u0000\U00010400\U0001F4A9; GREEK > UPSILON WITH HOOK , COMBINING ACUTE > ACCENT , NULL > , DESERET CAPITAL LETTER LONG I > , PILE OF POO > ) > - Encrypted key: > 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn > - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF > - Unencrypted private key (WIF): > 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 > - *Note:* The non-standard UTF-8 characters in this passphrase should > be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9= before > further processing > > > > > > -------------------------------------------------------------------------= ----- > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > --089e01536a5a4ca13704fe3b1907 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I don't know for sure if the test vector is corre= ct NFC form. =C2=A0But for what it's worth, the Pile of Poo character i= s pretty easily accessible on the iPhone and Android keyboards, and in this= string it's already in NFC form (f09f92a9 in the test result). =C2=A0I= 've certainly seen it in usernames around the internet, and wouldn'= t be surprised to see it in passphrases entered on smartphones, especially = if the author of a BIP38-compatible app includes a (possibly ill-advised) s= uggestion to have your passphrase "include special characters".

I haven't seen the NULL character on any smar= tphone keyboards, though - I assume the iOS and Android developers had the = foresight to know how much havoc that would wreak on systems assuming null-= terminated strings. =C2=A0It seems unlikely that NULL would be in a real-wo= rld passphrase entered by a sane user.


On Tue,= Jul 15, 2014 at 8:03 AM, Mike Hearn <mike@plan99.net> wrote:<= br>
[+cc aaron]

We recently added an implem= entation of BIP 38 (password protected private keys) to bitcoinj. It came t= o my attention that the third test vector may be broken. It gives a hex ver= sion of what the NFC normalised version of the input string should be, but = this does not match the results of the Java unicode normaliser, and in fact= I can't even get Python to print the names of the characters past the = embedded null. I'm curious where this normalised version came from.
Given that "pile of poo" is not a character I thin= k any sane user would put into a passphrase, I question the value of this t= est vector. NFC form is intended to collapse things like umlaut control cha= racters onto their prior code point, but here we're feeding the algorit= hm what is basically garbage so I'm not totally surprised that differen= t implementations appear to disagree on the outcome.

Proposed action: we remove this test vector as it does = not represent any real world usage of the spec, or if we desperately need t= o verify NFC normalisation I suggest using a different, more realistic test= string, like Z=C3=BCrich, or something written in Thai.



Test 3:
  • Passphrase =CF=92=CC=81=E2=90=80=F0=90=90=80=F0=9F=92=A9 (\u03D2= \u0301\u0000\U00010400\U0001F4A9;=C2=A0GREEK UPSILON WITH HOOK,=C2=A0COMBINING ACUTE ACCENT,= =C2=A0NULL,= =C2=A0DESERET = CAPITAL LETTER LONG I,=C2=A0PILE OF POO)
  • Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcD= Qn
  • Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
  • Unen= crypted private key (WIF): 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTM= SZ4
  • Note:=C2=A0The non-standard UTF-8 characters in this passphrase = should be NFC normalized to result in a passphrase of0xcf9300f090908= 0f09f92a9=C2=A0before further processing


<= /div>

-----------------------------------------------------------------------= -------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/b= ds
_______________________________________________
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment


--089e01536a5a4ca13704fe3b1907--