1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
helo=mx.sourceforge.net)
by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <mh.in.england@gmail.com>) id 1Y5Un1-0004Ia-0g
for bitcoin-development@lists.sourceforge.net;
Mon, 29 Dec 2014 07:30:55 +0000
Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com
designates 74.125.82.50 as permitted sender)
client-ip=74.125.82.50; envelope-from=mh.in.england@gmail.com;
helo=mail-wg0-f50.google.com;
Received: from mail-wg0-f50.google.com ([74.125.82.50])
by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1Y5Umw-0003CU-9G
for bitcoin-development@lists.sourceforge.net;
Mon, 29 Dec 2014 07:30:54 +0000
Received: by mail-wg0-f50.google.com with SMTP id a1so18410101wgh.9
for <bitcoin-development@lists.sourceforge.net>;
Sun, 28 Dec 2014 23:30:44 -0800 (PST)
MIME-Version: 1.0
X-Received: by 10.180.82.98 with SMTP id h2mr3982906wiy.7.1419791129297; Sun,
28 Dec 2014 10:25:29 -0800 (PST)
Sender: mh.in.england@gmail.com
Received: by 10.194.188.9 with HTTP; Sun, 28 Dec 2014 10:25:29 -0800 (PST)
Date: Sun, 28 Dec 2014 18:25:29 +0000
X-Google-Sender-Auth: 4jwgr-WMWDSQ9mQpRBoLHhJ8Y8Y
Message-ID: <CANEZrP1tGYOsShk5Z91_JwH7E0pX8WcyT_-ZBkrWBqZAyrcd1A@mail.gmail.com>
From: Mike Hearn <mike@plan99.net>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=f46d044283eceacc1c050b4ae114
X-Spam-Score: 0.3 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(mh.in.england[at]gmail.com)
0.8 DATE_IN_PAST_12_24 Date: is 12 to 24 hours before Received: date
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 WEIRD_PORT URI: Uses non-standard port number for HTTP
1.0 HTML_MESSAGE BODY: HTML included in message
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1Y5Umw-0003CU-9G
Subject: [Bitcoin-development] Cartographer
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Mon, 29 Dec 2014 07:30:55 -0000
--f46d044283eceacc1c050b4ae114
Content-Type: text/plain; charset=UTF-8
Hi there!
Lately we have been bumping up against the limitations of DNS as a protocol
for learning about the p2p network. As a proposal for how to address this,
I have written a new network crawler and seed:
https://github.com/mikehearn/httpseed
It implements a standard DNS seed with a minimal embedded DNS server (you
can find one running at dnsseed.vinumeris.com) and also has the following
extra features:
- Can serve seed data using gzipped, timestamped digitally signed
protocol buffers over HTTP. This fixes authentication, auditability,
malware false positives and extensibility. The signature uses secp256k1.
SSL is *not* used, to simplify deployment and to allow ISPs to cache the
results transparently when a future version sets cache control headers.
- Can additionally serve data in JSON, XML and HTML (examples for json
<http://vinumeris.com:8081/peers.json> xml
<http://vinumeris.com:8081/peers.xml> html
<http://vinumeris.com:8081/peers.html>) for ease of use with other
tools, like web browsers.
- Results can be restricted using query parameters, e.g. for a service
flags bit mask. Cartographer tests nodes that set service bit 2 to see if
they really support BIP 64, and this requirement can also be specified as
an argument to the query.
- Crawl speed can be specified in terms of successful connects per
second, rather than the number-of-threads approach used by other crawlers.
- Can export statistics and controls using JMX, so you can reconfigure
it at runtime and view charts of things like connects/sec or CPU usage
using any JMX console, like Mission Control.
- A client for it is in bitcoinj master branch.
To provide all these features Cartographer relies heavily on libraries and
is written in a concise new language called Kotlin <http://kotlinlang.org/>,
so it fits in about 650 lines of code. Kotlin is easy to learn for anyone
who knows Scala or Java, so it should be straightforward to hack on and
there is no chance of any buffer/heap exploits in the DNS, HTTP or Bitcoin
protocol stacks.
In the new year I will probably write a BIP describing the protocol. For
now you can see the definition here
<https://github.com/mikehearn/httpseed/blob/master/src/main/peerseeds.proto>
or just read the textual forms from the links above. It's pretty self
explanatory. I hope that in future other DNS seeds will start supporting
this protocol too, as it has many advantages.
Future versions might include data like how long the peer has been around,
node keys if we add auth/encrypt support to the p2p protocol and so on.
--f46d044283eceacc1c050b4ae114
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi there!<div><br></div><div>Lately we have been bumping u=
p against the limitations of DNS as a protocol for learning about the p2p n=
etwork. As a proposal for how to address this, I have written a new network=
crawler and seed:</div><div><br></div><div><a href=3D"https://github.com/m=
ikehearn/httpseed">https://github.com/mikehearn/httpseed</a><br></div><div>=
<br></div><div>It implements a standard DNS seed with a minimal embedded DN=
S server (you can find one running at <a href=3D"http://dnsseed.vinumeris.c=
om">dnsseed.vinumeris.com</a>) and also has the following extra features:</=
div><div><br></div><div><ul class=3D"" style=3D"padding:0px 0px 0px 2em;mar=
gin-top:0px;margin-bottom:16px;color:rgb(51,51,51);font-family:'Helveti=
ca Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif;line-he=
ight:20.4799995422363px"><li>Can serve seed data using gzipped, timestamped=
digitally signed protocol buffers over HTTP. This fixes authentication, au=
ditability, malware false positives and extensibility. The signature uses s=
ecp256k1. SSL is <b>not</b>=C2=A0used, to simplify deployment and to allow =
ISPs to cache the results transparently when a future version sets cache co=
ntrol headers.</li><li>Can additionally serve data in JSON, XML and HTML (e=
xamples for <a href=3D"http://vinumeris.com:8081/peers.json">json</a> <a hr=
ef=3D"http://vinumeris.com:8081/peers.xml">xml</a> <a href=3D"http://vinume=
ris.com:8081/peers.html">html</a>)=C2=A0for ease of use with other tools, l=
ike web browsers.</li><li>Results can be restricted using query parameters,=
e.g. for a service flags bit mask. Cartographer tests nodes that set servi=
ce bit 2 to see if they really support BIP 64, and this requirement can als=
o be specified as an argument to the query.</li><li><span style=3D"line-hei=
ght:20.4799995422363px">Crawl speed can be specified in terms of successful=
connects per second, rather than the number-of-threads approach used by ot=
her crawlers.</span><br></li><li>Can export statistics and controls using J=
MX, so you can reconfigure it at runtime and view charts of things like con=
nects/sec or CPU usage using any JMX console, like Mission Control.</li><li=
>A client for it is in bitcoinj master branch.</li></ul><div><font color=3D=
"#333333" face=3D"Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, san=
s-serif"><span style=3D"line-height:20.4799995422363px">To provide all thes=
e features Cartographer relies heavily on libraries and is written in a con=
cise new language called <a href=3D"http://kotlinlang.org/">Kotlin</a>, so =
it fits in about 650 lines of code. Kotlin is easy to learn for anyone who =
knows Scala or Java, so it should be straightforward to hack on and there i=
s no chance of any buffer/heap exploits in the DNS, HTTP or Bitcoin protoco=
l stacks.</span></font></div></div><div><br></div><div>In the new year I wi=
ll probably write a BIP describing the protocol. For now you can see the de=
finition <a href=3D"https://github.com/mikehearn/httpseed/blob/master/src/m=
ain/peerseeds.proto">here</a> or just read the textual forms from the links=
above. It's pretty self explanatory. I hope that in future other DNS s=
eeds will start supporting this protocol too, as it has many advantages.</d=
iv><div><br></div><div>Future versions might include data like how long the=
peer has been around, node keys if we add auth/encrypt support to the p2p =
protocol and so on.</div></div>
--f46d044283eceacc1c050b4ae114--
|
