From gbildson at limepeer.com Wed Dec 1 00:00:06 2004 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: <20041130232809.5CBB23FCF7@capsicum.zgp.org> Message-ID: David, Yeah, the code has been in beta for a while. It is UDP based. If all UDP is blocked then it won't work and wont be attempted. We do a lot to first ensure that you can receive solicited UDP which is what this feature relies on. i.e. If you send a packet to X via UDP and you are behind a NAT/Firewall, you can receive a response back. In LimeWire terms, you are then said to be firewall capable and your searches and responses indicate this. There are many ways to negotiate the initiation of the connection on both sides. LimeWire has a concept of a push proxy for firewalled hosts so we actually use that to deliver a special PUSH message that tells the host to initiate a UDP connection to ip:port. Both ends then start sending UDP messages at each other and shortly thereafter, they should both be able to receive those messages. A type of TCP style connection negotiation begins from there. Just to be clear, this is not proxying. The only thing that is proxied, is the PUSH message to trigger the actions of the uploader. The FAQ is out of date. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 6:28 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? How does the Firewall-to-Firewall portion of Limewire work? Does it use un-firewalled clients as relay servers? It doesn't sound like it, but I thought that's the only solution that truly works in all situations. The "features history" page mentions this on the entry for 8.12.2004: "Firewall to Firewall transfers allows two people behind firewalls to connect directly to each other and transfer data. This makes use of UDP, and a third party to coordinate the initial messaging. . Normally, firewalled users would only be able to download from other hosts who are not firewalled, which is of course severely limited. With firewall to firewall transfers, firewalled users can now access the full 100% of hosts." This implies something like the NAT-to-NAT trick works with firewalls also. I'm a little shaky on how UDP works with firewalls, do both clients initiate a conversation with a third party, and then the third party hands back information IP/port information of the pre-established out-bound connection? How does this work if the firewall simply blocks all UDP traffic? However, the website is either out of date or there's more to the story because the FAQ says: http://www.limewire.com/english/content/faq.shtml#fir1 "Q: What if I'm behind a firewall? A: LimeWire will work when a user is behind certain types of firewalls, but will not work behind certain other types. If you are behind a firewall, you will not be able to download anything from a user that's also behind a firewall. In general, if you can connect (you will see your "connection status" in the lower left hand corner of the application) using LimeWire, you should be able to download and upload files, but LimeWire will not work if you have either a web-only proxy or a SOCKS proxy." What's the full story? -david ---------------------------------------------------------------------------- -- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 2:12 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? If you believe that there are problems with LimeWire, you should submit them to bugs@limewire and they will be looked into promptly. If you have not already, you should also upgrade to version 4.2.3 to get rid of some potential startup issues with old GWebcaches. LimeWire is a "good new" p2p application - check out that firewall-to-firewall transfer in the new version. ;) Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of Digitalgruvmoves@aol.com Sent: Monday, November 29, 2004 9:10 PM To: p2p-hackers@zgp.org Subject: Re: [p2p-hackers] Why UDP and not TCP? Whats a good new p2p filesharing download to use? Limeware just started acting nuts. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20041130/b079474d/attachment.htm From dbarrett at quinthar.com Wed Dec 1 00:34:42 2004 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: Message-ID: <20041201003449.E28013FCA7@capsicum.zgp.org> That's pretty sweet. Do you know what fraction of firewalls "in the wild" allow for this capability? I'm no firewall nor security expert, but I was under the impression a typical corporate firewall blocks most TCP ports, and virtually (or even) all UDP ports. Do you have any stats on what fraction of firewall users are able to take advantage of this feature? -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 5:00 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? David, Yeah, the code has been in beta for a while. It is UDP based. If all UDP is blocked then it won't work and wont be attempted. We do a lot to first ensure that you can receive solicited UDP which is what this feature relies on. i.e. If you send a packet to X via UDP and you are behind a NAT/Firewall, you can receive a response back. In LimeWire terms, you are then said to be firewall capable and your searches and responses indicate this. There are many ways to negotiate the initiation of the connection on both sides. LimeWire has a concept of a push proxy for firewalled hosts so we actually use that to deliver a special PUSH message that tells the host to initiate a UDP connection to ip:port. Both ends then start sending UDP messages at each other and shortly thereafter, they should both be able to receive those messages. A type of TCP style connection negotiation begins from there. Just to be clear, this is not proxying. The only thing that is proxied, is the PUSH message to trigger the actions of the uploader. The FAQ is out of date. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 6:28 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? How does the Firewall-to-Firewall portion of Limewire work? Does it use un-firewalled clients as relay servers? It doesn't sound like it, but I thought that's the only solution that truly works in all situations. The "features history" page mentions this on the entry for 8.12.2004: "Firewall to Firewall transfers allows two people behind firewalls to connect directly to each other and transfer data. This makes use of UDP, and a third party to coordinate the initial messaging. . Normally, firewalled users would only be able to download from other hosts who are not firewalled, which is of course severely limited. With firewall to firewall transfers, firewalled users can now access the full 100% of hosts." This implies something like the NAT-to-NAT trick works with firewalls also. I'm a little shaky on how UDP works with firewalls, do both clients initiate a conversation with a third party, and then the third party hands back information IP/port information of the pre-established out-bound connection? How does this work if the firewall simply blocks all UDP traffic? However, the website is either out of date or there's more to the story because the FAQ says: http://www.limewire.com/english/content/faq.shtml#fir1 "Q: What if I'm behind a firewall? A: LimeWire will work when a user is behind certain types of firewalls, but will not work behind certain other types. If you are behind a firewall, you will not be able to download anything from a user that's also behind a firewall. In general, if you can connect (you will see your "connection status" in the lower left hand corner of the application) using LimeWire, you should be able to download and upload files, but LimeWire will not work if you have either a web-only proxy or a SOCKS proxy." What's the full story? -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 2:12 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? If you believe that there are problems with LimeWire, you should submit them to bugs@limewire and they will be looked into promptly. If you have not already, you should also upgrade to version 4.2.3 to get rid of some potential startup issues with old GWebcaches. LimeWire is a "good new" p2p application - check out that firewall-to-firewall transfer in the new version. ;) Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of Digitalgruvmoves@aol.com Sent: Monday, November 29, 2004 9:10 PM To: p2p-hackers@zgp.org Subject: Re: [p2p-hackers] Why UDP and not TCP? Whats a good new p2p filesharing download to use? Limeware just started acting nuts. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20041130/cd8a2735/attachment.html From justin at chapweske.com Wed Dec 1 01:00:33 2004 From: justin at chapweske.com (Justin Chapweske) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: <20041201003449.E28013FCA7@capsicum.zgp.org> References: <20041201003449.E28013FCA7@capsicum.zgp.org> Message-ID: <1101862833.20537.463.camel@bog> > I?m no firewall nor security expert, but I was under the impression a > typical corporate firewall blocks most TCP ports, and virtually (or > even) all UDP ports. Do you have any stats on what fraction of > firewall users are able to take advantage of this feature? I'm guessing that most users behind that type of firewall shouldn't be running a file sharing app anyway unless its approved by the IT department. -Justin From samnospam at bcgreen.com Wed Dec 1 01:15:46 2004 From: samnospam at bcgreen.com (Stephen Samuel (leave the email alone)) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] How firewall to firewall works In-Reply-To: <20041130232809.5CBB23FCF7@capsicum.zgp.org> References: <20041130232809.5CBB23FCF7@capsicum.zgp.org> Message-ID: <41AD1B42.5020800@bcgreen.com> TCP ia a connection based protocol, when you start up a connection one machine initiates the connection, and the other is the responds to the connection. The normal sequence is three packets used just to open the connection Me:Port 3000 SYN -> You:Port 80 Me:Port 3000 <- SYN/ACK You:Port 80 Me:Port 3000 ACK -> You:port 80 At that point the connection is open, and people send messages in both directions until a sequence is transmitted which closes the connection. Stateful firewalls take advantage of this sequence. It can recognize when a machine from inside the firewall is initiating a connection and allow more freedom for outbound connections than inbound connections. The other thing to note about a connection is that it is identified by a combination of the local IP address/port combination and the remote address/port combination so if the address of 'me' was 12.13.14.15 and the address of 'you' was 18.19.20.21 then the connection above would would be described by TCP: 12.13.14.15:3000 <-> 18.19.20.21:80 ============================ UDP communications, on the other hand, are connectionless. Technically, you just send a packet and the other side either recieves it or doesnt. There is no protocol inherent in udp that signals the beginning or end of a connection. Congestion control and replacement of lost or damaged packets would have to be done at the user level, rather than the OS level which TCP does. For this reason most firewalls that allow UDP simply presume that any packet sent outbound is initiating a connection (if one doesn't already exist). Any inbound packet would be ignored unless it was associated with a previously outbound packet (using the same port/IP-address pair used for TCP) The way to 'fool' such statefull Firewalls with UDP is to have both machines talk to an intermediary and agree on a set of IP addresses and port numbers to use to talk to each other. You end up with the following conversation. Me:Port 3000 Hello -> You:Port 8000 (blocked by YOU's firewall) Me:Port 3000 <- Hello You:Port 8000 Gets thru (matching ports) Me:Port 3000 Welcome -> You:port 8000 Gets thru (matching ports) The second and third packets get thru because the recieving machine has already sent a packet using the address/port set of address, and the associated firewall pretty much has to either presume that the connection is legitimate, or just not allow outbound UDP connections at all (or only allow connections to specific ports). If the first option is chosen by both machines' firewalls, then the 'firewall to firewall' connections should work. This only works for a portion of firewalled users, but a big enough portion to make the process worth trying. David Barrett wrote: > How does the Firewall-to-Firewall portion of Limewire work? Does it use > un-firewalled clients as relay servers? It doesn?t sound like it, but I > thought that?s the only solution that truly works in all situations. > > > > The ?features history? page mentions this on the entry for 8.12.2004: > > > > ?Firewall to Firewall transfers allows two people behind firewalls to > connect directly to each other and transfer data. *This makes use of > UDP, and a third party to coordinate the initial messaging.* ? Normally, > firewalled users would only be able to download from other hosts who are > not firewalled, which is of course severely limited. With firewall to > firewall transfers, firewalled users can now access the full 100% of hosts.? -- Stephen Samuel +1(604)876-0426 samnospam@bcgreen.com http://www.bcgreen.com/ Powerful committed communication. Transformation touching the jewel within each person and bringing it to light. From dbarrett at quinthar.com Wed Dec 1 01:17:29 2004 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: <1101862833.20537.463.camel@bog> Message-ID: <20041201011748.B72233FCA7@capsicum.zgp.org> > -----Original Message----- > From: Justin Chapweske > Sent: Tuesday, November 30, 2004 6:01 PM > To: Peer-to-peer development. > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > > I'm no firewall nor security expert, but I was under the impression a > > typical corporate firewall blocks most TCP ports, and virtually (or > > even) all UDP ports. Do you have any stats on what fraction of > > firewall users are able to take advantage of this feature? > > I'm guessing that most users behind that type of firewall shouldn't be > running a file sharing app anyway unless its approved by the IT > department. Heh, true, but the general problem applies to more than file sharing. In my case, I'm writing an application that would be used inside exactly this type of network, and the adoption-pattern generally starts with a couple guys wanting to test it out before they can convince IT to punch holes in the firewall. Right now I support a relay-service as a fallback, but I'd love to offer NAT-to-NAT and Firewall-to-Firewall where applicable. Thus the big question is my mind is: if I implement this feature, how many users would actually be able to take advantage of it? -david From dbarrett at quinthar.com Wed Dec 1 04:45:54 2004 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] How firewall to firewall works In-Reply-To: <41AD1B42.5020800@bcgreen.com> Message-ID: <20041201044603.856833FCA7@capsicum.zgp.org> > -----Original Message----- > From: Stephen Samuel (leave the email alone) > Sent: Tuesday, November 30, 2004 6:16 PM > To: Peer-to-peer development. > Subject: [p2p-hackers] How firewall to firewall works > > This only works for a portion of firewalled users, but a big enough > portion to make the process worth trying. Excellent overview, thanks. Do you know anyone at Limewire that might have more detailed statistics on this? I'm really quite curious. -david From john.casey at gmail.com Wed Dec 1 07:49:09 2004 From: john.casey at gmail.com (John Casey) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: Opaque Key Access in DHT's ?? In-Reply-To: <20041104012834.GA20788@ref.nmedia.net> References: <20041104012834.GA20788@ref.nmedia.net> Message-ID: Hi, Thanks paul. That has helped a lot. Sorry for the late reply but I thought all messages where going to the mailing list. I have re: cc'd your reply to the mailing list. On Wed, 3 Nov 2004 17:28:34 -0800, Paul Campbell wrote: > First off, there's nothing that says you can't route/search a "region" of > DHT space. Look at the various DHT-based broadcast protocols. > > Essentially, a broadcast protocol sends a message routed via the DHT > that consists of "send(lowID, highID)". > > On receipt by a node in the range, in order to implement broadcasting, the > node then forwards the message on to neighbors that cover any subranges not > covered by the receiving node, changing the lowID/highID ranges appropriately. > > A search would be something like: > > broadcast(fromID, "keylow", "keyhigh") > > Second, Bloom filters can summarize any number of items. The key is that the > false positive rate of the filter increases as the number of items increases. > So you can potentially summarize as many items as you want, but the false > positive rate will increase dramatically once you get somewhat beyond the > "capacity" of a particular Bloom filter. In order to reduce this problem, > either the number of bits in the filter has to be increased or the number of > items has to be decreased. > > There are a bunch of potential tuning parameters in Bloom filters (number of > hashes, compression details if compression is used), but eventually the > fundamental Shannon information rate starts to become a serious limiting > factor. > From ddtbhai at gmail.com Wed Dec 1 12:19:31 2004 From: ddtbhai at gmail.com (Donny T. Daniel) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] How firewall to firewall works In-Reply-To: <20041201044603.856833FCA7@capsicum.zgp.org> References: <41AD1B42.5020800@bcgreen.com> <20041201044603.856833FCA7@capsicum.zgp.org> Message-ID: <68370453041201041922e08beb@mail.gmail.com> Thanks for the really cool description of the process ... it was very well put ..! On Tue, 30 Nov 2004 21:45:54 -0700, David Barrett wrote: > > -----Original Message----- > > From: Stephen Samuel (leave the email alone) > > Sent: Tuesday, November 30, 2004 6:16 PM > > To: Peer-to-peer development. > > Subject: [p2p-hackers] How firewall to firewall works > > > > This only works for a portion of firewalled users, but a big enough > > portion to make the process worth trying. > > Excellent overview, thanks. Do you know anyone at Limewire that might have > more detailed statistics on this? I'm really quite curious. > > -david > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From em at em.no-ip.com Wed Dec 1 12:54:17 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] How firewall to firewall works References: <20041201044603.856833FCA7@capsicum.zgp.org> Message-ID: <015e01c4d7a4$e31f3c60$0200a8c0@em.noip.com> ----- Original Message ----- From: "David Barrett" To: "'Peer-to-peer development.'" Sent: Wednesday, December 01, 2004 12:45 PM Subject: RE: [p2p-hackers] How firewall to firewall works > > -----Original Message----- > > From: Stephen Samuel (leave the email alone) > > Sent: Tuesday, November 30, 2004 6:16 PM > > To: Peer-to-peer development. > > Subject: [p2p-hackers] How firewall to firewall works > > > > This only works for a portion of firewalled users, but a big enough > > portion to make the process worth trying. > > Excellent overview, thanks. Do you know anyone at Limewire that might > have more detailed statistics on this? I'm really quite curious. I would guess about 75%, if the statistics contained in http://www.brynosaurus.com/pub/os/nat.pdf are correct (see also: http://zgp.org/pipermail/p2p-hackers/2004-November/002171.html ). For the rest, one can always use proxying/forwarding through selfless non-NATted nodes. Enzo From gbildson at limepeer.com Wed Dec 1 17:50:34 2004 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: <20041201003449.E28013FCA7@capsicum.zgp.org> Message-ID: David, Our operating environment has not yet stabilized - early beta testers didn't get the proper port stability tests and our entire user base has not been upgraded. However, early indications are that 50 to 60 percent of firewalled users will benefit. Again though, the harshest firewalled users probably have not had a good experience with P2P software so we are working with a self selected crew. This method does work with the Windows XP firewall so we expect it to be widely successful as XP becomes further adopted. One improvement that we can make is to our existing scheme is to handle firewalls/NATs that use a sequential port assignment algorithm for each attempt. Rather than just trying a fixed port, we could try to negotiate a connection with that port +1, +2 and +3. I'm not sure what percentage of users would really benefit from this though. As has been mentioned here recently, we certainly have noticed that incoming TCP connections are often possible after the same type of UDP pinging. We have no numbers on that though. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 7:35 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? That's pretty sweet. Do you know what fraction of firewalls "in the wild" allow for this capability? I'm no firewall nor security expert, but I was under the impression a typical corporate firewall blocks most TCP ports, and virtually (or even) all UDP ports. Do you have any stats on what fraction of firewall users are able to take advantage of this feature? -david ---------------------------------------------------------------------------- -- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 5:00 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? David, Yeah, the code has been in beta for a while. It is UDP based. If all UDP is blocked then it won't work and wont be attempted. We do a lot to first ensure that you can receive solicited UDP which is what this feature relies on. i.e. If you send a packet to X via UDP and you are behind a NAT/Firewall, you can receive a response back. In LimeWire terms, you are then said to be firewall capable and your searches and responses indicate this. There are many ways to negotiate the initiation of the connection on both sides. LimeWire has a concept of a push proxy for firewalled hosts so we actually use that to deliver a special PUSH message that tells the host to initiate a UDP connection to ip:port. Both ends then start sending UDP messages at each other and shortly thereafter, they should both be able to receive those messages. A type of TCP style connection negotiation begins from there. Just to be clear, this is not proxying. The only thing that is proxied, is the PUSH message to trigger the actions of the uploader. The FAQ is out of date. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 6:28 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? How does the Firewall-to-Firewall portion of Limewire work? Does it use un-firewalled clients as relay servers? It doesn't sound like it, but I thought that's the only solution that truly works in all situations. The "features history" page mentions this on the entry for 8.12.2004: "Firewall to Firewall transfers allows two people behind firewalls to connect directly to each other and transfer data. This makes use of UDP, and a third party to coordinate the initial messaging. . Normally, firewalled users would only be able to download from other hosts who are not firewalled, which is of course severely limited. With firewall to firewall transfers, firewalled users can now access the full 100% of hosts." This implies something like the NAT-to-NAT trick works with firewalls also. I'm a little shaky on how UDP works with firewalls, do both clients initiate a conversation with a third party, and then the third party hands back information IP/port information of the pre-established out-bound connection? How does this work if the firewall simply blocks all UDP traffic? However, the website is either out of date or there's more to the story because the FAQ says: http://www.limewire.com/english/content/faq.shtml#fir1 "Q: What if I'm behind a firewall? A: LimeWire will work when a user is behind certain types of firewalls, but will not work behind certain other types. If you are behind a firewall, you will not be able to download anything from a user that's also behind a firewall. In general, if you can connect (you will see your "connection status" in the lower left hand corner of the application) using LimeWire, you should be able to download and upload files, but LimeWire will not work if you have either a web-only proxy or a SOCKS proxy." What's the full story? -david ---------------------------------------------------------------------------- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 2:12 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? If you believe that there are problems with LimeWire, you should submit them to bugs@limewire and they will be looked into promptly. If you have not already, you should also upgrade to version 4.2.3 to get rid of some potential startup issues with old GWebcaches. LimeWire is a "good new" p2p application - check out that firewall-to-firewall transfer in the new version. ;) Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of Digitalgruvmoves@aol.com Sent: Monday, November 29, 2004 9:10 PM To: p2p-hackers@zgp.org Subject: Re: [p2p-hackers] Why UDP and not TCP? Whats a good new p2p filesharing download to use? Limeware just started acting nuts. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20041201/519f24a5/attachment.htm From douwen at yahoo.com Thu Dec 2 14:10:30 2004 From: douwen at yahoo.com (Dou Wen) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Simple reliable UDP data transfer In-Reply-To: <64D04EB5-397B-11D9-A632-000D932C5880@locut.us> Message-ID: <20041202141030.48082.qmail@web40601.mail.yahoo.com> hi,Ian, maybe you could get some ideas from www.rakkarsoft.com, which implemented a reliable UDP APIs. But its C++ implemented! it has a Socket Layer which just likes TCP-Style APIs. Regards Dou Wen --- Ian Clarke wrote: > I am in the process of implementing a simple UDP > data transfer > algorithm in Java (or more precisely, replacing a > braindead > implementation with something slightly more > respectable). > > The requirement is simple, get 256k from one node to > another over UDP > reliably. It should be "TCP friendly", ie. its flow > control shouldn't > crowd out politer TCP traffic, and packets, for > obvious reasons, should > be around 1k in size. > > I have toyed with a variety of ideas, and done some > research, but I > wanted to see if anyone had any thoughts or advice > on the simplest way > I can implement something to meet these requirements > (I will probably > use a straight-forward TCP-style windowed approach). > > Cheers, > > Ian. > > -- > Founder, The Freenet Project > http://freenetproject.org/ > CEO, Cematics Ltd http://cematics.com/ > Personal Blog http://locut.us/~ian/blog/ > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > __________________________________ Do you Yahoo!? All your favorites on one personal page – Try My Yahoo! http://my.yahoo.com From clausen at gnu.org Fri Dec 3 23:55:26 2004 From: clausen at gnu.org (Andrew Clausen) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] residue class In-Reply-To: <001601c4d5bf$c2684660$9402000a@ictltbo> References: <001601c4d5bf$c2684660$9402000a@ictltbo> Message-ID: <20041203235525.GH2431@gnu.org> On Mon, Nov 29, 2004 at 11:01:44AM +0800, Lutianbo wrote: > Would you please give me some meterial about > residue class in group theory. I think residue class refers to the elements of finite fields constructed by taking the integers and modding them by a prime. (Other finite fields exist) This looks like a reasonable explanation: www.math.utah.edu/~bertram/4030/clocks.pdf Ignore all but the last sentence of the first paragraph. I think that the "equivalence classes" they talk about are the same as your "residue classes". Cheers, Andrew From webmaster at software-x.org Thu Dec 2 00:40:37 2004 From: webmaster at software-x.org (RLWagner) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] unsubscribe References: <001601c4d5bf$c2684660$9402000a@ictltbo> <20041203235525.GH2431@gnu.org> Message-ID: <000a01c4d807$8addaa10$040aa8c0@softwarex> ----- Original Message ----- From: "Andrew Clausen" To: "Peer-to-peer development." Sent: Friday, December 03, 2004 5:55 PM Subject: Re: [p2p-hackers] residue class > On Mon, Nov 29, 2004 at 11:01:44AM +0800, Lutianbo wrote: >> Would you please give me some meterial about >> residue class in group theory. > > I think residue class refers to the elements of finite fields > constructed by taking the integers and modding them by a prime. > (Other finite fields exist) > > This looks like a reasonable explanation: > > www.math.utah.edu/~bertram/4030/clocks.pdf > > Ignore all but the last sentence of the first paragraph. > > I think that the "equivalence classes" they talk about are the same > as your "residue classes". > > Cheers, > Andrew > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From dbarrett at quinthar.com Sat Dec 4 06:07:19 2004 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: Message-ID: <20041204060722.19B703FC1E@capsicum.zgp.org> Great, thanks for the detailed answer. I'd love to hear more stats on this as you learn them. Incidentally, does Limewire (or any of the major P2P networks, for that matter) have a centralized stat-gathering mechanism? It'd seem you could have all clients safely upload an installation platform/networking profile - even ongoing usage stats - to a central server (with user consent) without compromising the rest of the decentralized architecture. Just curious if you already have something like this in place, or any plans to do so. -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Wednesday, December 01, 2004 10:51 AM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? David, Our operating environment has not yet stabilized - early beta testers didn't get the proper port stability tests and our entire user base has not been upgraded. However, early indications are that 50 to 60 percent of firewalled users will benefit. Again though, the harshest firewalled users probably have not had a good experience with P2P software so we are working with a self selected crew. This method does work with the Windows XP firewall so we expect it to be widely successful as XP becomes further adopted. One improvement that we can make is to our existing scheme is to handle firewalls/NATs that use a sequential port assignment algorithm for each attempt. Rather than just trying a fixed port, we could try to negotiate a connection with that port +1, +2 and +3. I'm not sure what percentage of users would really benefit from this though. As has been mentioned here recently, we certainly have noticed that incoming TCP connections are often possible after the same type of UDP pinging. We have no numbers on that though. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 7:35 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? That's pretty sweet. Do you know what fraction of firewalls "in the wild" allow for this capability? I'm no firewall nor security expert, but I was under the impression a typical corporate firewall blocks most TCP ports, and virtually (or even) all UDP ports. Do you have any stats on what fraction of firewall users are able to take advantage of this feature? -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 5:00 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? David, Yeah, the code has been in beta for a while. It is UDP based. If all UDP is blocked then it won't work and wont be attempted. We do a lot to first ensure that you can receive solicited UDP which is what this feature relies on. i.e. If you send a packet to X via UDP and you are behind a NAT/Firewall, you can receive a response back. In LimeWire terms, you are then said to be firewall capable and your searches and responses indicate this. There are many ways to negotiate the initiation of the connection on both sides. LimeWire has a concept of a push proxy for firewalled hosts so we actually use that to deliver a special PUSH message that tells the host to initiate a UDP connection to ip:port. Both ends then start sending UDP messages at each other and shortly thereafter, they should both be able to receive those messages. A type of TCP style connection negotiation begins from there. Just to be clear, this is not proxying. The only thing that is proxied, is the PUSH message to trigger the actions of the uploader. The FAQ is out of date. Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett Sent: Tuesday, November 30, 2004 6:28 PM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] Why UDP and not TCP? How does the Firewall-to-Firewall portion of Limewire work? Does it use un-firewalled clients as relay servers? It doesn't sound like it, but I thought that's the only solution that truly works in all situations. The "features history" page mentions this on the entry for 8.12.2004: "Firewall to Firewall transfers allows two people behind firewalls to connect directly to each other and transfer data. This makes use of UDP, and a third party to coordinate the initial messaging. . Normally, firewalled users would only be able to download from other hosts who are not firewalled, which is of course severely limited. With firewall to firewall transfers, firewalled users can now access the full 100% of hosts." This implies something like the NAT-to-NAT trick works with firewalls also. I'm a little shaky on how UDP works with firewalls, do both clients initiate a conversation with a third party, and then the third party hands back information IP/port information of the pre-established out-bound connection? How does this work if the firewall simply blocks all UDP traffic? However, the website is either out of date or there's more to the story because the FAQ says: http://www.limewire.com/english/content/faq.shtml#fir1 "Q: What if I'm behind a firewall? A: LimeWire will work when a user is behind certain types of firewalls, but will not work behind certain other types. If you are behind a firewall, you will not be able to download anything from a user that's also behind a firewall. In general, if you can connect (you will see your "connection status" in the lower left hand corner of the application) using LimeWire, you should be able to download and upload files, but LimeWire will not work if you have either a web-only proxy or a SOCKS proxy." What's the full story? -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Greg Bildson Sent: Tuesday, November 30, 2004 2:12 PM To: Peer-to-peer development. Subject: RE: [p2p-hackers] Why UDP and not TCP? If you believe that there are problems with LimeWire, you should submit them to bugs@limewire and they will be looked into promptly. If you have not already, you should also upgrade to version 4.2.3 to get rid of some potential startup issues with old GWebcaches. LimeWire is a "good new" p2p application - check out that firewall-to-firewall transfer in the new version. ;) Thanks -greg -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of Digitalgruvmoves@aol.com Sent: Monday, November 29, 2004 9:10 PM To: p2p-hackers@zgp.org Subject: Re: [p2p-hackers] Why UDP and not TCP? Whats a good new p2p filesharing download to use? Limeware just started acting nuts. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20041203/4a3ba7d4/attachment.html From provaluator at yahoo.de Mon Dec 6 11:11:57 2004 From: provaluator at yahoo.de (prova) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Why UDP and not TCP? In-Reply-To: <20041204060722.19B703FC1E@capsicum.zgp.org> References: <20041204060722.19B703FC1E@capsicum.zgp.org> Message-ID: Am 04.12.2004 um 07:07 schrieb David Barrett: > Great, thanks for the detailed answer.? I?d love to hear more stats on > this as you learn them. > > ? > > Incidentally, does Limewire (or any of the major P2P networks, for > that matter) have a centralized stat-gathering mechanism?? I guess so. Although it looks like they have some kind of bot that scans the network. http://www.limewire.com/english/content/netsize.shtml Steffen > ? > > > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] > On Behalf Of Greg Bildson > Sent: Wednesday, December 01, 2004 10:51 AM > To: Peer-to-peer development. > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > ? > > David, > > ? > > Our operating environment has not yet stabilized - early beta testers > didn't get the proper port stability tests and our entire user base > has not been upgraded.? However, early indications are that 50 to 60 > percent of firewalled users will benefit.? Again though, the harshest > firewalled users probably have not had a good experience with P2P > software so we are working with a self selected crew.?? This method > does work with the Windows XP firewall so we expect it to be widely > successful as XP becomes further adopted.? > > ? > > One improvement that we can make is to our existing scheme is to > handle firewalls/NATs that use a sequential port assignment algorithm > for each attempt.? Rather than just trying a fixed port, we could try > to negotiate a connection with that port +1, +2 and +3.? I'm not sure > what percentage of users would really benefit from this though. > > ? > > As has been mentioned here recently, we certainly have noticed that > incoming TCP connections are often possible after the same type of UDP > pinging.? We have no numbers on that though. > > ? > > Thanks > > -greg > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett > Sent: Tuesday, November 30, 2004 7:35 PM > To: 'Peer-to-peer development.' > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > That?s pretty sweet.? Do you know what fraction of firewalls ?in the > wild? allow for this capability? > > ? > > I?m no firewall nor security expert, but I was under the impression a > typical corporate firewall blocks most TCP ports, and virtually (or > even) all UDP ports.? Do you have any stats on what fraction of > firewall users are able to take advantage of this feature? > > ? > > -david > > ? > > > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] > On Behalf Of Greg Bildson > Sent: Tuesday, November 30, 2004 5:00 PM > To: Peer-to-peer development. > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > ? > > David, > > ? > > Yeah, the code has been in beta for a while.? > > ? > > It is UDP based.? If all UDP is blocked then it won't work and wont be > attempted.? We do a lot to first ensure that you can receive solicited > UDP which is what this feature relies on.? i.e. If you send a packet > to X via UDP and you are behind a NAT/Firewall, you can receive a > response back.? In LimeWire terms, you are then said to be firewall > capable and your searches and responses indicate this.? > > ? > > There are many ways to negotiate the initiation of the connection on > both sides.? LimeWire has a concept of a push proxy for firewalled > hosts so we actually use that to deliver a special PUSH message that > tells the host to initiate a UDP connection to ip:port.? Both ends > then start sending UDP messages at each other and shortly thereafter, > they should both be able to receive those messages.? A type of TCP > style connection negotiation begins from there.? Just to be clear, > this is not proxying.? The only thing that is proxied, is the PUSH > message to trigger the actions of the uploader. > > ? > > The FAQ is out of date. > > ? > > Thanks > > -greg > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of David Barrett > Sent: Tuesday, November 30, 2004 6:28 PM > To: 'Peer-to-peer development.' > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > How does the Firewall-to-Firewall portion of Limewire work?? Does it > use un-firewalled clients as relay servers?? It doesn?t sound like it, > but I thought that?s the only solution that truly works in all > situations. > > ? > > The ?features history? page mentions this on the entry for 8.12.2004: > > ? > > ?Firewall to Firewall transfers allows two people behind firewalls to > connect directly to each other and transfer data. This makes use of > UDP, and a third party to coordinate the initial messaging. ? > Normally, firewalled users would only be able to download from other > hosts who are not firewalled, which is of course severely limited. > With firewall to firewall transfers, firewalled users can now access > the full 100% of hosts.? > > ? > > This implies something like the NAT-to-NAT trick works with firewalls > also.? I?m a little shaky on how UDP works with firewalls, do both > clients initiate a conversation with a third party, and then the third > party hands back information IP/port information of the > pre-established out-bound connection?? How does this work if the > firewall simply blocks all UDP traffic? > > ? > > However, the website is either out of date or there?s more to the > story because the FAQ says: > > ? > > http://www.limewire.com/english/content/faq.shtml#fir1 > > ? > > ?Q: What if I?m behind a firewall? > > ? > > A: LimeWire will work when a user is behind certain types of > firewalls, but will not work behind certain other types. If you are > behind a firewall, you will not be able to download anything from a > user that?s also behind a firewall. In general, if you can connect > (you will see your ?connection status? in the lower left hand corner > of the application) using LimeWire, you should be able to download and > upload files, but LimeWire will not work if you have either a web-only > proxy or a SOCKS proxy.? > > ? > > What?s the full story? > > ? > > -david > > ? > > > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] > On Behalf Of Greg Bildson > Sent: Tuesday, November 30, 2004 2:12 PM > To: Peer-to-peer development. > Subject: RE: [p2p-hackers] Why UDP and not TCP? > > ? > > If you believe that there are problems with LimeWire, you should > submit them to?bugs@limewire?and they will be looked into promptly.?? > If you have not already, you should also upgrade to version 4.2.3 to > get rid of some potential startup issues with old GWebcaches. > > ? > > LimeWire is a "good new" p2p application - check out that > firewall-to-firewall transfer in the new version.? ;) > > ? > > Thanks > > -greg > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of > Digitalgruvmoves@aol.com > Sent: Monday, November 29, 2004 9:10 PM > To: p2p-hackers@zgp.org > Subject: Re: [p2p-hackers] Why UDP and not TCP? > > Whats a good new p2p filesharing download to use? Limeware just > started acting nuts. > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 19068 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041206/f75b7a21/attachment.bin From mgp at ucla.edu Wed Dec 8 00:44:13 2004 From: mgp at ucla.edu (Michael Parker) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] p2p capacity estimates Message-ID: <41B64E5D.7010308@ucla.edu> Hi everybody, At http://www.pdos.lcs.mit.edu/p2psim/kingdata/ I was able to pick up a matrix consisting of the pairwise RTTs between 1740 nodes on the Internet. I was wondering if anyone knew of a similar file that had pairwise capacity estimates (bandwidth estimates) between a large number of nodes? The number of nodes doesn't have to be that large, since bandwidth estimation is more difficult and a study of over a thousand nodes would certainly be non-trivial. I am just looking for a set with a diverse population of end-users (e.g., dial-up, cable, DSL, T3) representative of what you might find on a p2p network, from which I can glean general characteristics and hopefully extrapolate to a larger number of nodes. If such a data set didn't exist, perhaps someone could point me toward some Gnutella/OpenNap/KaZaA traces which I might be able to use? Thanks, Michael Parker From travis at redswoosh.net Wed Dec 8 03:04:59 2004 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Your latest info Message-ID: <1102475099.22104.30059.sendUpdate@mx.plaxo.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: Travis Kalanick.vcf Type: text/x-vcard Size: 348 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041208/d9aa1eb9/TravisKalanick.vcf From travis at redswoosh.net Wed Dec 8 08:26:24 2004 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Before I get flamed Message-ID: <200412080826.iB88QpaL017690@be9.noc0.redswoosh.com> Sorry about plaxo-ing the list. Somehow I missed the p2p-hackers list when filtering addresses. T Travis Kalanick Red Swoosh, Inc. Founder, Chairman travis@redswoosh.net (v) 310.666.1429 (f) 253.322.9478 AIM: ScourTrav123 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20041208/6af0c000/attachment.html From tutschku at informatik.uni-wuerzburg.de Wed Dec 8 11:19:07 2004 From: tutschku at informatik.uni-wuerzburg.de (Kurt Tutschku) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] ETT Special Issue on P2P Network and P2P Services available Message-ID: <003f01c4dd17$bb5dced0$b86abb84@musa> Dear P2P researchers, as the coordinating guest editor, I'd like to draw your attention to recent research results on P2P published in the latest issue of the European Transactions of Telecommunication (ETT, Vol. 15, No. 6). The on-line version of the ETT special issue on "P2P Networks and P2P Services" is now available at: http://www3.interscience.wiley.com/cgi-bin/jhome/104087069 I'd like to acknowledge the support of my co-editors: Frank-Uwe Andersen, Hermann de Meer, and Konsuke Kawashima. Best Regards Kurt Tutschku --- Dr. Kurt Tutschku, Assistant Professor University of Wuerzburg Department of Distributed Systems Institute of Computer Science Am Hubland 97074 Wuerzburg Germany Tel.: +49-931-8886641 FAX.: +49-931-8886632 mailto:tutschku@informatik.uni-wuerzburg.de or mailto:kurttutschku@hotmail.com http://www-info3.informatik.uni-wuerzburg.de/staff/tutschku From jdd at dixons.org Wed Dec 8 15:19:53 2004 From: jdd at dixons.org (Jim Dixon) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] ETT Special Issue on P2P Network and P2P Services available In-Reply-To: <003f01c4dd17$bb5dced0$b86abb84@musa> References: <003f01c4dd17$bb5dced0$b86abb84@musa> Message-ID: On Wed, 8 Dec 2004, Kurt Tutschku wrote: > Dear P2P researchers, > > as the coordinating guest editor, I'd like to draw your attention to > recent research results on P2P published in the latest issue of the > European Transactions of Telecommunication (ETT, Vol. 15, No. 6). > > The on-line version of the ETT special issue on "P2P Networks and P2P > Services" is now available at: > > http://www3.interscience.wiley.com/cgi-bin/jhome/104087069 It appears that without a subscription you can only view a list of articles in the issue. If you register, you get access to a sample issue - but not to this special issue on P2P. -- Jim Dixon jdd@dixons.org tel +44 117 982 0786 mobile +44 797 373 7881 http://jxcl.sourceforge.net Java unit test coverage http://xlattice.sourceforge.net p2p communications infrastructure From farez at imetrix.co.uk Wed Dec 8 21:59:41 2004 From: farez at imetrix.co.uk (Farez Rahman) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Memory and reputation calculation Message-ID: <6.1.2.0.0.20041208215931.02b80300@pop.cs.ucl.ac.uk> Hi, Does anyone have any reference to work/paper which looked at how much history is useful in reputation systems? Perhaps some analysis of tradeoffs between size of memory and effective sample size? Cheers, Farez - trust . reputation . research - www.cs.ucl.ac.uk/staff/f.abdulrahman/ - socap.blogspot.com From gbildson at limepeer.com Wed Dec 8 22:49:14 2004 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Memory and reputation calculation In-Reply-To: <6.1.2.0.0.20041208215931.02b80300@pop.cs.ucl.ac.uk> Message-ID: The best trust work that I saw came out of Microsoft Research I believe. I'm not sure how much of it was ever put into papers though and have never really found good sources of it online. I don't recall your points of interest being addressed. Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Farez Rahman > Sent: Wednesday, December 08, 2004 5:00 PM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Memory and reputation calculation > > > Hi, > > Does anyone have any reference to work/paper which looked at how > much history is useful in reputation systems? Perhaps some > analysis of tradeoffs between size of memory and effective sample size? > > Cheers, > Farez > > - trust . reputation . research > - www.cs.ucl.ac.uk/staff/f.abdulrahman/ > - socap.blogspot.com > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From clausen at gnu.org Wed Dec 8 22:25:38 2004 From: clausen at gnu.org (Andrew Clausen) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: [trustcomp] Memory and reputation calculation In-Reply-To: <6.1.2.0.0.20041208220022.02b68620@pop.cs.ucl.ac.uk> References: <6.1.2.0.0.20041208220022.02b68620@pop.cs.ucl.ac.uk> Message-ID: <20041208222537.GA2441@gnu.org> On Wed, Dec 08, 2004 at 10:00:30PM +0000, Farez Rahman wrote: > Does anyone have any reference to work/paper which looked at how much > history is useful in reputation systems? Perhaps some analysis of > tradeoffs between size of memory and effective sample size? The first working paper on Chris Dellarocas' site claims that the history size doesn't matter: http://ccs.mit.edu/dell/reputation.html Cheers, Andrew ------------------------ Yahoo! Groups Sponsor --------------------~--> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar. Now with Pop-Up Blocker. Get it for free! http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/ngFolB/TM --------------------------------------------------------------------~-> _________ http://www.trustcomp.org Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/trustcomp/ <*> To unsubscribe from this group, send an email to: trustcomp-unsubscribe@yahoogroups.com <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/ From paul at nmedia.net Thu Dec 9 00:42:50 2004 From: paul at nmedia.net (paul@nmedia.net) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Automatic reputation systems for P2P security? Message-ID: <41B7593A.16202.5A7CCB@localhost> I've seen several papers referencing advogato, among other things, and it seems like reputation/trust systems solve a lot of problems related to P2P misbehavior. For instance, clients can track other clients that send out bogus files, that report a file and then refuse to share it, that create bogus queueing data (big problem with Emule/Edonkey networks), that might outright lie or otherwise cheat/steal and attempt to disrupt a Chord network, etc. It seems that scalar trust systems aren't going to do it because it is fairly easy to cheat by creating fake nodes, etc. So the real trick is the "group" or vector trust metrics. However, that may solve the theoretical issue but I haven't seen any real examples of implementation. For instance, most of the papers referring to Advogato and Advogato-like systems are based on the client-server model. And to implement trust networks as it appears that they are done now, the shear amount of data necessary makes them pretty darned unwieldy. In addition, it is relatively well known (but time/bandwidth consuming) for a node to detect misbehaving nodes. But translating that to a trust metric, or even how to handle that on an implementation level has not been published anywhere. SO...is there anything out there on this sort of idea, especially on the implementation side? I mean...if this can be done in reality, then it has a whole host of uses even just in the small world of file sharing networks. As it stands, any trust metric that's been tried so far is easily tampered with by the clients. From tutschku at informatik.uni-wuerzburg.de Thu Dec 9 08:02:55 2004 From: tutschku at informatik.uni-wuerzburg.de (Kurt Tutschku) Date: Sat Dec 9 22:12:44 2006 Subject: AW: [p2p-hackers] ETT Special Issue on P2P Network and P2P Servicesavailable In-Reply-To: Message-ID: <004c01c4ddc5$7d35a520$b86abb84@musa> Dear Jim, unfortunately we are living in a world of monetary policies, i.e. free access is to the journal is not possible. However, I'd advice you a true real-world peer-to-peer approach (perhaps this is somehow appropriate for this research topic ;-) and to ask the authors directly by email about their publications. They can give you also more information on their research. :-) Have a nice day & Best Regards Kurt > -----Urspr?ngliche Nachricht----- > Von: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org] Im Auftrag von Jim Dixon > Gesendet: Mittwoch, 8. Dezember 2004 16:20 > An: tutschku@informatik.uni-wuerzburg.de; Peer-to-peer development. > Cc: p2prg@ietf.org > Betreff: Re: [p2p-hackers] ETT Special Issue on P2P Network > and P2P Servicesavailable > > > On Wed, 8 Dec 2004, Kurt Tutschku wrote: > > > Dear P2P researchers, > > > > as the coordinating guest editor, I'd like to draw your > attention to > > recent research results on P2P published in the latest issue of the > > European Transactions of Telecommunication (ETT, Vol. 15, No. 6). > > > > The on-line version of the ETT special issue on "P2P > Networks and P2P > > Services" is now available at: > > > > http://www3.interscience.wiley.com/cgi-bin/jhome/104087069 > > It appears that without a subscription you can only view a > list of articles in the issue. If you register, you get > access to a sample issue - but not to this special issue on P2P. > > -- > Jim Dixon jdd@dixons.org tel +44 117 982 0786 mobile +44 > 797 373 7881 > http://jxcl.sourceforge.net Java unit > test coverage > http://xlattice.sourceforge.net p2p communications > infrastructure > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/P> eerToPeerConferences > From p2p-hackers at strufe.de Thu Dec 9 10:11:42 2004 From: p2p-hackers at strufe.de (Thorsten Strufe) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: AW: [P2Prg] ETT Special Issue on P2P Network and P2P Services available In-Reply-To: <004601c4ddc5$41835b80$b86abb84@musa> References: <004601c4ddc5$41835b80$b86abb84@musa> Message-ID: <41B824DE.2020006@strufe.de> Hi Kurt, Bill, I thought about 'complaining' for free access; we are doing research which is based on the possibility to review and scrutinize other researchers theses after all, but then fell back to using a big and commonly known internet search engine, which produced pdf-results for all but two or three of the papers... ;-) hth, best, thorsten From seth.johnson at RealMeasures.dyndns.org Thu Dec 9 19:28:34 2004 From: seth.johnson at RealMeasures.dyndns.org (Seth Johnson) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Don't Let the RIAA Put the Net at Risk! Message-ID: <41B8A762.894C691E@RealMeasures.dyndns.org> > http://www.nyfairuse.org/action/ftc/ Don't Let the RIAA Put the Net at Risk! Tell the FTC that the Internet is Peer to Peer, It is Ours, and We Intend to Keep It! Please forward this notice to any other concerned parties you may know. Please tell the FTC not to allow a few rich cartels and monopolies such as the RIAA, MPAA and Microsoft to put the Internet at risk: http://www.ftc.gov/bcp/workshops/filesharing/comments.htm What's Going On: The FTC has issued a call for participation announcing a workshop on "P2P Filesharing" technology to take place in Washington, DC this December 15th and 16th. An RIAA-sponsored CapAnalysis paper submitted to the FTC, calls for an investigation of "P2P Filesharing" applications for deceptive practices that affect the privacy and security of users, subjecting them to such risks as adware, viruses, exposure to undesirable material, impairments of computer function, and last but not least, liability to charges of copyright infringement. Congress is also calling the FTC to investigate these products. We call all those who know the Internet is a common good, who make productive use of it and who develop applications for it as a regular part of their daily lives, to join us in telling the FTC what the real sources of these risks are. Please tell the FTC not to allow a few rich cartels and monopolies such as the RIAA, MPAA and Microsoft to put the Internet at risk: http://www.ftc.gov/bcp/workshops/filesharing/comments.htm Get On the Bus! We are organizing a caravan of concerned citizens to travel to the nation's capital and defend our rights and powers against the RIAA, the MPAA and Microsoft. When we arrive: * We will call the FTC to protect Internet users by acting against a few rich cartels and monopolies that impede innovation and access to robust solutions, choice, transparency and control. * We will call the FTC to focus their attention on the real sources of the risks in question, and to respond to them appropriately. * We will pose the question to the FTC of how they can distinguish the applications they have selected for consideration at this workshop from the multitude of applications of the Internet and the ordinary functions of operating systems now in use on millions of interconnected desktops across the planet. * We will press the FTC to explain what risks are actually unique to the applications they have singled out. * We will call the FTC to separate copyright matters from consideration of the private interests of computer owners. * We will call the FTC to refer copyright policy to the appropriate body, the United States Congress. Please submit comments to the FTC here: http://www.ftc.gov/bcp/workshops/filesharing/comments.htm Please contact us to let us know you will join us in this action and to offer your assistance with travel, lodging and sustenance: http://www.nyfairuse.org/cgi-bin/nyfu/contactus Links: The FTC "P2P Filesharing" Workshop: http://www.ftc.gov/bcp/workshops/filesharing/index.htm The CapAnalysis/RIAA Paper: http://ipcentral.info/blog/P2P%20White%20Paper.doc House and Senate Members Urge FTC Action Against P2P: http://www.gnutellanews.com/article/13743 >From Clean System to Zombie Bot in Four Minutes: http://slashdot.org/article.pl?sid=04/11/30/1932245 In Praise of P2P: http://www.economist.com/displayStory.cfm?Story_id=3422905 New Yorkers for Fair Use - www.nyfairuse.org -- DRM is Theft! We are the Stakeholders! New Yorkers for Fair Use http://www.nyfairuse.org [CC] Counter-copyright: http://realmeasures.dyndns.org/cc I reserve no rights restricting copying, modification or distribution of this incidentally recorded communication. Original authorship should be attributed reasonably, but only so far as such an expectation might hold for usual practice in ordinary social discourse to which one holds no claim of exclusive rights. From seth.johnson at RealMeasures.dyndns.org Thu Dec 9 19:28:37 2004 From: seth.johnson at RealMeasures.dyndns.org (Seth Johnson) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] NYFU Volunteers Meeting for P2P Action, Saturday 11 December 2004 Message-ID: <41B8A765.1632E146@RealMeasures.dyndns.org> For those of you in the New York area. Seth
New Yorkers for Fair Use will meet at 6:30 pm Saturday 11 December 2004. NYFU and many organizations, tribes, and free lances will attend the FTC P2P Workshop on 15 and 16 December 2004: http://www.nyfairuse.org/action/ftc At this NYFU meeting we will make arrangements to get down to the FTC P2P Workshop and present our case. We will meet at the the Gyro Pizza and Bagel Place on the corner of Third Avenue and Eighth Street on the Island of the Manahattoes, likely in the back. The Gyro Place has several names. The location is also called Astor Place and also St. Marks and Third Avenue. There are two subway stops nearby: Eighth Street NRW, Astor Place Lexington Avenue Line. NYFU will be in Washington on 15 December 2004 to present to the FTC facts and principles not yet well understood among legislators, judges, and regulators. One of the things NYFU will argue for is that there are two distinct usages of the phrase "peer to peer", and that it is dangerous to conflate the two. We need your help to convey this, and more, to the FTC. If you believe you have the right to own a computer and use your computer as you wish in the privacy of your house, and you are willing to work in defense of this right, come to this meeting. If you believe that the First Amendment to the Constitution of the United States, and the Fourth too, apply to our use of our Net, and you are willing to work in the common defense of our Great Commons, come to this meeting. In particular, at this meeting we will make arrangements to get folk down to Washington DC in good time for the FTC Workshop, and we will get housing for those who stay overnight. So if you want to come down to Washington DC with New Yorkers for Fair Use, and speak before the FTC for Freedom of the Net, come to this meeting. New Yorkers for Fair Use DRM is Theft! We are the Stakeholders! http://www.nyfairuse.org
Distributed poC TINC: Jay Sulzberger Corresponding Secretary LXNY LXNY is New York's Free Computing Organization. http://www.lxny.org From muller at emse.fr Fri Dec 10 08:33:39 2004 From: muller at emse.fr (MULLER Guillaume) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: Memory and reputation calculation In-Reply-To: <20041209193503.BECE23FD5F@capsicum.zgp.org> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> Message-ID: <41B95F63.2090501@emse.fr> Hi all, Right, I would have cited Dellarocas' papers also because he is the only one I know that worked on this subject. However, IMHO, his claim that size of history doesn't matter is false. He took this conclusion in very a specific domain that is eBay-like market-places with very specific assumption (cf. cited paper). My idea is that size of history DOES matter. Let's imagine a system (even eBay-like) where every agent *knows* that the history is a list of the X last encounters experiences. Then it is easy to see that cheating 1/X times is a strategy that pays off (particularly in systems where ratings might be noisy). IMHO, the key point with respect to the history is that others should not be able guess its size. If it has a fixed size, I believe it doesn't matter if (and only if) other can guess its size (and therefore cannot use strategy as described above). However, I'm sorry I didn't have time to make any experimentations, but I'd like to hear if anybody has. Regards G. MULLER -- *************** http://www.guillaume-muller.tk/ *************** *MULLER Guillaume *** *** Office : 532 * *Phone: 33 4 77 42 66 84 ** ** 29 rue des Fr?res Ponchardier* *Fax: 33 4 77 42 66 66 *** *** 42023 Saint-?tienne C?DEX 2 * *Principe unixien : "faire une seule chose et la faire bien". * *************************************************************** From eugen at leitl.org Fri Dec 10 10:25:34 2004 From: eugen at leitl.org (Eugen Leitl) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] CodeCon CFP deadline nearing (fwd from rabbi@abditum.com) Message-ID: <20041210102534.GD9221@leitl.org> ----- Forwarded message from Len Sassaman ----- From: Len Sassaman Date: Fri, 10 Dec 2004 01:08:57 -0800 (PST) To: cypherpunks@minder.net Subject: CodeCon CFP deadline nearing CodeCon 4.0 February 11-13, 2005 San Francisco CA, USA www.codecon.org Call For Papers CodeCon is the premier showcase of cutting edge software development. It is an excellent opportunity for programmers to demonstrate their work and keep abreast of what's going on in their community. All presentations must include working demonstrations, ideally accompanied by source code. Presenters must be done by one of the active developers of the code in question. We emphasize that demonstrations be of *working* code. We hereby solicit papers and demonstrations. * Papers and proposals due: December 15, 2004 * Authors notified: January 1, 2005 Possible topics include, but are by no means restricted to: * community-based web sites - forums, weblogs, personals * development tools - languages, debuggers, version control * file sharing systems - swarming distribution, distributed search * security products - mail encryption, intrusion detection, firewalls Presentations will be a 45 minutes long, with 15 minutes allocated for Q&A. Overruns will be truncated. Submission details: Submissions are being accepted immediately. Acceptance dates are November 15, and December 15. After the first acceptance date, submissions will be either accepted, rejected, or deferred to the second acceptance date. The conference language is English. Ideally, demonstrations should be usable by attendees with 802.11b connected devices either via a web interface, or locally on Windows, UNIX-like, or MacOS platforms. Cross-platform applications are most desirable. Our venue will be 21+. To submit, send mail to submissions-2005@codecon.org including the following information: * Project name * url of project home page * tagline - one sentence or less summing up what the project does * names of presenter(s) and urls of their home pages, if they have any * one-paragraph bios of presenters, optional, under 100 words each * project history, under 150 words * what will be done in the project demo, under 200 words * slides to be shown during the presentation, if applicable * future plans General Chairs: Jonathan Moore, Len Sassaman Program Chair: Bram Cohen Program Committee: * Jeremy Bornstein, AtomShockwave Corp., USA * Bram Cohen, BitTorrent, USA * Jered Floyd, Permabit, USA * Ian Goldberg, Zero-Knowledge Systems, CA * Dan Kaminsky, Avaya, USA * Klaus Kursawe, Katholieke Universiteit Leuven, BE * Ben Laurie, A.L. Digital Ltd., UK * David Molnar, University of California, Berkeley, USA * Jonathan Moore, Mosuki, USA * Len Sassaman, Nomen Abditum Services, USA Sponsorship: If your organization is interested in sponsoring CodeCon, we would love to hear from you. In particular, we are looking for sponsors for social meals and parties on any of the three days of the conference, as well as sponsors of the conference as a whole and donors of door prizes. If you might be interested in sponsoring any of these aspects, please contact the conference organizers at codecon-admin@codecon.org. Press policy: CodeCon provides a limited number of passes to bona fide press. Complimentary press passes will be evaluated on request. Everyone is welcome to pay the low registration fee to attend without an official press credential. Questions: If you have questions about CodeCon, or would like to contact the organizers, please mail codecon-admin@codecon.org. Please note this address is only for questions and administrative requests, and not for workshop presentation submissions. ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041210/492e6363/attachment.pgp From paul at ref.nmedia.net Fri Dec 10 17:29:27 2004 From: paul at ref.nmedia.net (Paul Campbell) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: Memory and reputation calculation In-Reply-To: <41B95F63.2090501@emse.fr> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> <41B95F63.2090501@emse.fr> Message-ID: <20041210172927.GB8406@ref.nmedia.net> With regards to the history function, I recall seeing a paper (no idea where or what the title was) that looked at it a different way. The concern is that in a P2P environment, there's no central assumed tamper-proof central server. One must rely on the peers themselves for history. It would be relatively easy for a peer to simply erase and ignore bad history, or for peers to be able to collude to report false history, unless one of two things happens: 1. The vector/group concept of Advogato among others prevents collusion simply because there are no multple paths...the false history shows up as a self-referential structure and not as a web of trust links. The group/vector concept searches for multiple disjoint paths of trust, which lessens or destroys collusion. 2. That the history passed on by a peer should be serialized in such a way that it is tamper-proof. That is, the client can't selectively delete events from the history. For instance, a one-way accumulator-type function intertwined into the data performs the protection. It doesn't circumvent the possibility of a client simply deleting the last few events in the history (and nothing is going to stop a client from doing a snapshot to achieve this), but it at least makes such selective editting an all-or-nothing function. From bkn3 at columbia.edu Tue Dec 14 20:09:37 2004 From: bkn3 at columbia.edu (Brad Neuberg) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Skype's "Global Index" In-Reply-To: <20041210172927.GB8406@ref.nmedia.net> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> <41B95F63.2090501@emse.fr> <20041210172927.GB8406@ref.nmedia.net> Message-ID: <6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com> Skype claims to have a scalable, secure, globally distributed directory of Skype phone numbers they call the "Global Index". From the Skype FAQ: "P2P network technologies used by file-sharing applications would be almost suitable for decentralizing [phonebook directories], but those networks are fragmented in nature ? a search does not reach all nodes in the network. Clearly, in order to deliver high quality telephony with the lowest possible costs, a third generation of P2P technology (?3G P2P?), or Global Index (GI) was a necessary development and represents yet another paradigm shift in the notion of scaleable networks. The Global Index technology is a multi-tiered network where supernodes communicate in such a way that every node in the network has full knowledge of all available users and resources with minimal latency. " Does anyone know what techniques Skype might be using to actually achieve this? Brad Neuberg, bkn3@columbia.edu Senior Software Engineer, Rojo Networks Weblog: http://www.codinginparadise.org ===================================================================== Check out Rojo, an RSS and Atom news aggregator that I work on. Visit http://rojo.com for more info. Feel free to ask me for an invite! Rojo is Hiring! If you're interested in RSS, Weblogs, Social Networking, Java, Open Source, etc... then come work with us at Rojo. If you recommend someone and we hire them you'll get a free iPod! See http://www.rojonetworks.com/JobsAtRojo.html. From gojomo at bitzi.com Tue Dec 14 21:19:25 2004 From: gojomo at bitzi.com (Gordon Mohr) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Skype's "Global Index" In-Reply-To: <6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> <41B95F63.2090501@emse.fr> <20041210172927.GB8406@ref.nmedia.net> <6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com> Message-ID: <41BF58DD.2040403@bitzi.com> Brad Neuberg wrote: > Does anyone know what techniques Skype might be using to actually > achieve this? I would wager that it's a DHT, by another name, with their own pragmatic wrinkles to deal with observed real-net conditions. (That would be their pattern as established with the original Kazaa, which ran slightly ahead of Gnutella/published work with some necessary and pragmatic bootstrapping, alt-source, and supernode innovations.) But I'm just speculating. - Gordon From bkn3 at columbia.edu Wed Dec 15 01:24:24 2004 From: bkn3 at columbia.edu (Brad Neuberg) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Skype's "Global Index" In-Reply-To: <41BF58DD.2040403@bitzi.com> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> <41B95F63.2090501@emse.fr> <20041210172927.GB8406@ref.nmedia.net> <6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com> <41BF58DD.2040403@bitzi.com> Message-ID: <6.1.0.6.2.20041214172357.022e7210@pop.mail.yahoo.com> I found a presentation where some researchers seem to have reverse engineered some of the Skype protocol. Thought others might find this useful: http://mnet.cs.nthu.edu.tw/paper/Chance/041125.pdf At 01:19 PM 12/14/2004, you wrote: >Brad Neuberg wrote: >>Does anyone know what techniques Skype might be using to actually achieve >>this? > >I would wager that it's a DHT, by another name, with their own >pragmatic wrinkles to deal with observed real-net conditions. >(That would be their pattern as established with the original >Kazaa, which ran slightly ahead of Gnutella/published work with >some necessary and pragmatic bootstrapping, alt-source, and >supernode innovations.) > >But I'm just speculating. > >- Gordon >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences Brad Neuberg, bkn3@columbia.edu Senior Software Engineer, Rojo Networks Weblog: http://www.codinginparadise.org ===================================================================== Check out Rojo, an RSS and Atom news aggregator that I work on. Visit http://rojo.com for more info. Feel free to ask me for an invite! Rojo is Hiring! If you're interested in RSS, Weblogs, Social Networking, Java, Open Source, etc... then come work with us at Rojo. If you recommend someone and we hire them you'll get a free iPod! See http://www.rojonetworks.com/JobsAtRojo.html. From p.schow at comcast.net Wed Dec 15 02:19:15 2004 From: p.schow at comcast.net (Peter Schow) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Skype's "Global Index" In-Reply-To: <6.1.0.6.2.20041214172357.022e7210@pop.mail.yahoo.com> References: <20041209193503.BECE23FD5F@capsicum.zgp.org> <41B95F63.2090501@emse.fr> <20041210172927.GB8406@ref.nmedia.net> <6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com> <41BF58DD.2040403@bitzi.com> <6.1.0.6.2.20041214172357.022e7210@pop.mail.yahoo.com> Message-ID: <20041215021915.GA26603@panacea> On Tue, Dec 14, 2004 at 05:24:24PM -0800, Brad Neuberg wrote: > I found a presentation where some researchers seem to have reverse > engineered some of the Skype protocol. Thought others might find this > useful: > > http://mnet.cs.nthu.edu.tw/paper/Chance/041125.pdf Which I think they borrowed portions of: http://arxiv.org/pdf/cs.NI/0412017 to make. From em at em.no-ip.com Wed Dec 15 03:05:32 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Skype's "Global Index" References: <20041209193503.BECE23FD5F@capsicum.zgp.org><41B95F63.2090501@emse.fr> <20041210172927.GB8406@ref.nmedia.net><6.1.0.6.2.20041214120809.022ced10@pop.mail.yahoo.com><41BF58DD.2040403@bitzi.com> <6.1.0.6.2.20041214172357.022e7210@pop.mail.yahoo.com> Message-ID: <000d01c4e252$f230da20$0200a8c0@em.noip.com> ----- Original Message ----- From: "Brad Neuberg" To: "Peer-to-peer development." Sent: Wednesday, December 15, 2004 9:24 AM > I found a presentation where some researchers seem to have reverse > engineered some of the Skype protocol. Thought others might find this > useful: > > http://mnet.cs.nthu.edu.tw/paper/Chance/041125.pdf Thanks Brad, a very interesting document: it appears to confirm that the login server is centralized under Skype Inc.'s control. By the way, in matters of NAT traversal, does anybody know of any opensource implementation of TURN (http://www.ietf.org/internet-drafts/draft-rosenberg-midcom-turn-06.txt )? Vovida seems to provide only an implementation of STUN, which only works with a subset of NATting devices but provides no relaying for the rest of them. Enzo -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.296 / Virus Database: 265.5.3 - Release Date: 14/12/04 From travis at redswoosh.net Wed Dec 15 05:14:08 2004 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Red Swoosh/Travis Kalanick comments Message-ID: <4748.12.170.144.61.1103087648.squirrel@webmail.redswoosh.net> I made some comments today that were covered (sort of): http://news.google.com/news?hl=en&ned=us&ie=UTF-8&ncl=http://asia.cnet.com/news/personaltech/0,39037091,39209363,00.htm For the record, here are my comments (raw text). A careful read might bring some nuances to light . . . Thank you for the introduction Jon. Certainly I have a unique perspective on the litigation. The last time I was in a room with the MPAA was in a courtroom on the other side of a $250B lawsuit. Since then I?ve certainly turned a corner as a technologist. And though I have my own opinions on litigation and its effectiveness, I?m not here to discuss lawsuits. So, what am I here to talk about? I?m here to tell you that P2P technology is good for the content owners and the media industry. And I?m here to communicate how the industry can better fight piracy by beginning to promote and adopt the positive, non-infringing uses of the technology. This process of assimilation of distribution technologies like P2P is familiar. From the Xerox machine, to the tape-deck, to the VCR, and even the DVD, each of these new distribution technologies has been initially seen as a threat to content creators. The advent of VCR technology was challenged and ultimately resolved by the Supreme Court. Ultimately, the film industry turned their immense creative talent toward building business models around the VCR technology, and they found an entirely new billion dollar distribution window waiting for them, bigger than all box office receipts combined. An equally lucrative opportunity exists with non-infringing business models built on P2P. That enormous opportunity in P2P is the holy grail of digital distribution. High Quality. Low Cost. Universal Access. Instant Gratification. Without P2P, digital distribution of film and video is not viable. With P2P, billions of dollars in network upgrades can be spared, and higher quality entertainment services are possible many years sooner. Even in its short, controversial history it clearly has shown the kind of prodigious demand for digital distribution that represents another billion-dollar opportunity latent in waiting. Red Swoosh is a tangible example of the beginning of harnessing that demand. Across the almost 100 media customers of our technology, we are distributing millions of files of promotional studio content every month. With all of the noise of litigation there is a very important story that unfortunately won?t make it to the headlines. That story is that the media industry is beginning to see the power of P2P. Like Shawn Fanning with Snocap, and myself with Red Swoosh, the media industry too has begun to turn the corner. They are now embracing P2P?s non-infringing applications as flashpoints for innovation in their own businesses. The MPAA?s statement to the FTC is illuminating and I quote: ?it is quite possible that P2P technology could become the DVD of the future: offering great content to consumers with protections that preserve the rights of creators?. We should still be realistic. The roads between the technologists and the media may be converging, but there is still a lot of work ahead of us. The message we are sending to the entertainment industry is bold. The tools in the fight against piracy are no longer exclusive to litigation. In this new phase of technology adoption, it is time for the entertainment industry to actively support and utilize the non-infringing applications of P2P. By doing so they can set the course so that legitimate distribution will soon dwarf illicit distribution, in the value delivered to consumers, and the commerce generated with new business models. Eventually, the power of the partnership between traditional entertainment and innovative technology will win out and piracy will fade into the background. The message to the entertainment industry is that the next phase is about to begin, and the new secret weapon is to embrace the technology once feared most. In the arms race against piracy, supporting and promoting the non-infringing uses of P2P will prove one of the most effective strategies. On the eve of the P2P FTC hearings, we believe there is also a role for regulators as well. With malicious intent of certain actors there have arisen malicious business models from some of the more infringing applications of P2P. The harm caused by the bundling of adware, spyware, malware, and their kin has resulted in billions of dollars in real damages. The abuse of users? trust and privacy though has had a much greater negative impact on society and technological progress. Because of the aggressive and surreptitious activities by bad actors in the marketplace, consumers no longer feel safe online and now are reticent of participating in technological progress. We cannot expect major investment in new technologies and new innovations when the mass consumer and audience fears of damage to personal property and the misuse of their personal information. A marketplace that consumers are afraid to take part in is not a marketplace at all. The curtailing and constraint of these more malicious online activities will go a long way in convincing the stakeholders to participate in what will ultimately be a powerful marketplace indeed. In closing I would like to thank the MPAA for having me join them at this event. It shows that while they protect their members? copyrights, they are also making strides in understanding and promoting the productive uses of P2P technology. Travis Kalanick Red Swoosh, Inc. Founder, Chairman travis@redswoosh.net (v) 310.666.1429 (f) 253.322.9478 AIM: ScourTrav123 From coderman at peertech.org Wed Dec 15 06:11:56 2004 From: coderman at peertech.org (coderman) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems Message-ID: <41BFD5AC.1080904@peertech.org> http://pds.twi.tudelft.nl/~pawel/pub/btmeasurement.pdf Interesting results. It took 5 days of seeding a 1.87G file before it started completing among participating peers. Given the MPAA action against trackers and individuals this suggests their attacks will be effective against many in the current architecture. [I'm not going to argue copyright issues; I will simply state that where corporations and/or governments can influence availability for legitimate reasons they can also do the same for less noble purposes.] The need for decentralization of trackers is mentioned along with the caveat that such changes increase vulnerability to other attacks. "One of the big advantages of BitTorrent/Suprnova is the high level of integrity of both the content and the meta-data due to the working of its global components. We have shown that only 20 moderators combined with numerous other volunteers solve the fake-file problem on BitTorrent/Suprnova. However, this comes at a price: system availability is hampered by the global nature of these components. Decentralization would provide an obvious solution, but makes the meta-data more vulnerable. Also, a decentralized scheme such as in Kazaa has no availability problems but lacks integrity, since Kazaa is plagued with many fake files. Clearly, decentralization is an unsolved issue that needs further research." We have been talking about this in another channel and a few thoughts were discussed. There has been much talk of private social group networks / social networks with file sharing. These services are usually centralized and thus constrained by contributory / vicarious liability concerns: they would have to police the networks. Same set of issues. [or some like to claim decentralization while relying on a trusted certificate authority or other index. heh] The decentralized variants have their own set of problems, mainly identity and reputation as well as discovery / search. Consider a live linux dvd (4.8G / 8.5G) that contains all of the software you need to participate in a private network, pre distribution of keys for all the peers known to the one who created it, and an innate ability for the dvd to clone itself - this provides a distribution mechanism for keys (identity), applications themselves (which are currently being targeted) and also caching for 4-8G of resources (initial introduction / resource discovery / search in replicated indexes). Passing a few dvd-r's between friends could approach some significant bandwidth levels and provide a good base of seeds for things like ISO distribution where the initial seeding takes a significant amount of time. (this is something i'm playing with: a live linux ISO that includes a bittorrent client and the ability to seed itself once booted) --- There was a paper posted a few months back with an excellent survey of search in decentralized networks: http://uluru.ee.unsw.edu.au/~john/tr-unsw-ee-p2p-1-1.pdf A number of the methods mentioned could benefit from a replicating ISO bootstrap like described above. Regards, From coderman at peertech.org Wed Dec 15 06:21:51 2004 From: coderman at peertech.org (coderman) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <41BFD5AC.1080904@peertech.org> References: <41BFD5AC.1080904@peertech.org> Message-ID: <41BFD7FF.7060701@peertech.org> coderman wrote: > A number of the methods mentioned could benefit from a replicating > ISO bootstrap like described above. I forgot to mention another paper on percolation search which shows the benefits of aggressive caching for search / discovery efficiency (although this is of secondary focus in the paper) http://arxiv.org/abs/cond-mat/0406152 4-8G is a plentiful cache space... Regards, From greg at electricrain.com Wed Dec 15 09:02:19 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Red Swoosh/Travis Kalanick comments In-Reply-To: <4748.12.170.144.61.1103087648.squirrel@webmail.redswoosh.net> References: <4748.12.170.144.61.1103087648.squirrel@webmail.redswoosh.net> Message-ID: <20041215090219.GB12070@zot.electricrain.com> > That enormous opportunity in P2P is the holy grail of digital > distribution. High Quality. Low Cost. Universal Access. Instant > Gratification. Without P2P, digital distribution of film and video is not > viable. With P2P, billions of dollars in network upgrades can be spared, > and higher quality entertainment services are possible many years sooner. Those "billions of dollars in network upgrades" are shifted from the serving/hosting ISPs to the end peer/consumer ISPs. They don't magically disappear (though you could reasonably argue that their size changes). From list-p2phack at ruffledpenguin.org Wed Dec 15 09:41:55 2004 From: list-p2phack at ruffledpenguin.org (Adam Lydick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <41BFD5AC.1080904@peertech.org> References: <41BFD5AC.1080904@peertech.org> Message-ID: <1103103716.14163.19.camel@localhost.localdomain> On Tue, 2004-12-14 at 22:11 -0800, coderman wrote: > > We have been talking about this in another channel and a few > thoughts were discussed. > > There has been much talk of private social group networks / social > networks with file sharing. These services are usually centralized > and thus constrained by contributory / vicarious liability concerns: > they would have to police the networks. Same set of issues. > > [or some like to claim decentralization while relying on a trusted > certificate authority or other index. heh] > > The decentralized variants have their own set of problems, mainly > identity and reputation as well as discovery / search. Perhaps not the best possible world, but why not a decentralized model that is enhanced by (but does not require) a centralized authority? This seems to work fairly well for a number of existing networks. If your trusted authority of choice is subjected to legal or technical DoS attacks you can always fall back to another option: * durable identifiers that you've already acquired * an alternate "central" (federated might be a better term) authority * exchange durable, attack resistant identifiers (eg: content hash) over existing social channels (email sigs, blogs, IM, SMS, ...) * keyword (or other) search without trust metrics (at least the network still works, you just might have a higher signal:noise for certain queries) -- Adam Lydick From em at em.no-ip.com Wed Dec 15 10:03:43 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> Message-ID: <00bf01c4e28d$60589260$0200a8c0@em.noip.com> ----- Original Message ----- From: "Adam Lydick" To: "Peer-to-peer development." Sent: Wednesday, December 15, 2004 5:41 PM > On Tue, 2004-12-14 at 22:11 -0800, coderman wrote: > > > > > > > We have been talking about this in another channel and a few > > thoughts were discussed. > > > > There has been much talk of private social group networks / social > > networks with file sharing. These services are usually centralized > > and thus constrained by contributory / vicarious liability concerns: > > they would have to police the networks. Same set of issues. > > > > [or some like to claim decentralization while relying on a trusted > > certificate authority or other index. heh] > > > > The decentralized variants have their own set of problems, mainly > > identity and reputation as well as discovery / search. > > > > Perhaps not the best possible world, but why not a decentralized model > that is enhanced by (but does not require) a centralized authority? This > seems to work fairly well for a number of existing networks. > > If your trusted authority of choice is subjected to legal or technical > DoS attacks you can always fall back to another option: > > * durable identifiers that you've already acquired > * an alternate "central" (federated might be a better term) authority > * exchange durable, attack resistant identifiers (eg: content hash) over > existing social channels (email sigs, blogs, IM, SMS, ...) > * keyword (or other) search without trust metrics (at least the network > still works, you just might have a higher signal:noise for certain > queries) Why not use pseudonymous digital signatures on both file content's hash and metadata? The signing "moderators", self-appointed and identified only by the fingerprint of their public key, could be rated by reputation, with a subjective score that each user would adjust every time a endorsement is found to be truthful or misleading. Enzo From eugen at leitl.org Wed Dec 15 10:47:20 2004 From: eugen at leitl.org (Eugen Leitl) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Red Swoosh/Travis Kalanick comments In-Reply-To: <20041215090219.GB12070@zot.electricrain.com> References: <4748.12.170.144.61.1103087648.squirrel@webmail.redswoosh.net> <20041215090219.GB12070@zot.electricrain.com> Message-ID: <20041215104720.GU9221@leitl.org> On Wed, Dec 15, 2004 at 01:02:19AM -0800, Gregory P. Smith wrote: > Those "billions of dollars in network upgrades" are shifted from the > serving/hosting ISPs to the end peer/consumer ISPs. They don't > magically disappear (though you could reasonably argue that their size > changes). Bandwidth is cheap on the local loop. Stackable Fast Ethernet switches cost very little/port these days, even Level 3 switches. Peering is what is expensive. If these ISP be smart, they'd deploy custom P2P which limits traffic to their own network, only using dedicated gateways, to reduce redundant traffic outside of their network. -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041215/40f4a7d9/attachment.pgp From list-p2phack at ruffledpenguin.org Wed Dec 15 11:15:06 2004 From: list-p2phack at ruffledpenguin.org (Adam Lydick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems In-Reply-To: <00bf01c4e28d$60589260$0200a8c0@em.noip.com> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <00bf01c4e28d$60589260$0200a8c0@em.noip.com> Message-ID: <1103109306.14163.35.camel@localhost.localdomain> On Wed, 2004-12-15 at 18:03 +0800, Enzo Michelangeli wrote: > Why not use pseudonymous digital signatures on both file content's hash > and metadata? The signing "moderators", self-appointed and identified only > by the fingerprint of their public key, could be rated by reputation, with > a subjective score that each user would adjust every time a endorsement is > found to be truthful or misleading. > > Enzo That could also be a reasonable approach. By making the trusted authority pseudoanonymous, they become better hardened against DoS (which is the primary difficulty with centralized systems). However, only positive ratings tend to be useful, in such as system, as pseudoanonymous identities are "throwaway" by design. If I'm an evil attacker trying to raise the rating of my fake/hostile content, I'll just discard my current identity as soon as users identify it as hostile. Making identities somewhat expensive to create might be an interesting solution to this problem (eg: identity is public key + a public-key-bound proof of work), although I worry that attackers with a moderate amount of resources could still abuse the process. (a normal user isn't going to leave a small network of machines running 24/7 to generate new identities). In the end, I think that the best solution is to default to "untrusted/semi-trusted" and manually add authorities as you discover them to be helpful/truthful. I find "word of mouth" to be much more effective than any automated system when making trust decisions. (eg: the recommendation of a knowledgable co-worker vs. Google's PageRank, Amazon reviews, or epinions) - Adam From em at em.no-ip.com Wed Dec 15 12:50:55 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems References: <41BFD5AC.1080904@peertech.org><1103103716.14163.19.camel@localhost.localdomain><00bf01c4e28d$60589260$0200a8c0@em.noip.com> <1103109306.14163.35.camel@localhost.localdomain> Message-ID: <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> ----- Original Message ----- From: "Adam Lydick" To: "Peer-to-peer development." Sent: Wednesday, December 15, 2004 7:15 PM [...] > However, only positive ratings tend to be useful, in such as system, as > pseudoanonymous identities are "throwaway" by design. If I'm an evil > attacker trying to raise the rating of my fake/hostile content, I'll > just discard my current identity as soon as users identify it as > hostile. It may be preferable to blacklist for a while the identities found to be hostile: running into their evil deeds over and over again may be too time and bandwidth consuming. On the other hand, a newcomer must necessarily be given the benefit of doubt, or else it won't ever get a chance of increasing its reputation. > Making identities somewhat expensive to create might be an interesting > solution to this problem (eg: identity is public key + a > public-key-bound proof of work), although I worry that attackers with a > moderate amount of resources could still abuse the process. (a normal > user isn't going to leave a small network of machines running 24/7 to > generate new identities). They could, but the attack would be slowed down. One could use some Dworkian memory-bound POW in order to level the playing field. > In the end, I think that the best solution is to default to > "untrusted/semi-trusted" and manually add authorities as you discover > them to be helpful/truthful. I find "word of mouth" to be much more > effective than any automated system when making trust decisions. (eg: > the recommendation of a knowledgable co-worker vs. Google's PageRank, > Amazon reviews, or epinions) Sure, I was thinking of automating the usage of the list, not its construction: a user should always be able to import new identities through a semi-manual procedure. Identities themselves could be propagated peer-to-peer and endorsed with signatures by other already trusted identities, WoT-style. But it would also be nice to have a user-friendly UI that, after accessing a resource, asked: "Is the file I just fetched you a good one?" and if the answer is "no", it should automatically blacklist all the identities that had endorsed it, and maybe their endorsers as well. Enzo From jcea at argo.es Wed Dec 15 14:39:00 2004 From: jcea at argo.es (Jesus Cea) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <41BFD5AC.1080904@peertech.org> References: <41BFD5AC.1080904@peertech.org> Message-ID: <41C04C84.9000003@argo.es> coderman wrote: > "One of the big advantages of BitTorrent/Suprnova is the high level of > integrity of both the content and the meta-data due to the working of > its global components. We have shown that only 20 moderators > combined with numerous other volunteers solve the fake-file problem > on BitTorrent/Suprnova. However, this comes at a price: system > availability is hampered by the global nature of these components. The solution is obvious: use cryptographic signatures. I could download a torrent file from anywhere, check the signature against the suprnova public key and be sure that: a) The torrent was validated by suprnova. b) The torrent was not altered in any way. In fact, torrents could be signed by anyone. You simply check signatures from people you trust. No, signatures doesn't need to be integral part of the torrent file. You can generate "detached" signatures. So no coordination neccesary between torrent creators and signers. So suprnova could simply publish signatures using, for example, RSS or usenet. Or a daily signature ZIP via emule or something like that. -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From jdd at dixons.org Wed Dec 15 15:22:47 2004 From: jdd at dixons.org (Jim Dixon) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Red Swoosh/Travis Kalanick comments In-Reply-To: <20041215104720.GU9221@leitl.org> References: <4748.12.170.144.61.1103087648.squirrel@webmail.redswoosh.net> <20041215090219.GB12070@zot.electricrain.com> <20041215104720.GU9221@leitl.org> Message-ID: On Wed, 15 Dec 2004, Eugen Leitl wrote: > > Those "billions of dollars in network upgrades" are shifted from the > > serving/hosting ISPs to the end peer/consumer ISPs. They don't > > magically disappear (though you could reasonably argue that their size > > changes). > > Bandwidth is cheap on the local loop. The local loop is generally copper and expensive. Even if it's fibre, someone has to pay to lay it. In the UK, as I recall, digging a trench costs about 100 pounds ($185 or so) per meter. > Stackable Fast Ethernet switches cost > very little/port these days, even Level 3 switches. Peering is what is > expensive. Peering is settlement-free. That is, the only cost is infrastructure. It might be private, in which case the cost is the cable run (say a couple of hundred dollars). Or it might be through an Internet exchange, in which case the cost is your fee for belonging to the exchange. In either case, peering is very very low cost so long as you are exchanging a reasonable volume of traffic. Perhaps you mean transit, which is indeed expensive. When you pay the other guy to carry your traffic, it's transit. When he carries your traffic for free (and vice-versa), it's peering. > If these ISP be smart, they'd deploy custom P2P which limits traffic to their > own network, only using dedicated gateways, to reduce redundant traffic > outside of their network. It's difficult to understand this. ISPs need to provide a reliable service. This can be done only if they control the hardware. In fact almost all of the services provided by an ISP involve p2p technology. ISPs peer with one another using BGP4; global routing is managed by routers constantly chattering to one another, a process called BGP peering. Internal routing is done similarly, using OSPF, ISIS, or other p2p protocols. The domain name system has many peer to peer elements. Web traffic is invisibly proxied. But providing a reliable service is absolutely dependent upon professional management and control of hardware. As the term is generally used on this list, p2p involves distribution of control. How could an ISP rely upon p2p running on client machines? Wouldn't multicast make a lot more sense for distributing movies and such? -- Jim Dixon jdd@dixons.org tel +44 117 982 0786 mobile +44 797 373 7881 http://jxcl.sourceforge.net Java unit test coverage http://xlattice.sourceforge.net p2p communications infrastructure From coderman at gmail.com Wed Dec 15 18:19:07 2004 From: coderman at gmail.com (Martin Peck) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <1103103716.14163.19.camel@localhost.localdomain> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> Message-ID: <4ef5fec6041215101974b7f8f0@mail.gmail.com> On Wed, 15 Dec 2004 01:41:55 -0800, Adam Lydick wrote: > ... > Perhaps not the best possible world, but why not a decentralized model > that is enhanced by (but does not require) a centralized authority? This > seems to work fairly well for a number of existing networks. It is usually very easy to move from a fully decentralized model to a more centralized one. It is much harder to go the other direction. My point is that you need a fully decentralized system to avoid the weaknesses / vulnerabilities associated with a central identity server (CA, single signon, social net sites, etc). I agree that it would be useful to support other systems as well, though they would not be the base upon which identity is built, but rather augment the decentralized identities within the system... From coderman at gmail.com Wed Dec 15 18:24:09 2004 From: coderman at gmail.com (Martin Peck) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <41C04C84.9000003@argo.es> References: <41BFD5AC.1080904@peertech.org> <41C04C84.9000003@argo.es> Message-ID: <4ef5fec604121510243f3d320e@mail.gmail.com> On Wed, 15 Dec 2004 15:39:00 +0100, Jesus Cea wrote: > ... > The solution is obvious: use cryptographic signatures. Yes, works great. The hard problem is secure key distribution for strong peer identities. One reason I like the pre-distribution via DVD is that you can fit tens of thousand of pub keys on a disc with little space and it is exchanged out of band. You could leverage these peers for discovery of new ones online and be sure that those exchanges are suitably protected (since they are using sessions initialized with keys traded out of band on read-only media long before...) This is probably overkill for most peer network tasks, but in a wireless environment good key distribution becomes much more important given the ease of active man-in-the-middle attacks. From coderman at gmail.com Wed Dec 15 18:34:00 2004 From: coderman at gmail.com (Martin Peck) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems In-Reply-To: <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <00bf01c4e28d$60589260$0200a8c0@em.noip.com> <1103109306.14163.35.camel@localhost.localdomain> <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> Message-ID: <4ef5fec60412151034c7e996e@mail.gmail.com> On Wed, 15 Dec 2004 20:50:55 +0800, Enzo Michelangeli wrote: > ... > Sure, I was thinking of automating the usage of the list, not its > construction: a user should always be able to import new identities > through a semi-manual procedure. Identities themselves could be propagated > peer-to-peer and endorsed with signatures by other already trusted > identities, WoT-style. But it would also be nice to have a user-friendly > UI that, after accessing a resource, asked: "Is the file I just fetched > you a good one?" and if the answer is "no", it should automatically > blacklist all the identities that had endorsed it, and maybe their > endorsers as well. I am a huge fan of implicit feedback systems. One of the goals of the feedback filesystem I have been working on is to make the relative value of file resources determines implicit based on usage patterns. If you download a file and then read the first 10% of it, then delete it, that is a very strong indicator that the resource was a poor match or of low quality. If you download it, open all of it (full play, etc) and copy or rename, that indicates it is probably useful. Likewise you can track the number of reads this file has compared to others within a directory structure and see how popular they are based solely on observed interactions with the filesystem. Rating resources in this manner allows you to tie the feedback to peer identities (this peer has given me many good resources, while this other one has provided only crap) as well as recommender systems (these are my top favorite files, you might like them too). In feedbackfs I track usage by userID, application, and processID so that you can separate disparate resources into specific domains (video files opened by xine, music files opened by xmms, pdf's opened by userID $foo, etc) as you see fit. I think implicit feedback will become increasingly important and useful for reputation in decentralized networks. (And can be augmented by explicit feedback, but does not require it) From coderman at gmail.com Wed Dec 15 18:58:53 2004 From: coderman at gmail.com (Martin Peck) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems In-Reply-To: <4ef5fec60412151034c7e996e@mail.gmail.com> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <00bf01c4e28d$60589260$0200a8c0@em.noip.com> <1103109306.14163.35.camel@localhost.localdomain> <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> <4ef5fec60412151034c7e996e@mail.gmail.com> Message-ID: <4ef5fec60412151058495d3e1e@mail.gmail.com> On Wed, 15 Dec 2004 10:34:00 -0800, Martin Peck wrote: > ... > Rating resources in this manner allows you to tie the feedback to peer > identities (this peer has given me many good resources, while this > other one has provided only crap) as well as recommender systems > (these are my top favorite files, you might like them too). Another useful application of implicit feedback is for cache control. Aggressive caching is very important for decentralized networks, and "knowing when to forget" is a tricky problem. This is well described in the Elephant File System paper: http://citeseer.ist.psu.edu/santry99deciding.html The application of these techniques is well suited to other forms of cached information, metadata, indexes, etc. From list-p2phack at ruffledpenguin.org Wed Dec 15 19:23:02 2004 From: list-p2phack at ruffledpenguin.org (Adam Lydick) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <4ef5fec6041215101974b7f8f0@mail.gmail.com> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <4ef5fec6041215101974b7f8f0@mail.gmail.com> Message-ID: <1103138582.16331.3.camel@localhost.localdomain> On Wed, 2004-12-15 at 10:19 -0800, Martin Peck wrote: > On Wed, 15 Dec 2004 01:41:55 -0800, Adam Lydick > wrote: > > ... > > Perhaps not the best possible world, but why not a decentralized model > > that is enhanced by (but does not require) a centralized authority? This > > seems to work fairly well for a number of existing networks. > > It is usually very easy to move from a fully decentralized model to a > more centralized one. It is much harder to go the other direction. > > My point is that you need a fully decentralized system to avoid the > weaknesses / vulnerabilities associated with a central identity server > (CA, single signon, social net sites, etc). I agree that it would be > useful to support other systems as well, though they would not be the > base upon which identity is built, but rather augment the > decentralized identities within the system... I completely agree with that. My reply was poorly worded, as it implied that the central portion should be "designed in". As you noted, this would be unwise. From floydophone at gmail.com Wed Dec 15 21:13:02 2004 From: floydophone at gmail.com (Peter Hunt) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Kademlia Message-ID: <6654eac404121513133331d59b@mail.gmail.com> Can someone give me some links to Kademlia papers/pseudocode/example simple implementations? I've seen the presentation and a thirty page or so paper on it, but I haven't been able to Google much else on it. Thanks a lot! Peter Hunt From mgp at ucla.edu Wed Dec 15 22:05:47 2004 From: mgp at ucla.edu (Michael Parker) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Kademlia In-Reply-To: <6654eac404121513133331d59b@mail.gmail.com> References: <6654eac404121513133331d59b@mail.gmail.com> Message-ID: <41C0B53B.2080407@ucla.edu> http://www.cs.ucla.edu/~mgp/p2p/kademlia1.pdf This is the most descriptive and thorough paper on Kademlia I know of. Alternatively, you could look at the code of some open-source programs using Kademlia (eMule comes to mind), or simulators that implement Kademlia (such as p2psim). - Michael Parker Peter Hunt wrote: >Can someone give me some links to Kademlia papers/pseudocode/example >simple implementations? I've seen the presentation and a thirty page >or so paper on it, but I haven't been able to Google much else on it. > >Thanks a lot! > >Peter Hunt >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > From dbarrett at quinthar.com Wed Dec 15 23:33:54 2004 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems In-Reply-To: <4ef5fec60412151034c7e996e@mail.gmail.com> References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <00bf01c4e28d$60589260$0200a8c0@em.noip.com> <1103109306.14163.35.camel@localhost.localdomain> <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> <4ef5fec60412151034c7e996e@mail.gmail.com> Message-ID: On Wed, 15 Dec 2004, Martin Peck wrote: > I am a huge fan of implicit feedback systems. > ... > I think implicit feedback will become increasingly important and > useful for reputation in decentralized networks. (And can be > augmented by explicit feedback, but does not require it) I think Martin has touched on an under-discussed topic: obtaining reliable quality measures through pure observation. That's an interesting approach to do it at the file system level, but we might also tackle it at the application level. Are there any good examples of content-playback applications integrating with content-distribution systems to obtain implicit quality measures? I could imagine a decentralized, open protocol where every time I listen to a song (for example) WinAmp spits out a digitally signed "I played this song to completion, and here's its CRC" into the ether. The sum total of these certificates is the implict quality of the track. Naturally there are the standard reputation system problems, but assuming a decent reputation system existed already, this would be one way to leverage it for implicit quality measures. Is there anything like this already working or in the works? -david From kit at unadopted.com Wed Dec 15 23:48:12 2004 From: kit at unadopted.com (Continuity) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Kademlia In-Reply-To: <6654eac404121513133331d59b@mail.gmail.com> References: <6654eac404121513133331d59b@mail.gmail.com> Message-ID: <41C0CD3C.8040201@unadopted.com> Peter Hunt wrote: > Can someone give me some links to Kademlia papers/pseudocode/example > simple implementations? I've seen the presentation and a thirty page > or so paper on it, but I haven't been able to Google much else on it. I think there's a python implementation of it called SharkyPy, but I haven't looked at it. There's also this thread on this list about it not self-healing (ie re-joining a split network): http://zgp.org/pipermail/p2p-hackers/2003-August/001344.html which put me off it a little, but then I don't really understand it yet.. From adam at cypherspace.org Thu Dec 16 01:56:04 2004 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:12:44 2006 Subject: distributed document popularity metrics using amortizable hashcash (Re: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems) In-Reply-To: References: <41BFD5AC.1080904@peertech.org> <1103103716.14163.19.camel@localhost.localdomain> <00bf01c4e28d$60589260$0200a8c0@em.noip.com> <1103109306.14163.35.camel@localhost.localdomain> <017d01c4e2a4$d91bacc0$0200a8c0@em.noip.com> <4ef5fec60412151034c7e996e@mail.gmail.com> Message-ID: <20041216015604.GA29937@bitchcake.off.net> On Wed, Dec 15, 2004 at 03:33:54PM -0800, David Barrett wrote: > Are there any good examples of content-playback applications > integrating with content-distribution systems to obtain implicit > quality measures? > > I could imagine a decentralized, open protocol where every time I > listen to a song (for example) WinAmp spits out a digitally signed > "I played this song to completion, and here's its CRC" into the > ether. The sum total of these certificates is the implict quality > of the track. I proposed a way to do this without a reputation system: amorizable hashcash. http://www.hashcash.org/papers/amortizable.pdf It is designed to not need a reputation system. There are no certificates and each user essentially gets to vote on the document popularity (through implicit or explicit action) in a distributed fashion. To vote the user expends CPU time to create a hashcash stamp tied to the document id. (Eg lower priority background thread during playback for implicit, for some duration as a background thread after marking it as quality content for explicit). Stamps have a special property that you can add them, and probabilistically they resulting stamp is equal to the sum of the input stamps. (Adding the same stamp repeatedly has no effect). (I call this probabilistic addition "amortization"). The size of the stamp does not grow. Documents are requested from peers, and with the requester sends his view of the documents popularity stamp value. The server adds the requesters stamp to it's own stamp, and sends the new stamp in the response. So each peer sharing a document sums the stamps it sees, and each peer that downloads a document sends stamps it sees from other peers it is swarming from. So (for swarming) no additional connections are needed for popularity metrics. The only comms overhead is the the size of the stamp which is relatively compact. The exact size of the stamp depends on how you tune it -- you can reduce the error introduced by the randomness by increasing the size. The smallest stamp is going to be around 16 bytes, and reasonably low error rate ones maybe 100 bytes. What can cheaters do, there two types of cheating: 1. jammers trying to mark junk/mislabeled documents as high quality - We don't try to do anything about the jammer. Best case the p2p net has significantly more CPU resources than the jammer. Worst case the jammer is the RIAA or MPAA and they blow a bundle on server racks. But generally the kazza net would be a pretty decent super-computer: 1 million online peers is a lot of compute if you're paying for it. 2. non-participants who want to download and view the file, but for some reason do not want to contribute to it's popularity - this probably isn't a signifcant concern. It can be combatted to a limited extent as described in the paper see "fair amortizable cost-function" The paper in hind-sight does not really explain the above in much detail though this was the purpose of it. (The paper is mostly crypto math showing how and why it works). But the actual stuff is pretty simple overall. There is source code for basic hashcash (non-amortizable) here: http://www.hashcash.org/ Amortizable hashcash (with worst case error rate) is to "add" hashcash stamps x and y by retaining the larger x+y = max(x,y). (The addition is kind of logarithmic, but you can adjust for it to have a hits estimate). For lower error rate with let's say with 16 stamps you keep the largest 16 stamps you've seen: say X = (x1,...,x16) and Y = (y1,...,y16) then X+Y= top16(sort(X,Y)) (ie take the biggest 16 of the combined list). Higher level things: A. you can have negative votes also, just define other properties -- rather than quality vote, a negative vote saying a document is line noise or mis-labelled. B. you can vote on other things, eg peer aggregate contribution (in terms of MB shared), performance, reliability, uptime C. you could weight document popularity voting by the contribution level of the voter (the RIAA has to share lots of good stuff to be able to jam other stuff:-). D. other than contribution have a pseudonym that has a cost: Say that each user who joins the p2p has to create a relatively expensive stamp on their pseudonym. Now when they publish stamps their stamps are ignored unless accompanied by a pseudonym stamp. E. pseudonyms could be meta-voted on. If a jammer creates a pseudonym and starts to vote heavily (with rented super-computer time trying to compete with kazza users), his pseudonym gets negatively voted on and he has to discard the pseudonym and create another one. (Of course the jammer can go ahead and falsely vote against everyone else's pseudonyms, but it's 1 against many, and they all can vote against the jammer). What about the CPU waste? So one might think this leads to wasted CPU resources, but the scarce resources a p2p net is trying to conserve are: bandwidth, and human satisfaction (getting what they expect). I also proposed that in p2p sytems that do cacheing (and works for pre-fetching) that the cache replacement policy is biased to favor popular content. Adam > Naturally there are the standard reputation system problems, but > assuming a decent reputation system existed already, this would be one > way to leverage it for implicit quality measures. > > Is there anything like this already working or in the works? From em at em.no-ip.com Thu Dec 16 01:59:06 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Kademlia References: <6654eac404121513133331d59b@mail.gmail.com> Message-ID: <037301c4e312$e880fb20$0200a8c0@em.noip.com> My KadC library allows to access thye Kademlia-based DHT of Overnet/eDonkey1.0: http://kadc.sourceforge.net/ At the moment I haven't yet implemented the part that allows to participate to the DHT rather than simply using it, but it shouldn't be very hard to do. In other words, KadC in its present state allows to write applications behaving as Overnet "leaf nodes", rather than "supernodes/ultrapeers" (in Kazaa/Gnutella parlance). That's not very limiting because the number of Overnet/eDonkey peers online at any given time is around several hundred thousand (it's only a bit selfish :-) ): the application can freely publish and retrieve small dictionaries in the format that Overnet uses for metadata, but containing arbitrary data. If the index is properly chosen, no collision will ever occur with DHT records used for filesharing purposes. Enzo ----- Original Message ----- From: "Peter Hunt" To: Sent: Thursday, December 16, 2004 5:13 AM Subject: [p2p-hackers] Kademlia > Can someone give me some links to Kademlia papers/pseudocode/example > simple implementations? I've seen the presentation and a thirty page > or so paper on it, but I haven't been able to Google much else on it. > > Thanks a lot! > > Peter Hunt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From eugen at leitl.org Thu Dec 16 07:50:57 2004 From: eugen at leitl.org (Eugen Leitl) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] P2P In 15 Lines of Code Message-ID: <20041216075057.GL9221@leitl.org> Link: http://slashdot.org/article.pl?sid=04/12/15/1953227 Posted by: timothy, on 2004-12-15 21:00:00 from the practice-and-theory dept. nile_list writes "Edward Felten of the very fine [1]Freedom to Tinker has written a [2]15 line P2P program in Python. From the [3]post on Freedom to Tinker, "I wrote TinyP2P to illustrate the difficulty of regulating peer-to-peer applications. Peer-to-peer apps can be very simple, and any moderately skilled programmer can write one, so attempts to ban their creation would be fruitless." Matthew Scala, a reader of Freedom to Tinker, has responded with the [4]9 line MoleSter, written in Perl." IFRAME: [5]pos6 References 1. http://freedom-to-tinker.com/ 2. http://www.freedom-to-tinker.com/tinyp2p.html 3. http://www.freedom-to-tinker.com/archives/000738.html 4. http://ansuz.sooke.bc.ca/software/molester 5. http://ads.osdn.com/?ad_id=5819&alloc_id=12653&site_id=1&request_id=2200724 ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041216/77b94196/attachment.pgp From ian at locut.us Thu Dec 16 13:39:42 2004 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] P2P In 15 Lines of Code In-Reply-To: <20041216075057.GL9221@leitl.org> References: <20041216075057.GL9221@leitl.org> Message-ID: I haven't examined this too closely yet, but from the site: "The program creates a small-world network, which might be used by a group of friends or business associates to share files." I am wondering whether this is really a small-world network in the Kleinberg sense (http://citeseer.ist.psu.edu/kleinberg00smallworld.html), in which case it would be impressive and should be very scalable. My suspicion is that it is not, given that it appears to be a client-server system, given that they specifically state that it won't scale, and given that searches are conducted with regexps (where as conventional small-world based DHTs search for specific large numbers, not approximate regexps). As an aside, to the latter point, we have conducted research[1] where we have successfully generalised Freenet's search algorithm, which appears[2] to form a small world network (although more complicated than that modelled by Kleinberg), to search for "fuzzy" data rather than specific hashes as with a DHT. Having said that I think it is very unlikely that this is what they are doing here. Ian. [1] See http://freenetproject.org/kronfol_final_thesis.pdf [2] Expect to see a paper in the next few months providing mathematical support for this conjecture On 16 Dec 2004, at 07:50, Eugen Leitl wrote: > Link: http://slashdot.org/article.pl?sid=04/12/15/1953227 > Posted by: timothy, on 2004-12-15 21:00:00 > > from the practice-and-theory dept. > nile_list writes "Edward Felten of the very fine [1]Freedom to > Tinker > has written a [2]15 line P2P program in Python. From the [3]post on > Freedom to Tinker, "I wrote TinyP2P to illustrate the difficulty of > regulating peer-to-peer applications. Peer-to-peer apps can be very > simple, and any moderately skilled programmer can write one, so > attempts to ban their creation would be fruitless." Matthew Scala, a > reader of Freedom to Tinker, has responded with the [4]9 line > MoleSter, written in Perl." > > IFRAME: [5]pos6 > > References > > 1. http://freedom-to-tinker.com/ > 2. http://www.freedom-to-tinker.com/tinyp2p.html > 3. http://www.freedom-to-tinker.com/archives/000738.html > 4. http://ansuz.sooke.bc.ca/software/molester > 5. > http://ads.osdn.com/? > ad_id=5819&alloc_id=12653&site_id=1&request_id=2200724 > > ----- End forwarded message ----- > -- > Eugen* Leitl leitl > ______________________________________________________________ > ICBM: 48.07078, 11.61144 http://www.leitl.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE > http://moleculardevices.org http://nanomachines.net > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -- Founder, The Freenet Project http://freenetproject.org/ CEO, Cematics Ltd http://cematics.com/ Personal Blog http://locut.us/~ian/blog/ From jcea at argo.es Thu Dec 16 16:17:02 2004 From: jcea at argo.es (Jesus Cea) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] BitTorrent measurements / fully decentralized systems In-Reply-To: <4ef5fec604121510243f3d320e@mail.gmail.com> References: <41BFD5AC.1080904@peertech.org> <41C04C84.9000003@argo.es> <4ef5fec604121510243f3d320e@mail.gmail.com> Message-ID: <41C1B4FE.7080507@argo.es> > Yes, works great. The hard problem is secure key distribution for > strong peer identities. You usually are only concerned about user recognizion, not identification. That is, your interest is in "recognizing" users you trust, but you don't need the real identity. So the easy way could be simply use a reputation system. You download content and if the quality pleases you, you give points to the signers. If the content is a fake, you blacklist the signers. Then you can trust quality simply checking signatures in the "most trusted public keys" group. -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From floydophone at gmail.com Thu Dec 16 16:58:47 2004 From: floydophone at gmail.com (Peter Hunt) Date: Sat Dec 9 22:12:44 2006 Subject: [p2p-hackers] Re: Kademlia Message-ID: <6654eac404121608581e21d1e6@mail.gmail.com> Do you have another DHT algorithm you can suggest? What's the "best" one right now? Thanks in advance. From ian at locut.us Thu Dec 16 17:03:40 2004 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:12:45 2006 Subject: [p2p-hackers] Re: Kademlia In-Reply-To: <6654eac404121608581e21d1e6@mail.gmail.com> References: <6654eac404121608581e21d1e6@mail.gmail.com> Message-ID: <6F021FF6-4F84-11D9-96C4-000D932C5880@locut.us> On 16 Dec 2004, at 16:58, Peter Hunt wrote: > Do you have another DHT algorithm you can suggest? What's the "best" > one right now? And while you are at it, which is better - Vi or Emacs? :-) Ian. -- Founder, The Freenet Project http://freenetproject.org/ CEO, Cematics Ltd http://cematics.com/ Personal Blog http://locut.us/~ian/blog/ From hal at finney.org Thu Dec 16 18:38:46 2004 From: hal at finney.org (Hal Finney) Date: Sat Dec 9 22:12:45 2006 Subject: [p2p-hackers] P2P In 15 Lines of Code Message-ID: <20041216183846.71F7057E2D@finney.org> Ian Clarke writes: > "The program creates a small-world network, which might be used by a > group of friends or business associates to share files." > > I am wondering whether this is really a small-world network in the > Kleinberg sense > (http://citeseer.ist.psu.edu/kleinberg00smallworld.html), in which case > it would be impressive and should be very scalable. I think you're right, Ed Felten used the wrong phrase in describing this. Small-world networks have a fractal-like connection topology and scale very well. The term goes back at least to 1998. What Felten made was a network of the kind which is designed to be used by a small circle of friends, not part of a larger network. WASTE is probably the best known example of this kind of network. I'm not sure what the right name is for these. Sometimes they are called darknets but I think that term may be more generic, for any kind of network which is designed to resist surveillance. Hal Finney From bkn3 at columbia.edu Thu Dec 16 20:22:18 2004 From: bkn3 at columbia.edu (Brad Neuberg) Date: Sat Dec 9 22:12:45 2006 Subject: [p2p-hackers] Why BitTorrent is Important and SwarmStream is Doomed Message-ID: <6.1.0.6.2.20041216120912.023c0c08@pop.mail.yahoo.com> Hi folks. I've written up a short analysis on my blog of SwarmStream and BitTorrent titled "Why BitTorrent is Important and SwarmStream is Doomed". I am interested in people's feedback and/or criticism of these ideas. I am cross-posting this to the P2P Hackers list as well. The link: http://codinginparadise.org/weblog/2004/12/why-bittorrent-is-important-and.html Hope all is well, Brad Brad Neuberg, bkn3@columbia.edu Senior Software Engineer, Rojo Networks Weblog: http://www.codinginparadise.org ===================================================================== Check out Rojo, an RSS and Atom news aggregator that I work on. Visit http://rojo.com for more info. Feel free to ask me for an invite! Rojo is Hiring! If you're interested in RSS, Weblogs, Social Networking, Java, Open Source, etc... then come work with us at Rojo. If you recommend someone and we hire them you'll get a free iPod! See http://www.rojonetworks.com/JobsAtRojo.html. From hal at finney.org Thu Dec 16 21:56:59 2004 From: hal at finney.org (Hal Finney) Date: Sat Dec 9 22:12:49 2006 Subject: distributed document popularity metrics using amortizable hashcash (Re: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems) Message-ID: <20041216215659.E0E9B57E2C@finney.org> Adam Back writes: > I proposed a way to do this without a reputation system: amorizable > hashcash. http://www.hashcash.org/papers/amortizable.pdf > > It is designed to not need a reputation system. > > There are no certificates and each user essentially gets to vote on > the document popularity (through implicit or explicit action) in a > distributed fashion. To vote the user expends CPU time to create a > hashcash stamp tied to the document id. (Eg lower priority background > thread during playback for implicit, for some duration as a background > thread after marking it as quality content for explicit). This is a cool idea but I don't exactly understand your description of the details. > Documents are requested from peers, and with the requester sends his > view of the documents popularity stamp value. The server adds the > requesters stamp to it's own stamp, and sends the new stamp in the > response. The grammar here is confusing to me. How would a requester have a stamp before he has even downloaded the file? I thought someone would create a stamp after downloading, and after they have determined that the file is legit and not bogus. > So each peer sharing a document sums the stamps it sees, and each peer > that downloads a document sends stamps it sees from other peers it is > swarming from. So (for swarming) no additional connections are needed > for popularity metrics. The only comms overhead is the the size of > the stamp which is relatively compact. The idea here is that the downloader acts as a sort of clearinghouse for the stamps, combining them from all the peers he is swarm-downloading from, and sending back the summed stamp to all of the peers he is downloading from? This way the peers all get updates from each other, without having to connect directly to each other? I would have thought that the stamps would be used in two ways. First, a prospective downloader might see several candidate files in the network that all claim to be the one he wants. He wants to know which are legit and which are bogus, and the stamp value can give him a clue about this. So he'd want to compare stamps before downloading. And second, there then needs to be a feedback mechanism where, after downloading and inspecting the content, a new stamp is sent back into the network which allows the downloader to testify that the document was good (or bad, if negative stamps are used too). Anyway as I said it sounds like a good idea but I am a little unclear about the exact mechanisms and how the data would flow. I'd appreciate it if you could explain that again. Thanks, Hal Finney From coderman at gmail.com Thu Dec 16 22:22:15 2004 From: coderman at gmail.com (Martin Peck) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Why BitTorrent is Important and SwarmStream is Doomed In-Reply-To: <6.1.0.6.2.20041216120912.023c0c08@pop.mail.yahoo.com> References: <6.1.0.6.2.20041216120912.023c0c08@pop.mail.yahoo.com> Message-ID: <4ef5fec604121614227a8fa102@mail.gmail.com> On Thu, 16 Dec 2004 12:22:18 -0800, Brad Neuberg wrote: > Hi folks. I've written up a short analysis on my blog of SwarmStream and > BitTorrent titled "Why BitTorrent is Important and SwarmStream is > Doomed". I am interested in people's feedback and/or criticism of these > ideas. I think the two have very different problem domains. I can foresee both thriving in their respective environments. SwarmStream is more about accelerating content from authoritative servers via connection aggregation and intelligent caching. BitTorrent pertains more to decentralized distribution (no authoritative server, but instead a metadata descriptor with secure identifiers) using tit for tat to encourage cooperative behavior. While you could use one for the other purpose and vice versa, I expect each will be heading in more distinct directions over time rather than merging into the same problem space. And Justin touches on a legitimacy issue which will further play into the distinction between the two: SwarmStream is very much an enterprise solution; commercially supported, well managed with authoritative server configurations, etc. BitTorrent is often portrayed as a rogue piracy tool (in addition to its legitimate uses) which alienates the SwarmStream market to some degree. For these reasons I see them both serving the needs of their respective targets without cannibalizing each others user base. From adam at cypherspace.org Thu Dec 16 23:55:29 2004 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:12:49 2006 Subject: distributed document popularity metrics using amortizable hashcash (Re: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems) In-Reply-To: <20041216215659.E0E9B57E2C@finney.org> References: <20041216215659.E0E9B57E2C@finney.org> Message-ID: <20041216235529.GA29130@bitchcake.off.net> On Thu, Dec 16, 2004 at 01:56:59PM -0800, "Hal Finney" wrote: > > Documents are requested from peers, and with the requester sends his > > view of the documents popularity stamp value. The server adds the > > requesters stamp to it's own stamp, and sends the new stamp in the > > response. > > The grammar here is confusing to me. I was tired (2am) it was rushed. OK start again: Three cases A) selecting content, B) downloading content, C) voting on content. Even though it p2p I'll use server and client to mean the usual things. (A peer is just one or the other, or both in a given transaction). A) selecting content: the peer chooses the content that matches the search criteria and has the highest rating. B) downloading content: the client acts (as you said) as a clearing house for stamps from the servers it downloads from. Note it doesn't need to send stamps back to servers unless it does a calculation which gives a larger stamp than the server already has. C) voting on content: during playback (implicit) or after explicit voting action the peer votes on the content. We might have positive as well as negative votes, tracked separately, just add up their +ve and -ve normalized value to reach the overall value. (Normalization being what you have to do to factor out the logarithmic nature of the composite stamps. And that addition being standard floating point addition.) I tend to think it would be more effective to note vote directly on content but vote on the reputation of pseudonymous content raters. That way they build up a pretty heavy reputation which is going to be seriously expensive to compete with. > Anyway as I said it sounds like a good idea but I am a little unclear > about the exact mechanisms and how the data would flow. I'd appreciate > it if you could explain that again. Overall the basic mechanism is a distributed voting mechanism which introduces no extra communications purely for converging on and distributing the global sum of the votes. There can be multiple votes on any number of metrics, and meta-level votes (the content raters reputation). One could also build decay into the model so reputations fade unless maintained (or the popularity metric of a document fades). This is easy to do, just include the time in the stamp, and discard or reduce the value according to some formula of stamps as they age. eg. if you were using the document popularity metric to weight caches you might want to do this (depending on your availability criteria), eg to ensure that very historically very popular, but passe content does not hog cache space better used for new content. I suspect that films have this property ... every one wants it when its just out, after that rush it tapers off except for more classic genre setting films. As I mentioned (for others Hal knows all about the crypto) the actual crypto is pretty easy, and looking at the things you can do with it I think it would actually work pretty well in eg Kazaa or other similar networks if deployed. Maybe for meta-rating, you might even want to increase the weighting of rating with age, to make it hard for the johnny-come-lately jammers working for RIAA / MPAA et al to displace pseudonymous established players. (This would be a bit like Rivest & Shamir's micromint where the system accumulates advantage over outsiders attempting to forge). Also once you have established pseudonymous high rating rating service pseudonyms you could have cross-pseudonym ratings, which are conventional WoT signatures conveying big chunks of reputation. A jammer could spend lots of real-world money to buy enough CPU to compete with the bona-fide rating services. That reputation could be revoked with a signature revocation certificate if the rating service turned rogue (or got turned, private key compromised by jammer). Generally it provides an anchor point for a distribution focussed reputation system possibly solving the bootstrap problem. And people tend to use what works, so you could imagine a system where you pick rating services, and you don't add new ones unless the old ones stop rating enough content for you. The set of ratings services is probably small enough for people to apply human intelligence to in discussion forums etc. Of course the jammer may try to jam those also, but some of the good info will be exchanged out of band between social networks. Maybe we can even make it more expensive than it's worth to attack, the final measure of true security :-) Adam From hal at finney.org Fri Dec 17 06:42:33 2004 From: hal at finney.org (Hal Finney) Date: Sat Dec 9 22:12:49 2006 Subject: distributed document popularity metrics using amortizable hashcash (Re: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems) Message-ID: <20041217064233.4D28F57E2C@finney.org> Adam Back writes: > Three cases A) selecting content, B) downloading content, C) voting on > content. Even though it p2p I'll use server and client to mean the > usual things. (A peer is just one or the other, or both in a given > transaction). > [...] > C) voting on content: > > during playback (implicit) or after explicit voting action the peer > votes on the content. We might have positive as well as negative > votes, tracked separately, just add up their +ve and -ve normalized > value to reach the overall value. Would this require an extra communications step, to reconnect back into the network and provide feedback to the server(s) that you downloaded the song from? > I tend to think it would be more effective to note vote directly on > content but vote on the reputation of pseudonymous content raters. If we stick to the idea of voting on the content, are there some naming issues? I think the hashcash stamp would have to identify the content in some way. It's not clear to me if this should be by filename, by an internal title (e.g. from an MP3 tag), or by a crypto hash of the file itself. Maybe some combination of these? Hal Finney From adam at cypherspace.org Fri Dec 17 13:40:35 2004 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:12:49 2006 Subject: distributed document popularity metrics using amortizable hashcash (Re: [p2p-hackers] BitTorrent measurements / fully decentralizedsystems) In-Reply-To: <20041217064233.4D28F57E2C@finney.org> References: <20041217064233.4D28F57E2C@finney.org> Message-ID: <20041217134035.GA11702@bitchcake.off.net> On Thu, Dec 16, 2004 at 10:42:33PM -0800, "Hal Finney" wrote: > Adam Back writes: > > [...] > > C) voting on content: > > > > during playback (implicit) or after explicit voting action the peer > > votes on the content. We might have positive as well as negative > > votes, tracked separately, just add up their +ve and -ve normalized > > value to reach the overall value. > > Would this require an extra communications step, to reconnect back into > the network and provide feedback to the server(s) that you downloaded > the song from? Not necessarily, just associate it with the file locally and when the downloader acts as a server that information will get propagated. (This delays slightly but in a busy p2p net it'll get distributed fast enough anyway). Leechers would need an explicit connect (if someone disables sharing you would want to enable explicit connects). So one could have another optional step D): D) vote upload: optionally send the vote to one or more servers downloaded from. If user is operating in leech mode this connection is non-optional. > > I tend to think it would be more effective to note vote directly on > > content but vote on the reputation of pseudonymous content raters. > > If we stick to the idea of voting on the content, are there some naming > issues? I think the hashcash stamp would have to identify the content > in some way. It's not clear to me if this should be by filename, by > an internal title (e.g. from an MP3 tag), or by a crypto hash of the > file itself. Maybe some combination of these? Yes I was thinking the hash of the content and the document human readable name. (Well in fact the hash of the content should be a root hash from a Merkle Hash Tree based hash of the content to facilitate integrity verification during download so jammers can't send you bogus data that you only notice after downloading 1GB file and then don't know which chunks were bogus.) The voters should bind the name to their vote as well as the document hash otherwise you get the other kind of jamming: renaming good content so you get something high integrity but the wrong thing. Adam From kit at unadopted.com Sat Dec 18 20:38:06 2004 From: kit at unadopted.com (Continuity) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Re: Kademlia In-Reply-To: <6654eac404121608581e21d1e6@mail.gmail.com> References: <6654eac404121608581e21d1e6@mail.gmail.com> Message-ID: <41C4952E.8040003@unadopted.com> Peter Hunt wrote: > Do you have another DHT algorithm you can suggest? What's the "best" > one right now? Honestly, I don't know either enough about the subject or enough different algorithms to be able to answer that.. I'm quite (very) new to the subject.. I like some things about kademlia - up-time is inversely proportional to the probability of nodes going offline soon, for example.. so don't let me put you off looking at it, it's just that cleverer people than me have said it's not perfect :) Both chord and pastry(tapestry? I think it's the same 'family') seem active and well-researched, but I don't know much about their design. If anyone else could answer this question better, or at least give an overview of what realistic options developers have right now, that would be nice =) From mgp at ucla.edu Sun Dec 19 04:30:25 2004 From: mgp at ucla.edu (Michael Parker) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Re: Kademlia Message-ID: <200412190430.iBJ4UPX21567@webmail.my.ucla.edu> Kademlia, Chord, Pastry... They all have different underlying geometries, but there's not much difference between them when it comes to what matters: They all route messages in O(log N) time with O(log N) connectivity and state per node. Just a few comments though... First, the idea where Kademlia fills its tree with nodes that have been online the longest can also be emulated in Pastry -- If you have some row and column of your routing table filled with a live node on the network, and a new node contacts you to notify you of its arrival which also belongs in the same row and column, simply disregard the notification as opposed to replacing the current live entry (or just note its IP address for later in case your current entry dies). In this way, older nodes are not replaced by new nodes, just like in Kademlia. In the _default_ Chord algorithm, however, you can't do this -- fingers in your routing table must point to the first nodes that succeed your position plus half the ring, plus a quarter of the ring, plus an eighth of the ring, etc... Although this requirement can be relaxed while still maintaining O(log N) lookup time [see "Designing a DHT for low latency and high throughput" by Dabek et. al]. If you do that, then I suspect you can weave prioritizing oldest nodes first into Chord as well. Second, using the default implementation of Pastry (called FreePastry, available through Rice) performs poorly under networks with heavy churn. Basically, if nodes join or leave the network too fast, remaining nodes waste so much bandwidth and effort on keeping their leaf sets current that most messages fail delivery. The Bamboo DHT, which slightly modifies the Pastry join algorithm, uses an 'epidemic' approach to updating leaf sets and solves this problem. See Publications at www.bamboo-dht.org for details. Third, and this is just my naive opinion -- Kademlia seems more complicated to implement than Chord or Pastry. If I recall correctly, you have a concurrency parameter introducing parallel lookups, publishing and lookups across k nodes instead of 1, and kind of an ad hoc solution to imbalanced trees... This doesn't mean I'm bagging on Kademlia -- its advantages include being the most popular DHT in practice (see eMule -- so it is definitely a proven protocol _in practice_, which is huge), you don't have to worry about maintaining the consistency of a ring (although this may not be much of an issue anyway), and you can fill your routing table "passively". I'm just saying the protocol seems complicated in comparison, not that it's worse. Again, this is just me spouting off my naive opinion ;) So for high performance DHTs where anonymity doesn't matter, I think Kademlia, Chord, and Pastry (Bamboo) are all perfectly reasonable options. In my upcoming p2p app (heh, whenever that will be completed) I'm using the Bamboo protocol -- but this stems from learning Pastry before either Chord and Kademlia, so I felt more comfortable implementing it. Cheers, Michael Parker On Sat, 18 Dec 2004 20:38:06 +0000 Continuity wrote: > Peter Hunt wrote: > > Do you have another DHT algorithm you can suggest? What's the "best" > > one right now? > > Honestly, I don't know either enough about the subject or enough > different algorithms to be able to answer that.. I'm quite (very) new to > the subject.. I like some things about kademlia - up-time is inversely > proportional to the probability of nodes going offline soon, for > example.. so don't let me put you off looking at it, it's just that > cleverer people than me have said it's not perfect :) > > Both chord and pastry(tapestry? I think it's the same 'family') seem > active and well-researched, but I don't know much about their design. > > If anyone else could answer this question better, or at least give an > overview of what realistic options developers have right now, that would > be nice =) > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From bkn3 at columbia.edu Tue Dec 21 18:47:17 2004 From: bkn3 at columbia.edu (Brad Neuberg) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Exeem - Is it Real? Message-ID: <6.1.0.6.2.20041221104546.0236c4f0@pop.mail.yahoo.com> I've heard mention of a new BitTorrent client, named Exeem, that decentralizes tracking and searching. Does Exeem actually exist or is it a bit of an Internet rumour? Has anyone here actually used it or seen it? I'd like to learn more about it since its architecture sounds interesting. Thanks, Brad Brad Neuberg, bkn3@columbia.edu Senior Software Engineer, Rojo Networks Weblog: http://www.codinginparadise.org ===================================================================== Check out Rojo, an RSS and Atom news aggregator that I work on. Visit http://rojo.com for more info. Feel free to ask me for an invite! Rojo is Hiring! If you're interested in RSS, Weblogs, Social Networking, Java, Open Source, etc... then come work with us at Rojo. If you recommend someone and we hire them you'll get a free iPod! See http://www.rojonetworks.com/JobsAtRojo.html. From harmless at owbn.org Tue Dec 21 19:59:57 2004 From: harmless at owbn.org (Nobody Special) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Exeem - Is it Real? In-Reply-To: <6.1.0.6.2.20041221104546.0236c4f0@pop.mail.yahoo.com> References: <6.1.0.6.2.20041221104546.0236c4f0@pop.mail.yahoo.com> Message-ID: <3188.140.192.45.60.1103659197.squirrel@140.192.45.60> Suprnova's eXeem beta review and screenshots http://www.mitosis.com/sections/forum/showflat.php?Board=web_news&Number=43166 -- > I've heard mention of a new BitTorrent client, named Exeem, that > decentralizes tracking and searching. Does Exeem actually exist or is it a > bit of an Internet rumour? Has anyone here actually used it or seen > it? I'd like to learn more about it since its architecture sounds > interesting. > > Thanks, > Brad From eugen at leitl.org Tue Dec 21 22:38:50 2004 From: eugen at leitl.org (Eugen Leitl) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] [Beowulf] CFP: MPPS05 (New Submission Deadline) (fwd from srgadmin@cs.hku.hk) Message-ID: <20041221223849.GH9221@leitl.org> ----- Forwarded message from srg-admin ----- From: srg-admin Date: Tue, 21 Dec 2004 09:46:40 +0800 To: srg-cfp@cs.hku.hk, Cho Li Wang Cc: Subject: [Beowulf] CFP: MPPS05 (New Submission Deadline) User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040803 ******************** NEW SUBMISSION DEADLINE *************************** Call for Papers The First International Workshop on Mobility in Peer-to-peer Systems (MPPS05) (http://mx.nthu.edu.tw/~ctking/MPPS05.htm) in conjunction with The 25th IEEE International Conference on Distributed Computing Systems (ICDCS 2005) June 6-9, 2005 Columbus, Ohio, USA ----------------------------------------------------------------------------------- THEME Peer-to-peer (P2P) has emerged as a promising paradigm for developing large-scale distributed systems. P2P systems are characterized as being fully decentralized, self-organizing, and self-repairing. Early P2P systems were designed with an Internet-like network infrastructure in mind. As the emergence and prevalence of new wireless networking techniques, such as wireless mesh networks, wireless LANs, and 3G cellular networks, the need to move P2P paradigm into wireless networking and to support mobile computing is increasing. How does a P2P system encompass wired and heterogeneous wireless networks? How does a P2P system exploit and aggregate the resources in such an environment? How should a P2P system manage node mobility? How should a mobile P2P system ensure security and privacy? These issues become very challenging. The goal of the workshop is to examine the mobility issues in P2P systems over heterogeneous wired/wireless networks. TOPICS Topics of interest include, but are not limited to: * P2P systems over wireless mesh and ad hoc networks * mobile P2P applications and systems * power-aware and energy-efficient mobile P2P systems * topology-aware mobile P2P systems * mobility and resource management in mobile P2P systems * P2P systems for wireless grid * security in mobile P2P systems * trust and access control in mobile P2P systems * anonymity and anti-censorship for mobile P2P systems * modeling and analysis for mobile P2P systems PAPER SUBMISSION Authors are invited to submit an electronic version of original, unpublished manuscripts, not to exceed 20 double-spaced pages, to ctking@mx.nthu.edu.tw. Submissions should be in PDF or Postscript format. Submissions must be received by the deadline. All submitted papers will be refereed by reviewers in terms of originality, contribution, correctness, and presentation. The accepted papers will be published by the IEEE Computer Society Press. IMPORTANT DATES - Paper submission: December 31, 2004 (extended) - Author notification: February 1, 2005 - Final Manuscript: March 1, 2005 ORGANIZATION Workshop Chair: Lionel M. Ni, Hong Kong University of Science and Technology, Hong Kong Program Chairs: Chung-Ta King, National Tsing Hua University, Taiwan Jie Wu, Florida Atlantic University, USA Program Committee: James Aspnes, Yale University, U.S.A. Jiannong Cao, Hong Kong Polytechnic University, Hong Kong Bernady O. Apduhan, Kyushu Sangyo University, Japan Dan Grigoras, University College Cork, Ireland Hung-Chang Hsiao, National Tsing Hua University, Taiwan Charlie Hu, Purdue University, U.S.A. Yiming Hu, University of Cincinnati, U.S.A. Jehn-Ruey Jiang, National Central University, Taiwan Fabian Kuhn, ETH, Zurich Xiaoming Li, Peking University Yunhao Liu, Hong Kong University of Science and Technology, Hong Kong Chunqiang Tang, IBM Watson Research Center, U.S.A. Li Xiao, Michigan State University, U.S.A. Aaron Zollinger, ETH, Zurich ------------------------------------------------ _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20041221/89fcbcf9/attachment.pgp From dev at eth0.ch Thu Dec 23 07:10:55 2004 From: dev at eth0.ch (David E. Meier) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How reliable is UDP? Message-ID: <1699.217.162.71.141.1103785855.squirrel@217.162.71.141> Hi p2p-hackers, I know that UDP traffic is not reliable by design. However, I wonder how reliable UDP packets reach their destination. Does someone on the list have experience with UDP packet loss? Is this usually in the range of a percent or more or less? Or is it even a common case that all packets arrive? Dave. From samnospam at bcgreen.com Fri Dec 24 20:32:02 2004 From: samnospam at bcgreen.com (Stephen Samuel (leave the email alone)) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How reliable is UDP? In-Reply-To: <1699.217.162.71.141.1103785855.squirrel@217.162.71.141> References: <1699.217.162.71.141.1103785855.squirrel@217.162.71.141> Message-ID: <41CC7CC2.5010109@bcgreen.com> ping (or mtr) can generally give you a good estimation of packet loss in a connection. 1% packet loss is generally considered bad. More than that will cause severe performance penalties for things like NFS and also slow down TCP transmissions noticably. Local connections should generally have 0% packet loss (It's not unusuall for me to leave ping sessions running for days to a local box and get things like 80000 packets transmitted with 0 or 1 packet lost). From a quick test with 10 packets/second:(about a 2500 mile round trip). 1626 packets transmitted, 1626 received, 0% packet loss, time 164186ms rtt min/avg/max/mdev = 28.195/29.424/52.322/1.056 ms, pipe 2, ipg/ewma 101.037/29.204 ms I get similar results for a ping session transcontinental * transatlantic. 472 packets transmitted, 470 received, 0% packet loss, time 54703ms rtt min/avg/max/mdev = 200.554/202.114/230.265/2.298 ms, pipe 4, ipg/ewma 116.143/201.708 ms and pinging a chinese SPAM site: [samuel@me samuel]% ping -f -i.1 202.102.230.36 PING 202.102.230.36 (202.102.230.36) 56(84) bytes of data. .................................................................... --- 202.102.230.36 ping statistics --- 1762 packets transmitted, 1694 received, 3% packet loss, time 178508ms rtt min/avg/max/mdev = 466.896/479.661/621.438/6.357 ms, pipe 8, ipg/ewma 101.367/479.347 ms [root@me samuel]# David E. Meier wrote: > Hi p2p-hackers, > > I know that UDP traffic is not reliable by design. However, I wonder how > reliable UDP packets reach their destination. Does someone on the list > have experience with UDP packet loss? Is this usually in the range of a > percent or more or less? Or is it even a common case that all packets > arrive? Dave. -- Stephen Samuel +1(604)876-0426 samnospam@bcgreen.com http://www.bcgreen.com/ Powerful committed communication. Transformation touching the jewel within each person and bringing it to light. From em at em.no-ip.com Tue Dec 28 16:56:27 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? Message-ID: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> DHT's such as Kademlia work well when they are used to store large numbers of items with different keys. But let's suppose, for example, that a lot of nodes want to advertise their IP addresses as the place where people can find an instance of a very common resource, such as, say, an open proxy for some kind of protocol. All the records will then have the same key: Hash("Resource XYZ available here"), which means that the few unlucky DHT nodes with ID close to that hash will be requested to store the pointers for ALL the records. To some extent this already happens with filesharing applications: nodes with ID close to Hash("britney") are requested to store metadata information for all Britney Spear's MP3 files. However, those may amount, at most, to few hundred or maybe thousand records; but in the case I mentioned before the number of records may grow unbounded as more and more resource providers jpin the scheme. Are there techniques devised specifically to solve this problem? Enzo From paul at ref.nmedia.net Tue Dec 28 17:46:29 2004 From: paul at ref.nmedia.net (Paul Campbell) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? In-Reply-To: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> References: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> Message-ID: <20041228174629.GA7400@ref.nmedia.net> There's a simple solution to solve the "Britney Spears" problem in my opinion in the DHT model (randomized searches don't suffer from this problem as a result of their structural fault that they handle common files well but don't handle rare files well). Let's first recast your question in an even worse situation. Let's say that we directly implement an inverted index (keyword-->file) on the DHT. Even with common word elimination, this certainly and severely generates a very non-linear index space. But it is something that is very obvious and desirable since it also solves a major DHT and randomized P2P dilema: how to quickly search for files by keyword. So far, most proponents of DHT's believe that hashing is the answer. While hashing MASKS problems, it doesn't actually solve the problem. All solutions that I've seen appear to be pretty simple. The idea is to spread the location of a particular key out over a wider arc. Conceptually, one can spread it in multiple points (such as spreading it to equadistant points around the ring) or on multiple hierarchal rings (like the distributed sloppy hash tables...DSHT's although that was done for delay purposes). But the concept is essentially the same. The routing algorithm remains valid. Since the routing algorithm incrementally searches for the destination. It just remains to send a search that is in the form of incrementally refining an "area" to search in. As the search request passes through potential nodes, the search area gradually narrows. Storage is slightly more complicated. It is necessary to store a key/value pair in the appropriate AREA of the DHT. The storage targets the area and not the point. The above search mechanism (which has to be used to find the appropriate destination for a key/value pair anyways) also paradoxically nails down the appropriate area for storage. thecircle (from your terminology, it appears that's what you've been looking at) attaches a random bit string to the end of the hash to accomplish this and then spreads a file over a small area using that random hash (although a lot of this isn't actually implemented when you look at the code). The various "tree" designs out there just use the hash keys directly and split storage as necessary. I've seen a few designs that attempt to simply delete common files from the DHT and index those using a random P2P structure (thus taking advantage of both designs). What I've noticed in general is this. In order for the routing properties of a DHT to be preserved, it is necessary to maintain the node link structure. But that's not to say that you have to maintain the same structure at the key/value pair level of the system...that is, a direct one-to-one mapping is necessary. The "skip graph" structures specifically acknowledge this. Simply substitute "key/value pair" for "DNS name" and you convert those structures to the idea I'm referring to. In a similar way, one could simply store "first key/last key" values in addition to "node keys" and decouple the data structure from the DHT link structure. This is similar to index tabs in a rolodex...no matter how many cards are in the rolodex and no matter what the ordering, you maintain labels and you can approximately locate a card using the same search mechanism all the time. As to the basic problem of "too many keys with the same value", this is also pretty easy to deal with. Extend the key/value pairs so that for certain operations (searching), you use just the key as before. However, optionally treat the keys as "key+value" meta-keys. For instance, when storing a new key/value pair, use a "search-store" function that treats the key as "key"+"value" and the value as just "value". Then no matter how identical keys there are, there is still a valid ordering and the number of keys with the same values are no longer a limiting factor. Of course, if your system doesn't require a total ordering on all keys and values (the node balancing algorithms more or less do require this at least right now), then you can dispense with the "key+value" idea and simply ignore this issue. The one remaining problem is node balancing. The mechanisms that are already published seem to be pretty uniform in design. Nodes that undertake rebalancing can do so using one of two mechanisms. Either an empty node (that may have to be created by dumping it's current payload) is moved into position on the ring, or else incremental data movement is undertaken. Since the former incurs a much higher communication cost, the latter is given preference as much as possible. However, it has been shown that both mechansims are necessary for load balance to remain bounded. From paul at ref.nmedia.net Wed Dec 29 01:53:06 2004 From: paul at ref.nmedia.net (Paul Campbell) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How reliable is UDP? In-Reply-To: <1699.217.162.71.141.1103785855.squirrel@217.162.71.141> References: <1699.217.162.71.141.1103785855.squirrel@217.162.71.141> Message-ID: <20041229015306.GA27446@ref.nmedia.net> On Thu, Dec 23, 2004 at 08:10:55AM +0100, David E. Meier wrote: > Hi p2p-hackers, > > I know that UDP traffic is not reliable by design. However, I wonder how > reliable UDP packets reach their destination. Does someone on the list > have experience with UDP packet loss? Is this usually in the range of a > percent or more or less? Or is it even a common case that all packets > arrive? Dave. Pretty darned reliable. Remember...the base IP protocol goal is to deliver packets. Routers use packet dropping to signal congestion to the TCP protocol. So other than actual packet losses (which are so minimal that currently TCP ASSUMES all packet losses are congestion notifications), the only losses are due to the congestion signalling protocol. In reality, they could actually eventually be due to real congestion, but that would never happen since you of course implemented congestion control in your UDP protocol, right? :) It is a serious problem for wireless protocols since in the wireless environment, packet loss is a real issue. Hence things like the "TULIP" protocol have been developed to encapsulate TCP/IP at either the TCP or IP layer and implement their own ACK protocols (and sometimes ACK spoofing) specifically to cover up for standard TCP protocol weaknesses of assuming packet loss is always due to packet loss so that standard implementations don't get confused and give artificially low throughputs as a result. From ian at locut.us Wed Dec 29 12:03:45 2004 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? In-Reply-To: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> References: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> Message-ID: A simple and (IMHO) elegant approach used in Freenet (that can also been applied to DHTs) is for nodes to cache the responses to requests that they forward in a least-recently-requested cache. If a node receives a request for some data in its cache it responds immediately (updating the timestamp in the cache accordingly). The effect of this is that the load of responding to requests for popular information is spread among peers close to that information, the more popular it is, the more peers will cache it. As its popularity decreases it will be cached in fewer machines - and so it adapts to shifting file popularity. This approach is also employed by Dijjer (http://dijjer.org/). Ian. On 28 Dec 2004, at 16:56, Enzo Michelangeli wrote: > DHT's such as Kademlia work well when they are used to store large > numbers > of items with different keys. But let's suppose, for example, that a > lot > of nodes want to advertise their IP addresses as the place where people > can find an instance of a very common resource, such as, say, an open > proxy for some kind of protocol. All the records will then have the > same > key: Hash("Resource XYZ available here"), which means that the few > unlucky > DHT nodes with ID close to that hash will be requested to store the > pointers for ALL the records. To some extent this already happens with > filesharing applications: nodes with ID close to Hash("britney") are > requested to store metadata information for all Britney Spear's MP3 > files. > However, those may amount, at most, to few hundred or maybe thousand > records; but in the case I mentioned before the number of records may > grow > unbounded as more and more resource providers jpin the scheme. Are > there > techniques devised specifically to solve this problem? > > Enzo > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > -- Founder, The Freenet Project http://freenetproject.org/ CEO, Cematics Ltd http://cematics.com/ Personal Blog http://locut.us/~ian/blog/ From mllist at vaste.mine.nu Wed Dec 29 18:44:44 2004 From: mllist at vaste.mine.nu (Vaste) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? In-Reply-To: <20041228174629.GA7400@ref.nmedia.net> References: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> <20041228174629.GA7400@ref.nmedia.net> Message-ID: <41D2FB1C.4090508@vaste.mine.nu> Paul Campbell wrote: > Let's say that we directly implement an inverted index (keyword-->file) on > the DHT. Even with common word elimination, this certainly and severely > generates a very non-linear index space. But it is something that is very > obvious and desirable since it also solves a major DHT and randomized P2P > dilema: how to quickly search for files by keyword. > > So far, most proponents of DHT's believe that hashing is the answer. While > hashing MASKS problems, it doesn't actually solve the problem. > > All solutions that I've seen appear to be pretty simple. The idea is to spread > the location of a particular key out over a wider arc. Conceptually, one can > spread it in multiple points (such as spreading it to equadistant points > around the ring) or on multiple hierarchal rings (like the distributed > sloppy hash tables...DSHT's although that was done for delay purposes). But > the concept is essentially the same. > > The routing algorithm remains valid. Since the routing algorithm incrementally > searches for the destination. It just remains to send a search that is in > the form of incrementally refining an "area" to search in. As the search > request passes through potential nodes, the search area gradually narrows. Some examples: In a situation where a node typically only requires part of a key's value, one can split the key's value among several peers, (e.g. in chord an arc of the circle). E.g. in a filesharing network using a DHT to locate peers with the file (hash of file -> peers), you only need some peers, not all. Another example would be for AND-queries in an inverted index (keyword -> file) such as the one mentioned. For e.g. "spears britney" you don't need all matches for "spears". Here though, you want a specific subset of the "spears" set (the one matching "britney" as well) rather than any (random) subset. One approach could be to map the "spears"-subset of the network the same way as the whole network; i.e. after locating the spears-"area" one narrows it the same way (by the hash of another keyword), until one gets to an area small enough to fully retrieve it or until all keywords match (and thus you've found some subset of the "britney spears"-subset). /Vaste From mfreed at cs.nyu.edu Wed Dec 29 20:13:31 2004 From: mfreed at cs.nyu.edu (Michael J. Freedman) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? In-Reply-To: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> References: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> Message-ID: Enzo, This is exactly the problem for which "sloppy hashing" in Coral was designed. It can perform this load balancing at the DHT layer (not through aggressive app-layer caching like Freenet, CFS, PAST, etc.) Short paper: http://www.scs.cs.nyu.edu/coral/docs/coral-iptps03.pdf Expanded paper with integrating CDN: http://www.scs.cs.nyu.edu/coral/docs/coral-nsdi04.pdf The basic idea is that get() may retrieve only a subset of the values inserted under put (). And in fact, due to Coral's clustering algorithms, these retrieved values are likely those inserted by nearby nodes. We've used Coral primarly to build and deploy CoralCDN, a web content distribution network. Think that when a slashdotted page gets accessed via Coral, it disseminates to most/all web proxies within the CoralCDN network, yet even though all proxies are "announcing" themselves as caching the page, the most-loaded DHT node is only seeing at most log(n) puts (). (We in fact quickly prove this, see the expanded paper.) As an aside, Coral has seen its traffic quickly grow since we've announced "beta" release here back in late August. (Just add ".nyud.net:8090" to the end of any hostname in a URL.) It now gets 5-7 M requests per day from ~200,000 clients, serving several TB. I guess this list serves as a fast-track method for slashdotting! --mike On Wed, 29 Dec 2004, Enzo Michelangeli wrote: > Date: Wed, 29 Dec 2004 00:56:27 +0800 > From: Enzo Michelangeli > Reply-To: Peer-to-peer development. > To: Peer-to-peer development. > Subject: [p2p-hackers] How to solve the "Britney problem"? > > DHT's such as Kademlia work well when they are used to store large numbers > of items with different keys. But let's suppose, for example, that a lot > of nodes want to advertise their IP addresses as the place where people > can find an instance of a very common resource, such as, say, an open > proxy for some kind of protocol. All the records will then have the same > key: Hash("Resource XYZ available here"), which means that the few unlucky > DHT nodes with ID close to that hash will be requested to store the > pointers for ALL the records. To some extent this already happens with > filesharing applications: nodes with ID close to Hash("britney") are > requested to store metadata information for all Britney Spear's MP3 files. > However, those may amount, at most, to few hundred or maybe thousand > records; but in the case I mentioned before the number of records may grow > unbounded as more and more resource providers jpin the scheme. Are there > techniques devised specifically to solve this problem? > > Enzo > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > ----- "Not all those who wander are lost." www.michaelfreedman.org From em at em.no-ip.com Thu Dec 30 01:23:52 2004 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] How to solve the "Britney problem"? References: <00b401c4ecfe$792c2100$0200a8c0@em.noip.com> Message-ID: <01b401c4ee0f$54e30220$0200a8c0@em.noip.com> Thanks everybody for your helpful replies. Here is what I'm trying to achieve: My interest in DHT's originates primarily from my desire of building a decentralized (serverless) communications system able to handle both text IM and VoIP; I first mentioned this goal nine months ago in a post archived at http://zgp.org/pipermail/p2p-hackers/2004-March/001780.html . At present, I'm thinking of implementing it as a daemon simulating a SIP and RTP proxy bound to 127.0.0.1, so that any SIP/SIMPLE user agent (of which there are many around) could be used for the actual user interface. The daemon would then translate SIP messages and encapsulate the RTP flows (adding security as well) in a way the UA would be blissfully unaware of. This approach is similar to what I suggested at http://zgp.org/pipermail/p2p-hackers/2004-July/002018.html to run SIP over JXTA. In order to take advantage from a large installed base, I decided to make the DHT library interoperable with an existing and widely deployed application, and I chose Overnet, the Kademlia-based protocol developed by Metamachine for their eDonkey 1.0 (one of the reasons for that particular choice is that Ethereal has a dissector for that protocol, making my life easier during the debugging). That library is now released at http://kadc.sourceforge.net/ , although in the current version it only works in "leaf node" mode, accessing the DHT but not participating to it (the "store" and "find" RPC's are issued, but ignored when received from other nodes, as on the other hand NATted Overnet peers routinely do). Of course, the choice of an existing protocol also imposes limitations: I can play with the structure of keys and data (e.g., I can randomize some bits of the hash used as key before performing a "store"), but I can't modify the way records with a given key/value structure are handled by the peers composing the DHT. Also, Kademlia does not forward requests as Gnutella does, but uses a "redirection" model where the replies to individual "find" operations point the requesting node to other peers ever closer to the target. This prevents the use of application-level caching. The first sample application I've written, a DNS extender implemented as caching DNS proxy(http://kadc.sourceforge.net/apps.html#namecache ) is immune from the "Britney problem" because the distribution of the pseudo domain names stored in the DHT is likely to be relatively smooth, and as a result it works fairly well. However, NATted peers in a P2P communications system need not only a users directory, but also a way to discover resources such as NAT-traversal helper servers (a' la STUN) or actual proxies/reflectors (e.g., for SIP or RTP) that are advertised as locally available by non-NATted nodes. And here lies the problem: the information that those nodes need to publish has variable data ("my address is 1.2.3.4:5678") but a constant key ("I'm an available SIP proxy", "I'm an available STUN server" etc.). Of course, I could use a separate search protocol for those resources, perhaps a Gnutella-like one based on bounded flooding (as fast full-overlay-network searches are made unnecessary precisely by the genericity of the key), or perhaps one allowing sorted access like P-Trees or SkipGraphs to locate the nodes with the best available bandwidth; but before building myself a new screwdriver, I'd like to be sure that I can't somehow use the hammer I already have :-) Enzo ----- Original Message ----- From: "Michael J. Freedman" To: "Peer-to-peer development." Sent: Thursday, December 30, 2004 4:13 AM Subject: Re: [p2p-hackers] How to solve the "Britney problem"? > Enzo, > > This is exactly the problem for which "sloppy hashing" in Coral was > designed. It can perform this load balancing at the DHT layer (not > through aggressive app-layer caching like Freenet, CFS, PAST, etc.) [...] From mllist at vaste.mine.nu Fri Dec 31 04:18:20 2004 From: mllist at vaste.mine.nu (Vaste) Date: Sat Dec 9 22:12:49 2006 Subject: [p2p-hackers] Common interest, finding trading partners Message-ID: <41D4D30C.4040202@vaste.mine.nu> One of the reasons e.g. BitTorrent works so great is that when you receive a piece from someone, you can trade it for another piece with anyone else in the swarm without that piece. This works since everyone on a torrent (typically) are interested in the same file. With multi-file torrents this assumption is extended to several files as well. In the file-based filesharing world (ed2k, gnutella) the same assumption hold, but but only for one file. And with batchtorrents, this assumption might brake even with torrent; one might only want a few files, (and e.g. using Azureus not even request the other files). Still, two people downloading an episode from same tv-series are quite likely to be interested in the same files, and thus they might benefit from trading. Has any research been done on how these peers with common interests can find each other? Mainly it's about finding the peer with the most coinciding interest. Still, other factors play in, such as the resources available (does the peer lack trading partners, i.e. has bandwidth to spare?) in finding a good match. On the networklevel it might also be good to find a balance between finding the "best" peer and creating a well-connected network (avoiding cliques and bottlenecks). Moreover, as interest's change over time, how should this be handled? (The more coinciding the interests, the longer it should take for them to deviate from each other.) This is question is quite similar to finding a peer with pieces of a file you're interested in (that you don't already have). (The difference being that in the former, you search for _potential_ bearers of the piece.) Here the problem of matching a large number of preferences shows (namely the pieces; there are usually quite a few of them). The same things happens with many (especially small) files. In the search-layer I believe this is usually handled either not at all (random) or in a binary way (complete file vs. only pieces of it), and leaving the details to the strict peer-to-peer chatting. Would there be any point in using more detailed information of finished pieces in the search layer? Would there be any use to use different resolutions (e.g. 10 pieces resolution might be: piece 1-10: none finished, 11-20: all, 21-29: some)? An interesting take on this is the perspective of piece-based networks, where one searches for every piece separately. Here it's even more obvious how much one would benefit from finding people interested in the same file (the same pieces). Yet, introducing things such as patches, it would be pleasing to have a solution that didn't depend on peers wanting the exactly same pieces (defined by the file), but just roughly the same. /Vaste