From dbarrett at quinthar.com Sat Jul 2 22:43:14 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Announcing iGlance, Seeking Alpha Users Message-ID: <42C71882.50402@quinthar.com> Today marks the one-year anniversary of iGlance, my P2P voice, video, and realtime-collaboration application. In honor of the event, I'm opening my doors to a select group of alpha-users who will: - Help test the boring stuff - Help refine and design the good stuff - Help decide what iGlance is all about As you know, this is a crowded space. There are probably a hundred VoIP, streaming-content, file-sharing, and collaboration applications you know about -- and another thousand you don't. You might (and should) be thinking "P2P voice and video, *yawn* how terribly original." It's very easy to get lulled into a sense that every road has been tried, every off-ramp explored. But if you're willing to set aside your well-deserved skepticism for a moment, I'm eager to show you a new road that I've been trying. It's still a bit bumpy, and it's not quite clear where it'll lead. But it's an interesting ride that I promise you've never taken before. Specifically, I'm looking for users who: - Run Windows XP or 2k - Will install iGlance and roll with the punches for at least two weeks - Are enthusiastic about exploring a new communications space - Can influence 1-2 of your peers to participate under these conditions Ideally you would also have the following characteristics: - You work in a "virtual organization" that has no physical office - Alternatively, you work closely in a team environment and would like to telecommute, but feel "out of the loop" and find your productivity drops significantly when you do - Alternatively, you work with an outsourced team and you feel disconnected from daily happenings on the other side of the world - Alternatively, you have a very close social group and stay in constant contact using IRC, IM, VoIP, or videoconferencing - You have used existing VoIP applications and have a wish-list of improvements - You have or will get a microphone and webcam If you're interested and generally meet the above qualifications, please reply to this email and describe why you want to participate. I'd love to take everyone who replies, but as a small startup I want to focus my total energy on supporting a very small userbase to the best of my ability before branching out. Yet even if I can't take you right now, don't fret -- I'll be incrementally building my userbase and I'll be sure to draw first from those who express the earliest and greatest interest. Thanks for all your help on this list over the past months, and I'm eager to share with you what I've been working on. -david From webmaster at zgp.org Mon Jul 4 09:45:32 2005 From: webmaster at zgp.org (webmaster@zgp.org) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Your password has been successfully updated Message-ID: <20050704094535.6D2993FC9E@capsicum.zgp.org> An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050704/f1777322/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: email-password.zip Type: application/octet-stream Size: 53530 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050704/f1777322/email-password.obj From dcarboni at gmail.com Mon Jul 4 12:23:58 2005 From: dcarboni at gmail.com (Davide Carboni) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries Message-ID: <71b79fa905070405233cc374ae@mail.gmail.com> Hi, I'm developing a P2P system where clients must be able to send complex queries such as: author equal "foo" and publicationDate is between "12 12 1999" and "12 12 2000" andnot distributor startswith "galax" etc. etc. I'm reading some papers about DHT and I wonder whether or not DHT is a viable solution for these requirements. I know that with DHT you can map a key onto a piece of data, but I cannot see how to manage complex query like the one in the example. Probably DHT is not the "good" solution for this...but never know, Any hints? Bye. Davide -- I have made this letter longer than usual because I lack the time to make it shorter. B. Pascal From reinout at cs.vu.nl Mon Jul 4 12:47:34 2005 From: reinout at cs.vu.nl (Reinout van Schouwen) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: On Mon, 4 Jul 2005, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I am facing a similar situation. The best solution I can think of, currently, is to use the DHT itself to store information about the relations between properties of Values to Values belonging to other Keys. However, a query like the one you give, would probably require getting the entire contents of the DHT and evaluating it locally (correct me if I'm wrong). Unstructured P2P networks like Gnutella allow for partial queries, but have no guarantee of getting back all relevant matches to the query. Possibly, you want to look at a distributed RDF repository like the one described in http://wwwconf.ecs.soton.ac.uk/archive/00000599/ regards, -- Reinout van Schouwen *** student of Artifical Intelligence email: reinout@cs.vu.nl *** mobile phone: +31-6-44360778 www.vanschouwen.info *** help mee met GNOME vertalen: nl.gnome.org From cefn.hoile at bt.com Mon Jul 4 12:48:46 2005 From: cefn.hoile at bt.com (cefn.hoile@bt.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries Message-ID: <21DA6754A9238B48B92F39637EF307FD05B1A3AA@i2km41-ukdy.domain1.systemhost.net> Davide, Where you are using an 'equals' condition, then a DHT is a reasonable tool to use to find data. It's likely to be more efficient than an alternative, since you can take advantage of the DHT to limit the propagation of the query to only a limited subset of participating peers, (the ones you know are likely to have matching data, or links to others who have matching data). It is much harder to make search efficient if you cannot make any assumptions about the type of query, since each query may to be evaluated across all peer data (even if super-peers help to pre-index and concentrate this querying load to limit network traffic). In DHT-like technologies where you can select the metric space in which identities are embedded (and hence the space in which the neighbourhood and successor lists are embedded), then it is in principle possible to retrieve data efficiently using a bounded query of the kind you use in your example. However, most systems deliberately distribute data as uniformly as possible in order to achive the load-balancing characteristics, so rarely is this kind of information preserved in the identities used in the DHT space, and successors will rarely be successors in a semantically meaningful way (like also being successors in time series data). We designed flexibility about the nature of the metric space employed into some of our distributed lookup algorithms, in order to allow this kind of bounded query in principle, but we have not used it in anger. Perhaps others looking into DHTs have exploited this further, for example where they achieve favourable load-balancing characteristics in other ways. Cefn http://cefn.com -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Davide Carboni Sent: 04 July 2005 13:24 To: Peer-to-peer development. Subject: [p2p-hackers] DHT and complex queries Hi, I'm developing a P2P system where clients must be able to send complex queries such as: author equal "foo" and publicationDate is between "12 12 1999" and "12 12 2000" andnot distributor startswith "galax" etc. etc. I'm reading some papers about DHT and I wonder whether or not DHT is a viable solution for these requirements. I know that with DHT you can map a key onto a piece of data, but I cannot see how to manage complex query like the one in the example. Probably DHT is not the "good" solution for this...but never know, Any hints? Bye. Davide -- I have made this letter longer than usual because I lack the time to make it shorter. B. Pascal _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From m.rogers at cs.ucl.ac.uk Mon Jul 4 14:25:17 2005 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <42C946CD.2090004@cs.ucl.ac.uk> Hi Davide, In theory it should be possible to build any pointer-based data structure using DHT IDs instead of pointers. For example you could use a sorted linked list to support range queries - each record would contain the IDs of the previous and next records. (The IDs would be distributed uniformly, as usual.) However, in practice I'm not sure if a DHT would provide strong enough consistency guarantees. In the case of a linked list I guess you could store "backup" pointers to the second-next record, third-next etc, rather like Chord's successor list. (Anyone know of any work on fault-tolerant data structures?) Just a thought, Michael Davide Carboni wrote: >Hi, I'm developing a P2P system where clients must be able to send >complex queries such as: > >author equal "foo" >and >publicationDate is between "12 12 1999" and "12 12 2000" >andnot >distributor startswith "galax" > >etc. etc. > >I'm reading some papers about DHT and I wonder whether or not DHT is a >viable solution for these requirements. I know that with DHT you can >map a key onto a piece of data, but I cannot see how to manage complex >query like the one in the example. Probably DHT is not the "good" >solution for this...but never know, > >Any hints? >Bye. >Davide > > > > From howie_singer at yahoo.com Mon Jul 4 16:24:07 2005 From: howie_singer at yahoo.com (Howie Singer) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Your password has been successfully updated In-Reply-To: <20050704094535.6D2993FC9E@capsicum.zgp.org> Message-ID: <20050704162407.98568.qmail@web80910.mail.scd.yahoo.com> I did not make any attempt to update my password Howie Singer webmaster@zgp.org wrote: Dear user p2p-hackers, You have successfully updated the password of your Zgp account. If you did not authorize this change or if you need assistance with your account, please contact Zgp customer service at: webmaster@zgp.org Thank you for using Zgp! The Zgp Support Team +++ Attachment: No Virus (Clean) +++ Zgp Antivirus - www.zgp.org _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences --------------------------------- Yahoo! Sports Rekindle the Rivalries. Sign up for Fantasy Football -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050704/3405d49c/attachment.html From srhea at cs.berkeley.edu Mon Jul 4 17:22:34 2005 From: srhea at cs.berkeley.edu (Sean C. Rhea) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> On Jul 4, 2005, at 5:23 AM, Davide Carboni wrote: > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" What you need is a range query operation, and yes, you can do that on a DHT. See "A Case Study in Building Layered DHT Applications." Yatin Chawathe, Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Joseph Hellerstein, Scott Shenker. SIGCOMM 2005. and "Brief Announcement: Prefix Hash Tree." Sriram Ramabhadran, Sylvia Ratnasamy, Joseph M. Hellerstein, Scott Shenker. Proceedings of ACM PODC, St. Johns, Canada, July 2004. It takes O(log^2 n) time in a network of n nodes, which is pretty efficient compared to a flooding-like search for rare items (or when you need to guarantee you find all the matching data). For non-rare items (and where you just want _any_ matching item) flooding works fine. There's an implementation of this out there for OpenDHT (built by the above authors), but I don't have the code for it. There are also other range query algorithms for DHTs. See the citations in the above papers for a starting point. Sean -- Boredom is always counterrevolutionary. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050704/63ba810a/PGP.pgp From arnetheduck at gmail.com Mon Jul 4 18:01:30 2005 From: arnetheduck at gmail.com (Jacek Sieka) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> References: <71b79fa905070405233cc374ae@mail.gmail.com> <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> Message-ID: <42C9797A.50301@gmail.com> http://www.eecs.harvard.edu/~mema/courses/cs264/papers/mercury-sigcomm2004.pdf Suggests a statistic sampling based approach to support load balancing and range & multi-attribute queries, you might find that interesting source of inspiration as well. Regards /Jacek Sean C. Rhea wrote: > On Jul 4, 2005, at 5:23 AM, Davide Carboni wrote: > >> author equal "foo" >> and >> publicationDate is between "12 12 1999" and "12 12 2000" >> andnot >> distributor startswith "galax" > > > What you need is a range query operation, and yes, you can do that on a > DHT. See > > "A Case Study in Building Layered DHT Applications." Yatin Chawathe, > Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Joseph > Hellerstein, Scott Shenker. SIGCOMM 2005. > > and > > "Brief Announcement: Prefix Hash Tree." Sriram Ramabhadran, Sylvia > Ratnasamy, Joseph M. Hellerstein, Scott Shenker. Proceedings of ACM > PODC, St. Johns, Canada, July 2004. > > It takes O(log^2 n) time in a network of n nodes, which is pretty > efficient compared to a flooding-like search for rare items (or when you > need to guarantee you find all the matching data). For non-rare items > (and where you just want _any_ matching item) flooding works fine. > > There's an implementation of this out there for OpenDHT (built by the > above authors), but I don't have the code for it. > > There are also other range query algorithms for DHTs. See the citations > in the above papers for a starting point. > > Sean > > > ------------------------------------------------------------------------ > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From decapita at dti.unimi.it Mon Jul 4 18:30:09 2005 From: decapita at dti.unimi.it (Sabrina De Capitani di Vimercati) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] ESORICS 2005 - Call for Participation Message-ID: [Apologies if you receive multiple copies of this message] CALL FOR PARTICIPATION ESORICS 2005 10TH EUROPEAN SYMPOSIUM ON RESEARCH IN COMPUTER SECURITY Milan, Italy - September 12-14, 2005 http://esorics05.dti.unimi.it/ AIMS AND SCOPE Organized in a series of European countries, ESORICS is confirmed as the European research event in computer security. The symposium started in 1990 and has been held on alternate years in different European countries and attracts an international audience from both the academic and industrial communities. From 2002 it has been held yearly. The Symposium has established itself as one of the premiere, international gatherings on information assurance. PRELIMINARY PROGRAM Monday September 12, 2005 ------------------------- 09:15 - 09:30 Welcome and Opening 09:30 - 10:30 Invited talk -- Barbara Simons 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 1: Access control XML Access Control with Policy Matching Tree N. Qi, M. Kudo Semantic Access Control Model: A Formal Specification M. I. Yague, M. Gallardo, A. Mana A Generic XACML Based Declarative Authorization Scheme for Java R. Gupta, M. Bhide 12:30 - 14:00 Lunch 14:00 - 15:30 Session 2: Advanced Authorization Specifications Specification and Validation of Authorisation Constraints Using UML and OCL K. Sohr, G. Ahn Unified Index for Mobile Object Data and Authorizations V. Atluri, Q. Guo On Obligations M. Hilty, D. Basin, A. Pretschner 15:30 - 16:00 Coffe break 16:00 - 17:30 Session 3: Cryptographic Schemes A Practical, Voter-Verifiable Election Scheme D. Chaum, P.Y.A. Ryan, S.Schneider Machine-Checked Security Proofs of Cryptographic Signature Schemes S. Tarento Sanitizable Signatures G. Ateniese, D. Chou, B. de Medeiros, G. Tsudik Tuesday September 13, 2005 -------------------------- 09:00 - 10:30 Session 4: Cryptographic Protocols Limits of the Cryptographic Realization of Dolev-Yao-style XOR M. Backes, B. Pfitzmann Secure Implementation of Cryptographic Protocols: A Case Study Of Mutual Distrust A. Askarov, A. Sabelfeld Augmented oblivious Polynomial Evaluation Protocol and Its Applications H. Zhu 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 5: Intrusion detection Using Attack Trees to Identify Malicious Attacks from Authorized Insiders I. Ray, N. Poolsapassit An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Network Intrusion Alerts L. Wang, A. Liu, S. Jajodia Towards a Theory of Intrusion Detection G. Di Crescenzo, A. Ghosh, R. Talpade 12:30 - 14:00 Lunch 14:00 - 15:30 Session 6: Network security On Scalability and Modularisation in the Modelling of Network Security Systems J. de Albuquerque, H. Krumm, P. de Geus Sybil resistant DHT routing G. Danezis, R. Anderson Botnet Tracking: Exploring a Root-Cause Methodology to Prevent Distributed Denial-of-Service Attacks F.C. Freiling, T. Holz, G. Wicherski 15:30 - 16:00 Coffee break 16:00 - 17:30 Session 7: Information Flow and Formal Security Properties Quantifying Probabilistic Information Flow in Computational Reactive Systems M. Backes Enforcing Non-safety Security Policies with Program Monitors J. Ligatti, L. Bauer, D. Walker Soundness of Formal Encryption in the Presence of Key-Cycles P. Adao, G. Bana, J. Herzog, A. Scedrov Wednesday September 14, 2005 ---------------------------- 09:00 - 10:30 Session 8: Privacy and Data Protection Privacy Preserving Clustering S. Jha, L. Kruger, P. McDaniel Abstractions Preserving Parameter Confidentiality S. Gurgenas, P. Ochsenschlaeger, C. Rudolpah Minimal Disclosure in Hierarchical Hippocratic Databases with Delegation F. Massacci, J. Mylopoulos, N. Zannone 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 9: Security for protocols and devices Security Notions for Disk Encryption K. Gjosteen Local View Attack on Anonymous Communication M. Gogolewski, M. Klonowski, M. Kutylowski Browser Model for Security Analysis of Browser-Based Protocols T. Gross, B. Pfitzmann, A. Sadeghi REGISTRATION Online registration is available on the conference web page: http://esorics05.dti.unimi.it/registration.php ADDITIONAL INFORMATION On the web pages (http://esorics05.dti.unimi.it), you will find information about the program, the conference hotel and venue, and some travel and tourist information. We look forward to seeing you in Milan at ESORICS 2005. From p2p-hackers at ryanb.org Mon Jul 4 18:51:09 2005 From: p2p-hackers at ryanb.org (Ryan Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> References: <71b79fa905070405233cc374ae@mail.gmail.com> <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> Message-ID: On Mon, 4 Jul 2005, Sean C. Rhea wrote: > What you need is a range query operation, and yes, you can do that on a DHT. providing these kinds of operations on top of DHTs has been on my project wishlist for years. specifically, i've always thought tuplespaces had a particularly elegant design and API. they require a central server, though, which is usually a dealbreaker for distributed systems geeks. i'd love to see how close those papers might get to allowing a true tuplespaces API on top of a DHT...kind of the best of both worlds. jxtaspaces, JavaSpaces, and parts of Linda come close, but from what i've seen, their backing stores aren't nearly as mature or efficient as DHTs. more info: http://xml.coverpages.org/tupleSpaces.html -Ryan -- http://snarfed.org/space/ideas From hcai at cse.unl.edu Mon Jul 4 19:05:49 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: Message-ID: <200507041900.j64J0Whu024172@cse.unl.edu> Hi there, I know that some P2P systems as well as research prototypes use Bloom filters as content replications. However, using Bloom filters does not support substring matching such as "how" matches "however", and wildcard matching. Is there any solution for this already? Thanks Hailong From gbildson at limepeer.com Mon Jul 4 19:04:11 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <200507041900.j64J0Whu024172@cse.unl.edu> References: <200507041900.j64J0Whu024172@cse.unl.edu> Message-ID: <1120503851.42c9882b68d60@cyrus.limewire.com> Gnutella encodes length, length-1, length-2 and I believe length-3 to catch the standard suffixes and other common shortenings. Degenerate words and cases are avoided. Not optimal but better than nothing. Thanks -greg Quoting Hailong Cai : > Hi there, > > I know that some P2P systems as well as research prototypes use Bloom > filters as content replications. However, using Bloom filters does not > support substring matching such as "how" matches "however", and wildcard > matching. Is there any solution for this already? > > Thanks > > Hailong > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From hcai at cse.unl.edu Mon Jul 4 19:24:12 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <1120503851.42c9882b68d60@cyrus.limewire.com> Message-ID: <200507041918.j64JIthu024641@cse.unl.edu> I'm interested in more details. Do you know where I can find some documents on this? By "length 1", do you mean that "h" can match "how". If so, it seems we have too many matches for every query. Thanks hailong -----Original Message----- From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] Sent: Monday, July 04, 2005 2:04 PM To: Peer-to-peer development.; Hailong Cai Cc: 'Peer-to-peer development.' Subject: Re: [p2p-hackers] substring matching with Bloom filters Gnutella encodes length, length-1, length-2 and I believe length-3 to catch the standard suffixes and other common shortenings. Degenerate words and cases are avoided. Not optimal but better than nothing. Thanks -greg Quoting Hailong Cai : > Hi there, > > I know that some P2P systems as well as research prototypes use Bloom > filters as content replications. However, using Bloom filters does not > support substring matching such as "how" matches "however", and wildcard > matching. Is there any solution for this already? > > Thanks > > Hailong > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From gbildson at limepeer.com Mon Jul 4 19:25:43 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <200507041918.j64JIthu024641@cse.unl.edu> References: <200507041918.j64JIthu024641@cse.unl.edu> Message-ID: <1120505143.42c98d3763f51@cyrus.limewire.com> We route queries on the two last hops based on a bloom filter bitvector representing keywords in this way. For routing purposes, it is a true or a false so extra trues don't matter. We avoid encoding such short words in shorter forms so "h" would not be in the encoding. I'm not entirely clear on what you are replicating so perhaps that has some significance. Thanks -greg Quoting Hailong Cai : > > I'm interested in more details. Do you know where I can find some documents > on this? By "length 1", do you mean that "h" can match "how". If so, it > seems we have too many matches for every query. > > Thanks > hailong > > -----Original Message----- > From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] > Sent: Monday, July 04, 2005 2:04 PM > To: Peer-to-peer development.; Hailong Cai > Cc: 'Peer-to-peer development.' > Subject: Re: [p2p-hackers] substring matching with Bloom filters > > Gnutella encodes length, length-1, length-2 and I believe length-3 to catch > the > standard suffixes and other common shortenings. Degenerate words and cases > are > avoided. Not optimal but better than nothing. > > Thanks > -greg > > Quoting Hailong Cai : > > > Hi there, > > > > I know that some P2P systems as well as research prototypes use Bloom > > filters as content replications. However, using Bloom filters does not > > support substring matching such as "how" matches "however", and wildcard > > matching. Is there any solution for this already? > > > > Thanks > > > > Hailong > > > > > > > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > From gbildson at limepeer.com Mon Jul 4 19:31:01 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <1120505143.42c98d3763f51@cyrus.limewire.com> References: <200507041918.j64JIthu024641@cse.unl.edu> <1120505143.42c98d3763f51@cyrus.limewire.com> Message-ID: <1120505461.42c98e753ce55@cyrus.limewire.com> I would look at these two papers for insight. http://www.limewire.com/developer/query_routing/keyword%20routing.htm http://aeolusres.homestead.com/files/index.html We don't actually use the full depth routing mentioned in these papers but we did experiment with it. We use a simplified depth for the last two hops. Thanks -greg Quoting gbildson@limepeer.com: > We route queries on the two last hops based on a bloom filter bitvector > representing keywords in this way. For routing purposes, it is a true or a > false so extra trues don't matter. We avoid encoding such short words in > shorter forms so "h" would not be in the encoding. > > I'm not entirely clear on what you are replicating so perhaps that has some > significance. > > Thanks > -greg > Quoting Hailong Cai : > > > > I'm interested in more details. Do you know where I can find some documents > > on this? By "length 1", do you mean that "h" can match "how". If so, it > > seems we have too many matches for every query. > > > > Thanks > > hailong > > > > -----Original Message----- > > From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] > > Sent: Monday, July 04, 2005 2:04 PM > > To: Peer-to-peer development.; Hailong Cai > > Cc: 'Peer-to-peer development.' > > Subject: Re: [p2p-hackers] substring matching with Bloom filters > > > > Gnutella encodes length, length-1, length-2 and I believe length-3 to catch > > the > > standard suffixes and other common shortenings. Degenerate words and cases > > are > > avoided. Not optimal but better than nothing. > > > > Thanks > > -greg > > > > Quoting Hailong Cai : > > > > > Hi there, > > > > > > I know that some P2P systems as well as research prototypes use Bloom > > > filters as content replications. However, using Bloom filters does not > > > support substring matching such as "how" matches "however", and wildcard > > > matching. Is there any solution for this already? > > > > > > Thanks > > > > > > Hailong > > > > > > > > > > > > _______________________________________________ > > > p2p-hackers mailing list > > > p2p-hackers@zgp.org > > > http://zgp.org/mailman/listinfo/p2p-hackers > > > _______________________________________________ > > > Here is a web page listing P2P Conferences: > > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > > > > > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From john.casey at gmail.com Mon Jul 4 21:43:29 2005 From: john.casey at gmail.com (John Casey) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: There have been a few attempts at optimizing these types of join queries on DHTs before. You might like to have a look at "P. Reynolds and A. Vahdat. Efficient peer-to-peer keyword searching." who have created a system that performs incremental set intersection using bloom filters. The other one to look at would be "KSS keyword search set by Omprakash D Gnawali", which basically encodes all the different bi-gram and tri-gram term combinations available in a document. This is pretty excessive in terms of storage. But I guess a variation on the same theme could select higher quality bi-grams, and tri-grams with out storing all possible combinations. On 7/4/05, Reinout van Schouwen wrote: > On Mon, 4 Jul 2005, Davide Carboni wrote: > > > Hi, I'm developing a P2P system where clients must be able to send > > complex queries > > > I'm reading some papers about DHT and I wonder whether or not DHT is a > > viable solution for these requirements. > > I am facing a similar situation. > > The best solution I can think of, currently, is to use the DHT itself to > store information about the relations between properties of Values to > Values belonging to other Keys. > > However, a query like the one you give, would probably require getting > the entire contents of the DHT and evaluating it locally (correct me if > I'm wrong). Unstructured P2P networks like Gnutella allow for partial > queries, but have no guarantee of getting back all relevant matches to > the query. > > Possibly, you want to look at a distributed RDF repository like the one > described in http://wwwconf.ecs.soton.ac.uk/archive/00000599/ > > regards, > > -- > Reinout van Schouwen *** student of Artifical Intelligence > email: reinout@cs.vu.nl *** mobile phone: +31-6-44360778 > www.vanschouwen.info *** help mee met GNOME vertalen: nl.gnome.org > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From em at em.no-ip.com Tue Jul 5 13:38:14 2005 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <006801c58166$dba99ae0$0200a8c0@em.noip.com> ----- Original Message ----- From: "Davide Carboni" To: "Peer-to-peer development." Sent: Monday, July 04, 2005 8:23 PM Subject: [p2p-hackers] DHT and complex queries > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? One simple but quite inefficient way of doing this is to search first for the exact match (author equal "foo") and then filter the results. This assumes that the query always has the format "(key==value) AND ()". Overnet, the Kademlia derivative used by eDonkey 1.0, does precisely that: the "value" stored in the DHT, besides another key for the location search, is a dictionary of metadata made of pairs, with some tags being numeric (e.g., file size) and most others simple strings. The thing is less horrible than it may sound at first, because the filtering is performed "at the server side", in each peer being queried, rather than by the client that queried the DHT. This prevents a lot of unnecessary traffic. Most Overnet clients can only issue simple queries based on the AND of a number of conditions, but in my KadC library (http://kadc.sourceforge.net/ ) I have built a recursive descent parser that can accept arbitrarily complex expressions. I quote from http://kadc.sourceforge.net/Quickstart.html : [...] The filter's syntax is an arbitrarily complex expression involving keywords (case insensitive), "tagname=tagvalue" or "integertagname{relop}integertagvalue" (with being >, <, =, !=, >=, <=), &, | and ! boolean operators, and parentheses to force precedence. For example, the filter: billie&lester&(SIZE!=123|bitrate<=256)&!FORMAT=jpeg ...is parsed as: ((.TRUE. AND_NOT FORMAT=jpeg) AND ((bitrate<257 OR (SIZE>123 OR SIZE<123)) AND (lester AND billie))) For more detail, see the comments in KadCparser.c . [...] (the apparently bizarre translation depends of the lack of "NOT", "GE" and "LE" operators in the syntax of the filters handled by Overnet and eDonkey). Enzo From hcai at cse.unl.edu Tue Jul 5 19:34:19 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <006801c58166$dba99ae0$0200a8c0@em.noip.com> Message-ID: <200507051929.j65JT1hu006155@cse.unl.edu> Hello, everybody As we know, DHTs receive a lot of attention from academy while unstructured systems thrive on the market. I just want to know how these two types of P2P would evolve in the near future, say, in the following 5 years. Which one will prevail and which one will die, or maybe they may merge into some kind of hybrid system? Could you please share your thoughts on this? I don't know if this question has been asked before, but I want to know your ideas at present. A. DHT will succeed and unstructured systems will die B. Unstructured systems will continue growing and DHTs will fade away. C. They will merge into one. D. They will coexist but with different applications. For example, unstructured systems for file sharing, and DHTs for distributed storage system. E. Other Thanks Hailong From agthorr at barsoom.org Tue Jul 5 19:45:14 2005 From: agthorr at barsoom.org (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507051929.j65JT1hu006155@cse.unl.edu> References: <006801c58166$dba99ae0$0200a8c0@em.noip.com> <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: <20050705194514.GC3480@barsoom.org> On Tue, Jul 05, 2005 at 02:34:19PM -0500, Hailong Cai wrote: > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. I expect them to coexist, sometimes for different applications, but sometimes with both in use for the same application. Notably, one of the largest file sharing networks (eDonkey/eMule with 5 million simultaneous users) is DHT-based, though I don't see the unstructured systems vanishing from sight (Gnutella has grown by a factor of 5 in the last year). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From hcai at cse.unl.edu Tue Jul 5 20:05:30 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <20050705194514.GC3480@barsoom.org> Message-ID: <200507052000.j65K0Chu007617@cse.unl.edu> Maybe you mean Overnet? I remember that eDonkey is still unstructured system. Am I wrong? -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Daniel Stutzbach Sent: Tuesday, July 05, 2005 2:45 PM To: 'Peer-to-peer development.' Subject: Re: [p2p-hackers] DHT or unstructured? On Tue, Jul 05, 2005 at 02:34:19PM -0500, Hailong Cai wrote: > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. I expect them to coexist, sometimes for different applications, but sometimes with both in use for the same application. Notably, one of the largest file sharing networks (eDonkey/eMule with 5 million simultaneous users) is DHT-based, though I don't see the unstructured systems vanishing from sight (Gnutella has grown by a factor of 5 in the last year). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From agthorr at barsoom.org Tue Jul 5 20:06:49 2005 From: agthorr at barsoom.org (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507052000.j65K0Chu007617@cse.unl.edu> References: <20050705194514.GC3480@barsoom.org> <200507052000.j65K0Chu007617@cse.unl.edu> Message-ID: <20050705200648.GD3480@barsoom.org> On Tue, Jul 05, 2005 at 03:05:30PM -0500, Hailong Cai wrote: > Maybe you mean Overnet? I remember that eDonkey is still unstructured > system. Am I wrong? eDonkey (the network) is an unstructured network, and Overnet (the network) is a DHT network developed by the same others. However, eDonkey (the software) now uses Overnet (the network) for lookup. Likewise, eMule (the software) originally used eDonkey (the unstructured network) but now also uses Kad (the DHT network). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From mintar at web.de Tue Jul 5 20:31:34 2005 From: mintar at web.de (=?ISO-8859-15?B?TWFydGluIEf8bnRoZXI=?=) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <20050705200648.GD3480@barsoom.org> References: <20050705194514.GC3480@barsoom.org> <200507052000.j65K0Chu007617@cse.unl.edu> <20050705200648.GD3480@barsoom.org> Message-ID: <149337391.20050705223134@web.de> > eDonkey (the network) is an unstructured network, and Overnet (the > network) is a DHT network developed by the same others. However, > eDonkey (the software) now uses Overnet (the network) for lookup. > Likewise, eMule (the software) originally used eDonkey (the > unstructured network) but now also uses Kad (the DHT network). I'm not quite sure if I misunderstood you, but the server-based eDonkey network has not been abandoned. It is still used in parallel with Overnet (in the eDonkey client) and Kad (in the eMule client). If that is what you wanted to say, I'm sorry for wasting your time. Martin From marco at bice.it Wed Jul 6 14:44:32 2005 From: marco at bice.it (marco@bice.it) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] About Pastry Message-ID: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> Hi.. got a strange problem with FreePastry-1.4.1. I tried to compile and execute the test class found in www.cin.ufpe.br/~jbrj/files/pastry, but it didn't work... It doesn't find all the classes I got the right classpath and environment variables.. Oh.. jdk1.4.1 (both on winXP and on Solaris) How to make it works? Any help in order to get started with FreePastry? Thank you, and sorry for my bad english.. From jstewart at cs.rice.edu Wed Jul 6 17:30:40 2005 From: jstewart at cs.rice.edu (James Stewart) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] About Pastry In-Reply-To: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> References: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> Message-ID: <3E8F09E3-DEF0-4AB0-838F-CEDAE0DC24B0@cs.rice.edu> What classes does it not find? Have you tried the tutorials? http://freepastry.rice.edu/FreePastry/tutorial/ Jim On Jul 6, 2005, at 9:44 AM, marco@bice.it wrote: > Hi.. > got a strange problem with FreePastry-1.4.1. > > I tried to compile and execute the test class found in > www.cin.ufpe.br/~jbrj/files/pastry, but it didn't work... > It doesn't find all the classes > > I got the right classpath and environment variables.. > Oh.. jdk1.4.1 (both on winXP and on Solaris) > > How to make it works? > Any help in order to get started with FreePastry? > > Thank you, and sorry for my bad english.. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From mfreed at cs.nyu.edu Thu Jul 7 00:35:47 2005 From: mfreed at cs.nyu.edu (Michael J. Freedman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507051929.j65JT1hu006155@cse.unl.edu> References: <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: At the latest NSDI, Castro et al had a paper "Debunking some myths about structured and unstructured overlays" that argued that 'anything unstructured systems can do, structured systems can do better'. http://www.research.microsoft.com/~antr/MS/myths.pdf Basically, it wasn't providing specific algorithms, e.g., for prefix search as Sean suggested, but instead suggesting that you can leverage the structure for other than DHT queries. For example, (1) if you want a very rich query language -- which is the main reason people suggest using unstructured systems -- you can always improve performance by using the structured overlay to _direct_ the query, as opposed to random walks. (2) Explicitly provide better organization for super-peer hierarchies (heteroPastry), as opposed to ad-hoc techniques. I forget all the details, but it may be of some interest to system builders. Of course, there is more complexity in maintaining a structured system (in terms of implementation), although unstructured systems assume similar amounts of maintenance traffic to ensure good connectivity and liveness of peers. --mike On Tue, 5 Jul 2005, Hailong Cai wrote: > Date: Tue, 5 Jul 2005 14:34:19 -0500 > From: Hailong Cai > Reply-To: Peer-to-peer development. > To: 'Peer-to-peer development.' > Subject: [p2p-hackers] DHT or unstructured? > > Hello, everybody > > As we know, DHTs receive a lot of attention from academy while unstructured > systems thrive on the market. I just want to know how these two types of > P2P would evolve in the near future, say, in the following 5 years. Which > one will prevail and which one will die, or maybe they may merge into some > kind of hybrid system? Could you please share your thoughts on this? > I don't know if this question has been asked before, but I want to know your > ideas at present. > > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. > E. Other > > > Thanks > Hailong ----- www.michaelfreedman.org www.coralcdn.org From em at em.no-ip.com Thu Jul 7 06:32:45 2005 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? References: <20050705194514.GC3480@barsoom.org><200507052000.j65K0Chu007617@cse.unl.edu><20050705200648.GD3480@barsoom.org> <149337391.20050705223134@web.de> Message-ID: <000001c582be$421078a0$0200a8c0@em.noip.com> ----- Original Message ----- From: "Martin G?nther" To: "Peer-to-peer development." Sent: Wednesday, July 06, 2005 4:31 AM Subject: Re[2]: [p2p-hackers] DHT or unstructured? > > eDonkey (the network) is an unstructured network, and Overnet (the > > network) is a DHT network developed by the same others. However, > > eDonkey (the software) now uses Overnet (the network) for lookup. > > > Likewise, eMule (the software) originally used eDonkey (the > > unstructured network) but now also uses Kad (the DHT network). > > I'm not quite sure if I misunderstood you, but the server-based > eDonkey network has not been abandoned. It is still used in parallel > with Overnet (in the eDonkey client) and Kad (in the eMule client). If > that is what you wanted to say, I'm sorry for wasting your time. Also, please note that KAD cannot interoperate with Overnet, due to differences in the protocol. This is regrettable, and due in part to a (most likely deliberate) lack of assistance by MetaMachine, the original creators of the closed-source eDonkey, to the eMule developers. Meanwhile, the team that developed the multi-protocol program mldonlkey managed to reverse-engineer most of the Overnet protocol, initially with some errors that did nothing to support their popularity in the eDonkey camp ;-) Enzo From wolfgang.mueller at wiai.uni-bamberg.de Thu Jul 7 08:18:23 2005 From: wolfgang.mueller at wiai.uni-bamberg.de (Wolfgang =?iso-8859-1?q?M=FCller?=) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: References: <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: <200507071018.23315.wolfgang.mueller@wiai.uni-bamberg.de> On Thursday 07 July 2005 02:35, Michael J. Freedman wrote: > At the latest NSDI, Castro et al had a paper "Debunking some myths about > structured and unstructured overlays" that argued that 'anything > unstructured systems can do, structured systems can do better'. > > http://www.research.microsoft.com/~antr/MS/myths.pdf > > Basically, it wasn't providing specific algorithms, e.g., for prefix > search as Sean suggested, but instead suggesting that you can leverage the > structure for other than DHT queries. For example, (1) if you want a very > rich query language -- which is the main reason people suggest using > unstructured systems -- you can always improve performance by using the > structured overlay to _direct_ the query, as opposed to random walks. In the same vein, it is often overlooked that even *shipping* complex similarity queries is expensive, even if the peer reached does not contribute a result. The Li et al. paper http://pdos.csail.mit.edu/~rtm/papers/search_feasibility.ps assumes a bandwidth budget of about 1 megabyte per query. You can easily spend all that by shipping a 1000byte query to 1000 peers, so you're really tight on how much you are going to spend even disseminating the query. A first step towards reducing that communication cost is doing broadcast in a structured overlay (which contacts each peer once) instead of Gnutella-like flooding (which contacts each peer multiple times). Best, Wolfgang -- Dr. Wolfgang M?ller LS Medieninformatik Universit?t Bamberg Check out the SIG MM web site http://www.sigmm.org From ian at locut.us Thu Jul 7 16:03:22 2005 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> You should take a look at FASD: http://freenet.sourceforge.net/kronfol_final_thesis.pdf It uses a generalisation of Freenet's search algorithm to support "fuzzy" searching (in the case of FASD, using a series of keywords). Essentially, any query where you can determine, for a given key, whether one key is a closer match to the query than another key, or whether it is equally matched, can be searched for. So, for example, if the query is 'key is equal to numeric value 5', then the key '6' would be a closer match than the key '8'. If the query is 'key is equal to string value "hello"', then the key 'heppo' would be a closer match than the key 'fdfdf' (perhaps using the Levenshtein distance algorithm). Of course, keys can be anything, not just integers and strings. For example, a single key could be a set of key-value tuples (apologies for the overloaded use of 'key'). You can also combine queries using boolean operators to form new queries, provided you maintain a way to evaluate the relative closeness of two keys to the query. The FASD paper doesn't really go into these more complex queries, but it does demonstrate that Freenet's search algorithm generalises well, while maintaining its small-world scalability characteristics. Ian. On 4 Jul 2005, at 05:23, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? > Bye. > Davide > > > -- > I have made this letter longer than usual because I lack the time to > make it shorter. > B. Pascal > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From lemonobrien at yahoo.com Thu Jul 7 17:49:26 2005 From: lemonobrien at yahoo.com (Lemon Obrien) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> Message-ID: <20050707174926.75510.qmail@web53606.mail.yahoo.com> i think complex querry calls are redundant in ease of use; think google. Ian Clarke wrote:You should take a look at FASD: http://freenet.sourceforge.net/kronfol_final_thesis.pdf It uses a generalisation of Freenet's search algorithm to support "fuzzy" searching (in the case of FASD, using a series of keywords). Essentially, any query where you can determine, for a given key, whether one key is a closer match to the query than another key, or whether it is equally matched, can be searched for. So, for example, if the query is 'key is equal to numeric value 5', then the key '6' would be a closer match than the key '8'. If the query is 'key is equal to string value "hello"', then the key 'heppo' would be a closer match than the key 'fdfdf' (perhaps using the Levenshtein distance algorithm). Of course, keys can be anything, not just integers and strings. For example, a single key could be a set of key-value tuples (apologies for the overloaded use of 'key'). You can also combine queries using boolean operators to form new queries, provided you maintain a way to evaluate the relative closeness of two keys to the query. The FASD paper doesn't really go into these more complex queries, but it does demonstrate that Freenet's search algorithm generalises well, while maintaining its small-world scalability characteristics. Ian. On 4 Jul 2005, at 05:23, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? > Bye. > Davide > > > -- > I have made this letter longer than usual because I lack the time to > make it shorter. > B. Pascal > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences You don't get no juice unless you squeeze Lemon Obrien, the Third. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050707/f484365e/attachment.html From ian at locut.us Thu Jul 7 17:56:35 2005 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <20050707174926.75510.qmail@web53606.mail.yahoo.com> References: <20050707174926.75510.qmail@web53606.mail.yahoo.com> Message-ID: <15DEE022-4028-4789-BF19-C80EFB65D871@locut.us> On 7 Jul 2005, at 10:49, Lemon Obrien wrote: > i think complex querry calls are redundant in ease of use; think > google. It depends on the application, databases which only supported google- style keyword based searches would be next to useless. Either way, as FASD demonstrates, this technique can be used with simple google-style keyword-based searches. Ian. > > Ian Clarke wrote: > You should take a look at FASD: > > http://freenet.sourceforge.net/kronfol_final_thesis.pdf > > It uses a generalisation of Freenet's search algorithm to support > "fuzzy" searching (in the case of FASD, using a series of keywords). > > Essentially, any query where you can determine, for a given key, > whether one key is a closer match to the query than another key, or > whether it is equally matched, can be searched for. > > So, for example, if the query is 'key is equal to numeric value 5', > then the key '6' would be a closer match than the key '8'. > > If the query is 'key is equal to string value "hello"', then the key > 'heppo' would be a closer match than the key 'fdfdf' (perhaps using > the Levenshtein distance algorithm). > > Of course, keys can be anything, not just integers and strings. For > example, a single key could be a set of key-value tuples (apologies > for the overloaded use of 'key'). You can also combine queries using > boolean operators to form new queries, provided you maintain a way to > evaluate the relative closeness of two keys to the query. > > The FASD paper doesn't really go into these more complex queries, but > it does demonstrate that Freenet's search algorithm generalises well, > while maintaining its small-world scalability characteristics. > > Ian. > > On 4 Jul 2005, at 05:23, Davide Carboni wrote: > > > Hi, I'm developing a P2P system where clients must be able to send > > complex queries such as: > > > > author equal "foo" > > and > > publicationDate is between "12 12 1999" and "12 12 2000" > > andnot > > distributor startswith "galax" > > > > etc. etc. > > > > I'm reading some papers about DHT and I wonder whether or not DHT > is a > > viable solution for these requirements. I know that with DHT you can > > map a key onto a piece of data, but I cannot see how to manage > complex > > query like the one in the example. Probably DHT is not the "good" > > solution for this...but never know, > > > > Any hints? > > Bye. > > Davide > > > > > > -- > > I have made this letter longer than usual because I lack the time to > > make it shorter. > > B. Pascal > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > You don't get no juice unless you squeeze > Lemon Obrien, the Third. > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050707/0a261f23/attachment.htm From wolfgang.mueller at wiai.uni-bamberg.de Thu Jul 7 20:28:43 2005 From: wolfgang.mueller at wiai.uni-bamberg.de (Wolfgang Mueller) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <20050707174926.75510.qmail@web53606.mail.yahoo.com> References: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> <20050707174926.75510.qmail@web53606.mail.yahoo.com> Message-ID: <20050707202843.GA20571@portos.uni-bamberg.de> Hi > i think complex querry calls are redundant in ease of use; think google. Google is not the end of the line. Examples are location based services or similarity queries for multimedia data. Cheers, Wolfgang From dbarrett at quinthar.com Fri Jul 8 01:58:28 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 Message-ID: <42CDDDC4.9090001@quinthar.com> Can you recommend a good resource for getting started with Universal Plug-n-Play on Win32 for NAT traversal purposes? I see the various UPnP APIs (Control Point, Device Host, etc.) in the MSDN docs, but I don't see specific mention of how to apply them to NAT configuration. Rather, they seem to give an overall approach for configuring an arbitrary UPnP device, with details on specific devices left as "an exercise to the reader". Can you give me the 30-second overview of COM objects and function calls I need to use? Or can you refer me to a decent overview of the subject? If nothing else, do you know of some sample code that shows configuring a NAT device to allow incoming data? Thanks! -david From dbarrett at quinthar.com Fri Jul 8 05:17:07 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <42CDDDC4.9090001@quinthar.com> References: <42CDDDC4.9090001@quinthar.com> Message-ID: <42CE0C53.80208@quinthar.com> Ah, I'll answer my own question: It appears WinXP has another API in the Platform SDK named "Network Address Translation". It's a wrapper over the UPnP NAT functionality (thus you don't touch the UPnP API directly). Rather, you use these basic steps: 1) Create a UPnPNAT COM object and get its IUPnPNAT interface 2) Call IUPnPNAT::get_StaticPortMappingCollection( ) 3) IStaticPortMappingCollection::Add( ) to add configure a port forward At least, that's the theory. A C++ file showing it in action is here: http://cvs.sourceforge.net/viewcvs.py/dcplusplus/dcplusplus/windows/UPnP.cpp?rev=1.3 Not sure how well the whole thing works and it only appears to work under XP, but it's a place to start. -david David Barrett wrote: > Can you recommend a good resource for getting started with Universal > Plug-n-Play on Win32 for NAT traversal purposes? > > I see the various UPnP APIs (Control Point, Device Host, etc.) in the > MSDN docs, but I don't see specific mention of how to apply them to NAT > configuration. Rather, they seem to give an overall approach for > configuring an arbitrary UPnP device, with details on specific devices > left as "an exercise to the reader". > > Can you give me the 30-second overview of COM objects and function calls > I need to use? Or can you refer me to a decent overview of the subject? > If nothing else, do you know of some sample code that shows > configuring a NAT device to allow incoming data? > > Thanks! > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From sbest at best.com Fri Jul 8 21:06:50 2005 From: sbest at best.com (Scott C. Best) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <20050708190004.C47583FD76@capsicum.zgp.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> Message-ID: <20050708210457.I26504@kaboodle.org> David: Any mention in there how it handles multiple WinXP PC's on the LAN in contention for the same port-forwarding adjustment? cheers, Scott > Ah, I'll answer my own question: > > It appears WinXP has another API in the Platform SDK named "Network > Address Translation". It's a wrapper over the UPnP NAT functionality > (thus you don't touch the UPnP API directly). Rather, you use these > basic steps: > > 1) Create a UPnPNAT COM object and get its IUPnPNAT interface > 2) Call IUPnPNAT::get_StaticPortMappingCollection( ) > 3) IStaticPortMappingCollection::Add( ) to add configure a port forward > > At least, that's the theory. A C++ file showing it in action is here: > > http://cvs.sourceforge.net/viewcvs.py/dcplusplus/dcplusplus/windows/UPnP.cpp?rev=1.3 > > Not sure how well the whole thing works and it only appears to work > under XP, but it's a place to start. > > -david From dbarrett at quinthar.com Fri Jul 8 21:16:35 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <20050708210457.I26504@kaboodle.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> <20050708210457.I26504@kaboodle.org> Message-ID: <1120857399.24E01522@dj11.dngr.org> On Fri, 8 Jul 2005 2:10 pm, Scott C. Best wrote: > Any mention in there how it handles multiple WinXP PC's > on the LAN in contention for the same port-forwarding adjustment? No. Nor is there mention on lease durations, timeouts, or refresh periods. Horray for well-designed Microsoft APIs! For the first I assume Add( ) will just fail, and I guess I'll just try a handful of random assignments until it works (I don't see any ability to let it pick the mapping and tell me what it picked). For the second, no idea. -david From coderman at gmail.com Fri Jul 8 22:07:20 2005 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <1120857399.24E01522@dj11.dngr.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> <20050708210457.I26504@kaboodle.org> <1120857399.24E01522@dj11.dngr.org> Message-ID: <4ef5fec60507081507512c174e@mail.gmail.com> On 7/8/05, David Barrett wrote: > On Fri, 8 Jul 2005 2:10 pm, Scott C. Best wrote: > > Any mention in there how it handles multiple WinXP PC's > > on the LAN in contention for the same port-forwarding adjustment? > > No. Nor is there mention on lease durations, timeouts, or refresh > periods. Horray for well-designed Microsoft APIs! a useful example of UPnP NAT forwarding between XP and Linksys is given here (in VB): http://www.knoxscape.com/Upnp/NAT.htm you can see that there are a handful of common forwards available, which are returned by the call to get_StaticPortMappingCollection( ). they are probably not actively forwarded to anything unless you configured the services in XP and it did so for you. you can add additional forwards as mentioned with StaticPortMappingCollection::Add( ) but there is a low limit (5??) and whoever gets the mapping first wins. thus if you try to do this from another application you will see the assignment already in place when get_StaticPortMappingCollection( ) is called, and you would have to explicitly remove it before adding it to a different internal endpoint. i'm not aware of any timeouts for the forwards unless the host DHCP lease times out, and then even this is implementation / vendor dependant. From dbarrett at quinthar.com Fri Jul 8 23:42:10 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 References: <1120865042.B84CED1@di11.dngr.org> Message-ID: <1120866139.2DB61DBF@dk11.dngr.org> On Fri, 8 Jul 2005 3:08 pm, coderman wrote: > you can add additional forwards as mentioned with > StaticPortMappingCollection::Add( ) but there is a low limit (5??) and > whoever gets the mapping first wins. thus if you try to do this from > another application you will see the assignment already in place when > get_StaticPortMappingCollection( ) is called, and you would have to > explicitly remove it before adding it to a different internal > endpoint. Thanks for the link. As for a 5-port limit, that's pretty tight in a multi-PC LAN. Any idea if 5 is on the low side, or if they're all this restrictive? As for timeouts, I'm concerned that there's no apparent "cleanup" process to mappings. Say I configure a static mapping and the process is killed -- will that mapping stay forever? Naturally my app can be a bastard and just blow away any mappings that get in its way, but that's not the most elegant solution. -david From dbarrett at quinthar.com Sun Jul 10 01:30:38 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range Message-ID: <42D07A3E.7030408@quinthar.com> From range of ports should I randomly pick a UDP port mapping? The IANA states "The Dynamic and/or Private Ports are those from 49152 through 65535". Thus I'm considering just randomly selecting from within this range. However, I'm curious if you have any experience with this and and either encourage or dissuade me along this path? I personally don't care which port I use, so long as I don't stumble into some "known" minefield. Do firewalls often block this range? Do NATs do anything funky with this range? Is there any reason to use one UDP port over another? Should I secretly use on of the registered ranges to improve my NAT/firewall penetration odds? An alternate plan is to choose an unregistered range from the big IANA list (http://www.iana.org/assignments/port-numbers) but if there's no reason not to, I'd just as soon stick with the big unregistered dynamic range. Any suggestions? And finally, I'm going to try to configure a UPnP port-mapping using this randomly-selected port as my external NAT port. Does this change your recommendations at all? -david From dbarrett at quinthar.com Sun Jul 10 03:53:11 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Automatic proxy detection / tunneling Message-ID: <42D09BA7.80105@quinthar.com> What is the best method for a client to detect and use any web proxy that separates it from the Internet? Under Win32 I see the functions "WinHttpGetProxyForUrl( )" and "WinHttpGetIEProxyConfigForCurrentUser( )" which help discover the proxy. What I specifically don't see are: 1) What type of proxy do these functions return? The return value contains a list in the form of: ([=]["://"][":"]) What is the range of possible "schemes"? Will this be something like "socksv4" or "socksv5" or what? 2) If it's a SOCKSv5 proxy, how do I determine what kind of authentication method and credentials to use? 3) If it's not a SOCKS proxy, what kind is it? The HTTP protocol RFC mentions proxies and gateways (like the Squid web-cache), but they are implied to be wholly transparent. Am I wrong? What other kind of proxies exist? Basically, I want to tunnel out of restrictive enterprises that employ firewalls and web proxies. But I'm not entirely sure what comprises these deployments. Can you offer any hints to point me in the right direction? Thanks! -david However, this is Win32-dependent code. Do you know: 1) Is there a platform-independent library available that auto-detects web-proxies and executes any resulting PAC From afisk at speedymail.org Mon Jul 11 17:38:42 2005 From: afisk at speedymail.org (Adam Fisk) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range In-Reply-To: <42D07A3E.7030408@quinthar.com> References: <42D07A3E.7030408@quinthar.com> Message-ID: <42D2AEA2.5060309@speedymail.org> The only thing that comes to mind here is that some apps like RealPlayer and likely Windows Media Player and Quicktime select UDP ports from fairly broad ranges, at least the last time I checked. I don't remember what those ranges were, but it's quite likely they're choosing a subset of the dynamic range as well, so you might just want to take a second to check for an overlap. It's probably fine and not really something to worry about -- just putting it on your radar. -Adam David Barrett wrote: > From range of ports should I randomly pick a UDP port mapping? > > The IANA states "The Dynamic and/or Private Ports are those from 49152 > through 65535". Thus I'm considering just randomly selecting from > within this range. > > However, I'm curious if you have any experience with this and and > either encourage or dissuade me along this path? I personally don't > care which port I use, so long as I don't stumble into some "known" > minefield. Do firewalls often block this range? Do NATs do anything > funky with this range? Is there any reason to use one UDP port over > another? Should I secretly use on of the registered ranges to improve > my NAT/firewall penetration odds? > > An alternate plan is to choose an unregistered range from the big IANA > list (http://www.iana.org/assignments/port-numbers) but if there's no > reason not to, I'd just as soon stick with the big unregistered > dynamic range. > > Any suggestions? > > And finally, I'm going to try to configure a UPnP port-mapping using > this randomly-selected port as my external NAT port. Does this change > your recommendations at all? > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From dbarrett at quinthar.com Tue Jul 12 00:24:11 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range In-Reply-To: <42D2AEA2.5060309@speedymail.org> References: <42D07A3E.7030408@quinthar.com> <42D2AEA2.5060309@speedymail.org> Message-ID: <42D30DAB.4020501@quinthar.com> That's good to keep in mind, thanks. Adam Fisk wrote: > The only thing that comes to mind here is that some apps like RealPlayer > and likely Windows Media Player and Quicktime select UDP ports from > fairly broad ranges, at least the last time I checked. I don't remember > what those ranges were, but it's quite likely they're choosing a subset > of the dynamic range as well, so you might just want to take a second to > check for an overlap. It's probably fine and not really something to > worry about -- just putting it on your radar. > > -Adam > > > David Barrett wrote: > >> From range of ports should I randomly pick a UDP port mapping? >> >> The IANA states "The Dynamic and/or Private Ports are those from 49152 >> through 65535". Thus I'm considering just randomly selecting from >> within this range. >> >> However, I'm curious if you have any experience with this and and >> either encourage or dissuade me along this path? I personally don't >> care which port I use, so long as I don't stumble into some "known" >> minefield. Do firewalls often block this range? Do NATs do anything >> funky with this range? Is there any reason to use one UDP port over >> another? Should I secretly use on of the registered ranges to improve >> my NAT/firewall penetration odds? >> >> An alternate plan is to choose an unregistered range from the big IANA >> list (http://www.iana.org/assignments/port-numbers) but if there's no >> reason not to, I'd just as soon stick with the big unregistered >> dynamic range. >> >> Any suggestions? >> >> And finally, I'm going to try to configure a UPnP port-mapping using >> this randomly-selected port as my external NAT port. Does this change >> your recommendations at all? >> >> -david >> _______________________________________________ >> p2p-hackers mailing list >> p2p-hackers@zgp.org >> http://zgp.org/mailman/listinfo/p2p-hackers >> _______________________________________________ >> Here is a web page listing P2P Conferences: >> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences >> > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From distoage at sbbi.net Tue Jul 12 11:21:49 2005 From: distoage at sbbi.net (DiSToAGe) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <42CDDDC4.9090001@quinthar.com> References: <42CDDDC4.9090001@quinthar.com> Message-ID: <1121167309.7405.144.camel@station.local> Le jeudi 07 juillet 2005 ? 18:58 -0700, David Barrett a ?crit : > Can you recommend a good resource for getting started with Universal > Plug-n-Play on Win32 for NAT traversal purposes? > if you use java (or want to inspire from), perhaps I can suggest you our lib. We have developped UPNPLib a java based lib to use UPnP, you can find source too, and various documentations and examples about it and explain of UPnP usable for NAT traversal. http://www.sbbi.net/site/upnp/ (docs) http://www.sbbi.net/site/upnp/docs/discovery.html (our server using UPnP for NAT traversal) http://www.sbbi.net/site/jafs/docs/upnp-nat.html -- "Perhaps one day "computer science" will, like Yugoslavia, get broken up into its component parts. That might be a good thing. Especially if it meant independence for my native land, hacking." (hackers and Painters) [ Paul Graham ] From dbarrett at quinthar.com Tue Jul 12 21:01:07 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <1121167309.7405.144.camel@station.local> References: <42CDDDC4.9090001@quinthar.com> <1121167309.7405.144.camel@station.local> Message-ID: <42D42F93.3050701@quinthar.com> DiSToAGe wrote: > if you use java (or want to inspire from), perhaps I can suggest you our > lib. We have developped UPNPLib a java based lib to use UPnP, you can > find source too, and various documentations and examples about it and > explain of UPnP usable for NAT traversal. Thanks for the links. I *think* I've got the UPnP code working, but I don't have a UPnP network (and the half-dozen or so networks I've randomly tested on didn't appear to be UPnP compatible). Would you mind testing my application on your network to see if my UPnP code works? (I'm using C++, so alas I can't really use your lib.) Thanks! -david From huaiyu at mail.utexas.edu Wed Jul 13 20:39:35 2005 From: huaiyu at mail.utexas.edu (huaiyu@mail.utexas.edu) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] CollaborateCom-DCCS 2005 Call for papers Message-ID: <1121287175.42d57c07cde58@webmailapp1.cc.utexas.edu> [Apologies if you receive multiple copies of this message] ------------------------------------------------------------------------------- CALL FOR PAPERS First International Workshop on Data Consistency in Collaborative Systems (DCCS 2005) http://www.dataconsistency.org In conjunction with the IEEE CollaborateCom 2005 San Jose, USA, December 18, 2005 ------------------------------------------------------------------------------- SCOPE: Collaborative systems enable collaboration between people and computers to work more productively in acquiring, representing, organizing, archiving, protecting, accessing and communicating information. Consistency is one of the most important and classical issues in collaborative systems research, ranging from the chip level to the system level. Recent advances in collaborative systems have introduced several new challenges to the consistency maintenance in a decentralized, heterogeneous environment. This workshop aims to bring together researchers from various areas of computer science whose work is related to data replication, caching and consistency, including distributed systems, distributed database, Web caching and content delivery network, computer supported collaborative work (CSCW) systems, and so on. In particular, we are interested in the latest progress on data consistency in collaborative systems, including data consistency models, algorithms, systems, and applications. We solicit papers of original research including, but not limited to, the following topics: * Cache modeling and analysis * Data consistency models * Cache coherence protocol * File and storage caching * Web caching and content delivery network * Caching and consistency in P2P systems * Secure data consistency * Concurrency control algorithms performance analysis * Consistency issues in wireless sensor networks Submission Guidelines Authors are invited to submit papers that demonstrate original and unpublished research in these areas. Papers should be no more than 10 pages in length (including references and figures) with in IEEE double column format. All submissions should be in pdf format via the cocus submission web site. In order to submit a paper to the conference, you should first register on the conference management system: http://cocus.create-net.it:8080/confsys/loadRegister.do?act=completeReg Shortly after completing the registration, you will get by e-mail a password. Submission of a paper should be regarded as an undertaking that, should the paper be accepted, at least one of the authors will register and attend the workshop to present the work. Important Dates Paper Submission Deadline: September 1, 2005 Notification of Acceptance: October 1, 2005 Camera Ready Papers Due: October 15, 2005 Workshop Co-Chairs Du Li, Texas A&M University Email: lidu@cs.tamu.edu Weisong Shi, Wayne State University Email: weisong@wayne.edu Program Committee: Monica Brockmeyer, Wayne State University Anca Andrean Ivan, IBM T. J. Watson Research Center Du Li, Texas A&M University (Co-Chair) Huaiyu Liu, University of Texas at Austin Christine Morin, INRIA, France Guillaume Pierre, Vrije University, Netherlands Weisong Shi, Wayne State University (Co-Chair) Evan Speight, IBM Austin Research Lab Chengzheng Sun, Griffith University, Australia Li Xiao, Michigan State University From matthew at matthew.at Mon Jul 18 05:45:13 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement Message-ID: <200507180545.j6I5jKU63721@where.matthew.at> For the past year and a half here at amicima, we've been developing some new protocol technology particularly suited for multimedia transport and p2p applications. A few months ago, we made the decision to open-source this technology, and we've been getting it ready for public release. The main announcement of the first big piece, the Secure Media Flow Protocol (MFP) implementation, is just a couple weeks away (we hope... things can move slowly at non-funded software startups), but in the meantime, I thought that folks on this list might like to get a taste of what's to come, so here's some links, in a shameless self-promotion of our work: Technology overview: http://www.amicima.com/technology/index.html The downloads page: http://www.amicima.com/developers/downloads.html (Our BSD-licensed open-source object library for doing sophisticated object-oriented programming in plain ANSI C is already available for download there, the various MFP layers will be up as soon as we can get them ready to ship, dual-licensed... GPL and an available commercial license for proprietary apps) And some initial protocol documentation: http://www.amicima.com/developers/documentation.html (The implementation of the protocol described here will be our next release... coming very soon... runs without modification on Windows, Mac OS X, and Unix) We've used this to build several applications, including a Skype-like VOIP app that runs on Mac OS X and Windows XP. Using our underlying protocol and p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through NAT) computer-to-computer calls, by name, to mobile endpoints, including fast file transfer that doesn't impair the quality of the voice call, took less than a week to whip up a working prototype. If there's demand, we'll probably release that as a sample app with sources, once the MFP and MFPNet layers and some other supporting libraries are available for download. Matthew Kaufman amicima, Inc. matthew@amicima.com matthew@matthew.at Ps. One of the reasons I'm putting this preannouncement out on the p2p-hackers list is that several folks on the list have mailed in the past asking for more information about what we're doing at amicima and when they can play with the technology, and I've managed to delete about half my list of who those people were. From ardagna at dti.unimi.it Mon Jul 18 07:33:46 2005 From: ardagna at dti.unimi.it (Claudio Agostino Ardagna) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] ESORICS 2005 - Call for Participation Message-ID: <00df01c58b6b$07fab6c0$1100000a@Berlino> [Apologies if you receive multiple copies of this message] CALL FOR PARTICIPATION ESORICS 2005 10TH EUROPEAN SYMPOSIUM ON RESEARCH IN COMPUTER SECURITY Milan, Italy - September 12-14, 2005 http://esorics05.dti.unimi.it/ AIMS AND SCOPE Organized in a series of European countries, ESORICS is confirmed as the European research event in computer security. The symposium started in 1990 and has been held on alternate years in different European countries and attracts an international audience from both the academic and industrial communities. From 2002 it has been held yearly. The Symposium has established itself as one of the premiere, international gatherings on information assurance. PRELIMINARY PROGRAM Monday September 12, 2005 ------------------------- 09:15 - 09:30 Welcome and Opening 09:30 - 10:30 Invited talk -- Barbara Simons 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 1: Access control XML Access Control with Policy Matching Tree N. Qi, M. Kudo Semantic Access Control Model: A Formal Specification M. I. Yague, M. Gallardo, A. Mana A Generic XACML Based Declarative Authorization Scheme for Java R. Gupta, M. Bhide 12:30 - 14:00 Lunch 14:00 - 15:30 Session 2: Advanced Authorization Specifications Specification and Validation of Authorisation Constraints Using UML and OCL K. Sohr, G. Ahn Unified Index for Mobile Object Data and Authorizations V. Atluri, Q. Guo On Obligations M. Hilty, D. Basin, A. Pretschner 15:30 - 16:00 Coffe break 16:00 - 17:30 Session 3: Cryptographic Schemes A Practical, Voter-Verifiable Election Scheme D. Chaum, P.Y.A. Ryan, S.Schneider Machine-Checked Security Proofs of Cryptographic Signature Schemes S. Tarento Sanitizable Signatures G. Ateniese, D. Chou, B. de Medeiros, G. Tsudik Tuesday September 13, 2005 -------------------------- 09:00 - 10:30 Session 4: Cryptographic Protocols Limits of the Cryptographic Realization of Dolev-Yao-style XOR M. Backes, B. Pfitzmann Secure Implementation of Cryptographic Protocols: A Case Study Of Mutual Distrust A. Askarov, A. Sabelfeld Augmented oblivious Polynomial Evaluation Protocol and Its Applications H. Zhu 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 5: Intrusion detection Using Attack Trees to Identify Malicious Attacks from Authorized Insiders I. Ray, N. Poolsapassit An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Network Intrusion Alerts L. Wang, A. Liu, S. Jajodia Towards a Theory of Intrusion Detection G. Di Crescenzo, A. Ghosh, R. Talpade 12:30 - 14:00 Lunch 14:00 - 15:30 Session 6: Network security On Scalability and Modularisation in the Modelling of Network Security Systems J. de Albuquerque, H. Krumm, P. de Geus Sybil resistant DHT routing G. Danezis, R. Anderson Botnet Tracking: Exploring a Root-Cause Methodology to Prevent Distributed Denial-of-Service Attacks F.C. Freiling, T. Holz, G. Wicherski 15:30 - 16:00 Coffee break 16:00 - 17:30 Session 7: Information Flow and Formal Security Properties Quantifying Probabilistic Information Flow in Computational Reactive Systems M. Backes Enforcing Non-safety Security Policies with Program Monitors J. Ligatti, L. Bauer, D. Walker Soundness of Formal Encryption in the Presence of Key-Cycles P. Adao, G. Bana, J. Herzog, A. Scedrov Wednesday September 14, 2005 ---------------------------- 09:00 - 10:30 Session 8: Privacy and Data Protection Privacy Preserving Clustering S. Jha, L. Kruger, P. McDaniel Abstractions Preserving Parameter Confidentiality S. Gurgenas, P. Ochsenschlaeger, C. Rudolpah Minimal Disclosure in Hierarchical Hippocratic Databases with Delegation F. Massacci, J. Mylopoulos, N. Zannone 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 9: Security for protocols and devices Security Notions for Disk Encryption K. Gjosteen Local View Attack on Anonymous Communication M. Gogolewski, M. Klonowski, M. Kutylowski Browser Model for Security Analysis of Browser-Based Protocols T. Gross, B. Pfitzmann, A. Sadeghi REGISTRATION Online registration is available on the conference web page: http://esorics05.dti.unimi.it/registration.php ADDITIONAL INFORMATION On the web pages (http://esorics05.dti.unimi.it), you will find information about the program, the conference hotel and venue, and some travel and tourist information. We look forward to seeing you in Milan at ESORICS 2005. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050718/599a6786/attachment.html From ap at hamachi.cc Mon Jul 18 16:30:01 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507180545.j6I5jKU63721@where.matthew.at> References: <200507180545.j6I5jKU63721@where.matthew.at> Message-ID: <42DBD909.1020703@hamachi.cc> Matthew, I can't seem to find the mentioning of replay protection in your protocol. Also encrypted packets don't seem to carry any sort of HMAC, which is also a bit odd. Perhaps these details are just missing from the spec, in which case it might make sense to produce a separate security-oriented description of the protocol. Alex Matthew Kaufman wrote: > For the past year and a half here at amicima, we've been developing some new > protocol technology particularly suited for multimedia transport and p2p > applications. A few months ago, we made the decision to open-source this > technology, and we've been getting it ready for public release. > > The main announcement of the first big piece, the Secure Media Flow Protocol > (MFP) implementation, is just a couple weeks away (we hope... things can > move slowly at non-funded software startups), but in the meantime, I thought > that folks on this list might like to get a taste of what's to come, so > here's some links, in a shameless self-promotion of our work: > > Technology overview: > http://www.amicima.com/technology/index.html > > The downloads page: > http://www.amicima.com/developers/downloads.html > (Our BSD-licensed open-source object library for doing sophisticated > object-oriented programming in plain ANSI C is already available for > download there, the various MFP layers will be up as soon as we can get them > ready to ship, dual-licensed... GPL and an available commercial license for > proprietary apps) > > And some initial protocol documentation: > http://www.amicima.com/developers/documentation.html > (The implementation of the protocol described here will be our next > release... coming very soon... runs without modification on Windows, Mac OS > X, and Unix) > > We've used this to build several applications, including a Skype-like VOIP > app that runs on Mac OS X and Windows XP. Using our underlying protocol and > p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through > NAT) computer-to-computer calls, by name, to mobile endpoints, including > fast file transfer that doesn't impair the quality of the voice call, took > less than a week to whip up a working prototype. If there's demand, we'll > probably release that as a sample app with sources, once the MFP and MFPNet > layers and some other supporting libraries are available for download. > > Matthew Kaufman > amicima, Inc. > matthew@amicima.com > matthew@matthew.at > > Ps. One of the reasons I'm putting this preannouncement out on the > p2p-hackers list is that several folks on the list have mailed in the past > asking for more information about what we're doing at amicima and when they > can play with the technology, and I've managed to delete about half my list > of who those people were. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From matthew at matthew.at Mon Jul 18 19:42:37 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DBD909.1020703@hamachi.cc> Message-ID: <200507181942.j6IJgjU65031@where.matthew.at> Alex Pankratov: > > Matthew, > > I can't seem to find the mentioning of replay protection in > your protocol. Also encrypted packets don't seem to carry any > sort of HMAC, which is also a bit odd. Perhaps these details > are just missing from the spec, in which case it might make > sense to produce a separate security-oriented description of > the protocol. > > Alex The packet-level protocol spec is admittedly short on information about how exactly to use the protocol (and a followup document is being edited and will be up soon). However if you examine the brief description of the startup sequence, you get some hints as to how this works. The goal is to get an encrypted session up in as few round trip times as possible, so the session keying is overlapped with the session handshaking. Then flows can come up simply by sending data on them, so subsequent flows on an established session come up in even fewer round trip times. The Simplified Digital Certificate format is also compact, so that fragmentation can likely be avoided. So specifically, if you assume RSA and AES (or something else that lets you have public-key crypto and fast symmetric crypto, noting that the choice of cryptosystem is not mandated by the protocol, and in fact you could run with no cryptography and simply get replay resistance that is "as good as TCP" (but a protocol that would still be more resistant to "SYN flood" attacks))... A Responder Hello chunk could be forged, by replying with the correct tag data echo, an arbitrary cookie, and a copy of someone else's certificate. However, when the other end sends its Initiator Initial Keying, the session key to be used is encypted with the public key in that certificate, so unless you also have the matching private key, forging a Responder Hello doesn't help you except to possibly cause denial of service. Our implementation supports parallel-open, so if *any* of the endpoints being opened to is actually the right one, you'll still get a session up (better than TCP's situation in that case). And clearly we can't protect against the case where all packets to the valid destination are intercepted and discarded. The Initiator Initial Keying chunk is also signed using the private key associated with the Initator's certificate which is included, and that signature covers the Cookie Echo field... That protects against replay in one direction, in that the Responder got to choose the (new, unique) cookie data, and is forcing the Initiator to prove that they have the private key (and aren't simply replaying) by signing that unique data. At the same time, the Initiator gets to choose the (new, unique) Session Key, and by extension is choosing the Encrypted Session Key. The Responder is then forced to prove that they have their private key (and aren't simply replaying) by successfully decrypting that session key using their private key. If the Responder fails to do that, then it can't send a Responder Keying Ack encrypted with the proper key. Replay of session data after the session is up then only works if the session that is being replayed has a Session ID and a Session Key which match a session which is up, otherwise the data is discarded at the receiver as being unrecognized. We make the assumption that this is very unlikely except in the case where data is simply being replayed after a relatively short delay by a third party. This replay is also limited to entire intact packets, because the encryption prevents the contents from being modified or added to without damaging the packet such that it will be ignored at the receiver, unless the attacker was able to get the session key by a brute force attack on AES, or by monitoring the session startup and having the private RSA key of an endpoint. If the third party doing the replaying forges the source IP address to match the real sender, then it simply looks like duplicated packets, and the sequence numbering used on all the flows and acknowledgements is already prepared to deal just fine with duplicated or reordered packets, even for unreliable or unordered-reception flows, so there's no effect on the application. If the third party doing the replaying uses a new source IP address, then it looks to our implementation like IP mobility is occuring... that is, an endpoint has changed its IP address. However, in our IP mobility support, the return data flow is not moved to the "new" IP address until a unique and timestamped message sent to the "new" IP address can be echoed back (with modification) from that "new" address... Thus, unless the attacker can actually decrypt the session traffic, they will be unable to provide the correct return data, and so the packets from the "new" address will simply be ignored as above, except for the rate-limited probes to see if actual mobility has occurred. As for a per-packet HMAC, this is most valuable for non-encrypted data. An HMAC is a hash generated from the combination of the packet data and a shared secret... We essentially have this, in that a session key (a shared secret) is used to encrypt the packet and its checksum. If one wished to use the MFP protocol unencrypted, a valuable extension might be to provide for a separate HMAC, but we expect most users to run with encryption, since these days everything really should be encrypted, and with MFP there is no round-trip-time penalty to using it, nor any restriction on passing unreliable or unordered data like you'd have with SSL over TCP. Matthew Kaufman matthew@amicima.com matthew@matthew.at www.amicima.com From ap at hamachi.cc Tue Jul 19 04:08:56 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507181942.j6IJgjU65031@where.matthew.at> References: <200507181942.j6IJgjU65031@where.matthew.at> Message-ID: <42DC7CD8.50605@hamachi.cc> Matthew Kaufman wrote: > Alex Pankratov: > >>Matthew, >> >>I can't seem to find the mentioning of replay protection in >>your protocol. Also encrypted packets don't seem to carry any >>sort of HMAC, which is also a bit odd. Perhaps these details >>are just missing from the spec, in which case it might make >>sense to produce a separate security-oriented description of >>the protocol. >> >>Alex > > > The packet-level protocol spec is admittedly short on information about how > exactly to use the protocol (and a followup document is being edited and > will be up soon). However if you examine the brief description of the > startup sequence, you get some hints as to how this works. > [snip] Matthew, Thanks for a detailed reply. I guess my main concern with security portion of your protocol is that it does not follow certain design properties found in nearly all commonly used transport protocols like SSL/TLS, SSH2, ESP as well as a handful of others like IKE, JFK, etc. Few I can think of off hand are - * the use of MACs. Relying on validity marker in plaintext payload to detect corrupted packets is expensive computationally-wise in a worst case scenario, while HMAC-based authentication costs pennies. It also 'not good' from cryptographical perspective, but I cannot comment on that (note though how SSH was switched to using MACs in v2). Besides MACs are also required for implementing - * replay protection, which involves tagging packets with unique sequence numbers after the encryption but before the authentication. This way replayed or duplicate packets can be discarded without wasting any time on decryption or authentication at all. * session key being derived in 'symmetrical way', when both parties contribute to the key in equal degree. Basically security component of your protocol looks very different from existing mature protocols. And I would suspect that this will not facilitate adoption of MFP .. at least not in its secured form. Alex From dbarrett at quinthar.com Tue Jul 19 05:38:16 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507180545.j6I5jKU63721@where.matthew.at> References: <200507180545.j6I5jKU63721@where.matthew.at> Message-ID: <42DC91C8.7050108@quinthar.com> Matthew -- This is great; I'm glad to hear the release is imminent. I was *really* impressed with what I saw in your demo several months ago and I think it's great your opening up the code for others to license and use. I'm particularly interested in the excellent work you've done with congestion control and link-speed determination. However, excepting a couple curious flags, I didn't see these powerful features called out in the protocol description. Are these handled at the MFP layer, or higher up in the stack? Regardless, congratulations on taking the big step on opening it up; I hope to follow in your footsteps soon enough! -david Matthew Kaufman wrote: > For the past year and a half here at amicima, we've been developing some new > protocol technology particularly suited for multimedia transport and p2p > applications. A few months ago, we made the decision to open-source this > technology, and we've been getting it ready for public release. > > The main announcement of the first big piece, the Secure Media Flow Protocol > (MFP) implementation, is just a couple weeks away (we hope... things can > move slowly at non-funded software startups), but in the meantime, I thought > that folks on this list might like to get a taste of what's to come, so > here's some links, in a shameless self-promotion of our work: > > Technology overview: > http://www.amicima.com/technology/index.html > > The downloads page: > http://www.amicima.com/developers/downloads.html > (Our BSD-licensed open-source object library for doing sophisticated > object-oriented programming in plain ANSI C is already available for > download there, the various MFP layers will be up as soon as we can get them > ready to ship, dual-licensed... GPL and an available commercial license for > proprietary apps) > > And some initial protocol documentation: > http://www.amicima.com/developers/documentation.html > (The implementation of the protocol described here will be our next > release... coming very soon... runs without modification on Windows, Mac OS > X, and Unix) > > We've used this to build several applications, including a Skype-like VOIP > app that runs on Mac OS X and Windows XP. Using our underlying protocol and > p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through > NAT) computer-to-computer calls, by name, to mobile endpoints, including > fast file transfer that doesn't impair the quality of the voice call, took > less than a week to whip up a working prototype. If there's demand, we'll > probably release that as a sample app with sources, once the MFP and MFPNet > layers and some other supporting libraries are available for download. > > Matthew Kaufman > amicima, Inc. > matthew@amicima.com > matthew@matthew.at > > Ps. One of the reasons I'm putting this preannouncement out on the > p2p-hackers list is that several folks on the list have mailed in the past > asking for more information about what we're doing at amicima and when they > can play with the technology, and I've managed to delete about half my list > of who those people were. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From matthew at matthew.at Tue Jul 19 08:36:53 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DC7CD8.50605@hamachi.cc> Message-ID: <200507190837.j6J8b2U65776@where.matthew.at> > Matthew, > > Thanks for a detailed reply. I guess my main concern with > security portion of your protocol is that it does not follow > certain design properties found in nearly all commonly used > transport protocols like SSL/TLS, SSH2, ESP as well as a > handful of others like IKE, JFK, etc. It is best to compare this against IPSEC, rather than something like SSL, because we are operating without the ordered-and-reliable delivery that something like SSL depends on. (For instance, we can't have an internal sequence number that is not transmitted that must match at the far end, like SSH2 has, because we expect out-of-order delivery and lost packets to occur, and we can't just run all the hypothetical sequence numbers through an HMAC algorithm) > Few I can think of off hand are - > > * the use of MACs. Relying on validity marker in plaintext > payload to detect corrupted packets is expensive > computationally-wise in a worst case scenario, while > HMAC-based authentication costs pennies. Not true. See appendix B of rfc3723 (or other equivalent sources). For a 1024 byte packet, AES-CBC takes 27.97 cycles/byte and HMAC-SHA1 takes 24.71 cycles/byte. And we need good performance in the "not being attacked" case as well, so the real comparison is: HMAC in use: 24.71 cycles/byte when being attacked (discard after HMAC, before decrypt), 24.71 + 27.97 = 52.68 cycles/byte when not being attacked (the "expected" case). HMAC not in use: 27.97 cycles/byte when being attacked, 27.97 cycles/byte (the same) when not being attacked. Summary is that if you run SHA1 HMAC and AES, you save about 10% on cycles when you're being attacked and you pay for that small savings by using almost 90% more cycles when you're not. > It also 'not good' from cryptographical perspective, but I > cannot comment on that (note though how SSH was switched to > using MACs in v2). Besides MACs are also required for implementing - > > * replay protection, which involves tagging packets with > unique sequence numbers after the encryption but before the > authentication. Replay protection can also be placed inside the encryption in a similar fashion... The real question is, does putting it outside and using HMAC save you cycles in the being-attacked case (so that you can discard replays prior to decryption cost)? And the answer is "not enough, especially given the penalty in the non-attack case". Our flows have sequence numbers which prevent replay while a flow is active (and once a session is torn down, replay is ineffective), and we also can use the timestamp field for some additional replay protection. Because the attacker cannot "see inside" other packets in order to guess at good sequence numbers, we are better protected than TCP in this way, though not quite as protected as the anti-replay provisions of IPSEC. We might add more if we were convinced that it was necessary. Modified packets can of course be detected not only by the checksum being invalid, but also because it is unlikely that valid chunks would be found and decoded, even if a valid checksum were accidentally generated by a change to an encrypted packet. > This way replayed or duplicate packets can be discarded > without wasting any time on decryption or authentication at all. Well, you can't use the non-encrypted sequence number that you see until you check the HMAC... And that has non-zero cost, as described above. > * session key being derived in 'symmetrical way', when both > parties contribute to the key in equal degree. This is a valid argument, and we have previously discussed this internally and will again. The single issue that is caused by the asymmetry is that an attacker who has compromised one host's private RSA key can (about half the time, in a P2P environment... more or less in a client-server environment) use that information to extract the session key that is chosen, whereas it would require compromising both ends in order to extract a session key that was chosen with symmetry. The difficulty is putting everything in the small number of startup packets we have, and fitting those into non-fragmented datagrams, but this is probably something that will be addressed because it is a valid security concern in some threat cases. > Basically security component of your protocol looks very > different from existing mature protocols. And I would suspect > that this will not facilitate adoption of MFP .. at least not > in its secured form. It actually looks quite like IPSEC ESP without AH... Consider the session ID as the security association identifier and go from there. In reality, though, in order to determine if a secure network protocol is right for your application, you must consider the threats that you are protecting against, and compare that to the benefits. Is this as secure as some other protocols? Probably not. Does it have numerous advantages for certain applications? Absolutely. Will bugs be found in the first release of the implementation? I sure hope so, so that we can fix them! Matthew From lauri.pesonen at gmail.com Tue Jul 19 09:15:32 2005 From: lauri.pesonen at gmail.com (Lauri Pesonen) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507190837.j6J8b2U65776@where.matthew.at> References: <42DC7CD8.50605@hamachi.cc> <200507190837.j6J8b2U65776@where.matthew.at> Message-ID: On 7/19/05, Matthew Kaufman wrote: > > It actually looks quite like IPSEC ESP without AH... Consider the session ID > as the security association identifier and go from there. I'm sure you are aware that encryption only without authentication is considered a bad idea in crypto circles (even the IPSec RFCs warn against it), and I assume you have decided to go against the grain here for performance reasons. That does not change the fact that it's a bad idea. Recently there was an attack published on IPSec with ESP and without AH: http://www.uniras.gov.uk/niscc/docs/al-20050509-00386.html?lang=en And some discussion on sci.crypt: http://groups-beta.google.com/group/sci.crypt/browse_frm/thread/33dd95c9697f46c4?hl=en& Now the attack is a very special case and from a practical point of view and does not give the attacker a huge advantage. This might be even more true for MFP. But it does show that encryption without authentication is a bad idea and does give the attacker _an_ advantage. And as history shows, any advantage in crypto tends to grow bigger as attacks evolve in time. I think you should look again at your performance requirements and re-evaluate whether adding authentication incurs an unacceptable performance hit. Basically at the moment you are going against time-proven crypto thinking. > Matthew -- ! Lauri From ap at hamachi.cc Tue Jul 19 09:47:23 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507190837.j6J8b2U65776@where.matthew.at> References: <200507190837.j6J8b2U65776@where.matthew.at> Message-ID: <42DCCC2B.30009@hamachi.cc> Matthew Kaufman wrote: [snip] >>Few I can think of off hand are - >> >>* the use of MACs. Relying on validity marker in plaintext >>payload to detect corrupted packets is expensive >>computationally-wise in a worst case scenario, while >>HMAC-based authentication costs pennies. > > Not true. See appendix B of rfc3723 (or other equivalent sources). For a [snip] > Summary is that if you run SHA1 HMAC and AES, you save about 10% on cycles > when you're being attacked and you pay for that small savings by using > almost 90% more cycles when you're not. I have no doubt the authors of rfc3723 computed cycles/byte correctly, however running a quick test with openssl yields 36:5 ratio between AES-128-CBC and HMAC-SHA1. Note that this uses assembly optimized AES implementation and rather frivolious implementation of HMAC. If you consider a hardware implementation, then any modern ESP-capable crypto accelerator will be computing HMAC in parallel with decryption, which would result in zero overhead. > >>It also 'not good' from cryptographical perspective, but I >>cannot comment on that (note though how SSH was switched to >>using MACs in v2). Besides MACs are also required for implementing - >> >>* replay protection, which involves tagging packets with >>unique sequence numbers after the encryption but before the >>authentication. > > Replay protection can also be placed inside the encryption in a similar > fashion... The real question is, does putting it outside and using HMAC save > you cycles in the being-attacked case (so that you can discard replays prior > to decryption cost)? And the answer is "not enough, especially given the > penalty in the non-attack case". You don't need to check MAC to discard duplicate packet. If you are looking at seqno that you already saw, you can drop packet right away. [snip] > >>This way replayed or duplicate packets can be discarded >>without wasting any time on decryption or authentication at all. > > > Well, you can't use the non-encrypted sequence number that you see until you > check the HMAC... And that has non-zero cost, as described above. See above. >>* session key being derived in 'symmetrical way', when both >>parties contribute to the key in equal degree. > > This is a valid argument, and we have previously discussed this internally [snip] > >>Basically security component of your protocol looks very >>different from existing mature protocols. And I would suspect >>that this will not facilitate adoption of MFP .. at least not >>in its secured form. > > It actually looks quite like IPSEC ESP without AH... Consider the session ID > as the security association identifier and go from there. > > In reality, though, in order to determine if a secure network protocol is > right for your application, you must consider the threats that you are > protecting against, and compare that to the benefits. Agreed. > Is this as secure as > some other protocols? Probably not. Does it have numerous advantages for > certain applications? Absolutely. The problem here is in 'probably not'. The mere presence of 'probably' outweighs all benefits. It basically means that security services provided by the protocol may be susceptible to certain attack that may or may not be on the list of threats we are trying to defend against. Alex From matthew at matthew.at Tue Jul 19 16:31:58 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DCCC2B.30009@hamachi.cc> Message-ID: <200507191632.j6JGW7U66642@where.matthew.at> Alex Pankratov: > I have no doubt the authors of rfc3723 computed cycles/byte > correctly, however running a quick test with openssl yields > 36:5 ratio between AES-128-CBC and HMAC-SHA1. Note that this > uses assembly optimized AES implementation and rather > frivolious implementation of HMAC. We will be benchmarking several implementations today. Last night I was simply using the data I had available. Even at 36:5, HMAC isn't "free", and you do pay for the extra overhead *in addition* to the symmetric encryption *at both the sender and the receiver* in the non-attack case. There is nothing about our design which precludes adding an HMAC trailer, even a mandatory one, and we may add this to the specification after we see the results of the benchmarks and look at the possible attacks. > You don't need to check MAC to discard duplicate packet. If > you are looking at seqno that you already saw, you can drop > packet right away. But an attacker isn't going to replay packets at you with existing sequence numbers if he knows you're going to discard ones that you've already seen without further processing... The attacker will instead modify the packets in order to generate the maximum possible load at the receiver (since once you have sequence numbers, the remaining attack is a cryptographic load denial of service attack) > The problem here is in 'probably not'. The mere presence of 'probably' > outweighs all benefits. It basically means that security > services provided by the protocol may be susceptible to > certain attack that may or may not be on the list of threats > we are trying to defend against. This "probably" is also true of SSL, TLS, IPSEC, etc. Each one has already-known attack vectors and tradeoffs in their designs and implementation. And new ones are being found all the time. The IPSEC situation, for instance, is in many ways worse than this. It is also true of the underlying choice of cryptosystem... Using SHA-1 as your HMAC basis opens you to known attacks on SHA-1. Using RSA for your public-key crypto opens you up to brute-force attacks (and maybe more) against the private keys. But the good news is that what we're releasing will be available in source form, for inspection, and we are open to improving the security as known or suspected attacks against it materialize. (And if we stop being open to it, you can change the source yourself) Already, that's better than the cryptography that most existing P2P users are using (none, in the case of most p2p networks, or proprietary and unavailable for inspection, in a few cases, like Skype) Our primary goal in adding encryption and authentication to MFP was not to make the strongest possible transport... It was rather to get relatively far away from the present situation, where everyone doing P2P is running everything in plain text with no attempt at authentication at all. If by making a few changes we can make it all that much stronger, then sure, we'll do that. Matthew From matthew at matthew.at Tue Jul 19 17:21:12 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: Message-ID: <200507191721.j6JHLLU66696@where.matthew.at> Lauri Pesonen: > > I'm sure you are aware that encryption only without > authentication is considered a bad idea in crypto circles It is considered an idea which leaves open additional theoretical avenues of attack which are only partially mitigated by adding the authentication. (AES without HMAC is safe unless someone can corrupt the AES data in such a way as to cause a problem for you. Then you're safer with HMAC, *only* as long as the same attacker doesn't also have a working attack against your HMAC scheme... All you're doing is locking two doors instead of one... You're not preventing a great locksmith from getting through both) > (even the IPSec RFCs warn against it) The case of IPSEC is substantially different... The IPSEC RFCs warn against encryption without authentication because in IPSEC there are fields which *cannot be encrypted*, but which *can* be protected with the HMAC. The attack vectors against IPSEC when used with ESP and without AH largely rely on the possibility opened to manipulate the fields which are not encrypted and are then not protected with the AH. > and I assume you have > decided to go against the grain here for performance reasons. Performance and because the attack possibilities are different than in IPSEC. We are open to analyses that show how big the real vulnerability is in our work. And there's nothing about our protocol design that precludes adding an HMAC trailer if it were to be deemed necessary. > That does not change the fact that it's a bad idea. Recently > there was an attack published on IPSec with ESP and without AH: > > http://www.uniras.gov.uk/niscc/docs/al-20050509-00386.html?lang=en I've seen this... There are three basic attacks described in the paper, none of which appears to directly apply to our work. In #1, you perform a successful bit-flipping attack on the encryption to change the destination address on a tunneled packet, which is then decrypted by the tunnel endpoint and sent to the wrong final destination, which could be where the attacker is monitoring. We are potentially vulnerable to bit-flipping attacks, but you'd need to successfully flip bits such that the checksum stays valid and the chunk formatting stays valid, which is probably harder than modifying a single IP address in the header. Furthermore, because we don't decrypt at an intermediate tunnel, this attack cannot be used to redirect plaintext data anywhere, it could only be used to change what is being received, if it were carried out successfully (requires the ability to do this data modification *and* intercept the valid packets before those arrive). In #2 and #3 you damage the encrypted tunneled packet such that after it is decrypted and passed to the final recipient, the final recipient rejects it with a plaintext ICMP message which contains, as per the ICMP spec, a portion of the plaintext that was received. Both of these rely on the obvious danger of including decrypted plaintext in an error reply, and are much easier to carry out, because you can leave the authenticated part alone, damage the encrypted part, and get the plaintext out. Since we never generate plaintext replies, particularly plaintext copies of encrypted data, we don't have this issue. Also, we don't have a non-encrypted part that can stay undamaged. > Now the attack is a very special case and from a practical > point of view and does not give the attacker a huge > advantage. This might be even more true for MFP. But it does > show that encryption without authentication is a bad idea and > does give the attacker _an_ advantage. But in the IPSEC case that advantage comes primarily from the ability to manipulate the encrypted part separately from the unencrypted (and not authenticated if you're not using AH) part in some of the earlier published attacks, or the ability to leave the tunnel encapsulation intact while modifying the interior encrypted packet (as in the cases discussed above). > And as history shows, > any advantage in crypto tends to grow bigger as attacks > evolve in time. This is true. And over time, the algorithms become less secure against both analysis and brute-force attacks. > I think you should look again at your performance > requirements and re-evaluate whether adding authentication > incurs an unacceptable performance hit. Basically at the > moment you are going against time-proven crypto thinking. We will reevaluate this, but as I pointed out above, all that is going on here is adding a second door. Adding an HMAC simply means that someone who figures out a successful attack vector that depends on modifying the AES-encrypted data such that it is wrong but checksums and passes format checks correctly must now also have an attack on the HMAC that works too. What it doesn't do is change this from "insecure" to "perfectly secure", and since we don't know the odds that someone who can manipulate AES to their advantage also knows how to create HMAC collisions, we don't even know how much more secure it would be. Matthew From matthew at matthew.at Tue Jul 19 21:55:58 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: Message-ID: <200507192156.j6JLu7U66951@where.matthew.at> Lauri Pesonen [mailto:lauri.pesonen@gmail.com] (sent privately but permission to reply to the list granted): > > In the end it comes down to your threat model. If you think > a denial of service attack is probable then you should > implement a MAC. More specifically, if you think a denial of service attack is probable and your MAC is enough cheaper in CPU cycles than your decryption, add a MAC, or... > Getting past a CRC when modifying encrypted data is not improbable. > Getting this far means that you end up using resources in > handling invalid packets. This opens up a venue for a DoS > attack. Using a MAC after encryption prevents the DoS attack > at the cost of requiring more resources in handling valid > packets, as you stated earlier. There's also reasons other than a denial-of-service attack to want a MAC, because it does add one more hoop to jump through for an attacker, which with today's crypto technology probably is a significant improvement in security (though without it, the security is still pretty darn good). It is a bunch more CPU cycles though, so as I've pointed out before, deciding whether or not to do it requires analysis of the performance requirements vs. the threat model. > In any case it is seen as a good practise to implement a MAC > on top of encryption. In your case the CRC and message > structure provides some probabilistic protection against > tampered packets. This is not the case in all applications. No disagreement here. > One remaining question is why wouldn't you add a MAC to the protocol? Because the "protocol" doesn't need to have a MAC added. See below... > It would make a lot of crypto people very happy and the > performance impact is not that huge (i.e. your network link > will most probably get bogged down before your CPU cycles run > out). Obviously adding unnecessary features to your > implementation is silly, but a lot of crypto experts (I'm > definitely not one of them) seem to demand a MAC in all > encrypted communication protocols. As the other programmer here pointed out to me right after I sent my last email, because our entire payload is "encrypted", adding a trailing MAC is trivial (unlike in IPSEC, where the MAC needs to protect a different set of bytes than what the encryption covers)... In our implementation of the MFP protocol, ALL cryptographic operations are implemented by a crypto "plug-in". (This means you can easily substitute hardware-accelerated algorithms, proprietary or classified algorithms, etc.) Application developers who want the extra protection of wrapping the encrypted data with an HMAC may simply define an extended crypto plugin which adds a trailing HMAC after encryption (there are some good papers about why it needs to be done after, not before) on the transmit side and which checks the HMAC prior to continuing with decryption on the receive side. This causes no changes to the protocol itself, it simply changes the definition of the cryptosystem that is plugged in (which, like the initial session key choice, is application-defined). The secret key for the HMAC may either be the same key as the AES key, or may be defined as a partitioning of a larger exchanged key into the AES key part and the HMAC key part, and doesn't change the key exchange part of the protocol itself either, because key encryption and decryption are *also* defined by the implementation in the cryptographic plugin. We may or may not provide this HMAC-extended crypto plugin as the default or as one of the initial options that ships with the first release, but it is absolutely trivial for a developer to add if we don't. (And other implementations of the protocol, while they might not use the same crypto plug-in architecture are also free to add or not add a MAC) The only real open issue that has been raised on this list and which we are investigating is the issue with symmetric key choice vs. having one end choose. Changing this likely requires a small change to the protocol definition. Whether this is in version 1 or version 2 remains to be seen. Matthew Kaufman matthew@matthew.at matthew@amicima.com www.amicima.com From reinout at cs.vu.nl Thu Jul 21 12:41:42 2005 From: reinout at cs.vu.nl (Reinout van Schouwen) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] How to interpret FreePastry/Bunshin error message Message-ID: Hello all, I'm experimenting with Bunshin (a FreePastry-based DHT). I created one "BunshinConnection" object, stored a test key/value pair, created a BunshinSearch object, and then the following messages started scrolling over my console: Could not dispatch message [ [PEM ReplicaMessage from :[SNH: <0xDBE93F..> -> <0xD615C4..>/fspc003.cs.vu.nl/130.37.30.147:5009 [2924789513309533531]] with key <0x4D43A9..> values : {Object=world}] ] because the application address [StandardAddress: class rice.pastry.commonapi.PastryEndpoint-bunshin.BunshinImplbunshin] was unknown. Message is going to be dropped on the floor. My question is how I should interpret this error. What is wrong and how should I fix it? regards, -- Reinout van Schouwen *** student of Artifical Intelligence email: reinout@cs.vu.nl *** mobile phone: +31-6-44360778 www.vanschouwen.info *** help mee met GNOME vertalen: nl.gnome.org From adam at cypherspace.org Thu Jul 21 15:47:36 2005 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:13:00 2006 Subject: crc inside block-mode = insecure (Re: [p2p-hackers] amicima's MFP - preannouncement) In-Reply-To: <200507192156.j6JLu7U66951@where.matthew.at> References: <200507192156.j6JLu7U66951@where.matthew.at> Message-ID: <20050721154736.GA16660@bitchcake.off.net> Actually what you are doing if I understood (CRC? checksum inside block cipher) is a known insecure construct. SSH had a fairly major bug filed on this when the issue was discovered and it was fixed. Minimally you need a cryptographic checksum (SHA1 etc) inside the block cipher. But really I think you would just as well use HMAC-SHA1 as the extra cost over SHA1 is small. The simpler/safer way to add a MAC IMO is to put the MAC outside the encryption. Avoids issues such as Bleichenbacher's attack against some implementations of openSSL working from the interaction of MAC inside encryption and the padding mode. Adam On Tue, Jul 19, 2005 at 02:55:58PM -0700, Matthew Kaufman wrote: > > In any case it is seen as a good practise to implement a MAC > > on top of encryption. In your case the CRC and message > > structure provides some probabilistic protection against > > tampered packets. This is not the case in all applications. From dbarrett at quinthar.com Thu Jul 21 20:49:01 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification Message-ID: <42E00A3D.8030900@quinthar.com> How do you decide how fast to send UDP file-transfer packets? Lots of people on this list make file sharing applications. Many of those applications use UDP. A big drawback of using UDP for file transfers is you can't rely upon the link-speed detection capabilities of TCP. As a result, if you send UDP too slow, you're not maximizing your connection. But if you send it too fast, you'll congest your upstream and/or the remote peer's downstream connection, causing packet loss at the very least, as well as possibly denial of service. How have you resolved this problem? I'm using an approach where the the client (receiving peer) provides continuous feedback to the server (sending peer) as to how fast it should send data. Thus the client can control the rate at which it receives data from the server. But the question is: how does the client know how fast to request data? The overall approach I'm using is to request data at a certain rate, measure how fast it is actually received, calculate what new rate should be tried, repeat. However, I'm finding this surprisingly tricky. Either I ramp up too slow, or I ramp up and down erratically. I've considered some FSM approaches, and some formulaic approaches, and I still haven't come up with something I like. I've read the TCP RFC for guidance, but it leaves the specifics left as an exercise to the reader. Other than that, I'm not sure where else to read for inspiration. So I'm curious, how have you solved this problem? What can you advise? -david From wesley at felter.org Thu Jul 21 21:12:56 2005 From: wesley at felter.org (Wes Felter) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00A3D.8030900@quinthar.com> References: <42E00A3D.8030900@quinthar.com> Message-ID: <42E00FD8.1030103@felter.org> David Barrett wrote: > How do you decide how fast to send UDP file-transfer packets? Have you read about DCCP, TFRC, and TCP-like Congestion Control? Wes Felter - wesley@felter.org From dbarrett at quinthar.com Thu Jul 21 21:20:08 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00FD8.1030103@felter.org> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> Message-ID: <42E01188.2050501@quinthar.com> Wes Felter wrote: > David Barrett wrote: > >> How do you decide how fast to send UDP file-transfer packets? > > Have you read about DCCP, TFRC, and TCP-like Congestion Control? Nope, but I will. Thanks for the references! -david From p2phack at cilux.org Thu Jul 21 22:26:21 2005 From: p2phack at cilux.org (Duncan B. Cragg) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00A3D.8030900@quinthar.com> References: <42E00A3D.8030900@quinthar.com> Message-ID: <42E0210D.5030301@cilux.org> > How do you decide how fast to send UDP file-transfer packets? > So I'm curious, how have you solved this problem? What can you advise? > I would also recommend taking a look at Peter Amstutz's 'VIP' that he uses on his VOS project: http://interreality.org/cgi-bin/moinwiki/moin.cgi/VipDocumentation If that isn't enough, just try mailing Peter directly on tetron@interreality.org - I'm sure he'd be happy to chat. (I don't know him - I just subscribe to the VOS list). Duncan B. Cragg From agthorr at cs.uoregon.edu Thu Jul 21 21:59:20 2005 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00A3D.8030900@quinthar.com> References: <42E00A3D.8030900@quinthar.com> Message-ID: <20050721215920.GB1492@cs.uoregon.edu> On Thu, Jul 21, 2005 at 01:49:01PM -0700, David Barrett wrote: > However, I'm finding this surprisingly tricky. Either I ramp up too > slow, or I ramp up and down erratically. I've considered some FSM > approaches, and some formulaic approaches, and I still haven't come up > with something I like. I've read the TCP RFC for guidance, but it > leaves the specifics left as an exercise to the reader. Other than > that, I'm not sure where else to read for inspiration. TCP ramps up and down erratically. To a certain extent that's OK. Make sure to read *all* the TCP RFCs; there are several of them. I recommend reading in the following order: Van Jacobson, "Congestion Avoid and Control", SIGCOMM 1988. http://www-nrg.ee.lbl.gov/papers/congavoid.pdf RFC 2581 RFC 3782 RFC 3517 There's a standard experiment to test whether your implementation is "TCP-friendly" (i.e., that it won't starve TCP, nor will it be starved by TCP). Setup a network configuration like this: host ------ router ------------ router ----------- host Where the link between the two routers is the bottleneck link, and the host connections are much faster. Then, start up a large TCP flow between the two hosts (transfer a gigabyte file, for instance with HTTP/FTP), and start up a flow using your protocol. Plot the bandwidth each flowing is using as a function of time. They should stabilize so that both flows get around half the bottleneck bandwidth. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From gbildson at limepeer.com Thu Jul 21 22:18:22 2005 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00A3D.8030900@quinthar.com> Message-ID: I used a totally undocumented (as of yet) approach in our UDP based firewall-to-firewall transfer code. One of the main reasons it is undocumented is that a lot of it is essentially a trial and error based hack. However, I think a couple parts of it are interesting in the general sense. There are three basic things going on to determine our data rate: 1) We try to achieve 2% to 3% UDP failure rate. So, if you don't have enough failures then speed up the sending rate in small increments. In reality, we decrease the wait time between data packets and actually start making some multiple of wait times zero before a real wait occurs. The zero wait times is just a way of dealing with the fact that our current data message size is small (500 bytes) and fixed at the moment. 2) We try to coexist nicely with TCP. If your round trip times goes up above some factor of the baseline average, back off a little. This is imperfect to be sure but mostly where the trial and error approach comes in. What you will notice if you run an algorithm that continues to increase in aggression is that the round trip times will start to go up when you are crowding out TCP. To coexist nicely and maximize your own throughput, you should start to back off when your round trip times escalate by some factor (or multiple) over your baseline average round trip time. 3) Apply standard exponential backoff rules. If I did it over again, I would definitely just follow the rule book of TCP congestion control or some other good source. Writing your own congestion control is fun but dangerous. Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of David Barrett > Sent: Thursday, July 21, 2005 4:49 PM > To: Peer-to-peer development. > Subject: [p2p-hackers] UDP file transfer link speed identification > > > How do you decide how fast to send UDP file-transfer packets? > > Lots of people on this list make file sharing applications. Many of > those applications use UDP. A big drawback of using UDP for file > transfers is you can't rely upon the link-speed detection capabilities > of TCP. As a result, if you send UDP too slow, you're not maximizing > your connection. But if you send it too fast, you'll congest your > upstream and/or the remote peer's downstream connection, causing packet > loss at the very least, as well as possibly denial of service. > > How have you resolved this problem? I'm using an approach where the the > client (receiving peer) provides continuous feedback to the server > (sending peer) as to how fast it should send data. Thus the client can > control the rate at which it receives data from the server. > > But the question is: how does the client know how fast to request data? > The overall approach I'm using is to request data at a certain rate, > measure how fast it is actually received, calculate what new rate should > be tried, repeat. > > However, I'm finding this surprisingly tricky. Either I ramp up too > slow, or I ramp up and down erratically. I've considered some FSM > approaches, and some formulaic approaches, and I still haven't come up > with something I like. I've read the TCP RFC for guidance, but it > leaves the specifics left as an exercise to the reader. Other than > that, I'm not sure where else to read for inspiration. > > So I'm curious, how have you solved this problem? What can you advise? > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From dbarrett at quinthar.com Thu Jul 21 23:19:30 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E00FD8.1030103@felter.org> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> Message-ID: <42E02D82.7060007@quinthar.com> Wes Felter wrote: > Have you read about DCCP, TFRC, and TCP-like Congestion Control? These are great references, thank you very much. Not only are they good in isolation, but they helped fill in the cracks in my understanding of TCP. I'll record what I learned here for others who are similarly confused. As I do so, can you clarify I'm understanding it correctly? In essence, TCP maintains a "congestion window" (cwnd) that dictates how many unacknowledged packets can be "in flight". So for a window size of 1, the server sends one packet, and waits for an explicit ACK before sending the next. Assuming a 100ms round-trip latency, this means at most ten packets will be sent a second. As the server increases the in-flight window size (or as the latency decreases), the server sends at a faster pace. The server uses an algorithm named "Additive Increase, Multiplicative Decrease (AIMD)" to regulate cwnd. Basically, each time an ACK is received, cwnd is incremented by some amount. However each time cwnd is to be decreased, it backs off by some set fraction (more on this in a bit). This means it increases cautiously, and backs off aggressively. Because the server increases cwnd in response to each ACK, and because each packet (typically) generates its own ACK, this results in initial exponential increase in send rate -- even though cwnd is only incrementing linearly. It took me a bit to wrap my head around why, but this quote from RFC 2001 cleared me up: > The sender starts by transmitting one segment and waiting for its > ACK. When that ACK is received, the congestion window is incremented > from one to two, and two segments can be sent. When each of those > two segments is acknowledged, the congestion window is increased to > four. This provides an exponential growth, although it is not > exactly exponential because the receiver may delay its ACKs, > typically sending one ACK for every two segments that it receives. Somewhat counterintuitively, this exponential rate increase mode of TCP is called "slow start" -- named so because it starts transmitting very slowly, though increases quite quickly. Regardless, at some point it'll start sending at a faster rate than can be delivered. Routers will discard packets they can't forward, and thus the sender discovers it's sending too fast when it fails to receive an ACK for a given packet. In other words, packet loss is what notifies the sender that it needs to slow down. So once the sender detects a packet has been dropped, it backs off its cwnd by some fraction (TCP suggests that it back off to 50%, but [1] recommends only to 85%) before starting to increase again. Now, after the slow-start approach hit the ceiling and backed off some fraction, it could just restart with exponential growth again. This would again hit the ceiling and back off indefinitely, creating the up-and-down graph shown in [1], reminiscent of Sisyphus and his boulder. [1] http://www-lce.eng.cam.ac.uk/~ctk21/scalable/ However, it would be preferable to "level off" at the top send rate and just stick with that. For this reason after the initial slow-start approach hits the ceiling, it uses a different algorithm called "congestion avoidance". Thus rather than quickly shooting up to the ceiling with exponential growth, it just creeps up using linear growth until it experiences packet loss again. At that point it just keeps transmitting at that rate, as graphed in [2]. [2] http://www.csm.ornl.gov/~dunigan/netperf/kelly.html Now, all this is really simplified, and I recommend the above two sources, as well as RFC 2581 for more details. Furthermore, there are variations on how to tweak these algorithms for better performance ([1] describes one in particular). But this should get you started. Now, with all that out of the way, here's what I don't understand: (a) After "congestion avoidance" has been started (ie, cwnd has exceeded ssthresh), how does it determine when to stop incrementing the cwnd? I see that [2] clearly shows cwnd (and hence, the send rate) level off at some ceiling amount. And RFC 2581 clearly states "Congestion avoidance continues until congestion is detected." (ie, a packet loss occured). But this is easier said than done. Assuming TCP is a very simplified finite state machine (FSM) that starts in "slow start" state and transitions to "congestion avoidance" state when a packet loss is detected, does it likewise transition from "congestion avoidance" to "hold steady" once a second packet loss is detected? If so, under what circumstances does it transition from "hold steady" back to "collision avoidance" or "slow start"? Furthermore, does the cwnd get adjusted at all in "hold steady"? Thank again Wes for the references, and I hope my summary above is reasonably accurate and can help others better understand the topic. -david From agthorr at cs.uoregon.edu Thu Jul 21 23:32:25 2005 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E02D82.7060007@quinthar.com> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> Message-ID: <20050721233155.GH1492@cs.uoregon.edu> On Thu, Jul 21, 2005 at 04:19:30PM -0700, David Barrett wrote: > Assuming TCP is a very simplified finite state machine (FSM) that starts > in "slow start" state and transitions to "congestion avoidance" state > when a packet loss is detected, does it likewise transition from > "congestion avoidance" to "hold steady" once a second packet loss is > detected? If so, under what circumstances does it transition from "hold > steady" back to "collision avoidance" or "slow start"? Furthermore, > does the cwnd get adjusted at all in "hold steady"? There is no "hold steady" mode. If a loss is detected by timeout, cwnd is set to 1, ssthresh is cut in half, and it goes back to Slow Start. If a loss is detected via 3 duplicate ACKs, cwnd and ssthresh are both cut in half, then it goes into Fast Retransmit mode to recover the loss. Once the loss is recovered, it returns to congestion avoidance mode. Basically, in congestion avoidance mode you still oscillate, but you oscillate slower and closer to the goal. Also, this oscillation is from the sender's perspective. Ideally, the receiver sees a steady stream of packets. When the sender is sending too fast, their building up in a router's queue, until the router drops one. The sender cuts it window in half, but there's still a bunch of packets in the queue. The (ideal) net effect is a steady stream of packets arriving at the receiver. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From sg266 at cornell.edu Thu Jul 21 23:41:03 2005 From: sg266 at cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E02D82.7060007@quinthar.com> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> Message-ID: <1121989264.5716.48.camel@localhost.localdomain> On Thu, 2005-07-21 at 16:19 -0700, David Barrett wrote: > Now, after the slow-start approach hit the ceiling and backed off some > fraction, it could just restart with exponential growth again. [...] > Thus rather than quickly shooting up to the > ceiling with exponential growth, it just creeps up using linear growth > until it experiences packet loss again. In addition, if it just uses the exponential ramp-up and ramp-down then multiple TCPs do not converge to their fair-shares. Given the choice of multiplicative (M) and additive (A), increase (I) and decrease (D): I think only MIAD and AIMD converge to the full link bandwidth, and of that only AIMD converges to the fair share as well. [1] [1] http://www.cse.ohio-state.edu/~jain/papers/ftp/cong_av.pdf cheers, -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050721/65a31fbb/attachment.pgp From dbarrett at quinthar.com Thu Jul 21 23:58:52 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: References: Message-ID: <42E036BC.3080709@quinthar.com> Greg Bildson wrote: > I used a totally undocumented (as of yet) approach in our UDP based > firewall-to-firewall transfer code. One of the main reasons it is > undocumented is that a lot of it is essentially a trial and error > based hack. However, I think a couple parts of it are interesting in > the general sense. That's cool; I think Limewire has pretty well demonstrated its ability to transfer files. :) > There are three basic things going on to determine our data rate: > > 1) We try to achieve 2% to 3% UDP failure rate. So, if you don't > have enough failures then speed up the sending rate in small > increments. This is a novel approach. I hadn't considered intentionally targeting a certain packet loss; I was approaching it from a more cautious angle of finding the maximum throughput that creates no packet loss. > In reality, we decrease the wait time between data packets and > actually start making some multiple of wait times zero before a real > wait occurs. The zero wait times is just a way of dealing with the > fact that our current data message size is small (500 bytes) and > fixed at the moment. Could you restate this in another way? I didn't quite get what you were trying to say; I don't see how wait time is related to message size. Incidentally, why are your data packets so small? > 2) We try to coexist nicely with TCP. If your round trip times goes > up above some factor of the baseline average, back off a little. That's interesting that you monitor not only packet loss, but RTT. TCP makes no mention of monitoring latency. However, you're also operating more aggressively than TCP, so perhaps it doesn't need to. > What you will notice if you run an algorithm that continues to > increase in aggression is that the round trip times will start to go > up when you are crowding out TCP. This is an excellent real world lesson; thanks for sharing it. > 3) Apply standard exponential backoff rules. *Definitely.* > If I did it over again, I would definitely just follow the rule book > of TCP congestion control or some other good source. Writing your > own congestion control is fun but dangerous. I agree with you there. But TCP's congestion control algorithms are closely tied with it's use of ACK. Because I'm not ACK'ing data in the same fashion as TCP, I've got little choice but to tweak its congestion control. Regardless, TCP serves as a good model, even if the specifics change. -david From dbarrett at quinthar.com Fri Jul 22 00:00:48 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <1121989264.5716.48.camel@localhost.localdomain> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <1121989264.5716.48.camel@localhost.localdomain> Message-ID: <42E03730.2060605@quinthar.com> Saikat Guha wrote: > On Thu, 2005-07-21 at 16:19 -0700, David Barrett wrote: > >>Now, after the slow-start approach hit the ceiling and backed off some >>fraction, it could just restart with exponential growth again. [...] >>Thus rather than quickly shooting up to the >>ceiling with exponential growth, it just creeps up using linear growth >>until it experiences packet loss again. > > In addition, if it just uses the exponential ramp-up and ramp-down then > multiple TCPs do not converge to their fair-shares. Given the choice of > multiplicative (M) and additive (A), increase (I) and decrease (D): I > think only MIAD and AIMD converge to the full link bandwidth, and of > that only AIMD converges to the fair share as well. [1] > > [1] http://www.cse.ohio-state.edu/~jain/papers/ftp/cong_av.pdf Ahh, good point. Thanks for bringing that up. -david From greg at electricrain.com Fri Jul 22 00:21:29 2005 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E02D82.7060007@quinthar.com> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> Message-ID: <20050722002129.GM17762@zot.electricrain.com> > In other words, packet loss is what notifies the sender that it > needs to slow down. Be cautious of this fundamental assumption that TCP makes. This works well for wired networks where layer 2 is unlikely to drop packets. It falls over in the face of simple 802.11 based wireless layer 2 because a lost wireless packet is often due to signal interference that will not be reduced at all by decreasing the transmit rate. Also, for a modified TCP congestion avoidance algorithm that does a better job I recommend looking up Westwood+ TCP. -greg From agthorr at cs.uoregon.edu Fri Jul 22 05:05:57 2005 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E036BC.3080709@quinthar.com> References: <42E036BC.3080709@quinthar.com> Message-ID: <20050722050556.GA3652@cs.uoregon.edu> On Thu, Jul 21, 2005 at 04:58:52PM -0700, David Barrett wrote: > That's interesting that you monitor not only packet loss, but RTT. TCP > makes no mention of monitoring latency. However, you're also operating > more aggressively than TCP, so perhaps it doesn't need to. Measuring latency is critical to TCP's operation. It computes the weighted moving average of the RTT as well as the variance of the RTT estimates to compute the timeout interval. Otherwise, it wouldn't know how long to wait before declaring a packet lost. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From lemonobrien at yahoo.com Fri Jul 22 17:20:41 2005 From: lemonobrien at yahoo.com (Lemon Obrien) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <20050722050556.GA3652@cs.uoregon.edu> Message-ID: <20050722172041.47688.qmail@web53607.mail.yahoo.com> where can you get code to simulate tcp using udp? joel Daniel Stutzbach wrote: On Thu, Jul 21, 2005 at 04:58:52PM -0700, David Barrett wrote: > That's interesting that you monitor not only packet loss, but RTT. TCP > makes no mention of monitoring latency. However, you're also operating > more aggressively than TCP, so perhaps it doesn't need to. Measuring latency is critical to TCP's operation. It computes the weighted moving average of the RTT as well as the variance of the RTT estimates to compute the timeout interval. Otherwise, it wouldn't know how long to wait before declaring a packet lost. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences You don't get no juice unless you squeeze Lemon Obrien, the Third. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050722/7c78f6eb/attachment.html From wesley at felter.org Fri Jul 22 17:55:31 2005 From: wesley at felter.org (Wes Felter) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <20050722172041.47688.qmail@web53607.mail.yahoo.com> References: <20050722172041.47688.qmail@web53607.mail.yahoo.com> Message-ID: <42E13313.6010407@felter.org> Lemon Obrien wrote: > where can you get code to simulate tcp using udp? This was linked from one of the pages mentioned earlier in this thread: http://www.csm.ornl.gov/~dunigan/netperf/atou.html http://www.csm.ornl.gov/~dunigan/netperf/download/atou.tar.gz Wes Felter - wesley@felter.org From adam at cypherspace.org Fri Jul 22 19:23:49 2005 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:13:01 2006 Subject: use tcp over udp? (Re: [p2p-hackers] UDP file transfer link speed identification) In-Reply-To: <20050722172041.47688.qmail@web53607.mail.yahoo.com> References: <20050722050556.GA3652@cs.uoregon.edu> <20050722172041.47688.qmail@web53607.mail.yahoo.com> Message-ID: <20050722192349.GA10503@bitchcake.off.net> Why don't you just use the tcp stack's tcp implementation and tunnel the IP packets that TCP sends over upd? (I am not sure why ppl persist in reimplementing TCP... get it too aggressive or off by one in some area and your p2p app if widely deployed will make lots of people block your ports if you win heavily over tcp in bandwidth allocation). eg on linux I believe you can use tun or tap do this even in user space. Not sure how you'd do it on windows, but there ought to be some way to do it, given non-microsoft firewalls, vpns, ipsec etc. This is what zks did in their anonymous-IP overlay system. Also it is what cebolla does: http://www.cypherspace.org/cebolla/ There is source available on that link. Adam On Fri, Jul 22, 2005 at 10:20:41AM -0700, Lemon Obrien wrote: > where can you get code to simulate tcp using udp? > > joel > > Daniel Stutzbach wrote: > On Thu, Jul 21, 2005 at 04:58:52PM -0700, David Barrett wrote: > > That's interesting that you monitor not only packet loss, but RTT. TCP > > makes no mention of monitoring latency. However, you're also operating > > more aggressively than TCP, so perhaps it doesn't need to. > > Measuring latency is critical to TCP's operation. It computes the > weighted moving average of the RTT as well as the variance of the RTT > estimates to compute the timeout interval. Otherwise, it wouldn't > know how long to wait before declaring a packet lost. From dbarrett at quinthar.com Fri Jul 22 20:51:24 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <20050721233155.GH1492@cs.uoregon.edu> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> Message-ID: <42E15C4C.7040907@quinthar.com> Daniel Stutzbach wrote: > On Thu, Jul 21, 2005 at 04:19:30PM -0700, David Barrett wrote: > >>Assuming TCP is a very simplified finite state machine (FSM) that starts >>in "slow start" state and transitions to "congestion avoidance" state >>when a packet loss is detected, does it likewise transition from >>"congestion avoidance" to "hold steady" once a second packet loss is >>detected? If so, under what circumstances does it transition from "hold >>steady" back to "collision avoidance" or "slow start"? Furthermore, >>does the cwnd get adjusted at all in "hold steady"? > > There is no "hold steady" mode. > > If a loss is detected by timeout, cwnd is set to 1, ssthresh is cut in > half, and it goes back to Slow Start. > > If a loss is detected via 3 duplicate ACKs, cwnd and ssthresh are both > cut in half, then it goes into Fast Retransmit mode to recover the > loss. Once the loss is recovered, it returns to congestion avoidance > mode. > > Basically, in congestion avoidance mode you still oscillate, but you > oscillate slower and closer to the goal. > > Also, this oscillation is from the sender's perspective. Ideally, > the receiver sees a steady stream of packets. When the sender is > sending too fast, their building up in a router's queue, until the > router drops one. The sender cuts it window in half, but there's > still a bunch of packets in the queue. The (ideal) net effect is a > steady stream of packets arriving at the receiver. Ah, thanks for that overview. That makes perfect sense. -david From dbarrett at quinthar.com Sat Jul 23 04:12:23 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review Message-ID: <42E1C3A7.3060402@quinthar.com> In light of of my previous question on link speed identification of a UDP file-transfer protocol, I'd like to share my overall protocol and ask for feedback on how to improve it. Here's a (slightly long) overview of the protocol, followed by specific questions at the end: --- Rationale --- I'm using SIP and RTP as the basis for a P2P system that includes (among other things) VoIP and a file transfer feature. To simplify my networking layer, I'm using custom RTP profiles for all non-SIP (ie, non-signaling) traffic. Thus I'm creating a simple file transfer profile for RTP. --- Terminology --- - server = sending peer - client = receiving peer --- Priorities --- The overall priorities for my file transfer protocol are as follows: - Fast link speed discovery (to maximize file transfer speed) - Client-side bandwidth management (to free bandwidth for VoIP) - High efficiency (minimal bandwidth lost to overhead) - Minimal fragmentation (all packets fit within UDP MTU) - Play nice with TCP and other applications --- Protocol Overview --- The overall protocol works as follows: 1) Client sends GETINFO to the server 2) Server responds with INFO, which includes the length of the file, as well as a strong hash 3) Client send GETBLOCKS to the server, containing a bit vector describing which blocks the client hasn't yet received, as well as a rate (in blocks per second) at which the server should begin sending them 4) Server sends a series of BLOCK packets to the client, each of which contains a block identifier, sequence number, and 1024 bytes of the file (except for the final block, which might contain less) 5) Throughout the transfer, the client periodically adjusts the download rate by sending CHANGERATE requests to the server. 6) Throughout the transfer, the client occasionally sends an updated bit vector to the server in a new GETBLOCKS request. 7) At the end of the transfer, the client sends DONE and verifies it received the file correctly by testing the strong hash. --- Lessons from TCP --- I'd like to model the protocol on TCP as much as possible. But a direct application can't be done for at least two reasons: - TCP puts the server in charge of choosing the send rate (advised by the client), while I need to have the client decide (ie, the client needs the ability to instantly reduce file transfer bandwidth to free up room for an incoming VoIP stream about which the server has no knowledge). - TCP uses immediate, per-packet ACKs, which are overkill for an any-order bulk transfer. (Upstream bandwidth is an absolute premium so I need to conserve wherever possible.) Regardless, I like the TCP concepts of "slow start" and "congestion avoidance", and I'd like to apply them here. --- Adjusting the Send Rate --- Whenever the client determines data is being sent too fast or too slow, the client calculates a new send rate and requests it using CHANGERATE. If increasing, it can either use the "slow start" or "congestion avoidance" formula. If decreasing, it uses the "decrease" formula. These are defined as follows: - floor = Rate after the last decrease - rate = Current send rate - span = rate - floor - newRate_slowStart = rate + span - newRate_congestionAvoid = rate + CONSTANT - newRate_decrease = floor + span*FRAC The "floor" is a deviation from TCP (which always uses a zero floor), and is actually the top value of a stack that is adjusted as follows: - When decreasing after period of increase, push "newRate" on top - When decreasing twice in a row, pop the stack - If the stack is empty, reset to initial values The net result should ideally produce a graph like this: |- - - -|- -/|-/|/|/|/- - link speed | | / |/ | || / r| ||/ a| | t| | e| | | / |__-^ +-------------------------- time 0 a b c d e ... (As for the discontinuities at a-e, I'll use the notation rate(a) to mean the rate just before, and rate(a') to mean the adjusted rate immediately after.) 0-a: Slow start, exponential increase from scratch a-b: Congestion avoid, CONSTANT slope starting from FRAC*rate(a) b-c: Congestion avoid from rate(a')+FRAC*(rate(b)-rate(a')) c-d: Congestion avoid from rate(b')+FRAC*(rate(c)-rate(b')) d-e: Congestion avoid from rate(c')+FRAC*(rate(d)-rate(c')) In other words, it starts with exponential increase to quickly get in the right ballpark, and then switches to linear increase while it zeros in on the link speed. Eventually it'll oscillate with low magnitude right around the actual rate. --- Rate Change Trigger --- One big question is "how does the client decide when to change rates?" I don't have a good answer here, and I'd love your advice. It seems that there are a few options to decide when to increase: a) When it receives COUNT packets without packet loss b) When it receives some duration of packets without packet loss c) When it downloads at the full requested rate for some period Likewise, there is a similar list of options about when to decrease: a) When it detects a single packet loss b) When it is unable to download at the requested rate for some period I really don't know which to use. I've experimented with "increase:c" and "decrease:b", but I find the download rate is so erratic that it requires a long sample period to average out, which in turn makes the whole system respond very slowly. I'm considering using "increase:a" and "decrease:a" as it seems to work well for TCP, but I'm concerned about the impact of wireless networks (which I'm told have high packet loss). --- Questions for the Reader --- So, with all that played out, here are my specific questions for you: 1) What's a good value for CONSTANT? 2) What's a good value for FRAC? 3) What's a good value for COUNT? 4) Which option should I use to trigger increases? 5) Which option should I use to trigger decreases? 6) What are the ramifications using a non-zero "floor" (ie, being more aggressive than TCP)? 7) What would the ramifications be for using only "slow start", and skipping "collision avoidance" altogether? Likewise, I'm sure there are plenty of areas that deserve additional scrutiny, so feel encouraged to point them out. Thanks, I look forward to hearing your thoughts! -david From ap at hamachi.cc Sat Jul 23 05:36:40 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E1C3A7.3060402@quinthar.com> References: <42E1C3A7.3060402@quinthar.com> Message-ID: <42E1D768.6040307@hamachi.cc> David Barrett wrote: > In light of of my previous question on link speed identification of a > UDP file-transfer protocol, I'd like to share my overall protocol and > ask for feedback on how to improve it. Thanks, David. This is an interesting read. > 5) Which option should I use to trigger decreases? As Gregory P. Smith pointed out in a previous thread, using single packet loss as an congestion indicator does not work well in lossy networks. Like the wireless ones ... or certain Shaw cable segments in Vancouver area :) If your protocol allows for RTT estimation, you may try detecting a congestion by looking at packet loss _and_ RTT value. If the congestion is caused by the software which employs some sort of a packet queueing, you will see a noticable increase in RTT at a rate that immediately preceeds its maximum lossless value. IIRC in a simple scenario with two directly-connected Linux boxes RTT rapidly goes up 3-4 times before receiving box starts dropping packets. Alex PS. My experience is purely empirical. If anyone is aware of any formal research on RTT/congestion subject, I'd really like to hear about it. Thanks. From sg266 at cornell.edu Sat Jul 23 08:03:09 2005 From: sg266 at cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E1C3A7.3060402@quinthar.com> References: <42E1C3A7.3060402@quinthar.com> Message-ID: <1122105789.7354.78.camel@localhost.localdomain> On Fri, 2005-07-22 at 21:12 -0700, David Barrett wrote: > Likewise, there is a similar list of options about when to decrease: > a) When it detects a single packet loss > b) When it is unable to download at the requested rate for some period > ... > but I'm concerned about the impact of wireless networks In a wireless scenario, [1] shows that a single packet loss (a) or even a threshold percent of packet loss within some period (b) are not good indicators of congestion/contention -- i.e. cannot differentiate among them. TCP uses a) and performs poorly in the wireless scenario. Instead, [1] proposes looking at the number of _consecutive_ losses since such losses are temporally correlated. FastTCP [2] more or less ignores isolated losses and moreover uses delay based congestion signals. From your list of options, increase:a and increase:a-applied to decrease may be interesting (similar to [1]'s proposal). Though you may want to set the COUNT for the decrease part to something low (2-3 perhaps) to avoid crowding out regular TCP. Also, if you haven't already, I'd suggest looking at FastTCP. [1] http://lecs.cs.ucla.edu/~cerpa/papers/time-estimation-tech- report.pdf ... there is a MobiHoc '05 version of this paper somewhere [2] http://www.cs.caltech.edu/~weixl/research/control/ton2004.pdf -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050723/6975587d/attachment.pgp From dbarrett at quinthar.com Sat Jul 23 09:02:26 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <1122105789.7354.78.camel@localhost.localdomain> References: <42E1C3A7.3060402@quinthar.com> <1122105789.7354.78.camel@localhost.localdomain> Message-ID: <42E207A2.7020303@quinthar.com> Saikat Guha wrote: > Instead, > [1] proposes looking at the number of _consecutive_ losses since such > losses are temporally correlated. Ahh, consecutive losses is a great idea. 2-3 consecutive losses sounds like a good threshold for decreasing. How many consecutive successes would you recommend for increasing? > FastTCP [2] more or less ignores > isolated losses and moreover uses delay based congestion signals. In my glance through FastTCP, I found it rather dense, and it wasn't immediately apparent to me how it worked. Could you give the 30-second cheat-sheet to how to use and derive delay-based congestion signals? Thanks for the references and help! -david From dbarrett at quinthar.com Sat Jul 23 09:08:12 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E1D768.6040307@hamachi.cc> References: <42E1C3A7.3060402@quinthar.com> <42E1D768.6040307@hamachi.cc> Message-ID: <42E208FC.9010601@quinthar.com> Alex Pankratov wrote: > > If your protocol allows for RTT estimation, you may try detecting > a congestion by looking at packet loss _and_ RTT value. If the > congestion is caused by the software which employs some sort of a > packet queueing, you will see a noticable increase in RTT at a rate > that immediately preceeds its maximum lossless value. A downside of not using per-packet ACKs (or indeed, any ACKs) is that there's little opportunity to continously evaluate the RTT. However, you provide good incentive to change the protocol in order to support this. > IIRC in a simple scenario with two directly-connected Linux boxes > RTT rapidly goes up 3-4 times before receiving box starts dropping > packets. This is excellent real-world feedback. Thank you. I'll keep an eye out for this. -david From sg266 at cornell.edu Sat Jul 23 09:37:55 2005 From: sg266 at cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E207A2.7020303@quinthar.com> References: <42E1C3A7.3060402@quinthar.com> <1122105789.7354.78.camel@localhost.localdomain> <42E207A2.7020303@quinthar.com> Message-ID: <1122111476.7354.96.camel@localhost.localdomain> On Sat, 2005-07-23 at 02:02 -0700, David Barrett wrote: > How many consecutive successes > would you recommend for increasing? I may not be required -- say in one RTT you send X packets. The recipient acknowledges the ones it received (though some sort of bitmap for instance). If you find, say, 3 consecutive holes in that bitmap ... then you conclude a loss occurred and ramp-down, otherwise you ramp-up. -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050723/ca9d9ba4/attachment.pgp From sg266 at cornell.edu Sat Jul 23 09:57:29 2005 From: sg266 at cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E1C3A7.3060402@quinthar.com> References: <42E1C3A7.3060402@quinthar.com> Message-ID: <1122112649.7354.115.camel@localhost.localdomain> On Fri, 2005-07-22 at 21:12 -0700, David Barrett wrote: > The net result should ideally produce a graph like this: > > |- - - -|- -/|-/|/|/|/- - link speed > | | / |/ > | || / > r| ||/ > a| | > t| | > e| | > | / > |__-^ > +-------------------------- time > 0 a b c d e ... > > (As for the discontinuities at a-e, I'll use the notation rate(a) to > mean the rate just before, and rate(a') to mean the adjusted rate > immediately after.) > > 0-a: Slow start, exponential increase from scratch > a-b: Congestion avoid, CONSTANT slope starting from FRAC*rate(a) > b-c: Congestion avoid from rate(a')+FRAC*(rate(b)-rate(a')) > c-d: Congestion avoid from rate(b')+FRAC*(rate(c)-rate(b')) > d-e: Congestion avoid from rate(c')+FRAC*(rate(d)-rate(c')) Since the point to which you cut back always increases, this approach will not work with competing traffic or when the bottleneck link bandwidth decreases, and you'll keep pegging the network. Consider the line formed by joining the local minimas |/. The line is some sort of logarithmic curve (shown with +). |- - - -|- -/|-/|/+++++- - link speed | | / |/+++ | || / ++ r| ||/+ a| |+ t| | e| | | / |__-^ +-------------------------- time 0 a b c d e ... 1) Your approach will never send fewer packets than this line 2) The line will be monotonically increasing 3) It will increase up to the link speed over time When the "link speed" drops -- i.e. your fair share decreases (another competing TCP starts up) or the bottleneck link changes (routing in the internet changes), your congestion window will be stuck far above the real link speed. To fix this, they absolutely must be some set of transitions that bring the window down; in some cases, all the way down to 1. Perhaps when you declare a loss for a couple of RTTs (i.e. 2-3 consecutive packets are lost in a couple of successive RTT-size windows) then scale back to 1 and start from scratch? -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050723/84b791d2/attachment.pgp From matthew at matthew.at Sat Jul 23 16:15:28 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E1D768.6040307@hamachi.cc> Message-ID: <200507231615.j6NGFGU76183@where.matthew.at> Alex Pankratov: > PS. My experience is purely empirical. If anyone is aware of > any formal research on RTT/congestion subject, I'd really > like to hear about it. Thanks. Start with the papers about TCP Vegas. Matthew From matthew at matthew.at Sat Jul 23 16:20:25 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E207A2.7020303@quinthar.com> Message-ID: <200507231620.j6NGKCU76195@where.matthew.at> David Barrett: > Ahh, consecutive losses is a great idea. 2-3 consecutive > losses sounds like a good threshold for decreasing. How many > consecutive successes would you recommend for increasing? The only hard and fast rule is that you *must* perform fairly against TCP. (Unless your stated goal is to use the "leftover bandwidth" after TCP is running or to "crowd out" TCP deliberately. Since most TCP implementations will drop the send rate in half on a single loss, if you wait for more losses, you most likely need to drop more than half. The only way to know for sure if you're doing the right thing is to compare in both simulations and the real world, and see what happens when you compete with a big TCP flow. There's lots more details around doing better than TCP overall... True selective ack, sharing the congestion window on multiple flows between the same hosts, etc. Matthew From osokin at osokin.com Sat Jul 23 18:43:49 2005 From: osokin at osokin.com (Serguei Osokine) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <200507231620.j6NGKCU76195@where.matthew.at> Message-ID: On Saturday, July 23, 2005 Matthew Kaufman wrote: > The only hard and fast rule is that you *must* perform fairly against > TCP. (Unless your stated goal is to use the "leftover bandwidth" after > TCP is running... - which, as a matter of fact, is a perfectly reasonable goal to have in the P2P system design. It is a very common user complaint among BitTorrent and other P2P systems' users that their online experience suffers greatly when their P2P application is running. E-mail is slow, web surfing slows down to a crawl, and so on. The reason for that is fairly obvilus: a P2P application tends to have multiple open connections at all times, and its target is to keep them all constantly saturated, maximizing its network throughput. Swarm downloading, multiple downloads, multiple uploads - you name it. More often than not all these streams are TCP, so if you wish to get your mail or do some Web browsing, your new TCP connection becomes just another one among 5, 10, or 20 already saturated P2P ones. Hence its data transfer rate will be 6, 11, or 21 times slower than what you used to have without P2P application running, and your user experience will suffer accordingly. One way around this problem would be to have a UDP-based P2P data transfer protocol that would be less aggressive than TCP by design - in that case, its swarm links would still fully occupy the available bandwidth when the user does not use the machine for ineractive tasks, but would shrink back in the presence of the TCP streams, allowing them to take the full bandwidth to themselves self and allowing the user to have the same online experience with P2P as there was without it. Until that goal is achieved, I don't see P2P applications running on every user machine as a background task 24x7, which is where they belong to. Today they are too intrusive for that. I don't remember the LimeWire/BearShare UDP protocol details right away, but my recollection is that it was fairly naive - hopefully, naive enough to send much less data than it possibly could if it would have the "real" TCP-like AIMD mechanisms. This waste of bandwidth should not really hurt anyone when you're running ten swarming streams - their total throughput should be fairly close to the link capacity anyway. But at the same time, this naive approach might make these streams shrink in the presnece of the simultaneous TCP streams on the same link, thus achieving the "be invisible to the user" goal. It would be interesting to know whether LimeWire and BearShare did any experiments in that field - ideally, the long FTP transfer should be unaffected by the parallel P2P transfers (which should not make any progress in the meantime, of course). Greg, did you try measuring something like that? Best wishes - S.Osokine. 23 Jul 2005. -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On Behalf Of Matthew Kaufman Sent: Saturday, July 23, 2005 9:20 AM To: 'Peer-to-peer development.' Subject: RE: [p2p-hackers] UDP file transfer protocol peer review David Barrett: > Ahh, consecutive losses is a great idea. 2-3 consecutive > losses sounds like a good threshold for decreasing. How many > consecutive successes would you recommend for increasing? The only hard and fast rule is that you *must* perform fairly against TCP. (Unless your stated goal is to use the "leftover bandwidth" after TCP is running or to "crowd out" TCP deliberately. Since most TCP implementations will drop the send rate in half on a single loss, if you wait for more losses, you most likely need to drop more than half. The only way to know for sure if you're doing the right thing is to compare in both simulations and the real world, and see what happens when you compete with a big TCP flow. There's lots more details around doing better than TCP overall... True selective ack, sharing the congestion window on multiple flows between the same hosts, etc. Matthew _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From dbarrett at quinthar.com Sat Jul 23 18:54:02 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <1122112649.7354.115.camel@localhost.localdomain> References: <42E1C3A7.3060402@quinthar.com> <1122112649.7354.115.camel@localhost.localdomain> Message-ID: <42E2924A.5050600@quinthar.com> Saikat Guha wrote: > > Since the point to which you cut back always increases, this approach > will not work with competing traffic or when the bottleneck link > bandwidth decreases, and you'll keep pegging the network. > > Consider the line formed by joining the local minimas |/. The line is > some sort of logarithmic curve (shown with +). > > > |- - - -|- -/|-/|/+++++- - link speed > | | / |/+++ > | || / ++ > r| ||/+ > a| |+ > t| | > e| | > | / > |__-^ > +-------------------------- time > 0 a b c d e ... Don't you love ASCII art? :) > 1) Your approach will never send fewer packets than this line > 2) The line will be monotonically increasing > 3) It will increase up to the link speed over time Though I'll admit it wasn't clear from my original post, my intent to resolve this is encoded in this paragraph: > The "floor" is a deviation from TCP (which always uses a zero > floor), and is actually the top value of a stack that is adjusted > as follows: > - When decreasing after period of increase, push "newRate" on top > - When decreasing twice in a row, pop the stack > - If the stack is empty, reset to initial values Basically, the first time you decrease, you drop to the "floor" of your latest attempt to increase. If the floor itself is too fast, then you drop back to the floor before that, bottoming out at the initial state. Thus if the link speed does in fact remain constant and I'm not competing with anyone else, by design it should monotonoically increase with the link speed as a sort of asymptote. (Though I'm reconsidering this in light of our discussion about competing more aggressively than TCP.) But, if the link speed changes or competition increases, then it'll back off first a little, and then a lot, and eventually restart over from scratch. -david From dbarrett at quinthar.com Sat Jul 23 18:56:02 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <1122111476.7354.96.camel@localhost.localdomain> References: <42E1C3A7.3060402@quinthar.com> <1122105789.7354.78.camel@localhost.localdomain> <42E207A2.7020303@quinthar.com> <1122111476.7354.96.camel@localhost.localdomain> Message-ID: <42E292C2.5000709@quinthar.com> Ah, gotcha. Thanks. Saikat Guha wrote: > On Sat, 2005-07-23 at 02:02 -0700, David Barrett wrote: > >>How many consecutive successes >>would you recommend for increasing? > > > I may not be required -- say in one RTT you send X packets. The > recipient acknowledges the ones it received (though some sort of bitmap > for instance). If you find, say, 3 consecutive holes in that bitmap ... > then you conclude a loss occurred and ramp-down, otherwise you ramp-up. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From matthew at matthew.at Sat Jul 23 20:35:19 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E2924A.5050600@quinthar.com> Message-ID: <200507232035.j6NKZ6U76408@where.matthew.at> David Barrett: > Thus if the link speed does in fact remain constant and I'm > not competing with anyone else, by design it should > monotonoically increase with the link speed as a sort of > asymptote. (Though I'm reconsidering this in light of our > discussion about competing more aggressively than TCP.) > > But, if the link speed changes or competition increases, then > it'll back off first a little, and then a lot, and eventually > restart over from scratch. Your "thought experiment" here is flawed. The first case ("link speed constant, not competing with anyone else") *never* occurs in the real world. Your packets are *always* competing with other flows, in the output queue from your machine, LAN card drivers or hardware, maybe in a LAN switch, each router and switch encountered on the way, the drivers and stack on the receiving end. Always. And because the speed of light isn't infinite, there is always a non-zero RTT. And because of that, even with ack-every-packet, your information about the current link state is always out of date. All you can do is play fair with TCP when you and TCP both encounter approximately the same loss and RTT. If you try to ramp speed more smoothly with averaging math, or you get your feedback less often and so can't react as quickly, you have to do it more aggressively to make up for the times when you're sending faster than TCP is. But likewise, when another source of congestion goes away, you want to ramp up as fast as (but no faster than) TCP would, on average. Since congestion is always changing, you need to be prepared to reduce or increase your rate all the time. Then, once you've got that all right, go look at the problems with TCP itself on large delay*bandwidth connections with low but nonzero loss during startup, and solve that problem in a way that is compatible with the extentions to TCP which try to solve the same problem. Matthew From dbarrett at quinthar.com Sun Jul 24 05:58:57 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <20050721233155.GH1492@cs.uoregon.edu> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> Message-ID: <42E32E21.6060602@quinthar.com> Daniel Stutzbach wrote: > If a loss is detected via 3 duplicate ACKs, cwnd and ssthresh are both > cut in half, then it goes into Fast Retransmit mode to recover the > loss. Once the loss is recovered, it returns to congestion avoidance > mode. Can you clarify for me what precisely is a "duplicate ACK"? Specifically: - Does it literally mean two ACK packets that are byte-identical? Or just two ACKs that overlap? - Does the client ever send a duplicate ACK intentionally? Or is this just something that happens occasionally on bad connections? Basically, I just don't "get" what duplicate ACKs are, why I should switch fast transmit in response to three of them in a row (does this mean six identical ACK packets?), and so forth. Can you shed any light on this for me? Thanks! -david From dbarrett at quinthar.com Sun Jul 24 06:09:36 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: use tcp over udp? (Re: [p2p-hackers] UDP file transfer link speed identification) In-Reply-To: <20050722192349.GA10503@bitchcake.off.net> References: <20050722050556.GA3652@cs.uoregon.edu> <20050722172041.47688.qmail@web53607.mail.yahoo.com> <20050722192349.GA10503@bitchcake.off.net> Message-ID: <42E330A0.7090500@quinthar.com> Adam Back wrote: > Why don't you just use the tcp stack's tcp implementation and tunnel > the IP packets that TCP sends over upd? > ... > This is what zks did in their anonymous-IP overlay system. Also it is > what cebolla does: That's a damn good idea, actually. Can you give the 30-second architectural overview of how Cebolla exposes a virtual Linux TCP interface? -david From mgp at ucla.edu Sun Jul 24 06:27:02 2005 From: mgp at ucla.edu (Michael Parker) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E32E21.6060602@quinthar.com> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> <42E32E21.6060602@quinthar.com> Message-ID: <42E334B6.7070409@ucla.edu> David, Because TCP data can take different paths between a single source and destination, TCP packets may arrive out-of-order at the destination. Now TCP usually maintains a timer for each packet sent (usually set to the RTO); if it has not received an ACK for that packet's sequence number by the time this timer expires, it retransmits the packet. But in reality, this is naive, and causes too long a delay for lost packets. Say the sender sends TCP packets X-1, X, X+1, X+2, X+3, ... X+n. The receiver gets packet X-1, and acknowledges X-1. The receiver next receives X+1, instead of X as expected, so it buffers packet X+1 but actually acknowledges X-1 -- the last _cumulative, in-order_ packet it has received. This causes a duplicate X-1 at the sender, who can come to two conclusions: packet X experienced extra delay and so it is simply going to arrive out of order, or it was dropped in the network and requires retransmission. If it were pessimistic, it would assume the latter and retransmit immediately. By adopting a "3 duplicate ACK" policy, it instead takes a wait-and-see attitude. If the receiver gets X+2 next, it again buffers this packet and acknowledges X-1. This causes another duplicate ack at the sender, which again decides whether to retransmit packet X or not. If the sender declines to retransmit, and the receiver then received X, it now acknowledges X+2, since all packets up to and including X+2 have been received. When the sender receives this acknowledgment, it knows X has been received, and so it can cancel the timer for packet X. So, as you can see, the 3 duplicate ACK rule (empirically derived?) is just the threshold between the sender deciding that the packet is simply arriving out-of-order, and deciding it was lost and thus requiring retransmission. The gritty details of this might be wrong, but I'm pretty sure that's the gist of it. - Mike David Barrett wrote: > Daniel Stutzbach wrote: > >> If a loss is detected via 3 duplicate ACKs, cwnd and ssthresh are both >> cut in half, then it goes into Fast Retransmit mode to recover the >> loss. Once the loss is recovered, it returns to congestion avoidance >> mode. > > > Can you clarify for me what precisely is a "duplicate ACK"? > Specifically: > > - Does it literally mean two ACK packets that are byte-identical? Or > just two ACKs that overlap? > > - Does the client ever send a duplicate ACK intentionally? Or is this > just something that happens occasionally on bad connections? > > Basically, I just don't "get" what duplicate ACKs are, why I should > switch fast transmit in response to three of them in a row (does this > mean six identical ACK packets?), and so forth. Can you shed any > light on this for me? > > Thanks! > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From sg266 at cornell.edu Sun Jul 24 06:31:22 2005 From: sg266 at cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E32E21.6060602@quinthar.com> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> <42E32E21.6060602@quinthar.com> Message-ID: <1122186682.6677.17.camel@localhost.localdomain> On Sat, 2005-07-23 at 22:58 -0700, David Barrett wrote: > Can you clarify for me what precisely is a "duplicate ACK"? Specifically: > > - Does it literally mean two ACK packets that are byte-identical? Or > just two ACKs that overlap? Two inbound packets with the same # in the ACK field. > - Does the client ever send a duplicate ACK intentionally? Or is this > just something that happens occasionally on bad connections? The TCP receiver always ACK's (read: asks for) the first byte in the stream that it has not yet received. Say it is waiting for byte X and receives a packet starting with byte X+1400 ... then it will ACK/ask for a packet with byte X. Say the sender sends three packets starting with bytes X, X+1400, X+2800, X+3500 ... and the first and third packets are lost; the other two packets will be responded to with an ACK for X each -- thus making duplicate ACKs. The sender only learns that the first packet was lost; and that some two packets did reach the destination (since 2 ACKs were generated). The sender doesn't learn which packet other than the first one was lost. For plain old TCP, this is fine because the receiver doesn't queue up the packets it received since they are out of sequence .. and waits for the sender to retransmit them (even the ones that it did receive). SACK (selective ACK) sort of rectifies this by allowing the receiver to tell the sender which ones were dropped, in a limited way -- kinda like a bitmap but not quite. Now the way regular TCP handles infers isolated losses and congestion is as follows: If the sender doesn't receive an ACK for the byte it has not yet sent; it concludes that a loss occurred and cuts the windows to 1. If it receives dup ACKs, then it infers that some packets are going through fine and therefore there isn't a terrible lot of congestion. So it artificially inflates its window and keeps sending packets (at half the original rate instead of 1). See TCP Fast Retransmit and Fast Recovery [1]. [1] http://www.faqs.org/rfcs/rfc2001.html -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050723/3f7210b0/attachment.pgp From dbarrett at quinthar.com Sun Jul 24 07:36:39 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <1122186682.6677.17.camel@localhost.localdomain> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> <42E32E21.6060602@quinthar.com> <1122186682.6677.17.camel@localhost.localdomain> Message-ID: <42E34507.9040201@quinthar.com> Ahh, thank you Saikat and Michael. I didn't put together the pieces in a way that made sense; now I see: 1) Because the receiver generates an ACK for every received packet, and 2) Because the ACK always requests the first missing byte, therefore 3) Out of order or dropped packets generate duplicate ACKs. Likewise, I now understand why a timeout is more severe than a duplicate ACK, as a timeout indicates that *nothing* is getting through (because the receiver is generating no ACKs). Furthermore, I see why picking three duplicate ACKs (ie, three ACKs asking for the same byte) is a good threshold as it allows the client to accomodate some amount of out-of-order delivery without necessarily requiring retransmission. A couple followup questions: 1) Does the sender maintain a separate retransmission timer for each packet sent, or just one per connection? Is this timer reset each time it receives any ACK (even a duplicate ACK), or only when it receives the "correct" ACK? 2) What's the best resource for learning more about selective ACK in use in the Internet today, RFC 2018? What's the "state of the art" in selective ACK approaches (even if not widely deployed)? Thanks! -david Saikat Guha wrote: > On Sat, 2005-07-23 at 22:58 -0700, David Barrett wrote: > >>Can you clarify for me what precisely is a "duplicate ACK"? Specifically: >> >>- Does it literally mean two ACK packets that are byte-identical? Or >>just two ACKs that overlap? > > > Two inbound packets with the same # in the ACK field. > > > >>- Does the client ever send a duplicate ACK intentionally? Or is this >>just something that happens occasionally on bad connections? > > > The TCP receiver always ACK's (read: asks for) the first byte in the > stream that it has not yet received. Say it is waiting for byte X and > receives a packet starting with byte X+1400 ... then it will ACK/ask for > a packet with byte X. Say the sender sends three packets starting with > bytes X, X+1400, X+2800, X+3500 ... and the first and third packets are > lost; the other two packets will be responded to with an ACK for X each > -- thus making duplicate ACKs. The sender only learns that the first > packet was lost; and that some two packets did reach the destination > (since 2 ACKs were generated). The sender doesn't learn which packet > other than the first one was lost. For plain old TCP, this is fine > because the receiver doesn't queue up the packets it received since they > are out of sequence .. and waits for the sender to retransmit them (even > the ones that it did receive). > > SACK (selective ACK) sort of rectifies this by allowing the receiver to > tell the sender which ones were dropped, in a limited way -- kinda like > a bitmap but not quite. > > > Now the way regular TCP handles infers isolated losses and congestion is > as follows: > > If the sender doesn't receive an ACK for the byte it has not yet sent; > it concludes that a loss occurred and cuts the windows to 1. > > If it receives dup ACKs, then it infers that some packets are going > through fine and therefore there isn't a terrible lot of congestion. So > it artificially inflates its window and keeps sending packets (at half > the original rate instead of 1). See TCP Fast Retransmit and Fast > Recovery [1]. > > > [1] http://www.faqs.org/rfcs/rfc2001.html > > > > ------------------------------------------------------------------------ > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From decapita at dti.unimi.it Sun Jul 24 11:59:13 2005 From: decapita at dti.unimi.it (Sabrina De Capitani di Vimercati) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] ESORICS'05 - Call for Participation Message-ID: [Apologies if you receive multiple copies of this message] CALL FOR PARTICIPATION ESORICS 2005 10TH EUROPEAN SYMPOSIUM ON RESEARCH IN COMPUTER SECURITY Milan, Italy - September 12-14, 2005 http://esorics05.dti.unimi.it/ AIMS AND SCOPE Organized in a series of European countries, ESORICS is confirmed as the European research event in computer security. The symposium started in 1990 and has been held on alternate years in different European countries and attracts an international audience from both the academic and industrial communities. From 2002 it has been held yearly. The Symposium has established itself as one of the premiere, international gatherings on information assurance. PRELIMINARY PROGRAM Monday September 12, 2005 ------------------------- 09:15 - 09:30 Welcome and Opening 09:30 - 10:30 Invited talk -- Barbara Simons 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 1: Access control XML Access Control with Policy Matching Tree N. Qi, M. Kudo Semantic Access Control Model: A Formal Specification M. I. Yague, M. Gallardo, A. Mana A Generic XACML Based Declarative Authorization Scheme for Java R. Gupta, M. Bhide 12:30 - 14:00 Lunch 14:00 - 15:30 Session 2: Advanced Authorization Specifications Specification and Validation of Authorisation Constraints Using UML and OCL K. Sohr, G. Ahn Unified Index for Mobile Object Data and Authorizations V. Atluri, Q. Guo On Obligations M. Hilty, D. Basin, A. Pretschner 15:30 - 16:00 Coffe break 16:00 - 17:30 Session 3: Cryptographic Schemes A Practical, Voter-Verifiable Election Scheme D. Chaum, P.Y.A. Ryan, S.Schneider Machine-Checked Security Proofs of Cryptographic Signature Schemes S. Tarento Sanitizable Signatures G. Ateniese, D. Chou, B. de Medeiros, G. Tsudik Tuesday September 13, 2005 -------------------------- 09:00 - 10:30 Session 4: Cryptographic Protocols Limits of the Cryptographic Realization of Dolev-Yao-style XOR M. Backes, B. Pfitzmann Secure Implementation of Cryptographic Protocols: A Case Study Of Mutual Distrust A. Askarov, A. Sabelfeld Augmented oblivious Polynomial Evaluation Protocol and Its Applications H. Zhu 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 5: Intrusion detection Using Attack Trees to Identify Malicious Attacks from Authorized Insiders I. Ray, N. Poolsapassit An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Network Intrusion Alerts L. Wang, A. Liu, S. Jajodia Towards a Theory of Intrusion Detection G. Di Crescenzo, A. Ghosh, R. Talpade 12:30 - 14:00 Lunch 14:00 - 15:30 Session 6: Network security On Scalability and Modularisation in the Modelling of Network Security Systems J. de Albuquerque, H. Krumm, P. de Geus Sybil resistant DHT routing G. Danezis, R. Anderson Botnet Tracking: Exploring a Root-Cause Methodology to Prevent Distributed Denial-of-Service Attacks F.C. Freiling, T. Holz, G. Wicherski 15:30 - 16:00 Coffee break 16:00 - 17:30 Session 7: Information Flow and Formal Security Properties Quantifying Probabilistic Information Flow in Computational Reactive Systems M. Backes Enforcing Non-safety Security Policies with Program Monitors J. Ligatti, L. Bauer, D. Walker Soundness of Formal Encryption in the Presence of Key-Cycles P. Adao, G. Bana, J. Herzog, A. Scedrov Wednesday September 14, 2005 ---------------------------- 09:00 - 10:30 Session 8: Privacy and Data Protection Privacy Preserving Clustering S. Jha, L. Kruger, P. McDaniel Abstractions Preserving Parameter Confidentiality S. Gurgenas, P. Ochsenschlaeger, C. Rudolpah Minimal Disclosure in Hierarchical Hippocratic Databases with Delegation F. Massacci, J. Mylopoulos, N. Zannone 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 9: Security for protocols and devices Security Notions for Disk Encryption K. Gjosteen Local View Attack on Anonymous Communication M. Gogolewski, M. Klonowski, M. Kutylowski Browser Model for Security Analysis of Browser-Based Protocols T. Gross, B. Pfitzmann, A. Sadeghi REGISTRATION Online registration is available on the conference web page: http://esorics05.dti.unimi.it/registration.php ADDITIONAL INFORMATION On the web pages (http://esorics05.dti.unimi.it), you will find information about the program, the conference hotel and venue, and some travel and tourist information. We look forward to seeing you in Milan at ESORICS 2005. From coderman at gmail.com Sun Jul 24 14:23:53 2005 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <1122105789.7354.78.camel@localhost.localdomain> References: <42E1C3A7.3060402@quinthar.com> <1122105789.7354.78.camel@localhost.localdomain> Message-ID: <4ef5fec6050724072314d4477@mail.gmail.com> On 7/23/05, Saikat Guha wrote: > ... > In a wireless scenario, [1] shows that a single packet loss (a) or even > a threshold percent of packet loss within some period (b) are not good > indicators of congestion/contention -- i.e. cannot differentiate among > them. TCP uses a) and performs poorly in the wireless scenario. Instead, > [1] proposes looking at the number of _consecutive_ losses since such > losses are temporally correlated. FastTCP [2] more or less ignores > isolated losses and moreover uses delay based congestion signals. i've found the best results when the wireless transport layer takes the following 802.11 specific peculiarities into account: - if you are getting interference with other network / clients / microwaves the retransmission in layer 2 (802.11MAC) will cause intermittent single hop delays in excess of 1 or 2 seconds. - packet loss without congestion is the norm and not the exception; both intermittent latency and intermittent loss must be tolerated to a greater degree. a fun environment to tune a stack in. Best regards, [also, airhook is a nice example of how to optimize datagram transport for wireless environments] From paul at ref.nmedia.net Sun Jul 24 17:16:02 2005 From: paul at ref.nmedia.net (Paul Campbell) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: References: <200507231620.j6NGKCU76195@where.matthew.at> Message-ID: <20050724171602.GB31887@ref.nmedia.net> On Sat, Jul 23, 2005 at 11:43:49AM -0700, Serguei Osokine wrote: > One way around this problem would be to have a UDP-based P2P data > transfer protocol that would be less aggressive than TCP by design - in > that case, its swarm links would still fully occupy the available > bandwidth when the user does not use the machine for ineractive tasks, > but would shrink back in the presence of the TCP streams, allowing them > to take the full bandwidth to themselves self and allowing the user to > have the same online experience with P2P as there was without it. Until > that goal is achieved, I don't see P2P applications running on every > user machine as a background task 24x7, which is where they belong to. > Today they are too intrusive for that. Can't do that either. If you are less aggressive than TCP, then even if you solve the local link problem, you'll fail to get fair bandwith on the internal network connections. The basic problem also rears its head in the multicast vs. unicast protocols since there's obviously a tradeoff in fairness there as well...a multicast should theoretically be given a different treatment from a unicast since obviously the multicast affects a greater number of connections in spite of the fact that multicasting network impact is smaller (more destinations sharing the same stream). One possible way around this is to manage the ENTIRE load as X number of TCP links. An even better way would be to monitor internal and external link bandwith limits for the purpose of deciding how to maintain fairness at all levels. It can be done but the protocol is going to have to have a lot more intelligence built in than just AIMD. This somewhat reminds me of the cool process monitor tool that AmigaOS had. It showed a process monitor similar to what is built-in to Windows, Linux, etc. Except that then there were sliders on it where users could trivially tune process load on their machines. It seems like at some point it would be really handy to have this kind of capability built into a packet filtering application where you could adjust the relative importance and/or load of various applications to your desire. Then when you fire off that email client or click on a web page, it would recognize POP traffic and correspondingly give it more bandwidth vs. the P2P file transfer that is running in the background. This can't be done at the P2P app level because it needs to be able to differentiate traffic across the entire link. Obviously at the internal network level, this is meaningless since those kinds of capabilities are already built into the routers (where necessary) or not really capable of being used since at that level, it's a user vs. user or machine vs. machine question. From paul at ref.nmedia.net Sun Jul 24 17:19:59 2005 From: paul at ref.nmedia.net (Paul Campbell) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E334B6.7070409@ucla.edu> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> <42E32E21.6060602@quinthar.com> <42E334B6.7070409@ucla.edu> Message-ID: <20050724171959.GC31887@ref.nmedia.net> On Sat, Jul 23, 2005 at 11:27:02PM -0700, Michael Parker wrote: > David, > > Because TCP data can take different paths between a single source and > destination, TCP packets may arrive out-of-order at the destination. Now > TCP usually maintains a timer for each packet sent (usually set to the > RTO); if it has not received an ACK for that packet's sequence number by > the time this timer expires, it retransmits the packet. But in reality, > this is naive, and causes too long a delay for lost packets. Say the Fortunately, out-of-order conditions are so exceedingly rare that more recent protocols are essentially assuming that they don't happen. The amount of losses/overhead incurred by NOT assuming out-of-order=packet loss is less than 1%. I would have to dig up the report on this to show it, but the empirical information is out there. So under that scenario, any time that you are missing an ACK (out of order or not), it's a loss. From dbarrett at quinthar.com Mon Jul 25 08:34:51 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <20050724171959.GC31887@ref.nmedia.net> References: <42E00A3D.8030900@quinthar.com> <42E00FD8.1030103@felter.org> <42E02D82.7060007@quinthar.com> <20050721233155.GH1492@cs.uoregon.edu> <42E32E21.6060602@quinthar.com> <42E334B6.7070409@ucla.edu> <20050724171959.GC31887@ref.nmedia.net> Message-ID: <42E4A42B.3090508@quinthar.com> Paul Campbell wrote: > > Fortunately, out-of-order conditions are so exceedingly rare that more > recent protocols are essentially assuming that they don't happen. The amount > of losses/overhead incurred by NOT assuming out-of-order=packet loss is less > than 1%. I would have to dig up the report on this to show it, but the > empirical information is out there. So under that scenario, any time that > you are missing an ACK (out of order or not), it's a loss. Ah, interesting. This is one good way to simplify my implementation. Thanks! -david From elathan at ics.forth.gr Mon Jul 25 09:40:59 2005 From: elathan at ics.forth.gr (Elias Athanasopoulos) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] Gnutella Simulator Message-ID: Hello! I am looking for a Gnutella simulator, which is close enough to the modern Gnutella system. I found gnutellasim, which is based on top of ns, but there is a requirement for pdns-2.26, which I cannot locate (only pdns-2.27 is available from the project's home page). Is there any other alternative? Regards, -- Elias Athanasopoulos Distributed Computing Systems (DCS) Institute of Computer Science (ICS/FORTH) Heraklion, Crete A bug can become a feature by documenting it. From collectivefriction at gmail.com Mon Jul 25 11:01:52 2005 From: collectivefriction at gmail.com (Paul Sanderson) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] P2P systems overview and technical description Message-ID: <653034590507250401716b0d3a@mail.gmail.com> Hi All I am doing a little high level research into P2P and wondered where I could find a sort of high level, technical/semi-technical overview of how the different networks work. i.e. a sort of overview of the history, topography strengths and weaknesses of P2P in general and different systems specifically. From adam at cypherspace.org Mon Jul 25 11:15:04 2005 From: adam at cypherspace.org (Adam Back) Date: Sat Dec 9 22:13:01 2006 Subject: use tcp over udp? (Re: [p2p-hackers] UDP file transfer link speed identification) In-Reply-To: <42E330A0.7090500@quinthar.com> References: <20050722050556.GA3652@cs.uoregon.edu> <20050722172041.47688.qmail@web53607.mail.yahoo.com> <20050722192349.GA10503@bitchcake.off.net> <42E330A0.7090500@quinthar.com> Message-ID: <20050725111504.GA3347@bitchcake.off.net> On Sat, Jul 23, 2005 at 11:09:36PM -0700, David Barrett wrote: > Adam Back wrote: > >Why don't you just use the tcp stack's tcp implementation and tunnel > >the IP packets that TCP sends over upd? > > That's a damn good idea, actually. Can you give the 30-second > architectural overview of how Cebolla exposes a virtual Linux TCP > interface? Here's a forwarded response from Zach one of the authors of cebolla: | Sure can :) | | The most basic premise is that the application gets a file descriptor | that is used to read and write packets from a networking device in the | system. When a packet is delivered to this networking device, it is | made available to the application for read() on the fd. When the | application write()s to this file descriptor, the packet is injected | into the networking stack as though it just arrived at the device. | | So the trick is to configure this device<->fd path and then tell the | linux routing pieces to forward traffic for the TCP stream you're | interested in down that device. When the locally generated TCP traffic | is routed out the device, the application reads it, encapsulates it in | whatever goo it wants, and sends it out a UDP socket as usual. When | traffic comes back in to the application via UDP, the app trips off the | encapsulation and write()s the packet out the fd. This injects it back | into the local TCP stream. | | This stuff is no problem. The fd<->device path is setup with a magic | /dev/net/tun device. The routing is generally done by tagging streams | by user/endpoints/etc with iptables and then using iproute2 to configure | the routing subsystem to forward such tagged packets to the fake device. | | I don't know how much detail you're interested in, but there are some | tricky parts to this and the worst is propagating icmp for path mtu | discovery. | | Anyway, some resources: | | http://lxr.linux.no/source/Documentation/networking/tuntap.txt | http://lartc.org/howto/lartc.netfilter.html | | I couldn't find a strong description of the path mtu and icmp forwarding | problem that tunnels introduce, but section 3 of this rfc comes pretty | close: | | http://www.faqs.org/rfcs/rfc1853.html | | - z From trep at cs.ucr.edu Mon Jul 25 17:02:38 2005 From: trep at cs.ucr.edu (Thomas Repantis) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] P2P systems overview and technical description In-Reply-To: <653034590507250401716b0d3a@mail.gmail.com> References: <653034590507250401716b0d3a@mail.gmail.com> Message-ID: <20050725170238.GB1158@angeldust.chaos> You might be interested in the following technical report: http://www.hpl.hp.com/techreports/2002/HPL-2002-57R1.html Peer-to-Peer Computing Milojicic, Dejan S.; Kalogeraki, Vana; Lukose, Rajan; Nagaraja, Kiran; Pruyne, Jim; Richard, Bruno; Rollins, Sami; Xu, Zhichen HPL-2002-57R1 Cheers, Thomas On Mon, Jul 25, 2005 at 12:01:52PM +0100, Paul Sanderson wrote: > Hi All > > I am doing a little high level research into P2P and wondered where I > could find a sort of high level, technical/semi-technical overview of > how the different networks work. i.e. a sort of overview of the > history, topography strengths and weaknesses of P2P in general and > different systems specifically. -- http://www.cs.ucr.edu/~trep From trep at cs.ucr.edu Mon Jul 25 17:08:52 2005 From: trep at cs.ucr.edu (Thomas Repantis) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] Gnutella Simulator In-Reply-To: References: Message-ID: <20050725170852.GC1158@angeldust.chaos> Hi Elias, There are many different Gnutella simulators. I have used NeuroGrid in the past: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerSimulation Cheers, Thomas On Mon, Jul 25, 2005 at 12:40:59PM +0300, Elias Athanasopoulos wrote: > > Hello! > > I am looking for a Gnutella simulator, which is close enough to the > modern Gnutella system. I found gnutellasim, which is based on top > of ns, but there is a requirement for pdns-2.26, which I cannot locate > (only pdns-2.27 is available from the project's home page). > > Is there any other alternative? > -- http://www.cs.ucr.edu/~trep From trep at cs.ucr.edu Mon Jul 25 18:11:20 2005 From: trep at cs.ucr.edu (Thomas Repantis) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] Distributed Snapshots Message-ID: <20050725181120.GA1902@angeldust.chaos> Hi all, Can someone verify whether I am correct in my understanding of the following: The Chandy-Lamport algorithm [1] can produce a distributed snapshot of a system only if *all* nodes are running the algorithm. In other words, I am looking for a way to get a global snapshot of a system of nodes say A, B, and C. The caveat is that they might exchange messages with node D, which will not run a snapshot algorithm. I am not interested in the state of D, but I need a consistent global snapshot of A, B, and C. Am I correct in my understanding that Chandy-Lamport cannot help in this case? Is there any other way to get a consistent global snapshot of such a distributed system? Many thanks! Thomas [1] Distributed Snapshots: Determining Global States of a Distributed System K. Mani Chandy, Leslie Lamport ACM Transactions on Computer Systems 3, 1 (February, 1985), 63-75. http://research.microsoft.com/users/lamport/pubs/chandy.pdf -- http://www.cs.ucr.edu/~trep From dbarrett at quinthar.com Mon Jul 25 20:16:25 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <4ef5fec6050724072314d4477@mail.gmail.com> References: <42E1C3A7.3060402@quinthar.com> <1122105789.7354.78.camel@localhost.localdomain> <4ef5fec6050724072314d4477@mail.gmail.com> Message-ID: <42E54899.4060909@quinthar.com> coderman wrote: > i've found the best results when the wireless transport layer takes > the following 802.11 specific peculiarities into account: > - if you are getting interference with other network / clients / > microwaves the retransmission in layer 2 (802.11MAC) will cause > intermittent single hop delays in excess of 1 or 2 seconds. Am I understanding you correctly that even though UDP is unreliable, 801.11 still does retransmission under some circumstances, and when it does it introduces a 1-2 second delay? Does this delay only affect the packet being retransmitted, or does it affect it and all subsequent packets in the UDP flow? In all connections? > - packet loss without congestion is the norm and not the exception; > both intermittent latency and intermittent loss must be tolerated to a > greater degree. > > a fun environment to tune a stack in. Yes, no kidding. Have you tried using any congestion control technique that doesn't rely upon packet loss as an indicator? I'm still reading through the "TCP-Friendly" congestion control profile for DCCP that Wes recommended a while back. My understanding is it's "equation based" (as opposed to packet-loss triggered?) but I haven't read far enough to know what that means, though. -david From matthew at matthew.at Mon Jul 25 20:26:45 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E54899.4060909@quinthar.com> Message-ID: <200507252026.j6PKQZU79722@where.matthew.at> David Barrett: > Am I understanding you correctly that even though UDP is unreliable, > 801.11 still does retransmission under some circumstances, > and when it does it introduces a 1-2 second delay? Even if the delay isn't that long, the way 802.11 handles automatic speed adjustment when the signal level is low can create large delay variance ("jitter") > Does this > delay only affect the packet being retransmitted, or does it > affect it and all subsequent packets in the UDP flow? In all > connections? Typically you find that everything sent during a retransmit-due-to-respeed/loss gets buffered and sent later. > Yes, no kidding. Have you tried using any congestion control > technique that doesn't rely upon packet loss as an indicator? First problem with this theory is that you have to interoperate with TCP, which *does* rely upon packet loss as an indicator. The only real improvement to be had in this area is Explicit Congestion Notification, which isn't deployed widely enough to be useful. > I'm still reading through the "TCP-Friendly" congestion > control profile for DCCP that Wes recommended a while back. > My understanding is it's "equation based" (as opposed to > packet-loss triggered?) but I haven't read far enough to know > what that means, though. We've played with various equation-based schemes here. Almost all of them are designed to "smooth" the response to congestion, or to provide for congestion control with less frequent feedback than ack-every or ack-every-other as most TCP implementations do. As I've pointed out before, in order to be fair with a smoothed response, you must compensate properly for those times when, because of the smoothing, you're sending more than TCP would at that instant, by sending less later. And, much more important, and as I've also pointed out before, doing congestion control properly requires instantaneous feedback, or at the very least, feedback one RTT later than instantaneous. If you try to do equation-based but only give it loss data, say, every second, or every 10 seconds, what you'll find is that the loss data is *so* out of date that the response is totally decoupled from how real TCP would work. Matthew From gbildson at limepeer.com Mon Jul 25 22:14:02 2005 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: Message-ID: > It would be interesting to know whether LimeWire and BearShare > did any experiments in that field - ideally, the long FTP transfer > should be unaffected by the parallel P2P transfers (which should not > make any progress in the meantime, of course). Greg, did you try > measuring something like that? Serguei, I just noticed your question now as I read backwards through these emails. LimeWire's UDP goal was very minimal - make it work without blowing up the internet. I don't recall testing it against a steady ftp stream although that would be an interesting test. I did test it on a server with heavy load and it performed fine. Our goal was not to have zero impact on TCP connections; we just wanted to coexist reasonably well. > More often than not all these streams are TCP, so if you wish to > get your mail or do some Web browsing, your new TCP connection becomes > just another one among 5, 10, or 20 already saturated P2P ones. Hence > its data transfer rate will be 6, 11, or 21 times slower than what you > used to have without P2P application running, and your user experience > will suffer accordingly. We actually borrowed a trick from BearShare to get around this problem. We pause all TCP activity of the client for 1 second every other second (if the user is not idle I believe). By pause here, I mean that we stop reading and writing. This appears to leave room for other applications (browsers, email, etc) to do their thing - surprisingly with little effect on throughput. Note that during the pause, system buffers could still be getting filled up and emptied out to some extent at least. It was fairly surprising to find out just how well this approach worked. One caveat was that this approach worked horribly when applied to UDP file transfers - likely due to the smaller built in buffering capabilities and quick slow down when the buffer is close to being full. Thanks -greg From ap at hamachi.cc Tue Jul 26 05:52:53 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <200507231615.j6NGFGU76183@where.matthew.at> References: <200507231615.j6NGFGU76183@where.matthew.at> Message-ID: <42E5CFB5.7030306@hamachi.cc> Matthew Kaufman wrote: > Alex Pankratov: > >>PS. My experience is purely empirical. If anyone is aware of >>any formal research on RTT/congestion subject, I'd really >>like to hear about it. Thanks. > > > Start with the papers about TCP Vegas. > > Matthew > Thanks, that's a good read. I seem to be missing something though. I was reading your reply to David earlier today and you wrote - > First problem with this theory is that you have to interoperate with TCP, > which *does* rely upon packet loss as an indicator. > > The only real improvement to be had in this area is Explicit Congestion > Notification, which isn't deployed widely enough to be useful. So do I understand correctly that eventhough TCP Vegas tries to avoid the congestion (with a secondary goal of preventing a packet loss), it still somehow relies on the loss to control its Tx rate ? Alex From matthew at matthew.at Tue Jul 26 06:25:02 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <42E5CFB5.7030306@hamachi.cc> Message-ID: <200507260624.j6Q6OrU80218@where.matthew.at> Alex Pankratov: > So do I understand correctly that eventhough TCP Vegas tries > to avoid the congestion (with a secondary goal of preventing > a packet loss), it still somehow relies on the loss to > control its Tx rate ? As I remember (I haven't read the papers in some months), the Vegas implementation adds the use of detecting RTT increase to detect incipient congestion and to back off in that case in an effort to avoid loss, but when actual loss occurs, the rate is still dropped as per TCP Reno. My comments about UDP-based protocols interoperating well with TCP is based on the fact that most TCP implementations are Reno- or newReno-like. Matthew From ap at hamachi.cc Tue Jul 26 06:50:38 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer protocol peer review In-Reply-To: <200507260624.j6Q6OrU80218@where.matthew.at> References: <200507260624.j6Q6OrU80218@where.matthew.at> Message-ID: <42E5DD3E.80701@hamachi.cc> Matthew Kaufman wrote: > Alex Pankratov: > >>So do I understand correctly that eventhough TCP Vegas tries >>to avoid the congestion (with a secondary goal of preventing >>a packet loss), it still somehow relies on the loss to >>control its Tx rate ? > > As I remember (I haven't read the papers in some months), the Vegas > implementation adds the use of detecting RTT increase to detect incipient > congestion and to back off in that case in an effort to avoid loss, but when > actual loss occurs, the rate is still dropped as per TCP Reno. > > My comments about UDP-based protocols interoperating well with TCP is based > on the fact that most TCP implementations are Reno- or newReno-like. > Right, so I assumed. Thanks From matthew at matthew.at Tue Jul 26 07:11:38 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] amicima's MFP - update Message-ID: <200507260711.j6Q7BSU80358@where.matthew.at> Due to the large number of responses received both on-list and off-list from our last announcement, we're continuing the announcements on this list (please contact me if that's a problem for you). Additional documentation is now available on the amicima web site for interested developers. The header files for our MFP protocol implementation, MFPMsg message-passing layer and MFPMU event-loop library (all of which will be part of the initial MFP protocol implementation release) are now available at: http://www.amicima.com/developers/documentation.html In addition, the MFP Protocol document has been upgraded to version 1.1 after comments received here on the list. The version 1.1 protocol now supports a bilateral keying model in which each side contributes to the generation of the symmetric session key. It also now supports an optional Session Sequence Number chunk which may be used to implement stronger anti-replay measures. As discussed previously, the protocol itself does not specify a particular cryptosystem, and the cryptosystem implementation "plugs in" to our MFP protocol implementation, so HMAC, if desired, may be added at that point. Additional details about the default cryptosystem provided with the initial distribution will be available in the distribution documentation. We are working towards releasing the MFP protocol implementation by the end of this month, but as with all unfunded software startups, it might take a little longer, depending on circumstances. We've been working on this for 16 months, after all. The MFPNet layer will follow shortly thereafter. Matthew Kaufman matthew@matthew.at matthew@amicima.com http://www.amicima.com From tutschku at informatik.uni-wuerzburg.de Thu Jul 28 13:18:29 2005 From: tutschku at informatik.uni-wuerzburg.de (Kurt Tutschku) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] CFP: 3rd IEEE International Workshop on Mobile Peer to PeerComputing MP2P 06, (conjunction with the 4th IEEE PerCom 06) Message-ID: <20050728132151.CDC1E6F58E@nero.informatik.uni-wuerzburg.de> (Please apologize multiple copies) --------------------------------------------------------------------- Call-for-Paper 3rd IEEE International Workshop on Mobile Peer-to-Peer Computing (MP2P'06) http://www-info3.informatik.uni-wuerzburg.de/mp2p2006 Pisa, Italy, March 17, 2006 In conjunction with the 4th IEEE International Conference on Pervasive Computing and Communications (PerCom?06) http://www.percom.org/ ---------------------------------------------------------------------- Peer-to-Peer (P2P) computing is a networking and distributed computing paradigm which allows the sharing of computing resources and services by direct, symmetric interaction between computers. The advance in mobile wireless communication technology and the increasing number of mobile users, extend the arena of P2P computing to encompass mobile devices and wireless networks. The special characteristics of mobile environments, such as highly variable connectivity, disconnection, location-dependency, resource contraints, and diversity in wireless networks as well as carrier-grade performance requirements bring new challenges for research in mobile P2P computing. MP2P?06 is intended to serve as a continuing forum for scientists and engineers in academia and industry to exchange and discuss their experiences, new ideas, and research results about all aspects of mobile P2P computing. It will address the challenges, technologies, and architectures leading to real-world solutions that provide users with direct access and control of their critical peer-based information and services, regardless of location or device. The principal theme of MP2P?06 is the development of protocols, systems and architectures of mobile P2P architectures and the evaluation of their performance. Topics of particular interest include, but are not limited to: - Architecture and platforms for mobile P2P computing - Measurement studies on mobile P2P systems - Performance of mobile P2P services - Resource and service discovery in mobile P2P computing - Resource exchange mechanisms in mobile P2P application - Mobile P2P over different kinds of bearer services: 2.5/3G (GPRS/UMTS) / 802.11 (WLAN) - Mobile P2P & operator/provider requirements - Reliability and carrier-gradeness of mobile P2P services - Mobile P2P & fixed P2P system interworking - Issues of combining P2P services with mobility (mobile IP / MANET) - Peer access and control in mobile environment - Location dependent mobile P2P services - Data exchange and rendering techniques for mobile devices - Secure communication protocols for mobile P2P computing - Mobile P2P messaging systems - Peer-to-peer broadband wireless communications - Applications of mobile P2P - Nature-inspired algorithms for mobile P2P computing - Theoretical issues on mobile information diffusion - Mobile P2P video games - Stimulating cooperation in mobile computing Paper Submission: ----------------- Papers must be written in English and should not exceed 6 pages in IEEE proceedings style. Authors MUST submit their papers through the EDAS web site (http://www.edas.info/) in three steps: * Creation of a personal account on EDAS (if the author does not already have one) * Registration of the paper (requiring a short abstract of up to 150 words) * Upload of the paper. Only in pdf format Submission implies the willingness of at least one of the authors to register and present the paper. All submissions will be reviewed and selected based on their originality of the paper. Accepted papers must be presented at the workshop and will appear in a combined PerCom 2006 workshop proceedings published by IEEE Computer Society Press. Authors that present a system in their paper are highly encouraged to demonstrate also the system in the demo session of the workshop. Important Dates: ---------------- Papers due: 5pm EST, September 15, 2005 Notification of acceptance November 15, 2005 Camera-ready papers due December, 2005 Organizing Committee, Program Co-chairs: ---------------------------------------- Kurt Tutschku, Department of Distributed Systems, University of W?rzburg, Germany. Frank-Uwe Andersen, Siemens Communications, Berlin, Germany. Li Li, Communications Research Center Canada, Canada. Maria Papadopouli, Department of Computer Science, University of North Carolina at Chapel Hill, USA Steering Board -------------- - Jiannong Cao, Department of Computing, Hong Kong Polytechnic University. - Y. Charlie Hu, School of Electrical and Computer Engineering, Purdue University - Cecilia Mascolo, Department of Computer Science, University College London - Maria Papadopouli, Department of Computer Science, University of North Carolina at Chapel Hill Publicity Chair --------------- Kurt Tutschku, Department of Distributed Systems, University of W?rzburg, Germany Program Committee Members: -------------------------- - Frank-Uwe Andersen, Siemens Communications, Germany. - Christian Bettstetter, DoCoMo Euro-Labs,Germany. - Jiannong Cao, Department of Computing, Hong Kong Polytechnic University. - Luca Caviglione, CNIT - University of Genoa Research Unit, University of Genova, Italy. - Geoff Coulson, Computing Department, University of Lancaster, UK. - Hermann de Meer, Department of Computer Networks & Computer Communications, University of Passau, Germany. - Babak Esfandiari, Carleton University, Canada. - Stephen Hailes, Department of Computer Science, University College of London, UK - Felix Hernandez-Campos, Department of Computer Science, University of North Carolina at Chapel Hill, USA - Y. Charlie Hu, School of Electrical and Computer Engineering, Purdue University - Norihiro Ishikawa, NTT Docomo, Japan. - Valerie Issarny, INRIA, France. - Wolfgang Kellerer, DoCoMo Communications Laboratories Europe, Germany. - Kazuhiro Kitagawa, Keio University, Japan. - Thomas Kunz, Dept. of Systems and Comp. Engineering, Carleton University. Canada. - Li Li, Communications Research Center Canada, Canada. - Christoph Lindemann, FB Informatik IV, University of Dortmund , Germany. - Maria Papadopouli, Department of Computer Science, University of North Carolina at Chapel Hill - Rudolf Riedi, Department of Statistics, Rice University, USA - Pablo Rodriguez, Microsoft Research Ltd, Cambridge, UK. - George Roussos, School of Computer Science and Information Systems, Birkbeck College, University of London, UK - Shervin Shirmohammadi, School of Information Technology and Engineering (SITE), University of Ottawa, Canada. - Kurt Tutschku, Department of Distributed Systems, University of W?rzburg, Germany. --- Dr. Kurt Tutschku, Assistant Professor University of Wuerzburg Department of Distributed Systems Institute of Computer Science Am Hubland 97074 Wuerzburg Germany Tel.: +49-931-8886641 FAX.: +49-931-8886632 mailto:tutschku@informatik.uni-wuerzburg.de or mailto:kurttutschku@hotmail.com http://www-info3.informatik.uni-wuerzburg.de/staff/tutschku From gbildson at limepeer.com Fri Jul 29 19:53:59 2005 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: <42E036BC.3080709@quinthar.com> Message-ID: > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of David Barrett > > > In reality, we decrease the wait time between data packets and > > actually start making some multiple of wait times zero before a real > > wait occurs. The zero wait times is just a way of dealing with the > > fact that our current data message size is small (500 bytes) and > > fixed at the moment. > > Could you restate this in another way? I didn't quite get what you were > trying to say; I don't see how wait time is related to message size. > > Incidentally, why are your data packets so small? > > Its a little late in the game but let me just follow up with some history and some answers. My desire was to make the simplest possible reliable UDP protocol that performed well. I started by looking at the trivial file sharing protocol. This certainly worked but suffered from fairly poor performance given that there is an ack for each packet. Packet size wasn't specifically a constraint and I wanted to avoid UDP fragmentation problems so I chose to fix our initial implementation at 500 bytes of data per packet (with up to 4K allowed). To get more throughput, we just send multiple packets with zero delay. So, a real delay may only occur after 1, 2, 3, 4, 5, ... 14 packets are sent. Beyond those basic answers, we added a few very simple ideas from TCP and of our own making. Simple as in a 20 packet receipt window, including the highest contiguous sequence number in acks, suppressing rapid acks to reduce ack bandwidth, textbook rtt calculations, etc. Thanks -greg From dbarrett at quinthar.com Sat Jul 30 05:41:31 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] UDP file transfer link speed identification In-Reply-To: References: Message-ID: <42EB130B.6070906@quinthar.com> Greg Bildson wrote: > This certainly worked but suffered from fairly poor performance given > that there is an ack for each packet. Thanks for the overview, Greg, I understand now. But I'm curious about your ack/packet performance statement. Do you mean it consumes an unnecessary amount of upstream bandwidth, or do you mean it actually obtrains lower throughput? After a lot of research, I've finally settled on an approach much like the DCCP "TCP-like" congestion-control profile. In essence it uses selective ACK, cwnd, pipe, ssthresh, and other hallmarks of TCP congestion control. However, it doesn't offer reliability, thus offering an unreliable equivalent to TCP. At the moment I use one ACK per packet. However, the sender still sends many packets at a time; it doesn't wait a RTT for each new packet to be sent, nor does it delay delivery due to out-of-order arrival. Thus while I'm sure this approach consumes an unfortunate amount of upstream bandwidth, I don't believe it actually reduces downstream throughput. But I've done no head-to-head comparision against anything else, so I'd love your feedback on whether I'm missing some big downstream throughput performance penalty. Any ideas? -david From aharwood at cs.mu.oz.au Sun Jul 31 13:14:32 2005 From: aharwood at cs.mu.oz.au (Aaron Harwood) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] Gnutella Simulator In-Reply-To: References: Message-ID: We are just about to release a package called VPDNS which, among other things, will provide instructions for applying the gnutella patches to pdns-2.27. --Aaron On 25/07/2005, at 7:40 PM, Elias Athanasopoulos wrote: > > Hello! > > I am looking for a Gnutella simulator, which is close enough to the > modern Gnutella system. I found gnutellasim, which is based on top > of ns, but there is a requirement for pdns-2.26, which I cannot locate > (only pdns-2.27 is available from the project's home page). > > Is there any other alternative? > > Regards, > -- > Elias Athanasopoulos > Distributed Computing Systems (DCS) > Institute of Computer Science (ICS/FORTH) > Heraklion, Crete > > A bug can become a feature by documenting it. > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From matthew at matthew.at Sun Jul 31 23:31:58 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:01 2006 Subject: [p2p-hackers] amicima's MFP - source code released In-Reply-To: <200507260711.j6Q7BSU80358@where.matthew.at> Message-ID: <200507312332.j6VNW5U92198@where.matthew.at> We are very pleased to announce that we got it done by month-end... The first public release of amicima's MFP protocol implementation is now available for download at the amicima web site. This release builds and has been tested on Linux (Debian 3.1), FreeBSD (5.2.1), Mac OS X (10.4), Windows XP (XP SP2 w/Visual Studio.NET). The amicima MFP implementation is licensed under the GPL, with an available commercial license for developers of proprietary applications. Download page: http://www.amicima.com/developers/downloads.html In case you downloaded an earlier MObj release, you will need to update to the latest release of MObj (20050728), available on our downloads page before building MFP. In the next few days we will be publishing the MFPNet API header files and expect to release the MFPNet implementation by the end of August 2005. The MFPNet layer provides an API that allows MFP flows to be initiated to named endpoints without application knowledge of that endpoint's current location or IP address, automatically traversing NAT and firewall devices without application intervention. For the developer, this provides a simple interface to seamless, reconfiguration-free operation in all network environments, even when providing peer-to-peer services from behind NAT or firewall devices. After MFPNet is released, we plan to release the source code to a demo peer-to-peer VOIP + file transfer application for Windows which uses MFP and MFPNet. Matthew Kaufman - amicima, Inc. matthew@matthew.at matthew@amicima.com www.amicima.com