Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193] helo=mx.sourceforge.net) by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1TLiyB-0007Ry-Li for bitcoin-development@lists.sourceforge.net; Tue, 09 Oct 2012 23:12:11 +0000 X-ACL-Warn: Received: from mail-qa0-f54.google.com ([209.85.216.54]) by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1TLiyA-0003br-JT for bitcoin-development@lists.sourceforge.net; Tue, 09 Oct 2012 23:12:11 +0000 Received: by mail-qa0-f54.google.com with SMTP id y23so3918837qad.13 for ; Tue, 09 Oct 2012 16:12:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=L6YbKIZVArj0+OGGzNniC8chCe+wV1ueOHFvLvnTIms=; b=ExXr+TWYnzU94gA+EAaKZzM4osctAbfFSuueM7hF85xPiRFMRowlh1Vo3RH9m1RlHz c8km2aEq1muNC70oIKEz+Bc+3kNdiMavXTPzl2j/F0A1SSkNTvhrLlWp5/x2z0yjBhlL af0A3PwX78zx7QKkVvVhY9aH4oHZ3bD1rVSeOhcwfx3KBd8Ck65xr6or+LwipD/XiKK7 Bwss+Blsr+gT2Zl3UBRW4uoCZUaCAZbzRMUcKetY+EQShtJ0+z7kOtNgOInmcAruh7kM DZSXNFGAtdo8Qi/vjhX1iN0ZBUg+iib90rBgLdH1unBCsb3XlhjgBzLGyuAyCo+3kGwl FIoQ== MIME-Version: 1.0 Received: by 10.49.96.162 with SMTP id dt2mr52002420qeb.48.1349824325002; Tue, 09 Oct 2012 16:12:05 -0700 (PDT) Received: by 10.49.97.6 with HTTP; Tue, 9 Oct 2012 16:12:04 -0700 (PDT) X-Originating-IP: [2001:4830:1603:2:21c:c0ff:fe79:c8c2] Date: Tue, 9 Oct 2012 19:12:04 -0400 Message-ID: From: Jeff Garzik To: Bitcoin Development Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlgc4ePzIM2FpGr0WcvbOcdX1WZBPdnHl/sZCeP3twnCieOs0JbVLfwXBd0DCP0+lMQfFUk X-Spam-Score: 0.0 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. X-Headers-End: 1TLiyA-0003br-JT Subject: [Bitcoin-development] On bitcoin testing X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Oct 2012 23:12:11 -0000 Copying from a response posted to "Bitcoin software testing effort" https://bitcointalk.org/index.php?topic=117487.0 as it is relevant to a recent thread here... Any level of testing is useful and appreciated. Various types of testing that are helpful: * "it works" testing: Simply run the latest Release Candidate (or latest version, if released). Make sure all the basics work (for whatever definition of "basics" you desire). This is the level most accessible to casual users. * Major features testing: Develop a short checklist of must-work features, and organize volunteers to work together and go through that checklist, item by item. Test each major feature on each major platform. * Stress and fuzz testing: Attempt to "stress" the system somehow, or randomly corrupt bits of data. See what breaks. * Regression testing: Record bugs fixed, and develop automated test cases that successfully reproduce the bugs on older versions, and verify newer versions remain fixed. * Unit function testing: Rigorously exercise each C++ class to ensure it behaves as expected at a micro level. * Full peer automated testing: Automated testing of RPC and P2P functions is non-existent, because of the difficulty in doing so. Find a solution to this problem. * Data-driven tests: If possible, write software-neutral, data-driven tests. This enables clients other than the reference one (Satoshi client) to be tested. Embed tests in testnet3 chain, if possible. The community at large can be a big help simply by doing the first item: download and run the Release Candidates and the latest version, and report any problems. Even reporting success is fine by me, for example: "Version 0.7.1 works for me on Windows 7/32-bit" posted on a forum thread. It is always very difficult to organize any sort of testing regime with open source volunteers that come and go. Each volunteer chooses their level of involvement. Any amount of testing and test-case writing, large or small, is helpful to bitcoin. -- Jeff Garzik exMULTI, Inc. jgarzik@exmulti.com