Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of bluematt.me
	designates 173.246.101.161 as permitted sender)
	client-ip=173.246.101.161;
	envelope-from=bitcoin-list@bluematt.me; helo=mail.bluematt.me; 
Message-ID: <1365183223.2757.16.camel@localhost.localdomain>
From: Matt Corallo <bitcoin-list@bluematt.me>
To: Adam Ritter <aritter@gmail.com>
Date: Fri, 05 Apr 2013 13:33:43 -0400
In-Reply-To: <CAKuKjyUFBuPMPRV6R-u0iTa=8DWMN9vqdOxnr8o8kxg9rJtVBA@mail.gmail.com>
References: <CAKuKjyUFBuPMPRV6R-u0iTa=8DWMN9vqdOxnr8o8kxg9rJtVBA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Cc: bitcoin-development@lists.sourceforge.net
Subject: Re: [Bitcoin-development] Integration testing for BitCoin
Precedence: list

These tests are run on each pull requests and on each new commit to
master.  They arent very complete, but they do test a lot of block
acceptance rules.

https://github.com/TheBlueMatt/test-scripts

Matt

On Fri, 2013-04-05 at 12:24 -0500, Adam Ritter wrote:
> Hey guys,
> 
> I just bought some BitCoins after being lazy to do it for the last few
> years, but also looked at the client code and the messages that are
> going on this mailing list.
> I saw that there are quite some unit tests, but I didn't find
> integration test for BitCoin, and I believe that it's quite important
> for the future of BitCoin (making the current code more stable,
> testing attack scenarios, refactoring and extending code).
> 
> I have wrote some integration tests before at other projects, and they
> usually turned out useful, but I have 0 experience with the BitCoin
> development and codebase.
> I wrote a short document of what I think would be the safest way to do
> the testing (but not yet the tests themselves, as I don't have enough
> experience..I'd like to have something like testing that the wallets
> are empty, and after somebody mines she'll have more money..after she
> sends money to the other person, the other person will see it...things
> like this, just to get to know the code base).
> 
> What do you guys think?
> The plan is here:
> https://github.com/xiphias/bitcoin/blob/master/src/test/integration/README.md
> Please feel free to comment/fork, I'll try to write all your replies
> in the document as well.
> 
> Also here's the text to make it easier to comment:
> 
> Integration testing for bitcoin
> ================================
> 
> Tests that simulate multiple bitcoin users and can verify that the
> whole network of bitcoin clients work together
> to achieve the goals of Bitcoin. Also maybe [System
> testing](http://en.wikipedia.org/wiki/System_testing)
> would be a better name for the tests, but I'm not sure.
> 
> Goals
> ---------
> - Make the bitcoin code easier to refactor while increasing the guarantee
>  that it doesn't break the overall behaviour of the client.
> - Make it easier to have multiple experimental coins (for example
> LightCoin or PPCoin) in the codebase, while guaranteeing that the
> original BitCoin protocol doesn't break
> - Make it easy to test attack scenarios (like DOS,
>  releasing an incompatible BitCoin client), monitoring
> - Have tests without (or at least minimal number of) unreadable
> constants and unreadable fake data to make them easier to verify.
> 
> Proposed implementation
> ------------------------------------
> The first implementation should use the JSON-RPC interface and build
> up as much verification
> of BitCoin network as possible. It should be in C++, which makes it
> easier to move to the second implementation.
> The JSON-RPC calls should be hidden by a C++ interface.
> 
> The second implementation should use the same interface that was used
> for JSON-RPC,
> but using the BitCoin code directly (while not breaking the JSON-RPC tests).
> For this the BitCoin client has to be refactored as a library,
> getting rid of all global variables (and having them in a data structure),
> so that multiple BitCoin clients can be run in the same process.
> 
> The improvement of the second implementation should have dependency injection
> for the time and for finding/verifying a mined block,
> so that the tests don't need to use real CPU power for mining,
> and they can run faster and test more complex scenarios.
> 
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire 
> the most talented Cisco Certified professionals. Visit the 
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development