Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of mistfpga.net
	designates 208.91.199.220 as permitted sender)
	client-ip=208.91.199.220; envelope-from=steve@mistfpga.net;
	helo=us2.outbound.mailhostbox.com; 
Message-ID: <5062F4F8.6040504@mistfpga.net>
Date: Wed, 26 Sep 2012 13:28:40 +0100
From: steve <steve@mistfpga.net>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
	rv:15.0) Gecko/20120907 Thunderbird/15.0.1
MIME-Version: 1.0
To: Matt Corallo <bitcoin-list@bluematt.me>
References: <5061F8CC.9070906@mistfpga.net>
	<1348605677.2284.2.camel@localhost.localdomain>
In-Reply-To: <1348605677.2284.2.camel@localhost.localdomain>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Cc: Bitcoin Development List <bitcoin-development@lists.sourceforge.net>,
	Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Precedence: list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Matt,

Glad to have another ninja onboard :)

On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).

I have never heard of jenkins before.  I need to do some more digging.
is this the right thing?

https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin

Mantis on the other hand, I know exceptionally well.  I hate
duplication of work/data unless absolutely necessary.  I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)

So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?

Adding to this doesnt make sense.  Each one of these reporting methods
is for a different thing.  I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns].  Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.

The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.

> 
> They both run the same set of scripts, namely those at 
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).

I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases.  have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.

This is the outline of the testing that I setup for 0.7

https://secure.bettermeans.com/projects/4256/wiki

> 
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?

Yes, I would love to unify all build testing and testcases into one
place.  I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)

2 - Acceptance based testcases - these are testcases that should be
run for every build.  Check out the General Acceptance Tests in the
wiki link for examples and testcases

3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)

These three things alone are a massive task. and they still wont cover
everything.  I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])

Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.

> 
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).

Nice, I love testing.  I think we will get on :)

And I would rather go for interoperability between testing rather than
rewriting it all.

Cheers,

steve

> 
> Matt
> 
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>> 
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>> 
>> I am heavily into test driven development, and I have a strong 
>> background in requirements management, and automation.
>> 
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>> 
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>> 
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>> 
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>> 
>> This was a much longer post, but decided against it. :)
>> 
>> So, what I want to know is who wants to be a part helping me out
>> with all this? I am finalising the workflow flow diagrams and
>> they should be ready for inspection soon.
>> 
>> Anyone interested in helping out/reviewing processes? even if it
>> is just some encouragement, it is all greatly appreciated.
>> 
>> Drop me an email if you want access to the current setup and help
>> me review the different software for the bitcoin workflow
>> process.
>> 
>> cheers,
>> 
>> steve
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT
BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb
M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP
y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/
xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI
bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4=
=W5xI
-----END PGP SIGNATURE-----