1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
|
Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
helo=mx.sourceforge.net)
by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <steve@mistfpga.net>) id 1TGqkC-0008Fs-1h
for bitcoin-development@lists.sourceforge.net;
Wed, 26 Sep 2012 12:29:36 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of mistfpga.net
designates 208.91.199.220 as permitted sender)
client-ip=208.91.199.220; envelope-from=steve@mistfpga.net;
helo=us2.outbound.mailhostbox.com;
Received: from us2.outbound.mailhostbox.com ([208.91.199.220])
by sog-mx-3.v43.ch3.sourceforge.com with esmtp (Exim 4.76)
id 1TGqkA-0006Gb-NU for bitcoin-development@lists.sourceforge.net;
Wed, 26 Sep 2012 12:29:36 +0000
Received: from [10.10.10.55] (5ad2e75a.bb.sky.com [90.210.231.90])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
(Authenticated sender: steve@mistfpga.net)
by us2.outbound.mailhostbox.com (Postfix) with ESMTPSA id D56B4699585;
Wed, 26 Sep 2012 12:29:15 +0000 (GMT)
Message-ID: <5062F4F8.6040504@mistfpga.net>
Date: Wed, 26 Sep 2012 13:28:40 +0100
From: steve <steve@mistfpga.net>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
rv:15.0) Gecko/20120907 Thunderbird/15.0.1
MIME-Version: 1.0
To: Matt Corallo <bitcoin-list@bluematt.me>
References: <5061F8CC.9070906@mistfpga.net>
<1348605677.2284.2.camel@localhost.localdomain>
In-Reply-To: <1348605677.2284.2.camel@localhost.localdomain>
X-Enigmail-Version: 1.4.4
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.72 on 172.16.214.9
X-Spam-Score: -1.6 (-)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
-0.0 SPF_PASS SPF: sender matches SPF record
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1TGqkA-0006Gb-NU
Cc: Bitcoin Development List <bitcoin-development@lists.sourceforge.net>,
Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Wed, 26 Sep 2012 12:29:36 -0000
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Matt,
Glad to have another ninja onboard :)
On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).
I have never heard of jenkins before. I need to do some more digging.
is this the right thing?
https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin
Mantis on the other hand, I know exceptionally well. I hate
duplication of work/data unless absolutely necessary. I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)
So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?
Adding to this doesnt make sense. Each one of these reporting methods
is for a different thing. I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns]. Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.
The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.
>
> They both run the same set of scripts, namely those at
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).
I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases. have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.
This is the outline of the testing that I setup for 0.7
https://secure.bettermeans.com/projects/4256/wiki
>
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?
Yes, I would love to unify all build testing and testcases into one
place. I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)
2 - Acceptance based testcases - these are testcases that should be
run for every build. Check out the General Acceptance Tests in the
wiki link for examples and testcases
3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)
These three things alone are a massive task. and they still wont cover
everything. I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])
Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.
>
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).
Nice, I love testing. I think we will get on :)
And I would rather go for interoperability between testing rather than
rewriting it all.
Cheers,
steve
>
> Matt
>
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>>
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>>
>> I am heavily into test driven development, and I have a strong
>> background in requirements management, and automation.
>>
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>>
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>>
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>>
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>>
>> This was a much longer post, but decided against it. :)
>>
>> So, what I want to know is who wants to be a part helping me out
>> with all this? I am finalising the workflow flow diagrams and
>> they should be ready for inspection soon.
>>
>> Anyone interested in helping out/reviewing processes? even if it
>> is just some encouragement, it is all greatly appreciated.
>>
>> Drop me an email if you want access to the current setup and help
>> me review the different software for the bitcoin workflow
>> process.
>>
>> cheers,
>>
>> steve
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT
BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb
M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP
y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/
xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI
bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4=
=W5xI
-----END PGP SIGNATURE-----
|
