[asterisk-users] Testing Framework

[Matt Riddell]

Hi,

So, now that we've all complained about the state of testing of Open
Source versions of Asterisk, lets do something about it.

I propose we start with a list of things that we think should be tested
in Asterisk, and means to test them.

Maybe we could run certain tests based on the changes between minor
versions?

Anyway lets start.

Call Volumes

1) Call volume up to x channels from SIP to SIP (i.e. sipp)
2) Call volume up to x channels from IAX2 to SIP
3) Call volume up to x channels from IAX2 to IAX2

Application testing

4) Connect x calls between techs to Meetme (leave running for 1 hour)
5) Connect x concurrent calls to VoiceMail

Call Centre Testing

6) Send x calls to a queue with no agents in it, leave them holding for
x minutes
7) Run x calls against AMD connected to recorded known good files

Recording

8) Run x calls recording simultaneously from an automatically generated
call, play ulaw/alaw - compare outputs.

You get the idea.

If people can add to this list, I can start making a few scripts and
programs that will test them (as I'm sure others can).

If we end up with a complete list, I'm sure some of our individual QA
departments can take the responsibility for certain items.

The call volume ones are obviously going to either need a live person to
dial in at volume and check everything is ok, or a recording which can
later be checked.

I'm of the opinion that the majority of tests should test individual
components, but that we should also form some "Application Type"
frameworks so that we can test integration between Asterisk apps.

Any takers? Add to the list? If there is something you believe is
mission critical to your business, write up a test case for it, and
we'll all try to code something that can run automatically to test it.

If we try and keep to ANSI C for the testing apps, Digium should be able
to run them on their multi platform machines as well.

Should these tests be added to Asterisk-Addons or maintained outside of
the tree?

Anyway, what do you think? Feasible? I already have a few tests here and
I'm sure others have a few too. Lets put them all together and get a
framework going.

- --
Kind Regards,

Matt Riddell
Director
_______________________________________________

http://www.venturevoip.com (Great new VoIP end to end solution)
http://www.venturevoip.com/news.php (Daily Asterisk News - html)
http://feeds.venturevoip.com/AsteriskNews (Daily Asterisk News - rss)

[Russell Bryant]

Post by Matt Riddell
Should these tests be added to Asterisk-Addons or maintained outside of
the tree?

If people start writing test utilities, I would be happy to host them in a
subversion repository. Depending on the size of this stuff, it could probably
go into the main Asterisk repository. We'll see where things go ...

[Atis]

Post by Russell Bryant

Post by Matt Riddell
Should these tests be added to Asterisk-Addons or maintained outside of
the tree?

If people start writing test utilities, I would be happy to host them in a
subversion repository. Depending on the size of this stuff, it could probably
go into the main Asterisk repository. We'll see where things go ...

Our company have a scriptable testing utility (it's quite far from
being simple to use), but we could share it as basis for testing
framework or at least - as example for inspiration. I just talked to
management, and this week it will be discussed. I'll write you in
case of success.

Regards,
Atis

[dave cantera]

matt,
are you looking for unit testing of the * components or systems testing,
testing the finished product? or both?
I think you are onto something here... I hope it takes root. I would
say put it in the addons. it would be Great if digium takes it up. it
is a smart move for them to foster, cajole, nudge, and support it.
call volume I would leave to others as different processors, O/S,
builds, kernel versions, and configurations will have too many variables.

I was playing with the idea of monitoring multiple * systems. perhaps
we can start out with testing the components and then migrate the
project (future) to one pbx monitor the other. we will need scripts to
initiate some action, config to make some measurements, the scripts to
gather the results into a nice neat little summary report. you will
want to take the human aspect out of the picture as much as possible.
for example:

on pbx A

* create a recording in multiple formats .gsm, .wav, etc.
* initiate a script to generate 5,10, or 25 calls to pbx B and
play the file

on pbx B

* pbx B gets the calls, records them,
* copy the recordings from pbx A to pbx B (or have that already
done)
* have a wave analyzer compare the recordings to the original
files (you know I won't be writing that program! :)
* report on anomalies

*call
* *Technology
* *recording
delta
*
1
Zap Provider 1
2%
2
VoIP Provider 2
5%
3
VoIP Provider 2
15%
...
VoIP Provider 3
...


let me know what you think!
daveC

Hash: SHA1
Hi,
So, now that we've all complained about the state of testing of Open
Source versions of Asterisk, lets do something about it.
I propose we start with a list of things that we think should be tested
in Asterisk, and means to test them.
Maybe we could run certain tests based on the changes between minor
versions?
Anyway lets start.
Call Volumes
1) Call volume up to x channels from SIP to SIP (i.e. sipp)
2) Call volume up to x channels from IAX2 to SIP
3) Call volume up to x channels from IAX2 to IAX2
Application testing
4) Connect x calls between techs to Meetme (leave running for 1 hour)
5) Connect x concurrent calls to VoiceMail
Call Centre Testing
6) Send x calls to a queue with no agents in it, leave them holding for
x minutes
7) Run x calls against AMD connected to recorded known good files
Recording
8) Run x calls recording simultaneously from an automatically generated
call, play ulaw/alaw - compare outputs.
You get the idea.
If people can add to this list, I can start making a few scripts and
programs that will test them (as I'm sure others can).
If we end up with a complete list, I'm sure some of our individual QA
departments can take the responsibility for certain items.
The call volume ones are obviously going to either need a live person to
dial in at volume and check everything is ok, or a recording which can
later be checked.
I'm of the opinion that the majority of tests should test individual
components, but that we should also form some "Application Type"
frameworks so that we can test integration between Asterisk apps.
Any takers? Add to the list? If there is something you believe is
mission critical to your business, write up a test case for it, and
we'll all try to code something that can run automatically to test it.
If we try and keep to ANSI C for the testing apps, Digium should be able
to run them on their multi platform machines as well.
Should these tests be added to Asterisk-Addons or maintained outside of
the tree?
Anyway, what do you think? Feasible? I already have a few tests here and
I'm sure others have a few too. Lets put them all together and get a
framework going.
- --
Kind Regards,
Matt Riddell
Director
_______________________________________________
http://www.venturevoip.com (Great new VoIP end to end solution)
http://www.venturevoip.com/news.php (Daily Asterisk News - html)
http://feeds.venturevoip.com/AsteriskNews (Daily Asterisk News - rss)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG1yKhDQNt8rg0Kp4RAv5UAJ48tW28T5lWCQIPTwVimyvlhEPJowCgpnE6
OF3L2M/6Hc+YBNL1NFx6dzA=
=OXNn
-----END PGP SIGNATURE-----
_______________________________________________
--Bandwidth and Colocation Provided by http://www.api-digital.com--
asterisk-users mailing list
http://lists.digium.com/mailman/listinfo/asterisk-users

[Hariharan Veerappan]

I think the testing frame work includes both the components
and system testing.
I wish to add some more test even though all giants may aware,
since i wish to do some contribution to asterisk what ever i can.

i am plannig for the framework and addon as given below, expecting
techies advise in this,
Testing frame work may contain testing internally and externally,
i mean to say internally, some client may stick into the problem of
voice quality, when more phones on the PBX, that time, we may not
leave the system from the network, their call may get interrupted,
that time internal testing daemon will work for the performance analysis
till the lelvel of without affecting present calls.

externally means PBX system on production for performance analysis,
as per the test case started in the mail.

internal and external testing may be configurable at the run time, through
some verbose like variable configurable at run time, addon should have the
capability
of releasing the testing call bandwidth, whenever PBX gets new call, this
might be simple
example.

procesding with Testing frame work requirements,
call on volume,
1. SIP - SIP, multiple SIP clients support, through which we can either
direct the call for testing to another client registered in testing frame
work,
or return back to the different client registered in the same testing frame
work,
when the call incoming and outgoing call are handled in single framework
point,
wave analysis also cane be done with the script and performance can be
easily
evaluated.

On production performance testing, by connecting multiple testing framework
point to the PBX,
having the sending files in all the frame work, analysis and performance
evaluation can be
done very easily,

i also think that once it is done for SIP with compatiblity of like
channel driver, we can adapt IAX2, anything we want.

i think this type of testing would make the system stable and provide
good support on system on running also.

Hariharan.V.
R&D Engineer,
NEEVEE Technologies,

Post by dave cantera
matt,
are you looking for unit testing of the * components or systems testing,
testing the finished product? or both?
I think you are onto something here... I hope it takes root. I would
say put it in the addons. it would be Great if digium takes it up. it
is a smart move for them to foster, cajole, nudge, and support it.
call volume I would leave to others as different processors, O/S,
builds, kernel versions, and configurations will have too many variables.
I was playing with the idea of monitoring multiple * systems. perhaps
we can start out with testing the components and then migrate the
project (future) to one pbx monitor the other. we will need scripts to
initiate some action, config to make some measurements, the scripts to
gather the results into a nice neat little summary report. you will
want to take the human aspect out of the picture as much as possible.
on pbx A
* create a recording in multiple formats .gsm, .wav, etc.
* initiate a script to generate 5,10, or 25 calls to pbx B and
play the file
on pbx B
* pbx B gets the calls, records them,
* copy the recordings from pbx A to pbx B (or have that already
done)
* have a wave analyzer compare the recordings to the original
files (you know I won't be writing that program! :)
* report on anomalies
*call
* *Technology
* *recording
delta
*
1
Zap Provider 1
2%
2
VoIP Provider 2
5%
3
VoIP Provider 2
15%
...
VoIP Provider 3
...
let me know what you think!
daveC

Hash: SHA1
Hi,
So, now that we've all complained about the state of testing of Open
Source versions of Asterisk, lets do something about it.
I propose we start with a list of things that we think should be tested
in Asterisk, and means to test them.
Maybe we could run certain tests based on the changes between minor
versions?
Anyway lets start.
Call Volumes
1) Call volume up to x channels from SIP to SIP (i.e. sipp)
2) Call volume up to x channels from IAX2 to SIP
3) Call volume up to x channels from IAX2 to IAX2
Application testing
4) Connect x calls between techs to Meetme (leave running for 1 hour)
5) Connect x concurrent calls to VoiceMail
Call Centre Testing
6) Send x calls to a queue with no agents in it, leave them holding for
x minutes
7) Run x calls against AMD connected to recorded known good files
Recording
8) Run x calls recording simultaneously from an automatically generated
call, play ulaw/alaw - compare outputs.
You get the idea.
If people can add to this list, I can start making a few scripts and
programs that will test them (as I'm sure others can).
If we end up with a complete list, I'm sure some of our individual QA
departments can take the responsibility for certain items.
The call volume ones are obviously going to either need a live person to
dial in at volume and check everything is ok, or a recording which can
later be checked.
I'm of the opinion that the majority of tests should test individual
components, but that we should also form some "Application Type"
frameworks so that we can test integration between Asterisk apps.
Any takers? Add to the list? If there is something you believe is
mission critical to your business, write up a test case for it, and
we'll all try to code something that can run automatically to test it.
If we try and keep to ANSI C for the testing apps, Digium should be able
to run them on their multi platform machines as well.
Should these tests be added to Asterisk-Addons or maintained outside of
the tree?
Anyway, what do you think? Feasible? I already have a few tests here and
I'm sure others have a few too. Lets put them all together and get a
framework going.
- --
Kind Regards,
Matt Riddell
Director
_______________________________________________
http://www.venturevoip.com (Great new VoIP end to end solution)
http://www.venturevoip.com/news.php (Daily Asterisk News - html)
http://feeds.venturevoip.com/AsteriskNews (Daily Asterisk News - rss)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG1yKhDQNt8rg0Kp4RAv5UAJ48tW28T5lWCQIPTwVimyvlhEPJowCgpnE6
OF3L2M/6Hc+YBNL1NFx6dzA=
=OXNn
-----END PGP SIGNATURE-----
_______________________________________________
--Bandwidth and Colocation Provided by http://www.api-digital.com--
asterisk-users mailing list
http://lists.digium.com/mailman/listinfo/asterisk-users

--
My wife's sister is in California.
I should buy her a Videophone2008!
Truly, The Next Best Thing to Being There!
--
WorldWideVideoPhones.com
856.380.0894
_______________________________________________
--Bandwidth and Colocation Provided by http://www.api-digital.com--
asterisk-users mailing list
http://lists.digium.com/mailman/listinfo/asterisk-users

[Kyle Sexton]

Post by Matt Riddell
Hi,
I propose we start with a list of things that we think should be tested
in Asterisk, and means to test them.
Any takers? Add to the list? If there is something you believe is
mission critical to your business, write up a test case for it, and
we'll all try to code something that can run automatically to test it.

I think a good way to get new items on the testing list would be for
Digium/community to take a look at the bug tracker for released versions and ask
'Was there a way we could have caught this bug?'. Then add that test
case to the list. In my mind I would rather have too many tests than
not enough! :)