Plenary Lecture

Use of Orthogonal Arrays and Design of Experiments via Taguchi Methods in Software Testing

Assistant Professor Ljubomir Lazic
State University of Novi Pazar

Abstract: Software development project employs some Quality Control (QC) process to detect and remove defects. The final quality of the delivered software depends on the effort spent on all the QC stages. Given a quality goal, different combinations of efforts for the different QC stages may lead to the same goal. For the quality of the final software we use the commonly used measure of delivered defect density - the number of defects present in the final product normalized by the size of the product. One of the main objectives of a project is to achieve the desired quality goal with least amount of resources. Using defects as the defining metric for quality, we can view the process of a project as comprising of defect injection and removal stages. There are some stages like the requirements, design and coding, in which defects are injected. These defects are removed in various QC stages. A QC stage can be characterized by the defect removal rate of that stage. There can be many possible combinations of defect removal rates for the different QC stages that can achieve the same overall quality goal. The different combinations will have different implications on the total QC effort. Clearly, for a process designer or a project manager. A key problem is to select the amount of effort to be spent in each QC stage such that the desired quality goal is met with the minimum cost. We propose a model i.e. Optimal SQM for the cost of QC process and then view the resource allocation among different QC stages as an optimization problem. Software testing consumes 30-70% of the development resources; however, shipped products may still have many residual faults resulting in low reliability, high usage cost, and high maintenance cost. For software testing process optimization we apply Orthogonal Array-Based Robust Testing (OART) and Design of Experiments via Taguchi method.
To solve the problem of great number of test cases, and to force the configuration testing to be effective, combinatorial testing is proposed, using an OART Strategy as a systematic, statistical way of testing pair-wise interactions. This combinatorial approach to software testing uses models to generate a minimal number of test inputs so that selected combinations of input values are covered. The OART method can simultaneously reduce testing costs, product introduction delays, and faults going to the field by generating test cases that are more efficient and thorough in finding faults. Often the result is a 50% reduction in the number of tests and detection of more faults.
An advantage of the Taguchi method application in Software Testing is that it emphasizes a mean performance characteristic (Defect Removal Efficiency of a QC stage and cost of software Quality) value close to the target value rather than a value within certain specification limits, thus improving the product quality. Additionally, Taguchi's method for experimental design is straightforward and easy to apply to many engineering situations, making it a powerful yet simple tool. It can be used to quickly narrow down the scope of a research project or to identify problems in a manufacturing process from data already in existence. Also, the Taguchi method allows for the analysis of many different parameters without a prohibitively high amount of experimentation. For example, a process with 8 variables, each with 3 states, would require 6561 (38) experiments to test all variables. However using Taguchi's orthogonal arrays, only 18 experiments are necessary, or less than 0.3% of the original number of experiments. In this way, it allows for the identification of key parameters that have the most effect on the performance characteristic value so that further experimentation on these parameters can be performed and the parameters that have little effect can be ignored.
We give examples in this paper to show how optimal allocation of effort to each QC stage can be done to achieve a goal with minimum total effort. We also discuss how the model parameters can be obtained from process performance data that is often collected by organizations. We have also built a software that, given the project parameters, gives the optimal resource allocation schedule for any given overall quality goal.

Brief Biography of the Speaker: Ljubomir Lazic graduated from the University Electrical Engineering School, Serbia in 1979. In the 1980s he worked as Embedded Software and Hardware Test Engineer, Test Manager and Senior Researcher at Military Technical Testing Center (MTTC). He was a member of MTTC‘s Scientific Council, Belgrade, Former Yugoslavia and ICT Military Expert at Yugoslav Army Headquarters. Also in the 1990s, he has been working for a local telecommunications SIEMENS Company in Belgrade as Chief Engineer in Sales & Marketing Division, Installation & Commissioning Manager and Maintenance Manager. He continued to serve industry in a variety of roles, including consulting, executive education, and expert testimony. He is docent in Computer Science at the State University of Novi Pazar, Serbia (2007- current), and docent in Software Engineering, University Union of Belgrade (2006-2010). His research interests are in Software Engineering, Software Project Management, Software Testing, Human Computer Interaction, and Component Based Engineering. Current research interests, doing as a Project leader, in two projectssupported in part by the Ministry of Science and Technological Development of the Republic of Serbia under Grant No. TR-1318 (2008-2011) and TR-35026 (2011-2014) are: Optimal software project management, Software Metrics, Effort Estimation Modeling etc. He is author of about 90 papers published in international journals and conference proceedings, invited speaker (Keynote speaker at QA&TEST 2010, 9th International Conference on Software QA and Testing on Embedded Systems, 27-28-29 October - Bilbao, Spain,2010) and book chapters.

Bulletin Board


The Conference Program is online.

The Conference Guide is online.

The paper submission deadline has expired. Please choose a future conference to submit your paper.

Plenary Speakers

WSEAS Main Site

Publication Ethics and Malpractice Statement