Preamble

Ever since the dot com bust in 2000, software testing has received considerable attention partly because it became one of the top three key result areas (KRA) for the chief technology officer or the chief information officer of enterprises using technology as an enabler for business. As a result, testing has been partly unbundled from the IT stack of outsourcing. It still remains partly even after a decade, since some of the test types or levels are still bundled with development; while a few are unbundled and handled by the enterprise itself or by third party testing vendors.

Current status of Testing and Measurement

There are many influencing factors in this. It is vital to first understand the state of testing before one begins to discuss measurement.

Status of Testing

The practice has significantly matured in the last decade. The enterprises and the independent software vendors (ISV), the prime customer category, are recognising this. This is evident from their buying preferences shifting towards specialist test vendors from conventional test vendors. The status may be looked at from the following perspective:

Organising- This refers to the structure by which the various test types or levels are distributed across the various players in a way that would provide the best results for the enterprise. If one considers the conventional V model, the various test levels listed there are split across the relevant player- developer, integrator and enterprise.

Though in some instances, there are duplication of coverage and effort, in many cases, they are covered in the most optimal fashion.

Management

It is sad that record and management of tests, based on the model of organising followed, rest with the relevant player and are scarcely integrated to lend a one view of the entire chain. Even in this, the management begins once the cases are ready and not during the process of its construction. There is definitely more maturity in managing defects than the rest of the chain.

Process

Like in development, the maturity of the vendor is higher than the enterprise in testing too. There are more test vendors who have aligned their process to the established standards such as ITIL, TMM or T-Plan or variants derived from them than the enterprise themselves. The practice being such that, testing even when staffed with outsourced vendors,  is delivered within the enterprise and therefore having the need to adhere to their practices which has not sufficiently matured.

Methodology

Most places are still operating with the waterfall methodology. The adoption of agile methodology at enterprise level is still low while it is gaining some traction at the independent software vendor (ISV) segment. Even in this, some test types are done using agile while most are done with waterfall methodology.

Techniques

The testing is still person driven than by techniques. The level of adoption of combinatorial, model based, pair-wise or other techniques are still low. Some tools have emerged dealing with one or more of these techniques and yet the adoption has not caught on. Unless and until these tools are integrated with the available architecture, it is not going to be pervasive.

Principles

The application of test principles such as boundary value, equivalence partitioning and cause and effect are dependent on the tester. Though there are some design tools that facilitate this, they do not cover all test types and their adoption is quite low.

Status of measurement

At this time, measurement occurs at two end of the spectrum. On the one end, is the product or code walkthrough where the number of defects- essentially on the logic, construct and adherence to internal standards, serve as the factors, are measured in relation to the lines of code (LOC). At the other end, the defects during acceptance tests or leakage in to production are measured both with reference to the number of test cases or other metric of tests.

Each of these defects, in turn, would be graded by severity or other parameters in to four or five classes.

There is hardly any measurement of defects slippage across test levels or types. This, as stated earlier, arises due to the fact they are performed by different parties and there is no platform in which it is consolidated to get a one view on that.

Technology trends

It is imperative that this is set across the three levels- IT, testing and measurement. This would enable an assessment of whether they are coordinated or disconnected, and even if they are coordinated, whether there is a lag with regard to those.

IT

It is a reality that the world, especially the commercial world, is slowly but surely shifting towards an integrated electronic system. This system does not conform to the conventional pattern and therefore you notice that the play of devices, whether it be the front office channels to connect with the customer or the market as it is or the interface to third party system in the ecosystem or with the outsourced service providers, is evident.

Online has already become ubiquitous and the cloud technology for application development, service delivery across the spectrum of the life cycle or for hosted solution, though at an infancy, has already caught the attention and interest of the early adopters.

Testing

The trends can be addressed across the following elements:

Virtualisation

This is meant to convey the means by which the conventional merchandise whether called as hardware, peripherals, devices or any such name are made available as a software as well as the availability of many of the basic software whether it be operating systems, drivers, utilities and others are available in a single application.

This has enabled conduct of various test types possible at the integrator, developer or solution provider end without having the need to push that to the enterprise end for testing.

Tools

There is an exponential growth of availability of tools across the various components of the life cycle. They can be conveniently grouped under the following:

  • Specification: They address the non-code elements in the life cycle excepting the testing part. Typically, requirements, functional and technical specification come under this category. They eliminate the gaps in the transition across transition across these stages and in some instances have a linkage to the solution as well so that the definition becomes tight and not to have any loose ends.

Coding: These tools check for adherence to set criteria on the measurement of the quality of codes and also for any logical and construct errors. At this time there are a number of tools available for a specific language such as Java or C++ or Cobol than across a spectrum. They accomplish three important things:

  1. Simulate a code walkthrough and detect errors
  2. Grade or rank the code in terms of defined specification so that the standard norms for acceptance of codes could be set and managed; and
  3. Identify the quality of codes generated by a coder in and across projects so that the fitment of the skill level can be measured and monitored.


Design: These tools come with the model, principles and methodologies set for a test type and can therefore ensure a consistent quality of test generation against a specification. They also carry traceability across the stages in the life cycle. They accomplish three things:

  1. Ensure that the coverage is comprehensive;
  2. Consistent application of principles and techniques;
  3. Comprehension of test cases through structured and relevant methodologies.


  • Stubs and harness: These come up virtualisation of the covered elements and include support for various test types through bi-way integration with the application under test. They facilitate the various test types and can locate integration issues.

Data: They relate to data generation or masking. These help in test data generation as well as data masking of migrated data to be used for testing.

Measurement

The broad developments are:

  • Defects: These define the stages or phases of the life cycle, including the requirements, design, build, test, user acceptance and production and track the metrics. Additionally, they help to establish the standard and improve the phase where there are more defects so that the overall quality improves.
  • Coverage: This is to ensure that all factors relating to the requirements and all principles and techniques have been correctly applied.
  • Efficiency: This evaluates on two counts- number for the coverage and discovery of the maximum number of defects. This automatically puts pressure on the adoption of the relevant techniques and methodologies to make the best coverage with minimal cases.
  • Execution: This is to record the efficiency of execution. This would measure the improvement in productivity across cycles and even in a cycle through learning; makes a factor for defects.
  • Metric: Set the list of metrics to be measured, such as for defects, identification, rejection, injection resolution, slippage and fixing. This enables to locate where the gap is. For example, a defect is logged and is subsequently rejected through clarification that it is not a defect, the issue is with the logger and not the provider.

Challenges

The critical ones are:

Lack of platform: Most global outsourced application developers have a platform that integrates with at least their delivery centres in delivering a seamless development to the customer. At this time, there is no such platform for testing and this is an urgent requirement.

Metrics: The developments indicated in this article are based on thoughts circulating in the profession than empirical evidence of their usage. Unless, there is a greater usage of those metrics, the quality would be indifferent based on the players involved.

Concepts and tools:  While there is a general agreement on the status of the issues in the life cycle, there is very little effort made to adopt better techniques, principles and methodologies; this in spite of the spate of tools that have come to support that. There is a universal unfounded view that any of these would entail more effort and cost.

Heterogeneity : There are various generation of application and equipment with varying shades of adherence to standards and protocols that it becomes difficult to have a clear workable solution addressing all of them.

Future course

There are three distinct developments that augur well:

The concept of creating centres of excellence for quality/testing has really arrived; more progressive customers are moving in to that fold. It means, the environment for a life cycle quality management would emerge sooner.

A number of specialist firms dealing with testing or quality improvement have emerged and their clout is growing as is their count. It really means that a capable pool would be in place to meet the demands for a total quality revolution.

Platforms are emerging integrating the various test types, connectors to equipment and straddling the entire life cycle. At this time, the suite comprises of components bought and integrated or capable of being integrated in to one. It is likely that they would be revamped in a single platform since the time for that had come.