Parallel Apps Enable
Continuous 100% Coverage Testing
Parallel System Verification as an alternative to traditional testing.
November 05, 2020
Authors:
Brian Bernknopf – QA Consultants
Michael Cation – Fractal
Often, the “long pole in the tent” in traditional software testing is the duration it takes to properly set up test data and establish a scenario to be tested. Often, subtle but important changes to the code base and application under test are not able to be tested until later in the customer or workflow lifecycle, often transpiring over logical date changes, triggering backend jobs, etc. This is an intensive and repeated process that while often automated, still increases the “Time to Test” measure from the moment code and environments are delivered until the specific test cases can be executed.
Fractal Computing™ architectures and methodologies, which are modern approachs to software development focused on performance and simplicity and have multiple industry uses beyond QA, uniquely provides an alternative use for the creation of parallel test systems.
This approach, currently in use in the energy utility markets allows for rapid validation of system changes without the need for arduous and complex test cycles and scenarios, but rather, a completed end to end validation of core system processes with mathematical proof points of system correctness.
100% Test Coverage
With Fractal Computing™
A New Testing Paradigm
When QA Consultants beings a system QA assessment, we are often looking for the right balance between cost of quality and speed to market. Of course, doing that without a sacrifice to quality beyond the allowable risk levels of a particular system. When applicable, if it is a high transactional system or high volume processing product, we are also evaluating if a new emerging technique in QA, “Parallel System Verification”, is applicable. This process allows us to mathematically ‘prove’ that a system is performing as expected. Not by running hypothetical test scenarios, but by running real data, in high volume, with real expected results.
The higher the volume, the better the math.
In short, if two systems with the same data input produce the same results, the is high probability they are performing correctly.
Certainly, the system under test must be 100% separated from the parallel test system. And further, that parallel test system must also be validated against a core data set to be accurate.
Using new technologies such as Fractal Computing™ architectures and methodologies allows us to rapidly stand up a parallel system to deliver 100%, continuous test coverage as a viable alternative.
We are currently seeing that continuous testing with 100% test coverage has delivered significant benefits. Among these is the ability to essentially eliminate transactional production errors. Another benefit is a dramatic cost reduction in ongoing QA costs while increasing the surface area for testing critical new app features.
The Power of Parallel Execution: Continuous Parallel Testing and Coverage
One of the primary advantages of using Fractal Computing in the development of enterprise software is its ability to rapidly process significant amounts of data. Our software partners use this feature to run entire days, months, or years of input data through the system under test, as a parallel system. Here, two systems, with different code bases, are performing what should be identical tasks. The independent QA as a Service vendor, QA Consultants, measures every transaction from both systems to determine, line by line if they are identical.
This is a two-step process that first validates that the parallel application accurately matches the existing legacy system. Through this process multiple years of data are run and reconciled for correctness.
Once the parallel Fractal application is validated, the second step is to make the suggested software changes to both the legacy system and the parallel Fractal system. Then the data sets are re-run looking for expected and unexpected outcomes. The outcome differences are analyzed and confirmed for defects.
Our experience has shown a 100% accuracy rate for the Fractal parallel system in identifying production defects when there is a mismatch between the two systems.
When calculations from two independent systems, each with different code bases, reach the same conclusion, the probability that the final product – i.e. a utility bill -- is correct, approaches 100%.
The volume of data allows for mathematical certainty of system sameness.
Eliminate Testing Through Validation: Testing a System of Record as New Features are Added
As enhancements to the system of record become available, the Fractal Computing approach allows the QA team to build them into a version of the parallel system.
Because Fractal technology enables complex applications to be modified, configured, and delivered rapidly, it is practical to construct parallel systems for each set of feature enhancements. With this approach, every line item is reconciled for every calculation in real time. The system of record can be checked against an established parallel system and also reconciled against one or more additional parallel systems to ensure that all of them agree.
This process is continuous. The parallel systems can be validated 24/7. Across years of historical data.
When an anomaly appears, there is immediate feedback. As each new system is validated and then moved over to the role of “primary parallel system”, it is possible to introduce significant updates into a production system and test them at the level of every line item, on every bill or other output, for every customer for 100% test coverage of the output of the updated system.
Not 100% hypothetical coverage, 100% historical coverage. Complicate “edge” scenarios or complex processes are all accounted for. There is ZERO QA time spent on the creation of test data.
Further, the comparison of expected results to actual results also takes place in a Fractal Computing comparison solution allowing for rapid validation.
Since Fractal Computing applications are more efficient than traditional applications, the hardware requirements are minimal. This makes it economical to run multiple systems in parallel without breaking the program budget. It also makes it possible to do extensive retrospective testing – for example, running three years of transactions through each of the parallel systems on a regular basis to find anomalies.
It also opens the possibilities up for fraud detection and hypothetical scenario modeling.
Summary
Fractal systems for enterprise applications have been in production for 7 years. They have been used for 100% continuous testing with great success.
We have seized this technology and created a new way to test and validate transactional oriented applications on an ongoing\continuous basis to deliver a higher level of quality than what has been previously possible and at a speed that allows for rapid system deployments with low risk.
QA Consultants and Fractal believe that this unique approach to quality, validation, and velocity is a significant mindset change for IT organizations, and we further believe it is ground-breaking. It’s current use means that we think there would be some relevance in considering this approach as new systems come online or legacy systems go through refresh and modernization.
There are many options for ETL validation as well as traditional system maintenance and upgrades.