Smoke and Sanity Testing: A Definitive Guide [2019]

There is chaos in testers when it comes to the difference between Smoke and Sanity testing. It is also a very frequently asked Interview Question as well. This article will give a clear idea about:

• What is Build and Release in Software Testing?
• What is Smoke and Sanity Testing (with Example)?
• Who does Smoke testing do? When to perform Smoke testing along with Smoke Testing Techniques?
• Why and When to perform Sanity testing?
• Difference between Smoke and Sanity testing
• Importance of Smoke and Sanity testing for the Software Industry

 

 

Smoke and Sanity Testing

Smoke and Sanity testing very much go hand in hand with Build & Release, to better understand these, it is suggested to understand what is Build and Release and how these relate with the Smoke & Sanity Test.

 

What is Build in Software Testing?

In layman’s term, Build is the construction of something that has a visible and tangible result. Let’s try to understand this further with the help of an example.

If we are building a house, then –

• Firstly, we will construct the foundation and walls of the building; this is the first build.
• Then we will fix the defects in the walls and construct the ceiling followed by whitewash and woodwork etc. These are the second and third builds simultaneously.

This process continues until the entire completion of the whole building.

Similarly, in the software industry – A build is a growing software application, where the first build will start from scratch and will incorporate some features. It is called Build 1. The errors in the first build are corrected, and some new features added. It is called Build 2. This process continues until the software is fully developed and ready for use.

 

What is Release in Software Testing?

A Release is the final version of an application for customers, which is generally an outcome after multiple builds. A build, when tested and certified by the software testing team, is provided to the clients as “Release.” The Build is the part of an application while the release is a fully developed application.

Therefore, a fully developed software that is ready to be delivered to the customer after all the testing is called a Release. Also, to note that software can also have multiple releases. If any set of new functionalities or features gets added to the already produced software application, then that is also called a Release. That is why software applications have many different versions. Each version is a release.

 

 

What is Smoke testing?

Smoke tests are preliminary tests to reveal simple faults that are serious enough to reject a possible software version. These are the first tests performed in the build, and all the other types of tests follow it.

It is the process that comprises of a non-exhaustive set of tests which give confidence that the essential functions of an application/software are working fine. The outcome of this testing is used to decide if a build is stable enough to proceed with further testing because of which it is also known as Build Verification Testing.

The term “Smoke Testing” is derived from hardware testing; it is performed in the initial stage of hardware testing to verify fire or smoke is not ignited when the hardware power is turned on.

Smoke Testing is performed before accepting any build to the test environment, and generally, these are initial builds. It checks for major Priority1 (P1) issues. If the build is stable and no major P1 issues exist, testers accept the build and then perform later testing like Functional or Regression testing on the same build.

Smoke testing is carried out to find out whether significant functionalities of the system/application are working or not. If the main features break, then there is no point in carrying out further testing, thereby saving time and effort.

Let’s try to understand it with the help of an example:

 

Assume an example – An application for booking a radio taxi 

Some of the features/modules of this application can be:

• Log in to the application
• Book a cab
• Change password
• Display traveler name and contact number
• Make a payment

Now assume the developer develops the radio taxi software version 1 / Build 1 (40% complete), checks the application at their level and the build is given to the QA to test.

As a QA, we

• Prepared test cases for each module to test the application.
• Along with that, we also prepared one Smoke Test, which is End-2-End test, where user Login to the application, Book a cab and Make a payment. 

Scenario 1 – Login functionality breaks

As you are not able to log in so you will not be able to test most of the modules like Booking a cab or Making a Payment. Any attempt to execute test cases will be sheer wastage of time and effort. In this case, it is wise to Reject the Build.

Scenario 2 – Login functionality works fine, but you can log in with the wrong password.

It is a big security issue and needs reporting, but since it allows you to log in to the system and test the rest of the functionalities, so it is safe to Accept the Build.

This testing of critical features is known as Smoke Testing.

 

Who Performs Smoke testing?

The Quality Control team usually performs smoke tests, but in certain situations, the Development team can also perform it. The development team verifies the stability of the build before sending it to QA for further testing.

Also, this testing is carried out by testers on an environment other than the test environment when the last build is ready to be deployed as a release to make sure all the P1 issues have been fixed or happy path scenarios are working positively.

 

When to perform Smoke testing?

The smoke test is done immediately after the implementation of the build followed by other tests such as Functional tests (to test newly added functions), Regression tests and User Acceptance tests, etc. It is preliminary testing.

 

Smoke Testing Techniques 

1. Manual approach: Here, manual execution of the Smoke test cases takes place. Primarily the manual method is used when the product gets developed from scratch, and it is unstable. The scenario will be costly because it will take a lot of effort to execute the test scripts in each build.
2. Automation approach: Here, the Smoke test cases are automated and run with the help of automation tools. In some cases, the smoke scripts may be integrated with automated build creation tools such as Jenkins, so that each time a new compilation gets implemented, the smoke suite automatically starts the execution without manual intervention and without losing time.
3. Hybrid approach: The hybrid approach is a combination of manual and automation methods; we use both manually designed test cases and automation tools in the Hybrid approach.

 

 

What is Sanity Testing?

It is the process of testing the fixed bugs in an updated or new build. After receiving a build of the software, with small changes in the code, or functionality, the Sanity test is performed to check that the errors are corrected and no more problems occur due to these changes in the same module or feature.

Sanity testing gets generally performed on stable builds:

• Initial builds of software are relatively unstable; therefore, they undergo Smoke testing to verify whether critical functionalities are working fine or not. If Build surpasses Smoke testing, it further undergoes System and/or Regression testing. 
• After going through multiple rounds of Regression tests, the build gets relatively stable. If there are any bug fixes or new features introduced to this “stable build,” then Sanity testing is performed to re-test bug fixture. If that goes fine, then proper Regression testing or further testing is performed.

Let us continue with the same example of the radio taxi application to have a better understanding of Sanity testing.

In Build 7, few new features and bug fixes have been carried out in radio taxi application. Here are its details:-

New features – Build 7

• Drivers feedback
• GPS tracking of the ride
• Deals and offers for the ride

Bug Fixes – Build 7

• Book a Cab
• Make a Payment

QA will be retesting the particular features for which bug is fixed to ensure the given features are functioning smoothly. It is called Sanity testing.

Note: The strategy of Smoke and Sanity is used in different organizations interchangeably. Every organization uses these as per their benefits. So, in the above scenario, developer or QA can also carry out Smoke testing on all newly introduced features of Build 7 – Drivers Feedback, GPS tracking, and Deals & Offers. 

It is somewhere more significant than the Retesting and less than the Regression testing, as retesting focuses only on the defect which gets fixed while Regression testing focuses on all the features.

 

Who Performs Sanity testing?

Sanity testing is carried out by testers only.

 

Why and When to perform Sanity testing?

The purpose of this test is to determine that the proposed changes or functionality are working as expected. If the sanity test fails, the build is rejected by the testing team to save time and money. It is a subset of Monkey testing or Exploratory testing. In Monkey and Exploratory testing, random inputs are given to the system to perform specialized and in-depth testing. This testing does not follow a strictly defined process or test scripts.

A Sanity test is a narrow regression test that focuses on specific areas of functionality. This test is used to determine that the application still works right after the bug has been fixed or new feature added.

 

Sanity Testing Techniques: 

There is not any specialized technique that gets employed in Sanity testing. It is performed generally without test scripts or test cases, but manual performing of all the tests happens.

 

 

Difference between Smoke and Sanity testing

Both the terms Smoke and Sanity are used interchangeably to a great extent in the IT industry, but there are also a few critical differences between them, as explained below:

 

Feature	Smoke Testing	Sanity Testing
Test Coverage	Smoke testing is a shallow and broad approach.	Sanity testing is usually a narrow and in-depth approach.
Motive	It is designated to touch every part of the application quickly.	It basically emphasis on the small sections of the application and check whether it is working correctly after a minor change.
Technique	The test cases for smoke testing can be either manual or automated or sometimes a hybrid approach.	A sanity test is performed generally without test scripts or test cases but manually.
Performed By	The application developers or QA team perform this testing.	The QA team usually performs sanity.
Subset	It can be a subset of acceptance testing or regression testing.	Sanity testing is more of a Monkey and Exploratory testing.
Documentation	Properly documented. Smoke testing is usually managed separately as a Smoke Test Suite.	No proper documentation is available. Sanity testing is generally performed with the experience only and focus on the defect and nearby area.

 

 

Importance in the Software Industry:

Similarities	Explanation
Saves Time	Smoke and Sanity tests are efforts to save time by quickly determining if an application is working correctly or not. Also, it ensures that the compilation is eligible for rigorous testing.
Save cost	The saving of time and effort leads to saving the cost of testing the application. Employing the correct approach and eliminating the mistakes in the early stage, lowers the effort and time, thus minimizing the damage.
Integration risk	We perform end-to-end tests on each build so that functionality-based problems get discovered earlier. So, the risk of having integration issues minimizes.
Quality improvement	Here, the main problems are detected and corrected much earlier in the software test cycle, which increases the quality of the software.
Evaluation of progress	It is easier for project managers to assess the progress of development. Since with each compilation, we certify that the product from end to end is working correctly after the addition of new features.