The Testcase Execution application can be found in the Test Lead's Desktop, Tester's Desktop and any other role's Desktop that has been given access to this application. Once Testcase EAS is set up and assignments are made, the Local Tree of the Testcase Execution application shows the appropriate test cycle and the phases/systems/sub-systems and their testcases. These show up for a particular user only if they have a testcase assigned to them anywhere in the tree.
Selecting a testcase in the list of assigned testcases shows its details in the lower panel.
The user can view the entire testcase, execute it and then update its Status. There are 4 test states - Pass, Fail, WIP (Work in Progress), and Blocked. On selecting a test status, a "Defect" button pops up if a Defect Tracking System has been selected in the Administration. Clicking on that will allow the creation of a new defect.
The Notes field is a free form text field that allows any type of execution related notes to be entered. The Alt ID field is the alternate ID that a testcase might have had in a previous testcase system (this comes in via the Testcase Import mechanism). Other columns such as Executed On, Executed By, Coverage and Attachment can also be displayed.
In the Attachment window, clicking on the "+" button opens a Browse dialog to allow for browsing and attaching a file. Clicking on the "-" button allows for the deletion of an attachment. You can add multiple attachments. Selecting an attachment (or multiple attachments with the "Shift" or "Control" keys) and clicking on the "Preview" button will open an Image Previewer that will allow scrolling through the various attachments, displaying previews of file formats that are recognized.
The execution status of multiple testcases can be changed at once. This is achieved by selecting multiple testcases either via the Shift or Control keys. On doing so, a Change multiple drop-down menu appears that will allow status changes to all those selected testcases at once.
Zephyr allows kicking off or launching of automation scripts on remote target machines. For this to happen, testcases must have been marked as Automated and details about the automation script must have been added to that testcase in the Testcase Creation process. It is also required to have a remote agent - ZBot - installed on the target machine in order to launch the automated script on that remote machine.
Automation testcases can be executed by clicking on the "E" button that shows up for a testcase that has been marked Automated. A single testscript can be executed or a batch of them can executed at one time.
On selecting a single automation script to run, the following window is presented. Here the Path field can be modified (just in case it was entered incorrectly in the Testcase Creation or needs to be changed for other reasons). Changes made to this path here will not be written back to the original testcase.
System level variables can be sent to the ZBot/ZIP for this particular execution run. These are replaced by the actual values before being handed off to the ZBot/ZIP. These are entered in the ZBot Parameter field but are not mandatory. The available system variables to send are:
@@uid@@ = User ID who has been assigned to execute the testcase
@@rid@@ = Release ID of this particular release
@@cid@@ = Cycle ID of this particular execution cycle
@@ccname@@ = Name of this particular execution cycle
@@cpid@@ = Cycle-Phase ID of this particular execution cycle
@@build@@ = Build information of this particular execution cycle
@@env@@ = Environment information of this particular execution cycle
@@nid@@ = ID of the tree node to which this testcase(s) belongs to
@@nname@@ = Name of the tree node to which this testcase(s) belongs to
Multiple system level parameters can be added to this field separated by a space. This is not a mandatory field.
The next thing to do is to select a ZBot from the drop-down list of available ZBots (or target machines) on which to run the automation script. You can also select a Status for the testcase to be shown while it is being executed (WIP or Work-in-Progress in this example).
At this point, clicking on Run executes the script on the remote machine. Real-time Execution Messages are shown right next to the script in the grid above and ZBot Real-time status is shown in the message box below.
Upon completion of test script execution, results can be updated either manually or automatically.
Multiple testcases can be selected in the Testcase Execution application (by holding the Shift or Control key) and then clicking on the "E" button next to the Execute Multiple option. Upon doing that, the following window is displayed with the batch of test scripts added to it. They can be re-ordered by selecting them and using the Move up and down arrows.
System level variables can be sent to the ZBot/ZIP for this particular execution run. These are replaced by the actual values before being handed off to the ZBot/ZIP. These are entered in the ZBot Parameter field but are not mandatory. The available system variables to send are:
@@uid@@ = User ID who has been assigned to execute the testcase
@@rid@@ = Release ID of this particular release
@@cid@@ = Cycle ID of this particular execution cycle
@@ccname@@ = Name of this particular execution cycle
@@cpid@@ = Cycle-Phase ID of this particular execution cycle
@@build@@ = Build information of this particular execution cycle
@@env@@ = Environment information of this particular execution cycle
@@nid@@ = ID of the tree node to which this testcase(s) belongs to
@@nname@@ = Name of the tree node to which this testcase(s) belongs to
Multiple system level parameters can be added to this field separated by a space. This is not a mandatory field.
The ZBot can then be selected, initial execution status can be set and potentially, the option to execute multiple scripts in parallel can be chosen. This last option kicks of multiple scripts without waiting for the previous one to complete.
The rest of the process is very similar to executing single test scripts as described above, including updating results manually or automatically.
Automation test scripts can be kicked off singly or in a batch as described above. Based on the type of script written, automation execution can be over in seconds or could days to complete. Either way, once the automation is complete, results can be updated into Zephyr in one of two ways.
1. Manually update or promote results
This is a good option to employ in the initial stages of Automation. If your automation scripts are prone to being disrupted by external factors (environment, data, unstable code etc.), you will have a lot more failures (potential false-positives or false-negatives). If you were to automatically update these results, they might not represent the true state of your execution. Remember, once results are updated in Zephyr, metrics are automatically calculated and this information is pushed out in real-time to Dashboards. So, in these scenarios, its best to watch and complete your automation execution and when you are comfortable with the results, manually update the status of that execution.
2. Automatically update results
When you are comfortable with the state of your automation execution and confident in its ability to run without being disrupted by external factors (environment, data, unstable code etc.), automatically updating results is great option. This can be done via the execution utility available in ZBots that use a webservices API call to update the results in Zephyr. Please refer to that section for more details.