Azure DevOps

Overview

Integrate Endtest with Azure DevOps Pipelines to trigger automated end-to-end tests on every build, publish results in Azure DevOps, and prevent deployments when tests fail.

Recommended approach: install the official Endtest extension

The easiest way to integrate Endtest with Azure DevOps is to install the official Endtest for Azure DevOps extension from the Visual Studio Marketplace:

Install Endtest for Azure DevOps

The extension adds the native EndtestRun@1 pipeline task and an Endtest service connection type.

It can:

• Trigger one Endtest execution or multiple executions started through an Endtest label
• Store the Endtest App ID and App Code in a secure Azure DevOps service connection
• Wait for all Endtest executions to finish
• Publish an Endtest summary in the pipeline run
• Publish execution-level JUnit results in the Azure DevOps Tests section
• Expose execution hashes, result links, and test totals as output variables
• Fail the pipeline when Endtest reports failures or errors

The extension source code is available on GitHub.

Install the extension

1. Open the Endtest for Azure DevOps Marketplace page.
2. Select Get it free.
3. Choose the Azure DevOps organization where you want to install the extension.
4. Confirm the installation.
5. Open your Azure DevOps project.

You need permission to install extensions in the selected Azure DevOps organization.

Create an Endtest service connection

After installing the extension:

1. Open your Azure DevOps project.
2. Go to Project settings.
3. Open Service connections.
4. Select New service connection.
5. Choose Endtest.
6. Set the Endtest URL to:

https://app.endtest.io/

7. Enter your Endtest App ID and App Code.
8. Give the connection a recognizable name, such as:

Endtest Production

9. Save the connection.

Using a service connection keeps the App Code out of your pipeline YAML. Credentials already included in the API request are still supported for backward compatibility, but the service connection is recommended.

Add Endtest to an Azure Pipeline

Add the EndtestRun@1 task to your pipeline:

steps:
  - task: EndtestRun@1
    name: runEndtest
    displayName: Run Endtest tests
    inputs:
      endtestService: 'Endtest Production'
      apiRequest: 'https://app.endtest.io/api.php?action=runWeb&suite=YOUR_SUITE_ID&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes='
      timeoutMinutes: '30'
      pollIntervalSeconds: '30'
      resultsFormat: 'json-light'
      failPipeline: true
      publishSummary: true
      publishTestResults: true
      testRunTitle: 'Endtest automated tests'

Replace YOUR_SUITE_ID and the other configuration parameters with the values from the API request generated in Endtest.

The App ID and App Code can normally be omitted from apiRequest. The task adds them from the selected service connection when they are not already present.

Task inputs

Pipeline results

When the Endtest execution finishes, the task can publish:

• The test suite name and configuration
• The number of test cases
• Passed, failed, and error totals
• Start and end times
• Direct links to the Endtest results
• JUnit results in the Azure DevOps Tests section

When failPipeline is enabled, the task fails the pipeline if Endtest reports a failure or error. This allows the task to act as a deployment gate.

Output variables

Assign a name to the task, such as runEndtest, to use its output variables in later steps:

- script: |
    echo "Executions: $(runEndtest.executionCount)"
    echo "Hashes: $(runEndtest.hashes)"
    echo "Results: $(runEndtest.resultUrls)"
    echo "Status: $(runEndtest.status)"
  displayName: Show Endtest outputs
  condition: always()

The following output variables are available:

• $(runEndtest.executionCount)
• $(runEndtest.hashes)
• $(runEndtest.resultUrls)
• $(runEndtest.testCases)
• $(runEndtest.passed)
• $(runEndtest.failed)
• $(runEndtest.errors)
• $(runEndtest.status)

Running multiple Endtest executions

An Endtest API request can start more than one execution, for example when a label is used to run several configurations.

The Azure DevOps task detects all returned execution hashes, waits for every execution to finish, includes all result links in the pipeline summary, and publishes one Azure DevOps test result for each execution.

Manual integration with a Shell Script task

The Marketplace extension is recommended for most teams because it handles credentials, polling, summaries, Azure DevOps test-result publishing, output variables, and pipeline failure behavior.

You can still use the original do-it-yourself approach when you need a fully custom implementation or cannot install Marketplace extensions in your Azure DevOps organization.

You can use the Endtest API and a Shell Script task to integrate Endtest with Azure DevOps.

The following video demonstrates the manual integration method:

YAML snippet

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: Bash@3
  inputs:
    filePath: 'test.sh'
    arguments: 'app-id app-code "api-request-to-start-test-execution" number-of-loops'

Create test.sh at the root of your repo.

We recommend creating this file from a Linux environment, such as a real Linux machine or Windows Subsystem for Linux, so that line endings are correct.

Also, do not forget to run chmod +x test.sh before you commit it.

test.sh

#!/bin/bash
hash=$(curl -X GET --header "Accept: */*" "${3}")
for run in {1.."${4}"}
do
  sleep 30
  result=$(curl -X GET --header "Accept: */*" "https://app.endtest.io/api.php?action=getResults&appId=${1}&appCode=${2}&hash=${hash}&format=json")
  if [ "$result" == "Test is still running." ]
  then
    status=$result
    # Don't print anything
  elif [ "$result" == "Processing video recording." ]
  then
    status=$result
    # Don't print anything
  elif [ "$result" == "Stopping." ]
  then
    status=$result
  elif [ "$result" == "Erred." ]
  then
    status=$result
    echo $status
  elif [ "$result" == "" ]
  then
    status=$result
    # Don't print anything
  else
     echo "$result" | jq
     exit 0
  fi
done
exit

This sample script requires the following arguments:

1. The App ID for your account.
2. The App Code for your account.
3. The API Request for starting the test execution.
4. The number of times the API request for fetching the results will be sent, once every 30 seconds.

For example, if you know that your test execution usually takes 3 minutes, you should use the value 7.

7 x 30 seconds = 210 seconds > 3 minutes

Make the Shell script executable by running:

sudo chmod +x test.sh

You also need to install the jq package:

sudo apt-get install jq

The command for running the test.sh script locally has the following format:

./test.sh app-id app-code api-request-to-start-test-execution number-of-loops

For example, consider the following values:

App ID = 44233559

App Code = 22381137

Endtest API Request to start test execution = "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes="

Number of loops = 10

The command for running the test.sh script locally would look like this:

./test.sh 44233559 22381137 "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes=" 10

The YAML file in Azure DevOps would look like this:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: Bash@3
  inputs:
    filePath: 'test.sh'
    arguments: '44233559 22381137 "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all&notes=" 10'

Since you will be integrating Endtest into your current Azure DevOps Pipeline, your actual YAML file will also contain existing instructions that are not related to Endtest.

It will also contain instructions to read the JSON output from the Endtest API response and execute the required pipeline logic, such as marking a build as passed or failed.

The JSON with the results contains the following keys:

• test_suite_name: The name of the test suite.
• configuration: The configuration of the machine or mobile device on which the test was executed.
• test_cases: The number of test cases.
• passed: The number of assertions that have passed.
• failed: The number of assertions that have failed.
• errors: The number of errors that have been encountered.
• start_time: The timestamp for the start of the test execution.
• end_time: The timestamp for the end of the test execution.
• detailed_logs: The detailed logs for the test execution.
• screenshots_and_video: The URLs for the screenshots and the video recording of the test execution.
• test_case_management: The name, status, and external IDs for each test case.

You can also use the value of the hash variable to generate the link to the Results page for that test execution:

results=https://app.endtest.io/results?hash="$hash"

More details about the API are available in the Endtest API chapter.

Troubleshooting

The EndtestRun@1 task is not found

Confirm that the extension is installed in the same Azure DevOps organization where the pipeline is running.

Authentication fails

Check the App ID and App Code in the selected Endtest service connection. Also confirm that the pipeline is authorized to use that service connection.

The execution exceeds the pipeline timeout

Increase timeoutMinutes or verify that the Endtest execution is not waiting indefinitely for an unavailable browser, device, or environment.

Detailed Endtest logs are missing

Change resultsFormat from json-light to json. The lighter format is recommended for normal runs because it returns less data.

The pipeline passes even though a test should block deployment

Confirm that failPipeline is set to true and that Endtest returned a failure or error in the final result.