# Azure DevOps Integrate Endtest with Azure DevOps Pipelines to trigger end-to-end tests on every build, publish results, and gate deployments on failures. Part of the Endtest documentation (Integrations). Canonical URL: https://endtest.io/docs/integrations/azure-devops --- ## Overview Integrate Endtest with Azure DevOps Pipelines to trigger automated end-to-end tests on every build, publish results in Azure DevOps, and prevent deployments when tests fail. ## Recommended approach: install the official Endtest extension The easiest way to integrate Endtest with Azure DevOps is to install the official **Endtest for Azure DevOps** extension from the Visual Studio Marketplace: [Install Endtest for Azure DevOps](https://marketplace.visualstudio.com/items?itemName=endtest-technologies.endtest-azure-pipelines) The extension adds the native `EndtestRun@1` pipeline task and an Endtest service connection type. It can: - Trigger one Endtest execution or multiple executions started through an Endtest label - Store the Endtest App ID and App Code in a secure Azure DevOps service connection - Wait for all Endtest executions to finish - Publish an Endtest summary in the pipeline run - Publish execution-level JUnit results in the Azure DevOps **Tests** section - Expose execution hashes, result links, and test totals as output variables - Fail the pipeline when Endtest reports failures or errors The extension source code is available on [GitHub](https://github.com/endtest-technologies/endtest-azure-devops-extension). ## Install the extension 1. Open the [Endtest for Azure DevOps Marketplace page](https://marketplace.visualstudio.com/items?itemName=endtest-technologies.endtest-azure-pipelines). 2. Select **Get it free**. 3. Choose the Azure DevOps organization where you want to install the extension. 4. Confirm the installation. 5. Open your Azure DevOps project. You need permission to install extensions in the selected Azure DevOps organization. ## Create an Endtest service connection After installing the extension: 1. Open your Azure DevOps project. 2. Go to **Project settings**. 3. Open **Service connections**. 4. Select **New service connection**. 5. Choose **Endtest**. 6. Set the Endtest URL to: ```text https://app.endtest.io/ ``` 7. Enter your Endtest **App ID** and **App Code**. 8. Give the connection a recognizable name, such as: ```text Endtest Production ``` 9. Save the connection. Using a service connection keeps the App Code out of your pipeline YAML. Credentials already included in the API request are still supported for backward compatibility, but the service connection is recommended. ## Add Endtest to an Azure Pipeline Add the `EndtestRun@1` task to your pipeline: ```yaml steps: - task: EndtestRun@1 name: runEndtest displayName: Run Endtest tests inputs: endtestService: 'Endtest Production' apiRequest: 'https://app.endtest.io/api.php?action=runWeb&suite=YOUR_SUITE_ID&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' timeoutMinutes: '30' pollIntervalSeconds: '30' resultsFormat: 'json-light' failPipeline: true publishSummary: true publishTestResults: true testRunTitle: 'Endtest automated tests' ``` Replace `YOUR_SUITE_ID` and the other configuration parameters with the values from the API request generated in Endtest. The App ID and App Code can normally be omitted from `apiRequest`. The task adds them from the selected service connection when they are not already present. ## Task inputs | Input | Required | Default | Description | |---|---:|---|---| | `endtestService` | No | None | Recommended Endtest service connection containing the App ID and App Code. | | `apiRequest` | Yes | None | Complete Endtest API request used to start the execution. | | `timeoutMinutes` | Yes | `30` | Maximum time to wait for the Endtest execution. Allowed range: 1 to 1440 minutes. | | `pollIntervalSeconds` | Yes | `30` | Time between result checks. Allowed range: 5 to 300 seconds. | | `resultsFormat` | Yes | `json-light` | Use `json-light` for normal pipeline runs or `json` when detailed logs and media data are needed. | | `failPipeline` | No | `true` | Fails the Azure Pipeline when Endtest reports failures or errors. | | `publishSummary` | No | `true` | Adds an Endtest summary to the pipeline run. | | `publishTestResults` | No | `true` | Publishes one Azure DevOps test result for each Endtest execution. | | `testRunTitle` | No | `Endtest automated tests` | Title used for the Azure DevOps test run. | ## Pipeline results When the Endtest execution finishes, the task can publish: - The test suite name and configuration - The number of test cases - Passed, failed, and error totals - Start and end times - Direct links to the Endtest results - JUnit results in the Azure DevOps **Tests** section When `failPipeline` is enabled, the task fails the pipeline if Endtest reports a failure or error. This allows the task to act as a deployment gate. ## Output variables Assign a name to the task, such as `runEndtest`, to use its output variables in later steps: ```yaml - script: | echo "Executions: $(runEndtest.executionCount)" echo "Hashes: $(runEndtest.hashes)" echo "Results: $(runEndtest.resultUrls)" echo "Status: $(runEndtest.status)" displayName: Show Endtest outputs condition: always() ``` The following output variables are available: - `$(runEndtest.executionCount)` - `$(runEndtest.hashes)` - `$(runEndtest.resultUrls)` - `$(runEndtest.testCases)` - `$(runEndtest.passed)` - `$(runEndtest.failed)` - `$(runEndtest.errors)` - `$(runEndtest.status)` ## Running multiple Endtest executions An Endtest API request can start more than one execution, for example when a label is used to run several configurations. The Azure DevOps task detects all returned execution hashes, waits for every execution to finish, includes all result links in the pipeline summary, and publishes one Azure DevOps test result for each execution. ## Manual integration with a Shell Script task The Marketplace extension is recommended for most teams because it handles credentials, polling, summaries, Azure DevOps test-result publishing, output variables, and pipeline failure behavior. You can still use the original do-it-yourself approach when you need a fully custom implementation or cannot install Marketplace extensions in your Azure DevOps organization. You can use the [Endtest API](https://endtest.io/docs/advanced/how-to-use-the-endtest-api/) and a [Shell Script task](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/shell-script?view=azure-devops) to integrate Endtest with Azure DevOps. The following video demonstrates the manual integration method: [Video](https://endtest-videos.s3.us-west-2.amazonaws.com/documentation/endtest_azure_devops_docs.mp4) ### YAML snippet ```yaml trigger: - main pool: vmImage: ubuntu-latest steps: - task: Bash@3 inputs: filePath: 'test.sh' arguments: 'app-id app-code "api-request-to-start-test-execution" number-of-loops' ``` Create `test.sh` at the root of your repo. We recommend creating this file from a Linux environment, such as a real Linux machine or Windows Subsystem for Linux, so that line endings are correct. Also, do not forget to run `chmod +x test.sh` before you commit it. ### `test.sh` ```bash #!/bin/bash hash=$(curl -X GET --header "Accept: */*" "${3}") for run in {1.."${4}"} do sleep 30 result=$(curl -X GET --header "Accept: */*" "https://app.endtest.io/api.php?action=getResults&appId=${1}&appCode=${2}&hash=${hash}&format=json") if [ "$result" == "Test is still running." ] then status=$result # Don't print anything elif [ "$result" == "Processing video recording." ] then status=$result # Don't print anything elif [ "$result" == "Stopping." ] then status=$result elif [ "$result" == "Erred." ] then status=$result echo $status elif [ "$result" == "" ] then status=$result # Don't print anything else echo "$result" | jq exit 0 fi done exit ``` This sample script requires the following arguments: 1. The `App ID` for your account. 2. The `App Code` for your account. 3. The API Request for starting the test execution. 4. The number of times the API request for fetching the results will be sent, once every 30 seconds. For example, if you know that your test execution usually takes 3 minutes, you should use the value 7. ```text 7 x 30 seconds = 210 seconds > 3 minutes ``` Make the Shell script executable by running: ```bash sudo chmod +x test.sh ``` You also need to install the `jq` package: ```bash sudo apt-get install jq ``` The command for running the `test.sh` script locally has the following format: ```bash ./test.sh app-id app-code api-request-to-start-test-execution number-of-loops ``` For example, consider the following values: ```text App ID = 44233559 App Code = 22381137 Endtest API Request to start test execution = "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=" Number of loops = 10 ``` The command for running the `test.sh` script locally would look like this: ```bash ./test.sh 44233559 22381137 "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=" 10 ``` The YAML file in Azure DevOps would look like this: ```yaml trigger: - main pool: vmImage: ubuntu-latest steps: - task: Bash@3 inputs: filePath: 'test.sh' arguments: '44233559 22381137 "https://app.endtest.io/api.php?action=runWeb&appId=44233559&appCode=22381137&suite=380667&platform=windows&os=windows10&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=" 10' ``` > Since you will be integrating Endtest into your current Azure DevOps Pipeline, your actual YAML file will also contain existing instructions that are not related to Endtest. > > It will also contain instructions to read the JSON output from the Endtest API response and execute the required pipeline logic, such as marking a build as passed or failed. The JSON with the results contains the following keys: - `test_suite_name`: The name of the test suite. - `configuration`: The configuration of the machine or mobile device on which the test was executed. - `test_cases`: The number of test cases. - `passed`: The number of assertions that have passed. - `failed`: The number of assertions that have failed. - `errors`: The number of errors that have been encountered. - `start_time`: The timestamp for the start of the test execution. - `end_time`: The timestamp for the end of the test execution. - `detailed_logs`: The detailed logs for the test execution. - `screenshots_and_video`: The URLs for the screenshots and the video recording of the test execution. - `test_case_management`: The name, status, and external IDs for each test case. You can also use the value of the `hash` variable to generate the link to the Results page for that test execution: ```bash results=https://app.endtest.io/results?hash="$hash" ``` More details about the API are available in the [Endtest API](https://endtest.io/docs/advanced/how-to-use-the-endtest-api/) chapter. ## Troubleshooting ### The `EndtestRun@1` task is not found Confirm that the extension is installed in the same Azure DevOps organization where the pipeline is running. ### Authentication fails Check the App ID and App Code in the selected Endtest service connection. Also confirm that the pipeline is authorized to use that service connection. ### The execution exceeds the pipeline timeout Increase `timeoutMinutes` or verify that the Endtest execution is not waiting indefinitely for an unavailable browser, device, or environment. ### Detailed Endtest logs are missing Change `resultsFormat` from `json-light` to `json`. The lighter format is recommended for normal runs because it returns less data. ### The pipeline passes even though a test should block deployment Confirm that `failPipeline` is set to `true` and that Endtest returned a failure or error in the final result.