# CircleCI Run Endtest end-to-end tests in CircleCI with the official Endtest orb Part of the Endtest documentation (Integrations). Canonical URL: https://endtest.io/docs/integrations/circleci --- ## Overview The official Endtest CircleCI orb allows you to run Endtest web and mobile tests directly from a CircleCI workflow. The orb can: - Start one or more Endtest test executions. - Wait for all executions to finish. - Display the result of each execution in CircleCI. - Display aggregate results when multiple test suites are triggered. - Fail the CircleCI job when any execution reports failed assertions or errors. - Block a deployment when an important end-to-end test fails. You can view the orb in the [CircleCI Orb Registry](https://circleci.com/developer/orbs/orb/endtest/endtest). The source code is available in the [Endtest CircleCI Orb repository](https://github.com/endtest-technologies/endtest-circleci-orb). ## Requirements Before using the orb, you need: 1. An Endtest account. 2. Your Endtest `App ID`. 3. Your Endtest `App Code`. 4. An Endtest API Request for starting the desired test execution. 5. A CircleCI project connected to your source-code repository. Your `App ID` and `App Code` can be found on the [Endtest Settings page](https://app.endtest.io/settings). You can generate an API Request by following the instructions from the [Endtest API](https://endtest.io/docs/advanced/how-to-use-the-endtest-api) chapter. The API Request added to the CircleCI configuration must not contain the `appId` or `appCode` parameters. The orb adds those values securely at runtime. ## Add the environment variables Open your project in CircleCI and navigate to: ```text Project Settings → Environment Variables ``` Add these environment variables: ```text ENDTEST_APP_ID ENDTEST_APP_CODE ``` Their values should be: - `ENDTEST_APP_ID`: The App ID for your Endtest account. - `ENDTEST_APP_CODE`: The App Code for your Endtest account. CircleCI hides the values of environment variables after they are saved. The Endtest API Request itself is added visibly to `.circleci/config.yml`, without credentials. This allows team members to see which test suite, label, browser, operating system, resolution, and other settings are being used. ## Add the Endtest orb Create or update the `.circleci/config.yml` file in your repository: ```yaml version: 2.1 orbs: endtest: endtest/endtest@0.0.7 workflows: endtest-tests: jobs: - endtest/run: api_request: 'https://app.endtest.io/api.php?action=runWeb&suite=YOUR_SUITE_ID&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' number_of_loops: 10 ``` Replace `YOUR_SUITE_ID` with the ID of the Endtest test suite you want to execute. Do not add `appId` or `appCode` to the API Request. The orb reads them from the CircleCI environment variables and adds them securely. Always check the [Endtest page in the CircleCI Orb Registry](https://circleci.com/developer/orbs/orb/endtest/endtest) for the latest published version. ## Trigger multiple test suites An Endtest API Request can trigger multiple test suites by using a label instead of a suite ID. For example: ```yaml version: 2.1 orbs: endtest: endtest/endtest@0.0.7 workflows: critical-tests: jobs: - endtest/run: api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' number_of_loops: 10 ``` This request starts every test suite associated with the `Critical` label. The Endtest API may return multiple execution hashes: ```text EXECUTION_HASH_1,EXECUTION_HASH_2 ``` The orb automatically: 1. Detects every execution hash. 2. Waits for all test executions to finish. 3. Displays the result of each test suite separately. 4. Calculates aggregate totals. 5. Fails the CircleCI job if any execution reports failures or errors. No additional CircleCI configuration is required for multiple test executions. ## Using the default environment-variable names The default credential environment-variable names are: ```text ENDTEST_APP_ID ENDTEST_APP_CODE ``` When you use these names, you do not need to pass the `app_id` or `app_code` parameters: ```yaml version: 2.1 orbs: endtest: endtest/endtest@0.0.7 workflows: endtest-tests: jobs: - endtest/run: api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' number_of_loops: 10 ``` ## Using custom credential variable names You can use different names for the credential environment variables: ```yaml version: 2.1 orbs: endtest: endtest/endtest@0.0.7 workflows: endtest-tests: jobs: - endtest/run: app_id: MY_ENDTEST_APP_ID app_code: MY_ENDTEST_APP_CODE api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' number_of_loops: 10 ``` In that case, add these variables to the CircleCI project: ```text MY_ENDTEST_APP_ID MY_ENDTEST_APP_CODE ``` The values passed to `app_id` and `app_code` are the names of CircleCI environment variables, not the actual credentials. ## Legacy environment-variable configuration Older versions of the Endtest orb expected the entire API Request to be stored in an environment variable named: ```text ENDTEST_API_REQUEST ``` Version `0.0.7` continues to support this configuration for compatibility. When the visible `api_request` parameter is not provided, the orb reads the complete request from `ENDTEST_API_REQUEST`. The recommended configuration is to keep only the App ID and App Code in environment variables and place the API Request without credentials in `.circleci/config.yml`. ## Parameters | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `app_id` | Environment-variable name | `ENDTEST_APP_ID` | The variable containing your Endtest App ID. | | `app_code` | Environment-variable name | `ENDTEST_APP_CODE` | The variable containing your Endtest App Code. | | `api_request` | String | Empty | A visible Endtest API Request without `appId` or `appCode`. | | `api_request_env_var` | Environment-variable name | `ENDTEST_API_REQUEST` | Legacy fallback variable containing the complete API Request. | | `number_of_loops` | Integer | `10` | Maximum number of result checks. Each check occurs after a 30-second wait. | The `api_request` parameter takes priority over `api_request_env_var`. For security, the visible API Request must use: ```text https://app.endtest.io/api.php ``` The orb rejects visible requests that contain `appId` or `appCode`. ## Choosing the number of loops The orb checks the Endtest execution once every 30 seconds. For example, when an execution normally takes approximately three minutes, you can use: ```yaml number_of_loops: 7 ``` This allows up to 210 seconds: ```text 7 × 30 seconds = 210 seconds ``` Choose a value slightly higher than the normal duration of the test execution. When a request triggers multiple test suites, set the value according to the longest-running execution. For tests that normally finish within five minutes, you can use: ```yaml number_of_loops: 10 ``` ## Pipeline behavior The CircleCI job passes when every completed Endtest execution reports: ```text Failed: 0 Errors: 0 ``` The job fails when: - Any execution reports one or more failed assertions. - Any execution reports one or more errors. - An execution enters an error state. - The API returns an invalid response. - The number of returned results does not match the number of started executions. - The executions do not finish within the configured polling period. ## Use Endtest as a deployment gate The `endtest/run` job can be used as a deployment gate: ```yaml version: 2.1 orbs: endtest: endtest/endtest@0.0.7 jobs: deploy: docker: - image: cimg/base:stable steps: - run: name: Deploy application command: ./deploy.sh workflows: test-and-deploy: jobs: - endtest/run: name: endtest-tests api_request: 'https://app.endtest.io/api.php?action=runWeb&label=Critical&platform=windows&os=windows11&browser=chrome&browserVersion=latest&resolution=1280x1024&geolocation=sanfrancisco&cases=all¬es=' number_of_loops: 10 - deploy: requires: - endtest-tests ``` The `deploy` job only starts when all Endtest executions pass. ## Results displayed in CircleCI For every Endtest execution, the orb displays: - Test suite name. - Test configuration. - Number of test cases. - Number of passed assertions. - Number of failed assertions. - Number of errors. - Start time. - End time. - Execution hash. - Link to the Endtest Results page. When multiple test suites are triggered, the orb also displays an aggregate summary: ```text Endtest summary Executions: 2 Test Cases: 4 Passed: 12 Failed: 0 Errors: 0 ``` The Results page for an execution has the following format: ```text https://app.endtest.io/results?hash=EXECUTION_HASH ``` ## Manual API integration The official orb is the recommended way to integrate Endtest with CircleCI. For custom workflows that cannot use an orb, you can call the Endtest API directly from a shell script. The script must: 1. Send the API Request that starts one or more test executions. 2. Save the execution hash or comma-separated execution hashes returned by Endtest. 3. Call the `getResults` API endpoint using the App ID, App Code, and execution hashes. 4. Continue checking until every execution finishes. 5. Return a non-zero exit code when any execution fails. More information about direct API usage is available in the [Endtest API](../advanced/how-to-use-the-endtest-api) chapter. ## Additional resources - [Endtest CircleCI Orb](https://circleci.com/developer/orbs/orb/endtest/endtest) - [Endtest CircleCI Orb source code](https://github.com/endtest-technologies/endtest-circleci-orb) - [Endtest API documentation](../advanced/how-to-use-the-endtest-api) - [Endtest Settings](https://app.endtest.io/settings)