Skip to main content

Test Quarantining

Once you've detected flaky tests, the next step is to fix them. However, you may have other priorities - here lies a dilemma. Flaky tests impact developer time, but there are other deliverables as well.

BuildPulse enables you to temporarily disable tests that flake beyond a configured threshold, to mitigate the impact of flaky tests. You can actually save the time consumed shown on our dashboard:

  1. Configure quarantine disruption percentage: Breached tests will automatically be marked as quarantined, and served via the Quarantine API. A test can also be manually quarantined on the Dashboard.
  2. Query the Quarantine API for flaky tests
  3. Disable/skip Tests
  4. Track & Re-enable tests (Jira/Linear/GitHub Issues)

Configure Thresholds​

To configure your flaky test threshold:

  1. Navigate to the BuildPulse Flaky Test Dashboard.

  2. Click on top-right dropdown, and navigate to Settings > Repository Settings > Thresholds.

  3. Set the disruption percentage which will determine quarantine status for a flaky test. There are a variety of thresholds that configure detection and quarantining behavior.

    Git-Based Flaky Tests:

    • Quarantine Minimum Count: Minimum number of results that have to be seen before a test is quarantined (default: 10, not configurable)
    • Quarantine Disruption Percentage: If the disruption percentage exceeds this value, the test is quarantined (must be configured).

    Statistical Flaky Tests:

    • Detection Minimum Count: Minimum number of results that have to be seen before a test is quarantined. This can be configured (default: 10).
    • Detection Disruption Percentage: If the test case failure rate exceeds this value, the test is marked statistically flaky. This can be configured (default: 30%).
    • Quarantine Disruption Percentage: This inherits the same value is Git-Based Quarantine Disruption Percentage, and determins if the test is quarantined.

Query the Quarantine API​

  1. Generate an API key at Settings > General > API Key.
  2. Query the API for flaky tests:

URL: https://api.buildpulse.io/v1/flaky/tests

Authorization: Authorization: Bearer ${BUILDPULSE_API_KEY}

Query Parameters:

  • repository (required, or repository_id): The name of your repository/project
  • repository_id (required, or repository): The ID of your repository/project. You can find if you navigate to the repository > Flaky Tests tab > View Setup Guide (bottom left) > Other Providers > Step 2 (see Repository ID).
  • quarantine: Set to false to return all flaky tests (default: true)
  • include: Expand flaky test attributes. Supported values: tags, disruptiveness_ratio, nondeterministic_negative_result_count, quarantine_date, quarantine_type, time_consumed (microseconds), quarantined_by. Example: include=tags,disruptiveness_ratio
  • after: Pagination key. Value will be null on the last page.
  1. Sample Response:
{
    "count": 1,
    "tests": [
        {
            "id": 434140152,
            "name": "System Posts Lifecycle Basic posts flow can edit a post",
            "suite": "rspec",
            "class": "spec.system.posts_lifecycle_spec",
            "file": "./spec/system/posts_lifecycle_spec.rb"
            "disruptor_type": "default" # 'default' for git-based, 'statistical' for statistical flaky tests, 'disruptor_type_manual' for manually quarantined tests
        }
    ],
    "metadata": {
        "after": null,
        "limit": 25
    }
}

Disable/Skip Tests​

Please consult the documentation of your test framework for disabling individual tests.

Critical Tests​

Test cases can be marked as 'critical', to prevent them from being quarantined automatically. Manual quarantining will still work, and tests that are already quarantined will remain so.

To mark a test case as 'critical', navigate to the test case page and toggle the checkbox in the test case metadata panel.

Track & Re-enable tests (Jira/Linear/GitHub Issues)​

BuildPulse will preserve quarantined tests beyond the data retention period, and are tagged as 'quarantined' in the BuildPulse Flaky Test Dashboard.

BuildPulse will also automatically create issues in your project management system when a test is quarantined.

A test is re-enabled and resolved by either moving the created issue into the 'Done' column, or can also be manually resolved. This will remove the test from the quarantined list, and will no longer be ignored by your test framework.