How to setup CodeRabbit Verify - Quickstart guide

Last updated: November 18, 2026

Overview

CodeRabbit Verify is a public beta product that runs functional tests on the whole repository or individual pull requests (PRs) based on auto-generated user journeys in a sandboxed copy of the repo code. Agentic QA results are reported with videos, screenshots, and logs with actionable follow up with Linear, Jira, or Coding Agent to fix any failures from the test runs. A repository-wide test run requires a configured infrastructure environment whose snapshot is prepared in the CodeRabbit app before the first run.

Prerequisites

  • The organization uses CodeRabbit Cloud on GitHub Cloud, Bitbucket Cloud, or Gitlab.com.

  • A running application test can be seen in a live browser view from within the CodeRabbit web app. Native mobile apps are not supported.

  • Verify usage is free during the beta for participating organizations, including free Agent minutes to fix bugs caught in the test runs.

Set up PR testing

  1. In the CodeRabbit app, create an infrastructure environment, assign the repository, and prepare the environment until the status is Ready. Creating environments requires Coding Agent write permissions.

  2. Select Verify in the sidebar and select Add repository. Choose the repository, branch, and Ready environment to add.

  3. Select Review > Verify and confirm that Enable PR testing is on (enabled by default).

  4. Comment @coderabbitai verify on a PR. Results appear in the PR comment and in the Verify view of Change Stack. This will run tests only on the PR changes, as opposed to full repo-wide testing described in previous steps.

  5. Optionally, turn on Auto Verify under Automatic runs to test PRs when opened. Select Also verify new commits to include new pushes.

  6. For more detail, visit our Verify documentation.

Set up User Journeys and Schedules

  1. On the Verify tab select your repository.

  2. Select the Journey Testing tab.

    1. Under User Journeys, select Add user journey to add a user journey manually.

    2. Under Schedules, select Create schedule.

  3. Visit our User Journeys and Schedules documentation for more detail.

Reviewing run results

  • Passed: The scenario completed without a finding.

  • Finding: The scenario detected a bug or blocker, with evidence attached. A finding can be turned into a Linear or Jira ticket or sent to CodeRabbit Agent for a fix.

  • Inconclusive: The scenario could not be completed. To resolve, open the agent trace and check for missing credentials, unavailable services, startup failures, or unsupported dependencies. Correct the relevant environment setup, then run Verify again.

Current limitations

  • Native mobile apps (iOS, Android) are not supported.

  • Tests that span multiple repositories are not supported.

  • Testing guidelines cannot be loaded through MCP connections.

  • Verify runs the application in a CodeRabbit sandbox. Running against an externally deployed environment is not supported.