1 Overview

Radicle is a peer-to-peer collaboration system built on top of the Git version control system. Radicle has support for integrating with continuous integration (CI) systems, using an architecture where a "broker" listens to events about changes to repositories stored in a node, and launching the appropriate "adapter" for each change, according to its configuration.

This means each node can opt into running CI for what projects and changes according to the interests of the person whose node it is.

The delegates for a repository might run CI on all patches to make merge decisions with more confidences.
Someone else, who is contributing to a project, might only care about patches they themselves created, and only run CI for those.
A third party might run CI for projects they use, to know if it's OK to deploy to their production systems.

Radicle provides its own, very simple "native CI" solution. It's just good enough for the Radicle project to use itself. In addition, there are adapters that integrate with external CI systems.

1.1 Goal of Radicle CI

Context: The user is a software developer working on a project that uses Radicle for version control. The project has an automated test suite, and in-repository configuration for how to build the project and run the test suite, in a format suitable for the CI engine being used.

In the long run, the goal for CI in Radicle is "anything that makes it easier, more fun, and faster to produce working software", but that's not a concrete goal.

At this stage in the development of Radicle CI has two concrete goals:

When I create a patch to propose a change, I am automatically told if the project branch with my committed changes fails to build or pass its test suite. I can also manually check what the status of that process ("CI run") is, and find out what the build log is, to investigate any problems.
- This is "build and test the patch branch".
When a project delegate merges my patch, both they and I are automatically told if the merge fails due to a merge conflict, or if, after the merge the project no longer builds or its test suite fails.
- This is "build and test the master branch after the merge". This is useful, because sometimes a merged change breaks the build or the test suite, even when there are no merge conflicts.

It is not yet clear how notifications will work.

1.2 Components of Radicle CI

Using CI with Radicle requires several co-operating components:

The Radicle node (radicle-node).
The Radicle CI broker (cib).
An adapter the runs CI, possibly using an external CI service.
A management tool (cibtool).

Pikchr diagram

The node emits an event (RefsFetched), when it has fetched updates to one of the repositories it has. The event specifies the repository, and every Git ref. The CI broker parses this event to extract information about the kind of change: branch has created? updated? deleted? Likewise for tags and patches, etc. These become "CI events", to distinguish them from "node events".

The node operator can specify filters on CI events, and if an event gets past a filter, the CI broker will run an external program called an "adapter". The adapter has the responsibility of running CI for the change, and to report to the CI broker if it succeeded or failed.

Communication with the adapter is via the adapter's standard input and output, using single-line JSON messages.

UML diagram

The CI broker also generates HTML report pages to list all the CI runs it has performed.

2 The adapter

The adapter process reads the request to perform a CI run on a specific commit in a repository, and responds with the id of the run, then later with the status of the finished run.

For native CI, the adapter actually is the CI engine and performs the CI run itself. For external CI, the adapter process does whatever it needs to do to get the external CI engine instance to perform the CI run. If the CI engine calls back via web hook to notify of the run finishing, the adapter process needs to receive the call and process it.

External CI engines allow complex pipelines to be written and support a variety of workflows. Different jobs or tasks can be triggered based on different events (e.g. push, patch created, patch updated, etc.), to satisfy different workflow needs. (Only a few of these are actually implemented yet, but the scaffolding to support more is there.)

Some examples of real-world use-cases:

trigger "fast" tests on every push to any branch
trigger "relatively fast" tests only when a Patch is created / updated
trigger "full test suite" on every push to the default branch (e.g. main)

In order to allow developers using Radicle the same flexibility that they are used to on other forges, we want the broker to pass on whatever information it already has from the node events to the adapters, so they can pass it on to external CI systems, as appropriate.

3 Report generation

The CI broker has an SQLite database file for persistent storage of information of the CI runs it triggers. This is used to generate report pages, among other things.

The report page generation is done in its own thread, separate from the main thread of the CI broker. This allows the reporting to happens independently of what the main thread is doing. In particular, it means the report generation happens even while the main thread is busy running an adapter.

When it comes to per-run logs, the adapter can include a URL to one in the first response message. The URL will be included as a link in the report HTML.

4 Sample messages to and from CI adapter

Note: the JSON objects below are formatted on multiple lines to make them easier to read. The actual wire format is one line per message.

4.1 Trigger on branch create or update

{
  "request": "trigger",
  "version": 1,
  "event_type": "push",
  "repository": {
    "id": "rad:z2Et4G1QwmUoqxmesdbdPRcThzFXy",
    "name": "acme",
    "description": "Acme's repository",
    "private": false,
    "default_branch": "master",
    "delegates": [
      "did:key:z6MknwfiqsqycvkeQ9u8xH6iT9ajCVERsnd2fYsxDgQAeZGL"
    ]
  },
  "pusher": {
    "id": "did:key:z6MknwfiqsqycvkeQ9u8xH6iT9ajCVERsnd2fYsxDgQAeZGL",
    "alias": null
  },
  "before": "532c92e4ec973d79c25652c581ae82d238e46a06",
  "after": "532c92e4ec973d79c25652c581ae82d238e46a06",
  "branch": "refs/namespaces/z6MknwfiqsqycvkeQ9u8xH6iT9ajCVERsnd2fYsxDgQAeZGL/refs/heads/master",
  "commits": [
    "532c92e4ec973d79c25652c581ae82d238e46a06"
  ]
}

4.2 Trigger on patch create or update

{
  "request": "trigger",
  "version": 1,
  "event_type": "patch",
  "repository": {
    "id": "rad:z4QWN49BpecRRzq11dMhNYd6NbLUn",
    "name": "acme",
    "description": "Acme's repository",
    "private": false,
    "default_branch": "master",
    "delegates": [
      "did:key:z6Mkmy8jdkuDScdGtGYW6M3vNvmcz2w2ZcR58UDaJbAvcoJb"
    ]
  },
  "action": "Created",
  "patch": {
    "id": "1e4673a0d3d973747d00b2d8ffe31dbdbcd5c729",
    "author": {
      "id": "did:key:z6Mkmy8jdkuDScdGtGYW6M3vNvmcz2w2ZcR58UDaJbAvcoJb",
      "alias": null
    },
    "title": "my patch title",
    "state": {
      "status": "open",
      "conflicts": []
    },
    "before": "f2de534b5e81d7c6e2dcaf58c3dd91573c0a0354",
    "after": "532c92e4ec973d79c25652c581ae82d238e46a06",
    "commits": [
      "532c92e4ec973d79c25652c581ae82d238e46a06"
    ],
    "target": "f2de534b5e81d7c6e2dcaf58c3dd91573c0a0354",
    "labels": [],
    "assignees": [],
    "revisions": [
      {
        "id": "1e4673a0d3d973747d00b2d8ffe31dbdbcd5c729",
        "author": {
          "id": "did:key:z6Mkmy8jdkuDScdGtGYW6M3vNvmcz2w2ZcR58UDaJbAvcoJb",
          "alias": null
        },
        "description": "my patch description",
        "base": "f2de534b5e81d7c6e2dcaf58c3dd91573c0a0354",
        "oid": "532c92e4ec973d79c25652c581ae82d238e46a06",
        "timestamp": 1757432355
      }
    ]
  }
}

4.3 Response when run has started

{
  "response": "triggered",
  "run_id": {
    "id": "xyzzy"
  },
  "info_url": null
}

4.4 Response when run has started, with info URL

{
  "response": "triggered",
  "run_id": {
    "id": "xyzzy"
  },
  "info_url": "https://ci.example.com/runid/log.html"
}

4.5 Response on successful finish

{
  "response": "finished",
  "result": "success"
}

4.6 Response on failure finish

{
  "response": "finished",
  "result": "failure"
}

Radicle CI broker

Table of Contents