1 Introduction

The Radicle CI broker runs CI for repositories in the local Radicle node. This is the user guide for the CI broker.

The CI broker helps users run validation on changes to their software project, by automating the building and testing of the projects when anything in the repository changes. This is often called "continuous integration".

(Technically, "continuous integration" is the software development practice to merge changes into the main line of development frequently, at least daily, to avoid painful merge conflict resolutions. However, for this guide we say "CI" to mean "when repository changes, perform these actions", which is a more generic, and quite popular definition, if not very purist.)

2 Overview

The Radicle node stores Git repositories and synchronizes them with other Radicle nodes. The CI broker connects to its local node and gets "node events" whenever anything changes in the node. The relevant change for the CI broker is that a Git references ("refs") in a repository have been created, updated, or deleted. For now, these are branches. Later, Radicle and the CI broker will support other references, such as tags.

There are no node events for Git repositories being created or deleted. It's not possible to create a Radicle repository without creating a branch, so just looking at references is enough.

The CI broker looks at the reference changes and refines them into "CI events", which are more suitable for the kind CI use that the CI broker is meant to enable, than "this ref changed", which is quite low level.

The CI events are filtered, and events that are allowed by the filter trigger a CI run, which runs another program called the CI adapter. The adapter arranges for a CI system or CI engine to execute CI for the code change captured in the CI event. There are adapters for different CI systems, such as GitHub Actions, Concourse, Kraken, and also the "native adapter", that runs a shell script locally on the host where the adapter runs. Each adapter may require a different configuration to suit the CI system it targets.

3 Getting started

To use Radicle CI, you first need to have a Radicle node where you want to run CI. Due to technical limitations, this can't be your usual node, because it won't react to changes you push to it, only to changes it receives from other nodes.

Once you've installed a Radicle node and it is running:

Install the Radicle CI broker. Currently this is easiest to by building the latest release from source. This requires the Rust compiler and tools to be installed.

cargo install radicle-ci-broker --locked

Install the Radicle native CI adapter. This is the simplest adapter to get to work, even if you don't want to use it later.

cargo install radicle-native-ci --locked

Create a configuration file. You can call the file anything you like (the example below assumes cib.yaml). You should ensure the Radicle node is below is your normal node. You should make sure the report_dir field points to a directory that exists. The example below assumes _rad user; adjust paths as necessary.

db: /home/_rad/ci-broker.db
report_dir: /srv/http
queue_len_interval: 1min
adapters:
  native:
    command: /bin/radicle-native-ci
    env:
      RADICLE_NATIVE_CI: /home/_rad/native-ci.yaml
      PATH: /bin:/home/_rad/.radicle/bin:/home/_rad/.cargo/bin
    sensitive_env: {}
triggers:
  - adapter: native
    filters:
    - !And
      - !HasFile ".radicle/native.yaml"
      - !Node z6MkgEMYod7Hxfy9qCvDv5hYHkZ4ciWmLFgfvm3Wn1b2w2FV
      - !Or
        - !DefaultBranch
        - !PatchCreated
        - !PatchUpdated

Create a configuration file for the native CI adapter. Place it in the location specified in cib.yaml above in the RADICLE_NATIVE_CI environment variable.

state: /srv/http
log: /home/_rad/native-ci.log
base_url: http://setup-ci/

Start the CI adapter:

cib --config cib.yaml process-events

You can also set up a web server to serve the files in report_dir directory over HTTP (with TLS, by preference). They are static files, which are easy to serve. Any web server can do it. You can copy the files to another server too, if you prefer.

To test this, tell your seed node to seed the Radicle CI example project. (You can also seed any other repository, but the example is there to make it easy to try this out.) You will need to do this from a different shell session (terminal window or tab) than the previous command:

rad seed rad:z28U8KUBvVSMQc13NydX3LBDsdEdQ

Then push a patch to this repository, from your usual node, not the CI node.

rad clone rad:z28U8KUBvVSMQc13NydX3LBDsdEdQ
cd radicle-ci-example
git switch -c patch
date > date
git add date
git commit -m trigger
git push rad HEAD:refs/patches

Check the report file to see that it works.

/home/_rad/native-ci.log
/srv/http/index.html
With your web browser the URL to where the report directory gets published.

If all works, excellent. If not, and you need help, drop by the Radicle Zulip chat for ask for help.

Next, you probably want to consider what adapter you want to use. See the radicle-ci-integrations-docs repository for a list.

4 Configuration

The CI broker must be started with a configuration file. The config sub-command tells cib to write out the actual run-time configuration, as computed from the specified configuration file and built-in defaults.

cib --config /etc/radicle-ci-broker/config.yaml config

The output will be in JSON (which also works as YAML input):

{
  "default_adapter": "dummy",
  "adapters": {
    "dummy": {
      "command": "../dummy.sh",
      "env": {},
      "sensitive_env": {}
    }
  },
  "filters": [],
  "triggers": null,
  "report_dir": "html",
  "status_update_interval_seconds": null,
  "db": "x.db",
  "max_run_time": {
    "secs": 3600,
    "nanos": 0
  },
  "queue_len_interval": {
    "secs": 3600,
    "nanos": 0
  }
}

The configuration fields are:

field	summary
`adapters`	list of CI adapters
`db`	database
`default_adapter`	this will become deprecated, use `triggers` instead
`filters`	this will become deprecated, use `triggers` instead
`max_run_time`	maximum duration of a CI run
`queue_len_interval`	how often the event queue length should be logged
`report_dir`	directory where HTML and JSON report pages are written
`status_update_interval_seconds`	how often should `status.json` be updated?
`triggers`	see the "Triggering CI" chapter

5 CI events

The CI broker currently supports a small set of CI events. There will be more.

In the tables below, the fields have the following meanings:

from_node -- the node from which the event originated
repo -- the ID of the repository concerned
branch -- the name of the branch created of updated
tip -- the newest commit in the branch or patch
old_tip -- the previous newest tip, before the change

5.1 `BranchCreated`

A branch has been created. This may mean the repository has also been created, but that is not certain.

Event	fields	field types
`BranchCreated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`branch`	`BranchName`
	`tip`	`Oid`

5.2 `BranchUpdated`

A branch has been updated.

Event	fields	field types
`BranchUpdated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`branch`	`BranchName`
	`tip`	`Oid`
	`old_tip`	`Oid`

5.3 `BranchDeleted`

A branch has been deleted.

Event	fields	field types
`BranchDeleted`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`branch`	`BranchName`
	`tip`	`Oid`

5.4 `TagCreated`

An annotated Git tag has been created.

Event	fields	field types
`TagCreated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`tag_name`	`RefString`
	`tip`	`Oid`

5.5 `TagUpdated`

An annotated Git tag has been updated.

Event	fields	field types
`TagUpdated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`tag_name`	`RefString`
	`tip`	`Oid`
	`old_tip`	`Oid`

5.6 `TagDeleted`

An annotated Git tag has been deleted.

Event	fields	field types
`TagDeleted`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`tag_name`	`RefString`
	`tip`	`Oid`

5.7 `PatchCreated`

A patch has been created.

Event	fields	field types
`PatchCreated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`patch`	`PatchId`
	`new_tip`	`Oid`

5.8 `PatchUpdated`

A patch has been updated.

Event	fields	field types
`PatchUpdated`	`from_node`	`NodeId`
	`repo`	`RepoId`
	`patch`	`PatchId`
	`new_tip`	`Oid`

6 Event filters

The CI broker configuration can use the following conditions, and AND/OR/NOT operators to build a filter expression: if the expression evaluates as "true", the event is allowed and will trigger a CI run. Otherwise it is discarded and does not trigger a CI run.

Condition	Meaning
`Allow`	Change is allowed
`And` or `AllOf`	Change is allowed if all the operands are true
`BranchCreated`	Branch was created
`BranchDeleted`	Branch was deleted
`BranchUpdated`	Branch was updated
`TagCreated`	Annotated tag was created
`TagDeleted`	Annotated tag was deleted
`TagUpdated`	Annotated tag was updated
`Branch`	Event refers to a specific Git branch
`DefaultBranch`	Event refers to a default branch of the repository
`Deny`	Changes is not allowed
`HasFile`	Commit in event contains named file
`Node`	Event originated from a specific node, identified by ID
`Not` or `NoneOf`	Change is allowed is the operand expressions are is false
`Or` or `AnyOf`	Change is allows if any of the operands is true
`PatchCreated`	Patch was created
`PatchUpdated`	Patch was updated
`Patch`	Event refers to a specific patch, identified by ID
`Repository`	Event refers to a specific repository, identified by ID

6.1 Example

The following example is a snippet of YAML for the CI broker configuration file to match events that refer to the main in the CI broker repository.

filters:
  - !And
    - !Repository "rad:zwTxygwuz5LDGBq255RA2CbNGrz8"
    - !Branch "main"

The conditions are expressed using the !Foo syntax in YAML. Foo must be one of the operands from the table above. Simple values are expressed as doubly quoted strings, and lists of operands are sub-lists in YAML syntax.

The filters field is a list of filter expressions that implicitly joined together using '!Or` -- in other words, if any of the expressions in the list allows the event, the whole list allows the events.

7 Triggering CI

The Radicle CI broker can be configured to trigger CI runs in two ways:

a global filter and a default adapter
a list of triggers, each of which has its own filter, and specifies an adapter to use

Example:

adapters:
  foo:
    command: foo-adapter
    env:
      RADICLE_CI_FOO: foo-ci.yaml
  bar:
    command: bar-adapter
    env:
      RADICLE_CI_BAR: bar-ci.yaml
default_adapter: foo
filters:
  - !Branch "main"
triggers:
  - adapter: foo
    filters:
      - !Branch "staging"
  - adapter: bar
    filters:
      - !PatchCreated
      - !PatchUpdated

The above configuration specifies two aapters, foo and bar.

foo is used for the main branch and staging branch
bar is used for patches

Thus, if the main branch changes, the foo adapter is run. If a branch other than main or staging changes, CI is not run.

7.1 Configuring adapters

The adapters field in the configuration file gives a name to each adapter. The name only matters within the configuration file, it has no other significance, but each name must be different. For each adapter the following fields are specified:

command - the command by which the adapter is invoked
- this is only the name of the command, without any options
- the executable is found via the PATH, or the name can be an absolute path
env is a mapping of name/value pairs; when the command is invoked, it is given each name as an environment variable, set to the value in the mapping
sensitive_env is like env, but the values are never logged or output; this is to prevent accidental leaking of secrets

Both env and sensitive_env are optional.

7.2 Default adapter

The default_adapter field can be set to the name of an adapter in the adapters field. That adapter will be used if the filters filters allow an event.

If filters isn't set, default_adapter is not used.

7.3 Triggers

The triggers field contains a sequence conditions (filters) that are used to decide if an adapter is to be run. A trigger has two fields:

filters is a list of event filters, just like the top level filters field, with the same syntax, predicates, and meaning
adapter names the adapter to run if the filters allow an event

If there are several triggers that are allowed, every adapter is run, in sequence. If an earlier adapter run fails, the later ones are still run. However, only the first error is returned to the CI broker.

Radicle CI broker user guide

Table of Contents

1 Introduction

2 Overview

3 Getting started

4 Configuration

5 CI events

5.1 `BranchCreated`

5.2 `BranchUpdated`

5.3 `BranchDeleted`

5.4 `TagCreated`

5.5 `TagUpdated`

5.6 `TagDeleted`

5.7 `PatchCreated`

5.8 `PatchUpdated`

6 Event filters

6.1 Example

7 Triggering CI

7.1 Configuring adapters

7.2 Default adapter

7.3 Triggers

Radicle CI broker user guide

Table of Contents

1 Introduction

2 Overview

3 Getting started

4 Configuration

5 CI events

5.1 BranchCreated

5.2 BranchUpdated

5.3 BranchDeleted

5.4 TagCreated

5.5 TagUpdated

5.6 TagDeleted

5.7 PatchCreated

5.8 PatchUpdated

6 Event filters

6.1 Example

7 Triggering CI

7.1 Configuring adapters

7.2 Default adapter

7.3 Triggers

5.1 `BranchCreated`

5.2 `BranchUpdated`

5.3 `BranchDeleted`

5.4 `TagCreated`

5.5 `TagUpdated`

5.6 `TagDeleted`

5.7 `PatchCreated`

5.8 `PatchUpdated`