Eclipse openDuT

Eclipse openDuT provides an open framework to automate the testing and validation process for automotive software and applications in a reliable, repeatable and observable way. Eclipse openDuT is hardware-agnostic with respect to the execution environment and accompanies different hardware interfaces and standards regarding the usability of the framework. Thereby, it is supporting both on-premise installations and hosting in a cloud infrastructure. Eclipse openDuT considers an eventually distributed network of real (HIL) and virtual devices (SIL) under test. Eclipse openDuT reflects hardware capabilities and constraints along with the chosen test method. Test cases are not limited to a specific domain, but it especially realizes functional and explorative security tests.

User Manual

Learn how to use openDuT and its individual components.

User Manual for CARL

CARL provides the backend service for openDuT. He manages information about all the DUTs and coordinates how they are put configured.

CARL also serves a web frontend, called LEA, for this purpose.

Setup of CARL

Currently, our setup is automated via Docker Compose.
See the folder .ci/deploy/localenv/ in our repository for more details.

If you want to use CARL and its components on a separate machine, i.e. a Raspberry PI or any other machine, this guide will show all necessary steps, to get CARL up and running.

Hosting Environment

  1. Install Git, if not already installed and checkout OpenDuT repository 
    git clone https://github.com/eclipse-opendut/opendut.git
  2. Install pwgen, docker.io and docker-compose-v2
  3. Optional: Change the repository CARL should be pulled from an image in .ci/deploy/localenv/docker-compose.yml. By default, CARL is pulled from ghcr.io in version 0.1.0.
  4. Set /etc/hosts file: Add the following lines to the /etc/hosts file on the host system to access the services from the local network. 
    192.168.56.9 opendut.local
192.168.56.9 auth.opendut.local
192.168.56.9 netbird.opendut.local
192.168.56.9 netbird-api.opendut.local
192.168.56.9 signal.opendut.local
192.168.56.9 carl.opendut.local
192.168.56.9 nginx-webdav.opendut.local
  5. Start the local test environment using docker compose. 
    # configure project path
export OPENDUT_REPO_ROOT=$(git rev-parse --show-toplevel)
# start provisioning and create .env file
docker compose --file ${OPENDUT_REPO_ROOT:-.}/.ci/deploy/localenv/data/provision/docker-compose.yml up --build
# start the environment
docker compose --file ${OPENDUT_REPO_ROOT:-.}/.ci/deploy/localenv/docker-compose.yml --env-file ${OPENDUT_REPO_ROOT:-.}/.ci/deploy/localenv/data/secrets/.env up --detach --build
    In this step secrets are going to be created and all containers are getting started.
    The secrets which were created during the first docker compose command can be found in .ci/deploy/localenv/data/secrets/.env.

If everything worked and is up and running, you can follow the EDGAR Setup Guide.

Configuration

  • If you followed the setup guide for CARL, there is no need to manually create this carl.toml file.
  • To configure CARL, you can create a configuration file under /etc/opendut/carl.toml.
    The possible configuration values and their defaults can be seen here:
[network]
bind.host = "0.0.0.0"
bind.port = 8080
remote.host = "localhost"
remote.port = 8080

[network.tls]
certificate = "/etc/opendut/tls/carl.pem"
key = "/etc/opendut/tls/carl.key"
ca = "/etc/opendut/tls/ca.pem"

[network.oidc]
enabled = false

[network.oidc.client]
id = "tbd"
secret = "tbd"
# issuer url that CARL uses
issuer.url = "https://keycloak/realms/opendut/"
# issuer url that CARL tells the clients to use (required in test environment)
issuer.remote.url = "https://keycloak/realms/opendut/"
issuer.admin.url = "https://keycloak/admin/realms/opendut/"
scopes = ""

[network.oidc.lea]
client.id = "opendut-lea-client"
issuer.url = "https://keycloak/realms/opendut/"
scopes = "openid,profile,email"

[peer]
disconnect.timeout.ms = 30000
can.server_port_range_start = 10000
can.server_port_range_end = 20000
ethernet.bridge.name.default = "br-opendut"

[serve]
ui.directory = "opendut-lea/"

[vpn]
enabled = true
kind = ""

[vpn.netbird]
url = ""
ca = ""
auth.type = ""
auth.secret = ""
timeout.ms = 10000
retries = 5

[logging]
stdout = true

[opentelemetry]
enabled = false
collector.endpoint = ""
service.name = "opendut-carl"

[opentelemetry.metrics]
interval.ms = 60000
cpu.collection.interval.ms = 5000

User Manual for CLEO

CLEO is a CLI tool to create/read/update/delete resources in CARL.

By using a terminal you will be able to configure your resources via CLEO.

CLEO can currently access the following resources:

  • Cluster configurations
  • Cluster deployments
  • Peers
  • Devices (DuTs)
  • Container executors

Every resource can be created, listed, described and deleted. Some have additional features such as an option to generate a setup-key or search through them.

In general, CLEO offers a help command to display usage information about a command. Just use opendut-cleo help or opendut-cleo <subcommand> --help.

Setup for CLEO

  • Download the opendut-cleo binary for your target from the openDuT GitHub project: https://github.com/eclipse-opendut/opendut/releases
  • Unpack the archive on your target system.
  • Add a configuration file /etc/opendut/cleo.toml (Linux) and configure at least the CARL host+port. The possible configuration values and their defaults can be seen here:
 [network]
carl.host = "localhost"
carl.port = 8080

[network.tls]
ca = "/etc/opendut/tls/ca.pem"
domain.name.override = ""


[network.oidc]
enabled = false

[network.oidc.client]
id = "opendut-cleo-client"
issuer.url = "https://keycloak/realms/opendut/"
scopes = "openid,profile,email"
secret = "<tbd>"

Download CLEO from CARL

It is also possible to download CLEO from one of CARLs endpoints. The downloaded file contains the binary for CLEO for the requested architecture, the necessary certificate file, as well as a setup script.

The archive can be requested at https://{CARL-HOST}/api/cleo/{architecture}/download.

Available architectures are:

  • x86_64-unknown-linux-gnu
  • armv7-unknown-linux-gnueabihf
  • aarch64-unknown-linux-gnu

This might be the go-to way, if you want to use CLEO in your pipeline. Once downloaded, extract the files with the command tar xvf opendut-cleo-{architecture}.tar.gz. It will then be extracted into the folder which is the current work directory. You might want to use another directory of your choice.

A setup string can be retrieved from LEA and used with the following command.

opendut-cleo setup <String> --persistent=<type>

The persistent flag is optional. Without the flag, the needed environment variables will be printed out to the terminal.
If the persistent flag is set to user or without a value, a configuration file will be written to ~/.config/opendut/cleo/config.toml, with it being set to system the cleo configuration file will be written to /etc/opendut/cleo.toml.

Setup via script

The script used to run CLEO will not set the environment variables for CLIENT_ID and CLIENT_SECRET. This has to be done by the users manually. This can easily be done by entering the following commands:

export OPENDUT_CLEO_NETWORK_OIDC_CLIENT_ID={{ CLIENT ID VARIABLE }} 
export OPENDUT_CLEO_NETWORK_OIDC_CLIENT_SECRET={{ CLIENT SECRET VARIABLE }}

These two variables can be obtained by logging in to Keycloak.

The tarball contains the cleo-cli.sh shell script. When executed it starts CLEO after setting the following environment variables:

OPENDUT_CLEO_NETWORK_OIDC_CLIENT_SCOPES
OPENDUT_CLEO_NETWORK_TLS_DOMAIN_NAME_OVERRIDE
OPENDUT_CLEO_NETWORK_TLS_CA
OPENDUT_CLEO_NETWORK_CARL_HOST
OPENDUT_CLEO_NETWORK_CARL_PORT
OPENDUT_CLEO_NETWORK_OIDC_ENABLED
OPENDUT_CLEO_NETWORK_OIDC_CLIENT_ISSUER_URL
SSL_CERT_FILE

SSL_CERT_FILE is a mandatory environment variable for the current state of the implementation and has the same value as the OPENDUT_CLEO_NETWORK_TLS_CA. This might change in the future.

Using CLEO with parameters works by adding the parameters when executing the script, e.g.:

./cleo-cli.sh list peers

TL;DR

  1. Download archive from https://{CARL-HOST}/api/cleo/{architecture}/download
  2. Extract tar xvf opendut-cleo-{architecture}.tar.gz
  3. Add two environment variable export OPENDUT_CLEO_NETWORK_OIDC_CLIENT_ID={{ CLIENT ID VARIABLE }} and export OPENDUT_CLEO_NETWORK_OIDC_CLIENT_SECRET={{ CLIENT SECRET VARIABLE }}
  4. Execute cleo-cli.sh with parameters

Additional notes

  • The CA certificate to be provided for CLEO depends on the used certificate authority used on server side for CARL.

Auto-Completion

You can use auto-completions in CLEO, which will fill in commands when you press TAB.

To set them up, run opendut-cleo completions SHELL where you need to replace SHELL with the shell that you use, e.g. bash, zsh or fish.
Then you need to pipe the output into a completions-file for your shell. See your shell's documentation for where to place these files.

Commands

Listing resources

To list resources you can decide whether to display the resources in a table or in JSON-format. The default output format is a table which is displayed by not using the --output flag.

opendut-cleo list --output=<format> <openDuT-resource>

Creating resources

To create resources it depends on the type of resource whether an ID or connected devices have to be added to the command.

opendut-cleo create <resource>

Generating PeerSetup Strings

To create a PeerSetup, it is necessary to provide the PeerID of the peer:

opendut-cleo generate-setup-string <PeerID>

Decoding PeerSetup Strings

If you have a peer setup string, and you want to analyze its content, you can use the decode command.

opendut-cleo decode-setup-string <String>

Describing resources

To describe a resource, the ID of the resource has to be provided. The output can be displayed as text or JSON-format.

opendut-cleo describe --output=<output format> <resource> --id

Finding resources

You can search for resources by specifying a search criteria string with the find command. Wildcards such as '*' are also supported.

opendut-cleo find <resource> "<at least one search criteria>"

Delete resources

Specify the type of resource and its ID you want to delete in CARL.

opendut-cleo delete <resource> --id <ID of resource>

Usage Examples

CAN Example

# CREATE PEER
    opendut-cleo create peer --name "$NAME" --location "$NAME"

# CREATE NETWORK INTERFACE
    opendut-cleo create network-interface --peer-id "$PEER_ID" --type can --name vcan0

# CREATE DEVICE
    opendut-cleo create device --peer-id "$PEER_ID" --name device-"$NAME"-vcan0 --interface vcan0 

# CREATE SETUP STRING
    opendut-cleo generate-setup-string --id "$PEER_ID"

Ethernet Example

# CREATE PEER
    opendut-cleo create peer --name "$NAME" --location "$NAME"

# CREATE NETWORK INTERFACE
    opendut-cleo create network-interface --peer-id "$PEER_ID" --type eth --name eth0

# CREATE DEVICE
    opendut-cleo create device --peer-id "$PEER_ID" --name device-"$NAME"-eth0 --interface eth0 

# CREATE SETUP STRING
    opendut-cleo generate-setup-string --id "$PEER_ID"

CLEO and jq

jq is a command line tool to pipe outputs from json into pretty json or extract values. That is how jq can automate cli-applications.

Basic jq

  • jq -r removes " from strings.
  • [] constructs an array
  • {} constructs an object

e.g. jq '[ { "name:" .[].name, "id:" .[].id } ]' or: jq '[ .[] | { title, name } ]'

input

opendut-cleo list --output=json peers

output

[
  {
    "name": "HelloPeer",
    "id": "90dfc639-4b4a-4bbb-bad3-6f037fcde013",
    "status": "Disconnected"
  },
  {
    "name": "Edgar",
    "id": "defe10bb-a12a-4ad9-b18e-8149099dd044",
    "status": "Connected"
  },
  {
    "name": "SecondPeer",
    "id": "c3333d4e-9b1a-4db5-9bfa-7a0a40680f1a",
    "status": "Disconnected"
  }
]

input

opendut-cleo list --output=json peers | jq '[.[].name]'

output
jq extracts the names of every json element in the list of peers.

[
  "HelloPeer",
  "Edgar", 
  "SecondPeer" 
]

which can also be put into an array with cleo list --output=json peers | jq '[.[].name']

input

opendut-cleo list --output=json peers | jq '[.[] | select(.status=="Disconnected")]'

output

[
    {
      "name": "HelloPeer",
      "id": "90dfc639-4b4a-4bbb-bad3-6f037fcde013",
      "status": "Disconnected"
    },
    {
      "name": "SecondPeer",
      "id": "c3333d4e-9b1a-4db5-9bfa-7a0a40680f1a",
      "status": "Disconnected"
    }
]

input

opendut-cleo list --output=json peers | jq '.[] | select(.status=="Connected") | .id' | xargs -I{} cleo describe peer -i {}

output

Peer: Edgar
  Id: defe10bb-a12a-4ad9-b18e-8149099dd044
  Devices: [device-1, The Device, Another Device, Fubar Device, Lost Device]

Get the number of the peers

opendut-cleo list --output=json peers | jq 'length'

Sort peers by name

opendut-cleo list --output=json peers | jq 'sort_by(.name)'

EDGAR

EDGAR hooks your DuT up to openDuT. It is a program to be installed on a Linux host, which is placed next to your ECU. A single-board computer, like a Raspberry Pi, is good enough for this purpose.

Within openDuT, EDGAR is a Peer in the network.

Setup

  • Download the opendut-edgar binary for your target from the openDuT GitHub project: https://github.com/eclipse-opendut/opendut/releases

  • Unpack the archive on your target system.

  • If you want to use CAN, follow the steps in CAN Setup before continuing.

  • If you want to use a self-hosted backend server without DNS name or with a self-signed certificate, follow the steps in Self-Hosted Backend Server before continuing.

  • EDGAR comes with a scripted setup, which you can initiate by running:

    opendut-edgar setup managed <SETUP-STRING>

You can get the <SETUP-STRING> from LEA or CLEO after creating a Peer.

This will configure your operating system and start the EDGAR Service, which will receive its configuration from CARL.

Download EDGAR from CARL

You can download the EDGAR binary as tarball from one of CARLs endpoints.

The archive can be requested at https://{CARL-HOST}/api/edgar/{architecture}/download.

Available architectures are:

  • x86_64-unknown-linux-gnu
  • armv7-unknown-linux-gnueabihf
  • aarch64-unknown-linux-gnu

Once downloaded, extract the files with the command tar xvf opendut-edgar-{architecture}.tar.gz. It will then be extracted into the folder which is the current work directory. You might want to use another directory of your choice.

CAN Setup

  • Only if you want to use CAN, it is mandatory to set the environment variable OPENDUT_EDGAR_SERVICE_USER.

    export OPENDUT_EDGAR_SERVICE_USER=root

  • It is possible that there are no bridges configured, in that case use those two commands:

    # Replace vcan0 with your bridge name
ip link add dev vcan0 type vcan
ip link set dev vcan0 up

  • Install wireguard-tools:

    sudo apt install wireguard-tools
# You can use it with `sudo wg`
# To view a more comprehensive information use `/opt/opendut/edgar/netbird/netbird status -d`

  • Test CAN connection with multiple EDGARs, execute on EDGAR leader:

    candump -d can0

    On EDGAR peer execute:

    cansend can0 01a#01020304

    Now you should see a can frame on leader side:

    root@host:~# candump -d can0
can0  01A   [4]  01 02 03 04

Cannelloni

  1. Install the following packages:
sudo apt install -y python3-can can-utils
  • Download Cannelloni from here: https://github.com/eclipse-opendut/cannelloni/releases/
  • Unpack the Cannelloni tarball and copy the files into your filesystem like so: 
    sudo cp libcannelloni-common.so.0 /lib/
sudo cp libsctp.so* /lib/
sudo cp cannelloni /usr/local/bin/

Self-Hosted Backend Server

DNS

If your backend server does not have a public DNS entry, you will need to adjust the /etc/hosts/ file, by appending entries like this (using your server's IP address):

123.456.789.101 opendut.local
123.456.789.101 carl.opendut.local
123.456.789.101 auth.opendut.local
123.456.789.101 netbird.opendut.local
123.456.789.101 netbird-api.opendut.local
123.456.789.101 signal.opendut.local

Now the following command should complete without errors:

ping carl.opendut.local

Self-Signed Certificate with Unmanaged Setup

If you plan to use the unmanaged setup and your NetBird server uses a self-signed certificate, follow these steps:

  1. Create the certificate directory on the OLU: mkdir -p /usr/local/share/ca-certificates/

  2. Copy your NetBird server certificate onto the OLU, for example, by running the following from outside the OLU:

    scp certificate.crt root@10.10.4.1:/usr/local/share/ca-certificates/

    Ensure that the certificate has a file extension of "crt".

  3. Run update-ca-certificates on the OLU.
    It should output "1 added", if everything works correctly.

  4. Now the following commands should complete without errors:

    curl https://netbird-api.opendut.local

Troubleshooting

  • In case of issues during the managed setup see:

    journalctl -u opendut-edgar.service

  • Sometimes it might be necessary to stop and re-start the EDGAR service:

    # Stop service
sudo systemctl stop opendut-edgar.service
# Start service
sudo systemctl start opendut-edgar.service
# Restart service
sudo systemctl restart opendut-edgar.service
# Check status
systemctl status opendut-edgar.service

  • Netbird service on host machine. It might happen that Netbird is not able to connect, in that case stop it and re-run EDGAR managed setup:

    # Stop service
sudo systemctl stop netbird.service
# Start service
sudo systemctl start netbird.service

  • More log files / statements can be found in the corresponding Docker containers:

    docker logs localenv-keycloak-1
docker logs localenv-carl-1
# To display all running containers use:
docker ps

  • Netbird logs are available on host machine

    cat /var/lib/netbird/client.log
cat /var/lib/netbird/netbird.err
cat /var/lib/netbird/netbird.out

  • EDGAR might start with an old IP, different from command sudo wg would print. In that particular case stop netbird service and opendut-edgar service and re-run the setup. This might happen to all EDGARs. If this is not enough, and it keeps getting the old IP, it is necessary to set up all devices and clusters from scratch.

    sudo wg

  • If this ERROR appears: ERROR opendut_edgar::service::cannelloni_manager: Failed to start cannelloni instance for remote IP 100.80.171.237: 'No such file or directory (os error 2)'. (OPTIONAL) Copy cannelloni to custom location. This is the way to go, if the step before is not possible to be done. This can happen for whatever reason, i.e. missing root rights.

    export LD_LIBRARY_PATH=/your-path-to-cannelloni/cannelloni (just folder, not the binary)
export PATH={all_other_PATH_variables}:/your-path-to-cannelloni/cannelloni
vim /etc/systemd/system/opendut-edgar.service 
systemctl daemon-reload
systemctl restart opendut-edgar.service

    Copy the two Environment variables into your opendut-edgar.service file.

    [Unit]
...

[Service]
...
Environment="LD_LIBRARY_PATH=/opt/opendut/edgar/cannelloni"
Environment="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/opendut/edgar/cannelloni"


[Install]
...

Test Execution

In a nutshell, test execution in openDuT works by executing containerized (Docker or Podman) test applications on a peer and uploading the results to a WebDAV directory. Test executors can be configured through either CLEO or LEA.

The container image specified by the image parameter in the test executor configuration can either be a container image already present on the peer or an image remotely available, e.g., in the Docker Hub.

A containerized test application is expected to move all test results to be uploaded to the /results/ directory within its container and create an empty file /results/.results_ready when all results have been copied there. When this file exists, or when the container exits and no results have been uploaded yet, EDGAR creates a ZIP archive from the contents of the /results directory and uploads it to the WebDAV server specified by the results-url parameter in the test executor configuration.

In the testenv launched by THEO, a WebDAV server is started automatically and can be reached at http://nginx-webdav/. In the Local Test Environment, a WebDAV server is also started automatically and reachable at http://nginx-webdav.opendut.local.

Note that the execution of executors is only triggered by deploying the cluster.

Test Execution using CLEO

In CLEO, test executors can be configured either by passing all configuration parameters as command line arguments...

$ opendut-cleo create container-executor --help 
Create a container executor using command-line arguments

Usage: opendut-cleo create container-executor [OPTIONS] --peer-id <PEER_ID> --engine <ENGINE> --image <IMAGE>

Options:
    --peer-id <PEER_ID>          ID of the peer to add the container executor to
-e, --engine <ENGINE>            Engine [possible values: docker, podman]
-n, --name <NAME>                Container name
-i, --image <IMAGE>              Container image
-v, --volumes <VOLUMES>...       Container volumes
    --devices <DEVICES>...       Container devices
    --envs <ENVS>...             Container envs
-p, --ports <PORTS>...           Container ports
-c, --command <COMMAND>          Container command
-a, --args <ARGS>...             Container arguments
-r, --results-url <RESULTS_URL>  URL to which results will be uploaded
-h, --help                       Print help

...or by providing a JSON-formatted configuration file.

$ opendut-cleo apply container-executor --help 
Create a container executor using a JSON-formatted configuration file

Usage: opendut-cleo apply container-executor <CONFIG_FILE>

Arguments:
<CONFIG_FILE>  Path to the JSON-formatted executor configuration file

Options:
-h, --help  Print help

Configuration File Format

A JSON configuration file for a test executor that is passed to CLEO may look as follows.

{
    "peer-id": "26ada545-e834-4af3-8b66-af860ad19dbe",
    "container": {
        "engine": "docker",
        "name": "nmap-test",
        "image": "nmap-test",
        "envs": {
            "ENVVAR1": "someenvironmentvariable",
            "ENVVAR2": "anotherenvironmentvariable"
        },
        "args": [
            "-A",
            "-T4",
            "127.0.0.1"
        ],
        "ports":  [],
        "volumes": [],
        "command": "",
        "devices": []
    },
    "results-url": "http://nginx-webdav:80/"
}

Test Execution Through LEA

In LEA, executors can be configured via the tab Executor during peer configuration, using similar parameters as for CLEO.

Developer Manual

Learn how to get started, the workflow and tools we use, and what our architecture looks like.

Getting Started

Development Setup

Install the Rust toolchain: https://www.rust-lang.org/tools/install

You may need additional dependencies. On Ubuntu/Debian, these can be installed with:

sudo apt install build-essential pkg-config libssl-dev

To see if your development setup is generally working, you can run cargo ci check in the project directory.
Mind that this runs the unit tests and additional code checks and may occasionally show warnings/errors related to those, rather than pure build errors.

Tips & Tricks

  • cargo ci contains many utilities for development in general.

  • To view this documentation fully rendered, run cargo ci doc book open.

  • To have your code validated more extensively, e.g. before publishing your changes, run cargo ci check.

Starting Applications

  • Run CARL (backend): 
    cargo carl

You can then open the UI by going to https://localhost:8080/ in your web browser.

  • Run CLEO (CLI for managing CARL):

    cargo cleo

  • Run EDGAR (edge software):

    cargo edgar service

    Mind that this is in a somewhat broken state and may be removed in the future,
    as it's normally necessary to add the peer in CARL and then go through edgar setup.
    For a more realistic environment, see test-environment.

UI Development

Run cargo lea to continuously build the newest changes in the LEA codebase.
Then you can simply refresh your browser to see them.

Git Workflow

Pull requests

Update Branch

Our goal is to maintain a linear Git history. Therefore, we prefer git rebase over git merge1. The same applies when using the GitHub WebUI to update a PR's branch.

Screenshot of the GitHub UI showing a PR's update branch dropdown

  1. Update with merge commit:

    The first option creates a merge commit to pull in the changes from the PR's target branch and this is against our goal of a linear history, so we do not use this option.

  2. Update with rebase:

    The second option rebases the changes of the feature branch on top of the PR's target branch. This is the preferred option we use.

Rebase and Merge

As said above, our goal is to maintain a linear Git history. A problem arises when we want to merge pull requests (PR), because the GitHub WebUI offers ineligible options to merge a branch:

Screenshot of the GitHub UI showing a PR's merge button dropdown

  1. Create a merge commit:

    The first option creates a merge commit to pull in the changes from the PR's branch and this is against our goal of a linear history, so we disabled this option.

  2. Squash and merge:

    The second option squashes all commits in the PR into a single commit and adds it to the PR's target branch. With this option, it is not possible to keep all commits of the PR separately.

  3. Rebase and merge:

    The third option rebases the changes of the feature branch on top of the PR's target branch. This would be our preferred option, but "The rebase and merge behavior on GitHub deviates slightly from git rebase. Rebase and merge on GitHub will always update the committer information and create new commit SHAs"2. This doubles the number of commits and spams the history unnecessarily. Therefore, we do not use this option either.

The only viable option for us is to rebase and merge the changes via the command line. The procedures slightly differ according to the location of the feature branch.

Feature branch within the same repository

This example illustrates the procedure to merge a feature branch fubar into a target branch development.

  1. Update the target branch with the latest changes.

    git pull --rebase origin development

  2. Switch to the feature branch.

    git checkout fubar

  3. Rebase the changes of the feature branch on top of the target branch.

    git rebase development

    This is a good moment to run test and validation tasks locally to verify the changes.

  4. Switch to the target branch.

    git checkout development

  5. Merge the changes of the feature branch into the target branch.

    git merge --ff-only fubar

    The --ff-only argument at this point is optional, because we rebased the feature branch and git automatically detects, that a fast-forward is possible. But this flag prevents a merge-commit, if we messed-up one of the previous steps.

  6. Push the changes.

    git push origin development

Feature branch of a fork repository

This example illustrates the procedure to merge a feature branch foo from a fork bar of the user doe into a target branch development.

  1. Update the target branch with the latest changes.

    git pull --rebase origin development

  2. From the project repository, check out a new branch.

    git checkout -b doe-foo development

  3. Pull in the changes from the fork.

    git pull git@github.com:doe/bar.git foo

  4. Rebase the changes of the feature branch on top of the target branch.

    git rebase development

    This is a good moment to run test and validation tasks locally to verify the changes.

  5. Switch to the target branch.

    git checkout development

  6. Merge the changes of the feature branch into the target branch.

    git merge --ff-only fubar

    The --ff-only argument at this point is optional, because we rebased the feature branch and git automatically detects, that a fast-forward is possible. But this flag prevents a merge-commit if we messed-up one of the previous steps.

  7. Push the changes.

    git push origin development
1

Except git merge --ff-only.

Building a Release

To build release artifacts for distribution, run:

cargo ci distribution

The artifacts are placed under target/ci/distribution/.

To build a docker container of CARL and push it to the configured docker registry:

cargo ci carl docker --publish

This will publish opendut-carl to ghcr.io/eclipse-opendut/opendut-carl:x.y.z. The version defined in opendut-carl/Cargo.toml is used as docker tag by default.

Alternative platform

If you want to build artifacts for a different platform, use the following:

cargo ci distribution --target armv7-unknown-linux-gnueabihf

The currently supported target platforms are:

  • x86_64-unknown-linux-gnu
  • armv7-unknown-linux-gnueabihf
  • aarch64-unknown-linux-gnu

Alternative docker registry

Publish docker container to another container registry than ghcr.io.

export OPENDUT_DOCKER_IMAGE_HOST=other-registry.example.net
export OPENDUT_DOCKER_IMAGE_NAMESPACE=opendut
cargo ci carl docker --publish --tag 0.1.1

This will publish opendut-carl to 'other-registry.example.net/opendut:opendut-carl:0.1.1'.

Test Environment

openDuT can be tricky to test, as it needs to modify the operating system to function and supports networking in a distributed setup.
To aid in this, we offer a virtualized test environment for development.
This test environment is set up with the help of a command line tool called theo. THEO stands for Test Harness Environment Operator.

It is recommended to start everything in a virtual machine, but you may also start the service on the host with docker compose if applicable. Setup of the virtual machine is done with Vagrant, Virtualbox and Ansible. The following services are included in docker:

  • carl
  • edgar
  • firefox container for UI testing (accessible via http://localhost:3000)
    • includes certificate authorities and is running in headless mode
    • is running in same network as carl and edgar (working DNS resolution!)
  • netbird
  • keycloak

Operational modes

There are two ways of operation for the test environment:

Test mode

Run everything in Docker (Either on your host or preferable in a virtual machine). You may use the OpenDuT Browser to access the services. The OpenDuT Browser is a web browser running in a docker container in the same network as the other services. All certificates are pre-installed and the browser is running in headless mode. It is accessible from your host via http://localhost:3000.

OpenDuT-VM

Development mode

Run CARL on the host in your development environment of choice and the rest in Docker. In this case there is a proxy running in the docker environment. It works as a drop-in replacement for CARL in the docker environment, which is forwarding the traffic to CARL running in an integrated development environment on the host.

OpenDuT-VM

Getting started

Set up the virtual machine

Then you may start the test environment in the virtual machine.

There are some known issues with the test environment (most of them on Windows):

Start testing

Once you have set up and started the test environment, you may start testing the services.

User interface

The OpenDuT Browser is a web browser running in a docker container. It is based on KasmVNC base image which allows containerized desktop applications from a web browser. A port forwarding is in place to access the browser from your host. It has all the necessary certificates pre-installed and is running in headless mode. You may use this OpenDuT Browser to access the services.

  • Open following address in your browser: http://localhost:3000
  • Usernames for test environment:
    • LEA: opendut:opendut
    • Keycloak: admin:admin123456
    • Netbird: netbird:netbird
    • Grafana: admin:admin
  • Services with user interface:
    • https://carl
    • https://netbird-dashboard
    • https://keycloak
    • http://grafana

THEO Setup in Vagrant

You may run all the containers in a virtual machine, using Vagrant. This is the recommended way to run the test environment. It will create a private network (subnet 192.168.56.0/24). The virtual machine itself has the IP address: 192.168.56.10. The docker network has the IP subnet: 192.168.32.0/24. Make sure those network addresses are not occupied or in conflict with other networks accessible from your machine.

Requirements

  • Install Vagrant

    Ubuntu / Debian

    sudo apt install vagrant

    On most other Linux distributions, the package is called vagrant.

  • Install VirtualBox (see https://www.virtualbox.org)

    sudo apt install virtualbox

  • Create or check if an ssh key pair is present in ~/.ssh/id_rsa

    mkdir -p ~/.ssh
ssh-keygen -t rsa -b 4096 -C "opendut-vm" -f ~/.ssh/id_rsa

Setup virtual machine

  • Either via cargo: 
    cargo theo vagrant up
  • Login to the virtual machine 
    cargo theo vagrant ssh

Warning Within the VM the rust target directory is overridden to /home/vagrant/rust-target to avoid hard linking issues. When running cargo within the VM, output will be placed in this directory!

  • Ensure a distribution of openDuT is present

    • By either creating one yourself (on the host) 
      cargo ci distribution
    • Or by copying one to the target directory target/ci/distribution/x86_64-unknown-linux-gnu/ 
      mkdir -p target/ci/distribution/x86_64-unknown-linux-gnu/

  • Start test environment

    cargo theo testenv start

Setup THEO on Windows

This guide will help you set up THEO on Windows.

Requirements

The following instructions use chocolatey to install the required software. If you don't have chocolatey installed, you can find the installation instructions here. You may also install the required software manually or e.g. use the Windows Package Manager winget (Hashicorp.Vagrant, Oracle.VirtualBox, Git.Git).

  • Install vagrant and virtualbox

    choco install -y vagrant virtualbox

  • Install git and configure git to respect line endings

    choco install git.install --params "'/GitAndUnixToolsOnPath /WindowsTerminal'"

  • Create or check if a ssh key pair is present in ~/.ssh/id_rsa

    mkdir -p ~/.ssh
ssh-keygen -t rsa -b 4096 -C "opendut-vm" -f ~/.ssh/id_rsa

Info
Vagrant creates a VM which mounts a Windows file share on /vagrant, where the openDuT repository was cloned. The openDuT project contains bash scripts that would break if the end of line conversion to crlf on windows would happen. Therefore a .gitattributes file containing
*.sh text eol=lf
was added to the repository in order to make sure the bash scripts also keep the eol=lf when cloned on Windows. As an alternative, you may consider using the cloned opendut repo on the Windows host only for the vagrant VM setup part. For working with THEO, you can use the cloned opendut repository inside the Linux guest system instead (/home/vagrant/opendut).

Setup virtual machine

  • Add the following environment variables to point vagrant to the vagrant file
    Git Bash: 
    export OPENDUT_REPO_ROOT=$(git rev-parse --show-toplevel)
export VAGRANT_DOTFILE_PATH=$OPENDUT_REPO_ROOT/.vagrant
export VAGRANT_VAGRANTFILE=$OPENDUT_REPO_ROOT/.ci/docker/Vagrantfile
    PowerShell: 
    $env:OPENDUT_REPO_ROOT=$(git rev-parse --show-toplevel)
$env:VAGRANT_DOTFILE_PATH="$env:OPENDUT_REPO_ROOT/.vagrant"
$env:VAGRANT_VAGRANTFILE="$env:OPENDUT_REPO_ROOT/.ci/docker/Vagrantfile"
  • Set up the vagrant box (following commands were tested in Git Bash and Powershell)
vagrant up

Info
If the virtual machine is not allowed to create or use a private network you may disable it by setting the environment variable OPENDUT_DISABLE_PRIVATE_NETWORK=true.

  • Connect to the virtual machine via ssh (requires the environment variables)
vagrant ssh

Additional notes

You may want to configure a http proxy or a custom certificate authority. Details are in the Advance usage section.

THEO Setup in Docker

Requirements

  • Install Docker

    Ubuntu / Debian

    sudo apt install docker.io

    On most other Linux distributions, the package is called docker.

  • Install Docker Compose v2

    Ubuntu / Debian

    sudo apt install docker-compose-v2

    Alternatively, see https://docs.docker.com/compose/install/linux/.

  • Add your user into the docker group, to be allowed to use Docker commands without root permissions. (Mind that this has security implications.)

    sudo groupadd docker  # create `docker` group, if it does not exist
sudo gpasswd --add $USER docker  # add your user to the `docker` group
newgrp docker  # attempt to activate group without re-login

    You may need to log out your user account and log back in for this to take effect.

  • Create a distribution of openDuT

cargo ci distribution
  • Start containers
cargo theo testenv start
  • Start edgar cluster
cargo theo testenv edgar start

Use virtual machine for development

OpenDuT-VM

  • Start vagrant on host: cargo theo vagrant up

  • Connect to virtual machine from host: cargo theo vagrant ssh

  • Start developer test mode in opendut-vm: cargo theo dev start

  • Once keycloak and netbird are provisioned, generate run configuration for CARL in opendut-vm: cargo theo dev carl-config

    • which should give an output similar to the following:
OPENDUT_CARL_NETWORK_REMOTE_HOST=carl
OPENDUT_CARL_NETWORK_REMOTE_PORT=443
OPENDUT_CARL_VPN_ENABLED=true
OPENDUT_CARL_VPN_KIND=netbird
OPENDUT_CARL_VPN_NETBIRD_URL=https://192.168.56.10/api
OPENDUT_CARL_VPN_NETBIRD_CA=<ca_certificate_filepath>
OPENDUT_CARL_VPN_NETBIRD_AUTH_SECRET=<dynamic_api_secret>
OPENDUT_CARL_VPN_NETBIRD_AUTH_TYPE=personal-access-token
OPENDUT_CARL_VPN_NETBIRD_AUTH_HEADER=Authorization
  • You may also use the toml configuration (also printed from the carl-config command) file in a special configuration file on your host at ~/.config/opendut/carl/config.toml.
  • Use the environment variables in the run configuration for CARL
    • Run CARL on the host: cargo ci carl run
    • Run LEA on the host: cargo ci lea run
  • Or start CARL in your IDE of choice and add the environment variables to the run configuration.

Use virtual machine for testing

This mode is used to test a distribution of OpenDuT.

OpenDuT-VM

  • Ensure a distribution of openDuT is present

    • By either creating one yourself on your host: 
      cargo ci distribution
    • Or in the opendut-vm. Within the VM the rust target directory is overridden to /home/vagrant/rust-target. Therefore, you need the to copy the created distribution to the expected location. 
      cargo ci distribution
mkdir -p /vagrant/target/ci/distribution/x86_64-unknown-linux-gnu/
cp ~/rust-target/ci/distribution/x86_64-unknown-linux-gnu/* /vagrant/target/ci/distribution/x86_64-unknown-linux-gnu/
    • Or by copying one to the target directory target/ci/distribution/x86_64-unknown-linux-gnu/ 
      # ensure directory is present
mkdir -p target/ci/distribution/x86_64-unknown-linux-gnu/
# copy distribution to target directory

  • Login to the virtual machine from your host (assumes you have already set up the virtual machine)

    cargo theo vagrant ssh

  • Start test environment in opendut-vm:

    cargo theo testenv start

  • Start a cluster in opendut-vm:

    cargo theo testenv edgar start

    This will start several EDGAR containers and create an OpenDuT cluster.

Known Issues

Copying data to and from the OpenDuT Browser

The OpenDuT Browser is a web browser running in a docker container. It is based on KasmVNC base image which allows containerized desktop applications from a web browser. When using the OpenDuT Browser, you may want to copy data to and from the OpenDuT browser inside your own browser. On Firefox this is restricted, and you may use the clipboard window on the left side of the OpenDuT Browser to copy data to your clipboard.

Cargo Target Directory

When running cargo tasks within the virtual machine, you may see following error:

warning: hard linking files in the incremental compilation cache failed. copying files instead. consider moving the cache directory to a file system which supports hard linking in session dir

This is mitigated by setting a different target directory for cargo in /home/vagrant/.bashrc on the virtual machine:

export CARGO_TARGET_DIR=$HOME/rust-target

Vagrant Permission Denied

Sometimes vagrant fails to insert the private key that was automatically generated. This might cause this error (seen in git-bash on Windows):

$ vagrant ssh
vagrant@127.0.0.1: Permission denied (publickey).

This can be fixed by overwriting the vagrant-generated key with the one inserted during provisioning:

cp ~/.ssh/id_rsa .vagrant/machines/opendut-vm/virtualbox/private_key

Vagrant Timeout

If the virtual machine is not allowed to create or use a private network it may cause a timeout during booting the virtual machine.

Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
  • You may disable the private network by setting the environment variable OPENDUT_DISABLE_PRIVATE_NETWORK=true and explicitly halt and start the virtual machine again.
export OPENDUT_DISABLE_PRIVATE_NETWORK=true
vagrant halt
vagrant up

Vagrant Custom Certificate Authority

When running behind an intercepting http proxy, you may run into issues with SSL certificate verification.

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)

This can be mitigated by adding the custom certificate authority to the trust store of the virtual machine.

  • Place certificate authority file here: resources/development/tls/custom-ca.crt
  • And re-run the provisioning of the virtual machine.
export CUSTOM_ROOT_CA=resources/development/tls/custom-ca.pem
vagrant provision

Ctrl+C in Vagrant SSH

When using cargo theo vagrant ssh on Windows and pressing Ctrl+C to terminate a command, the ssh session may be closed.

Netbird management invalid credentials

If keycloak was re-provisioned after the netbird management server, the management server may not be able to authenticate with keycloak anymore.

# docker logs netbird-management-1
[...]
2024-02-14T09:51:57Z WARN management/server/account.go:1174: user 59896d1b-45e6-48bb-ae79-aa17d5a2af94 not found in IDP
2024-02-14T09:51:57Z ERRO management/server/http/middleware/access_control.go:46: failed to get user from claims: failed to get account with token claims user 59896d1b-45e6-48bb-ae79-aa17d5a2af94 not found in the IdP

docker logs edgar-leader
[...]
Failed to create new peer.
[...]
  Received status code indicating an error: HTTP status client error (403 Forbidden) for url (http://netbird-management/api/groups)

This may be fixed by destroying the netbird service:

cargo theo testenv destroy --service netbird

Afterward you may restart the netbird service:

cargo theo testenv start
# or 
cargo theo dev start

No space left on device

Error writing to file - write (28: No space left on device)

You may try to free up space on the virtual machine by (preferred order):

  • Cleaning up the cargo target directory: 
    cargo clean
ls -l $CARGO_TARGET_DIR
  • removing old docker images and containers: 
    docker system prune --all
# and eventually with volumes
docker system prune --all --volumes

Advanced Usage

Use vagrant directly

Run vagrant commands directly instead of through THEO:

  • or directly via Vagrant's cli (bash commands run from the root of the repository): 
    export OPENDUT_REPO_ROOT=$(git rev-parse --show-toplevel)
export VAGRANT_DOTFILE_PATH=$OPENDUT_REPO_ROOT/.vagrant
export VAGRANT_VAGRANTFILE=$OPENDUT_REPO_ROOT/.ci/docker/Vagrantfile
vagrant up
  • provision vagrant with desktop environment 
    ANSIBLE_SKIP_TAGS="" vagrant provision

Re-provision the virtual machine

This is recommended after potentially breaking changes to the virtual machine.

  • Following command will re-run the ansible playbook to re-provision the virtual machine. Run from host:
cargo theo vagrant provision
  • Destroy test environment and re-create it, run within the virtual machine:
cargo theo vagrant ssh
cargo theo testenv destroy
cargo theo testenv start

Cross compile THEO for Windows on Linux

cross build --release --target x86_64-pc-windows-gnu --bin opendut-theo
# will place binary here
target/x86_64-pc-windows-gnu/release/opendut-theo.exe

Proxy configuration

In case you are working behind a http proxy, you need additional steps to get the test environment up and running. The following steps pick up just before you start up the virtual machine with vagrant up. A list of all domains used by the test environment is reflected in the proxy shell script: .ci/docker/vagrant/proxy.sh. It is important to note that the proxy address used shall be accessible from the host while provisioning and within the virtual machine.

If you have a proxy server on your localhost you need to make this in two steps:

  • Use proxy on your localhost

    • Configure vagrant to use the proxy localhost. 
      # proxy configuration script, adjust to your needs
source .ci/docker/vagrant/proxy.sh http://localhost:3128
    • Install proxy plugin for vagrant 
      vagrant plugin install vagrant-proxyconf
    • Then starting the VM without provisioning it. This should create the vagrant network interface with network range 192.168.56.0/24. 
      vagrant up --no-provision

  • Use proxy on private network address 192.168.56.1

    • Make sure this address is allowing access to the internet: 
      curl --max-time 2 --connect-timeout 1 --proxy http://192.168.56.1:3128 google.de
    • Redo the proxy configuration using the address of the host within the virtual machine's private network: 
      # proxy configuration script, adjust to your needs
source .ci/docker/vagrant/proxy.sh http://192.168.56.1:3128
    • Reapply the configuration to the VM 
      $ vagrant up --provision
Bringing machine 'opendut-vm' up with 'virtualbox' provider...
==> opendut-vm: Configuring proxy for Apt...
==> opendut-vm: Configuring proxy for Docker...
==> opendut-vm: Configuring proxy environment variables...
==> opendut-vm: Configuring proxy for Git...
==> opendut-vm: Machine not provisioned because `--no-provision` is specified.

  • Unset all proxy configuration for testing purposes (non-permanent setting in the shell)

    unset http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY

  • You may also set the docker proxy configuration in your environment manually:

    • ~/.docker/config.json 
      {
  "proxies": {
    "default": {
      "httpProxy": "http://x.x.x.x:3128",
      "httpsProxy": "http://x.x.x.x:3128",
      "noProxy": "localhost,127.0.0.1,netbird-management,netbird-dashboard,netbird-signal,netbird-coturn,keycloak,edgar-leader,edgar-*,carl,192.168.0.0/16"
    }
  }
}
    • /etc/docker/daemon.json 
      {
  "proxies": {
    "http-proxy": "http://x.x.x.x:3128",
    "https-proxy": "http://x.x.x.x:3128",
    "no-proxy": "localhost,127.0.0.1,netbird-management,netbird-dashboard,netbird-signal,netbird-coturn,keycloak,edgar-leader,edgar-*,carl,192.168.0.0/16"
  }
}

Custom root certificate authority

This section shall provide information on how to provision the virtual machine when running behind an intercepting http proxy. This is also used in the docker containers to trust the custom certificate authority. All certificate authorities matching the following path will be trusted in the docker container: ./resources/development/tls/*-ca.pem.

The following steps need to be done before provisioning the virtual machine.

  • Place certificate authority file here: resources/development/tls/custom-ca.crt
  • Optionally, disable private network definition of vagrant, if this causes errors.
export CUSTOM_ROOT_CA=resources/development/tls/custom-ca.pem
export OPENDUT_DISABLE_PRIVATE_NETWORK=true  # optional
vagrant provision

Secrets for test environment

This repository contains secrets for testing purposes. These secrets are not supposed to be used in a production environment. There are two formats defined in the repository that document their location:

  • ~/.gitguardian.yml
  • .secretscanner-false-positives.json

Alternative strategy to avoid this: auto-generate secrets during test environment setup.

GitGuardian

Getting started with ggshield

  • Install ggshield 
    sudo apt install -y python3-pip
pip install ggshield
export PATH=~/.local/bin/:$PATH
  • Login to https://dashboard.gitguardian.com
  • Either use PAT or service account (https://docs.gitguardian.com/api-docs/service-accounts)
  • Goto API -> Personal access tokens
    • and create a token
  • Use API token to login: ggshield auth login --method token

Scan repository

  • See https://docs.gitguardian.com/ggshield-docs/getting-started

  • Scan repo

    ggshield secret scan repo ./

  • Ignore secrets found in last run and remove them or document them in .gitguardian.yml

    ggshield secret ignore --last-found

  • Review changes in .gitguardian.yml and commit

Overview

overview.excalidraw.svg

Components

  • CARL (Control And Registration Logic)
  • EDGAR (Edge Device Global Access Router)
  • LEA (Leasing ECU Access)
  • CLEO (Command-Line ECU Orchestrator)
  • DUT (Device under test)

Functional description

openDuT provisions an end-to-end encrypted private network between Devices under Test (DuT), Test Execution Engines, RestBus simulations, and other devices. To achieve this, openDuT uses Edge Device Global Access Router (EDGAR), which can tunnel the Ethernet traffic (Layer 2) of the connected devices into the openDuT network using Generic Routing Encapsulation (GRE). CAN traffic is tunnelled between EDGAR instances using cannelloni. EDGAR registers with the Control and Registration Logic (CARL) and reports the type and status of its connected devices. Multiple EDGARs can be linked to clusters via the graphical Leasing ECU Access (LEA) UI or the Command-Line ECU Orchestrator (CLEO) of CARL, and the openDuT cluster can be provisioned for the user.

opendut-functional-diagram.svg

openDuT uses NetBird technology and provides its own NetBird server, including a TURN server in CARL and NetBird clients in the EDGARs. The NetBird clients of the clustered EDGARs automatically build a WireGuard network in star topology. If a direct connection between two EDGARs is not possible, the tunnel is routed through the TURN server in CARL.

edgar-gre-bridging.excalidraw.svg

Within EDGAR, the openDUT ETH Bridge manages Ethernet communication and routes outgoing packets to the GRE-Bridge(s). The GRE-Bridges encapsulate the packets and send them over fixed-assigned sources to fixed-assigned targets. When encapsulating, GRE writes the source and header information and the protocol type of the data packet into the GRE header of the packet. This offers the following advantages: different protocol types can be sent, network participants can be in the same subnet, and multiple VLANs can be transmitted through a single WireGuard tunnel.

CAN interfaces on EDGAR are connected by means of the openDUT CAN Bridge, which is effectively a virtual CAN interface connected to the individual interfaces by means of can-gw rules. Between the leading EDGAR and each other EDGAR, a cannelloni tunnel is established, linking the CAN bridges of different EDGAR instances together.

CARL

EDGAR

Setup

Service

Deployment

Architecture Overview

Cluster

ClusterState

Cluster

Cluster Creation

message ClusterConfiguration {
  ClusterId id = 1;
  ClusterName name = 2;
  opendut.types.peer.PeerId leader = 3;
  repeated opendut.types.topology.DeviceId devices = 4;
}

Cluster Deployment

message ClusterAssignment {
  ClusterId id = 1;
  opendut.types.peer.PeerId leader = 3;
  repeated PeerClusterAssignment assignments = 4;
}

message PeerClusterAssignment {
  opendut.types.peer.PeerId peer_id = 1;
  opendut.types.util.IpAddress vpn_address = 2;
  opendut.types.util.Port can_server_port = 3;
  repeated opendut.types.util.NetworkInterfaceDescriptor device_interfaces = 4;
}

Peer

PeerState

Telemetry

Architecture Overview