BeetleboxCI Enterprise Installation

BeetleboxCI Enterprise is an on-premises CI solution for customers who have large scale computational needs for their CI applications. BeetleboxCI is capable on running on bare metal or on a virtual machine, and it operates on a Kubernetes cluster. Kubernetes is used to schedule user-defined jobs.

Mininum requirements for installing BeetleboxCI Enterprise:

An x64/x86_64/AMD64/Intel 64 system
At least 4 CPUs
At least 16GB RAM

Recommended:

An x64/x86_64/AMD64/Intel 64 system
At least 12 CPUs
At least 32GB RAM

Introduction

This guide describes the process of installing the enterprise version of BeetleboxCI.

Services

BeetleboxCI exposes a number of services:

Frontend web application accessed through a browser. (Port 32767)

A local container registry to store images to run jobs. (Port 5000)

An NFS server to allow other nodes to access shared files (Port 2049)

beetleboxci-db - Stores information about users, jobs etc.

beetleboxci-redis-server - Manages the queue of jobs to be run.

beetleboxci-redis-worker-main - Orchestrates user defined jobs.

beetleboxci-redis-worker-poll - Monitors pipelines for changes.

beetleboxci-redis-scheduler - Handles cron jobs.

beetleboxci-nginx - Proxy that handles web server requests.

beetleboxci-web - Backend web server.

metrics-server - Monitors performance of user-defined jobs.

registry - Docker registry for storing custom container images.

Ports

The application exposes a number of ports:

53 - TCP/UDP - Kubernetes DNS server

179 - TCP - Calico CNI

443 - TCP - Metrics server

2049 - TCP - Network File System

2379 - TCP - etcd API

2380 - TCP - etcd internal communication

5000 - TCP - Docker registry

5432 - TCP - PostgreSQL

6443 - TCP - Kubernetes API server

6379 - TCP - Redis

10250 - TCP - Kubelet API

10251 - TCP - Kube-scheduler API

10252 - TCP - Kube-controller-manager API

32767 - TCP - Web app traffic

Platform

BeetleboxCI Enterprise is designed to be deployed onto a server with a Kubernetes cluster.

Scripted Installation

BeetleboxCI provides a set of scripts through which you can install the application on a server from scratch if your server is running Ubuntu Server 22.04.1. These scripts set up kubernetes for you as well as installing BeetleboxCI using a helm chart.

It is likely that many server setups will require manual installation.

Installing BeetleboxCI

You can sign up and download the installer here: BeetleboxCI.

Before you begin, please ensure that you know the hostname, fully qualified domain name (FQDN) and IP address of each of the servers that will form the cluster.

The hostname may only contain lowercase alphanumeric characters, hyphens and dots. You can find the commands to change the hostname of your machine here: Changing the hostname. Please note that other applications that depend on the current hostname may stop working if the hostname is changed.

On a Linux machine:

Hostname: The hostname can be obtained by typing the command hostname into a Linux terminal.
Fully Qualified Domain Name: The fully qualified domain name would be managed by the network administrator in your organisation if the server is part of a company intranet. It may be of a similar form to hostname.companyname.com. For machines not on a network domain, the FQDN would usually be hostname.local.
IP Address: The IP address can be obtained by typing ifconfig into a terminal or by running hostname -I | awk '{print $1}', which will usually list several network adapters. The IP address to use is the one that corresponds to the main network interface (Ethernet/WiFi) of your server.

Extract the tarball using a terminal and enter the folder that gets extracted:

tar -xzvf BeetleboxCIv*.tar.gz
cd dist/
tar -xzvf BeetleboxCIv*.tar.gz # Extract the tarball inside this one too
cd BeetleboxCIv*

Open the 000_params_example.sh parameter file and edit the following parameters (you may copy this file to a different location and rename it for future reference):

BeetleboxCI application parameters:

Parameter	Description
CONTROL_NODE_FQDN	Fully qualified domain name of the server where the BeetleboxCI application is installed
BBX_DEPLOY_NODE	Hostname of the server where the BeetleboxCI application is installed
BBX_DEPLOY_NODE_IP	IP address of the server where the BeetleboxCI application is installed

Network File System (NFS) parameters:

Parameter	Description
NFS_SERVER_FQDN	Fully qualified domain name of the server where the NFS is located
NFS_SERVER_NAME	Hostname of the server where the NFS is located
NFS_SERVER_IP	IP address of the server where the NFS is located
NFS_SERVER	On multi-node clusters, we recommend setting this to the same value as the NFS_SERVER_FQDN. On single node clusters, this can be set to the IP of the server if the IP is static or otherwise set to the same value as NFS_SERVER_FQDN.

Container Registry parameters:

Parameter	Description
REGISTRY_FQDN	Fully qualified domain name of the server where the container registry is hosted
REGISTRY_NODE_NAME	Hostname of the server where the container registry is hosted
REGISTRY_IP	IP address of the server where the container registry is hosted
REGISTRY_URL	On multi-node clusters, we recommend setting this to the same value as the REGISTRY_FQDN. On single node clusters, this can be set to the IP of the server if the IP is static or otherwise set to the same value as REGISTRY_FQDN or REGISTRY_NODE_NAME.

If you are only using a single node cluster, then the BeetleboxCI application, the NFS and the Container Registry will share the same parameter values. During the installation, the parameter file will be copied to the Beetlebox install path (default: /mnt/beetleboxci)

Run the 00_init_control_node.sh script. You will be prompted for the path of the parameter file which you edited earlier. This script installs the Kubernetes software and sets up a cluster on your machine.
```
./00_init_control_node.sh
```
Now you can restart your machine. Restarting is required for the containerisation backend to function correctly, otherwise the following scripts will not work correctly.

Run the following command to check that all the modules are running correctly — they may take a few minutes to stabilise, so you may need to run this command again:
```
kubectl get po --all-namespaces
```
Run the 10_install_nfs.sh script. This installs the software needed for a network file system to run on the machine.
```
./10_install_nfs.sh
```
Run the 11_create_nfs.sh script. You will be prompted to enter the path of the parameter file again. This sets up a network file system on your machine to allow communication between containers in BeetleboxCI.
```
./11_create_nfs.sh
```
Run the 40_create_certs. script. You will be prompted to enter the path of the parameter file. This creates the SSL certificates needed to set up the local container registry.
```
./40_create_certs.sh
```
Run the 41_create_registry.sh script. You will be prompted to enter the path of the parameter file.
```
./41_create_registry.sh
```
Run the 60_install_app.sh script. You will be prompted to enter the path of the parameter file. This builds and starts the BeetleboxCI application on the Kubernetes cluster. During the installation, the parameter file will be copied to the Beetlebox install path (default: /mnt/beetleboxci)
```
./60_install_app.sh
```

To access the application within the (virtual) machine where you installed BeetleboxCI, open a web browser and navigate to http://127.0.0.1:32767. If you need to access the BeetleboxCI web application from another machine (on the same local network) other than the control plane node, then you will need to replace 127.0.0.1 with the IP address or hostname of the control plane node.
If you are accessing the app for the first time, you will be prompted to create a superuser account. You can log in once you create the superuser account.

Upgrading BeetleboxCI

You can download the latest version of the installer here: BeetleboxCI.

The tarball contains a self-extracting executable that can be used to upgrade to a newer version, and the steps to upgrade are below:

Extract the tarball using a terminal and enter the folder that gets extracted:
```
tar -xzvf BeetleboxCIv*.tar.gz
cd dist_self_extracting/
```
Run the ./BeetleboxCIv*.run file. You will be prompted to enter the path of the parameter file. You can now locate this parameter file in the Beetlebox installation directory (default: /mnt/beetleboxci).
```
./BeetleboxCIv*.run
```
Run the following command to check that all the modules are running correctly — they may take a few minutes to stabilise, so you may need to run this command again:
```
kubectl get po --all-namespaces
```
You are now ready to use the latest version of BeetleboxCI.

Manual Installation

In some cases, you may already have a server with Kubernetes installed on it, and in other cases, the exact dependencies required by the automated scripts may not be available. In this case, please get in touch with us to discuss.