Pipeline YAML Reference

This is the reference for the configuration of config.yaml files. The entire build, simulation and deployment process can be specified by the configuration file (config.yaml). The configuration file is stored in the directory .bbx/config.yaml in the the root folder of your repo. They are written in YAML syntax. If you are not familiar with YAML, please see this guide.

runners

Required. The runners that are used in this pipeline. Runners provide the execution environment for your jobs to run in. Multiple different runners maybe used in the same pipeline so that the optimal choice is used for a particular job.

Key	Required	Type	Description
<runner_name>	Y	Map	The ID of a runner.

runners.<runner_name>

Required. Each runner must have an ID, which is unique. <runner_name> must start with a letter and only contain alphanumeric characters.

Key	Required	Type	Description
image	Y	Map	The runner image used for this runner.

runners.<runner_name>.image

Required. The runner image used for a particular runner. Images provide the OS, the specification, the pre-installed libraries and tools for a runner. Names of runner images, as well as additional information may be found in the runner store.

Example

runners:
 first-runner:
  image: work1-virtualbox:5000/ubuntu-vitis-2020-1:latest

jobs

Required. A workflow is made of one or more jobs. Each job is provided with a runner environment and will by default run in parallel with another job, unless a depends field is present. BeetleboxCI does not enforce any limit on the number of jobs that may run in parallel (unless there is lack of hardware resources on the machine to run a job).

Key	Required	Type	Description
<job_name>	Y	Map	The name of the job.

jobs.<job_name>

Required. Each job must have an associated ID, which is unique. The job name is used as key and its map is used as configuration data. <job_name> must start with a letter and only contain alphanumerical characters.

Key	Required	Type	Description
current_working_directory	N	String	The current working directory of the runner (job container). If it is not set, the default value of ~/ will be used.
device	N	String	The name of the device from the device registry to be used in the job to be run.
depends	N	Map	Identifies the name of the jobs that must successfully complete before this job is run.
input	N	Map	Artifacts that are to be retrieved from the artifact store for use by the job.
output	N	Map	The files that are to be saved from the job environment and stored as artifacts in the artifact store.
privileged	N	Boolean	Specifies whether the runner (job container) for the job runs with elevated privileges. This allows the runner to directly access hardware devices on the computer, such as USB and serial ports. Any job that requires communication with a registered device requires a privileged runner.
resource_spec	N	String	The amount of resources to allocate to this job.
runner	Y	String	The name of the runner to be used for this job.
steps	Y	List	Steps are the tasks that a job is to perform.

jobs.<job_name>.current_working_directory

Sets the current working directory of the image. When running jobs, Git repositories that are cloned will be automatically placed in this directory. If not set, the default value of ~/ will be used.

jobs:   
 first-job:
  current_working_directory: /tools/Xilinx/Vivado/2020.1/

This example will set the current working directory to the Vivado install path and will place the Git repository at this location.

jobs.<job_name>.depends

Depends are used to determine what jobs need to successfully complete for this job to start execution.

Key	Required	Type	Description
<dependency_name>	Y	String	The name of a dependency, on which <job_name> runs once the dependency has passed. <dependency_name> and <job_name> must be part of the same workflow.
<dependency_name>	Y	Map	The name of a dependency, on which <job_name> runs once the dependency meets a specified condition. <dependency_name> and <job_name> must be part of the same workflow.

Example

jobs:   
 first-job:
 second-job:
  depends:
   - first-job 
 third-job:
 second-job:
  depends:
   - first-job  
   - second-job:

In this example, first-job must complete successfully before second-job may start.

third-job must wait for both first-job and second-job.

jobs.<job_name>.depends.<dependency_name> (Map)

Required. The depends field, if it exists, must have at least one dependency.

Key	Required	Type	Description
when	Y	String	The condition which the <dependency_name> must meet in order for <job_name> to run.

Example

jobs:   
 first-job:
 second-job:
 third-job:
 fourth-job:
 fifth-job:
  depends:
   - first-job
   - second-job:
       when: on-pass
   - third-job:
       when: on-fail
   - fourth-job:
       when: on-completion       

In this example, before fifth-job can run:

• first-job must pass.
• second-job must pass.
• third-job must fail.
• fifth-job must complete (either pass or fail).

  If the conditions are not met, fifth-job will not run.

jobs.<job_name>.device

The name of the device from the device registry to be used in the job to be run.

Example

jobs:
 run-hw:
  device: raspberry-pi

jobs.<job_name>.input

input indicate the folder and files that are to be downloaded onto the runner before the job executes.

Key	Required	Type	Description
artifact	Y	List	The artifacts that are to be downloaded onto the runner.

jobs.<job_name>.input.artifact

Used to indicate the artifacts that are to be downloaded from the artifact store onto the runner before job execution.

Example

jobs:   
 first-job:
  input: 
   artifact: 
    - example1.zip
    - example2.zip

jobs.<job_name>.node_selector

Use to indicate which node of the cluster a particular job should run on. The node_selector value is a dict which much match the labels on a particular node, for a job to be placed on that node. To view the labels of a particular node, run the command kubectl describe node [name-of-node]. The labels will be listed in the form key=value, but in the config.yaml, it must be written as key: value. If this field is not specified, the job will automatically be placed onto any of the available nodes in the cluster.

Format

jobs:   
 first-job:
  node_selector: 
   key: value

Example

jobs:   
 first-job:
  node_selector: 
   kubernetes.io/hostname: name-of-node

jobs.<job_name>.output

This is used for folders and files that are to be uploaded onto the artifact store once the runner has finished execution.

Key	Required	Type	Description
artifact	Y	List	The artifacts that are to be downloaded onto the runner.

jobs.<job_name>.output.artifact

Identifies the artifacts that are to be uploaded onto the artifact store once the runner has finished execution.

Example

jobs:   
 first-job:
  output: 
   artifact: 
    - example1.zip
    - example2.zip

jobs.<job_name>.resource_spec

The amount of resources to allocate to this job. Defaults to small if unspecified.

Key	CPU	RAM	Type
micro	0.5	1GiB	String
small	1.0	4GiB	String
medium	2.0	8GiB	String
large	4.0	16GiB	String
xlarge	8.0	32GiB	String
1.5xlarge	12.0	48GiB	String
2xlarge	16.0	64GiB	String
3xlarge	24.0	96GiB	String
4xlarge	32.0	128GiB	String

Example

jobs:   
 first-job:
  resource_spec: micro

jobs.<job_name>.runner

Required. The runner that the job is to execute on. The runner provides the OS, specification, pre-installed libraries and tools for the job.

Example

jobs:   
 first-job:
  runner: first-runner

jobs.<job_name>.steps

Required. Each executes a series of tasks known as steps, which are able to execute commands on the runner. Each step runs in the same environment and has full access to the tools and filesystem of that runner. There are two types of steps: run and transfer. Every step must have one (but not both) of these types specified.

Key	Required	Type	Description
run	Y	Map	Specify a run type step.
transfer	Y	Map	Specify a transfer type step.

jobs.<job_name>.steps.run

Invoke one or more command-line programs through providing the command string. By default, commands are executed in a bash shell.

Key	Required	Type	Description
name	Y	String	The name of the step.
command	Y	String	The specific command to be run.
on-device	N	Boolean	Determines whether the command will be executed directly in the device terminal or not.
pass	N	String	For on-device steps only: The step immediately passes if this string is detected in the terminal output.
fail	N	String	For on-device steps only: The step immediately fails if this string is detected in the terminal output.

jobs.<job_name>.steps.run.name

Required. The name of the step to be displayed.

Example

jobs:   
 hello_world:
  steps:
   - run:
      name: Say Hello

jobs.<job_name>.steps.run.command

Required. The commands to be executed. By default command is executed on a bash shell.

Example

jobs:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"

jobs.<job_name>.steps.run.on-device

Required. Determines whether the command will be executed directly in the device terminal or not.

Example

jobs:   
 hello_world:
  device: raspberry-pi
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True

jobs.<job_name>.steps.run.pass

Required. For on-device steps only: The step immediately passes if this string is detected in the terminal output.

Example

jobs:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True
      pass: Hello World

jobs.<job_name>.steps.run.fail

Required. For on-device steps only: The step immediately fails if this string is detected in the terminal output.

Example

jobs:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True
      fail: Error

jobs.<job_name>.steps.transfer

Invoke one or more command-line programs through providing the command string. By default, commands are executed in a bash shell.

Key	Required	Type	Description
name	Y	String	The name of the step.
SOURCE	N	String	The location from where files are transferred. The SOURCE may also be hardcoded in the device configuration.
DESTINATION	N	String	The location to where files are transffered. The DESTINATION may also be hardcoded in the device configuration.

jobs.<job_name>.steps.transfer.name

Required. The name of the step to be displayed.

Example

jobs:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files

jobs.<job_name>.steps.transfer.SOURCE

Required. The location from where files are transferred. The SOURCE may also be hardcoded in the device configuration. The example shows files being copied from a runner (on the server) to a device.

Example

jobs:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files
      SOURCE: /PATH/TO/SOURCE/FOLDER/*
      DESTINATION: root@192.168.1.92:/PATH/TO/DESTINATION/FOLDER
      

jobs.<job_name>.steps.transfer.DESTINATION

Required. The location to where files are transfered. The DESTINATION may also be hardcoded in the device configuration. The example code files being copied from a device to the runner (on the server)

Example

jobs:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files
      SOURCE: root@192.168.1.92:/PATH/TO/SOURCE/FOLDER/*
      DESTINATION: /PATH/TO/DESTINATION/FOLDER

jobs.<job_name>.volumes

Use to indicate if any directories from the node which a job runs on should be mounted as a read-only volume inside the job container. This is useful for cases where jobs must access large amounts of data or very large (>50GB) software programs. Volume names cannot contain underscores.

Example

jobs:   
 random-job:
  volumes:
   - mount:
      name: volume1
      path: /tmp/firstvolume
   - mount:
      name: volume2
      path: /tmp/secondvolume  
   - mount:
      name: volume3
      path: /opt/thirdvolume        

post-hooks

A workflow may contain post hooks, which are run at the end of a workflow to perform various cleanup operations. Each post-hook is provided with a runner environment. A workflow may contain at most two post hooks.

Key	Required	Type	Description
<post_hook_name>	Y	Map	The name of the post hook.

post-hooks.<post_hook_name>

Required. Each job must have an associated ID, which is unique. The post hook name is used as key and its map is used as configuration data. <post_hook_name> must start with a letter and only contain alphanumerical characters.

Key	Required	Type	Description
current_working_directory	N	String	The current working directory of the runner (job container). If it is not set, the default value of ~/ will be used.
device	N	String	The name of the device from the device registry to be used in the job to be run.
input	N	Map	Artifacts that are to be retrieved from the artifact store for use by the job.
output	N	Map	The files that are to be saved from the job environment and stored as artifacts in the artifact store.
privileged	N	Boolean	Specifies whether the runner (job container) for the job runs with elevated privileges. This allows the runner to directly access hardware devices on the computer, such as USB and serial ports. Any job that requires communication with a registered device requires a privileged runner.
resource_spec	N	String	The amount of resources to allocate to this job.
runner	Y	String	The name of the runner to be used for this job.
steps	Y	List	Steps are the tasks that a job is to perform.

post-hooks.<post_hook_name>.current_working_directory

Sets the current working directory of the image. When running post hooks, Git repositories that are cloned will be automatically placed in this directory. If not set, the default value of ~/ will be used.

post-hooks:   
 first-hook:
  current_working_directory: /tools/Xilinx/Vivado/2020.1/

This example will set the current working directory to the Vivado install path and will place the Git repository at this location.

post-hooks.<post_hook_name>.device

The name of the device from the device registry to be used in the post hook to be run.

Example

post-hooks:
 run-hw:
  device: raspberry-pi

post-hooks.<post_hook_name>.input

input indicate the folder and files that are to be downloaded onto the runner before the post hook executes.

Key	Required	Type	Description
artifact	Y	List	The artifacts that are to be downloaded onto the runner.

post-hooks.<post_hook_name>.input.artifact

Used to indicate the artifacts that are to be downloaded from the artifact store onto the runner before post hook execution.

Example

post-hooks:   
 first-hook:
  input: 
   artifact: 
    - example1.zip
    - example2.zip

post-hooks.<post_hook_name>.node_selector

Use to indicate which node of the cluster a particular post hook should run on. The node_selector value is a dict which much match the labels on a particular node, for a post hook to be placed on that node. To view the labels of a particular node, run the command kubectl describe node [name-of-node]. The labels will be listed in the form key=value, but in the config.yaml, it must be written as key: value. If this field is not specified, the post hook will automatically be placed onto any of the available nodes in the cluster.

Format

post-hooks:   
 first-hook:
  node_selector: 
   key: value

Example

post-hooks:   
 first-hook:
  node_selector: 
   kubernetes.io/hostname: name-of-node

post-hooks.<post_hook_name>.output

This is used for folders and files that are to be uploaded onto the artifact store once the runner has finished execution.

Key	Required	Type	Description
artifact	Y	List	The artifacts that are to be downloaded onto the runner.

post-hooks.<post_hook_name>.output.artifact

Identifies the artifacts that are to be uploaded onto the artifact store once the runner has finished execution.

Example

post-hooks:   
 first-hook:
  output: 
   artifact: 
    - example1.zip
    - example2.zip

post-hooks.<post_hook_name>.resource_spec

The amount of resources to allocate to this psot hook. Defaults to small if unspecified.

Key	CPU	RAM	Type
micro	0.5	1GiB	String
small	1.0	4GiB	String
medium	2.0	8GiB	String
large	4.0	16GiB	String
xlarge	8.0	32GiB	String
1.5xlarge	12.0	48GiB	String
2xlarge	16.0	64GiB	String
3xlarge	24.0	96GiB	String
4xlarge	32.0	128GiB	String

Example

post-hooks:   
 first-hook:
  resource_spec: micro

post-hooks.<post_hook_name>.runner

Required. The runner that the post hook is to execute on. The runner provides the OS, specification, pre-installed libraries and tools for the post hook.

Example

post-hooks:   
 first-hook:
  runner: first-runner

post-hooks.<post_hook_name>.steps

Required. Each executes a series of tasks known as steps, which are able to execute commands on the runner. Each step runs in the same environment and has full access to the tools and filesystem of that runner. There are two types of steps: run and transfer. Every step must have one (but not both) of these types specified.

Key	Required	Type	Description
run	Y	Map	Specify a run type step.
transfer	Y	Map	Specify a transfer type step.

post-hooks.<post_hook_name>.steps.run

Invoke one or more command-line programs through providing the command string. By default, commands are executed in a bash shell.

Key	Required	Type	Description
name	Y	String	The name of the step.
command	Y	String	The specific command to be run.
on-device	N	Boolean	Determines whether the command will be executed directly in the device terminal or not.
pass	N	String	For on-device steps only: The step immediately passes if this string is detected in the terminal output.
fail	N	String	For on-device steps only: The step immediately fails if this string is detected in the terminal output.

post-hooks.<post_hook_name>.steps.run.name

Required. The name of the step to be displayed.

Example

post-hooks:   
 hello_world:
  steps:
   - run:
      name: Say Hello

post-hooks.<post_hook_name>.steps.run.command

Required. The commands to be executed. By default command is executed on a bash shell.

Example

post-hooks:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"

post-hooks.<post_hook_name>.steps.run.on-device

Required. Determines whether the command will be executed directly in the device terminal or not.

Example

post-hooks:   
 hello_world:
  device: raspberry-pi
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True

post-hooks.<post_hook_name>.steps.run.pass

Required. For on-device steps only: The step immediately passes if this string is detected in the terminal output.

Example

post-hooks:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True
      pass: Hello World

post-hooks.<post_hook_name>.steps.run.fail

Required. For on-device steps only: The step immediately fails if this string is detected in the terminal output.

Example

post-hooks:   
 hello_world:
  steps:
   - run:
      name: Say Hello
      command: echo "Hello World"
      on-device: True
      fail: Error

post-hooks.<post_hook_name>.steps.transfer

Invoke one or more command-line programs through providing the command string. By default, commands are executed in a bash shell.

Key	Required	Type	Description
name	Y	String	The name of the step.
SOURCE	N	String	The location from where files are transferred. The SOURCE may also be hardcoded in the device configuration.
DESTINATION	N	String	The location to where files are transffered. The DESTINATION may also be hardcoded in the device configuration.

post-hooks.<post_hook_name>.steps.transfer.name

Required. The name of the step to be displayed.

Example

post-hooks:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files

post-hooks.<post_hook_name>.steps.transfer.SOURCE

Required. The location from where files are transferred. The SOURCE may also be hardcoded in the device configuration. The example shows files being copied from a runner (on the server) to a device.

Example

post-hooks:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files
      SOURCE: /PATH/TO/SOURCE/FOLDER/*
      DESTINATION: root@192.168.1.92:/PATH/TO/DESTINATION/FOLDER
      

post-hooks.<post_hook_name>.steps.transfer.DESTINATION

Required. The location to where files are transfered. The DESTINATION may also be hardcoded in the device configuration. The example code files being copied from a device to the runner (on the server)

Example

post-hooks:   
 hello_world:
  device: raspberry-pi
  steps:
   - transfer:
      name: Copy files
      SOURCE: root@192.168.1.92:/PATH/TO/SOURCE/FOLDER/*
      DESTINATION: /PATH/TO/DESTINATION/FOLDER

post-hooks.<post_hook_name>.volumes

Use to indicate if any directories from the node which a post hook runs on should be mounted as a read-only volume inside the job container. This is useful for cases where post hooks must access large amounts of data or very large (>50GB) software programs. Volume names cannot contain underscores.

Example

post-hooks:   
 random-job:
  volumes:
   - mount:
      name: volume1
      path: /tmp/firstvolume
   - mount:
      name: volume2
      path: /tmp/secondvolume  
   - mount:
      name: volume3
      path: /opt/thirdvolume        

workflows

Required. A workflow is used to orchestrate and manage jobs.

Key	Required	Type	Description
<workflow_name>	Y	Map	The name of a workflow.

workflows.<workflow_name>

Required. The unique name of a workflow. <workflow_name> must start with a letter and only contain alphanumeric characters.

Key	Required	Type	Description
jobs	Y	List	The jobs that are to be run in this workflow
post-hooks	N	List	The tasks that are to be run optionally after the completion of a workflow
triggers	N	List	The events that cause the workflow to run

workflows.<workflow_name>.jobs

Required. The jobs that are to be run in this workflow.

workflows:
 workflow-example:
  jobs:
   - job-example-1
   - job-example-2

workflows.<workflow_name>.post-hooks

The post-hooks that are to be run optionally after the completion of a workflow.

Key	Required	Type	Description
<post_hook_name>	Y	Map	The name of a post hook that will be run at the end of a workflow.

workflows.<workflow_name>.post-hooks.<post_hook_name> (Map)

Required. The name of the post hook. <post_hook_name> must start with a letter and only contain alphanumeric characters.

Key	Required	Type	Description
when	Y	List	The conditions under which this post hook will run. Acceptable values are on-workflow-pass or on-workflow-fail.

workflows:
 workflow-example:
  jobs:
   - hello-world1
   - hello-world2
  post-hooks:
   - hello-world-hook1:
      when: on-workflow-pass
   - hello-world-hook2:
      when: on-workflow-fail

In this example, hello-world-hook1 will run if the jobs in the workflow pass, and hello-world-hook2 will run if any of the jobs in the workflow fail, as the workflow is deemed to have failed.

workflows.<workflow_name>.triggers

The events that cause the workflow to run. If not specified, the workflow defaults to a manual trigger.

Key	Description
manual	The workflow will run when the user presses the run button in the webapp. If no triggers are specified, the workflow can still be run manually using the run button.
pull-request	The workflow will run whenever there is a pull re the branch of the repository which the config file is in.
push	The workflow will run whenever there is a push to the branch of the repository which the config file is in.
schedule	The workflow will run periodically according to the schedule specified by the cron string. The cron string uses the standard cron syntax.

workflows:
 workflow-example:
  jobs:
   - job-example-1
   - job-example-2
  triggers:
   - manual
   - push
   - schedule:
      cron: "* * * * *"

Reserved keywords

There are certain strings that should not be used in a BeetleboxCI config file, as these are used internally by BeetleboxCI. Using these keywords in your config file may cause errors or unpredictable behaviour. The keywords are listed in the table below:

Key
__line__