<!-- SPDX-FileCopyrightText: 2022 eclipse foundation SPDX-License-Identifier: EPL-2.0 -→

Gitlab Runner As Code - GRAC!

GRAC! is a infrastructure as code of gitlab runner in kubernetes infrastructure for projects hosted by the Eclipse Foundation at [gitlab.eclipse.org](https://gitlab.eclipse.org).

What is GRAC?

The goal of GRAC! is to make the administration of hundreds of gitlab runner instances on a Kubernetes based infrastructure bearable. It uses automation, templates and a configuration-as-code approach. GRAC! consist of a set of custom shell scripts and a [jsonnet](https://jsonnet.org) generation. It relies also on the Docker CLI, OpenShift CLI.

Why GRAC?

With the successfull implementation of jenkins instance as code at Eclipse Foundation under the name of [JIRO](https://github.com/eclipse-cbi/jiro), and over +250 instances in production, Gitlab Continuous Integration has been request by projects to be supported. So as a natural extension, GRAC! is Born, under the woods: same approach and technologies.

As JIRO, it’s expected a lower administration overhead for recurring task like setting up a new gitlab runner for projects/groups, configure, allowing ressources, handle specifics requests, etc.

Before starting

Instance notion

Each groups or projects in gitlab link to the instance reference in all grac documentation

Project: https://gitlab.eclipse.org/eclipsefdn/it/releng/gitlab-runner-service/gitlab-runner-as-code * Instance: eclipsefdn.it.releng.gitlab-runner-as-code

Group: https://gitlab.eclipse.org/eclipsefdn/it/releng * Instance: eclipsefnd.it.releng

Generation in directory: ./instances

This notion will affect generation in cluster * Ex: ef-grac-eclipsefnd-it-releng namespace

Kubernetes Context

All commands in GRAC depends on kube current-context.

Before launching a command, it is recommend to verify the current target : kubectl config current-context or kubectx

All generation files will be attach to this context

CBI Configuration

Create file ~/.cbi/config

{
  "kubeconfig": {
    "path": "~/.kube/config",
    "contextMapping": {
      "okd-c1": "okd-c1",
      "okd-c2": "okd-c2",
    }
  },
  "password-store": {
    "cbi-dir": "~/.password-store-cbi",
    "it-dir": "~/.password-store"
  },
  "gitlab.com-token": "XXXXXXXXXXXXXX",
 }

kubeconfig: allow to configure path to kubernetes configuration file. Default: ~/.kube/config

password-store: configuration of pass tool.

gitlab.com-token: Personal Acces Token from gitlab. Need to be create from profil UI gitlab. [PAT](https://gitlab.eclipse.org/-/profile/personal_access_tokens).

This key is dynamic and can be configure depending on your gitlab instance. ex: gitlab.eclipse.org-token.

password-store is only used to retrieve gitlab runner registration token for shared runner store from administration UI.

Kube configuration

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://api.xxxx.org
  name: okd-c1
- cluster:
    certificate-authority-data: ...
    server: https://api.xxxx.org
  name: okd-c2
contexts:
- context:
    cluster: okd-c1
    namespace: ef-grac-...
    user: sebastien.heurtematte@eclipse-foundation.org/xxxx
  name: okd-c1
- context:
    cluster: okd-c2
    namespace: ef-grac-...
    user: sebastien.heurtematte@eclipse-foundation.org/xxxx
  name: okd-c2
current-context: okd-c2
kind: Config
preferences: {}
users:
- name: sebastien.heurtematte@eclipse-foundation.org/xxxxx
  user:
    token: sha256~XXXXXXX
- name: sebastien.heurtematte@eclipse-foundation.org/xxxx
  user:
    token: sha256~XXXXXXX

Build docker image

docker build . --tag eclipsecbi/grac

Execute command

docker run \
    -v "${PWD}":/app \
    -v ~/.cbi/config:/home/grac/.cbi/config \
    -v ~/.kube/config:/home/grac/.kube/config \
    -v ~/.password-store-cbi:/home/grac/.password-store-cbi \
    -v /etc/timezone:/etc/timezone:ro \
    -v /etc/localtime:/etc/localtime:ro \
    --env GITLAB_URL=https://gitlab.com
    --env PERSONAL_ACCESS_TOKEN=XXXXXXXXXXX
    --env KUBECONFIG=/home/grac/.kube/config \
    --env PASSWORD_STORE_DIR=/home/grac/.password-store-cbi \
    --network host \
    eclipsecbi/grac make deploy instance=eclipsefdn.it.releng.gitlab-runner-as-code

or directly with the script grac.sh

./grac.sh deploy eclipsefdn.it.releng.gitlab-runner-as-code
All command execute depends on Kubernetes context. see current context : kubectl config current-context or kubectx

Tasks

| Task | Description | |-------------|--------------| | create | Create directory for the project/group under ./instance with by default config.jsonnet and grac.jsonnet configure | | genconfig | Generation of all jsonnet file under ./instance/project_name/env_name/target | | k8s | Generation of configmap which is based on templating processor, genconfig is also execute with this task | | deploy | Deploy all configuration to kubernetes cluster and restart deployment after a reconfiguration | | restart | Restart all deployment from kubernetes cluster | | delete | Delete all configuration from kubernetes cluster | | replay | Delete, regenerate an instance and redeploy | | reload | Regenerate an instance and redeploy | | clean | Delete target directory generate by k8s or genconfig tasks |

Tasks *-all

All tasks define have an equivalent with -all suffix. Ex: create-all, deploy-all, k8s-all, …​

They apply to all instances of the directory ./instances. And depends on the kubernetes context.

Add clean up pod with ttl

Kubernetes configuration file pod-cleanup-example.yml allow to define policy around gitlab runner pod espacially if they tay stuck for any reason. It’s based on this project [gitlab-runner-pod-cleanup](https://gitlab.com/gitlab-org/ci-cd/gitlab-runner-pod-cleanup).

Annotation should be added to pod:

grac configuration podAnnotations to add:

  kubernetes+:{
    podAnnotations+:{
      'pod-cleanup.gitlab.com/ttl':'6h',
    },
  },

use a namespace RegEx

A RegEx can be define to apply the policy on specific namespace:

ex:

  - name: POD_CLEANUP_KUBERNETES_NAMESPACES
    value: ef-grac*

grac configuration prefix to add:

  project+: {
    prefix: 'ef-grac',