Semaphore UI¶

This directory contains the Rancher Fleet bundle that deploys Semaphore UI for running the GEM Ansible playbooks from inside the cluster.

Semaphore is the operator UI and task runner. It is not the source of truth for the automation logic. Playbooks, roles, inventories, and group variables live in the Ansible repository configured by ansibleRepo in values.yaml; this chart only deploys Semaphore and bootstraps Semaphore projects/templates that point at that repository.

Deployment Summary¶

• Fleet bundle: fleet/semaphore
• Namespace bundle: fleet/semaphore/namespace
• Namespace: semaphore
• Fleet release name: semaphore-ui
• Chart name: semaphore-ui
• Ingress: https://semaphore.gem.mintfit.hamburg
• Image: semaphoreui/semaphore:v2.18.9
• Database: SQLite at /etc/semaphore/semaphore.sqlite
• Storage: Longhorn PVC, 5Gi, ReadWriteOnce
• Admin password Secret: semaphore-admin, key admin-password
• Bootstrap: Helm post-install/post-upgrade Job

Fleet applies this local Helm chart through fleet.yaml:

defaultNamespace: semaphore
name: semaphore
dependsOn:
  - name: semaphore-namespace

helm:
  chart: .
  releaseName: semaphore-ui
  waitForJobs: true
  valuesFiles:
    - ./values.yaml

waitForJobs: true is important because the bootstrap Job is part of the deployment contract. A Fleet rollout is only healthy when the Semaphore pod is ready and the bootstrap Job has completed.

What Gets Deployed¶

The chart renders these Kubernetes resources:

• Deployment with one Semaphore UI replica.
• Longhorn-backed PVC used for Semaphore state and working repository clones.
• ClusterIP Service on port 3000.
• nginx Ingress with cert-manager TLS.
• ServiceAccount, ClusterRole, and ClusterRoleBinding for Kubernetes node inventory access.
• Opaque chart Secret with the Ansible repository URL, branch, deploy key, kubeconfig, and per-project environment data.
• Post-install/post-upgrade bootstrap Job.

The Deployment uses strategy: Recreate and replicaCount: 1 because the PVC is ReadWriteOnce and SQLite is not a multi-writer database. Do not scale this deployment beyond one replica unless Semaphore is moved to an external database that supports it.

The pod is pinned to the administration node through node affinity:

mintfit.io/group.administration_node: "true"

Some maintenance playbooks intentionally exclude the administration node so that Semaphore is not disrupted while it is orchestrating work.

Runtime Layout¶

The PVC is mounted into two separate sub-paths:

kubectl and helm are downloaded by init containers into an emptyDir at /tools. The main container prepends /tools and the Semaphore Ansible virtual environment to PATH, so bootstrapped playbooks can call ansible-playbook, kubectl, and helm.

The chart also mounts a kubeconfig at:

/etc/semaphore-kube/config

Each Semaphore project receives KUBECONFIG=/etc/semaphore-kube/config and its own KUBE_CONTEXTS value. The dynamic Ansible inventory uses those contexts to decide which Kubernetes nodes belong to that project.

Before First Deploy¶

Create the namespace and the stable admin password Secret before Fleet applies this bundle. The password is intentionally not generated by Fleet, because the password stored in Kubernetes and the password stored inside Semaphore's database must not drift on repeated syncs.

kubectl --context gem-cluster-01 create namespace semaphore \
  --dry-run=client -o yaml \
| kubectl --context gem-cluster-01 apply -f -

kubectl --context gem-cluster-01 -n semaphore create secret generic semaphore-admin \
  --from-literal=admin-password='<choose-a-strong-password>'

Generate a strong password locally:

openssl rand -base64 32

If the Secret already exists, leave it in place. Rotating only the Kubernetes Secret does not automatically rotate the password inside Semaphore's SQLite database.

Configuration¶

Most deployment settings live in values.yaml.

Important values:

Treat the values that contain deploy keys, kubeconfig tokens, SMTP passwords, webhook URLs, or other credentials as secret material. Do not add new plaintext secrets unless the repository policy explicitly allows it.

Bootstrapped Projects¶

The bootstrap Job is idempotent and runs after every Fleet/Helm install or upgrade. It logs into Semaphore through the local service API and reconciles projects, repositories, deploy keys, inventories, environments, views, and task templates.

The default projects are:

Template sets:

• cluster: Kubernetes node management for an RKE2 workload cluster.
• cluster-mgmt: cluster plus management-cluster tasks such as Rancher Manager upgrade and management control-plane network migration.
• standalone: safer subset for non-cluster hosts; omits Kubernetes service restart, drain, Rancher join, and cluster upgrade templates.

To add or change a project, edit bootstrap.projects and let Fleet sync the bundle. Each project needs:

• name: UI project name.
• slug: lowercase stable identifier used for Secret keys and logs.
• kubeContexts: comma-separated kubeconfig contexts, or empty for no cluster nodes.
• templateSet: cluster, cluster-mgmt, or standalone.
• extraEnv: optional per-project environment variables.

Inventory Model¶

Semaphore uses the Ansible repository inventory at inventory/.

For cluster projects, the dynamic inventory script queries Kubernetes nodes for every context in KUBE_CONTEXTS before each run. This means adding, removing, or relabeling Kubernetes nodes is reflected on the next task execution; there is no separate inventory refresh CronJob in this chart.

For the standalone project, KUBE_CONTEXTS is empty, so no Kubernetes nodes are emitted. Only static/manual inventory entries are available. The default standalone project sets:

extraEnv:
  INVENTORY_INCLUDE_BASTION: "1"

This exposes the bastion/manual hosts only where they are intended to be used.

Verifying A Rollout¶

Check Fleet first in Rancher. The bundle should become active after the Helm release and bootstrap Job complete.

Useful cluster checks:

kubectl --context gem-cluster-01 -n semaphore get deploy,pod,svc,ingress,pvc
kubectl --context gem-cluster-01 -n semaphore get jobs
kubectl --context gem-cluster-01 -n semaphore logs job/semaphore-ui-bootstrap

Depending on Helm's generated fullname, the bootstrap Job may be named semaphore-ui-bootstrap or semaphore-semaphore-ui-bootstrap. List Jobs in the namespace if the exact name differs:

kubectl --context gem-cluster-01 -n semaphore get jobs

The bootstrap log should contain one header per project:

=== project 'GEM Cluster Ansible' ...
=== project 'GEM Mgmt Ansible' ...
=== project 'Standalone Nodes Ansible' ...

On first install it logs creating ...; on later syncs it usually logs updating ....

Check the application:

kubectl --context gem-cluster-01 -n semaphore rollout status deploy/semaphore-ui
kubectl --context gem-cluster-01 -n semaphore get ingress

Then open:

https://semaphore.gem.mintfit.hamburg

Log in as admin using the password from the semaphore-admin Secret.

Operator Workflow¶

1. Select the project for the cluster or host type you want to operate on.
2. Start with Check node SSH access after any inventory, key, kubeconfig, or network change.
3. Use Show node system summary or Inspect node configuration before maintenance.
4. For disruptive tasks, always add a runtime CLI arg such as --limit=<host-or-group>.
5. Prefer --check --diff for templates that support Ansible check mode.
6. Use survey variables for typed choices such as reboot confirmation, drain behavior, Rancher target version, or report sections.
7. Review task output before starting a broader run.

Good runtime Args examples:

--limit=gem-c01-w03
--check
--diff
--tags=packages
--extra-vars=auto_reboot=true
--extra-vars=drain_node=false

Prefer --limit=gem-c01-w03 as one argument. If Semaphore displays separate argument rows, --limit and gem-c01-w03 can also be entered as separate arguments. Avoid entering --limit gem-c01-w03 as one combined string.

Common Templates¶

Read-only templates:

• Check node SSH access
• Show node system summary
• Inspect node configuration
• Collect node diagnostic archive

Baseline maintenance:

• Apply baseline packages and MOTD
• Apply UFW firewall baseline
• Manage team user accounts
• Configure security-only auto updates
• Upgrade OS packages with node drain for cluster projects
• Upgrade OS packages for standalone hosts

Disruptive restarts:

• Restart container runtime
• Restart Kubernetes node service
• Restart node networking
• Reboot node with drain

Node lifecycle:

• Prepare node for RKE2
• Install approved kernel
• Configure static private network
• Join node to Rancher/RKE2
• Reset node for Rancher rejoin

Management-cluster only:

• Migrate management CP networking
• Upgrade Rancher Manager

High-risk templates include survey confirmations in addition to Args. For example, Reboot node with drain requires --limit=<single-node> and reboot_node_confirm=true.

Updating Semaphore Or Templates¶

For deployment-level changes:

1. Edit values.yaml or templates in this directory.
2. Commit and push the change.
3. Let Rancher Fleet sync the bundle.
4. Verify the Semaphore pod and bootstrap Job.

For task descriptions, survey variables, views, or template wiring, update templates/configmap.yaml. Manual edits in the Semaphore UI are useful for short-lived testing, but they are not durable chart state; the bootstrap Job can replace them during the next Fleet sync.

For playbook logic, roles, or inventory behavior, update the Ansible repository instead. Semaphore pulls that repository when tasks run.

Database Migration Note¶

Older Semaphore deployments used BoltDB. BoltDB is deprecated upstream and removed in Semaphore 2.19. This bundle uses SQLite and sets:

SEMAPHORE_MIGRATE_FROM_BOLTDB=/etc/semaphore/database.boltdb

If an old /etc/semaphore/database.boltdb file exists on the PVC, Semaphore can import it on first SQLite startup.

Troubleshooting¶

If the UI is unavailable:

kubectl --context gem-cluster-01 -n semaphore describe pod -l app.kubernetes.io/name=semaphore-ui
kubectl --context gem-cluster-01 -n semaphore logs deploy/semaphore-ui
kubectl --context gem-cluster-01 -n semaphore describe ingress

If bootstrap fails:

kubectl --context gem-cluster-01 -n semaphore get jobs,pods
kubectl --context gem-cluster-01 -n semaphore logs job/semaphore-ui-bootstrap

Common causes:

• semaphore-admin Secret missing or wrong key name.
• Kubeconfig does not contain a context referenced by bootstrap.projects.
• Ansible deploy key cannot clone ansibleRepo.url.
• ansibleAppVersion does not match the installed Semaphore image path.
• The PVC is not attachable to the node where the pod is scheduled.

If an Ansible task cannot find nodes, check the project environment in Semaphore and verify KUBE_CONTEXTS, then run the read-only SSH and system summary templates.