Implement remaining server infrastructure changes #9

New issue

Open

opened 2026-06-15 17:50:19 -06:00 by Xeplosion · 1 comment

Xeplosion commented

2026-06-15 17:50:19 -06:00

Owner

Copy link

Type of chore

Security hardening

What needs doing and why?

Following the infrastructure and CI/CD security design work, a number of changes
need to be implemented on both servers and in Forgejo that are not yet in place.
The current setup uses a single shared cicd user and org-level SSH credentials
which does not match the agreed security model.

Forgejo — secrets and runners

Remove SSH_PRIVATE_KEY and SSH_USER from org-level secrets
Add SSH_PRIVATE_KEY and SSH_USER as repo-level secrets on each service repo
Register hub runner in Forgejo with label hub
Register VPS runner in Forgejo with label vps
Verify runner labels route correctly via runs-on: in workflows

VPS (node-i440fx)

Create svc-nginx user (/usr/sbin/nologin, no sudo except systemctl reload nginx)
Create svc-bungeecord user (/usr/sbin/nologin, no sudo except systemctl restart bungeecord)
Create forgejo-runner user (no SSH, no sudo, owns runner workspace only)
Generate deploy keypair for node-nginx — register public key in svc-nginx authorized_keys with command= restriction
Generate deploy keypair for node-bungeecord — register public key in svc-bungeecord authorized_keys with command= restriction
Write root-owned deploy.sh for svc-nginx (git pull + systemctl reload nginx)
Write root-owned deploy.sh for svc-bungeecord (git pull + systemctl restart bungeecord)
Add sudoers.d/svc-nginx entry
Add sudoers.d/svc-bungeecord entry
Add private deploy keys as SSH_PRIVATE_KEY in node-nginx and node-bungeecord repo secrets
Update SSH_USER in each repo to match service username

Hub (hub-x9scl-f)

Create svc-nginx user
Create svc-forgejo user
Create svc-mc-survival user
Create svc-mc-modded user
Create svc-zulip user
Create svc-uptime-kuma user
Create cicd user (no SSH, no sudo)
Generate deploy keypair per service — register public keys with command= restrictions
Write root-owned deploy.sh per service user
Add sudoers.d entry per service user
Add private deploy keys as SSH_PRIVATE_KEY in each service repo secret
Update SSH_USER in each repo to match service username

Forgejo migration (VPS → Hub)

Install Forgejo binary on hub
Stop Forgejo on VPS
Copy /var/lib/forgejo/ data to hub
Start and verify Forgejo on hub
Update DNS — git.xegames.online points to hub (or VPS with proxy to hub)
Update git.conf on VPS nginx to proxy to hub over WireGuard
Re-register VPS runner to point at hub-hosted Forgejo URL
Verify both runners appear in Forgejo Actions admin panel
Remove Forgejo from VPS once confirmed stable on hub

`.forgejo` repo

Create docs branch
Push README.md, CONVENTION.md, ISSUES.md, LABELS.md, PULL_REQUESTS.md, INFRASTRUCTURE-HUB.md, INFRASTRUCTURE-VPS.md to docs branch
Set docs as the default branch
Add .forgejo/CODEOWNERS (.* @xeplosion) to main branch
Add .forgejo/CODEOWNERS to each template branch
Enable branch protection on docs branch (disable direct push, 1 required approval)
Verify branch protection is active on main and all template branches
Update documentation to reflect change from forgejo-runner user to cicd user

Risk or impact

High impact — the Forgejo migration has a downtime window and requires careful
sequencing to avoid losing runner registrations or repo data. The per-service
user changes affect all CI/CD deployments and must be coordinated with secret
rotation in Forgejo.

Do the Forgejo migration during a low-activity window. Have a rollback plan:
keep the VPS Forgejo instance stopped but intact until the hub instance is
confirmed stable for at least 48 hours.

Additional context

Security model: INFRASTRUCTURE-HUB.md, INFRASTRUCTURE-VPS.md on docs branch
Deploy key pattern and command= restriction details: INFRASTRUCTURE-VPS.md
Runner topology and runs-on: label routing: INFRASTRUCTURE-HUB.md

Checklist

I have described why this is worth doing now
I have noted any risk or impact on other work

### Type of chore Security hardening ### What needs doing and why? Following the infrastructure and CI/CD security design work, a number of changes need to be implemented on both servers and in Forgejo that are not yet in place. The current setup uses a single shared `cicd` user and org-level SSH credentials which does not match the agreed security model. ### Forgejo — secrets and runners - [x] Remove `SSH_PRIVATE_KEY` and `SSH_USER` from org-level secrets - [ ] Add `SSH_PRIVATE_KEY` and `SSH_USER` as repo-level secrets on each service repo - [x] Register hub runner in Forgejo with label `hub` - [x] Register VPS runner in Forgejo with label `vps` - [ ] Verify runner labels route correctly via `runs-on:` in workflows ### VPS (node-i440fx) - [x] Create `svc-nginx` user (`/usr/sbin/nologin`, no sudo except `systemctl reload nginx`) - [x] Create `svc-bungeecord` user (`/usr/sbin/nologin`, no sudo except `systemctl restart bungeecord`) - [x] Create `forgejo-runner` user (no SSH, no sudo, owns runner workspace only) - [x] Generate deploy keypair for `node-nginx` — register public key in `svc-nginx` `authorized_keys` with `command=` restriction - [x] Generate deploy keypair for `node-bungeecord` — register public key in `svc-bungeecord` `authorized_keys` with `command=` restriction - [ ] Write root-owned `deploy.sh` for `svc-nginx` (`git pull` + `systemctl reload nginx`) - [ ] Write root-owned `deploy.sh` for `svc-bungeecord` (`git pull` + `systemctl restart bungeecord`) - [x] Add `sudoers.d/svc-nginx` entry - [x] Add `sudoers.d/svc-bungeecord` entry - [ ] Add private deploy keys as `SSH_PRIVATE_KEY` in `node-nginx` and `node-bungeecord` repo secrets - [ ] Update `SSH_USER` in each repo to match service username ### Hub (hub-x9scl-f) - [x] Create `svc-nginx` user - [x] Create `svc-forgejo` user - [x] Create `svc-mc-survival` user - [x] Create `svc-mc-modded` user - [x] Create `svc-zulip` user - [x] Create `svc-uptime-kuma` user - [x] Create `cicd` user (no SSH, no sudo) - [x] Generate deploy keypair per service — register public keys with `command=` restrictions - [ ] Write root-owned `deploy.sh` per service user - [x] Add `sudoers.d` entry per service user - [ ] Add private deploy keys as `SSH_PRIVATE_KEY` in each service repo secret - [ ] Update `SSH_USER` in each repo to match service username ### Forgejo migration (VPS → Hub) - [x] Install Forgejo binary on hub - [x] Stop Forgejo on VPS - [x] Copy `/var/lib/forgejo/` data to hub - [x] Start and verify Forgejo on hub - [x] Update DNS — `git.xegames.online` points to hub (or VPS with proxy to hub) - [x] Update `git.conf` on VPS nginx to proxy to hub over WireGuard - [x] Re-register VPS runner to point at hub-hosted Forgejo URL - [x] Verify both runners appear in Forgejo Actions admin panel - [x] Remove Forgejo from VPS once confirmed stable on hub ### `.forgejo` repo - [x] Create `docs` branch - [x] Push `README.md`, `CONVENTION.md`, `ISSUES.md`, `LABELS.md`, `PULL_REQUESTS.md`, `INFRASTRUCTURE-HUB.md`, `INFRASTRUCTURE-VPS.md` to `docs` branch - [x] Set `docs` as the default branch - [x] Add `.forgejo/CODEOWNERS` (`.* @xeplosion`) to `main` branch - [x] Add `.forgejo/CODEOWNERS` to each template branch - [x] Enable branch protection on `docs` branch (disable direct push, 1 required approval) - [x] Verify branch protection is active on `main` and all template branches - [ ] Update documentation to reflect change from `forgejo-runner` user to `cicd` user ### Risk or impact High impact — the Forgejo migration has a downtime window and requires careful sequencing to avoid losing runner registrations or repo data. The per-service user changes affect all CI/CD deployments and must be coordinated with secret rotation in Forgejo. Do the Forgejo migration during a low-activity window. Have a rollback plan: keep the VPS Forgejo instance stopped but intact until the hub instance is confirmed stable for at least 48 hours. ### Additional context - Security model: `INFRASTRUCTURE-HUB.md`, `INFRASTRUCTURE-VPS.md` on `docs` branch - Deploy key pattern and `command=` restriction details: `INFRASTRUCTURE-VPS.md` - Runner topology and `runs-on:` label routing: `INFRASTRUCTURE-HUB.md` ### Checklist - [x] I have described why this is worth doing now - [x] I have noted any risk or impact on other work

Xeplosion added the

Rows
Columns