Overhaul .forgejo meta-repo — workflows, sync, docs, and security #11

New issue

Open

opened 2026-06-23 22:38:11 -06:00 by Xeplosion · 0 comments

Xeplosion commented

2026-06-23 22:38:11 -06:00

Owner

Copy link

Type of chore

Other

What needs doing and why?

A full audit of the docs, main, and template-script branches surfaced
broken behavior, two conflicting architectural models, weak/inconsistent
comments, and undocumented security choices. This is the umbrella issue tracking
the cleanup. Canonical rule: the infrastructure docs win over the README
wherever they conflict.

Tags below: [decision] needs a call · [bug] broken · [docs] · [chore]
formatting/naming/comments · [feature] new.

1. Decisions

[decision] Deploy logic lives in git (orchestrate from the hub
runner), not in a root-owned server-side deploy.sh. Accepts the security
tradeoff below; deployments remain per-implementation via in-repo
service-start.sh / healthcheck.sh / xegames.config.yml.
[decision] .template-sync.yml derives its file list from
sync-config.yml at runtime rather than hand-maintaining a per-repo
paths: list. (Examine directory-entry expansion for ISSUE_TEMPLATE/.)
[decision] create-labels.sh — write it (read labels.yml, upsert via
the Forgejo labels API) or strip the references in LABELS.md /
labels.yml.
[decision] Whether the bespoke/legacy hub services (hub-nginx, hub-zulip,
…) migrate to the in-git deploy model too, or keep their forced-command
deploy.sh for now. Document whichever.

2. Rewrite all workflows from scratch

Comments and conventions on the existing workflows are poor across the board.
Rather than patch, rewrite each with consistent structure, accurate behavior,
and comments that explain what each step does and why.

[chore] Rewrite with the new names + descriptive name: fields:
- sync-shared.yml → sync-shared-files.yml — "Sync Shared Files → Template Branches (on push to main)"
- sync-shared-manual.yml → sync-shared-files-manual.yml — "Sync Shared Files → Template Branches (manual, one branch or all)"
- sync-templates.yml → publish-template-branch.yml — "Publish Template Branch → Template Repo (on push)"
- sync-templates-manual.yml → publish-template-branch-manual.yml — "Publish Template Branch → Template Repo (manual, one or all)"
- sync-full.yml → sync-everything.yml — "Full Sync — Shared Files + All Template Repos (manual)"
- sync-implementers.yml → template-sync.yml — "Template Sync — Pull Shared Files from .forgejo (scheduled)"
- sync-implementers-manul.yml → template-sync-manual.yml — "Template Sync — Pull Shared Files from .forgejo (manual)"
- deploy-production.yml (keep) — "Deploy to Production (push to main → target server)"
- deploy-test.yml (keep) — "Deploy to Test (push to dev → test server)"
- rollback.yml (keep) — "Rollback Deployment (manual, or auto on failed healthcheck)"
- healthcheck.yml (keep) — "Scheduled Healthcheck (production + test)"
- release.yml (keep) — "Release — Notes + Artifacts (on tag v*)"
- ci-script.yml (keep) — "CI — Script / Infra (runs validate.sh)"
- promote.yml (keep) — "Promote Mirror → Main (mirror-pull repos)"
[chore] Every workflow gets a header block (purpose, trigger, where it
runs) and inline comments on non-obvious steps.
Behavioral requirements the rewrite must satisfy (these are the old bugs):
[bug] Deploy/ops workflows run on runs-on: hub (no ubuntu-latest
runner exists).
[bug] Host values come from vars.{VPS_HOST,HUB_HOST,SSH_PORT} (org
variables), SSH creds from per-repo secrets.{SSH_USER,SSH_PRIVATE_KEY}.
[bug] Single deploy-target source: the DEPLOY_TARGET secret everywhere
(drop .deploy.target from xegames.config.yml).
[bug] template-sync workflows call scripts/template-sync.sh, and the
manual variant's PR head matches the script (template-sync/auto).
[feature] sync-shared-files-manual.yml takes a branch input (one
branch or all) replacing the dead full_sync boolean; sync-everything.yml
calls it with the default all.
[bug] Workflows that use yq install it (deploy/rollback/healthcheck/
release currently assume it's present).
[chore] Standardize: yq install path, secrets vs env-var token usage,
name: spacing, indentation.

3. Sync system correctness

[bug] sync-config.yml: fix template-issuess typo; fix "master" →
"main" comment.
[bug] Reconcile service-start.sh / service-stop.sh sync targets
(README says script/node/python/php; config lists script/node only).
[bug] sync-shared-files.yml paths: trigger: fix CODEOWNERS →
SYNCED_CODEOWNERS; add scripts/template-sync.sh and
workflows/template-sync*.yml.
[bug] Add template-sync.yml, template-sync-manual.yml, and
scripts/template-sync.sh to sync-config.yml so implementer repos
actually receive the pull-sync mechanism.
[feature] template-sync.sh: derive paths from sync-config.yml at
runtime (decision #1), with directory expansion.
[docs] Document additive-only sync + the manual "remove a file from
template branches" procedure.
[feature] Add validate-template-branches.yml + required-files.yml
(fail if a branch is missing a required branch-specific file). (Drafts
produced; tune the per-branch lists.)
[feature] Disk-size validation workflow. Scheduled; one job per
runner — runs-on: hub checks zpool list / zfs list / df -h /,
runs-on: vps checks df -h /. Threshold breach → Zulip alert and/or open
an issue. No SSH (each runner checks its own host).

4. Documentation review & comment standardization

[docs] Validate every documentation file against current behavior.
Inventory (gather the ones not yet uploaded):
- main: README.md
- docs: CONVENTION.md, INFRASTRUCTURE-HUB.md, INFRASTRUCTURE-VPS.md,
ISSUES.md, LABELS.md, PULL_REQUESTS.md (+ a README.md if one exists)
- per template branch (need to gather): template-issues/README.md,
template-script/README.md, template-dotnet/README.md,
template-node/README.md, template-python/README.md,
template-php/README.md
- inline doc-comments: sync-config.yml, xegames.config.yml,
labels.yml, .template-sync.yml
[chore] Standardize comments in all non-synced files on main —
consistent REFERENCE STUB — NOT SYNCED banner + house style on
validate.sh, build.sh, ci-script.yml, promote.yml, labels.yml,
xegames.config.yml, .template-sync.yml.
[bug] Fix script header comments naming non-existent workflows
(validate.sh "ci.yml", service-start.sh / healthcheck.sh "deploy.yml").
[docs] Fix known inconsistencies:
- ISSUES.md: board column "Completed" → "Done"
- ISSUES.md: templates live at repo-root ISSUE_TEMPLATE/, not .forgejo/ISSUE_TEMPLATE/
- README branch-protection user Xeplosion → xeplosion
- labels.yml: Kind/Feature color 0075ca → #0075ca
- INFRASTRUCTURE-HUB.md: usable pool size "~930GB" vs "~806GB" → reconcile
- promote.yml undocumented in README (add or remove)
- Document SYNC_GPG_KEY, FORGEJO_READ_TOKEN, OWNER / REPO_NAME
[chore] CONVENTION.md set -e / #!/bin/bash rule vs stubs that omit
set -e and template-sync.sh using env bash — reconcile.
[chore] CRLF vs .editorconfig eol = lf — normalize to LF + add
.gitattributes.

5. Security documentation (priority)

[docs] Add a Security section to the main README explaining how the
security model drove the structure, covering:
- per-repo unique deploy keys (leaked key = one service only)
- nologin service users, home isolation, one sudo command each
- the in-git deploy tradeoff (repo-controlled commands; mitigated by branch
protection + required review + synced authoritative workflows)
- GPG-signed sync commits; per-repo RELEASE_TOKEN
- why org variables vs secrets are split the way they are
[docs] Rewrite README Org Secrets to the infra model (org vars:
SSH_PORT/VPS_HOST/HUB_HOST; org secrets: Zulip; per-repo secrets:
SSH_PRIVATE_KEY/SSH_USER/DEPLOY_TARGET/RELEASE_TOKEN).
[docs] Update INFRASTRUCTURE-HUB.md / INFRASTRUCTURE-VPS.md to reflect
the in-git deploy model for templated stacks; clarify forced-command
deploy.sh as the legacy/bespoke pattern (per decision #1).

Risk or impact

Rewriting the sync/publish workflows touches live automation — do it on a
branch, validate via manual dispatch against one template branch before
replacing the originals, and keep the old files until the new ones are proven.
The deploy-model decision changes the trust boundary on the servers; the
security-docs work must land alongside it so the rationale is recorded.
Several items depend on the open decisions in section 1.

Additional context

Infra docs (INFRASTRUCTURE-HUB.md / INFRASTRUCTURE-VPS.md, validated
2026-06-20) are canonical for architecture.
Per-branch template READMEs are not yet gathered — needed for the section 4
doc review.
Suggested slicing into child PRs/issues: (1) mechanical doc + comment fixes,
(2) workflow rewrite, (3) sync correctness + .template-sync.yml, (4) deletion
docs + validation + disk-size workflow, (5) security docs + infra reconciliation.

Checklist

I have described why this is worth doing now
I have noted any risk or impact on other work

### Type of chore Other ### What needs doing and why? A full audit of the `docs`, `main`, and `template-script` branches surfaced broken behavior, two conflicting architectural models, weak/inconsistent comments, and undocumented security choices. This is the umbrella issue tracking the cleanup. Canonical rule: **the infrastructure docs win** over the README wherever they conflict. Tags below: `[decision]` needs a call · `[bug]` broken · `[docs]` · `[chore]` formatting/naming/comments · `[feature]` new. --- #### 1. Decisions - [x] `[decision]` **Deploy logic lives in git** (orchestrate from the `hub` runner), not in a root-owned server-side `deploy.sh`. Accepts the security tradeoff below; deployments remain per-implementation via in-repo `service-start.sh` / `healthcheck.sh` / `xegames.config.yml`. - [x] `[decision]` **`.template-sync.yml` derives its file list from `sync-config.yml` at runtime** rather than hand-maintaining a per-repo `paths:` list. (Examine directory-entry expansion for `ISSUE_TEMPLATE/`.) - [ ] `[decision]` `create-labels.sh` — write it (read `labels.yml`, upsert via the Forgejo labels API) **or** strip the references in `LABELS.md` / `labels.yml`. - [ ] `[decision]` Whether the bespoke/legacy hub services (hub-nginx, hub-zulip, …) migrate to the in-git deploy model too, or keep their forced-command `deploy.sh` for now. Document whichever. #### 2. Rewrite all workflows from scratch Comments and conventions on the existing workflows are poor across the board. Rather than patch, rewrite each with consistent structure, accurate behavior, and comments that explain **what** each step does and **why**. - [x] `[chore]` Rewrite with the new names + descriptive `name:` fields: - `sync-shared.yml` → `sync-shared-files.yml` — "Sync Shared Files → Template Branches (on push to main)" - `sync-shared-manual.yml` → `sync-shared-files-manual.yml` — "Sync Shared Files → Template Branches (manual, one branch or all)" - `sync-templates.yml` → `publish-template-branch.yml` — "Publish Template Branch → Template Repo (on push)" - `sync-templates-manual.yml` → `publish-template-branch-manual.yml` — "Publish Template Branch → Template Repo (manual, one or all)" - `sync-full.yml` → `sync-everything.yml` — "Full Sync — Shared Files + All Template Repos (manual)" - `sync-implementers.yml` → `template-sync.yml` — "Template Sync — Pull Shared Files from .forgejo (scheduled)" - `sync-implementers-manul.yml` → `template-sync-manual.yml` — "Template Sync — Pull Shared Files from .forgejo (manual)" - `deploy-production.yml` (keep) — "Deploy to Production (push to main → target server)" - `deploy-test.yml` (keep) — "Deploy to Test (push to dev → test server)" - `rollback.yml` (keep) — "Rollback Deployment (manual, or auto on failed healthcheck)" - `healthcheck.yml` (keep) — "Scheduled Healthcheck (production + test)" - `release.yml` (keep) — "Release — Notes + Artifacts (on tag v*)" - `ci-script.yml` (keep) — "CI — Script / Infra (runs validate.sh)" - `promote.yml` (keep) — "Promote Mirror → Main (mirror-pull repos)" - [x] `[chore]` Every workflow gets a header block (purpose, trigger, where it runs) and inline comments on non-obvious steps. Behavioral requirements the rewrite must satisfy (these are the old bugs): - [x] `[bug]` Deploy/ops workflows run on `runs-on: hub` (no `ubuntu-latest` runner exists). - [x] `[bug]` Host values come from `vars.{VPS_HOST,HUB_HOST,SSH_PORT}` (org variables), SSH creds from per-repo `secrets.{SSH_USER,SSH_PRIVATE_KEY}`. - [x] `[bug]` Single deploy-target source: the `DEPLOY_TARGET` secret everywhere (drop `.deploy.target` from `xegames.config.yml`). - [x] `[bug]` `template-sync` workflows call `scripts/template-sync.sh`, and the manual variant's PR head matches the script (`template-sync/auto`). - [x] `[feature]` `sync-shared-files-manual.yml` takes a `branch` input (one branch or `all`) replacing the dead `full_sync` boolean; `sync-everything.yml` calls it with the default `all`. - [x] `[bug]` Workflows that use `yq` install it (deploy/rollback/healthcheck/ release currently assume it's present). - [x] `[chore]` Standardize: yq install path, `secrets` vs env-var token usage, `name:` spacing, indentation. #### 3. Sync system correctness - [ ] `[bug]` `sync-config.yml`: fix `template-issuess` typo; fix "master" → "main" comment. - [ ] `[bug]` Reconcile `service-start.sh` / `service-stop.sh` sync targets (README says script/node/python/php; config lists script/node only). - [x] `[bug]` `sync-shared-files.yml` `paths:` trigger: fix `CODEOWNERS` → `SYNCED_CODEOWNERS`; add `scripts/template-sync.sh` and `workflows/template-sync*.yml`. - [ ] `[bug]` Add `template-sync.yml`, `template-sync-manual.yml`, and `scripts/template-sync.sh` to `sync-config.yml` so implementer repos actually receive the pull-sync mechanism. - [x] `[feature]` `template-sync.sh`: derive paths from `sync-config.yml` at runtime (decision #1), with directory expansion. - [ ] `[docs]` Document additive-only sync + the manual "remove a file from template branches" procedure. - [ ] `[feature]` Add `validate-template-branches.yml` + `required-files.yml` (fail if a branch is missing a required branch-specific file). *(Drafts produced; tune the per-branch lists.)* - [x] `[feature]` **Disk-size validation workflow.** Scheduled; one job per runner — `runs-on: hub` checks `zpool list` / `zfs list` / `df -h /`, `runs-on: vps` checks `df -h /`. Threshold breach → Zulip alert and/or open an issue. No SSH (each runner checks its own host). #### 4. Documentation review & comment standardization - [ ] `[docs]` **Validate every documentation file** against current behavior. Inventory (gather the ones not yet uploaded): - `main`: `README.md` - `docs`: `CONVENTION.md`, `INFRASTRUCTURE-HUB.md`, `INFRASTRUCTURE-VPS.md`, `ISSUES.md`, `LABELS.md`, `PULL_REQUESTS.md` (+ a `README.md` if one exists) - per template branch (**need to gather**): `template-issues/README.md`, `template-script/README.md`, `template-dotnet/README.md`, `template-node/README.md`, `template-python/README.md`, `template-php/README.md` - inline doc-comments: `sync-config.yml`, `xegames.config.yml`, `labels.yml`, `.template-sync.yml` - [ ] `[chore]` **Standardize comments in all non-synced files on `main`** — consistent `REFERENCE STUB — NOT SYNCED` banner + house style on `validate.sh`, `build.sh`, `ci-script.yml`, `promote.yml`, `labels.yml`, `xegames.config.yml`, `.template-sync.yml`. - [ ] `[bug]` Fix script header comments naming non-existent workflows (`validate.sh` "ci.yml", `service-start.sh` / `healthcheck.sh` "deploy.yml"). - [ ] `[docs]` Fix known inconsistencies: - `ISSUES.md`: board column "Completed" → "Done" - `ISSUES.md`: templates live at repo-root `ISSUE_TEMPLATE/`, not `.forgejo/ISSUE_TEMPLATE/` - README branch-protection user `Xeplosion` → `xeplosion` - `labels.yml`: `Kind/Feature` color `0075ca` → `#0075ca` - `INFRASTRUCTURE-HUB.md`: usable pool size "~930GB" vs "~806GB" → reconcile - `promote.yml` undocumented in README (add or remove) - Document `SYNC_GPG_KEY`, `FORGEJO_READ_TOKEN`, `OWNER` / `REPO_NAME` - [ ] `[chore]` `CONVENTION.md` `set -e` / `#!/bin/bash` rule vs stubs that omit `set -e` and `template-sync.sh` using `env bash` — reconcile. - [ ] `[chore]` CRLF vs `.editorconfig` `eol = lf` — normalize to LF + add `.gitattributes`. #### 5. Security documentation (priority) - [ ] `[docs]` Add a **Security** section to the `main` README explaining how the security model drove the structure, covering: - per-repo unique deploy keys (leaked key = one service only) - `nologin` service users, home isolation, one `sudo` command each - the in-git deploy tradeoff (repo-controlled commands; mitigated by branch protection + required review + synced authoritative workflows) - GPG-signed sync commits; per-repo `RELEASE_TOKEN` - why org **variables** vs **secrets** are split the way they are - [ ] `[docs]` Rewrite README **Org Secrets** to the infra model (org vars: `SSH_PORT`/`VPS_HOST`/`HUB_HOST`; org secrets: Zulip; per-repo secrets: `SSH_PRIVATE_KEY`/`SSH_USER`/`DEPLOY_TARGET`/`RELEASE_TOKEN`). - [ ] `[docs]` Update `INFRASTRUCTURE-HUB.md` / `INFRASTRUCTURE-VPS.md` to reflect the in-git deploy model for templated stacks; clarify forced-command `deploy.sh` as the legacy/bespoke pattern (per decision #1). ### Risk or impact - Rewriting the sync/publish workflows touches live automation — do it on a branch, validate via manual dispatch against one template branch before replacing the originals, and keep the old files until the new ones are proven. - The deploy-model decision changes the trust boundary on the servers; the security-docs work must land alongside it so the rationale is recorded. - Several items depend on the open decisions in section 1. ### Additional context - Infra docs (`INFRASTRUCTURE-HUB.md` / `INFRASTRUCTURE-VPS.md`, validated 2026-06-20) are canonical for architecture. - Per-branch template READMEs are not yet gathered — needed for the section 4 doc review. - Suggested slicing into child PRs/issues: (1) mechanical doc + comment fixes, (2) workflow rewrite, (3) sync correctness + `.template-sync.yml`, (4) deletion docs + validation + disk-size workflow, (5) security docs + infra reconciliation. ### Checklist - [x] I have described why this is worth doing now - [x] I have noted any risk or impact on other work

Xeplosion added the

Rows
Columns