Server Manager

The Server Manager plugin provides a web UI for managing remote Joinery production servers. Operations include status checks, backups, database copies, and applying updates -- all from the admin interface at /admin/server_manager.

The system has two components:

• PHP plugin (plugins/server_manager/) -- admin UI, job creation, command generation
• Go agent (/home/user1/joinery-agent/) -- generic step executor that polls the job queue and runs commands via SSH

Quick Start

1. Install the plugin

The plugin is already in the plugins/ directory. From the admin panel:

1. Go to /admin/admin_plugins
2. Click Actions on "Server Manager" and choose Install
3. Click Actions again and choose Activate

2. Install and start the Go agent

Build

cd /home/user1/joinery-agent
make release VERSION=1.0.0

This compiles the binary and packages it into joinery-agent-installer.sh — a self-extracting script that handles both fresh installs and upgrades.

Install

sudo bash joinery-agent-installer.sh --verbose

This creates:

• /usr/local/bin/joinery-agent — the binary
• /etc/systemd/system/joinery-agent.service — systemd unit
• /etc/joinery-agent/joinery-agent.env — configuration (from example, first install only)

Configure (usually not needed)

The agent reads database credentials directly from Globalvars_site.php — no manual configuration required on a standard Joinery install.

The default config path is /var/www/html/joinerytest/config/Globalvars_site.php. If your install is at a different path, set it in the env file:

sudo nano /etc/joinery-agent/joinery-agent.env
# Set: JOINERY_CONFIG=/var/www/html/mysite/config/Globalvars_site.php

Other optional settings in the env file:

Start

sudo systemctl start joinery-agent
sudo systemctl status joinery-agent

The dashboard at /admin/server_manager should now show Agent Status: Online.

If anything is wrong, the agent logs to systemd journal:

journalctl -u joinery-agent -f

Common startup errors are self-explanatory — missing DB_NAME, wrong password, or plugin tables not installed. Each error message tells you exactly what to fix.

Upgrade

cd /home/user1/joinery-agent
make release VERSION=1.x.x
sudo bash joinery-agent-installer.sh --verbose

The installer auto-detects upgrades: stops the service, swaps the binary, restarts, and rolls back automatically if the new version fails to start.

3. Add managed nodes

Go to /admin/server_manager/node_add (or click Add Node on the dashboard). There are two ways to add nodes:

Auto-detect (recommended)

The Auto-Detect Joinery Servers panel scans a remote host for Joinery instances automatically. Enter:

1. SSH Host -- the server IP (e.g., 23.239.11.53)
2. SSH Key Path -- path to the private key on the control plane (defaults to /home/user1/.ssh/id_ed25519_claude)
3. Click Detect

Detected instances appear as cards with Add This Node buttons. Clicking one auto-fills the entire form below -- just click Add Node to save.

Auto-detect requires the Go agent to be running (it executes the SSH commands, not PHP).

Manual

Fill in the form fields directly:

Admin Pages

All pages are at /admin/server_manager/... and require permission level 10 (superadmin).

The UI is organized around a dashboard + node detail pattern. The dashboard shows the fleet overview; clicking a node opens a tabbed detail page with all operations for that node.

Node Detail Tabs

The node detail page (/admin/server_manager/node_detail?mgn_id=N&tab=...) has five tabs:

Dashboard Features

The dashboard shows:

• Agent Status -- online/offline indicator with version and last heartbeat time
• Managed Nodes -- cards with health-based status dots (green=healthy, yellow=warning, red=problem, gray=no data), key metrics, and action buttons
• Publish Upgrade -- build upgrade archives from control plane source code (node-independent)
• Recent Jobs -- latest 20 jobs across all nodes

• Red: Last check failed, disk > 90%, or PostgreSQL not accepting connections
• Yellow: Disk > 80% or load average > 5
• Green: All metrics healthy
• Gray: Never checked or no data

Job Types

Note on bare-metal nodes with user1 SSH: When a bare-metal install completes, install.sh disables root SSH and the node's mgn_ssh_user is automatically updated to user1. Subsequent jobs run as user1 with NOPASSWD sudo. All backup/restore commands that need root-level paths (e.g. /backups/) use sudo automatically.

One-Click Node Install

Dashboard → Install New Node opens a form that provisions a fresh Joinery site on an SSH-accessible server in a single click. Two modes:

• Fresh: empty Joinery site with default schema. Admin picks the domain. Default admin login is [email protected] / changeme123, with usr_force_password_change=true so the first login forces a new password.
• From Backup: fresh install + restore of a source node's DB and project files. Target inherits the source's domain — admin cuts over DNS after install. Use source admin credentials to log in.

The mgn_install_state column tracks the lifecycle: installing → NULL (success) or install_failed (failure). On failure, the node detail page surfaces a Retry Install button; the target must be cleaned manually (e.g. rm -rf /var/www/html/SITENAME) before retry because install.sh refuses to overwrite an existing site. Postgres passwords are auto-generated and stored in the target's Globalvars_site.php — Server Manager does not capture or display them.

Docker notes:

• The reverse proxy step (manage_domain.sh) is skipped when the domain is a bare IP address — a routable hostname is required for Apache ServerName-based virtual hosting. With an IP domain, the site is accessible directly on its mapped port.
• backup_project.sh requires rsync. The bare-metal and Docker install scripts install rsync as part of the essential packages (install.sh line ~948). Sites installed before this was added can install it manually with apt install rsync.
• After a Docker install, mgn_container_name is automatically recorded in the control plane DB so future jobs correctly use docker exec to reach the site.

SSL Management

SSL State

Each node tracks its TLS certificate state in mgn_ssl_state:

Automatic Detection

check_status jobs include an SSH step that checks for a Let's Encrypt cert under /etc/letsencrypt/live/{domain}/. JobResultProcessor updates mgn_ssl_state and stores ssl_domain, ssl_expiry_raw, and ssl_expiry_ts in mgn_last_status_data. State transitions:

• CERT_FOUND → sets state to active (from any prior state)
• CERT_MISSING → clears state to null only if currently null or active; never overwrites pending or failed

Manual Provisioning

The Overview tab shows an SSL Setup card when mgn_ssl_state is not active and the node has a domain in its site URL. The card:

1. Resolves the domain via DNS and shows whether it points to the node's host IP
2. Enables the Provision SSL button when DNS is ready (or when the host IP is not configured)
3. On submit: creates a provision_ssl job, sets mgn_ssl_state = 'pending', redirects to job detail

Automated Provisioning (installs only)

For nodes installed via Install New Node, ProvisionPendingSsl (scheduled hourly) watches for nodes with mgn_ssl_state = 'pending', checks DNS, and kicks off provision_ssl jobs automatically. After ~16 hours of DNS check failures it flips state to failed. Manual provisioning via the Setup card is the fallback.

Backup Targets

Backup targets define where backup files are uploaded after creation. Each node can optionally have a backup target assigned. If no target is set, backups remain local only on the remote server.

Supported Providers

Nodes with no backup target leave backups local-only on the remote server.

Configuration

1. Go to /admin/server_manager/targets and click Add Target
2. Select a provider, enter bucket name, path prefix, and credentials
3. Go to a node's Overview tab, expand Edit Connection Settings, and select the target from the Backup Target dropdown
4. Save — backups for this node will now auto-upload after creation

Upload Path Structure

All providers use: {prefix}/{node_slug}/{filename}

Example: joinery-backups/empoweredhealthtn/empoweredhealthtn-04_11_2026.sql.gz.enc

Credential Storage

Credentials are stored in the bkt_credentials JSON column on the bkt_backup_targets table using a unified shape for every provider:

{"access_key": "...", "secret_key": "...", "region": "...", "endpoint": "..."}

For node-side operations (upload, delete, download), the credentials are embedded into a self-contained PHP script that is piped to the node via a heredoc'd php -- invocation — never written to a file on the node and never visible in process listings as positional arguments. The S3Signer.php and node_uploader.php source is composed at job-build time by JobCommandBuilder::build_node_uploader_script().

Backup Browser

The Backups tab on each node includes a file browser that lists backup files from both local storage and the cloud target. Features:

• Scan for Backups — creates a list_backups job to scan local /backups/ on the node
• Unified file table — shows filename, size, date, and location (Local / Cloud / Both)
• Delete — single Delete button per row that removes the file from every location it exists in (local, cloud, or both); the confirmation dialog names the file and locations explicitly
• Restore Full Project — for .tar.gz archives, see the restore_project row in the Job Types table

Backup Encryption

Default Behavior

Encryption is enabled by default on both Database Backup and Full Project Backup forms. The existing backup_database.sh script handles encryption using AES-256-CBC with a key from ~/.joinery_backup_key on the remote server.

B2 Enforcement

When a node's backup target is Backblaze B2, encryption is mandatory. The UI replaces the checkbox with a message, and the server-side enforces it regardless of form input.

Auto-Generated Keys

If an encryption key doesn't exist on the remote server, the backup job auto-generates one:

1. The first step checks for ~/.joinery_backup_key
2. If missing, generates a random 32-byte base64 key with openssl rand -base64 32
3. Saves it with 600 permissions
4. Logs ENCRYPTION_KEY_GENERATED in the job output (the key value itself is never in the output)

Key Security

The encryption key never touches the control plane. It exists only on the remote server. This ensures that compromising the B2 bucket or the control plane database does not expose the decryption key.

To retrieve the key (for decrypting backups on another machine), SSH to the remote server:

cat ~/.joinery_backup_key

How It Works: Smart Plugin, Dumb Agent

All job-type intelligence lives in JobCommandBuilder.php. The Go agent is a generic executor that understands four primitives: ssh, scp, local, and api.

When an admin triggers an operation:

1. PHP looks up the node's connection details (host, SSH key, container, etc.)
2. JobCommandBuilder::build_<type>() generates an ordered array of steps
3. PHP writes a job row with the steps in mjb_commands (JSON)
4. Go agent picks up the job, executes each step in order, streams output
5. Agent marks job completed or failed
6. JobResultProcessor optionally parses the output into structured data

{
    "steps": [
        {"type": "ssh", "label": "Check disk usage", "cmd": "df -h /"},
        {"type": "ssh", "label": "Check memory", "cmd": "free -m"},
        {"type": "ssh", "label": "Check uptime", "cmd": "uptime"},
        {"type": "ssh", "label": "Check PostgreSQL", "cmd": "pg_isready"},
        {"type": "ssh", "label": "Check Joinery version",
         "cmd": "grep VERSION /var/www/html/site/public_html/includes/version.php"},
        {"type": "ssh", "label": "Container stats",
         "cmd": "docker stats --no-stream empoweredhealthtn", "on_host": true}
    ]
}

The agent doesn't know this is a "status check." It just runs each step's command via SSH, captures output, and moves on.

Adding a New Job Type

Adding a new operation requires PHP changes only -- no Go rebuild needed.

1. Add a static method to JobCommandBuilder:

// plugins/server_manager/includes/JobCommandBuilder.php
public static function build_restart_apache($node) {
    return [
        ['type' => 'ssh', 'label' => 'Restart Apache',
         'cmd' => 'systemctl restart apache2'],
        ['type' => 'ssh', 'label' => 'Verify Apache status',
         'cmd' => 'systemctl is-active apache2'],
    ];
}

1. Add a UI trigger (button/form) in the appropriate admin view that calls:

$steps = JobCommandBuilder::build_restart_apache($node);
$job = ManagementJob::createJob($node->key, 'restart_apache', $steps, null, $session->get_user_id());
header('Location: /admin/server_manager/job_detail?job_id=' . $job->key);

1. Optionally add a result processor method in JobResultProcessor if you want to parse the output into structured data.

Step Fields Reference

Management API (Read-Only)

Every Joinery instance exposes a namespaced read-only HTTP surface at /api/v1/management/*. The control plane prefers this over SSH for observability operations (check_status, list_backups, fetch_backup) because it's faster, parallelizable, and auditable.

Endpoints (all under /api/v1/management/, all GET, all JSON except backups/fetch which streams binary):

Authentication uses the existing API key system (stg_api_keys — same bcrypt flow as public CRUD). The gate is user-level: the key's owning user must have usr_permission >= 10 (superadmin). apk_permission is NOT the gate here — it's the CRUD-axis permission and is orthogonal. A superadmin's key with apk_permission=1 can call management endpoints (read-only across both axes); a permission-5 admin's key cannot, regardless of apk_permission.

Adding a management key for a node: on the target node, Admin → API Keys → New Key, owner = a superadmin user, apk_permission = 1, IP-restrict to the control plane's egress IP. Paste the public/secret pair into the node's Overview tab on the control plane's Server Manager ("API Credential" panel).

> IP restriction on docker-prod nodes: for sites fronted directly by host Apache (no Cloudflare), the container now reads the real client IP via mod_remoteip + the host's X-Forwarded-For: %{REMOTE_ADDR}s header, so IP restriction works end-to-end. For Cloudflare-fronted sites, the container sees Cloudflare's edge IP — IP restriction is not yet meaningful in that case (a future spec will trust Cloudflare's ranges and read CF-Connecting-IP).

Build-time routing: JobCommandBuilder::build_<op>() dispatches to build_<op>_api() or build_<op>_ssh() based on has_api($node, $op), which checks: (1) credentials stored on the node row, (2) a matching build_<op>_api exists, (3) a fresh /health probe succeeds. No runtime fallback — a job is decided at build-time and runs that path or fails. The existing SSH implementation stays in place; clearing the stored credentials or breaking /health routes the next job back to SSH automatically.

Adding a new management endpoint: drop a file under includes/management_api/<name>_handler.php with <name>_handler($request) + <name>_handler_api() meta function. Nested paths mirror directories (backups/list_handler.php → GET /api/v1/management/backups/list). Parallels the action-endpoint convention in logic/*_logic.php.

TLS verification is strict by default. The mgn_tls_insecure boolean on mgn_managed_nodes opts a single node out for dev/local instances without a cert from a trusted CA. Audit: SELECT mgn_slug FROM mgn_managed_nodes WHERE mgn_tls_insecure = true.

Data Models

ManagedNode (mgn_managed_nodes)

Represents a remote Joinery instance. Key fields:

• mgn_name -- Display name (e.g., "Empowered Health Production")
• mgn_slug -- Short identifier, unique (e.g., "empoweredhealthtn")
• mgn_host -- SSH host (IP or hostname)
• mgn_ssh_user, mgn_ssh_key_path, mgn_ssh_port -- SSH connection details
• mgn_container_name -- Docker container name (null for bare metal)
• mgn_web_root -- Path to public_html inside the server/container
• mgn_last_status_data -- JSON from last status check (disk, memory, load, etc.)
• mgn_joinery_version -- Last known version string
• mgn_bkt_backup_target_id -- FK to backup target (null = local only)

ManagementJob (mjb_management_jobs)

Represents a queued, running, or completed operation. Key fields:

• mjb_mgn_node_id -- Target node (FK to mgn_managed_nodes, null for local-only jobs)
• mjb_job_type -- Label for display/filtering (e.g., "backup_database")
• mjb_status -- pending, running, completed, failed, or cancelled
• mjb_commands -- JSON with the step array the agent executes
• mjb_output -- Progressive text output (appended during execution)
• mjb_result -- Structured JSON populated by JobResultProcessor after completion
• mjb_current_step / mjb_total_steps -- Progress tracking

$job = ManagementJob::createJob(
    $node_id,               // target node ID (or null for local)
    'backup_database',      // job type label
    $steps,                 // array of step dicts from JobCommandBuilder
    ['encryption' => true], // parameters (stored for reference/re-run)
    $session->get_user_id() // who triggered it
);

BackupTarget (bkt_backup_targets)

Configured storage target for backups. Key fields:

• bkt_name -- Display name (e.g., "Production B2")
• bkt_provider -- b2, s3, or linode
• bkt_bucket -- Bucket name (required)
• bkt_path_prefix -- Path prefix within the bucket (default: joinery-backups)
• bkt_credentials -- JSON with the unified shape {access_key, secret_key, region, endpoint} for every provider; B2's region/endpoint are auto-detected at save time
• bkt_delete_local -- Whether to delete local backup after successful upload
• bkt_enabled -- Whether this target is active

AgentHeartbeat (ahb_agent_heartbeats)

Single-row table tracking agent liveness. Updated every 30 seconds by the Go agent. The dashboard checks ahb_last_heartbeat to show online/offline status.

Safety Constraints

1. Auto-backup before destructive operations -- copy_database, restore_database, and restore_project automatically prepend backup steps. restore_project snapshots both the current database (auto_pre_project_restore_*.sql.gz) and the current project tree (auto_pre_project_restore_*.tar.gz) to /backups/ before overwriting; either can be skipped if the corresponding component is unchecked in the form. If any pre-backup step fails, the destructive steps never run.

1. Per-node concurrency lock -- The agent skips jobs if another job is already running on the same node, preventing conflicts.

1. Stale job recovery -- On agent startup, any orphaned running jobs are marked failed with a descriptive message.

1. Step timeout -- 30-minute default per step, overridable. On timeout, the SSH session is killed.

1. Single-threaded agent -- One job at a time. Queued jobs run sequentially.

1. Remote credentials at runtime -- Database credentials for backup/copy/restore are extracted from each node's Globalvars_site.php at execution time, never stored on the control plane.

AJAX Endpoints

/ajax/job_status

Polled by the job detail page for live output.

Parameters:

• job_id (int) -- job to query
• output_offset (int) -- character position; only new output since this offset is returned

{
    "success": true,
    "status": "running",
    "new_output": "=== [Step 2/5] Check memory ===\n...",
    "output_offset": 1234,
    "current_step": 2,
    "total_steps": 5,
    "error_message": null
}

The UI polls every 2 seconds while a job is running and stops when status is completed or failed.

/ajax/backup_actions

Used by the backup browser on the Backups tab.

/ajax/discover_nodes

Used by the auto-detect panel on the Add Node page. Creates and polls discover_nodes jobs.

Troubleshooting

Agent shows Offline on dashboard

• Check the agent is running: sudo systemctl status joinery-agent
• Check logs: journalctl -u joinery-agent -f
• Verify DB credentials in /etc/joinery-agent/joinery-agent.env match those in Globalvars_site.php

• Agent is not running or can't connect to the database
• Another job is running on the same node (per-node lock)

• Verify SSH key path on the node record matches an actual key file
• Test manually: ssh -i /path/to/key root@host "echo ok"
• For container nodes, verify the container name is correct

• The agent crashed or was restarted mid-job. Check journalctl for the crash cause.
• The partially-completed job should be inspected manually. Use Re-run to retry.

File Reference

Plugin (plugins/server_manager/)

Go Agent (/home/user1/joinery-agent/)