PSA/docs/getting-started/setup_guide.md

# Complete Setup Guide

This guide provides step-by-step instructions for setting up the PSA system using Docker Compose, supporting both Community Edition (CE) and Enterprise Edition (EE).

> Note: The instructions below focus on the CE prebuilt images. Full EE setup guidance, including any edition-specific overrides, is being prepared and will be added soon.

## Prerequisites

- Docker Engine 24.0.0 or later
- Docker Compose v2.20.0 or later
- Git
- Text editor for configuration files

## Choose a Release

1. Clone the repository if you have not already:
   ```bash
   git clone https://github.com/nine-minds/alga-psa.git
   cd alga-psa
   ```
2. Fetch the latest remote refs and list release branches sorted by version:
   ```bash
   git fetch origin --prune
   git branch -r --list 'origin/release/*' --sort='version:refname'
   ```
3. Release branches follow the `release/<version>` pattern, including release candidates such as `release/1.0-rc3`. If you want the newest release branch, use the last entry from the command output above.
4. Check out the release branch you want to deploy:
   ```bash
   git checkout <release-branch>
   ```

## Initial Setup

1. Resolve the container image for the current checkout:
   ```bash
   ./scripts/set-image-tag.sh
   ```
   This checks for a published CE image matching the current `HEAD` short SHA. If it exists, the script pulls it and writes `.env.image` alongside `server/.env` with `ALGA_IMAGE_TAG=<sha>`. If no matching image exists, the script builds the CE image locally from the current checkout, tags it with that same SHA, and writes `.env.image`. If the worktree has uncommitted changes, the script builds locally with a `<sha>-local` tag so the image cannot be confused with the published commit image. Re-run the script whenever you upgrade or switch commits. If the local image build needs a larger Node.js heap, run `NEXT_BUILD_MAX_OLD_SPACE_SIZE=16384 ./scripts/set-image-tag.sh`.
2. Create required directories:
   ```bash
   mkdir -p secrets/tenants
   ```

## Secrets Configuration

1. Create secret files in the `secrets/` directory:

   Use single quotes around secret values to prevent shell expansion of special characters (for example `$`, `!`, `*`, and backticks).
   If a secret contains a single quote (`'`), use a quoted heredoc instead:
   ```bash
   cat > secrets/email_password <<'EOF'
   your-secret-value
   EOF
   ```

Database secrets (replace placeholders with strong values):
```bash
echo 'your-secure-admin-password' > secrets/postgres_password
echo 'your-secure-app-password' > secrets/db_password_server
echo 'your-secure-hocuspocus-password' > secrets/db_password_hocuspocus
```

Redis Secret:
```bash
echo 'your-secure-password' > secrets/redis_password
```

Authentication secret:
```bash
echo 'your-32-char-min-key' > secrets/alga_auth_key
```

Security Secrets:
```bash
echo 'your-32-char-min-key' > secrets/crypto_key
echo 'your-32-char-min-key' > secrets/token_secret_key
echo 'your-32-char-min-key' > secrets/nextauth_secret
```

Email & OAuth Secrets:
```bash
echo 'your-email-password' > secrets/email_password
echo 'your-client-id' > secrets/google_oauth_client_id
echo 'your-client-secret' > secrets/google_oauth_client_secret
```

2. Set proper permissions:
```bash
chmod 600 secrets/*
```

## Environment Configuration

1. Copy the appropriate environment template:

```bash
cp .env.example server/.env
```

2. Open `server/.env` in your editor and confirm these core settings (adjust as needed):
   - `DB_TYPE=postgres` (required)
   - `DB_USER_ADMIN=postgres`
   - `DB_NAME_HOCUSPOCUS=hocuspocus`
   - `DB_USER_HOCUSPOCUS=hocuspocus_user`
   - `HOST=http://localhost:3000` (use your public domain in production)
   - `LOG_LEVEL=INFO`
   - `LOG_IS_FORMAT_JSON=false`
   - `LOG_IS_FULL_DETAILS=false`
   - `EMAIL_ENABLE=false` (set to `true` when you are ready to send mail)
   - `EMAIL_FROM=noreply@example.com`
   - `EMAIL_HOST=smtp.gmail.com`
   - `EMAIL_PORT=587`
   - `EMAIL_USERNAME=noreply@example.com`
   - `NEXTAUTH_URL=http://localhost:3000`
   - `NEXTAUTH_SESSION_EXPIRES=86400`

   Optional: enable collaborative editing by setting `REQUIRE_HOCUSPOCUS=true`.

Note: The system performs validation of these environment variables at startup. Missing or invalid values will prevent the system from starting.



## Docker Compose Configuration

> All commands in this section assume you have run `./scripts/set-image-tag.sh` and that `.env.image` sits alongside `server/.env`. Always pass both env files so Compose pulls the correct prebuilt image.

```bash
docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
  --env-file server/.env --env-file .env.image up -d
```

> Compose now creates a project-scoped network automatically, so multiple checkouts can run in parallel without fighting over `sebastian_app-network`. If you previously ran the stack from another directory, bring that one down first to clean up the old shared network.

> Note: The `-d` flag runs containers in detached/background mode. Remove the `-d` flag if you want to monitor the server output directly in the terminal.

### Initial Login Credentials

The first successful boot seeds a sample workspace admin account and prints its credentials to the server logs. Tail the logs right after the stack starts so you can copy the values for your first login:

```bash
docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
  --env-file server/.env --env-file .env.image logs -f
```

Look for a banner similar to the following (password redacted here for safety—yours will show the real value):

```
sebastian_server_ce  | 2025-02-10 15:12:23 [INFO   ]: *******************************************************
sebastian_server_ce  | 2025-02-10 15:12:23 [INFO   ]: ******** User Email is -> [ glinda@emeraldcity.oz ]  ********
sebastian_server_ce  | 2025-02-10 15:12:23 [INFO   ]: ********       Password is -> [ ****REDACTED**** ]   ********
sebastian_server_ce  | 2025-02-10 15:12:23 [INFO   ]: *******************************************************
```

> Copy the credentials before stopping the logs. After you sign in, update the password for production use.

The CE stack now includes the `workflow-worker` service by default, giving you a production-like asynchronous processing setup without additional compose overrides. The `ALGA_IMAGE_TAG` value determines which server image is used; `./scripts/set-image-tag.sh` ensures it matches the current checkout by either pulling the published image for `HEAD` or building that image locally.

## Production Setup (Persistent Storage)

For production-like deployments, persist both your database and uploaded documents to named Docker volumes. This keeps data safe across container restarts and image updates.

- Database volume: `postgres_data` (mounted at `/var/lib/postgresql/data`)
- Documents/files volume: `files_data` (mounted at `/data/files`)

The CE prebuilt compose now includes these volumes by default. When you run the compose command above, Docker will automatically create and attach them.

Recommended environment config for storage (add to `server/.env`):
```bash
STORAGE_DEFAULT_PROVIDER=local
STORAGE_LOCAL_BASE_PATH=/data/files
```

Verify volumes:
```bash
docker volume ls | grep -E "postgres_data|files_data"
```

### Network Exposure

The shared base compose file exposes Postgres, PgBouncer, Redis, and the application ports to the host for local convenience. For hardened environments, either remove or override the `ports:` entries with a compose override file, bind them to `127.0.0.1` behind a reverse proxy/firewall, or enforce host-level firewall rules that only allow trusted networks. Restart the stack after applying your chosen approach.

### Backups

- Postgres (logical backup using pg_dump — recommended). Replace the container name if you customized it.
  ```bash
  PGPASSWORD=$(cat secrets/postgres_password) \
  docker exec -e PGPASSWORD=${PGPASSWORD} $(docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image ps -q postgres) \
    pg_dump -U postgres -d server -Fc -f /tmp/pg_backup.dump
  docker cp $(docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image ps -q postgres):/tmp/pg_backup.dump ./pg_backup_$(date +%F).dump
  ```

- Postgres (quick snapshot of the data volume — use when DB is stopped):
  ```bash
  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image stop server pgbouncer postgres
  docker run --rm -v <project>_postgres_data:/var/lib/postgresql/data -v "$PWD":/backup alpine \
    tar czf /backup/postgres_volume_$(date +%F).tar.gz -C /var/lib/postgresql/data .
  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image start postgres pgbouncer server
  ```

- Files/documents volume:
  ```bash
  docker run --rm -v <project>_files_data:/data/files -v "$PWD":/backup alpine \
    tar czf /backup/files_volume_$(date +%F).tar.gz -C /data/files .
  ```

Note: Volume names are prefixed by your Compose project (e.g., `<project>_postgres_data`). If you customized `APP_NAME` or use `-p` with compose, check with `docker volume ls` and substitute accordingly.

### Restores (brief)

- Postgres (pg_restore). Create an empty database first if needed.
  ```bash
  PGPASSWORD=$(cat secrets/postgres_password) \
  docker cp ./pg_backup.dump $(docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image ps -q postgres):/tmp/pg_backup.dump
  docker exec -e PGPASSWORD=${PGPASSWORD} $(docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image ps -q postgres) \
    pg_restore -U postgres -d server --clean --if-exists /tmp/pg_backup.dump
  ```

- Files/documents volume:
  ```bash
  docker run --rm -v <project>_files_data:/data/files -v "$PWD":/backup alpine \
    sh -c "rm -rf /data/files/* && tar xzf /backup/files_volume.tgz -C /data/files"
  ```

### Notes

- The application’s local storage provider writes to `/data/files` inside the server container. Using the named volume `files_data` keeps those assets across restarts without host-permission tweaks.
- `docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml --env-file server/.env --env-file .env.image down` followed by `... up -d` is safe for restarts. Avoid adding `-v` unless you explicitly intend to wipe Postgres/files volumes.
- To inspect the volume contents from the host, use `docker run --rm -v <project>_postgres_data:/var/lib/postgresql/data busybox ls /var/lib/postgresql/data` (replace the volume name if you changed `APP_NAME` or pass `-p`).

## Monitoring

You can monitor the initialization process through Docker logs:
```bash
docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
  --env-file server/.env --env-file .env.image logs -f
```

## Troubleshooting

### Postgres authentication loop
- Continuous `password authentication failed for user "postgres"` or `role "hocuspocus_user" does not exist` messages mean the secrets on disk no longer match the credentials stored inside the `postgres_data` volume.
- If you need to keep existing data, sync the passwords and recreate the missing role:
  ```bash
  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
    --env-file server/.env --env-file .env.image exec postgres \
    psql -U postgres -c "ALTER ROLE postgres WITH PASSWORD '$(cat secrets/postgres_password)';"

  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
    --env-file server/.env --env-file .env.image exec postgres \
    psql -U postgres -c "DO $$ BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'hocuspocus_user') THEN CREATE ROLE hocuspocus_user LOGIN PASSWORD '$(cat secrets/db_password_hocuspocus)'; ELSE ALTER ROLE hocuspocus_user WITH PASSWORD '$(cat secrets/db_password_hocuspocus)'; END IF; END $$;" 
  ```
- To start fresh (wipes the database), stop the stack and remove the named volumes before bringing it back up:
  ```bash
  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
    --env-file server/.env --env-file .env.image down -v
  docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
    --env-file server/.env --env-file .env.image up -d
  ```
- After credentials are in sync, the `setup` container will finish running and migrations plus seed data will be applied automatically.

## Verification

1. Check service health:
```bash
docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
  --env-file server/.env --env-file .env.image ps
```

2. Access the application:
- Development: http://localhost:3000
- Production: https://your-domain.com

3. Verify logs for any errors:
```bash
docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
  --env-file server/.env --env-file .env.image logs [service-name]
```

## Common Issues & Solutions

### Environment Validation Issues
- Check all required variables are set
- Verify DB_TYPE is set to "postgres"
- Ensure LOG_LEVEL is a valid value
- Verify email addresses are valid
- Check numeric values are > 0
- Verify URLs are valid

### Database Connection Issues
- Verify secret files exist and have correct permissions
- Check database host/port configuration
- Ensure PostgreSQL container is running
- Verify postgres_password for admin operations
- Verify db_password_server for application access
- Check RLS policies if access is denied

### Redis Connection Issues
- Verify redis_password secret exists
- Check redis host/port configuration
- Ensure Redis container is running

### Authentication Issues
- Verify alga_auth_key secret exists and is properly configured
- Ensure authentication key is at least 32 characters long
- Check permissions on alga_auth_key secret file

### Hocuspocus Issues
- Check REQUIRE_HOCUSPOCUS setting
- Verify service availability if required
- Check connection timeout settings
- Verify database access

### Service Startup Issues
- Check service logs for specific errors
- Verify all required secrets exist
- Ensure correct environment variables are set
- Verify database users and permissions

## Security Checklist

✓ All secrets created with secure values
✓ Secret files have restricted permissions (600)
✓ Environment files configured without sensitive data
✓ Production environment uses HTTPS
✓ Database passwords are strong and unique
✓ Redis password is configured
✓ Authentication key (alga_auth_key) is properly configured
✓ Encryption keys are at least 32 characters
✓ RLS policies properly configured
✓ Database users have appropriate permissions
✓ Environment variables properly validated

## Production/Public Deployment Configuration

When deploying for public access (not localhost), additional configuration is required:

### Authentication URL Configuration
The `NEXTAUTH_URL` environment variable must match your public domain:

For local development:
```bash
NEXTAUTH_URL=http://localhost:3000
```

For production deployment:
```bash
NEXTAUTH_URL=https://your-domain.com
HOST=https://your-domain.com
```

### SSL/TLS Configuration
For production deployments:
1. Ensure your domain has valid SSL certificates
2. Configure your reverse proxy (nginx, Apache, etc.) for HTTPS
3. Update `NEXTAUTH_URL` to use `https://` protocol
4. Verify OAuth providers (if used) allow your production domain

### Reverse Proxy Configuration

The docker-compose stack runs the Next.js `server` container on port 3000 and
the `hocuspocus` container (real-time notifications and collaborative editing)
on port 1234 — separately, with no built-in reverse proxy in front. Your proxy
must route **two paths** for the application to work fully:

- `/hocuspocus` → `hocuspocus:1234` (with WebSocket upgrade headers)
- everything else → `server:3000`

If `/hocuspocus` is missing, browsers will show a "Reconnecting to server…"
indicator on the notifications bell and live notification updates will fall
back to 30-second polling.

#### nginx
```nginx
server {
    server_name your-domain.com;
    # ... your SSL/TLS config ...

    location /hocuspocus {
        proxy_pass http://hocuspocus:1234;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 3600s;   # long-lived WS connections
        proxy_send_timeout 3600s;
    }

    location / {
        proxy_pass http://server:3000;
        proxy_set_header Host $host;
        # ... usual headers ...
    }
}
```

#### Caddy
```
your-domain.com {
    reverse_proxy /hocuspocus* hocuspocus:1234
    reverse_proxy * server:3000
}
```
WebSocket upgrade is automatic in Caddy. Order matters — the `/hocuspocus*`
matcher must come before the catch-all.

#### Nginx Proxy Manager
In the proxy host's **Custom Locations** tab, add a location with:
- Define Location: `/hocuspocus`
- Scheme: `http`
- Forward Hostname/IP: the docker host (or hocuspocus container if reachable)
- Forward Port: `1234`

Make sure **Websockets Support** is enabled on the proxy host. The default
"Forward Hostname / IP" + "Forward Port" on the Details tab continues to
serve the catch-all to the `server` container on port 3000.

#### Cloudflare Tunnel
Add a separate ingress rule for the `/hocuspocus` path pointing to
`http://hocuspocus:1234`, and ensure WebSockets are enabled in the tunnel
configuration.

### Email Configuration for Production
Update email settings for production notifications:
```bash
EMAIL_ENABLE=true
EMAIL_FROM=noreply@your-domain.com
EMAIL_HOST=your-smtp-server.com
EMAIL_USERNAME=noreply@your-domain.com
```

### Security Considerations
- Use strong, unique secrets (different from development)
- Ensure all secret files have proper permissions (600)
- Configure firewall rules appropriately
- Regular backup procedures
- Monitor access logs
cies

## Upgrading

When upgrading from a previous version:

1. Backup all data:
   ```bash
   docker compose --env-file server/.env --env-file .env.image exec postgres \
     pg_dump -U postgres server > backup.sql
   ```

2. Fetch the latest release branches, choose the target `release/*` branch, and check it out:
   ```bash
   git fetch origin --prune
   git branch -r --list 'origin/release/*' --sort='version:refname'
   git checkout <release-branch>
   ```

3. Run `./scripts/set-image-tag.sh` again so `.env.image` updates to the new checkout SHA, pulling the matching published image when available or building it locally when it is not.

4. Restart the stack. The image has already been pulled or built by `./scripts/set-image-tag.sh`:
   ```bash
   docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
     --env-file server/.env --env-file .env.image down
   docker compose -f docker-compose.prebuilt.base.yaml -f docker-compose.prebuilt.ce.yaml \
     --env-file server/.env --env-file .env.image up -d
   ```

5. Review changes in:
   - Docker Compose files
   - Environment variables
   - Secret requirements
   - Database schema
   - RLS policies
   - Protocol Buffer definitions (EE only)

6. Update configurations as needed and verify the application starts cleanly before removing the old backups.

## Additional Resources

- [Configuration Guide](configuration_guide.md)
- [Development Guide](development_guide.md)
- [Docker Compose Documentation](docker_compose.md)
- [Secrets Management](../security/secrets_management.md)
- [Entrypoint Scripts](entrypoint_scripts.md)