All Images

Follow One Point Cloud Request

Figure 1

Request flow: a browser asks Apache on the EC2 server for a collection page; Apache returns the HTML plus Potree's JavaScript from the same server; the Potree viewer running in the browser then requests the point cloud data directly from an S3 bucket, sending the page's address in the Referer header; S3 returns the data only if that Referer is the www site.

First Converge and Second Run

Variables, Templates and Handlers

Organize a Role, Validate Inputs

Break, Diagnose, Recover

Operate the Service

Dataverse Case Study: Why We Built This Way

Figure 1

Architecture diagram: a browser/user on the internet sends a request into an AWS box containing an EC2 instance running Apache httpd (reverse proxy and SSL termination), which forwards to Payara running the Dataverse WAR file, which talks to Solr for search. Payara also connects out to RDS PostgreSQL for metadata and S3 for file storage, both outside the EC2 instance.
The Dataverse stack

Tooling Setup

Terraform: The Infrastructure Layer

Figure 1

Diagram: Tim's environment applies to S3 bucket ucla-tim-terraform-state, and Jamie's environment applies to S3 bucket ucla-library-terraform-state. Both environments take a lock against a shared DynamoDB table called terraform-locks before applying.
Tim and Jamie’s operator environments each apply against their own S3 state bucket. Both share a DynamoDB lock table, so the shared table prevents same-operator races but not cross-operator conflicts.

Ansible: Configuration and Idempotency

Figure 1

Decision flow: after make ansible ENV=x runs, if it did not fail, the environment is configured and module-level idempotency held. If it failed partway through, the wrong move is re-running ansible-playbook in place, which risks unguarded shell/command tasks failing again or corrupting state. The right move is terraform destroy followed by make rebuild, producing a clean environment rebuilt from scratch.
A failed ansible-playbook run should end in terraform destroy and rebuild, not a second in-place run.

The Dataverse Stack: Payara, Solr, and the Data Layer

Secrets and Environment Configuration

Figure 1

Flow diagram: a secret starts as an encrypted vault block in group_vars/<env>.yml, is decrypted by Ansible at playbook runtime, is set as a Payara JVM option in domain.xml, and is read by Dataverse at startup.
A secret moves from an encrypted group_vars block, through ansible-vault decryption at runtime, into a Payara JVM option, and is finally read by Dataverse at startup.

The Makefile: Daily Operations

Figure 1

Pipeline diagram of the 7 make rebuild steps in order: 1 Destroy (terraform destroy), 2 Provision (terraform apply), 3 DNS propagation (manual A record update), 4 Ansible (site.yml), 5 Restore DB from S3 (restore-db.sh), 6 Start Payara (over SSH), 7 Wait then reindex (make reindex). A dashed line points from step 7 to make test, labeled as a separate, manual step not included in the rebuild pipeline.
The seven make rebuild steps run in sequence, ending at reindex. make test is a separate step run manually afterward, not part of the rebuild pipeline.

Testing and Validation

Using AI Tools in Infrastructure Work

The Migration Arc: 5.14 to 6.8

Figure 1

Pipeline diagram of the 7 migration phases in order: Phase 1 Production baseline (complete), Phase 2 Rebuild and hardening (1 of 3 done), Phase 3 Expanded test coverage, Phase 4 Jamie onboarding, Phase 5 Shibboleth/SSO, Phase 6 Maintenance window plan, Phase 7 DNS cutover.
The seven migration phases run in sequence, each gated by test/baseline passage before the next begins. Phase 2 is only one-third complete as of this writing.