Key Points

Follow One Point Cloud Request

  • A point cloud page involves three systems: Apache serves the page and viewer code, the viewer runs in the browser, and S3 serves the data directly to the browser.
  • curl with -o /dev/null -w "%{http_code}" checks one step at a time; adding -H "Referer: ..." reproduces what a browser sends.
  • Find where a failure happens from evidence first; the fix often belongs somewhere other than where the symptom appears.
  • A health check is only as good as the failures it would catch.

First Converge and Second Run

  • A converge applies the described state; recap ok includes successful tasks that reported changed.
  • A second run should report changed=0, but verify the resulting state as well.
  • creates, removes, and when can prevent execution; changed_when only changes reporting and handler notification.
  • Molecule’s idempotence step checks the playbook; its verify step checks the result. You need both, and an outside-in check for what staging can’t see.

Variables, Templates and Handlers

  • Variables carry requirements; templates render them into managed files.
  • Changed template output can notify handlers; reporting no change suppresses notification.
  • Check the served result as well as configuration syntax.

Organize a Role, Validate Inputs

  • Reuse a role by changing inputs and content, not by copying infrastructure tasks.
  • Input validation catches some errors early; it does not replace service checks.
  • HTML publication and successful point-cloud rendering are different outcomes.

Break, Diagnose, Recover

  • A 200 response does not establish correct content.
  • A verifier only checks requirements it actually asserts.
  • Repair source, inspect the result, and rebuild to test reproducibility.

Operate the Service

  • Monitors provide bounded evidence, including early warnings.
  • Verify a release with a requirement-specific check.
  • Static-content rollback does not establish data-migration rollback safety.

Dataverse Case Study: Why We Built This Way

  • Dataverse needs five components: Payara (app server), Solr (search), PostgreSQL via RDS (metadata), S3 (file content), Apache (proxy and SSL).
  • Terraform provisions the AWS infrastructure; Ansible configures what runs on it.
  • The three repos are terraform-dataverse, dataverse-ansible, and dataverse-infrastructure.
  • Infrastructure as code makes the system reproducible, reviewable, and rebuildable.
  • Many decisions in this codebase were shaped by the 5.14 to 6.8 migration, and that context appears throughout the lesson.

Tooling Setup

  • Public learners need only the core setup; operator onboarding is separate.
  • Repository paths, credentials, inventory, and permissions must match the current project.
  • Existing Vault ciphertext needs its existing password, not a newly generated one.
  • Configuration, connectivity, and privilege escalation are distinct checks.

Terraform: The Infrastructure Layer

  • Terraform manages EC2, RDS, S3, security groups, an Elastic IP resource, and IAM for this project.
  • State is stored remotely in S3, but Tim and Jamie each have their own bucket/key: state is isolated per operator, not shared.
  • Each operator has their own environment directory; both use shared modules.
  • An Elastic IP resource exists, but it’s tied to the instance’s lifecycle, so it does not yet survive make rebuild. That’s still open work (roadmap 02-01).
  • terraform.tfvars is gitignored per environment, but is not currently secret-free in practice: a known gap (audit F8).

Ansible: Configuration and Idempotency

  • dataverse-ansible installs and configures Payara, Solr, Apache, and Dataverse; site.yml is the entry point, and the whole repo is treated as one role.
  • Individual modules (like dnf) are idempotent, but this role, as a whole, is not safe to re-run against a live instance. The operating rule is destroy-and-rebuild, not re-run-in-place.
  • group_vars (all.yml, dev.yml, test.yml, staging.yml) provides environment-specific values without duplicating the role; DOI config lives under nested pid:/doi: blocks, not flat keys.
  • The Ansible inventory is generated from Terraform output (ansible_user: rocky): gitignored, regenerated on every apply, never hand-edited.
  • Unguarded shell/command tasks are the reason re-running isn’t safe. Prefer modules, and guard shell tasks with creates:/when: when you can’t avoid them.

The Dataverse Stack: Payara, Solr, and the Data Layer

  • Payara is a Jakarta EE application server; Dataverse runs as a WAR file inside it.
  • Most Dataverse configuration is set as Payara JVM options, managed by Ansible.
  • The Payara log at payara6/glassfish/domains/domain1/logs/server.log is the first place to look when things go wrong.
  • RDS holds metadata; S3 holds file content; Solr holds the search index.
  • Always run make reindex after a database restore: Solr does not update itself.
  • make reindex/baseline/test currently depend on the admin API being open over public HTTPS: a known Critical security exposure (F1/F5), not a stable design choice.

Secrets and Environment Configuration

  • Secrets are vaulted inline inside group_vars/<env>.yml (!vault | blocks): there is no separate group_vars/all/vault.yml file.
  • S3 access uses an IAM instance profile, not vaulted AWS credentials: there’s no access key to leak.
  • Test environments use self-signed certificates (letsencrypt.certbot.test_cert: true); production would use Let’s Encrypt.
  • DOI config is two nested blocks, pid: and doi:, not one flat provider variable. FAKE is used everywhere non-production; EZID is planned for Phase 7 and has no group_vars file yet.
  • Vaulting is incomplete today: test.yml has real unvaulted CHANGE_ME_USE_VAULT placeholders. Don’t assume every environment is equally secret-safe.

The Makefile: Daily Operations

  • The Makefile is the daily operations interface: rarely run Terraform or Ansible directly.
  • make rebuild ENV=<env> DB_PASS=<pass> destroys and recreates EC2, RDS, and S3 together, then restores the database from an S3 dump. It does not preserve data by default: the restore step is what puts data back.
  • make baseline ENV=<env> captures a timestamped snapshot to baseline-snapshots/, with dataset, file, and S3 counts.
  • make reindex ENV=<env> rebuilds the Solr index after any database restore, and depends on the admin API being open over public HTTPS (a known security gap, Dataverse stack).
  • Always specify ENV=: the Makefile will error without it.

Testing and Validation

  • The test suite (dataverse-ansible/tests/integration/, run via make test) covers API health, search, S3 round-trip, and Solr indexing as real pytest classes.
  • PID/DOI-specific validation is thin today: a TestPIDConfiguration class exists but only runs under the migration marker and mostly checks settings exist, not FAKE-vs-EZID behavior.
  • Baseline comparisons detect count discrepancies; they do not prove byte integrity or correct record-to-object mappings.
  • Every baseline field must match exactly except downloads, which is deliberately informational-only.
  • A final write-freeze baseline anchors the cutover comparison; an older planning snapshot is supporting context.

Using AI Tools in Infrastructure Work

  • Check current institutional tool and data policies separately from general learning guidance.
  • Never share credentials, private hostnames, database connection strings, or SSH keys with any AI tool.
  • AI is particularly strong with config systems like Terraform and Ansible, and that is exactly where the risk of losing contact is highest.
  • Automation bias is real: over-trusting AI output because it is usually right, until it isn’t.
  • Strategies for staying grounded: write it down in your own words, ask for explanation before output, validate everything, practice without the tool periodically, teach it to someone else.
  • Writing this lesson is itself one of those strategies.

The Migration Arc: 5.14 to 6.8

  • Dataverse 6.x requires Java 17, a Solr schema rebuild, and updated DOI/S3 configuration.
  • The 7-phase plan gates each phase with tests before proceeding to the next, but “current phase” doesn’t mean “current phase complete.” Phase 2 is one-third done as of this writing.
  • Phases 1-3 establish the foundation; phases 4-6 harden for production; phase 7 is cutover.
  • Keep both sides write-frozen through validation and routing changes. New writes require reconciliation on rollback, independently of DOI registration.
  • DNS TTL reduction and a persistent Elastic IP would together make the cutover switchover fast and predictable, but the EIP isn’t persistent yet, and file-byte synchronization must be implemented and rehearsed before the illustrative cutover sequence becomes a runbook.