Key Points

Introduction: Why We Built This Way

Dataverse needs five components: Payara (app server), Solr (search), PostgreSQL via RDS (metadata), S3 (file content), Apache (proxy and SSL).
Terraform provisions the AWS infrastructure; Ansible configures what runs on it.
The three repos are terraform-dataverse, dataverse-ansible, and dataverse-infrastructure.
Infrastructure as code makes the system reproducible, reviewable, and rebuildable.
Many decisions in this codebase were shaped by the 5.14 to 6.8 migration – that context appears throughout the lesson.

Five tools required: Terraform, Ansible, AWS CLI, Make, uv.
AWS profile is ucla-library-dsc – set AWS_PROFILE in your shell before running anything.
Clone dataverse-infrastructure first, then run make bootstrap – it nests the other two repos inside it. They are not siblings.
Ansible Vault needs a local .vault-password file (openssl rand -base64 24 > .vault-password in dataverse-ansible) before any vaulted group_vars can be decrypted.
terraform init must succeed before any other Terraform command will work.
terraform plan is always safe – it shows changes without making them.

Terraform manages EC2, RDS, S3, security groups, an Elastic IP resource, and IAM for this project.
State is stored remotely in S3, but Tim and Jamie each have their own bucket/key – state is isolated per operator, not shared.
Each operator has their own environment directory; both use shared modules.
An Elastic IP resource exists, but it’s tied to the instance’s lifecycle, so it does not yet survive make rebuild – that’s still open work (roadmap 02-01).
terraform.tfvars is gitignored per environment, but is not currently secret-free in practice – a known gap (audit F8).

dataverse-ansible installs and configures Payara, Solr, Apache, and Dataverse; site.yml is the entry point, and the whole repo is treated as one role.
Individual modules (like dnf) are idempotent – but this role, as a whole, is not safe to re-run against a live instance. The operating rule is destroy-and-rebuild, not re-run-in-place.
group_vars (all.yml, dev.yml, test.yml, staging.yml) provides environment-specific values without duplicating the role; DOI config lives under nested pid:/doi: blocks, not flat keys.
The Ansible inventory is generated from Terraform output (ansible_user: rocky) – gitignored, regenerated on every apply, never hand-edited.
Unguarded shell/command tasks are the reason re-running isn’t safe – prefer modules, and guard shell tasks with creates:/when: when you can’t avoid them.

Payara is a Jakarta EE application server; Dataverse runs as a WAR file inside it.
Most Dataverse configuration is set as Payara JVM options, managed by Ansible.
The Payara log at payara6/glassfish/domains/domain1/logs/server.log is the first place to look when things go wrong.
RDS holds metadata; S3 holds file content; Solr holds the search index.
Always run make reindex after a database restore – Solr does not update itself.
make reindex/baseline/test currently depend on the admin API being open over public HTTPS – a known Critical security exposure (F1/F5), not a stable design choice.

Secrets are vaulted inline inside group_vars/<env>.yml (!vault | blocks) – there is no separate group_vars/all/vault.yml file.
S3 access uses an IAM instance profile, not vaulted AWS credentials – there’s no access key to leak.
Test environments use self-signed certificates (letsencrypt.certbot.test_cert: true); production would use Let’s Encrypt.
DOI config is two nested blocks, pid: and doi:, not one flat provider variable. FAKE is used everywhere non-production; EZID is planned for Phase 7 and has no group_vars file yet.
Vaulting is incomplete today: test.yml has real unvaulted CHANGE_ME_USE_VAULT placeholders. Don’t assume every environment is equally secret-safe.

The Makefile is the daily operations interface – rarely run Terraform or Ansible directly.
make rebuild ENV=<env> DB_PASS=<pass> destroys and recreates EC2, RDS, and S3 together, then restores the database from an S3 dump. It does not preserve data by default – the restore step is what puts data back.
make baseline ENV=<env> captures a timestamped snapshot to baseline-snapshots/, with dataset, file, and S3 counts.
make reindex ENV=<env> rebuilds the Solr index after any database restore – and depends on the admin API being open over public HTTPS (a known security gap, Episode 5).
Always specify ENV= – the Makefile will error without it.

The test suite (dataverse-ansible/tests/integration/, run via make test) covers API health, search, S3 round-trip, and Solr indexing as real pytest classes.
PID/DOI-specific validation is thin today: a TestPIDConfiguration class exists but only runs under the migration marker and mostly checks settings exist, not FAKE-vs-EZID behavior.
Baseline comparisons verify data integrity by comparing counts before and after migration, saved to baseline-snapshots/ (no latest.json).
Every baseline field must match exactly except downloads, which is deliberately informational-only.
Jamie’s pre-migration production baseline is the anchor for the final Phase 7 comparison.

Use your institution’s licensed AI tools for work; at UCLA, that is Gemini via Google Workspace.
Never share credentials, private hostnames, database connection strings, or SSH keys with any AI tool.
AI is particularly strong with config systems like Terraform and Ansible – and that is exactly where the risk of losing contact is highest.
Automation bias is real: over-trusting AI output because it is usually right, until it isn’t.
Strategies for staying grounded: write it down in your own words, ask for explanation before output, validate everything, practice without the tool periodically, teach it to someone else.
Writing this lesson is itself one of those strategies.

Dataverse 6.x requires Java 17, a Solr schema rebuild, and updated DOI/S3 configuration.
The 7-phase plan gates each phase with tests before proceeding to the next – but “current phase” doesn’t mean “current phase complete.” Phase 2 is one-third done as of this writing.
Phases 1-3 establish the foundation; phases 4-6 harden for production; phase 7 is cutover.
Rollback is straightforward until the DNS switch and EZID activation – that window is the target.
DNS TTL reduction and a persistent Elastic IP would together make the cutover switchover fast and predictable – but the EIP isn’t persistent yet, and the automated cutover sequence has no step that moves file bytes, a gap found after the roadmap was written.