All in One View

Content from Introduction: Why We Built This Way

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What is this lesson and who is it for?
  • What infrastructure does Dataverse need to run?
  • How are Terraform, Ansible, and the Dataverse application connected?

Objectives

  • Describe the UCLA Dataverse infrastructure stack and what each component does.
  • Explain the division of responsibility between Terraform and Ansible.
  • Navigate the three repositories that make up this infrastructure.
  • Understand what the 5.14 to 6.8 migration involves and why it shaped these decisions.

What this lesson is

This lesson traces the infrastructure that runs the UCLA Library Dataverse instance. It was written during the migration from Dataverse 5.14 to 6.8 – not as abstract documentation, but as a way to make explicit what was built, why each decision was made, and how the pieces fit together.

The audience is people doing the work or being onboarded to it: DSC staff, DataSquad students, and anyone who will operate or hand off this system. It assumes comfort with the command line and some exposure to cloud services or configuration management, but not infrastructure expertise.

The stack at a glance

Running Dataverse requires several components working together:

+------------------------------------------------------------+
|                           AWS                              |
|                                                            |
|  +------------------------------------------------------+  |
|  |                    EC2 instance                      |  |
|  |                                                      |  |
|  |  +----------------+    +--------------------------+  |  |
|  |  |  Apache httpd  |    |  Payara (app server)     |  |  |
|  |  |  reverse proxy +---->                          |  |  |
|  |  |  + SSL         |    |  Dataverse (WAR file)    |  |  |
|  |  +----------------+    +-------------+------------+  |  |
|  |                                      |               |  |
|  |                        +-------------v------------+  |  |
|  |                        |  Solr (search index)     |  |  |
|  |                        +--------------------------+  |  |
|  +------------------------------------------------------+  |
|                                                            |
|  +---------------------+   +----------------------------+ |
|  |  RDS (PostgreSQL)   |   |  S3 (file storage)         | |
|  +---------------------+   +----------------------------+ |
+------------------------------------------------------------+

EC2 is a virtual machine running Rocky Linux 9. Payara, Solr, and Apache all run here.

RDS is a managed PostgreSQL database hosted by AWS. Dataverse stores all its metadata here: datasets, files, users, permissions, version histories. The database is the source of truth for everything except the actual file content.

S3 is object storage for the data files that users upload. Dataverse stores file metadata in RDS and file content in S3. The two must stay in sync – a file record in the database pointing to a missing S3 object is a broken dataset.

Payara is a Jakarta EE application server. Dataverse runs as a WAR (Web Application Archive) file deployed inside Payara, similar to how a web application runs inside Tomcat but targeting the Jakarta EE ecosystem. Most Dataverse configuration is applied via Payara JVM options or through the Dataverse API at first boot.

Solr is a search engine that powers Dataverse’s dataset and file search. It maintains its own index independently of the database. After any database restore, the Solr index must be explicitly rebuilt – it will not update itself. A running Dataverse with a stale or empty Solr index will appear to have no datasets.

Apache httpd acts as a reverse proxy in front of Payara and handles SSL termination. Requests from browsers hit Apache first, then are forwarded to Payara on a non-public port. Apache also handles URL rewriting and HTTP to HTTPS redirects.

The three repositories

This infrastructure is managed across three repositories:

Repository What it does
terraform-dataverse Provisions AWS resources: EC2, RDS, S3, security groups, Elastic IP, IAM roles
dataverse-ansible Configures the EC2 instance: installs and configures Payara, Solr, Apache, and Dataverse
dataverse-infrastructure Makefile targets, tests, baseline scripts, and runbooks that tie the other two together

These map onto two concerns:

  • Infrastructure (Terraform): what AWS resources exist and how they are connected
  • Configuration (Ansible): what software is installed on those resources and how it is configured

You run Terraform first to provision the resources, then Ansible to configure them. The dataverse-infrastructure repo is where you do both – its Makefile calls out to Terraform and Ansible so you rarely interact with either tool directly.

Why infrastructure as code

Before Terraform and Ansible, changes to the Dataverse environment meant clicking through the AWS console or running commands directly on the server by hand. That works until something breaks:

  • You cannot reproduce exactly what you did
  • The next person does not know what state the server is in
  • Rebuilding from scratch after a failure is slow and depends on memory

Infrastructure as code puts the desired state of the system in version-controlled files. Terraform describes what AWS resources should exist. Ansible describes what should be installed and how it should be configured. Running them again should produce the same result – this property is called idempotency, and it is one of the central ideas in Episode 4.

The migration context

This lesson was developed during the migration of the UCLA Library Dataverse instance from version 5.14 to 6.8. Several decisions in the codebase – the Makefile targets, the baseline scripts, the FAKE DOI provider configuration, the 7-phase migration plan – exist because of constraints the migration imposed.

Where those decisions appear in later episodes, we explain why they were made. The migration itself is the subject of the final episode.

Challenge

Take stock

Before moving on, open the three repositories in your browser:

In each one, find:

  1. The main configuration file or entry point
  2. At least one file that corresponds to something in the stack diagram above

Some things to look for:

  • terraform-dataverse: environments/tim/main.tf or environments/jamie/main.tf for the per-operator Terraform config
  • dataverse-ansible: site.yml for the role entry point (the whole repo is treated as one Ansible role); group_vars/ for environment-specific configuration
  • dataverse-infrastructure: Makefile for the operations entry point; scripts/baseline-capture.sh for the baseline tooling
Challenge

Who owns what

Without looking back at the table above, answer from memory:

  1. If a dataset’s files return a 404 because an S3 bucket policy is wrong, which repo do you fix?
  2. If Solr comes up with an empty index after a database restore, which repo (or manual step) is responsible for rebuilding it, and why doesn’t it happen automatically?
  1. terraform-dataverse – bucket policy and IAM are AWS resources, which is Terraform’s domain, not Ansible’s.
  2. Neither repo does it automatically. Solr’s index is built from what’s in RDS, and Solr has no way to know the database changed underneath it. Rebuilding is a separate, explicit step (make reindex in dataverse-infrastructure) that must be run after any restore – see Episode 5.
Key Points
  • Dataverse needs five components: Payara (app server), Solr (search), PostgreSQL via RDS (metadata), S3 (file content), Apache (proxy and SSL).
  • Terraform provisions the AWS infrastructure; Ansible configures what runs on it.
  • The three repos are terraform-dataverse, dataverse-ansible, and dataverse-infrastructure.
  • Infrastructure as code makes the system reproducible, reviewable, and rebuildable.
  • Many decisions in this codebase were shaped by the 5.14 to 6.8 migration – that context appears throughout the lesson.

Content from Tooling Setup

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What tools do I need installed to work with this infrastructure?
  • How do the three repositories fit together on my local machine?
  • How do I configure AWS credentials for the correct profile?

Objectives

  • Install and verify Terraform, Ansible, AWS CLI, and Make.
  • Configure an AWS credentials profile named ucla-library-dsc.
  • Clone all three repositories and orient yourself in each.
  • Run terraform init and confirm Ansible can reach a target host.

What you need

Five tools are required before you can work with this infrastructure:

Tool Purpose Version
Terraform Provision AWS resources >= 1.5
Ansible Configure the EC2 instance >= 2.14
AWS CLI Authenticate to AWS, access S3 >= 2.x
Make Run Makefile targets system
uv Python package manager, runs the integration test suite latest

Installation instructions are on the Setup page.

AWS credentials

All AWS calls go through the ucla-library-dsc named profile. This includes:

  • Terraform, which creates and destroys resources
  • The baseline scripts, which read S3 object counts
  • Any direct aws CLI commands you run

Set the profile with an environment variable:

BASH 

export AWS_PROFILE=ucla-library-dsc

The Makefile and scripts default to ucla-library-dsc if AWS_PROFILE is not set, but setting it explicitly avoids surprises when running commands outside the Makefile.

To verify your credentials are working:

BASH 

aws sts get-caller-identity --profile ucla-library-dsc

A successful response shows your IAM user or role ARN, account ID, and user ID.

Callout

Jamie’s profile may differ

Jamie’s AWS profile may use a different name than ucla-library-dsc. The Makefile reads from ${AWS_PROFILE:-ucla-library-dsc}, so setting AWS_PROFILE in your shell before running any make target is the safest approach for both operators.

Cloning the repositories

dataverse-infrastructure is the orchestration repo – clone it first, then let it clone the other two as children of itself. It has a bootstrap target for exactly this:

BASH 

git clone https://github.com/ucla-data-science-center/dataverse-infrastructure
cd dataverse-infrastructure
make bootstrap

make bootstrap clones terraform-dataverse and dataverse-ansible into the dataverse-infrastructure directory (not as siblings next to it) and wires up an upstream remote on dataverse-ansible pointing at the generic gdcc/dataverse-ansible role this one is forked from. The Makefile’s paths (terraform-dataverse/environments/$(ENV), dataverse-ansible) all assume this nested layout – if you clone the child repos somewhere else, nothing in the Makefile will find them.

Initializing Terraform

From inside your operator environment directory:

BASH 

cd terraform-dataverse/environments/tim   # or jamie
terraform init

terraform init does three things:

  • Downloads the AWS provider plugin
  • Configures the remote state backend (an S3 bucket where Terraform stores its state file)
  • Validates the configuration syntax

After init, run:

BASH 

terraform plan

This shows what Terraform would create, change, or destroy – without making any changes. Read the plan output before running terraform apply. A plan that shows unexpected deletions is worth pausing on.

Setting up Ansible Vault

Secrets in group_vars (database passwords, admin passwords, API tokens) are encrypted with Ansible Vault. Before Ansible can decrypt them, you need a local vault password file:

BASH 

cd dataverse-ansible
openssl rand -base64 24 > .vault-password

.vault-password is gitignored – it never gets committed, and if you lose it the vault-encrypted secrets in group_vars are unrecoverable. Save its contents somewhere durable (a password manager, not just your laptop) before doing anything else. Without this file, ansible-playbook fails the moment it hits a vaulted variable.

Verifying Ansible

Ansible runs against an inventory file generated by Terraform. After terraform apply, Terraform writes an inventory file to a path the Makefile knows about.

To check Ansible can reach the host:

BASH 

ansible -i <inventory-file> all -m ping

A successful response: dataverse | SUCCESS => {"ping": "pong"}

If this fails, the most common causes are:

  • SSH key not loaded (ssh-add ~/.ssh/your-key)
  • Security group not open on port 22 (check in AWS console or Terraform config)
  • EC2 instance not yet fully booted (wait 60 seconds and retry)
Challenge

Credentials check

Run the AWS identity check and terraform init commands above. If either fails, note the error message and check the Setup troubleshooting section. The most common issues are a missing profile in ~/.aws/credentials or a role without sufficient IAM permissions.

Challenge

Two ways to fail before you even start

Without checking the episode, answer from memory:

  1. You clone dataverse-infrastructure and run git clone on the other two repos yourself, as siblings next to it. Then you run make rebuild ENV=tim. What happens, and why?
  2. You have all three repos cloned correctly and terraform apply succeeds. Then ansible-playbook fails immediately on a vaulted variable. What file is missing, and what command creates it?
  1. The Makefile can’t find terraform-dataverse or dataverse-ansible – it expects them cloned inside dataverse-infrastructure (via make bootstrap), not as sibling directories next to it. Targets fail with missing-path errors.
  2. dataverse-ansible/.vault-password is missing. Create it with openssl rand -base64 24 > .vault-password from inside dataverse-ansible, and save a copy somewhere durable – if it’s lost, the vaulted secrets can’t be recovered.
Key Points
  • Five tools required: Terraform, Ansible, AWS CLI, Make, uv.
  • AWS profile is ucla-library-dsc – set AWS_PROFILE in your shell before running anything.
  • Clone dataverse-infrastructure first, then run make bootstrap – it nests the other two repos inside it. They are not siblings.
  • Ansible Vault needs a local .vault-password file (openssl rand -base64 24 > .vault-password in dataverse-ansible) before any vaulted group_vars can be decrypted.
  • terraform init must succeed before any other Terraform command will work.
  • terraform plan is always safe – it shows changes without making them.

Content from Terraform: The Infrastructure Layer

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What AWS resources does Terraform manage for this project?
  • How is Terraform state stored and shared between operators?
  • How are Tim’s and Jamie’s environments organized?

Objectives

  • Read a Terraform config and identify the AWS resource types it defines.
  • Explain what remote state is and why it matters for a multi-operator project.
  • Describe what an Elastic IP is and why the one defined here doesn’t yet simplify the rebuild cycle.
  • Run terraform plan and interpret the output.

What Terraform manages

Terraform is responsible for the AWS resources that exist – the “what” of the infrastructure. For this project, those resources are:

  • EC2 instance: the virtual machine where Dataverse and its supporting services run
  • RDS instance: the managed PostgreSQL database
  • S3 buckets: file storage for Dataverse, plus a separate bucket for migration assets and backups
  • Security groups: firewall rules controlling what traffic reaches the EC2 instance
  • Elastic IP: a static IP address attached to the EC2 instance (more on this below)
  • IAM roles and policies: permissions for the EC2 instance to read and write S3

Terraform does not install software. It creates the resources. Once the EC2 instance exists, Ansible takes over and configures it.

Repository layout

The terraform-dataverse repo is organized by operator environment:

terraform-dataverse/
  environments/
    tim/        <- Tim's dev environment
    jamie/      <- Jamie's dev environment
  modules/      <- shared resource definitions used by both environments

Each environment directory has its own main.tf, variables.tf, and terraform.tfvars. The environments are mostly identical – they share modules – but use different resource names and sizes so Tim and Jamie can work independently without affecting each other.

Remote state

By default, Terraform stores its state in a local file called terraform.tfstate. This is a problem for a team: if two people run Terraform from different machines, their state files diverge and Terraform loses track of what actually exists in AWS.

This project uses remote state – but each operator has their own bucket and key, not a shared one:

Tim:   s3://ucla-tim-terraform-state/terraform-dataverse/tim/terraform.tfstate
Jamie: s3://ucla-library-terraform-state/terraform-dataverse/jamie/terraform.tfstate

Tim’s apply and Jamie’s apply cannot see or affect each other’s state at all – they are backed by different buckets. This is deliberate: it means destroying or rebuilding one operator’s environment can’t touch the other’s, which matters a lot given how often make rebuild tears an environment down and recreates it (Episode 6). Both backends still use a shared DynamoDB table (terraform-locks) for locking, which prevents two terraform apply runs against the same state from racing each other – but that only protects an operator against themselves (e.g. two terminal tabs), not against each other.

s3://ucla-dataverse-migration-assets/ is a different bucket entirely – it holds database dumps and migration assets, not Terraform state. Don’t confuse the two.

Callout

Never edit the state file directly

The state file is JSON and technically editable, but Terraform is the only thing that should write to it. Editing it by hand can put Terraform in an inconsistent state that is painful to recover from. If state gets out of sync, use terraform state subcommands to inspect and repair it.

Elastic IP

An Elastic IP (EIP) is a static IP address you reserve in AWS and attach to an EC2 instance.

Without an Elastic IP, every time you run make rebuild – which destroys and recreates the EC2 instance – the instance gets a new public IP address. That means:

  • The Ansible inventory file needs to be updated before Ansible can run
  • Any DNS records pointing at the old IP are wrong
  • You spend time tracking down the current IP instead of doing work

In principle, an Elastic IP fixes this: reserve the address once, reassociate it with whatever instance exists, and the address survives even when the instance doesn’t.

That’s not what happens today, though. The aws_eip.dataverse resource in modules/dataverse_ec2/main.tf is defined in the same module as the EC2 instance, associated directly to aws_instance.dataverse.id. When make rebuild runs terraform destroy, it tears down the whole module – instance and EIP together – so the address does not survive a rebuild yet. Roadmap item 02-01 (“Elastic IP resource… for both environments”) is still unchecked for exactly this reason: having an aws_eip resource isn’t the same as having a persistent one. The real make rebuild output today prints a new IP and pauses for you to update the DNS A record by hand (Makefile, the rebuild target) – which is the friction this feature is meant to remove, and hasn’t yet.

Variables and tfvars

Terraform configurations use variables to avoid hardcoding values that differ between environments.

  • variables.tf: declares the variables and their types (like a function signature)
  • terraform.tfvars: provides values for those variables (like the function call)

The terraform.tfvars file for each environment is gitignored, not committed – each operator creates their own from terraform.tfvars.example during setup (Episode 2). That’s intentional, because in practice it’s not secret-free: Tim’s real tfvars includes a plaintext db_password. This is a known, flagged gap (the security audit calls it out as finding F8) – the intended design keeps secrets in Ansible Vault, not Terraform, but db_password currently lives in both places, and the Terraform copy isn’t encrypted. Don’t treat “it’s gitignored” as equivalent to “it’s safe” – gitignore keeps a file out of version control, it doesn’t encrypt what’s on disk.

Challenge

Read a Terraform plan

From terraform-dataverse/environments/tim, run terraform plan.

Answer these questions from the output:

  1. How many resources does Terraform plan to create, change, or destroy?
  2. Which resource type appears the most?
  3. Find the security group resource. What ports does it open, and to what CIDR range?

The exact output will depend on current state. Things to look for:

  • aws_instance (the EC2 instance)
  • aws_security_group and aws_security_group_rule (firewall rules)
  • aws_eip and aws_eip_association (Elastic IP)
  • aws_db_instance (RDS)
  • Port 22 (SSH), 80 (HTTP), 443 (HTTPS), and 8080/4848 (Payara) in the security group rules
Challenge

State isolation and IP stability

Without checking the episode:

  1. Can Tim accidentally destroy Jamie’s environment by running terraform destroy in his own environment directory? Point to the specific piece of config that proves your answer.
  2. Does today’s public EC2 IP survive make rebuild ENV=tim? Why or why not?
  1. No – each environment’s main.tf points at a different S3 bucket/key for its backend (ucla-tim-terraform-state vs. ucla-library-terraform-state, different keys). Terraform only knows about resources tracked in the state it’s pointed at, so Tim’s destroy has no way to reach anything in Jamie’s state file.
  2. No. aws_eip.dataverse lives in the same module as aws_instance.dataverse and is associated directly to it, so terraform destroy removes both together. The address changes on every rebuild until roadmap item 02-01 makes the EIP persistent.
Key Points
  • Terraform manages EC2, RDS, S3, security groups, an Elastic IP resource, and IAM for this project.
  • State is stored remotely in S3, but Tim and Jamie each have their own bucket/key – state is isolated per operator, not shared.
  • Each operator has their own environment directory; both use shared modules.
  • An Elastic IP resource exists, but it’s tied to the instance’s lifecycle, so it does not yet survive make rebuild – that’s still open work (roadmap 02-01).
  • terraform.tfvars is gitignored per environment, but is not currently secret-free in practice – a known gap (audit F8).

Content from Ansible: Configuration and Idempotency

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What does dataverse-ansible configure on the EC2 instance?
  • What does idempotency mean in practice, and why does it matter for operations?
  • How are environment-specific values managed without duplicating configuration?

Objectives

  • Explain the structure of the dataverse-ansible role.
  • Describe what group_vars does and how environment overrides work.
  • Run the playbook and read the output.
  • Explain the difference between module-level idempotency and playbook-level re-run safety, and why this role only has the first.

What Ansible does

Ansible is responsible for configuration – the “what is installed and how is it set up” layer. After Terraform creates the EC2 instance, Ansible connects to it over SSH and:

  • Installs system packages (Java, Python, curl, and others)
  • Installs and configures Payara
  • Installs and configures Solr
  • Installs and configures Apache with SSL
  • Deploys the Dataverse WAR file into Payara
  • Applies Dataverse configuration via the Dataverse API
  • Sets JVM options in Payara for Dataverse to use

All of this is defined in the dataverse-ansible role – a structured collection of tasks, templates, handlers, and variable files.

Role structure

The dataverse-ansible repo is an Ansible role. The key directories are:

dataverse-ansible/
  site.yml        <- the entry point playbook
  tasks/          <- what to do (~90 flat task files, one per concern)
  handlers/       <- actions triggered by other tasks (e.g., restart Payara)
  templates/      <- config file templates with variable substitution
  defaults/       <- default variable values
  group_vars/     <- environment-specific overrides

site.yml at the repo root is the entry point – the repo’s own comment describes the whole thing as “this repository itself is the Dataverse ansible role.” Instead of one tasks/main.yml dispatching to sub-files, tasks/ holds a flat collection of per-service files (payara.yml, solr.yml, dataverse-prereqs.yml, and so on) that site.yml pulls in directly. There’s no top-level vars/ directory in this role – just defaults/ for role defaults and group_vars/ for environment overrides.

Idempotency (the concept) vs. this role (the reality)

An idempotent operation produces the same result whether you run it once or ten times. For Ansible, this means: running the playbook on an already-configured server should make no changes, because everything is already in the desired state. Individual Ansible modules are generally built this way:

YAML 

- name: Install Java
  dnf:
    name: "java-{{ java.version }}-openjdk-devel"
    state: present   # "present" means "install if not already installed"

The dnf module checks whether the package is installed before trying to install it. If it is already installed, Ansible reports ok and moves on. If not, it installs and reports changed. That’s real, and it’s how most tasks in dataverse-ansible behave.

But module-level idempotency is not the same as playbook-level re-run safety, and this role does not have the second one. The repo’s own operating rule, stated plainly in CONTEXT.md: “Ansible is NOT idempotent. You must fully destroy the environment before re-running. Do not re-run ansible-playbook against an existing instance.” In practice that means: after any make rebuild, if something fails partway through, the fix is terraform destroy and start over – not “just run make ansible again and let idempotent modules sort it out.”

Callout

Why re-running isn’t safe here

shell and command tasks run every time unless guarded with creates: or when: – and a role this size has a number of them: first-boot Dataverse API calls, database schema bootstrapping, Let’s Encrypt certificate issuance. Any one of those re-running against an already-configured instance can fail outright or leave the system in a state none of the individual ok/changed reports would have predicted. The safe mental model for this specific role is destroy-and-rebuild, not re-run-in-place – treat the per-module idempotency as a nice property of individual steps, not a guarantee about the playbook as a whole.

group_vars and environment overrides

The group_vars/ directory holds variable files that override defaults for specific environments. The real files are all.yml (non-secret defaults shared everywhere), dev.yml, test.yml, staging.yml, and TEMPLATE.yml (a starting point for a new environment) – there is no production.yml yet.

Dev and test both point at the FAKE DOI provider, but the real variables are nested under pid: and doi: blocks, not a single flat dataverse_doi_provider key:

YAML 

# group_vars/dev.yml
pid:
  authority: "10.5072"
  protocol: doi
  shoulder: "FK2/"

doi:
  provider: FAKE
  baseurl: "https://mds.test.datacite.org/"
  username: "testaccount"
  password: "notmypassword"   # not vaulted in dev -- see Episode 6

The same playbook runs against every environment; the variables control which behavior each environment gets. This is how we keep dev, test, and staging separate without maintaining separate copies of the role.

The inventory

Ansible needs to know which host to connect to. The inventory file lists hosts and their connection details. For this project, the inventory is generated by Terraform output, and the real EC2 user is rocky (Rocky Linux 9), not ubuntu:

[dataverse]
ec2-12-34-56-78.us-west-2.compute.amazonaws.com ansible_user=rocky ansible_ssh_private_key_file=~/.ssh/dataverse-key.pem

The Makefile generates and uses this inventory automatically when you run make rebuild. CONTEXT.md flags this file specifically: it’s gitignored and regenerated by Terraform on every apply – never edit it by hand, your edits will just be overwritten.

Challenge

Run the real playbook and read the output

There is no --check-mode wrapper for this role today – make ansible ENV=tim runs the real thing:

BASH 

cd dataverse-infrastructure
make ansible ENV=tim

(Under the hood: cd dataverse-ansible && ansible-playbook -i ../<inventory> site.yml.)

Read through the output and identify:

  1. Which tasks report changed and which report ok?
  2. Find one shell or command task in tasks/. Does it have a creates: or when: guard?
  3. If this run failed halfway through, per CONTEXT.md what is the supported way to recover?

ok means the module checked state and found nothing to do. changed means it modified something. A freshly-provisioned instance should show mostly changed on first run; re-running against the same still-fresh instance would show more oks for the guarded tasks – but that’s not a scenario this role is meant to be run in twice.

Some shell/command tasks are guarded, some aren’t – that inconsistency is exactly why the repo-wide rule exists.

Per CONTEXT.md: destroy and rebuild (make rebuild), not re-run in place. There is no supported partial-recovery path.

Challenge

Module idempotency vs. playbook safety

In your own words: what’s the difference between “this dnf task is idempotent” and “this playbook is safe to re-run against a live instance”? Why can the first be true while the second is false?

Module idempotency is a per-task property: a well-written module checks current state before acting, so running it twice in a row causes no harm. Playbook-level re-run safety depends on every task in the run having that property, including shell/command tasks that don’t check anything by default. One unguarded task – a schema bootstrap, a cert request, a first-boot API call – is enough to make the whole playbook unsafe to re-run, even though most of its individual tasks are perfectly idempotent on their own.

Key Points
  • dataverse-ansible installs and configures Payara, Solr, Apache, and Dataverse; site.yml is the entry point, and the whole repo is treated as one role.
  • Individual modules (like dnf) are idempotent – but this role, as a whole, is not safe to re-run against a live instance. The operating rule is destroy-and-rebuild, not re-run-in-place.
  • group_vars (all.yml, dev.yml, test.yml, staging.yml) provides environment-specific values without duplicating the role; DOI config lives under nested pid:/doi: blocks, not flat keys.
  • The Ansible inventory is generated from Terraform output (ansible_user: rocky) – gitignored, regenerated on every apply, never hand-edited.
  • Unguarded shell/command tasks are the reason re-running isn’t safe – prefer modules, and guard shell tasks with creates:/when: when you can’t avoid them.

Content from The Dataverse Stack: Payara, Solr, and the Data Layer

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What is Payara and why does Dataverse use it?
  • How does Dataverse store and retrieve data across RDS, S3, and Solr?
  • What breaks when Solr is out of sync, and how do you fix it?

Objectives

  • Describe Payara’s role as a Jakarta EE application server.
  • Explain how Dataverse configuration is applied via JVM options and the API.
  • Describe how RDS, S3, and Solr each serve different data needs.
  • Know when and why a Solr reindex is required.

Payara: the application server

Payara is a Jakarta EE application server – a runtime environment for Java web applications. It is a community fork of GlassFish, maintained specifically for Jakarta EE compatibility. Dataverse chose it because Dataverse is a Jakarta EE application and Payara has continued to receive active maintenance after GlassFish development slowed.

Think of Payara the way you might think of a Python WSGI server or a Node.js process: it is the process that loads the application, handles incoming requests, manages database connections, and keeps the application running.

Dataverse is distributed as a WAR file – a Web Application Archive. Ansible deploys this WAR file into Payara during installation. Payara unpacks it and starts serving requests.

Payara ports

Payara listens on several ports by default:

Port Purpose
8080 HTTP (application)
8181 HTTPS (application)
4848 Admin console
9009 Debug

Apache sits in front of ports 80 and 443. Requests come in through Apache and are proxied to Payara on port 8080 or 8181. Port 4848 is the Payara admin console – it should not be publicly accessible and is locked down in the security group.

Configuring Dataverse through Payara JVM options

Most Dataverse configuration is not stored in a config file. Instead, it is set as JVM options in Payara’s domain configuration (domain.xml). These are key-value pairs that Dataverse reads at startup:

-Ddataverse.files.s3-bucket-name=ucla-dataverse-storage
-Ddataverse.files.storage-driver-id=s3
-Ddataverse.db.host=dev-dataverse-db.cb4k4a6gqn27.us-west-2.rds.amazonaws.com

Ansible sets these JVM options through the Payara admin API during the configure step. You can also inspect or change them manually through the Payara admin console on port 4848, but Ansible is the source of truth – any manual changes will be overwritten on the next playbook run.

First boot and the Dataverse API

Some configuration can only be applied after Dataverse is running. During first boot, Ansible calls the Dataverse API to:

  • Set the root dataverse name and contact email
  • Configure the DOI provider (FAKE for test environments, real EZID for production)
  • Set storage credentials
  • Apply any site-level settings

This is why make rebuild takes several minutes even after Payara is up – the playbook is waiting for Dataverse to finish its own initialization before it can call the API.

Common Payara problems

  • Payara not responding after deploy: Dataverse startup takes 2-5 minutes on first boot. Check the Payara log at /usr/local/payara6/glassfish/domains/domain1/logs/server.log (Payara 6, since the move to Dataverse 6.8 – older docs and issues may still say payara5). Look for Dataverse started or exception stack traces.
  • Out of memory: Payara JVM heap settings are configured in Ansible group_vars. Default is 2GB – increase if you see OutOfMemoryError in the Payara log.
  • WAR deploy failure: The WAR file checksum or version may not match what Payara expects. Check the Ansible payara.yml task output for errors during the deploy step.
Callout

Payara vs. GlassFish

Dataverse documentation and older issues sometimes reference GlassFish commands and paths. Payara is API-compatible – the asadmin command and most paths are the same. If you find a GlassFish-specific workaround in an older issue, it will almost always apply to Payara as well.

The data layer: RDS, S3, and Solr

Dataverse splits its data across three storage systems, each handling a different type:

RDS (PostgreSQL)

RDS stores all structured metadata:

  • Dataset records (titles, descriptions, authors, versions, publication dates)
  • File records (names, checksums, file type, which dataset they belong to)
  • User accounts and permissions
  • Dataverse collection hierarchy
  • Workflow state and notifications

RDS is the authoritative record for what datasets and files exist. When you restore a database backup, you are restoring this metadata.

S3

S3 stores the actual file content – the bytes of every data file users have uploaded. The connection between a file record in RDS and its content in S3 is a storage identifier stored in the database.

After a database restore, Dataverse uses the storage identifiers to serve files from S3. The files in S3 do not change when you restore the database – only the metadata does.

This also means that if a file is deleted from S3 but its record remains in RDS, Dataverse will show the file as available but fail when users try to download it.

Callout

The baseline scripts check both

The make baseline command captures both database counts (via the Dataverse Metrics API) and S3 object counts (via aws s3 ls). Comparing pre- and post-migration baselines verifies that both the database and the storage bucket survived the migration intact.

Solr

Solr is a full-text search engine. Dataverse uses it to power all search and browse functionality – when you type a query in Dataverse or browse a collection, the results come from Solr, not directly from the database.

Solr maintains its own index: a data structure optimized for fast search that is built from the content in the database. This index is separate from and derived from the database.

The index can go out of sync. If you restore the database to an earlier state, the Solr index may contain entries for datasets that no longer exist, or be missing entries for datasets that were added. After any database restore, you must reindex:

BASH 

make reindex ENV=tim

A Dataverse instance with a stale Solr index will appear to have no datasets – or the wrong datasets. This is one of the most common sources of confusion after a rebuild.

The reindex process reads all dataset metadata from the database and sends it to Solr. Depending on how many datasets you have, this can take from seconds to hours.

Callout

make reindex only works because the admin API is open – and that’s a problem

Look at the real target: it’s curl -X DELETE https://$HOST/api/admin/index over public HTTPS. That works only because Dataverse’s admin API is currently unauthenticated and reachable from the internet on this instance – which the infrastructure security audit flags as a Critical finding (F1): the same open admin API also allows dataset destroy on an instance that will eventually hold production research data. make baseline and make test have the identical dependency (F5).

These aren’t separable problems. Closing F1 (blocking the admin API at the proxy, the correct fix) breaks reindex, baseline, and test unless they’re rewritten first to go over SSH instead (ssh rocky@$IP 'curl -s localhost:8080/api/admin/...'). That SSH-based rewrite is planned work, not yet done. Until it lands, every reindex you run is quietly depending on an exposure that shouldn’t exist on a prod-data instance.

Challenge

Trace a file access

Given what you know about the data layer, answer these questions:

  1. A user uploads a file to Dataverse. What gets written to RDS? What gets written to S3?
  2. A user searches for “climate data.” Which storage system answers that query?
  3. You restore the database from a week-old backup. What is the state of S3? What is the state of Solr?
  4. After the restore in question 3, what must you do before the system is usable again?
  1. RDS gets a new file record (name, checksum, storage identifier, dataset reference). S3 gets the file bytes at the storage identifier path.
  2. Solr answers the search query. The database is not involved in search.
  3. S3 is unchanged – it still has all files ever uploaded, including those added in the past week. Solr still has the current index built from the current database before the restore.
  4. Run make reindex ENV=<env> to rebuild the Solr index from the restored database state.
Challenge

Why reindex works today (and why that’s not fine)

  1. What HTTP call does make reindex actually make, and what does it depend on being true about the server?
  2. If that dependency were removed tomorrow (admin API blocked at the proxy), what would break, and what’s the planned fix?
  1. curl -X DELETE https://$HOST/api/admin/index – a public-HTTPS call to Dataverse’s admin API. It depends on that API being unauthenticated and reachable from outside the instance, which it currently is (audit finding F1, Critical).
  2. make reindex, make baseline, and make test would all start failing (F5) – they all hit admin/metrics endpoints the same way. The planned fix is rewriting those calls to go over SSH to localhost:8080 on the instance instead of public HTTPS.
Key Points
  • Payara is a Jakarta EE application server; Dataverse runs as a WAR file inside it.
  • Most Dataverse configuration is set as Payara JVM options, managed by Ansible.
  • The Payara log at payara6/glassfish/domains/domain1/logs/server.log is the first place to look when things go wrong.
  • RDS holds metadata; S3 holds file content; Solr holds the search index.
  • Always run make reindex after a database restore – Solr does not update itself.
  • make reindex/baseline/test currently depend on the admin API being open over public HTTPS – a known Critical security exposure (F1/F5), not a stable design choice.

Content from Secrets and Environment Configuration

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • How are secrets kept out of the repository?
  • What configuration is different between staging and production?
  • What is a FAKE PID provider and why does it exist?

Objectives

  • Explain what Ansible Vault does and how to use it.
  • Describe the differences between test and production configuration.
  • Identify which config values are environment-specific vs. shared.
  • View a vaulted file’s contents using the vault password.

Ansible Vault

Ansible Vault is a tool for encrypting sensitive values so they can be stored in the repository without exposing secrets. The vault password decrypts them at playbook runtime.

Common secrets stored in the vault for this project:

  • Database password for the RDS instance (dataverse_postgresql_password)
  • Dataverse admin password (dataverse_adminpass)
  • EZID credentials (production DOI registration)

Notably absent: AWS credentials for S3 writes are not a vaulted secret at all. The EC2 instance authenticates to S3 through an IAM instance profile (s3.use_iam_role: true in group_vars) – there’s no access key to leak in the first place, which is the safer design and worth naming as deliberate, not an oversight.

There’s no separate group_vars/all/vault.yml file – secrets are inline, encrypted in place inside the same flat group_vars/<env>.yml files as everything else, using !vault | blocks:

YAML 

dataverse_adminpass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          663365396...

To decrypt and view one value, or a whole file:

BASH 

ansible-vault view group_vars/dev.yml

To edit a vaulted value in place:

BASH 

ansible-vault edit group_vars/dev.yml

Both commands prompt for the vault password (the .vault-password file from Episode 2). That file itself is never committed to the repository.

Callout

Never commit unencrypted secrets

If you accidentally add a plaintext secret to the repo, treat it as compromised and rotate it. Remove it from git history using git filter-branch or git filter-repo, then notify the team. The vault exists to prevent this – if in doubt, vault it.

Test cert vs. real SSL

In production, Dataverse serves HTTPS using a Let’s Encrypt certificate obtained via Certbot. Certbot contacts Let’s Encrypt’s servers to verify domain ownership and issue the certificate.

In test environments, running Certbot would:

  • Fail if the test EC2 instance is not reachable on a public domain
  • Hit Let’s Encrypt rate limits during rapid rebuilds
  • Register a real certificate for a temporary hostname

Instead, test environments use a self-signed certificate. In group_vars, this is controlled by a nested key under letsencrypt.certbot, not a flat variable:

YAML 

letsencrypt:
  certbot:
    test_cert: true   # dev.yml, test.yml -- staging certs during iterative rebuilds
    # test_cert: false  # TEMPLATE.yml, staging.yml default -- flip to false only at production cutover (Phase 7)

When test_cert: true, Ansible generates a self-signed certificate locally and skips Certbot entirely. Browsers will show a security warning for self-signed certs – that is expected.

FAKE PID provider

DOI registration is the process of minting a persistent identifier for a dataset and registering it with a DOI service (UCLA uses EZID, which registers with DataCite).

In production, every published dataset gets a real, resolving DOI. In test environments, you do not want to:

  • Mint real DOIs that point to a test server
  • Hit EZID’s API during rapid rebuilds and testing
  • Risk cluttering the production DOI namespace with test records

Dataverse has a built-in FAKE PID provider for exactly this purpose. It generates DOI-like identifiers (they look like DOIs but do not resolve) without contacting any external service.

In group_vars, this isn’t one variable but two nested blocks – pid: (the identifier format) and doi: (the registration service):

YAML 

pid:
  authority: "10.5072"
  protocol: doi
  shoulder: "FK2/"

doi:
  provider: FAKE     # dev.yml, test.yml
  # provider: EZID   # production, not configured yet -- no production group_vars file exists

The FAKE provider is used in all non-production environments throughout the migration. The switch to real EZID happens only at Phase 7 (DNS cutover) – and since there’s no production.yml yet, that switch requires writing production config, not just flipping a value in an existing file.

Environment configuration summary

Setting dev / test (tim, jamie) staging / TEMPLATE default Production
SSL certificate Self-signed (test_cert: true) Let’s Encrypt (test_cert: false) Not yet configured
DOI provider FAKE (no external calls) FAKE EZID (planned, Phase 7)
Vaulted? dev.yml partially vaulted staging.yml/test.yml have unvaulted CHANGE_ME_USE_VAULT placeholders –
S3 access IAM instance profile (no credentials to vault) same same
Callout

Vaulting is a work in progress, not a finished state

Don’t assume every environment’s secrets are actually encrypted right now. test.yml currently has dataverse_adminpass: "CHANGE_ME_USE_VAULT" and dataverse_postgresql_password: "CHANGE_ME_USE_VAULT" in plaintext – literal placeholder strings, not real secrets, but also not vaulted. all.yml (the shared defaults every environment inherits unless it overrides them) has real plaintext defaults too, like adminpass: admin. Treat “is this value vaulted in dev.yml” and “is this value vaulted in test.yml” as two separate questions with two different answers today.

Challenge

Spot the difference

Open group_vars/dev.yml and group_vars/test.yml in the dataverse-ansible repo.

  1. Which top-level keys are shared structure (same key, different value) between the two files?
  2. Find dataverse_adminpass in each file. Is it vaulted (!vault |) in both? If not, which one isn’t, and what does that tell you about deploy-readiness?
  3. Find the pid:/doi: blocks. What’s the DOI provider in each?
Key Points
  • Secrets are vaulted inline inside group_vars/<env>.yml (!vault | blocks) – there is no separate group_vars/all/vault.yml file.
  • S3 access uses an IAM instance profile, not vaulted AWS credentials – there’s no access key to leak.
  • Test environments use self-signed certificates (letsencrypt.certbot.test_cert: true); production would use Let’s Encrypt.
  • DOI config is two nested blocks, pid: and doi:, not one flat provider variable. FAKE is used everywhere non-production; EZID is planned for Phase 7 and has no group_vars file yet.
  • Vaulting is incomplete today: test.yml has real unvaulted CHANGE_ME_USE_VAULT placeholders. Don’t assume every environment is equally secret-safe.

Content from The Makefile: Daily Operations

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What can I do from the Makefile?
  • What does make rebuild actually do, step by step?
  • When do I run make baseline vs. make reindex?

Objectives

  • List the main Makefile targets and what each does.
  • Trace the steps of make rebuild in order.
  • Run make baseline and inspect the snapshot it produces.
  • Know when each operational target should be used.

Why a Makefile

The dataverse-infrastructure Makefile is the daily operations interface for this project. It wraps Terraform, Ansible, and the baseline scripts behind named targets so you rarely need to invoke those tools directly.

A Makefile target is a named command. Running make <target> executes the commands associated with that target. For example:

BASH 

make rebuild ENV=tim

…runs a sequence of Terraform and Ansible commands in the right order, with the right arguments, for Tim’s environment.

The Makefile lives in dataverse-infrastructure/. Most targets require an ENV argument that specifies which operator environment to work with.

Main targets

make rebuild

Destroys and recreates the entire environment – EC2, RDS, and S3 together – then configures it and restores data. It requires DB_PASS and prints a real warning before it runs, because of exactly the misconception this section used to encode: this is not an EC2-only operation.

The real 7 steps, from the Makefile itself:

  1. Destroy – terraform destroy tears down the whole per-environment module: EC2, RDS, and the S3 bucket all go together. Nothing about this step is EC2-scoped.
  2. Provision – terraform init -upgrade then terraform apply builds all of it back from scratch: new EC2 instance, new empty RDS database, new empty S3 bucket.
  3. DNS propagation – prints the new EC2 IP and pauses for you to manually update the DNS A record (this is the Elastic IP gap from Episode 3 in practice – the IP really did change, and nothing updates DNS automatically yet).
  4. Ansible – runs site.yml to install and configure Payara, Solr, Apache, and Dataverse.
  5. Restore the database from S3 – scripts/restore-db.sh pulls the latest dump and loads it into the just-created, currently-empty RDS instance. This step exists because step 1 wiped the database – without it, rebuild would hand you an empty Dataverse.
  6. Start Payara – over SSH (ssh rocky@$IP 'sudo systemctl start payara').
  7. Wait, then reindex – polls the app until it responds, then runs make reindex to rebuild Solr from the just-restored database.

There is no test-suite step at the end – rebuild ends at reindex and tells you to make logs to monitor. Running make test afterward is a separate, manual step.

Use make rebuild when:

  • Starting fresh after a failed or degraded environment
  • Testing infrastructure changes that require a clean stack
  • Validating a new Ansible configuration from scratch

BASH 

make rebuild ENV=tim DB_PASS=<dataverse_postgresql_password>
Callout

Nothing survives a rebuild by default – the restore step is what saves you

The single most consequential fact about make rebuild: it destroys RDS and S3 along with EC2, and the reason the environment isn’t empty afterward is step 5, restoring from a database dump in S3 (ucla-dataverse-migration-assets) – a separate bucket from the one terraform destroy just deleted. The security/reliability audit of this repo calls this restore-every-rebuild pattern “the single most valuable reliability practice here” – it means every rebuild is implicitly a disaster-recovery drill, proving the backup actually works. But it also means a stale or missing dump turns rebuild into “spin up an empty Dataverse,” not “restore my environment.” There’s no confirmation step that checks dump freshness before restoring (a known gap, audit F4).

make baseline

Captures a timestamped JSON snapshot of the current state of the Dataverse instance. The snapshot includes:

  • Dataset count (published, draft, total)
  • File count
  • User count
  • Dataverse collection count
  • S3 object count and total bytes

BASH 

make baseline ENV=tim

The snapshot is saved to baseline-snapshots/baseline_<timestamp>.json and optionally uploaded to S3 if BASELINE_UPLOAD_BUCKET is set:

BASH 

BASELINE_UPLOAD_BUCKET=ucla-dataverse-migration-assets make baseline ENV=jamie

Jamie’s production baseline (run before migration begins) is the anchor for the Phase 7 post-cutover comparison. Upload it to S3 so it is durable and shared.

make baseline-compare

Compares two baseline snapshots and reports differences:

BASH 

make baseline-compare BEFORE=baseline-snapshots/before.json AFTER=baseline-snapshots/after.json

A passing comparison shows matching counts across all fields – with one deliberate exception: downloads/guestbook-history drift is treated as informational only, not a failure, since download counts can legitimately keep changing between the two snapshots. Any discrepancy in dataset or file counts, though, is a problem to investigate before declaring the migration complete.

make reindex

Triggers a full Solr reindex of all datasets from the database.

BASH 

make reindex ENV=tim

Run this after:

  • Any database restore
  • A make rebuild that restored from a backup
  • Noticing that Dataverse search returns no results or stale results

Reindexing triggers Dataverse to read all dataset metadata from RDS and send it to Solr. The time it takes depends on how many datasets exist.

Under the hood this is a curl -X DELETE to Dataverse’s admin API over public HTTPS – which only works today because that API is currently open to the internet, a Critical security finding covered in depth in Episode 5. make baseline and make test share the same dependency.

make test

Runs the pytest test suite against the target environment:

BASH 

make test ENV=tim

The tests cover API smoke tests, S3 connectivity, Solr health, and basic CRUD operations. See Episode 8 for detail on what the tests check.

The ENV argument

Most targets require ENV=<operator> to specify which environment to use. Valid values are tim and jamie. This controls:

  • Which Terraform environment directory is used (environments/tim or environments/jamie)
  • Which Ansible inventory file is used
  • Which group_vars overrides are applied

Running make rebuild without ENV will error. Always specify it.

Challenge

Trace a rebuild – for real this time

Open dataverse-infrastructure/Makefile and find the rebuild target. Without relying on memory of this episode, answer from the actual Makefile:

  1. What happens to the RDS database during step 1? What restores it, and from where?
  2. Does the target end with a test run? What does it actually end with?
  3. What manual action does step 3 require from the operator, and what does that tell you about the Elastic IP’s current state (Episode 3)?
  1. terraform destroy in step 1 deletes the RDS instance along with EC2 and S3 – the whole module goes together. Step 5, scripts/restore-db.sh, restores it from a dump pulled from the ucla-dataverse-migration-assets S3 bucket (a separate bucket from the one that just got destroyed).
  2. No. It ends with step 7 (wait for the app, then make reindex) and a message pointing you to make logs to monitor. make test is a separate command you run yourself afterward.
  3. Step 3 pauses and prints the new EC2 IP, asking you to update the DNS A record by hand before continuing. That’s only necessary because the Elastic IP doesn’t yet survive a rebuild (Episode 3) – if it did, this manual step wouldn’t exist.
Key Points
  • The Makefile is the daily operations interface – rarely run Terraform or Ansible directly.
  • make rebuild ENV=<env> DB_PASS=<pass> destroys and recreates EC2, RDS, and S3 together, then restores the database from an S3 dump. It does not preserve data by default – the restore step is what puts data back.
  • make baseline ENV=<env> captures a timestamped snapshot to baseline-snapshots/, with dataset, file, and S3 counts.
  • make reindex ENV=<env> rebuilds the Solr index after any database restore – and depends on the admin API being open over public HTTPS (a known security gap, Episode 5).
  • Always specify ENV= – the Makefile will error without it.

Content from Testing and Validation

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • How do we know the migration worked?
  • What does the test suite check?
  • How do baseline comparisons verify data integrity?

Objectives

  • Run the pytest suite and read its output.
  • Explain what each test category checks.
  • Compare two baseline snapshots and interpret the diff.
  • Know what must pass before each migration phase can proceed.

Why testing matters here

For a database migration, “did it work?” is not obvious. Dataverse can appear to start successfully while silently missing datasets, serving stale search results, or failing on file downloads.

The test suite and baseline comparisons are the answer to that question. Together they check:

  • The application is running and accepting requests
  • The database has the expected number of datasets, files, and users
  • S3 has the expected number of objects
  • Solr is indexed and returning results
  • File upload and download work end to end

No migration phase is complete until these pass.

The pytest suite

Tests live in dataverse-ansible/tests/integration/ – the child repo, not the orchestration repo. There’s no tests/ directory in dataverse-infrastructure itself.

Run the full suite:

BASH 

make test ENV=tim

That target expands to, roughly:

BASH 

cd dataverse-ansible && uv run pytest tests/integration -v --dataverse-url=https://<hostname>

Note uv, not a bare pytest – and the --dataverse-url flag is required, since these tests hit a live instance over the network rather than running against localhost.

Test categories (real classes, from test_smoke.py)

TestAPIHealth: Check that the Dataverse API responds on expected endpoints. These are the fastest tests and fail first if Payara is down or misconfigured.

TestS3Integration: Verify that Dataverse can read from and write to S3. A test file is uploaded via the API and then downloaded. If the IAM instance profile or bucket config is wrong, this fails.

TestSolrIndexing: Check that Solr is running and the index is not empty. An empty index does not cause Dataverse to error – it just silently returns no search results. This test catches that.

TestSearch, TestDataverse, TestDatasets, TestWebInterface, TestAuthentication: round out the smoke suite – basic CRUD, page rendering, and login checks.

Baseline count comparison is a separate step (make baseline-compare), not a pytest class – it compares two JSON snapshots, not live API responses.

PID/DOI checks exist, but are narrower than they sound. test_migration.py has a TestPIDConfiguration class, but it only runs with the migration pytest marker (make ansible-migration, not the default make test), and it mostly checks that PID settings exist in the database – it doesn’t distinguish FAKE from a real EZID connection. A dedicated DOI/FAKE-provider validation test is planned (roadmap 03-03) but not yet built.

Reading test output 

PASSED tests/test_api.py::test_dataverse_api_version
PASSED tests/test_s3.py::test_s3_upload_download
FAILED tests/test_solr.py::test_solr_index_not_empty
  AssertionError: Solr returned 0 results -- did you run make reindex?

Any FAILED test is a blocker. Fix the underlying problem before proceeding. The error message usually tells you what to do.

Baseline comparisons

Capturing baselines

Run make baseline before and after any significant operation:

BASH 

# before migration
make baseline ENV=jamie
mv baseline-snapshots/baseline_<timestamp>.json baseline-snapshots/pre-migration.json

# ... do migration work ...

# after migration
make baseline ENV=jamie
mv baseline-snapshots/baseline_<timestamp>.json baseline-snapshots/post-migration.json

There’s no latest.json – every capture gets its own timestamped filename, so renaming (or tracking the filename make baseline prints) is how you keep pre/post straight.

Comparing

BASH 

make baseline-compare BEFORE=baseline-snapshots/pre-migration.json AFTER=baseline-snapshots/post-migration.json

A clean comparison looks like:

datasets:    1,247  ->  1,247  OK
files:      18,903  -> 18,903  OK
users:         142  ->    142  OK
s3_objects: 18,903  -> 18,903  OK
s3_bytes:   84.2GB  ->  84.2GB OK

Any mismatch is a problem to investigate – except one field. downloads/guestbook history is deliberately treated as informational-only in baseline-compare.sh, not a failure: download counts legitimately keep incrementing as long as the instance is live, so a “drift” there doesn’t mean data was lost. Datasets and files are the fields that must match exactly. Common causes of a real mismatch:

  • Solr not yet reindexed (run make reindex, then re-run the comparison)
  • A dataset was published or retracted between captures (check the timing)
  • Files missing from S3 (check S3 directly with aws s3 ls)
  • Database restore captured different data than expected (verify backup timestamp)
Callout

The Phase 7 comparison is the final gate

The baseline captured by Jamie before any migration work (run with BASELINE_UPLOAD_BUCKET set) is compared against the post-cutover production environment. If these do not match, the migration is not complete. Do not finalize the cutover until the comparison passes.

Challenge

Interpret a failing test run

Imagine you run make test ENV=tim and see:

PASSED tests/test_api.py::test_dataverse_api_version
FAILED tests/test_s3.py::test_s3_upload_download
  ConnectionError: Could not connect to S3 endpoint
PASSED tests/test_solr.py::test_solr_index_not_empty
  1. What does this tell you about the state of the environment?
  2. What are the two most likely causes of the S3 failure?
  3. What would you check first?

The API and Solr are working, so Payara is up and Solr is indexed. S3 connectivity is the problem – Dataverse cannot reach S3.

Two most likely causes:

  1. The S3 bucket name is wrong in the JVM options, or the IAM instance profile attached to the EC2 instance doesn’t grant the right permissions on that bucket (check group_vars for the bucket name, and the Terraform IAM role/policy for permissions – there are no AWS access keys to check, since this uses an instance profile, not vaulted credentials)
  2. The security group does not allow outbound HTTPS to S3 (check the Terraform security group config)

Start with the bucket name and IAM policy – they’re the most common source of S3 config problems. Check the Payara log for AmazonS3Exception or AccessDenied messages.

Challenge

The one field that’s allowed to drift

Without checking the episode: a baseline-compare run shows datasets: 1,247 -> 1,247 OK, files: 18,903 -> 18,903 OK, but downloads: 3,401 -> 3,512. Is this a failure? Why or why not?

Not a failure. downloads/guestbook history is treated as informational-only by baseline-compare.sh – download counts naturally keep incrementing while an instance is live and serving traffic, so a difference there reflects normal usage, not lost or corrupted data. Datasets and files matching exactly is what actually gates a migration phase.

Key Points
  • The test suite (dataverse-ansible/tests/integration/, run via make test) covers API health, search, S3 round-trip, and Solr indexing as real pytest classes.
  • PID/DOI-specific validation is thin today: a TestPIDConfiguration class exists but only runs under the migration marker and mostly checks settings exist, not FAKE-vs-EZID behavior.
  • Baseline comparisons verify data integrity by comparing counts before and after migration, saved to baseline-snapshots/ (no latest.json).
  • Every baseline field must match exactly except downloads, which is deliberately informational-only.
  • Jamie’s pre-migration production baseline is the anchor for the final Phase 7 comparison.

Content from Using AI Tools in Infrastructure Work

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • Where does AI actually help with infrastructure work, and where does it mislead?
  • What should I never share with an AI tool?
  • How do I stay technically grounded when AI is doing a lot of the work?

Objectives

  • Identify tasks where AI assistance is genuinely useful in this context.
  • Identify tasks where AI output requires careful verification or should not be trusted.
  • Apply a consistent habit of sanitizing inputs before sharing anything with an AI tool.
  • Name at least two strategies for staying grounded when AI accelerates your output.

What this episode is about

AI tools are useful in infrastructure work. They are also easy to misuse – not through dramatic failures, but through subtle ones: plausible-looking config that is slightly wrong, outdated API syntax, or confident answers to questions the tool cannot actually know.

This episode covers two things. First, the practical mechanics: what to ask, what to verify, and what to keep out of the conversation entirely. Second, something less often discussed: what happens to your own understanding when AI does a lot of the heavy lifting, and what you can do about it.

What your institution has licensed

Before using any AI tool for work, check what your institution has approved. Institutional licenses typically include data handling agreements that consumer-grade tools do not. Using an unlicensed tool for work-related content – especially anything involving infrastructure, access credentials, or user data – may violate your institution’s acceptable use policy.

At UCLA, the institution licenses Gemini for work use via Google Workspace. This is the appropriate tool for UCLA Library staff working on infrastructure tasks. When in doubt, check with your IT or information security team about what is approved.

Using a licensed tool does not mean anything goes – it means the data handling terms are known and accepted. The practices in this episode still apply.

What not to share

Some content should never be pasted into an AI tool, regardless of which tool or what license covers it:

  • Secrets and credentials: passwords, API tokens, Ansible Vault contents, AWS access keys
  • Private hostnames and IPs: internal RDS endpoints, EC2 instance addresses, VPN addresses
  • Database connection strings: these combine a hostname, port, database name, username, and password
  • SSH private keys
  • Personally identifiable information: user account data, emails, names from the database

For infrastructure work, this means sanitizing before sharing. Replace real values with placeholders:

YAML 

# what you have
db_host: dev-dataverse-db.cb4k4a6gqn27.us-west-2.rds.amazonaws.com
db_password: "actual-password-here"

# what you share with AI
db_host: <rds-endpoint>
db_password: "<redacted>"

The AI does not need the real values to help you. Hostnames and passwords do not change what an Ansible task should look like or why a Terraform plan is failing.

Where AI helps

Config systems and infrastructure-as-code

This is where AI tools are genuinely strong, and it is directly relevant to this lesson. Terraform and Ansible are pattern-heavy, well-documented, and structurally consistent. AI tools have seen enormous amounts of both. They are good at:

  • Generating correct HCL resource blocks from a description
  • Explaining what an Ansible module does and what its options mean
  • Debugging Terraform plan output or Ansible task failures
  • Translating “here is what I want to happen” into the right module or resource type

This is not a coincidence. Config systems are exactly the kind of domain where AI excels: large training corpus, consistent syntax, well-defined semantics, errors with recognizable patterns.

Understanding unfamiliar tools and concepts

When you encounter something new, AI gives you a faster orientation than documentation alone. “What is Payara and how does it relate to GlassFish?” or “explain what Ansible handlers do and when they run” produces a usable explanation quickly, tailored to your context.

Explaining error messages

Paste a sanitized error message and ask what it means. For common errors from Terraform, Ansible, Payara, or AWS, AI tools often recognize the pattern and explain the root cause and likely fixes faster than searching.

Drafting documentation and runbooks

AI is useful for turning rough notes into structured documentation – filling in a runbook outline, writing a glossary entry, turning a list of steps into a coherent procedure. This lesson was partly developed that way.

Talking through a problem

Describing a problem before asking a specific question often helps clarify your own thinking. “I ran make rebuild and Payara is up but Dataverse search returns no results – what are the possible causes?” returns a structured list you can work through.

Where AI misleads

Version-specific configuration

AI training data has a cutoff. Dataverse, Payara, Terraform provider syntax, and Ansible module arguments all change between versions. The AI may confidently give you configuration syntax that was correct in an older version and is now wrong or deprecated.

Always check the version-specific documentation for anything configuration-related. Do not assume AI-generated config is correct for the version you are running.

Your specific environment

The AI does not know your S3 bucket names, your RDS endpoint, your Terraform state, or what is currently deployed. When you need environment-specific answers, the tools are: AWS console, Terraform state, Ansible --check mode, and the Payara log. AI is a supplement to these, not a replacement.

Plausible but wrong

AI tools produce fluent, confident output even when they are wrong. A hallucinated Terraform resource argument or a non-existent Ansible module option looks exactly like a correct one. The text is convincing; the output may not work.

This is the central risk. Never copy AI-generated configuration into a production context without running it through the validation steps: terraform plan, --check mode, the test suite.

Security decisions

Do not outsource security decisions to an AI tool. Questions like “is this IAM policy safe?” benefit from AI explanation, but the final judgment belongs to a person who understands the actual threat model and has access to the full context.

The deeper risk: losing contact

Because AI is particularly good at config systems – Terraform, Ansible, CI pipelines – it is possible to move very fast without fully understanding what was built. The acceleration is real. So is the risk.

This is not unique to AI. It is a well-studied pattern in automation research. When a system handles the hard parts reliably, people stop building the mental model that lets them catch when it is wrong. In aviation and medicine, this is called automation bias: over-trusting automated output because it is usually right, even when evidence of an error is present. The cognitive load of staying engaged is high, and the tool usually gets it right – until it doesn’t.

For infrastructure work, the specific version looks like this:

  • AI writes your Ansible task. It works. You move on.
  • Six months later, the task behaves unexpectedly in a new context.
  • You do not have the mental model to know why, because you never built it.

The output was correct. The understanding was not transferred.

A useful framing here comes from chess: after computers became stronger than humans, the best human-computer teams were not grandmasters who let the engine decide everything, but players who kept their own judgment engaged and used the engine to check it. The technical term in HCI research is cognitive offloading – and the question is not whether to do it, but which parts you can safely offload and which you must own.

For this work, the parts you need to own:

  • Why idempotency matters and how to verify it
  • What each layer (Terraform, Ansible, Dataverse) is responsible for
  • What the test suite is actually asserting
  • What the migration phases gate and why they are sequenced the way they are

These are not things AI can hold for you. If you cannot explain them without opening a chat window, you do not have them yet.

Strategies for staying grounded

Write it down in your own words (this lesson)

Writing this lesson is itself a strategy. Turning what was built into structured explanation – not AI-generated, but your own reconstruction – forces contact with the material. If you cannot explain the Solr reindex requirement in a callout block, you do not understand it yet.

This is the oldest study technique there is, and it works: retrieval practice, or the “protege effect.” Teaching something consolidates the understanding in a way that reading or watching does not.

Ask for explanation, not just output

Instead of “write me an Ansible task to configure Payara JVM options,” try: “explain how Ansible configures Payara JVM options, then show me an example task.” The explanation gives you something to evaluate the output against.

If the explanation does not make sense, do not use the output.

State your versions and read the diff

When asking for configuration, include the versions you are running: “Terraform 1.7, AWS provider 5.x, Ansible 2.16, Dataverse 6.3.” This reduces outdated syntax.

Then read the generated output line by line, the same way you would read a code review. If you cannot explain what a line does, look it up before using it.

Validate before applying – always

Every piece of AI-generated configuration should go through the same path as everything else: terraform plan, --check mode, the test suite.

Make this non-negotiable. AI output is a draft, not a finished product.

Practice without the tool periodically

Work through a piece of configuration or a debugging problem without AI assistance, even if it takes longer. This is not about refusing help – it is about checking that the skill is still there when you need it. Pilots practice manual landings even though automation is usually better. The same logic applies.

Pair with another person

When another person is present – Jamie, a student, a colleague – explain what you are doing and why. This is not about the other person checking your work; it is about the act of explaining. If you cannot explain it, you do not understand it. Onboarding Leigh or a new DataSquad student is actually useful for this: teaching someone who does not have the context forces you to articulate things you have internalized.

Callout

A question worth asking periodically

Could you rebuild this environment – Terraform, Ansible, Dataverse configuration – from scratch without AI assistance if you had to?

Not immediately, and not perfectly. But could you trace the steps, know what to look for, and understand why each piece is there?

If the answer is uncertain, that is a signal. Not to stop using AI, but to spend some time with the material on your own terms.

Challenge

Sanitize a config snippet

Take the following snippet from a group_vars file:

YAML 

dataverse_db_host: prod-dataverse-db.abc123xyz.us-west-2.rds.amazonaws.com
dataverse_db_password: "s3cur3P@ssw0rd!"
dataverse_s3_bucket: ucla-dataverse-production
dataverse_doi_authority: "10.25346"

Rewrite it as you would share it with an AI tool when asking for help with a configuration problem. What is safe to share? What should be replaced?

YAML 

dataverse_db_host: <rds-endpoint>
dataverse_db_password: "<redacted>"
dataverse_s3_bucket: <s3-bucket-name>
dataverse_doi_authority: "10.25346"

The DOI authority (10.25346) is a public value – safe to share. The RDS endpoint, password, and bucket name should be replaced. The bucket name is not a credential, but it identifies your institution’s storage; use a placeholder when the real name is not needed to answer the question.

Challenge

Name the failure mode

An AI-written Ansible task worked fine when it was added. Six months later, it misbehaves in a new context and you can’t figure out why.

  1. Without re-reading the section above, name the specific failure mode this episode uses for why you can’t debug it.
  2. Name one concrete practice from this episode that would have prevented it.
  1. Automation bias (over-trusting output that’s usually correct, so you stop building the mental model needed to catch it when it’s wrong) is the underlying pattern; the proximate cause here is that the understanding was never transferred – the task worked, so you moved on without owning why it worked.
  2. Any of: asking for an explanation before accepting the output, reading the generated config line by line, or writing your own explanation of what the task does (retrieval practice) at the time it was added – not six months later when it breaks.
Challenge

Which tool, and why it matters

Which AI tool is UCLA’s licensed option for this kind of work, and why does that matter operationally (not just for compliance) when the config you’re discussing includes real hostnames, bucket names, or infrastructure details?

Gemini via Google Workspace. It matters beyond compliance because a licensed institutional tool comes with data-handling terms your institution has actually reviewed – an unlicensed consumer tool may retain or train on what you paste in, which matters even after redacting credentials, since hostnames, bucket-naming patterns, and architecture details are still information about UCLA’s infrastructure.

Key Points
  • Use your institution’s licensed AI tools for work; at UCLA, that is Gemini via Google Workspace.
  • Never share credentials, private hostnames, database connection strings, or SSH keys with any AI tool.
  • AI is particularly strong with config systems like Terraform and Ansible – and that is exactly where the risk of losing contact is highest.
  • Automation bias is real: over-trusting AI output because it is usually right, until it isn’t.
  • Strategies for staying grounded: write it down in your own words, ask for explanation before output, validate everything, practice without the tool periodically, teach it to someone else.
  • Writing this lesson is itself one of those strategies.

Content from The Migration Arc: 5.14 to 6.8

Last updated on 2026-07-30 | Edit this page

Overview

Questions

  • What changes between Dataverse 5.14 and 6.8?
  • Why is the migration done in phases?
  • What happens on cutover day and what does rollback look like?

Objectives

  • Explain the 7-phase migration plan and why it is sequenced the way it is.
  • Describe what changes between Dataverse 5.14 and 6.8.
  • Identify the rollback decision points and what each one requires.
  • Explain what the Elastic IP and DNS TTL reduction accomplish at cutover.

What changes in 6.x

Dataverse 6.x introduced several significant changes from the 5.x series:

  • Storage subsystem refactor: S3 configuration moved to a new format; the storage driver ID must be set explicitly
  • DOI/PID provider changes: The way persistent identifiers are configured changed; the FAKE provider configuration syntax differs
  • Solr schema updates: The Solr index schema changed; the old index is incompatible and must be rebuilt from scratch
  • Java version requirement: Dataverse 6.x requires Java 17 (up from Java 11 in 5.x)
  • API changes: Some API endpoints changed or were deprecated; any integrations need to be verified

These changes mean a 5.x database cannot simply be imported into a 6.x instance without migration steps. The migration process handles database schema updates automatically – Dataverse runs schema migrations on startup – but the Solr index must be rebuilt manually.

The 7-phase plan

The migration is structured in seven phases. Each phase ends with a gate: the test suite and (where applicable) baseline comparisons must pass before the next phase begins.

Phase 1: Production Baseline Capture (complete)

Capture a timestamped snapshot of the production 5.14 instance before any migration work begins. This snapshot is the anchor for the final comparison after cutover.

  • Run by Jamie against production with BASELINE_UPLOAD_BUCKET set
  • Stored in S3 for durability

Phase 2: Rebuild + Infrastructure Hardening (current, partially complete)

Goal: Tim’s dev environment rebuilds cleanly with an Elastic IP (no DNS wait on each rebuild), FAKE DOI provider, test_cert: true enforced, and Solr reindex as an explicit post-restore gate. Three plans, and they’re not all done:

  • 02-01 Elastic IP resource in Terraform – not started. This is the item covered in Episode 3: an aws_eip resource exists, but it’s tied to the instance’s lifecycle and doesn’t survive terraform destroy, so rebuilds still require a manual DNS update.
  • 02-02 FAKE DOI provider config + test_cert: true + Solr reindex make target – done.
  • 02-03 Full make rebuild ENV=tim cycle validation (clean rebuild, reindex, smoke tests pass) – not started.

So “Phase 2 complete” would mean all three; right now it’s one of three. Don’t treat this phase as finished just because the rebuild command runs.

Phase 3: Expanded Test Coverage

Extend the test suite with:

  • Baseline count comparison tests
  • S3 file upload and download round-trip
  • DOI/FAKE validation
  • Explicit Solr reindex gate

These tests gate every subsequent phase.

Phase 4: Jamie’s Environment Onboarding

Jamie can independently:

  • Destroy and rebuild her environment from scratch
  • Run the full test suite
  • Do this using only the runbook (not by asking Tim)

This proves the process is operator-independent. Required before production cutover.

Phase 5: Shibboleth / SSO

Configure Shibboleth authentication via Ansible, register the SP metadata with the UCLA campus IdP, deploy to Jamie’s environment, and validate the full login flow.

This phase has a long lead time – campus IT needs to register the SP metadata. Coordination starts in Phase 2 to avoid blocking the cutover schedule.

Phase 6: Maintenance Window Planning

Document the complete cutover procedure before executing it:

  • Step-by-step cutover sequence with timing estimates
  • Rollback decision points and what each one requires
  • DOI/EZID switch procedure
  • User communication templates
  • DNS TTL reduction checklist

Nothing on cutover day should be improvised. This phase produces the document that the team follows on cutover day.

Phase 7: DNS Cutover

The production migration itself:

  1. Reduce DNS TTL to 60 seconds (done days in advance)
  2. Open maintenance window; notify users
  3. Run final production DB dump
  4. Restore DB to 6.x environment
  5. Run make rebuild ENV=jamie against 6.x
  6. Run Solr reindex
  7. Run full test suite; baseline comparison must pass
  8. Switch DNS A record to the Elastic IP on the 6.x instance
  9. Switch DOI/EZID configuration from FAKE to real
  10. Monitor for 24 hours
  11. Terminate old 5.14 instance

Rollback is possible until step 9 (EZID switch). If anything fails before that point, switch DNS back to the old instance and the 5.14 instance is back up within the TTL window.

Callout

A gap this sequence doesn’t cover: the actual file bytes

Steps 3-6 move the database – dump, restore, rebuild, reindex. None of them move the files themselves from the old instance’s storage to the new one’s S3 bucket. The infrastructure security/reliability audit flags this as a High-severity gap (F6): today, file-bytes migration exists only as manual prose in the ansible repo’s migration guide (an aws s3 sync command a human is expected to run and remember), and the integrity checks in step 7 (baseline comparison) count database rows and S3 object counts – they don’t verify that a given file is actually downloadable. A cutover that follows only the 11 steps above could produce a production Dataverse where datasets exist, search works, and downloads 404. A round-trip file-retrievability test is planned (roadmap 03-02) but isn’t itself a migration step – something still has to actually move the bytes, and that’s not automated yet. Resolve this before Phase 7, not during it.

Rollback decision points

Point Rollback action
Before DNS switch Switch DNS back; old instance still running
After DNS switch, before EZID Switch DNS back; new DOIs minted as FAKE can be re-minted
After EZID switch Rollback is complex; requires DOI management coordination

The goal is to not reach a point where rollback is complex. The test suite gate after the DB restore (step 7) is the last clean opportunity to stop.

DNS TTL and the Elastic IP

DNS TTL (Time To Live) controls how long resolvers cache the DNS record. If the TTL is 3600 seconds (one hour) and you switch the DNS A record, some users will still be routed to the old IP for up to an hour.

The procedure reduces TTL to 60 seconds several days before cutover. At that TTL, the propagation delay after the DNS switch is at most 60 seconds.

The plan is for an Elastic IP to give the new 6.x instance a known, stable address before cutover day, so the DNS change is a single A record update. That depends on 02-01 landing first, though (see Phase 2 above) – as of today the EIP doesn’t survive a rebuild, so “known, stable IP before cutover” isn’t true yet. If cutover happened this week, step 8 below would still need the same manual “read the new IP off the Terraform output” step every other rebuild requires.

Challenge

Phase check

For each phase below, identify what must be true before that phase can start:

  1. Phase 3 (Expanded Test Coverage)
  2. Phase 4 (Jamie’s Environment Onboarding)
  3. Phase 7 (DNS Cutover)
  1. Phase 3 requires Phase 2 complete: Tim’s env must rebuild cleanly with Elastic IP, FAKE DOI, and test_cert enforced.
  2. Phase 4 requires Phase 3 complete: the expanded test suite must pass on Tim’s env before onboarding Jamie.
  3. Phase 7 requires Phases 1-6 complete: production baseline captured, Jamie can rebuild independently, Shibboleth validated, and the cutover runbook is written and reviewed.
Challenge

Beyond the 7 phases

  1. Is Phase 2 done? Name the one item of three that’s actually complete.
  2. Besides the 7 roadmap phases, what other body of work now also has to be resolved before Phase 7 can safely run – and why wasn’t it part of the original roadmap?
  1. No. Only 02-02 (FAKE DOI + test_cert + reindex target) is done. The Elastic IP (02-01) and full rebuild validation (02-03) are both still unstarted.
  2. The infrastructure security/reliability audit, run after this roadmap was written, found real gaps the 7 phases don’t cover: an open admin API on an internet-facing instance (Critical), unencrypted RDS storage, a stale/manual backup pipeline, and no automated file-bytes migration step (the gap covered above). None of these were anticipated when the roadmap’s phases were scoped – they surfaced from auditing what was actually built, not from the migration plan itself. They now gate Phase 7 alongside the original 6 phases.
Key Points
  • Dataverse 6.x requires Java 17, a Solr schema rebuild, and updated DOI/S3 configuration.
  • The 7-phase plan gates each phase with tests before proceeding to the next – but “current phase” doesn’t mean “current phase complete.” Phase 2 is one-third done as of this writing.
  • Phases 1-3 establish the foundation; phases 4-6 harden for production; phase 7 is cutover.
  • Rollback is straightforward until the DNS switch and EZID activation – that window is the target.
  • DNS TTL reduction and a persistent Elastic IP would together make the cutover switchover fast and predictable – but the EIP isn’t persistent yet, and the automated cutover sequence has no step that moves file bytes, a gap found after the roadmap was written.