The Dataverse Stack: Payara, Solr, and the Data Layer

Last updated on 2026-07-30 | Edit this page

Estimated time: 40 minutes

Overview

Questions

  • What is Payara and why does Dataverse use it?
  • How does Dataverse store and retrieve data across RDS, S3, and Solr?
  • What breaks when Solr is out of sync, and how do you fix it?

Objectives

  • Describe Payara’s role as a Jakarta EE application server.
  • Explain how Dataverse configuration is applied via JVM options and the API.
  • Describe how RDS, S3, and Solr each serve different data needs.
  • Know when and why a Solr reindex is required.

Payara: the application server

Payara is a Jakarta EE application server – a runtime environment for Java web applications. It is a community fork of GlassFish, maintained specifically for Jakarta EE compatibility. Dataverse chose it because Dataverse is a Jakarta EE application and Payara has continued to receive active maintenance after GlassFish development slowed.

Think of Payara the way you might think of a Python WSGI server or a Node.js process: it is the process that loads the application, handles incoming requests, manages database connections, and keeps the application running.

Dataverse is distributed as a WAR file – a Web Application Archive. Ansible deploys this WAR file into Payara during installation. Payara unpacks it and starts serving requests.

Payara ports

Payara listens on several ports by default:

Port Purpose
8080 HTTP (application)
8181 HTTPS (application)
4848 Admin console
9009 Debug

Apache sits in front of ports 80 and 443. Requests come in through Apache and are proxied to Payara on port 8080 or 8181. Port 4848 is the Payara admin console – it should not be publicly accessible and is locked down in the security group.

Configuring Dataverse through Payara JVM options

Most Dataverse configuration is not stored in a config file. Instead, it is set as JVM options in Payara’s domain configuration (domain.xml). These are key-value pairs that Dataverse reads at startup:

-Ddataverse.files.s3-bucket-name=ucla-dataverse-storage
-Ddataverse.files.storage-driver-id=s3
-Ddataverse.db.host=dev-dataverse-db.cb4k4a6gqn27.us-west-2.rds.amazonaws.com

Ansible sets these JVM options through the Payara admin API during the configure step. You can also inspect or change them manually through the Payara admin console on port 4848, but Ansible is the source of truth – any manual changes will be overwritten on the next playbook run.

First boot and the Dataverse API

Some configuration can only be applied after Dataverse is running. During first boot, Ansible calls the Dataverse API to:

  • Set the root dataverse name and contact email
  • Configure the DOI provider (FAKE for test environments, real EZID for production)
  • Set storage credentials
  • Apply any site-level settings

This is why make rebuild takes several minutes even after Payara is up – the playbook is waiting for Dataverse to finish its own initialization before it can call the API.

Common Payara problems

  • Payara not responding after deploy: Dataverse startup takes 2-5 minutes on first boot. Check the Payara log at /usr/local/payara6/glassfish/domains/domain1/logs/server.log (Payara 6, since the move to Dataverse 6.8 – older docs and issues may still say payara5). Look for Dataverse started or exception stack traces.
  • Out of memory: Payara JVM heap settings are configured in Ansible group_vars. Default is 2GB – increase if you see OutOfMemoryError in the Payara log.
  • WAR deploy failure: The WAR file checksum or version may not match what Payara expects. Check the Ansible payara.yml task output for errors during the deploy step.
Callout

Payara vs. GlassFish

Dataverse documentation and older issues sometimes reference GlassFish commands and paths. Payara is API-compatible – the asadmin command and most paths are the same. If you find a GlassFish-specific workaround in an older issue, it will almost always apply to Payara as well.

The data layer: RDS, S3, and Solr

Dataverse splits its data across three storage systems, each handling a different type:

RDS (PostgreSQL)

RDS stores all structured metadata:

  • Dataset records (titles, descriptions, authors, versions, publication dates)
  • File records (names, checksums, file type, which dataset they belong to)
  • User accounts and permissions
  • Dataverse collection hierarchy
  • Workflow state and notifications

RDS is the authoritative record for what datasets and files exist. When you restore a database backup, you are restoring this metadata.

S3

S3 stores the actual file content – the bytes of every data file users have uploaded. The connection between a file record in RDS and its content in S3 is a storage identifier stored in the database.

After a database restore, Dataverse uses the storage identifiers to serve files from S3. The files in S3 do not change when you restore the database – only the metadata does.

This also means that if a file is deleted from S3 but its record remains in RDS, Dataverse will show the file as available but fail when users try to download it.

Callout

The baseline scripts check both

The make baseline command captures both database counts (via the Dataverse Metrics API) and S3 object counts (via aws s3 ls). Comparing pre- and post-migration baselines verifies that both the database and the storage bucket survived the migration intact.

Solr

Solr is a full-text search engine. Dataverse uses it to power all search and browse functionality – when you type a query in Dataverse or browse a collection, the results come from Solr, not directly from the database.

Solr maintains its own index: a data structure optimized for fast search that is built from the content in the database. This index is separate from and derived from the database.

The index can go out of sync. If you restore the database to an earlier state, the Solr index may contain entries for datasets that no longer exist, or be missing entries for datasets that were added. After any database restore, you must reindex:

BASH 

make reindex ENV=tim

A Dataverse instance with a stale Solr index will appear to have no datasets – or the wrong datasets. This is one of the most common sources of confusion after a rebuild.

The reindex process reads all dataset metadata from the database and sends it to Solr. Depending on how many datasets you have, this can take from seconds to hours.

Callout

make reindex only works because the admin API is open – and that’s a problem

Look at the real target: it’s curl -X DELETE https://$HOST/api/admin/index over public HTTPS. That works only because Dataverse’s admin API is currently unauthenticated and reachable from the internet on this instance – which the infrastructure security audit flags as a Critical finding (F1): the same open admin API also allows dataset destroy on an instance that will eventually hold production research data. make baseline and make test have the identical dependency (F5).

These aren’t separable problems. Closing F1 (blocking the admin API at the proxy, the correct fix) breaks reindex, baseline, and test unless they’re rewritten first to go over SSH instead (ssh rocky@$IP 'curl -s localhost:8080/api/admin/...'). That SSH-based rewrite is planned work, not yet done. Until it lands, every reindex you run is quietly depending on an exposure that shouldn’t exist on a prod-data instance.

Challenge

Trace a file access

Given what you know about the data layer, answer these questions:

  1. A user uploads a file to Dataverse. What gets written to RDS? What gets written to S3?
  2. A user searches for “climate data.” Which storage system answers that query?
  3. You restore the database from a week-old backup. What is the state of S3? What is the state of Solr?
  4. After the restore in question 3, what must you do before the system is usable again?
  1. RDS gets a new file record (name, checksum, storage identifier, dataset reference). S3 gets the file bytes at the storage identifier path.
  2. Solr answers the search query. The database is not involved in search.
  3. S3 is unchanged – it still has all files ever uploaded, including those added in the past week. Solr still has the current index built from the current database before the restore.
  4. Run make reindex ENV=<env> to rebuild the Solr index from the restored database state.
Challenge

Why reindex works today (and why that’s not fine)

  1. What HTTP call does make reindex actually make, and what does it depend on being true about the server?
  2. If that dependency were removed tomorrow (admin API blocked at the proxy), what would break, and what’s the planned fix?
  1. curl -X DELETE https://$HOST/api/admin/index – a public-HTTPS call to Dataverse’s admin API. It depends on that API being unauthenticated and reachable from outside the instance, which it currently is (audit finding F1, Critical).
  2. make reindex, make baseline, and make test would all start failing (F5) – they all hit admin/metrics endpoints the same way. The planned fix is rewriting those calls to go over SSH to localhost:8080 on the instance instead of public HTTPS.
Key Points
  • Payara is a Jakarta EE application server; Dataverse runs as a WAR file inside it.
  • Most Dataverse configuration is set as Payara JVM options, managed by Ansible.
  • The Payara log at payara6/glassfish/domains/domain1/logs/server.log is the first place to look when things go wrong.
  • RDS holds metadata; S3 holds file content; Solr holds the search index.
  • Always run make reindex after a database restore – Solr does not update itself.
  • make reindex/baseline/test currently depend on the admin API being open over public HTTPS – a known Critical security exposure (F1/F5), not a stable design choice.