Version 2.5 is a legacy release, and these documents are no longer being maintained.

Asynchronous jobs and worker management

AtoM relies on Gearman in order to execute certain long-running tasks asynchronously to guarantee that web requests are handled promptly and work loads can be distributed across multiple machines.

For example, the ingest of Archivematica DIPs is processed asynchronously. See Upload DIP for more details.

You’ll need a good understanding of all the components involved to find out what the best configuration and arrangement of services will be for you. You could configure it so every service runs in the same machine, or scale up to many machines across the network. You can always start small in a single-node environment and add more hardware later.

Gearman Job Server

The first thing that you need to do is to install the job server (Gearman Job Server):

sudo apt-get install gearman-job-server

It’s up to you to install the job server in the same machine where you are running AtoM or in another machine in the same network. If you are doing the latter, make sure that you update /etc/default/gearman-job-server accordingly:

PARAMS="--listen=* --port=4730"

Note

By default, Gearman’s job queues are stored in memory. However you are free to deploy the queue storage in a different machine or replace it with a durable solution like MySQL or SQLite3. For more configuration options available, visit: http://gearman.org/manual/job_server/.

Gearman Job Worker

In AtoM, a worker is just a CLI task that you can run in a terminal or supervise with specific tools like upstart, supervisord or circus. The worker will wait for jobs that are assigned by the job server.

The simplest way to run a worker is from your terminal:

php symfony jobs:worker

A better way to run a worker is to use a process supervisor like systemd (included in Ubuntu 16.04 and 18.04). This is documented below.

systemd (Ubuntu 16.04, 18.04)

Create the following service (/usr/lib/systemd/system/atom-worker.service):

[Unit]
Description=AtoM worker
After=network.target
# High interval and low restart limit to increase the possibility
# of hitting the rate limits in long running recurrent jobs.
StartLimitIntervalSec=24h
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/usr/share/nginx/atom
ExecStart=/usr/bin/php7.2 -d memory_limit=-1 -d error_reporting="E_ALL" symfony jobs:worker
KillSignal=SIGTERM
Restart=on-failure
RestartSec=30

Important

If you are not using PHP 7.2, be sure to update the ExecStart filepath in the [Service] section of the sample configuration block above! Currently it assumes PHP 7.2 is being used, and will not work for installations using PHP 7.0 without modification.

Now reload systemd:

sudo systemctl daemon-reload

You can control the service execution status with the following commands:

sudo systemctl enable atom-worker   # Enables the worker (on boot)
sudo systemctl start atom-worker    # Starts the worker
sudo systemctl stop atom-worker     # Stops the worker
sudo systemctl restart atom-worker  # Restarts the workers
sudo systemctl status atom-worker   # Obtains current status

You can have access to the journal of our new atom-worker unit as follows:

sudo journalctl -f -u atom-worker

This is going to be useful in case you need to troubleshoot the worker.

Tip

If the worker hits the start rate limit (3 starts in 24h) to be able to start it again after fixing the issue, the failed status has to be cleared:

sudo systemctl reset-failed atom-worker
sudo systemctl start atom-worker

Other considerations

An AtoM worker needs to know where the job server is running, which is defined in an application setting under config/gearman.yml and defaults to 127.0.0.1:4730.

Note that the job server will perfectly handle multiple workers running simultaneously and the work load will be distributed across all available workers. If there are no workers available because they are busy completing other tasks, the job server will store the job in the queues and deliver them once a worker becomes available.

If you’re planing to connect multiple AtoM instances to the same Gearman server, make sure to set a different string value for the workers_key setting located in config/app.yml. This will avoid collisions between those instances and the workers will only take the jobs that belong to their related AtoM install.

Whenever you change any of these settings, make sure that the Symfony cache is cleared and the workers are restarted.