Queue health during large jobs

Shopware moves time-consuming tasks to a message queue instead of processing it during a request. The worker is the component that processes that queue in the background. Some jobs put thousands of messages into the queue at once. A full reindex with bin/console dal:refresh:index --use-queue, which also updates the products in the OpenSearch index when OpenSearch is enabled, is the most common one. Large imports and bulk updates behave the same way. The command returns as soon as the messages are queued, long before the work is done. From the outside this can look like a broken shop while the queue is simply still draining.

This page describes how the worker behaves, what to watch while such a job runs, and when to open a ticket.

How the worker processes the queue

Shopware uses three queues, called transports. Each one has its own worker. In Grafana, the deployment variable lists them under these names:

Transport	Stored in	Worker deployment	Used for
async	Valkey	<application>-store-worker-async	Indexing, mails, and everything else processed asynchronously
low_priority	Database	<application>-store-worker-low-priority-<hash>	Messages Shopware marks as low priority
failed	Database	<application>-store-worker-failed	Messages that failed after all retries. This worker retries them.

Each worker runs as one or more pods, and the number of pods follows the number of messages waiting in its queue:

• With an empty queue, the worker has zero pods. A pod starts within seconds of the first message.
• One pod is added for roughly every 300 waiting messages, up to the maximum your plan allows.
• When the queue is empty, the pods are removed after a short cooldown.

INFO

Indexing messages are batches. One product indexing message covers 50 products, so a catalog of a few thousand products is only a few dozen messages and drains within minutes on one pod. The scheduled task runner also queues a few messages every five minutes, so a worker pod appearing briefly every five minutes is expected.

Start the job

Run the reindex as a command in a dedicated container, so you can follow its output afterward. The first command prints the command ID that the second one needs:

sw-paas command create --script 'bin/console dal:refresh:index --use-queue'
sw-paas command logs --command-id <command-id>

Note the time you started it. The command only queues the work, so it reports SUCCEEDED long before indexing is done.

Where to look

Run sw-paas open grafana to get the Grafana URL and credentials, then open the Application Resources dashboard with a time range that covers the job start. The Message Queues row shows all three queues for the whole application. For the pod-level panels, set deployment to the async worker, since that is where indexing goes.

Panel	Healthy during a large job	Needs attention
Messages per Queue, Queue Depth	async jumps when the job starts, then falls steadily towards zero.	async plateaus or grows for several minutes while worker pods are running. failed is growing.
Available Replicas per Component	Worker replicas rise after the queue fills and drop to zero when it is empty.	Worker replicas stay at zero while async is above zero for more than a minute.
Pods & Autoscaling	Running pods are below the dashed plan maximum, or touch it briefly.	Running pods sit on the dashed line for the whole job and the queue drains slowly. The plan is the limit.
Component Issues	All components: running, or scaling while pods change.	error, not-found or unknown for the worker.
Memory Usage (of limits)	Below the limit. Memory grows during indexing and drops when the worker process restarts itself, which it does regularly.	Pods approaching 100% of the limit.
OOM-Killed Containers (24h), Restarts (24h)	Zero. The worker process restarting itself does not count here.	Above zero. A killed worker leaves its current message unacknowledged, and the queue redelivers it only after an hour. Each kill leaves the queue stuck above zero a little longer.

INFO

Metrics are collected once a minute. On a small catalog the reindex can finish between two samples, so the dashboard may show the worker pod but never the queue. If the queue is visible and shrinking, the job is progressing.

Check the dependencies

If the worker looks healthy but async is not moving, check what the worker depends on.

Valkey dashboard. Valkey is the Redis-compatible store the platform provides. The async queue lives in the worker Valkey instance, which also stores carts. Select it. On Memory Used, the dashed line is maxmemory. The instance uses the noeviction policy, so at the limit it rejects writes instead of evicting keys, and the eviction counter stays flat. Check memory usage together with Valkey OOM or write errors in the worker and storefront logs. Evictions on Evicted / Expired Keys mean another instance or policy is selected. Key expiration is normal.

Database dashboard. Indexing reads heavily from the database. ACU Utilization shows how much of your plan's database capacity is in use. Sustained values near the maximum slow every message and affect storefront requests too.

PHP-FPM dashboard. Select the storefront component. The storefront scales on Pool Utilization, so a value that stays high during the job means the reindex is competing with customer traffic.

Read the worker logs

sw-paas application logs --component worker --follow --since 15m

A healthy run is a steady sequence of Received message, handled successfully and OpenSearch Request Success lines. When the queue stalls, filter for errors:

sw-paas application logs --component worker --since 15m --query '{component="worker"} |= "ERROR"'

The Logs dashboard in Grafana offers the same filter with component set to worker and level to ERROR. Memory exhausted errors, database timeouts, or OpenSearch connection errors repeating on every retry explain a growing failed queue.

Check from inside the application

Open a terminal session in your application with sw-paas exec --new, see Executing Commands, and run:

bin/console messenger:stats
bin/console messenger:failed:show

The first shows pending messages per transport and matches the dashboard. The second lists failed messages with their error, which is the fastest way to the root cause of a growing failed queue.

Is the search index ready?

Two different jobs touch the OpenSearch index, and they are tracked differently.

DAL refresh. dal:refresh:index --use-queue writes products into the current OpenSearch index in place. It does not build a replacement index and does not switch the alias. On a shop that already has an index, search keeps working throughout. On a shop where OpenSearch was just enabled, the first product batch creates the index and makes it live immediately, so search returns partial results that grow as the queue drains. Results appearing is not a sign that the whole catalog is indexed. Only an empty queue is, see How to set up OpenSearch. In both cases es:status does not track this job and shows completed throughout. Its progress is the async queue depth and the worker logs described above. The job is done when the queue is empty and the worker logged no errors.

Full rebuild. bin/console es:index builds a new index next to the old one, queues the indexing work, and switches the alias to the new index at the end. The old index keeps serving until then, so search does not return empty results during the rebuild. es:status tracks this job:

+-----------------+-----------+
| Name            | Status    |
+-----------------+-----------+
| Cluster Status  | green     |
| Available Nodes | 6         |
| Indexing        | completed |
+-----------------+-----------+

Read the status with these two cases in mind:

• An index already exists. The command records the rebuild before it queues any messages, so es:status shows in progress with a progress bar from the start. It shows completed once all documents are written, but prints a warning that the alias will swap at the end of the indexing process until the switch has happened.
• No index exists yet. Shopware records nothing, so es:status shows completed throughout, with no warning, while the worker is still building the index. The new index is live from the start, so search returns partial results that grow as the build progresses. Use queue depth and worker logs to tell when it is complete.

The alias switch is done by a Shopware scheduled task, and the platform runs scheduled tasks every five minutes, so the warning should disappear within about ten minutes after the queue drains. If it persists, check the Scheduled Tasks row on the Application Resources dashboard and the runner's output with sw-paas application logs --component scheduled-task. Declare the index ready when the queue is empty, the worker logged no errors, no alias warning is printed, and storefront search returns results.

Empty search results during either job on a shop that already has an index are a search failure, not a draining queue.

When to open a ticket

Open a support ticket when one of the following is true:

• The worker is pinned at the plan maximum for the whole job and the queue drains too slowly for your needs. This is a sizing conversation.
• Worker pods are OOM-killed or restarted repeatedly.
• Valkey memory on the worker instance has reached maxmemory, or the logs show Valkey OOM or write errors.
• The worker shows error, not-found or unknown under Component Issues.

Include the start time, the command you ran, a screenshot of Queue Depth and Pods & Autoscaling covering the job, and the output of messenger:failed:show if the failed queue is above zero.

WARNING

A queue that drains steadily without errors or resource pressure usually does not require a ticket. If any of the criteria above apply, open a ticket even if the queue is still draining.

Reduce the impact next time

Long-running messages such as indexing delay time-critical messages such as order confirmation emails while they share the async queue. Route them to a separate transport so short messages keep flowing during a reindex. See Split message queues.