Skip to content

Queueing: Comparison of Lower/Upper Queues

Analogy: SQL/APIv4 and Lower/Upper Queue Systems

  • The Lower Queue System is like CiviCRM's SQL layer. It is a building-block. It's always there, but it's not always best to use.
  • The Upper Queue System is like CiviCRM's APIv4. It's recommended for most use-cases. And it is literally built on APIv4.

CiviCRM's queueing system evolved significantly from its original design (v4.2; Lower Queue) and its later design (v5.47; Upper Queue). The later design builds on top of the original design, and many concepts overlap. But the developer-experience is different.

Lower Queue Upper Queue
Originated circa v4.2 Originated circa v5.47
Lower-level concepts Higher-level concepts
Write a polling-loop Use a standard polling-loop
Runners Registrations, Agents, and Hooks
Useful for system-maintenance (upgrade, install, etc) Useful for higher-level logic (imports, exports, messaging, etc)

This page will break-down the differences in more detail. It aims to serve two audiences:

  • Developers who understand the old way (Lower Queue) and want to switch to the new way (Upper Queue).
  • Developers who have read the new docs (Upper Queue) and need to maintain code in the old way (Lower Queue).

Gist

  • In Lower Queue, you wrote a runner with PHP code. It provided a polling-loop to call $queue and handle each item.
  • In Upper Queue, you define a registration record in civicrm_queue. A standard agent reads that record and respects the configuration-options. It fires hooks to handle each item.

Strengths

  • Lower Queue has fewer dependencies. It doesn't even rely on the core-container. It can be used at tricky times (e.g. installation and upgrade).
  • Upper Queue is more standardized and more configurable. If multiple extensions are providing multiple queues to multiple organizations, then the standardized+configurable options will be easier to adapt.

Similarities

  • Driver classes
    • Both use CRM_Queue_Queue_* to add and consume queue-items.
    • Both provide a factory-function to get the CRM_Queue_Queue_* objects.
  • Items
    • Both use the method createItem().
    • Both support item-data using CRM_Queue_Task, array, etc.
    • Both support item-options using weight, release_time, etc.

Differences

  • Registration records (MySQL table civicrm_queue with lease_time, batch_limit, etc)
    • The Lower Queue does not require registration records. They did not exist originally. Today, they are optional.
    • The Upper Queue requires registration records.
  • Driver factory
    • The Lower Queue originally encouraged you to instantiate drivers by calling CRM_Queue_Service::singleton()->create().
    • The Upper Queue encourages you to instantiate drivers by calling Civi::queue().
    • These two methods are the same thing, except for the option is_persistent:
      • Civi::queue() defaults to is_persistent=>TRUE. (Do read registration records.)
      • CRM_Queue_Service::singleton()->create() defaults to is_persistent=>FALSE. (Do not read registration records.)
  • Runners vs Agents/Hooks
    • In the Lower Queue, a "runner" is responsible for both (a) polling the queue to claim an item and (b) evaluating the item. It is built directly on CRM_Queue_Queue_* drivers. There are two built-in task-runners. To use different payloads or change behaviors, you must implement a custom runner.
    • In the Upper Queue, the responsibilities are split. The "agent" is responsible for polling the queue to claim an item. The hooks are responsible for evaluating the item. You can change behaviors with settings and hooks.