cbq

A protocol-based queueing system for ColdBox

Requirements

Adobe 2018+ or Lucee 5+ ColdBox 6+

Batch UUID library

Batch IDs use the Java UUID Generator bundled in cbq/lib. Add that directory to your application's Application.cfc Java load paths before creating batches (adjust the path to your module installation):

this.javaSettings = {
    loadPaths : [ expandPath( "/modules/cbq/lib" ) ],
    reloadOnChange : false
};

If you already configure this.javaSettings, append this directory to its existing loadPaths. Restart the application after changing Java load paths. A ClassNotFoundException for com.fasterxml.uuid.Generators means this bundled library is missing from the application's classpath; changing the Java version alone does not add it. Use a Java version supported by your CFML engine.

Definitions

Queue Connection

A queue connection defines how to connect to a backend service like Redis, RabbitMQ, or even a database. Any given queue connection can have multiple "queues" which are named stacks of queued jobs or messages to be delivered.

Queue

A named stack of jobs or messages to be delivered. A queue connection must have at least one queue which is usually "default". A queue connection can have as many queues as desired. This is mostly used later when defining queue workers to scale different queues at different priorities.

Queue Provider

A queue provider is how a queue connection connects to a backend service like Redis, RabbitMQ, or a database. It implements the necessary interface to send the jobs and to work the queues. A queue provider can be used multiple times in a single application to define multiple queue connections with different configuration options.

A queue provider must extend the AbstractQueueProvider and implement the required abstract methods:

• public any function push( required string queue, required AbstractJob job, numeric delay, numeric attempt )
• public function function startWorker( required WorkerPool pool )

Additionally, the Queue Provider can use the following hooks to do additional processing or cleanup:

• private void function beforeJobRun( required AbstractJob job )
• private void function afterJobFailed( required any id, AbstractJob job, WorkerPool pool )

Job

A job is a CFC that follows the IDispatchableJob interface (easily done by extending the AbstractJob component). It defines how to serialize the job using a memento pattern and deserialize the job from the queue. It also holds the data needed to execute the job and a handle method that is called when working the job from the queue. Job components exist in the context of your application so you have access to all the models, services, and helpers you have already written. (Raw string messages can also be dispatched via cbq. The message will need to be handled directly by your queue worker.)

component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        sleep( 1000 ); // do some processing work
        log.info( "sending email - #this.getBody()#" );
    }

}

Providers

cbq provides the following providers out of the box:

• SyncProvider
• ColdBoxAsyncProvider
• DatabaseProvider

Future planned providers include:

• Mock (for testing)
• RabbitMQ
• Redis
• Couchbase

(See the ROADMAP for other planned protocols.)

Each of the providers takes different configuration when creating a connection. Refer to the specific provider documentation for details.

Database compatibility

For MySQL, DBProvider@cbq requires MySQL 8.0 or later. Its reservation query uses FOR UPDATE SKIP LOCKED, which MySQL 5.7 does not support. See MySQL locking reads. cbq does not ship a MySQL 5.7 fallback grammar. Upgrade MySQL before enabling database workers; removing the lock clause changes concurrent worker behavior.

Workers with different job mappings

Use separate queues and explicitly configure each worker pool's queues when workers have different job components. An unknown mapping is logged with the job ID and worker pool. Database workers release that reservation without consuming an attempt or marking the job failed, then continue the polling pass. A worker with the required mapping can claim it. Queue routing prevents incompatible workers from repeatedly claiming the same job. Configure your LogBox appenders to send worker logs to a shared destination when central logging is needed.

Batch result accounting

Run migration 2000_01_01_000011_track_processed_batch_jobs.cfc before deploying this version, including on custom batch tables. Batch results are recorded once per job ID under a row lock; repeated success or failure callbacks do not consume another pending job or repeat lifecycle callbacks. The new nullable processedJobIds column records results going forward. Existing failed IDs remain protected, but successful job IDs from before the migration cannot be reconstructed. Drain existing batches before upgrading if they might receive duplicate callbacks. This accounting does not make a job's external side effects execute exactly once.

Database polling

DBProvider@cbq polls once every five seconds by default. Each poll fetches only as many jobs as the worker pool currently has available slots. For short jobs, polling can therefore determine backlog drain time even when transport is fast.

Set pollIntervalMilliseconds in the connection properties to configure the delay between completed polling passes. It must be a positive integer; the default is 5000. This configures each native database watcher on that connection and takes effect when workers are registered. Worker quantity, job timeout, retry backoff and ownership checks are unchanged.

newConnection( "mail" )
    .setProvider( "DBProvider@cbq" )
    .setProperties( { "pollIntervalMilliseconds": 250 } );

A shorter interval also increases polling while the queue is idle. Measure database load, worker capacity, transport limits and queue age before choosing it. This setting does not increase the worker pool's concurrency or guarantee a delivery time.

Installation and Setup

To install cbq, install it from ForgeBox:

box install cbq

You can configure cbq in your moduleSettings inside config/ColdBox.cfc as follows:

moduleSettings = {
    "cbq" : {
        // The path the custom config file to register connections and worker pools
        "configPath" : "config.cbq",
        // Flag if workers should be registered.  If your application only pushes to the queues, you can set this to false.
        "registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
        // The interval to poll for changes to the worker pool scaling.  Defaults to 0 which turns off the scheduled scaling feature.
        "scaleInterval" : 0
    }
};

Most of the configuration for cbq happens inside the cbq config file, located at config/cbq.cfc by convention.

component {

    function configure() {
        newConnection( "default" )
            .setProvider( "SyncProvider@cbq" );

		newWorkerPool( "default", "default" );
    }

}

In configure you define one or more Connections. You must have at least one Connection called default.

New Connections are created using the newConnection function. It is a builder pattern object. Only a provider must be set. The other setters are optional.

newConnection( connectionName )
    .setProvider( providerMapping )
    .onQueue( name = "default" )
    .markAsDefault( /* true / false */ );    .

You can also define worker pools inside configure to work on queues for a given Connection that you defined previously.

New Worker Pools are created using the newWorkerPool function. It is a builder pattern object. All of the setters are optional.

newWorkerPool( name, connectionName )
    .quantity( numberOfWorkers )
    .onQueue( name = "default" )
    .backoff( backoffTimeInSeconds )
    .timeout( timeoutTimeInSeconds )
    .maxAttempts( maxNumberOfAttempts );

The config file follows ColdBox's environment overrides by calling a method matching the environment name if it is found. Inside that method you can use the withConnection and withWorkerPool methods to change the properties of connections defined in configure and worker pools defined in work:

component {

	function configure() {
		newConnection( "default" )
			.setProvider( "DBProvider@cbq" );

		newWorkerPool( "default", "default" )
			.setTimeout( 5 )
			.setMaxAttempts( 5 )
			.setQuantity( 3 );
	}

	function development() {
		withConnection( "default" )
			.setProvider( "SyncProvider@cbq" );

		withWorkerPool( "default" )
			.setMaxAttempts( 1 )
			.setQuantity( 1 );
	}

}

Usage

Job Components

Jobs are CFCs that extend cbq.models.Jobs.AbstractJob. You need to define a handle method that is ran when the Job is processed.

// GreetingJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        // this is ran when the job is processed
        log.debug( "Hello world!" );
    }

}

To dispatch a job to a queue to be worked, call the dispatch method on a Job instance. Dispatching a job serializes it and sends it to the configured connection. It will later be picked up by a worker and processed.

getInstance( "GreetingJob" ).dispatch();

A Job is sent to the queue with any of its properties serialized. You can set the properties of a job by calling the setProperties method and passing a struct of properties.

getInstance( "GreetingJob" )
    .setProperties( { "greeting": "Hello" } )
    .dispatch();

Additional Job-level properties can be set before dispatching to override Worker Pool defaults on a per-Job basis.

getInstance( "GreetingJob" )
    .setProperties( { "greeting": "Hello" } )
    .setDelay( 10 ) // delay processing this job for 10 seconds
    .dispatch();

CBQ Model

A cbq model exists to make certain job- and dispatch-related actions easier to perform. You can access it by injecting cbq@cbq or simply @cbq. Here are the methods available to you:

job

Creates a Job instance.

Name	Type	Required	Default	Description
job	string or Job instance	true		A job instance or mapping string to a Job instance. Additionally, any string may be provided here, even if it doesn't exist as a CFC. If so, cbq will create a NonExecutableJob with the given mapping. This can only be used if the instance dispatching the jobs will never work the jobs.
properties	struct	false	{}	A struct of properties for the new Job.
chain	Job[]	false	[]	An array of Job instances to chain after this one.
queue	string	false	null	The queue to run this Job on. Overrides the Job queue and the default queue, if provided.
backoff	numeric	false	null	The backoff amount in seconds between Job attempts. Overrides the Job backoff and the default backoff, if provided.
timeout	numeric	false	null	The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided.
maxAttempts	numeric	false	null	The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided.

dispatch

Creates a Job instance and immediately dispatches it.

Name	Type	Required	Default	Description
job	string or Job instance OR array of Job instances	true		A job instance or mapping string to a Job instance. Additionally, any string may be provided here, even if it doesn't exist as a CFC. If so, cbq will create a NonExecutableJob with the given mapping. This can only be used if the instance dispatching the jobs will never work the jobs. If an array of Job instances are passed, this forwards it on to chain and dispatches the chain.
properties	struct	false	{}	A struct of properties for the new job.
chain	Job[]	false	[]	An array of Job instances to chain after this one.
queue	string	false	null	The queue to run this Job on. Overrides the Job queue and the default queue, if provided.
backoff	numeric	false	null	The backoff in seconds amount between Job attempts. Overrides the Job backoff and the default backoff, if provided.
timeout	numeric	false	null	The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided.
maxAttempts	numeric	false	null	The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided.

chain

Creates a Job Chain and returns the first Job in the chain. To dispatch the chain, you must call dispatch on the returned Job.

Name	Type	Required	Default	Description
chain	Job[]	false	[]	An array of Job instances to chain after this one.

Opt-in database enqueue batches

Dispatcher@cbq.bulkDispatch accepts batchSize from 1 to 100. The default is 1 and preserves the existing order: announce onCBQJobAdded, reset the job's attempt, then push that job before announcing the next.

transaction {
    getInstance( "Dispatcher@cbq" ).bulkDispatch(
        jobs = jobs,
        connectionName = "db",
        queueName = "mail",
        batchSize = 100
    );
}

With a larger batch size, each job is still announced separately and becomes its own queue row, with its own payload and lifecycle. Events for one chunk run before that chunk is persisted. Choose the default if a job-added interceptor needs to read earlier jobs from the same chunk. An explicit queue overrides job queues as before; otherwise each job keeps its effective queue. As with existing bulkDispatch, this does not use each job's backoff as an initial delay.

The DB provider writes at most 100 rows (500 bindings) per insert. Other providers continue through their ordinary push implementation unless they implement pushMany(entries). Entries contain the ordinary push arguments: queueName, job, and optionally delay and attempts. DBProvider.pushMany also bounds direct calls to 100 rows per insert. It uses the configured table, query options and datasource.

The caller owns transaction boundaries. Wrap the dispatch with related database changes when all chunks must commit or roll back together. Without a surrounding transaction, a later event, serialization or insert failure can leave earlier chunks persisted.