Automated queue creation within a swiftlet

[Chad12]

I am trying to automate the one-time creation of a new persistent queue from within my Extension Swiftlet (SwiftMQ 7.5.1). Following the example in docs/router/swiftletapi/kernel/queue/index.html I am using the code:

QueueManager manager = (QueueManager)SwiftletManager.getInstance().getSwiftlet("sys$queuemanager");
if (!manager.isQueueDefined(queueName)) {
    try {
        manager.createQueue(queueName, (ActiveLogin)null);
	manager.getQueueForInternalUse(queueName).setPersistenceMode(AbstractQueue.PERSISTENT);
	log.info("Could not find queue ["+queueName+"], now created.");
    } catch (Exception e) {
        log.warn("Unexpected exception, ignoring...", e);
    }
}

And then re-saving configuration with SwiftletManager.getInstance().saveConfiguration(); as previously suggested to me. However when I create the queue in this way, the new queue does not appear to be saved to the configuration file. Strangely, a corresponding JNDI alias is saved, even though the queue itself isn't saved. Am I missing something here or doing something wrong?

[IIT Software]

Queues which you create via the Swiftlet API are system queues. Regular queues are create via the management tree only (CLI "new").

System queues in turn are created and managed by their corresponding queue controller. If enable the queue manager trace space, you should see which queue controller is used to create the queue. It is picked by a matching queue name with queue controller predicate top-down the queue controllers. The last one with predicate "%" matched for all.

You might use this last one or, if you really need different settings, create your own queue controller with matches with your queue name:

http://www.swiftmq.com/products/router/swiftlets/sys_queuemanager/controller/index.html
 

[Chad12]

The documentation and API really does not make any of that clear. It's confusing that queueManager.isDefined(queueName) can return true for a regular queue, and you can delete a regular queue when .createQueue is not able to actually create a regular queue. It seems the name is inconsistent and/or misleading?

Since I require persistent queues, I am assuming I need regular queues rather than system queues, but can't actually find documentation of the differences. Is there a functional difference between those queues defined in the config file and those "system" queues that may be created at startup?

i.e is there going to be a functional difference if I switched my extension swiftlet to always "create" all of my necessary queues at startup, vs. having them as regular queues defined in the config file? I am presuming it will affect the ability of those queues to have messages persisted across restarts, but I'm not really sure now.

If the answer is such that I can't have persistent "system" queues managed by my Swiftlet and thus have to use the CLI to manage regular queues, is there then a way I can create a regular one within an extension swiftlet during startup? When i try to do so using the CLI it seems unable to reliably get a CLI connection, usually timing out after 60 seconds. I presume this is somehow related to doing so while the router is still starting up; i.e. the management tree isn't fully enabled?

[IIT Software]

Well, I admin it's a bit 'underdocumented'. The reason is that's still an internal API. But no excuses.

Actually it's intended that an Extension Swiftlet creates and removes resources by putting corresponding CLI commands in the resp. sections of the "cli" element of its config.xml (excerpt from docs/router/swiftletapi/deploy/index.html):



In your case you create your queues in "before-install" and remove it in "after-remove". Only in case you can't do it this way, e.g. if you determine the queue name programmatically, you will do it with API-calls directly in your Swiftlet's startup method.

Concerning regular and system queues. Regular queues are user-defined queues, defined via routerconfig.xml, CLI, Explorer. Queues created from Swiftlets are system queues. Swiftlets usually just create it and give users full control over it via the queue controllers. For example, the input queues for the JNDI Swiftlet & Scheduler Swiftlet are created this way.

[Chad12 ()]

Yes, using before-cli etc is only useful for static configuration. We need our system to be centrally administered, so static XML configuration across various SwiftMQ boxes isn't really suitable; hence my attempts to load and reconfigure our extension swiftlet dynamically. I'm still not really clear on the functional differences between regular and system queues; i.e what you can and can't use them for, but I'm going to assume now that I need regular queues, so they can potentially interact with swiftlets other than our main extension swiftlet. You mention

IIT Software wrote:

if you determine the queue name programmatically, you will do it with API-calls directly in your Swiftlet's startup method

- as I said above, this is what I am now trying to do, however I am getting some sporadic errors like:

2009-07-05 21:18:44,130 [main] ERROR - Error creating new queue
com.swiftmq.tools.requestreply.TimeoutException: Request timeout occured (60000) ms
	at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.request(Unknown Source)
	at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.create(Unknown Source)
	at com.swiftmq.admin.mgmt.EndpointFactory.createEndpoint(Unknown Source)
	at com.swiftmq.admin.cli.CLI.createEndpoint(Unknown Source)
	at com.swiftmq.admin.cli.CLI.init(Unknown Source)
	at com.swiftmq.admin.cli.CLI.(Unknown Source)
	at MySwiftlet.verifyQueueExists(MySwiftlet.java:690)
	...
	at MySwiftlet.startup(MySwiftlet.java:97)
	at com.swiftmq.swiftlet.SwiftletManager.startUpSwiftlet(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.loadExtensionSwiftlet(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletDeployer.start(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletDeployer$4.swiftletStarted(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.fireSwiftletManagerEvent(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.startUpSwiftlet(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.startKernelSwiftlet(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.startKernelSwiftlets(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.initSwiftlets(Unknown Source)
	at com.swiftmq.swiftlet.SwiftletManager.startRouter(Unknown Source)
	at com.swiftmq.Router.main(Unknown Source)

I cannot work out how to reliably repeat these errors, but it does make me a little worried. Are there potentially any timing issues with creating CLI connections during the startup of a Swiftlet? In the timeout above, I notice that my cli.bat connections to the same router only say "Router 'myrouter' is available for administration" after the above call has timed out. How do you suggest I debug this further? Secondly, what's the best way to internally (Swiftlet API) obtain the correct provider URL to obtain a queue connection to use with the CLI? I'm currently using the below, but given I'm inside a Swiftlet, surely there's a better way to obtain the listener port at least? Or should I just use an intravm connection factory instead? I'm not sure from the documentation whether it's intended for this purpose.

Hashtable env = new Hashtable();
			env.put(Context.INITIAL_CONTEXT_FACTORY,"com.swiftmq.jndi.InitialContextFactoryImpl");
env.put(Context.PROVIDER_URL,"smqp://localhost:4207/timeout=10000");
InitialContext ctx = new InitialContext(env);
QueueConnectionFactory connectionFactory = (QueueConnectionFactory)ctx.lookup("QueueConnectionFactory");
connection = connectionFactory.createQueueConnection();
ctx.close();
CLI cli = new CLI(connection);

[IIT Software]

There is no functional difference between a system queue and a regular queue. Regular queues are created via the management tree. The Queue Manager Swiftlet implementation (a) provides the management functionality for this and then (b) uses the same methods you can use from other Swiftlets to create the actual queue. (a) is not accessible from other Swiftlets because it's an implementation detail of the Queue Manager Swiftlet implementation. (b) is part of the Queue Manager Swiftlet interface and thus accessible.

System queues are internal queues. Usually their names are prefixed with "rt$", "tmp$" etc. to make them inaccessible from JMS clients. There are a few exceptions like the swiftmqscheduler queue which must be accessible but these queues are still system queues. If you want to change the default attributes for system queues, you must do that via the resp. queue controller.

You can of course use CLI within a Swiftlet. Just use a intravm connection. I don't know why you got a request timeout but a general rule is: don't block threads. So putiing it into a AsyncTask will unblock it.

The intention of Swiftlets is to add router functionality and not to develop application. General rule is, if you can do it with JMS, do it with JMS. If you can't do it with JMS, create a Swiftlet. If you do it with JMS, you can hot deploy it like a Swiftlet, can use intravm connections, use CLI (as you do above) to create your resources etc. There is no difference except that you use a standard interface with JMS and without a vendor lockin, and a proprietary interface with Swiftlet API and a vendor lockin.

[Chad12]

Thanks for the regular vs. system queue explanation.

Regarding swiftlet vs. JAC; since I am enhancing an existing application heavily used in a production environment since SwiftMQ 4, for which I had no involvement in design of; and no mandate to re-architect as a separate application outside SwiftMQ I'm not sure it's worth discussing that route!

Anyway; I was deliberately blocking the startup of my swiftlet to ensure queues are created before startup, but using a separate thread to make the CLI connection does appear to have stopped me from being able to replicate the timeout issue.

I also noticed that my other external CLI.bat connections to the router got the same timeouts occasionally too while trying to reconnect; so I'm guessing there may be some issue with the CLI implementation if it tries to connect at a particular point in the router's startup?

com.swiftmq.tools.requestreply.TimeoutException: Request timeout occured (60000) ms
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.request(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory$EndpointCreator.create(Unknown Source)
        at com.swiftmq.admin.mgmt.EndpointFactory.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.CLI.createEndpoint(Unknown Source)
        at com.swiftmq.admin.cli.CLI.init(Unknown Source)
        at com.swiftmq.admin.cli.CLI.reconnected(Unknown Source)
        at com.swiftmq.jms.v750.ConnectionImpl$1.run(Unknown Source)

If it helps with any investigation you might want to do, the thread dump is here; you can see the main startup thread is waiting in QueueReceiverImpl, but not going anywhere.

[IIT Software]

CLI works fine, no doubt. You block the SwiftletManager thread starting your Swiftlet. The SwiftletManager stays in startKernelSwiftlet. I don't know how you have configured the router starting your Swiftlet but it must be via the Deploy Swiftlet (hot deploy). Then you would only block a thread of the Timer Swiftlet during deployment which is ok.

[Chad12]

Sorry, I'm not really sure what you're saying or implying. While I understand that the SwiftletManager blocks during startup, the thread dump doesn't indicate anything else is waiting for that lock - so it doesn't seem to explain why the CLI (sometimes) times out. The code and thread dump indicates the CLI interface is waiting for a response that it never receives; and the sporadic nature of the problem indicates there is a timing issue; i.e. sometimes it's making a request before some aspect of the router is ready to receive it.

The problem definitely goes away if I use AsyncTasks; but this is suboptimal as I want to fail startup/deployment if queues cannot be created for any reason and handling this asynchronously does not really give me the behaviour I want. If I must do say due to an indeterminate router startup/deployment performance, then I will do so - but I wanted to understand why this was happening.

We haven't configured our swiftlet in any "special" way; its' just defined as another swiftlet in the router config, per the documentation at docs/router/swiftletapi/deploy/index.html. It doesn't indicate any choice other than hot deployment as a possibility.

[IIT Software]

The thread dump shows that it is starting a Kernel Swiftlet, informs the deployer and starts it. Deployer starts Extension Swiftlet which startup method blocks because it creates a JMS connection. The reason why it blocks seems to be that one or more required Kernel Swiftlets like Network, JMS are not yet started because the startup sequence of Kernel Swiftlets hangs due to the blocking of the Extension Swiftlet's startup method. That's what I see in detail.

To solve that you need to register a SwiftletManagerListener at the SwiftletManager in your startup method and wait until every thing is up and then create your CLI connection.

I really do not understand why you don't use the proper ways by either (a) putting your CLI commands in your config.xml or, if you use generated queue names,  to (b) create system queues by passing the control to a queue controller.

(a) is actually the way to go. That queue names may change in future isn't a strong argument. Just change it in the config.xml and redeploy. You'd need to change it anyway.

[Chad12]

OK; regarding startup - I thought that might be the case.

Queue names changing isn't really the issue. This is a public forum, so I can only say so much, but suffice to say that our business logic demands that we create and remove quite large numbers of queues dynamically based on changes made to centralised configuration for our Extension Swiftlet; configuration that must sit outside of SwiftMQ as it affects several elements of our architecture. Additionally, business rules demand that we cannot restart SwiftMQ (and our extension swiftlet in particular) when changes are made to this centralised configuration - it must be propagated and hot-deployed across multiple SwiftMQ/Swiftlet instances and be able to support creation of any necessary resources such as new queues. A set of commands in config.xml is completely unsuitable for this purpose - this project is about automation and removing the need for our operations team to manually change SwiftMQ XML configuration as part of their day-to-day activities; since it's time consuming, risky, and non-scalable.

I don't want to go down the system queues route because it's poorly documented; I don't really understand it; and the current production system uses regular queues (albeit statically managed). Additionally, if there is a startup problem with the Extension Swiftlet I can also not afford for queues to not be created/visible/available to other external QueueProducers which rely on these queues, which seems to be the case with system queues.

[IIT Software]

Chad12 wrote:

Additionally, if there is a startup problem with the Extension Swiftlet I can also not afford for queues to not be created/visible/available to other external QueueProducers which rely on these queues, which seems to be the case with system queues.

In that case shouldn't you use statically defined regular queues (in the routerconfig.xml)?

It is not that difficult with system queues. Regular queues, created from users, use the attributes of the new queue entity. There is no such entity for internal (system) queues. In earlier releases we had just default attributes like cache-size etc which we did use for those queues. If the value was changed, the value of all system queues were changed. This wasn't suitable and we wanted more fine control. That's why we now have queue controllers. When an internal (system) queue is created, the Queue Manager goes top-down the queue controller list and picks that one where the queue name matches with the SQL-Like predicate of the queue controller.

That's it!

It now uses the attributes of this queue controller to create the queue and attaches the queue to the controller. If now someone changes the cache-size attribute value of this queue controller, the cache-size is changed for all queues created from and attached to this controller.

So what you can do is to create your own queue controller(s) and put it in the list with a SQL-Like predicate that matches your queue name. For example, use a prefix like "chad-" for your queue names and define a SQL-Like predicate "chad-%" for the queue controller. Define as many queue controllers as you want but keep in mind that they are compared top down and the first match is chosen (at least store it before the last "%" which matches for all).

This is well documented.

[Chad12]

IIT Software wrote:

Chad12 wrote:

Additionally, if there is a startup problem with the Extension Swiftlet I can also not afford for queues to not be created/visible/available to other external QueueProducers which rely on these queues, which seems to be the case with system queues.

In that case shouldn't you use statically defined regular queues (in the routerconfig.xml)?

Well, exactly - so that's why I said I didn't think system queues were appropriate; and that I need to create new queues that I can save to the configuration inside my Swiftlet's code; and you appear to be agreeing with that now.

Regarding documentation; respectfully; I disagree. The system queue controllers may be documented, but nowhere I could find talks about the pros and cons or when to use a regular queue vs. a system queue with respect to an Extension Swiftlet. I couldn't find any information regarding their recommended use with Extension Swiftlets in the Swiftlet API documentation with our distribution; nor the API JavaDoc.

[IIT Software]

That's a misunderstanding. I don't agree that you create regular queues from within your Swiftlet. Rather I meant that you have your queues statically defined in the routerconfig.xml.

Concerning docs. I agree there could be more. However, all queues created from Swiftlets are system queues and thus will use queue controllers. Before you start struggling with SwiftletManagerListener etc I highly recommend that you at least try to define your own queue controllers.

[Chad12]

This is going round in circles; which means either I obviously have no idea what you're talking about or you don't understand my requirements, or both. In any case, I'm giving up - I'm just getting more confused and frustrated and allocated time I have to complete this is disappearing fast.

I still don't understand how system queues and queue controllers could possibly meet the non-functional requirements I have defined above, so I cannot go down a path that uses them. Building a solution on components you don't understand within short time frames is a sure recipe for disaster. I understand the behaviour of regular queues, and that is what is currently in use, so even if I have to use messy internal Asynchronous calls via the CLI to manage them, I'd rather go down that route. It works. I understand it. For now, that's going to have to be good enough.

[IIT Software]

I understand your requirements.

You could also use something like this in your startup method:

    MgmtSwiftlet  mgmtSwiftlet = (MgmtSwiftlet) SwiftletManager.getInstance().getSwiftlet("sys$mgmt");
    CLIExecutor cliexec = mgmtSwiftlet.createCLIExecutor();
    QueueManager manager = (QueueManager)SwiftletManager.getInstance().getSwiftlet("sys$queuemanager");
    if (!manager.isQueueDefined(queueName))
    {
       cliexec.execute("cc /sys$queuemanager/queues");
       cliexec.execute("new "+queueName+" persistence-mode persistent");
       SwiftletManager.getInstance().saveConfiguration();
    }

I did not test it but it should actually work.

[Chad12]

Oh wow; that's a gem; being able to get a direct executor from the MgmtSwiftlet. That appears to have completely solved my problems. I can't replicate any timeout/blocking issues using that approach so don't appear to need to use an AsyncTask either which simplifies the code immensely - also a lot faster than doing JNDI and JMS to use new CLI()!

That's heaps better - thanks!