Problem with scheduler not starting job at specified time

[Karl Cassaigne]

Hi,

We've just upgraded some of our routers to version 7.5.2 and we were surprised to see that all scheduled jobs (simple "Application Invoker" jobs at fixed time) are not running at the specified time expression.

For example, we have a job with the following time expression "at 3:30", the job is effectively marked for execution the following day at 3:30 but effectively runs at 14:28 every day !!

I tried resetting the time expression, rebooting the router, the behaviour is the same except the time at which the job runs... it runs every day at the same time but this is not the specified time !!!

Our servers are all synchronized with a NTP server.

Thanks for your help,
Karl

[IIT Software]

We did not change anything in the Scheduler Swiftlet.

What was the release from which you did the upgrade?

Check with SwiftMQ Explorer under Scheduler Swiftlet/Usage/Active Job Schedules what the "Next Start" property of your schedules is set to.

For me it sounds like a system time problem with a 11:58h difference. Verify the SwiftMQ info.log which time it uses if you connect with Explorer. Also check warning/error.log whether you see any time change entries.

[Karl Cassaigne]

We upgraded from version 7.2.0 to 7.5.2 (it's the HA version).

This upgrade is part of a globale migration of our servers to another hosting provider.

Note that we also upgraded the JDK from version 5.0 to 6.0.

The "Next start" property display the good value but the actual execution time is wrong (cf. screen capture attached).

The system time as reported in the SwiftMQ logs is ok and no time change is reported in the logs.
swiftmq_scheduler.jpg

[IIT Software]

Just checked the code. We calculate the next start time which is formatted and set in the "Next Start" property. This value is ok. Then we create a timer which is fired on nextStart-System.currentTimeMillis() but it is fired with a 12 h difference.

The only explanation I have is a wrong default Locale or a permanent change of system time due to NTP.

[IIT Software]

Forgot to mention that we always use the default Locale, both in Timer and Scheduler Swiftlet, and we had no change in these Swiftlets between 7.2.0 and 7.5.2. For me it seems to be an environment issue (OS, JDK...).

[Karl Cassaigne]

The locale was set on the linux box (LC_ALL=fr_FR) but the problem still remains.

I have done another test on a 7.5.2 router (not HA) on a windows box with JDK 1.5 and it works fine.

Since we are facing some other issues (I will open a new topic) we will try downgrading the JDK to version 1.5 on the linux box to see if it helps...

[Karl Cassaigne]

I have some new information about this problem : I got the same issue on a windows box running SwiftMQ 7.2.0 and with NTP synchronisation configured to happen every week.

What I observed is the following : whatever the OS, JDK and SwiftMQ version, if there is any kind of NTP synchronisation enabled on the OS, then we get this line in SwiftMQ log file :

2009-10-13 00:59:38.106/sys$timer/INFORMATION/System time has changed, reordering timer task queue

and just after that, all scheduled tasks execution is corrupted by a large amount even if the time correction was negligeable (1 second or two maximum).

How is SwiftMQ supposed to react to system time change ? Can you confirm there's a bug in the scheduled task management in this case ? In this case, will you consider providing a fix ?

Thanks for your help,
Best regards

[IIT Software]

The Timer Swiftlet detects system time changes and reorders its timers accordingly. But it seems the Scheduler Swiftlet doesn't so this is definitely a bug and needs to be fixed by re-computing all schedules when a system time has been changed.