Supervisor spontaneously shuts down workers

[danielwhite]

I'm running a loopback application using strong-supervisor under the following conditions:

• strong-supervisor: 2.0.0
• Node: 0.10.36
• CentOS 6
• ESX Virtual Machine

The supervisor is being started through an init script that starts it with the following line:

start() {
  ... check for already running process ...

  daemon --check "$NAME" --user $USER --pidfile $PIDFILE "DEBUG=${DEBUG} sl-run --detach --cluster cpu --pid $PIDFILE  --log $LOGFILE --control $CTLFILE $APPLICATION_START"
}

Everything runs as expected, but after a few days I'm seeing the supervisor shut down without any operator initiating it (as far as my other logs go):

2015-06-09T08:25:41.152Z pid:28720 worker:0 INFO supervisor size set to 0
2015-06-09T08:25:43.080Z pid:28720 worker:0 INFO supervisor stopped worker 2 (pid 28732)
2015-06-09T08:25:43.089Z pid:28720 worker:0 ERROR supervisor worker id 2 (pid 28732) expected exit with 2
2015-06-09T08:25:44.079Z pid:28720 worker:0 INFO supervisor stopped worker 1 (pid 28730)
2015-06-09T08:25:44.083Z pid:28720 worker:0 INFO supervisor resized to 0
2015-06-09T08:25:44.105Z pid:28720 worker:0 ERROR supervisor worker id 1 (pid 28730) expected exit with 2

At this point, I'm running it with debug logging enabled, hoping it happens again.

In the meantime, any ideas what might be the culprit?

[rmg]

It appears the cluster master is being told to shrink the cluster down to 0 workers... which is really weird, considering it's not even running under strong-pm.

Some DEBUG strings to try, if you don't want to go wide open with * would be strong-supervisor:*,strong-control-channel:*,strong-cluster-control:*.

[rmg]

Oh, are you using log rotation? If so, you'll want to use SIGUSR2 for that. If you send SIGHUP it will restart the workers, which would probably produce log messages like the ones you are seeing. The DEBUG flags above will show for sure what the source of the resize event is.

[sam-github]

Also, you are asking both daemon to write the $PIDFILE, and sl-run to write it, I would rely on the daemon, and remove the --detach, the daemon tool should be able to deal with tools that do not detach themselves.

[danielwhite]

@sam-github That's what I initially thought, too, but the daemon function in /etc/init.d/functions only checks the pid file specified. The reason I specify it is because I am naming the pid file after our service rather than after sl-run.

From /etc/init.d/functions:

__pids_var_run() {
    local base=${1##*/}
    local pid_file=${2:-/var/run/$base.pid}
        ...
}

[danielwhite]

@rmg No log rotation yet.

Will continue monitoring with the DEBUG for now.

[danielwhite]

So I've manage to have it occur again, but this time I can correlate it to our automated deployments issuing a restart:

service <name> restart

This just boils down to stop and start. It works fine from the console, so it's possible that there is some sort of race condition here.

Either way, this simply seems to be a case of user error. Thanks for the help. :)