[Cluster] AssertionError: Resource leak detected.

[jshkurti]

Hello guys.

I'm doing some intensive use of cluster module and I'm still getting this error sometimes (nodejs v0.12.0).

AssertionError: Resource leak detected.
  at removeWorker (cluster.js:346:9)
  at ChildProcess.<anonymous> (cluster.js:366:34)
  at ChildProcess.g (events.js:199:16)
  at ChildProcess.emit (events.js:110:17)
  at Process.ChildProcess._handle.onexit (child_process.js:1067:12)

I can see that it happens when there is still some handles left after all workers are deleted.
https://github.com/joyent/node/blob/master/lib/cluster.js#L347-L348
I also noticed that you call removeHandlesForWorker() only on 'disconnect' event, not on 'exit'.
https://github.com/joyent/node/blob/master/lib/cluster.js#L382

Is it possible that 'exit' event could be fired before 'disconnect' and thus be the reason for this bug to happen ?

Is this a Node.js bug or am I misusing the cluster module at some point ?
If so, what scenario could possibly trigger this bug ? Could you show me a sample code which intentionally triggers this exception ?

Thanks a lot.

[hertzg]

I think i have the same issue

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

assert.js:86
  throw new assert.AssertionError({
        ^
AssertionError: Resource leak detected.
  at removeWorker (cluster.js:346:9)
  at ChildProcess.<anonymous> (cluster.js:366:34)
  at ChildProcess.g (events.js:199:16)
  at ChildProcess.emit (events.js:110:17)
  at Process.ChildProcess._handle.onexit (child_process.js:1067:12)

[jshkurti]

@hertzg Do you, by any chance, have a .removeAllListeners('disconnect') or .removeListener('disconnect', ...) in your code ?

[hertzg]

@jshkurti I'm using removeAllListeners and removeListener but not for disconnect events on cluster, process or any specific workers.

[jshkurti]

Hey @hertzg , I did a Pull Request to fix this (which actually works) : #9418

But until it's accepted and merged I have found a work around.
Everytime I create a worker, I apply this function to it :

  function workAround(worker) {
    var listeners = null;

    listeners = worker.process.listeners('exit')[0];
    var exit = listeners[Object.keys(listeners)[0]];

    listeners = worker.process.listeners('disconnect')[0];
    var disconnect = listeners[Object.keys(listeners)[0]];

    worker.process.removeListener('exit', exit);
    worker.process.once('exit', function(exitCode, signalCode) {
      if (worker.state != 'disconnected')
        disconnect();
      exit(exitCode, signalCode);
    });
  }

Works like a charm ;)

[hertzg]

@jshkurti I'm unable to reproduce this issue again but will keep that in mind. Thanks

[megastef]

@jshkurti THX! - happened to me with a simple test script on ARM. It happens when a worker terminates (in my case there was an Error in the http request handler).

[Splaktar]

We ran into this with one of our servers last night :(
I see that the PR was never merged.

Is this fixed in v4.0.0?

[herkyl]

@Splaktar I had the same error today on a machine running v4.0.0