An unusually large amount of memory usage

[pflannery]

When creating 10000 small tasks I find that the process memory peeks around 170MB. When I run the same 10000 tasks with setImmediate it's doesn't go over 40MB.
So its seems that task group is generating an unusually large amount of memory overhead.

Here the code I was testing with.

var TaskGroup = require("taskgroup").TaskGroup;
var testArray = Array(10000).join().split(',');
var grp = TaskGroup.create();

testArray.forEach(function(value, index) {
  grp.addTask(function() {
    console.log("task"+index);
  });
  /*
  setImmediate(function() {
    console.log("task"+index);
  });
  */
});

grp.run();

I only tested on Windows 8.1 so not sure what a Linux box will output

[balupton]

Perhaps linked to #12

[balupton]

To be honest, I would be open to migrating TaskGroup to a pre-processor that actually cares about performance. CoffeeScript no longer cares about it at all, which is really sad. Perhaps CoffeeScript Redux is an option, otherwise we could look at TypeScript, or maybe just native javascript (not that it is actually a good language - unlike compared to 5 years ago)

/ ref https://developers.google.com/v8/experiments

@pflannery I've started a discussion about this here: https://discuss.bevry.me/t/move-from-coffeescript-to-es6/30/1

[pflannery]

Seeing as the topic has gone and the ssl is telling me that it's not valid. I will just reply here for now :)

I agree, I think moving to native ES5\ES6 will give us more control over what code we use. I'm all for it.
I find coffee classes are a real pain when analysing them in a heap snapshot because they are all called ctor instead of the class names I define. This makes it very difficult.

I've been using v8-profiler to take snapshots of the test code I posted above and below you see that a Task instance is around 8K and the _events property generated by the EventEmitter is adding a 5-6kb per Task. So I'm wonder if its worth replacing the event emitter for something that is much lighter in weight?

[balupton]

https://discuss.bevry.me/t/move-from-coffeescript-to-es6/30 is the new link :-)

[pflannery]

Just want to correct myself slightly, EventEmitter isn't adding 5-6k per task, instead it just seems that its referencing 5-6k.

[balupton]

@pflannery can you do a screencast sometime of how you come up with that profile information - I've spent ages trying to figure it out and no luck!

[pflannery]

Here's the code for generating the heap snapshot..

var fs = require('fs');
var profiler = require('v8-profiler');

function saveSnapshot(fileName) {
  var buffer = '';
  var snapshot = profiler.takeSnapshot("test");
  var t = snapshot.serialize(
    function iterator(data, length) {
      buffer += data;
    },function complete(){
      fs.writeFileSync(fileName, buffer)
    }
  );
}

var TaskGroup = require("./taskgroup").TaskGroup;
var testArray = Array(10000).join().split(',');
var grp = TaskGroup.create();

testArray.forEach(function(value, index) {
  grp.addTask(function() {
    // console.log("task"+index);
  }); 
});

grp.on("completed", function(){
  saveSnapshot("completed.heapsnapshot")
});

grp.run();

[pflannery]

also here is a Memory Management Masterclass video on https://www.youtube.com/watch?v=LaxbdIyBkL0

[pflannery]

I've created a gist with javascript that generates the cpu profile and ensures the deopt reasons will render in latest chrome - https://gist.github.com/pflannery/8e38a06a844a7fc362dc

setup instructions

• create a temp folder
• npm install taskgroup
• npm install v8-profiler
• run node taskgroup-cpuprofiling.js
• open the generated complete.cpuprofile file in chrome-dev-tools -> profiler

[balupton]

Sweet. I'll give it a go (watching the talk right now). I'm wondering if it is best to just do the performance tracking inside the browser with a browserified/webpack'd build of taskgroup. Your thoughts?

[pflannery]

I've never used browserified or webpack so don't know what the outcome would be.

[pflannery]

@balupton It wouldn;t let me post post than three times on the discuss thread and it wont let me add any more links in the post either so I've replied here

I've been analysing the heap generated from running the my test and i've seen that commenting out the following lines takes the heap snapshot from 114MB down to 31MB. (the snapshot was taken once during the taskgroup completed event)

Lines here and Lines here

So from that I think this confirms that the EventEmitter (taking up 53MB) is a big part of this problem and it also seems that the Map copying (taking up 14MB) is adding to it too.

[balupton]

Interesting. A few notes:

Commenting out those events means that the taskgroup of 31MB doesn't work...

Perhaps the downsize of the commented out map options are actually because it disables domains.

My other thought is that it the completed event will be too early, as the normal garbage collection wouldn't have kicked in yet, it will have to be once the taskgroup has completed and the scope of the taskgroup instance has been forgotten. I'll do some evaluating of this today. Keep up the good work.