Garbage Collector and how does it work

[raresn]

Hey @ajwfrost,
Our interface freezez now and then and it is super annoying. (tested on a windows app)
Upon some Scout investigation, we noticed that the issue arrises when GC decies to come in and just kills the processing time with huge spikes.

Is there any way to avoid this? Make it run less often? Or any other parameters we can use it with? Run it in background? :)

Thanks.

[ylazy]

If your GC run often, so there're something are eating your RAM run often in your app. You need to optimize your code. You need to proactively destroy your objects when you doesn't need it anymore. Call gc when you close a tab, a dialog...

Furthermore, almost all of your classes should be "destroyable". In a framework that I'm building, I have DestroyableTextField, DestroyableContainer, DestroyableDispatcher... that implement IDestroyable. And I have a method called destroy() that can remove the object completely from memory. It can also check if the object was removed completely or not, and what object was preventing the destroying progress.

And don't forget to publish your app in release mode.

[ajwfrost]

The pause due to GC would tend to be the "collection" phase i.e. destroying all the objects that have no valid references left and cleaning up their memory. This can trigger destructors too (less of a thing in AS3, but it does happen under the hood in C++). And it's all done at once, single-threaded, hence the potential pause in the runtime if there is a lot to do there.

So per @ylazy's comments, the best things to do here would be to ensure it has less to do, when it does this! The two options really are:

1. Manually call the GC requests (from flash.system.System class) when you know there's going to be a bit of idle time. There's of course a downside to calling this too often though, as overall it has more work to do, which is why the runtime isn't garbage collecting fully on every frame...

2. Ensure that this type of GC mechanism isn't used so much, by using the reference-counted mechanisms instead. Within the memory management system in AIR, there are two ways we know if an object is being used or not: how many references are there to it, and is it reachable from some 'root' point. If you can ensure that the reference count on an object goes down to zero, then it will be tidily cleaned up at the end of every frame, which is an optimal "little and often" approach.

Of course, you can also improve things by not actually getting rid of objects - so e.g. object caching, if you know you'll need to use objects of the same type again in the future.

If you have two objects that reference each other, then they won't ever get collected by this reference-count mechanism. This is where we end up having to do the 'slow' garbage collection, where we gradually go through all the objects from the root and mark them as being "used". Then once that phase has finished (little and often, i.e. incremental marking) we get to the phase where we collect all the objects that were not marked as "used" and this is where you see the pause.

This is also where Scout can be useful, if you look at the allocations and deallocations between some of your frames, you can see how many of each type of object are being created vs destroyed, which gives an indication of what bits of code to look at to see if they can be refactored so that you don't end up with hanging references or just relying on this mark-and-sweep form of garbage collection..

Your final question - "run it in the background" - I'm not actually completely sure why the collection is done all at once, whilst the marking can be done incrementally. I suspect it's related to the fact that these objects may reference each other and we don't want to have such references hanging around whilst we're halfway through some of the destructor calls. But we can check that..

[Adrian-S]

@ajwfrost We have been using scout to detect bottlenecks, but our app is data intensive. Objects get created and destroyed, and we accept we don't have the best object pool management. But let's take this code for example:

        starling_internal static function fromPool(type:String, bubbles:Boolean=false, data:Object=null):Event
        {
            if (sEventPool.length) return sEventPool.pop().reset(type, bubbles, data);
            else return new Event(type, bubbles, data);
        }

Here the GC was "caused" by pop(). Probably this is an anonymous variable created and returned, and at some point is discarded even though it returns to the pool. I've notice this after removing a lot of elements from the stage. You might notice this function is in the starling.events.Event. I don't know if it can be written better and it still generates a problem.

The huge delay we are seeing could potentially be worsen by the fact that we have thousands of data objects in memory. A quick test reveled that if we generate 10k new Date() in an array on enter frame and we "lose" the array, than the GC is called every second and takes about 31ms to clear, getting a FPS of 22.7 (okay). But if we first generate 10 million dates in an array and we keep it, with the same code that generates 10k dates on enter frame, now the GC is called less often (~7 seconds) and it takes 191ms and the FPS is 4.6. Code below.

It's even worse in our app, having moments with 0.7 fps and 90% caused by GC. Probably because of a more complex reference tree. Since event the code above adds to the problem, and all the String concatenations that are out of our control.
So I would say the GC would needs to soften down. Here are some ideas:

• Run on different thread, something that will not impact the code.
• Run only when there is available frame budget.
• Could moving data handling in a different Worker (never used them), get us a parallel GC that will not influence the rendering / fps.

We are trying to optimize as much as possible, we moved data from memory to DB, and DB had some issues too, and we are moving some back to memory / cache. We have created a spaghetti monster, but GC is still our biggest enemy.

Code I used for testing:

	var now:Date = new Date();
	var memory:Array = new Array();
	for (var i:uint = 0; i < 10000000; i++)
	{
		var d:Date = new Date;
		d.time = now.time;
		memory.push(d);
		now.time--;
	}
	
	this.addEventListener(Event.ENTER_FRAME, function (e:Event):void
	{
		var memoryTemp:Array = new Array();
		for (var i:uint = 0; i < 10000; i++)
		{
			var d:Date = new Date;
			d.time = now.time;
			memoryTemp.push(d);
			now.time--;
		}
	});

[ajwfrost]

Hi

Thanks for the info and analysis. Some interesting points there.

The GC "pause" will currently happen at any point when it's needed i.e. when the marking has completed, and the next step is to do the collection. Which makes me think the people who designed this assumed that it would be a very quick operation...! But, it does mean, this is nothing to do with the pop() or event handler, it's just a function that's called any time there's any memory-related activity happening to see whether we can sneak in a small (!) bit of GC effort..

The huge delay we are seeing could potentially be worsen by the fact that we have thousands of data objects in memory. A quick test reveled that if we generate 10k new Date() in an array on enter frame and we "lose" the array, than the GC is called every second and takes about 31ms to clear, getting a FPS of 22.7 (okay). But if we first generate 10 million dates in an array and we keep it, with the same code that generates 10k dates on enter frame, now the GC is called less often (~7 seconds) and it takes 191ms and the FPS is 4.6. Code below.

So this shows the fact that it's having to work through all the objects to mark what's available and what's not available. This is the "incremental" part and so it doesn't take any more of your application's time, it just takes longer, if you have more objects. The interesting metric though is that the actual clear takes longer too - I can see how the time between GC collections would increase, but am not so sure what would make the collection time itself change.

Running on another thread is a goal we can have but I'm not entirely sure how feasible that is, similarly the idea of only doing it when we have time is dependent upon us being able to split it up into smaller chunks (in the same way as the incremental marking, if we could do incremental collection it would be good!)

Could moving data handling in a different Worker (never used them), get us a parallel GC that will not influence the rendering / fps.

I am 90% sure that this would help, because the Worker is running in its own thread with its own GC and so should have completely separate marking/collection activities. Might be worth confirming that though, before starting to refactor an application..

---

I think with the cost/example case that you've provided there, we can do some experiments, to see

• why that overall collection time is increased if there a more (non-collected) objects around
• whether we can do collection in an incremental fashion

And longer term we can see whether there is an option to handle GC operations on another thread..

thanks

[Adrian-S]

If marking is incremental and takes longer to finish, than it makes sense that more data is accumulated to be removed, and the collection takes longer. The test creates the same amount of data (10k items / frame), and without extra data in memory, the collection is 7 times faster and collection is also called 7 times more.

Array pop() and push() both move items to new reference, from inside the array to an outside object. Scout reports what operation created that reference. In the code I gave push() is reported as slow. After I filled all the memoryTemp array, I went trough it and set every item inside to null (code below). This made Scout report that the GC was called on root, but nothing changed as performance.

Here is the new version:

	this.addEventListener(Event.ENTER_FRAME, function (e:Event):void
	{
		var memoryTemp:Array = new Array();
		for (var i:uint = 0; i < 10000; i++)
		{
			var d:Date = new Date;
			d.time = now.time;
			memoryTemp.push(d);
			now.time--;
		}
		for each (var item:Date in memoryTemp)
		{
			item = null; 
		}
	});

Also there are deallocations on each frame. So that is data that we know we are going to delete. What is interesting with my example is that GC is constantly called on every frame and that can be seen on Scout graph.

• normally takes a constant 7-8ms each frame, probably tracking deallocations - this was constant on all tests.
• than it starts to take 12-14ms every 2-3 frames, probably scanning objects and marking
• than waits a bit and does the +190ms that I hate, probably removing data

So making the collection incremental too would have a lesser impact on the app. I would probably just allocate 1ms for collection each frame. Or something automatic based on the amount of data produced. For example in the last N frames we produced/marked an average of X items, the we should limit the collection at twice that amount.

[Adrian-S]

@ajwfrost Generating just 1000 objects on enter frame, with full memory, made the GC to be called again after 50 seconds. And what I think might be the "scanning objects and marking" was now ~30 frames apart.

Some other conclusions:

• GC is called when there is about 20% memory of garbage. Still has the same ugly impact.
• We don't need to generate junk on enter frame, calling GC on a timer will have the same impact with memory full.
• I now believe scanning for "objects that reference each other" is the main issue and it's done on collection.

What is the GC doing here?

Or CPU here:

Clearly something is done in background while execution continues, but only after it messed up with the main as3 thread.

I hope you can identify what is being processed and can "split it up into smaller chunks". If you can't split it, can you stop and resume from an index? Even if something is skipped it might be caught on the next parse.

If we call System.gc() in our app, is hell right now. But if you add some parameters, we could be testing even the a random collection type (you can call it "silly_random"). Just need to make sure it can't run more than a specified amount of time, so the second parameter could be an int [1-50] ms.

Thanks.

[ylazy]

for each (var item:Date in memoryTemp)
{
item = null;
}

Why did you do that? This code did nothing. If you want to remove all items of memoryTemp from memory, use:

memoryTemp.length = 0;

this code is better:

var now:Date = new Date();
var memoryTemp:Array = [];
var d:Date;
var i:uint;

this.addEventListener(Event.ENTER_FRAME, function (e:Event):void
{				
	for (i = 0; i < 10000; ++i)
	{
		d = new Date();
		d.time = now.time;
		memoryTemp[i] = d;
		now.time --;
	}
});

It runs at 60 FPS on my machine in release mode.

[Adrian-S]

Try it out with a full fixed memory (will take some time to start):

	var now:Date = new Date();
	var memory:Array = new Array();
	for (var i:uint = 0; i < 10000000; i++)
	{
		var d:Date = new Date;
		d.time = now.time;
		memory.push(d);
		now.time--;
	}
	
	this.addEventListener(.....);

The item = null; was my attempt to manually delete each object from the array. The only difference it made was that GC was now "blamed" on root and not on the Array.push(). This shows with telemetry on. Telemetry adds to the problem but the issue remains the same, Other than that, memoryTemp is just there to collect some junk and give it to the GC to test how GC works. With a full memory it gets bad.

@ajwfrost here is the graph (app was ran with no debugging, and no telemetry):

I get 60fps too except that one frame.

We do a simple screen transition and suddenly we get a stutter and app visibly freezes.

[raresn]

@ajwfrost anything that it is clearly done wrong here?

[ajwfrost]

calling System.gc() each frame with a lot of memory and 150ms execution time is... big no no.

Is that the GC execution time even if you call it each frame?! I would have expected that number to be a lot lower if this was called regularly.. the delays we're seeing in the test cases are down to the stack memory which shouldn't be significant if you call the 'delayed' gc() method.?

Why is there one single frame that looks like a "stop-the-world garbage collection", after what it looks like the incremental garbage collection?

The incremental part works on the 'heap' memory i.e. objects that you're creating and are properties of other objects etc. So, you don't really notice that because it happens a tiny bit per frame. It goes through and marks objects as "reachable" so we know not to collect them. But when all the incremental marking is done, there is then a final 'mark' pass of anything on the 'stack' memory i.e. local objects that are referenced in any of the functions in the AS3 call stack. This is where we see things taking time when we have a large array sat in one of the enclosing functions.

This part is where we're looking to see whether we can delay that final "mark the stack and then collect anything unused" to the exit point of a frame/event handler, i.e. when all the application's ActionScript has finished and the runtime is (hopefully) about to idle. Which I think is what that System.gc() is also set up to do: it does request a stop-the-world garbage collect, but the trigger for the actual Collect() call happens as you exit a call in to the VM.

I think we may need to look at this using real-world test cases rather than this kind of array-based test, just to see in practice where the time is taken up. As it's perfectly possible for us to optimise for a specific case, but for that to then not help at all in the real world applications.

thanks

[Adrian-S]

@ajwfrost graph is from normal enter frame and garbage generator, no debugging and no telemetry. GC is not called.

• Without the garbage generator, and using a timer it would be just a 150ms spike on every timer event.
• Without the garbage generator, and using enter frame to call gc(); it would be just a solid 150ms block each frame. (app becomes unusable)

In a real test I've noticed String generates the most garbage. For our app, is by far the most crappy thing. Everything generates garbage when manipulating strings. Array.join() and + have the same problem. And than it's blamed on something else when GC is called.

This is the graph from real test (debugger and telemetry ON):

You may notice GC spends 357ms from something related to Event.fromPool and Vector.pop. It is most time spend from the frame budget. I honestly doubt it's reported correctly, or that I can fix the as3 code.
Memory is not freed, even worse, we have ~3000 string allocations and deallocations, and that's happening on Responder callback (I think DB response) in just 12ms. Memory is not an issue, I know most of the allocations are used by trace functions (string concatenation), and these only get called on debugging. So they can be fixed.

We don't see some of these issues with test data (smaller). Only if our client has a lot of data this starts to happen. This is how we created the test and found out GC is slow for no reason with same amount of garbage. Calling gc() for any reason is pointless. The "incremental part" is not part of that AS3 function.

I don't want to escalate this any further. I will continue to optimize what I can, and hopefully that will be enough to keep delivering a good product with the framework we love.

[ajwfrost]

Hi

Thanks - timely report, as we had got some results from some testing this morning which I'd been meaning to post. Before that though - a quick comment on your call stack showing the garbage collection within Vector.pop() .. this would basically be the fact that GC functions are meant to be called little and often, so the fact that there's an allocation or deallocation happening there is enough for the runtime to try to do a little bit more of the incremental marking. The issue then happens where this has just finished and it decides to then do the final - all-at-once - completion of the GC process.

From what we've found in testing some of these (admittedly, carefully crafted) test cases, the GC takes a lot longer if it has to also scan an ActionScript call stack. In your case it's 10 functions deep and we don't know what's been allocated or referenced in any of those functions.. so this could well mean that extra time is spent marking things that don't need to be marked.

A full gc() on every frame I guess would be a bit bad because this does the whole 'incremental marking' thing all at once, as well as the final stack-marking (well, hopefully nothing there because of how the gc() call is implemented as a delayed call) and the collect/sweep. But .. if we were to have the incremental marking working as it is now, but then to delay the final collect/sweep until we had no AS3 stack, then it might help.

Which is what we had tested earlier... I'm actually going to get these test re-done because they've got a lot of logging which is impacting performance I think, but the fairly obvious result from what we've tried is that instead of the GC activity (in any one frame) being fairly variable and topping 40ms in some cases, it becomes a lot more consistent and limits itself to under 20ms.

I'm also wondering whether we do have something odd happening in an Array object... the enter-frame handler for our test case includes the code:

	for (var i:uint = 0; i < 10000; i++)
	{
		var d:Date = new Date();
		d.time = now.time;
		memoryTemp.push(d);
		now.time--;
	}
	for ( i = 0; i < 10000; i++)
	{
		memoryTemp[i] = null;
	}

To my mind, this should create a new date, set its 'time' property, and push it onto an array. And then it should go through the array and set all the array references to zero. At which point, I would expect all those Date objects to be cleaned up because there are no outstanding references to them...

Which makes me wonder why I'm seeing a saw-tooth memory usage that continues across frames, and why there is such a need for mark-and-sweep garbage collection.... something else to investigate!

Regarding the String usage... we can dig a little more into how this is actually used internally, particularly when concatenation etc is done. But I think there's already quite a lot of thought and optimization gone into it..

thanks

[ajwfrost]

FYI, results from a 'clean' build with that basic test (loads up a bit of memory to start with but not excessive amounts, and then creates 10k Date objects each frame...):

Current situation:

With a change to delay the FinishIncrementalMark function until we're not in a frame handler:

Actual differences seem pretty minimal but it's seemingly smoother (and the "AS3" parts no longer include the GC time, hence those differences..)
Probably we should also update this so that it's not in any event handler, not just the enterFrame ones!

This perhaps is where we need a "labs" kind of construct so that we can provide runtime versions for you to try out, to see how they actually behave in "real world" testing!

[ajwfrost]

Quick update -> those Date objects are being collected via ZCT, which is what I thought. So the "Array" set-objects-to-null is working as you would have hoped!

[raresn]

@ajwfrost we are more than happy to help with whatever you can throw at us to test or whatever! This might actually help overall performance of air quite a bit :)

[pnico]

Is the Array maybe getting resized/reallocated sometimes during push() and pop()?

[Adrian-S]

One array is fixed with 10 million Date objects and it's there to fill memory. The other one is created and deleted, on each frame, just to create junk and trigger garbage collection.

If the fixed array is not created, automatic garbage collection is much faster, with the same amout of junk created on each frame.

From tests with full memory marking is done in chunks over time and it has a lower impact on fps. But collection takes about 200ms and from what I understand, it should just delete what was marked and that should be the same amount of data in both cases.

[pnico]

Yeah I was just wondering if a bunch of needless array resizing could be happening with the temporary 10k arrays during the loop. Wild guess though. Could just be a condition when memory is tight and GC is trying to compact the heap, or something else 🤷‍♀️

[ajwfrost]

The one thing I was wondering is the construction of the test function .. it's an anonymous enter-frame (or timer) callback, sitting within the scope of the containing function where that big initial memory was allocated.

So, when we call the callback to allocate the temporary array, we do have that large array still within the scope stack.

Potentially if the callback is separate, it should be quicker..

[2jfw]

Semi-Offtopic, @ajwfrost, I was wondering, if we could get a specific method to manually dispose/destroy an object directly/immediately rather than making sure it will get garbage collected by nulling references etc? Not sure if applicable though...

[ajwfrost]

Manually forcing the destruction of an object may be tricky.. most unused object should normally be disposed very quickly because of the reference counting; but if you had a reference to it from somewhere else (like in these test cases, with variables available via the scope stack) then this could cause problems....

Larger memory-using objects like ByteArray or BitmapData, you can dispose or set the extra memory usage to zero.

Potentially we could add in some hooks so that - when developing/debugging an app - you can dynamically see what objects are around and what reference counts they have?

[ylazy]

Yeah I also thinked about it...

If we destroy an object directly via a method, e.g. called System.destroy(object) so what will happen with other objects or events that are strong refering to the object? Maybe we will run into unexpected errors or behaviors.

So the method System.destroy(object) must be called if object is the only remaining strong reference, and the method should only be worked if there's only one strong reference, and then it will remove the object from memory completely and also set null the object.

Or, at least there should be some method to control when the gc work, e.g. System.pauseGC(), System.resumeGC, System.maxPrivateMemory = 1024*1024*1024; // GC only runs if the app use more than 1GB of memory

[2jfw]

ajwfrost wrote:

Potentially we could add in some hooks so that - when developing/debugging an app - you can dynamically see what objects are around and what reference counts they have?

This would be really helpful; I thought that Scout with Advanced Telemetry would do so, but I think it's not going down so "deep" in the evaluation of objects. Could it be, that old FlashBuilder had such an in-depth profiling view where you could exactly see all this?

ylazy wrote:

If we destroy an object directly via a method, e.g. called System.destroy(object) so what will happen with other objects or events that are strong refering to the object? Maybe we will run into unexpected errors or behaviors.

At least in debug mode, this could actually also be a good hint to see where a potentially non-removed reference is.

[raresn]

@ajwfrost should we expect any improvement here? Just trying to get some sort of a conclusion.
Thank you.

[ajwfrost]

In terms of changing how the GC currently works .. we looked at the idea of only doing the collection work when we've exited the AS3 calls, to minimise the stack scans, but this actually had a worse result in some of the test cases despite helping in the initial one (which had a massive array of objects on the stack).

The current plan is to see whether we can make the collection work more incremental - although of course, the stack scan cannot be, so this may present a challenge.

The other thing to do is to provide more tools to help people understand what's happening, what objects are involved and how to then mitigate this. I gather that there were some Flash Builder tools that provided some of this kind of profiling; Adobe are checking to see if we could get some of those modules somehow - although we did have a quick check in the source to see the hooks that they might be using (as Flash Builder would have been running a customised build of Flash Player for this) and there would definitely be extra effort to get those working in AIR. Or per the above we may look at adding this to Scout..

And then extra control over GC via some new AS3 APIs, which is something we'd look at for AIR 52.0 because of the API update.

But in the short term, regarding how it actually works, I'm afraid there's unlikely to be much we can do...
One thing that just came to mind is whether the parameters around GC could be made less strict, given the improvements in phones/etc and memory that's available. So we can look at that (or expose those to developers) to see if that would mean fewer attempts to do GC in the first place. As long as people are happy with large memory use overall, of course..

thanks