Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable linking across missing segments with motile based tracking #85

Open
constantinpape opened this issue Oct 30, 2023 · 1 comment
Open

Comments

@constantinpape
Copy link
Owner

The default tracking with motile from https://github.com/constantinpape/elf/blob/master/elf/tracking/motile_tracking.py#L5, which just uses the overlap to construct edge costs, works well for many tracking problems.
However, it is currently limited by the fact that it cannot cross gaps (i.e. can only link objects if they are present in each consecutive frame or slice).
Such gaps often happen due to:

  • Occlusion
  • Objects going in and out of frame
  • Segmentation errors

We should extend the tracking functionality so that objects can be tracked with gaps. I can see two possible implementations for this.

1. Including edges that link objects across more than one frame in the tracking problem:

This could be done by extending compute_edges_from_overlap to include a parameter that determines the maximal gap to be closed (in terms of frames), and then computing the edges and overlap for the corresponding adjacent frames instead of just the nearest adjacent frame.
This should also include an option for a penalty that grows with the linking distance, so that links with smaller gaps are preferred.

2. Adding new functionality that links tracklets

This would add a second layer problem on top of the first tracking problem, where the nodes correspond to tracklets (the result from the first tracking run), and edges can link the tracklets and can allow linking across gaps.
The advantage of this approach is that more expressive features can be computed for the edges between tracklets, e.g. based on how well the local trajectories align.

I think we should go with the first option for now:

  • it is much easier to implement, since we just need to update the function that computes the edges and can just re-use the rest of the current functionality.
  • for the simple overlap based features it should also yield better results, because the gap closing is solved together with the rest of the tracking problem rather than as a second separate step.

We should then also add a function that post-processes the gaps after tracking and inserts the interpolated segmentation masks.

@constantinpape
Copy link
Owner Author

The compute_edges functionality was now moved here: https://github.com/constantinpape/elf/blob/master/elf/tracking/tracking_utils.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant