Custom Labels not shown in CVAT's AI Tools sidebar after Model Update on Nuclio

[bilalcodehub]

Dear All,

I completed an annotation task using the active learning (AL) pipeline within the MONAILabel and CVAT integration for the endoscopy app. After training, the updated models were stored inside the MONAILabel docker container. To make the model available in CVAT, I needed to update it in Nuclio so I copied the entire endoscopy app folder from the MONAILabel docker to my host machine’s CVAT directory using the following command:

docker cp $(docker container ls | grep 'youthful_herschel' | awk '{print $1}'):opt/monai/apps/endoscopy cvat

Next, I ran the update_cvat_model.sh script on the host to update tooltracking model on the Nuclio's docker:

./update_cvat_model.sh tooltracking

The output was:

Fetched and Published latest model: tooltracking to the nuclio function container.

Based on this, I assumed that both the updated model and the custom labels (used during model fine-tuning) would now be reflected in CVAT. However, when running a new job in CVAT and attempting to use the AI models from the sidebar, I still see the default Tool label instead of the custom labels utilised during fine-tuning with MONAILabel.

Here’s a screenshot for reference:

Is there something else I need to update or configure to ensure CVAT picks up the custom labels after deploying the fine-tuned model on Nuclio?

Your guidance would be greatly appreciated!

Many thanks and kind regards,
Bilal

[SachidanandAlle]

https://github.com/Project-MONAI/MONAILabel/blob/main/plugins%2Fcvat%2Fendoscopy%2Ftooltracking.yaml#L21

If you have more labels then you need to update above and also tool tracking infer in monailabel

[bilalcodehub]

Thanks for the response.

I updated the labels in the MONAILabel Docker file located at /usr/local/monailabel/plugins/cvat/endoscopy/tooltracking.yaml and restarted CVAT, Nuclio, and MONAILabel, but the issues in CVAT still persist. It still displays the Tool label in AI Tools sidebar.

[SachidanandAlle]

Are these labels not inherited from CVAT Project?

[bilalcodehub]

Are these labels not inherited from CVAT Project?

I’m not entirely sure how label inheritance works in CVAT, but from my understanding, the labels should be inherited, as they were initially created by MONAILabel from a CSV file when setting up the CVATDatastore. CVAT itself is displaying the labels correctly in the left sidebar under the “Labels” tab, so they seem to be recognized by CVAT.

When I imported existing annotations into a CVAT task, the labels were applied to the images without any errors, and everything looked good. After completing the task, MONAILabel initiated the training process as expected, and again, there were no issues during that phase. At that point, I assumed the labels were properly propagated across all necessary components in both MONAILabel and CVAT.

However, after updating the fine-tuned model on Nuclio, I expected CVAT to automatically pick up these custom labels when running a new job. Unfortunately, it didn’t. CVAT still shows the default “Tool” label in the AI models panel, instead of the custom labels used during model fine-tuning.

Following your reply, I updated the custom labels in the tooltracker.yaml file, as shown below:

I also modified the labels in the lib/infers/tooltracking.py file to reflect the custom ones:

Despite these changes, CVAT still doesn’t seem to recognise the custom labels post-deployment. I’m wondering if there’s another configuration or step I might be missing to ensure CVAT picks up these updated labels when the model is deployed in Nuclio.

Any guidance on how to resolve this and ensure CVAT recognises the fine-tuned labels would be greatly appreciated!

Many thanks,
Bilal

[SachidanandAlle]

This is when MONAILabel creates a project. If it's an existing project, then you might want to edit the project properties and add new labels.

[bilalcodehub]

Thank you for the clarification.

Our custom labels were created successfully in CVAT for the MONAILabel-generated projects. However, when using the AI Tools for annotation and selecting Detector followed by tooltracking, the dropdown menu for label mapping on the left only shows a single class—Tool.

I’m unsure where this Tool sidebar is pulling these labels from (probably Nuclio), as it doesn’t seem to reflect the custom labels used during model fine-tuning. We updated model weights in Nuclio and I hope changing tooltracking.yaml allows it to access these labels now. Shall I run .deploy.sh script again?

Any insights on where the AI Tool gets its label information and how I might ensure it picks up the correct labels?

[SachidanandAlle]

correct. u need to delete previous once before u deploy again.
http://server:8070/projects/cvat/functions

[SachidanandAlle]

Have you tried to check what is the result of infer from monailabel? Does it have all the multi labels correctly produced? Json/XML output for annotations/polygons

[bilalcodehub]

I have ensured that the same set of labels is used across all configuration files in the endoscopy app. Although CVAT isn’t showing any errors and appears to be processing, no segmentation results are displayed on the images. Additionally, nothing is appearing on the MONAILabel console, as you can see below:

[2024-11-12 16:23:27,460] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels
[2024-11-12 16:24:28,152] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels
[2024-11-12 16:25:28,841] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels
[2024-11-12 16:26:29,526] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels
[2024-11-12 16:27:30,215] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels
[2024-11-12 16:28:30,904] [405] [Thread-5] [INFO] (monailabel.datastore.cvat:185) - No Tasks with completed status (current: annotation) to refresh/download the final labels

How can I check the inference from MONAILabel to see if multi-labels are correctly produced? I haven’t done this before, so I would appreciate any guidance.

[SachidanandAlle]

Last week released a new rc1 where the integration is slightly modified for cvat and monailabel.

Now on cvat it's just a call to monailabel server.
GPU is only needed on monailabel server.

That should help you to debug everything at monailabel side to understand what is happening.

Otherwise in your set up you have to look into nuclio docker logs for that function for debugging.

[SachidanandAlle]

And if you are switching to new RC candidate then you make sure you delete the previous function and deploy it again.

[bilalcodehub]

Thank you for the information; this sounds like a significant improvement. I’m happy to try the latest version where these enhancements have been made.

Can I obtain the new RC by updating to the latest version of the MONAILabel Docker image? Also, do I need to update the CVAT Docker as well?

[SachidanandAlle]

Only monailabel docker needs update. CVAT side no changes needed. However just delete the previous monailabel function (for the model) in nuclio and install the latest version (from monailabel main branch)

…

On Tue, Nov 12, 2024, 7:53 AM Muhammad Bilal ***@***.***> wrote: Thank you for the information; this sounds like a significant improvement. I’m happy to try the latest version where these enhancements have been made. Can I obtain the new RC by updating to the latest version of the MONAILabel Docker image? Also, do I need to update the CVAT Docker as well? — Reply to this email directly, view it on GitHub <#1765 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABX7YK52UYNRFESX423PL7T2AIP6PAVCNFSM6AAAAABQTDGCPOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRSHA2DAMY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[bilalcodehub]

Thank you, I’m on it.

Quick question: I’m currently using the latest versions of Nuclio (nuctl-1.13.12-linux-amd64) and CVAT (v2.7.5) with MONAILabel. This shouldn’t cause any issues, should it?

[SachidanandAlle]

Recommended to use latest cvat

[bilalcodehub]

Thank you, @SachidanandAlle, for your support—it all worked out perfectly! I am now successfully using the latest CVAT with Nuclio with Monailabel.

I would like to share a few suggestions to make the endoscopy app more accessible and user-friendly for researchers:

1. Simplified Customization: Currently, modifying labels requires changes in multiple places. For example, you need to update labels in the main.py, lib/inferer/tooltracking.py, and the Nuclio configuration file (cvat/endoscopy/tooltracking.yml). These files also use different formats for constructing this information, which increases the likelihood of errors due to the cognitive burden. It would be great if the endoscopy app could function more like the radiology app, where custom variables like labels can be defined once and picked up wherever needed.
2. Automatic Model Deployment: After models are trained during the Active Learning cycle, there should be an automated way for MONAILabel to push the updated models to Nuclio without requiring manual intervention from developers.

These changes could greatly improve usability and reduce the setup complexity for researchers. Thank you for considering this feedback, and I appreciate all the work the team has done on this project. Well-done!

[SachidanandAlle]

Automatic Model Deployment: - The new RC version should solve this problem as all the model/inference is part of MONIA Label and Nuclio is just a wrapper to call monailabel. You don't have to re-deploy unless you change anything on the yaml file. Otherwise any newly trained model gets served via monailabel.

For the first one, yes. I do agree that few things can be more simplified. If you have any ideas please feel free to contribute as well.
I have not revisited CVAT (as dataset) interaction completely. I will keep your suggestions in mind when I am simplifying anything on that front.

[bilalcodehub]

Automatic Model Deployment: - The new RC version should solve this problem as all the model/inference is part of MONIA Label and Nuclio is just a wrapper to call monailabel. You don't have to re-deploy unless you change anything on the yaml file. Otherwise any newly trained model gets served via monailabel.

You mean even during the first-time installation of CVAT/Nuclio, MONAILabel automatically handles the deployment? Or is this feature only applicable for re-deployments after models have been trained and already added to Nuclio?

[bilalcodehub]

For the first one, yes. I do agree that few things can be more simplified. If you have any ideas please feel free to contribute as well.
I have not revisited CVAT (as dataset) interaction completely. I will keep your suggestions in mind when I am simplifying anything on that front.

@SachidanandAlle: Can you please guide how to contribute to the MONAILabel?

[SachidanandAlle]

Please create and issue to describe and then corresponding Pull Request :)

Before PR please make sure to run precommit checks and unit tests locally