Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Autoware compatibility #1

Merged
merged 8 commits into from
Jun 13, 2024

Conversation

kaancolak
Copy link
Collaborator

@kaancolak kaancolak commented Nov 30, 2023

Changes

  • Voxel encoder refactored to make trained models compatible with Autoware
  • Add PyTorch to ONNX converter for deploying trained CenterPoint model Autoware
  • TierIV dataset loader added
  • Custom Nuscenes Dataset metric evaluator added to evaluate TierIV dataset

Blocks merging of:

@kaancolak kaancolak requested review from yukke42 and miursh November 30, 2023 14:38
@kaancolak kaancolak self-assigned this Nov 30, 2023
@yukke42
Copy link

yukke42 commented Dec 12, 2023

@kaancolak Apologies for the late review 🙇
Could you share a trained pytorch checkpoint to check onnx conversion and try it on autoware?

@kaancolak
Copy link
Collaborator Author

Thank you @yukke42 -san, I shared over google drive:
checkpoint

@yukke42
Copy link

yukke42 commented Dec 20, 2023

@yukke42
Copy link

yukke42 commented Dec 20, 2023

@kaancolak
How about using the official project-style instead of forking the repository?
It's more simply and easier to manage codes.

@kaancolak
Copy link
Collaborator Author

Hi @yukke42 -san, I am so sorry for the delay, I was working on another project. 🙇

Before the implementation, I asked for suggestions on how to implement it. But I completely agree with you, the formal project style is simpler and easier to track. Subfolders of mmdet3d could be implemented with the project style as inherited from the main repo. But for tools, probably we need to hard copy to the original repository. If you want to go with this way, I can convert the repository to the official project style.

@yukke42
Copy link

yukke42 commented Jan 15, 2024

But for tools, probably we need to hard copy to the original repository.

@kaancolak In that case, it would be simpler and easier to manage codes by creating the project style folder in this repository.
We don't need to copy any files and we can sync with upstream easily.
How about proceding in this way?

@kaancolak
Copy link
Collaborator Author

@yukke42 -san, thank you, I just want to confirm.

  1. We will create a project-style folder under the fork of the repository that will contain specific changes for training (like a model, train config, metric, etc.)
  2. Apply specific changes for tools over the fork (like tier4 dataset converter, create data, etc.)

@kaancolak kaancolak marked this pull request as draft January 22, 2024 06:08
Signed-off-by: Kaan Çolak <[email protected]>
@kaancolak
Copy link
Collaborator Author

@yukke42 -san, friendly ping. 🙇

I updated the repos, could you check and review it?

@kaancolak kaancolak marked this pull request as ready for review January 22, 2024 09:00
Signed-off-by: Kaan Çolak <[email protected]>
@kaancolak
Copy link
Collaborator Author

Hi @miursh -san,
I guess @yukke42 -san doesn't have enough time to review. Could you assign a proper reviewer?

@miursh
Copy link

miursh commented Feb 15, 2024

@kaancolak
Thanks for your reminder. I'm going to ask some of my team if anyone can review this.

@Shin-kyoto Shin-kyoto requested review from Shin-kyoto and kminoda and removed request for yukke42 February 19, 2024 11:48
@Shin-kyoto
Copy link

@kaancolak

Your PR has many areas that can be improved by applying pre-commit (for example, missing a newline at the end of the file, etc.). Could you add CI to apply this automatically?

self.z_offset = self.vz / 2 + point_cloud_range[2]
self.point_cloud_range = point_cloud_range

def forward(self, features: Tensor, num_points: Tensor, coors: Tensor,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaancolak
This function is very long, and it is really difficult for reviewer to check. So could you cover this function by test?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this function is hard copied from the original repository, it just contains a small adjustment to make it compatible with Autoware's centerpoint TensorRT implementation. Autoware's implementation doesn't contain voxel center z as an encoder input but the original mmdetection3d implementation contains it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaancolak

Thank you for the quick response! This repository is open-source and will receive PRs from various contributors. To ease the burden on reviewers, adding tests seems to be the simplest solution for the future. What do you think? It's also fine to include tests that are already present in the original code.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Shin-kyoto -san, thank you.

IMO, we can add tests for customized operations like "autoware_voxel_encoder", and "T4Dataset" operations, etc. , original parts of the implementations have already been checked with the test under mmdet main repository.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaancolak

I also think that we can add tests for customized operations. PillarFeatureNetAutoware is also customized class and test will be really helpful when we review the PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Shin-kyoto -san, I added tests for the T4 dataset and autoware_voxel_encoder, PillarFeatureNetAutoware base class of autoware_voxel_encoder.

You can basically run with

cd mmdetection3d/ && pytest -s projects/AutowareCenterPoint/tests/

Relevant files:
https://github.com/autowarefoundation/mmdetection3d/pull/1/files#diff-0b5c1ac645064857b8ac247c5d193291b74c6130bfdfa4c22aed90d2bd0df58a

@kaancolak
Copy link
Collaborator Author

@Shin-kyoto - san, thank you so much for your quick review and advice. I will apply an automatic CI check to a repository.

@xmfcx xmfcx closed this Feb 19, 2024
@xmfcx xmfcx reopened this Feb 19, 2024
Copy link

@kminoda kminoda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution 🙏

@kminoda
Copy link

kminoda commented Feb 21, 2024

@kaancolak Hi, as reviewed above, currently the repository does not work due to this commit: 37fc84d.

Given that the contract between us was till the end of this month, please prioritize to make the whole pipeline work as intended as soon as possible so that we can review at least from the user's perspective.
Improving the code quality e.g. by adding a test (as proposed here) would also be nice, but that is the second priority.

Signed-off-by: Kaan Çolak <[email protected]>
Signed-off-by: Kaan Çolak <[email protected]>
@kaancolak
Copy link
Collaborator Author

Hi @kminoda @Shin-kyoto , I apologize for not addressing this matter earlier as I was dealing with other issues. I have made updates to the CI actions and changed the dataset name to "T4Dataset." It is now prepared for review.

Signed-off-by: Kaan Çolak <[email protected]>
@xmfcx
Copy link

xmfcx commented Mar 26, 2024

@Shin-kyoto -san, could you review this again?

@mitsudome-r
Copy link
Member

I talked with AutoCore and @beginningfan could follow the instructions and see if they can retrain the model.

nusc, train_scenes, val_scenes, True, max_sweeps=max_sweeps)
return train_nusc_infos

dataset_config = root_path + version + '.yaml'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version of dataset_config here does not match the T4Dataset. I downloaded the dataset here, and the yaml file inside does not have the version data passed in. Should we change the yaml file name of the dataset?

Copy link
Collaborator Author

@kaancolak kaancolak Apr 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beginningfan with which command did you get this error? T4Dataset is the dataset format, and sample_dataset is a version of it in there. So sorry for the late response, I missed notification.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaancolak Oh, my fault. I forgot to click on review changes before, which caused the comment to remain in a pending state。
I used default version param, which is v1.0. It will be ok when using sample_dataset for the version. Then I suggest writing the T4Dataset version in the document, or modifying the default parameters of the T4Dataset version number.

Copy link

@beginningfan beginningfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaancolak
Copy link
Collaborator Author

LGTM

Thank you so much @beginningfan, do you have any plans for reviewing documentation?

@beginningfan
Copy link

LGTM

Thank you so much @beginningfan, do you have any plans for reviewing documentation?

@kaancolak I've read the documentations before when testing, and it's basically fine. I still recommend adding the version of T4Dataset in the documentation to avoid confusion. You can add it to the document of this PR or your other PRs.

@kaancolak
Copy link
Collaborator Author

still recommend adding the version of T4Dataset in the documentation to avoid confusion.

I added an explanation to the training documentation.

https://github.com/autowarefoundation/autoware.universe/pull/5570/files#diff-ef509aa43435872925d0134c14c088da6f51549d59e84cc5bd3e74fd4fa333f9R207

@xmfcx xmfcx merged commit b89c542 into autowarefoundation:main Jun 13, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants