MlflowLogger: Invalid metric name: 'val/AR@100'. Names may only contain alphanumerics, underscores (_), dashes (-), periods (.), spaces ( ), and slashes (/)

[ihoholko]

Current mlflow logger doesn't support metrics logging from CocoMetric's proposal:

Metrics

val_evaluator:
  ann_file: ${data_root}/equifish_annotations/valdataset_toy.json
  type: CocoMetric
  metric: ['bbox', 'proposal']
  iou_thrs: [0.2]

Logger

# @package modules.MLflowVisBackend

type: MLflowVisBackend
save_dir: temp_dir #${paths.output_dir} #(str): The root directory to save the files produced by the backend.
exp_name: ${name}             #(str, optional): The experiment name. Default to None.
run_name: ${run_name}         #(str, optional): The run name. Default to None.
tags: null                 #(dict, optional): The tags to be added to the experiment. Default to None.
params: null                  #(dict, optional): The params to be added to the experiment. Default to None.
tracking_uri: http://${oc.env:MLFLOW_USERNAME}:${oc.env:MLFLOW_PASSWORD}@${oc.env:MLFLOW_HOST}
artifact_suffix: ['.json', '.log', '.py', 'yaml']     # (Tuple[str] or str, optional): The artifact suffix. Default to ('.json', '.log', '.py', 'yaml').

Are there some flags to fix this behaviour?

Env

mmcv                     2.0.0
mmdet                    3.0.0                /home/oem/Projects/aimosaic-roi/3rd/mmdetection
mmengine                 0.7.3
smmap                    5.0.0

[ihoholko]

@hhaAndroid

Fixed it by adding in

    scalar_dict_fixed = dict()
    for k, v in scalar_dict.items():
    scalar_dict_fixed[k.replace('@', '_at_')] = v

    @VISBACKENDS.register_module()
    class MLflowVisBackend(BaseVisBackend):
    
    ....
    
    @force_init_env
    def add_scalars(self,
                    scalar_dict: dict,
                    step: int = 0,
                    file_path: Optional[str] = None,
                    **kwargs) -> None:
        """Record the scalar's data to mlflow.

        Args:
            scalar_dict (dict): Key-value pair storing the tag and
                corresponding values.
            step (int): Global step value to record. Default to 0.
            file_path (str, optional): Useless parameter. Just for
                interface unification. Default to None.
        """
        assert isinstance(scalar_dict, dict)
        assert 'step' not in scalar_dict, 'Please set it directly ' \
                                          'through the step parameter'
        scalar_dict_fixed = dict()
        for k, v in scalar_dict.items():
            scalar_dict_fixed[k.replace('@', '_at_')] = v
        
        self._mlflow.log_metrics(scalar_dict_fixed, step)

It helped, of course. But is there a better way? Or should we make a Pull request?

[hhaAndroid]

@ihoholko This implementation should be put in mmengine, you can go to mmengine to create a pr?

[ihoholko]

open-mmlab/mmengine#1125

[zimonitrome]

Why is this not added yet???