How to get the original dataset name with username?

[npuichigo]

Feature request

The issue is related to ray data ray-project/ray#49008 which it requires to check if the dataset is the original one just after load_dataset and parquet files are already available on hf hub.

The solution used now is to get the dataset name, config and split, then load_dataset again and check the fingerprint. But it's unable to get the correct dataset name if it contains username. So how to get the dataset name with username prefix, or is there another way to query if a dataset is the original one with parquet available?

@lhoestq

Motivation

ray-project/ray#49008

Your contribution

Would like to fix that.

[lhoestq]

Hi ! why not pass the dataset id to Ray and let it check the parquet files ? Or pass the parquet files lists directly ?

[npuichigo]

I'm not sure why ray design an API like this to accept a Dataset object, so they need to verify the Dataset is the original one and use the DatasetInfo to query the huggingface hub. I'll advise the ray data team to use dataset id instead of dataset for the HuggingFaceDatasource API.