model weights are always downloaded repeatedly.

[1193749292]

Take facebook/opt-6.7b as an example. The following information is displayed each time you run the facebook/opt-6.7b directory.
/path/model/facebook/opt-6.7b' model weights not found in cache or outdated. Downloading from huggingface.co

llm = ff.LLM( "/path/model/model/facebook/opt-6.7b",cache_path="/path/model/model/facebook/opt-6.7b/half-precision")

When specifying the LLM, the weights_path and clean_cache parameters do not exist. Is there any parameter that can avoid this situation or other methods?

[jiazhihao]

@goliaro Can you take a look at this issue?

[goliaro]

Let me take a look!

[goliaro]

@1193749292 I just double checked, but I'm currently not running into the issue above on my machine. If you are still running into the issue, could you post the script that you are using to run the LLM, so I can help you debug the issue?

Btw, if you are using the instructions below:

llm = ff.LLM( "/path/model/model/facebook/opt-6.7b",cache_path="/path/model/model/facebook/opt-6.7b/half-precision")

two things to note are that:

• for the model name, unless you have previously downloaded the opt-6.7b model from huggingface, and you explicitly want to load the model from that local folder, you should use facebook/opt-6.7b rather than /path/model/model/facebook/opt-6.7b. After the first time, FlexFlow will automatically detect that the model is in the cache, and won't redownload
• for the cache folder, you don't need to specify a path, unless you do not want FlexFlow to use the standard caching path ~/.cache/flexflow. If you do decide to pass a cache_path, note that this should be the path to where all flexflow models, including weights, tokenizers, configs, etc... are stored. FlexFlow will automatically create subfolders for the current model. So if you pass cache_path="/path/model/model/facebook/opt-6.7b/half-precision", FlexFlow will look for the opt-6.7b weights in folder /path/model/model/facebook/opt-6.7b/half-precision/weights/facebook/opt-6.7b/half-precision/, for example. Noting that the folder does not exist, it will proceed to download and convert the weights again

In conclusion, my recommendation is to use:

llm = ff.LLM("facebook/opt-6.7b")

leaving the cache_path blank. Let me know if it still doesn't work!

[1193749292]

@goliaro Thank you very much for your answer. I executed the script in the same directory as flexflow and it worked.

And then I want to ask, is there a way to omit unnecessary output when executing a lot of things?

[goliaro]

@1193749292 We'll add a non-verbose inference mode soon. In the meantime, feel free to comment out the print statements that you don't need

[xinlong-yang]

@1193749292 We'll add a non-verbose inference mode soon. In the meantime, feel free to comment out the print statements that you don't need

hi, I wonder how to modify the cache path, I don't want cache model's weights in ~/.cache/flexflow, thanks!

[lockshaw]

hi, I wonder how to modify the cache path, I don't want cache model's weights in ~/.cache/flexflow, thanks!

Moved to flexflow/flexflow-serve#19.