Phi-3 mini support?

[vackosar]

Not the most powerful, but a useful model:

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

[casper-hansen]

Hi @vackosar, I am open to PRs from the community. For now, I will not have time to include it.

[vackosar]

@casper-hansen thank you. Is there any guidance on how to do that for architecture like Phi-3?

[TechxGenus]

I checked its architecture and it shouldn't be very hard to implement basic quantization. But its position encoding is special (LongRoPE) and implementing the fusion layer might need more work.

[vackosar]

It seems that the 4k version is the base and maybe uses standard Rope. That would reduce needed effort. Is there any way to test?

https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/tree/main

[casper-hansen]

This can easily be ported to AutoAWQ if someone has the time.
mit-han-lab/llm-awq#183

[vackosar]

Somebody implemented a merge request here:
#481

But it was not merged yet.