MI100 Support #24

LoggerHead22 · 2023-12-01T22:03:38Z

Hi, the documentation says that this implementation is compatible only with the MI200 and MI300 GPUs. But what about the MI100 gpu?

The code contains such conditions that formally match the MI100 with the gfx908 architecture.

bool is_gfx90x = dprops->major == 9 && dprops->minor == 0;
bool is_gfx94x = dprops->major == 9 && dprops->minor == 4;
TORCH_CHECK(is_gfx90x || is_gfx94x, "FlashAttention only supports AMD MI200 GPUs or newer.");

Will this code be compatible with MI100 in practice? If not, are there any plans to add such support? Or what are the reasons that keep you from adding support for the MI100?

The text was updated successfully, but these errors were encountered:

dejay-vu · 2023-12-05T09:36:07Z

Hi @LoggerHead22, this code appears to be a logic fault, thanks for noting.

We haven't tested the FA on MI100 since we did most of our testing on MI250&MI300 so we are limiting the support archs. I am not sure whether it will work correctly on MI100 but you can try by adding gfx908 to the valid archs. I suppose the building process will be fine.

LoggerHead22 · 2023-12-06T21:55:44Z

Thanks for the clarification @howiejayz . Your advice really helped, the code is compiled for mi100 and runs.

However, I encountered an error during the build, which is caused by the logic of the patch.

hipified_header_filepath = HIPIFY_FINAL_RESULT[header_filepath].hipified_path
      AttributeError: 'dict' object has no attribute 'hipified_path'

This seems logical, because a dict is being created here and then we try to take its hipified_path attribute.

Replacing dict with an object of the HipifyResult class in patch helped me.

ccbadd · 2024-01-11T15:09:26Z

Has this patch been merged to the main branch or do we need to apply it in order to test?

ehartford · 2024-01-16T07:11:53Z

I need mi100 support

ehartford · 2024-01-16T19:13:16Z

Hi @LoggerHead22, this code appears to be a logic fault, thanks for noting.

We haven't tested the FA on MI100 since we did most of our testing on MI250&MI300 so we are limiting the support archs. I am not sure whether it will work correctly on MI100 but you can try by adding gfx908 to the valid archs. I suppose the building process will be fine.

If you need hardware for testing mi100, I volunteer my server for this purpose. I have 8x mi100 with infinity fabric.

[email protected]

ehartford · 2024-01-22T21:08:58Z

Hi @sabreshao @howiejayz can you please give me a path forward?

I have a bunch of mi100s and I would like them to be hot. Without flash attention, I am blocked.

Maybe you could show me where in the code I would add it? give me some advice?

dejay-vu · 2024-01-23T10:11:13Z

Hi @LoggerHead22, this code appears to be a logic fault, thanks for noting.

We haven't tested the FA on MI100 since we did most of our testing on MI250&MI300 so we are limiting the support archs. I am not sure whether it will work correctly on MI100 but you can try by adding gfx908 to the valid archs. I suppose the building process will be fine.

Hi @ehartford! Currently I have no time to test FA on MI100 but could you try build and run based on this comment?

TNT3530 · 2024-01-25T14:23:35Z

I was able to compile flash attention for the MI100 using the docker image. Simply adding gfx908 to the target arch array (or in my case, removing everything BUT native and gfx908) makes it run fine. (Note: this also applies to the vLLM ROCm docker image, which was my use case)

Attempts to compile outside of docker seem to fail on ROCm 6.0 due to this issue, though I was unable to downgrade back to 5.7 to test on my machine.

luizanao · 2024-01-26T18:19:10Z

I managed to build MI100 (gfx908) as well but the env var didn't work @TNT3530 . This is because the setup is protected against unknown architectures and gfx908 is not listed. I will open a PR for adding that since gfx908 definitely works.

luizanao · 2024-01-26T18:34:16Z

Here's my PR, you folks might benefit from it: #38

ehartford · 2024-02-26T04:31:55Z

How do I install flash attention for mi100? How is the procedure from the README.md different?

luizanao · 2024-02-26T16:28:55Z

@ehartford passing the card arch to the build should be enough: export GPU_ARCHS="gfx908"

gittb · 2024-07-25T16:45:06Z

Also curious if support for Mi100 was finalized.

ehartford · 2024-07-25T17:44:24Z

This is awesome! Can't wait to try it!

jamestwhedbee · 2024-09-26T20:07:11Z

just realized Mi100 support was removed

jamestwhedbee · 2024-09-26T20:07:34Z

@jayz0123 was that intentional

IMbackK · 2024-11-03T20:36:11Z

I can confirm that when this is patched away again to allow mi100 to build the package, the latest main builds and works fine on gfx908 at least for the dimensions i tried. So this restriction seams pretty silly, and its quite puzzling why mi100 was removed from the array again given it still works fine.

ehartford · 2024-11-03T22:06:35Z

Then - someone doesn't want it to work on mi100

ehartford · 2024-11-03T23:05:00Z

I can confirm that when this is patched away again to allow mi100 to build the package, the latest main builds and works fine on gfx908 at least for the dimensions i tried. So this restriction seams pretty silly, and its quite puzzling why mi100 was removed from the array again given it still works fine.

Could you please make a PR that enables mi100 so I can test it?

ruboot · 2024-11-24T12:42:37Z

I can confirm that when this is patched away again to allow mi100 to build the package, the latest main builds and works fine on gfx908 at least for the dimensions i tried. So this restriction seams pretty silly, and its quite puzzling why mi100 was removed from the array again given it still works fine.

Could you please make a PR that enables mi100 so I can test it?

pytest test_flash_attn_ck.py
/usr/local/lib/python3.10/dist-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================================================================================ test session starts =============================================================================================
platform linux -- Python 3.10.12, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/power/shared/code/flash-attention/tests
configfile: pyproject.toml
plugins: asyncio-0.24.0, anyio-4.6.2.post1, typeguard-4.3.0
asyncio: mode=strict, default_loop_scope=None
collected 410996 items

test_flash_attn_ck.py ................................................................................................................................................................................ [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
...................................................................................................................................................................................................... [ 0%]
.............................................................................................................................Fatal Python error: Aborted

Thread 0x00007f15117fd640 (most recent call first):

Thread 0x00007f1511ffe640 (most recent call first):

Thread 0x00007f15127ff640 (most recent call first):

Thread 0x00007f15187ff640 (most recent call first):

Thread 0x00007f1c7f115000 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 91 in _flash_attn_forward
File "/usr/local/lib/python3.10/dist-packages/torch/_library/custom_ops.py", line 367 in wrapped_fn
File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 632 in _fn
File "/usr/local/lib/python3.10/dist-packages/torch/_compile.py", line 32 in inner
File "/usr/local/lib/python3.10/dist-packages/torch/_library/custom_ops.py", line 324 in backend_impl
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 721 in redispatch
File "/usr/local/lib/python3.10/dist-packages/torch/_library/autograd.py", line 40 in forward_no_grad
File "/usr/local/lib/python3.10/dist-packages/torch/_library/autograd.py", line 113 in autograd_impl
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116 in call
File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 458 in forward
File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 575 in apply
File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 1012 in flash_attn_qkvpacked_func
File "/home/power/shared/code/flash-attention/tests/test_flash_attn_ck.py", line 93 in test_flash_attn_qkvpacked
File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in call
File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 1627 in runtest
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 174 in pytest_runtest_call
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in call
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 242 in
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 341 in from_call
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 241 in call_and_report
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 132 in runtestprotocol
File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in call
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 362 in pytest_runtestloop
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in call
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 337 in _main
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 283 in wrap_session
File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 330 in pytest_cmdline_main
File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in call
File "/usr/local/lib/python3.10/dist-packages/_pytest/config/init.py", line 175 in main
File "/usr/local/lib/python3.10/dist-packages/_pytest/config/init.py", line 201 in console_main
File "/usr/local/bin/pytest", line 33 in

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 24)
Aborted (core dumped)

linchen111 · 2025-01-09T13:09:10Z

I can build it on MI100 ,but faied on FA test

sabreshao added the pre-mi200 label Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MI100 Support #24

MI100 Support #24

LoggerHead22 commented Dec 1, 2023

dejay-vu commented Dec 5, 2023

LoggerHead22 commented Dec 6, 2023

ccbadd commented Jan 11, 2024

ehartford commented Jan 16, 2024

ehartford commented Jan 16, 2024

ehartford commented Jan 22, 2024

dejay-vu commented Jan 23, 2024

TNT3530 commented Jan 25, 2024 •

edited

Loading

luizanao commented Jan 26, 2024

luizanao commented Jan 26, 2024 •

edited

Loading

ehartford commented Feb 26, 2024 •

edited

Loading

luizanao commented Feb 26, 2024 •

edited

Loading

gittb commented Jul 25, 2024 •

edited

Loading

ehartford commented Jul 25, 2024

jamestwhedbee commented Sep 26, 2024

jamestwhedbee commented Sep 26, 2024

IMbackK commented Nov 3, 2024 •

edited

Loading

ehartford commented Nov 3, 2024

ehartford commented Nov 3, 2024

ruboot commented Nov 24, 2024

linchen111 commented Jan 9, 2025

MI100 Support #24

MI100 Support #24

Comments

LoggerHead22 commented Dec 1, 2023

dejay-vu commented Dec 5, 2023

LoggerHead22 commented Dec 6, 2023

ccbadd commented Jan 11, 2024

ehartford commented Jan 16, 2024

ehartford commented Jan 16, 2024

ehartford commented Jan 22, 2024

dejay-vu commented Jan 23, 2024

TNT3530 commented Jan 25, 2024 • edited Loading

luizanao commented Jan 26, 2024

luizanao commented Jan 26, 2024 • edited Loading

ehartford commented Feb 26, 2024 • edited Loading

luizanao commented Feb 26, 2024 • edited Loading

gittb commented Jul 25, 2024 • edited Loading

ehartford commented Jul 25, 2024

jamestwhedbee commented Sep 26, 2024

jamestwhedbee commented Sep 26, 2024

IMbackK commented Nov 3, 2024 • edited Loading

ehartford commented Nov 3, 2024

ehartford commented Nov 3, 2024

ruboot commented Nov 24, 2024

linchen111 commented Jan 9, 2025

TNT3530 commented Jan 25, 2024 •

edited

Loading

luizanao commented Jan 26, 2024 •

edited

Loading

ehartford commented Feb 26, 2024 •

edited

Loading

luizanao commented Feb 26, 2024 •

edited

Loading

gittb commented Jul 25, 2024 •

edited

Loading

IMbackK commented Nov 3, 2024 •

edited

Loading