Add API flag for whether an AQL queue should be allocated in device memory.

[benvanik]

In 3baaa6e @saleelk added the ability to allocate an AQL queue in device memory via the HSA_ALLOCATE_QUEUE_DEV_MEM environment variable. I'd like to be able to use this programmatically with a queue creation flag as some of my queues are better in system memory and others in device memory (particularly those where I'm producing for the queue on the same device consuming it). In addition, as a middleware layer I don't control the environment and options that are set via them are generally not accessible.

hsa_queue_create doesn't seem to have a flags arg and I'm not sure of the best route for adding one such that creation could be controlled. Maybe an hsa_amd_queue_create?

[saleelk]

yeah, I got dragged elsewhere and sort of paused on this end. Yes, we would need a new API that takes flags. I had a conversation with ROCr owner about this, but need to start that over again

[benvanik]

That'd be wonderful! I haven't yet gotten my code working with it as I'm getting GPU faults when HSA_ALLOCATE_QUEUE_DEV_MEM=1, but that's likely my issue (or an issue with some of my queues needing to be in host memory and some in device memory). Thanks for pushing on this: I'm round-tripping through system memory for queue operations and it's taking an eternity :)

[saleelk]

i've looped you in on an internal chat

[atgutier]

@saleelk or @dayatsin-amd are either of you taking a look at this? If not, I can create a PR for this.

[benvanik]

Related to this, currently the amd_queue_t ends up in host memory and only the ringbuffer is going into device memory with the flag enabled. The queue struct is hot (write_dispatch_id/read_dispatch_id, etc) and we want that colocated with the ringbuffer in device memory as well.