adhoc jobs block other jobs from being processed in the queue

[jchilcher-godaddy]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

When you submit an adhoc job all other jobs (other adhoc, and job templates) are stuck in a pending state until the running adhoc job completes.

AWX version

23.4.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

Firefox

Steps to reproduce

1. submit any adhoc job (I used a shell command sleep 30; whoami)
2. submit a job from "Templates"
3. observe the job queue

Expected results

all jobs in the unified job queue should process according to capacity. I expect to see more than one job in a running state.

Actual results

The adhoc job is in a running state. All other jobs are stuck in pending waiting for the adhoc command to finish.

Additional information

I am using a container group to run jobs.
Capacity of the managed nodes have been visually verified to be < 25% cpu and mem usage.
I tried creating a duplicate container group for job templates, which still was stuck in pending, but now with a message waiting for adhoccommand-371 to finish

[AlanCoding]

Are all jobs blocked, or only those running in the same inventory as the ad hoc command? Because if that's the case, then this is expected behavior.

awx/awx/main/scheduler/dependency_graph.py

Lines 88 to 90 in d8a28b3

	def job_blocked_by(self, job):
	project_block = self.get_item(self.PROJECT_UPDATES, job.project_id)
	inventory_block = self.get_item(self.INVENTORY_UPDATES, job.inventory_id)

[jchilcher-godaddy]

adhoc jobs appear to block only when running jobs from the same inventory, yes. I had no idea that was expected. I assumed that a job was a job and they could all be run simultaneously (excluding the inventory/project updates).

Seems like a fair workaround would be to duplicate the inventory and use one for tempates and the other for adhoc jobs, does that sound about right?

[AlanCoding]

Discussion has come up before, and we would prefer to allow these rules to be fully user-customizable, @chrismeyersfsu was a particular advocate for this. There may be some existing related issues. The dependency_graph.py contains most of this type of blocking logic, which I would prefer to call "exclusion rules" to distinguish from other types of dependencies like update-on-launch dependencies. As you can see, that file isn't terribly complicated, but I don't quite know how to put that in a language where a user could customize it with a dictionary-valued setting, for example.

[fosterseth]

I think we could change the way we mark adhocommands so that they aren't treated as inventory updates. Instead we can introduce a mark_adhoccommand method. We should still block on inventory updates though.

awx/awx/main/scheduler/dependency_graph.py

Line 141 in d8a28b3

self.mark_inventory_update(job)

[imDarshanGK]

Hey, is this issue still open? I can work on this if needed..
Also saw #14645