Support --pipeline=/path/to/pipeline.yaml with a "direct router"

[markmc]

Currently --pipeline takes an alias or a directory, resolving to a directory containing knowledge.yaml, grounded_skills.yaml, and freeform_skills.yaml and the appropriate pipeline is chosen based on the data provided (aka "taxonomy routing"

There is a use case where, during experimentation, the user may want to run a specific pipeline directly

See instructlab/dev-docs#109 (comment) where we discussed a "router" design for this use case

This would allow ilab data generate support the starting point for data generation tasks use case described in aakankshaduggal@61eff48 rather adding a new entrypoint

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[bbrowning]

We will want to support directly specifying the pipeline to run (as opposed to a folder). Keeping this from going stale.

[bbrowning]

We will expose an API to run a single pipeline, and likely a CLI within SDG itself to do the same. However, that does not mean we'll necessarily do an ilab data generate --pipeline=/path/to/pipeline.yaml as that discussion would take place in the instructlab/instructlab repository once we have an API to run a single pipeline within SDG.