Chain bronze and silver layer in same pipeline

[ravi-databricks]

Provide option while launching dataflowpipeline by which both bronze/silver can be launched under single dlt.

[ravi-databricks]

Once Direct publishing mode becomes PuPr then can be merged to release branch.

[ravi-databricks]

Use CLI to test this feature. We have added broze_silver layer option which will launch single dlt with bronze/silver layers.

[ravi-databricks]

Introducing new layer option as bronze_silver for invoking dataflowpipeline invoke_dlt_pipeline function.
This feature will only work if customer create DLT pipelines using Direct publishing mode feature which is currently in private preview.

While creating DLT-META pipeline please select Direct publishing mode then provide layer=bronze_silver

Launch DLT-META pipeline using CLI:
Integrated this feature in CLI which will have option if customer wants to launch bronze_silver dlt pipeline.

Step1: databricks labs dlt-meta onboard
Step2: databricks labs dlt-meta deploy

This will launch bronze and silver layer into single DLT pipeline as below: