[Regression] Sporadic "cannot open relation with oid ###" error

[mikealfare]

Is this a regression in a recent version of dbt-redshift?

• I believe this is a regression in dbt-redshift functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

When executing a dbt run with multiple models using multiple threads, I sometimes receive an error message like:

Database Error
  could not open relation with OID 123456789

I can resolve it by rerunning the failed models manually. I can also reduce how often I need to run this manually by increasing the retries parameter.

Expected/Previous Behavior

Prior to moving to the Redshift driver, I did not see this error. I would expect the same behavior with the Redshift driver.

Steps To Reproduce

Create several models and run them in parallel. The more threads and the smaller the models, the more likely you'll see it. Because it's not deterministic, you may not see it every time.

Relevant log output

No response

Environment

- OS: any
- Python: any
- dbt-redshift (working version): Postgres driver versions (<1.5)
- dbt-redshift (regression version): Redshift driver versions (1.5+)

Additional Context

I believe this is happening because the Redshift driver does not handle multi-statement SQL as a single transaction whereas the Postgres driver does handle multi-statement SQL as a single transaction. While the Redshift approach complies with the DBAPI 2.0 spec (and the Postgres approach does not), this adapter was built with the assumptions associated with the Postgres driver. When we updated from Postgres to Redshift, we did not account for all of the places where the Postgres-based assumptions no longer hold.