add var for CALL + update way we disable source freshness

[fivetran-jamie]

PR Overview

This PR will address the following Issue/Feature:

• [Feature] Add using_twilio_call flag #4 by way of merging Feature: add flag for call models #5
• [Feature] Update collect_freshness config in src_twilio.yml to account for dbt macro logic #3

This PR will result in the following new package version:

v0.1.1

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Features

• Added the ability to disable models related to the CALL and MESSAGE source tables. Refer to the README for more details (PR #5).

Under the Hood

• Adjusted the case of loader name in in src_twilio.yml (PR #5).
• Adjusted the way we dynamically disable source freshness tests for tables that may be missing to use dbt's native config.enabled flag (PR #6).
• Updated the pull request templates (PR #6).
• Included auto-releaser GitHub Actions workflow to automate future releases (PR #6).

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt run –full-refresh && dbt test
• dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• The appropriate issue has been linked, tagged, and properly assigned
• All necessary documentation and version upgrades have been applied
• docs were regenerated (unless this PR does not include any code or yml updates)
• BuildKite integration tests are passing
• Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

@raphaelvarieras confirmed dbt build -s package:twilio_source was successful, but i'll also validate given i made additional updates:

using_twilio_call Addition
By default, the call models still run:

After setting using_twilio_call: False in my dbt_project.yml, call models are not executed:

Freshness Update
On main (with functional but non-native code for disabling sources), dbt source freshness yields 3 runs (note that i am using stale data)

adding using_twilio_call: false to my dbt_project.yml makes it so that dbt source freshness yields 3 runs, but now with the call test passing (which is expected from our custom freshness macro)

On this branch, dbt source freshness without variable configs still gives us 3 failing tests

And after setting using_twilio_call: false, we only see 2 source freshness tests being run now that dbt has natively disabled the call source

If you had to summarize this PR in an emoji, which would it be?

📞

[fivetran-jamie]

question for you @raphaelvarieras -- i'm curious about the using_twilio_message=False use case, as i realized that all of the models in the twilio transform package are dependent on message data.

have you found yourself or others to use Twilio data without any messages?

[raphaelvarieras]

have you found yourself or others to use Twilio data without any messages?

No, not me or anybody I know. I added this in reaction of the commit from @fivetran-reneeli from last year where they added a meta is_enabled flag on the YAML definition of that source. I figured the intention was to cover this model as well with these flags at some point. I agree the dbt_twilio package itself would not be much use without this model though.

[fivetran-jamie]

@raphaelvarieras cool yeah I see, then I may revert the addition of using_twilio_message specifically since the dbt_twilio package depends on it so heavily

[fivetran-joemarkiewicz]

Great work on this PR @fivetran-jamie! I just have a few small suggestions and one main one to consider making this a minor upgrade instead of a patch. Let me know if you have any questions.

[fivetran-joemarkiewicz]

One small request to update the CHANGELOG. Otherwise LGTM!

[fivetran-joemarkiewicz]

Can you add the red lights here to bring attention to the breaking change

[fivetran-jamie]

added!

[fivetran-avinash]

A few small notes before approving

[fivetran-avinash]

Do you know why we're capitalizing MESSAGE here? The source table is message in the dbt_project.yml. Is it because the customer is using Snowflake?

[fivetran-jamie]

just capitalizing it to highlight it as a source table (as opposed to a package model or a column)

[fivetran-avinash]

Similar question regarding CALL vs. call.

[fivetran-jamie]

just a stylistic choice - i tend to capitalize raw source tables in documentation to highlight that they're not package models or columns

[raphaelvarieras]

I just noticed another issue and I wonder if we want to try to address it in this release maybe? I noticed today that the NUM_SEGMENTS and NUM_MEDIA columns are VARCHAR in the MESSAGE table instead of NUMBER. Should we consider adding a cast to number in the staging models?

[fivetran-jamie]

@raphaelvarieras that is a good callout, but would you mind creating a new issue for that? I'd prefer to tackle in it another release to ensure we have ample time to figure out each field we'd want to (and logically should) cast as an integer, as I see that other tables have quite a few numeric fields cast as strings (ex: duration and price in CALL and count and usage in USAGE_RECORD)