Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SiS compatibility for SMG app #181

Merged
merged 71 commits into from
Oct 18, 2024
Merged
Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
32ced90
first draft
sfc-gh-hbrown Sep 24, 2024
7dabbb1
Delete .DS_Store
sfc-gh-jsummer Sep 25, 2024
558874d
Add support for OSS connection with env vars or st.connection toml; u…
sfc-gh-jsummer Sep 25, 2024
c78ec2b
Reconfigure environment.yml for SiS to be tested with working SiS imp…
sfc-gh-jsummer Sep 26, 2024
2328bf1
Reconfigure connection creation with env vars
sfc-gh-jsummer Sep 26, 2024
2b1e053
Update height to integer
sfc-gh-jsummer Sep 26, 2024
f719f2f
Remove unnecessary snowflake qualification
sfc-gh-jsummer Sep 26, 2024
46454f8
Remove unnecessary qualification
sfc-gh-jsummer Sep 26, 2024
b997529
Remove database and schema context setting
sfc-gh-jsummer Sep 26, 2024
791b722
Consolidate key session state vars from env vars
sfc-gh-jsummer Sep 26, 2024
bf55082
Remove SiS rqmt for user name and add type hints
sfc-gh-jsummer Sep 27, 2024
d9ba20b
Add new Code UI
sfc-gh-hbrown Sep 27, 2024
c1cc5b8
Merge branch 'PORT_WORK' of https://github.com/Snowflake-Labs/semanti…
sfc-gh-hbrown Sep 27, 2024
042d51f
Remove extraneous env vars set for various connections
sfc-gh-jsummer Sep 27, 2024
945d33c
Delete .DS_Store
sfc-gh-jsummer Sep 27, 2024
b551944
Merge remote-tracking branch 'origin/PORT_WORK' into PORT_WORK
sfc-gh-jsummer Sep 27, 2024
7e6c5f4
Update yaml_editor
sfc-gh-jsummer Sep 27, 2024
fac48a3
Add scaffolding for directory restructuring
sfc-gh-jsummer Sep 27, 2024
b8ee9b1
Set host_name for all connections
sfc-gh-jsummer Sep 27, 2024
1db3838
Add session creation
sfc-gh-jsummer Sep 27, 2024
6b7cc63
Replace stage upload/download with snowpark
sfc-gh-jsummer Sep 27, 2024
80f2c35
Remove commented blocks
sfc-gh-jsummer Sep 27, 2024
9cc9ee4
Update environment.yml
sfc-gh-hbrown Sep 27, 2024
c55ed9d
Initial overhaul commit
sfc-gh-jsummer Sep 28, 2024
7c4bd04
Initial SiS deployment commit
sfc-gh-jsummer Sep 30, 2024
5757721
Reorg project structure and fix VQR flow
sfc-gh-jsummer Oct 1, 2024
3d9a739
Move shared_utils into sibling dir and remove unsupported column alig…
sfc-gh-jsummer Oct 1, 2024
d42b806
Remove temp file var
sfc-gh-jsummer Oct 1, 2024
5c51f6d
Add links and update partner sections for dbt SQL models
sfc-gh-jsummer Oct 1, 2024
75f08ec
Update streamlit deploy for snowcli 3.0
sfc-gh-jsummer Oct 1, 2024
85c7ca5
Add strictyaml
sfc-gh-jsummer Oct 1, 2024
1e3505f
Merge remote-tracking branch 'origin/main' into PORT_WORK
sfc-gh-jsummer Oct 1, 2024
8f0ac58
Continue rebase work
sfc-gh-jsummer Oct 1, 2024
2b66876
Fix downgraded streamlit errors
sfc-gh-jsummer Oct 1, 2024
1a559ce
fully qualify information_schema query
sfc-gh-cnivera Oct 1, 2024
305d3a1
maintain join state
sfc-gh-cnivera Oct 1, 2024
f187889
revert use database
sfc-gh-cnivera Oct 1, 2024
1ce0360
Qualify show tables and views with db name
sfc-gh-jsummer Oct 2, 2024
41067a1
Qualify all model generation functions
sfc-gh-jsummer Oct 2, 2024
20d3c7e
Add package support for looker
sfc-gh-jsummer Oct 2, 2024
53eff4e
Add full snowCLI deployment instructions and resolve incompatibility …
sfc-gh-jsummer Oct 2, 2024
1d57458
Fix API in SiS
sfc-gh-jsummer Oct 2, 2024
5afc4f5
Add stage upload functionality for dbt upload in SiS
sfc-gh-jsummer Oct 3, 2024
bca8e09
Fix typo
sfc-gh-jsummer Oct 3, 2024
34f9282
Add open flag
sfc-gh-jsummer Oct 3, 2024
d0cac6d
Complete functional support for partner integrations
sfc-gh-jsummer Oct 3, 2024
b0ed92e
Revise setup instructions
sfc-gh-jsummer Oct 3, 2024
1f5ecfc
Add non-cli setup for SiS
sfc-gh-jsummer Oct 3, 2024
89ea283
Rename deployment sections
sfc-gh-jsummer Oct 3, 2024
39c03e8
Add instructions for python 3.8-3.11
sfc-gh-jsummer Oct 3, 2024
3611006
Fix typos
sfc-gh-jsummer Oct 3, 2024
ba90ebb
fix test
sfc-gh-cnivera Oct 3, 2024
003f1f5
fix usage
sfc-gh-cnivera Oct 3, 2024
a9140d3
regen poetry
sfc-gh-cnivera Oct 3, 2024
ec5ab48
Align pyproject.toml with environment.yml and regen poetry.lock
sfc-gh-jsummer Oct 3, 2024
f4d44d4
Fix import
sfc-gh-jsummer Oct 4, 2024
412820e
Adjust yaml_editor fonts to mirror code
sfc-gh-jsummer Oct 4, 2024
4cad76c
Merge remote-tracking branch 'origin/main' into PORT_WORK
sfc-gh-jsummer Oct 4, 2024
ae545e5
Remove directory change in snowCLI deployment
sfc-gh-jsummer Oct 4, 2024
d1a0fb7
Add notes to require SiS Python 3.9+ enablement and snowCLI version 3.0+
sfc-gh-jsummer Oct 7, 2024
3a1c5fe
Mitigate pycache PUT for protos
sfc-gh-jsummer Oct 10, 2024
f1e9b80
Merge remote-tracking branch 'origin/main' into PORT_WORK
sfc-gh-cnivera Oct 14, 2024
4808930
fix lint
sfc-gh-cnivera Oct 14, 2024
41afc01
touchups
sfc-gh-cnivera Oct 14, 2024
dcbd336
fix tests
sfc-gh-cnivera Oct 14, 2024
fe52031
fix deps
sfc-gh-cnivera Oct 15, 2024
0fa5a4e
Addd one-liner to open app from terminal in SiS
sfc-gh-jsummer Oct 15, 2024
f53f52a
Fix dbt download from stage
sfc-gh-jsummer Oct 15, 2024
b097a89
Cleanup looker instructions
sfc-gh-jsummer Oct 15, 2024
e7cf8a8
catch empty name and tables
sfc-gh-cnivera Oct 16, 2024
89171d7
Merge remote-tracking branch 'origin/main' into PORT_WORK
sfc-gh-cnivera Oct 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
* @sfc-gh-cnivera @sfc-gh-jsummer @sfc-gh-twhite
/admin_apps/ @sfc-gh-cnivera @sfc-gh-jsummer
/semantic_model_generator/ @sfc-gh-nsehrawat @sfc-gh-rehuang @sfc-gh-dasilva @sfc-gh-cnivera @sfc-gh-yayin
4 changes: 1 addition & 3 deletions .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ name: Semantic Model Format & Lint

on:
pull_request:
paths:
- "semantic_model_generator/**"
- "admin_apps/**"
- "*"

jobs:
build:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Local python environment
pyvenv
*.DS_Store

# Notebook intermediate state
*.ipynb_checkpoints
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ setup: check-deps shell ## Install dependencies into your poetry environment.

# app
run_admin_app:
python -m streamlit run admin_apps/app.py
python -m streamlit run app.py

# Installs dependencies for the admin app.
setup_admin_app:
Expand Down
295 changes: 88 additions & 207 deletions README.md

Large diffs are not rendered by default.

42 changes: 0 additions & 42 deletions admin_apps/README.md

This file was deleted.

Empty file removed admin_apps/journeys/__init__.py
Empty file.
53 changes: 19 additions & 34 deletions admin_apps/app.py → app.py
Original file line number Diff line number Diff line change
@@ -1,80 +1,64 @@
import streamlit as st
from snowflake.connector import DatabaseError
from snowflake.connector.connection import SnowflakeConnection

# set_page_config must be run as the first Streamlit command on the page, before any other streamlit imports.
st.set_page_config(layout="wide", page_icon="💬", page_title="Semantic Model Generator")

from admin_apps.shared_utils import ( # noqa: E402
from app_utils.shared_utils import ( # noqa: E402
GeneratorAppScreen,
get_snowflake_connection,
set_sit_query_tag,
set_account_name,
set_host_name,
set_user_name,
set_streamlit_location,
set_snowpark_session,
)
from semantic_model_generator.snowflake_utils.env_vars import ( # noqa: E402
SNOWFLAKE_ACCOUNT_LOCATOR,
SNOWFLAKE_HOST,
SNOWFLAKE_USER,
assert_required_env_vars,
)


@st.dialog(title="Setup")
def env_setup_popup(missing_env_vars: list[str]) -> None:
"""
Renders a dialog box to prompt the user to set the required environment variables.
Args:
missing_env_vars: A list of missing environment variables.
"""
formatted_missing_env_vars = "\n".join(f"- **{s}**" for s in missing_env_vars)
st.markdown(
f"""Oops! It looks like the following required environment variables are missing: \n{formatted_missing_env_vars}\n\n
Please follow the [setup instructions](https://github.com/Snowflake-Labs/semantic-model-generator?tab=readme-ov-file#setup) to properly configure your environment. Restart this app after you've set the required environment variables."""
)
st.stop()


@st.dialog(title="Connection Error")
@st.experimental_dialog(title="Connection Error")
def failed_connection_popup() -> None:
"""
Renders a dialog box detailing that the credentials provided could not be used to connect to Snowflake.
"""
st.markdown(
f"""It looks like the credentials provided for `{SNOWFLAKE_USER}` could not be used to connect to the account `{SNOWFLAKE_ACCOUNT_LOCATOR}` at host `{SNOWFLAKE_HOST}`. Please verify your credentials in the environment variables and try again."""
f"""It looks like the credentials provided could not be used to connect to the account."""
)
st.stop()


def verify_environment_setup() -> None:
def verify_environment_setup() -> SnowflakeConnection:
"""
Ensures that the correct environment variables are set before proceeding.
"""
missing_env_vars = assert_required_env_vars()
if missing_env_vars:
env_setup_popup(missing_env_vars)

# Instantiate the Snowflake connection that gets reused throughout the app.
try:
with st.spinner(
"Validating your connection to Snowflake. If you are using MFA, please check your authenticator app for a push notification."
):
get_snowflake_connection()
return get_snowflake_connection()
except DatabaseError:
failed_connection_popup()


if __name__ == "__main__":
from admin_apps.journeys import builder, iteration, partner
from journeys import builder, iteration, partner

st.session_state["sis"] = set_streamlit_location()

def onboarding_dialog() -> None:
"""
Renders the initial screen where users can choose to create a new semantic model or edit an existing one.
"""

# Direct to specific page based instead of default onboarding if user comes from successful partner setup
if (
st.session_state.get("partner_setup", False)
and st.session_state.get("partner_tool", None) == "looker"
):
builder.show()
st.markdown(
"""
<div style="text-align: center;">
Expand Down Expand Up @@ -115,12 +99,13 @@ def onboarding_dialog() -> None:
)
partner.show()

verify_environment_setup()
conn = verify_environment_setup()
set_snowpark_session(conn)

# Populating common state between builder and iteration apps.
st.session_state["account_name"] = SNOWFLAKE_ACCOUNT_LOCATOR
st.session_state["host_name"] = SNOWFLAKE_HOST
st.session_state["user_name"] = SNOWFLAKE_USER
set_account_name(conn, SNOWFLAKE_ACCOUNT_LOCATOR)
set_host_name(conn, SNOWFLAKE_HOST)
set_user_name(conn, SNOWFLAKE_USER)

# When the app first loads, show the onboarding screen.
if "page" not in st.session_state:
Expand Down
Binary file added app_utils/looker_sdk.zip
Binary file not shown.
Loading
Loading