We just published our first Snowflake Native app — Twing Pulse — to the Snowflake Marketplace! It features a few of the most useful queries from our original much-loved product, Twing Data. Now Snowflake users can access a portion of our useful warehouse and workload metrics instantly, with none of the friction or security concerns that come with external integrations. It’s also free for now, so check it out and let us know what you think!
Fresh after the experience from 0 to 1, we thought it would be helpful to condense the mental model, development processes we used, and submission/review processes into a guide for those who are also interested in building their own Snowflake app.
Mental Model
Published Native Apps are run directly on the consumer account. When the app code is ready to publish, the developer generates or updates an application package based on that code. Then that package is published either privately or to the marketplace. Publishing requires three review processes, which we’ll cover later on.
In development, it’s possible to create an application for testing directly from the application package, without publishing. Here is the flow I have been using:
Initial Setup
The file tree below includes everything you need to get a repo started:
On the Snowflake side, it goes without saying you’ll need a Snowflake account and, depending on the type of app you’re building, some data to test with.
Because testing within Snowflake with data was such an important part of my development workflow, I set up an integration with GitHub from the outset. Here are the queries you’ll need to set up a Snowflake secret, use it to create an API integration and a Snowflake Git Repository:
Next, pull your repo in from GitHub and verify:
Finally, build your application package and application from the git repo. Make sure to build using the path to your app if not at the root of the repo:
Required Components
setup_script.sql: A script that creates objects within the application object that are required by the app. It runs when a consumer installs or upgrades the app, or when the provider creates/upgrades the application object for testing the package.
manifest.yml: Contains configuration information and the setup_script.sql path, to be used by the application package to create and manage a Snowflake app.
environment.yml (required if using Streamlit): Lists dependencies and versions used within the app. The channel must be set to Snowflake, and it’s recommended to specify Streamlit versions here.
Optionally, you might also consider a marketplace.yml file to prevent the user from ending up with an unusable app by validating region resources before install/upgrade. It also adds resource requirements to the marketplace listing.
Permission Management
“As per our enforced requirements, all account-level privileges and references listed in the application package manifest file must be requested from the consumer through Snowsight or the Python Permission SDK. The account level privileges listed in the manifest should be requested via the Permissions SDK in a Streamlit. You can do this by following these steps. ”
Snowflake Marketplace Operations Rep (via submission review email)
Certain app functionality requires explicit privileges to be granted to the app by the consumer, such as reading or modifying consumer data. These privileges must be listed in the manifest, and also be requested by the app. The Permissions SDK is used to easily embed permission requests in the Streamlit UI. Here’s a function you can call directly within your Streamlit entrypoint file. Just be sure to import Snowflake.permissions as permissions first.
Application Versioning
Snowflake Native Apps support versioning through application packages, allowing developers to maintain and upgrade apps safely. Best practices include keeping at most two active versions at a time, using versioned schemas for stateless code and regular schemas for persistent data, and ensuring backward compatibility between versions. Release directives can be used to control which version consumers install, and upgrades should be coordinated carefully to avoid disrupting active users.
Frontend Development: Streamlit
While it’s not technically required for a Native App to have a UI, we opted for one. In Snowflake Native, that means Streamlit. Now, Streamlit is great for what it is, allowing devs to quickly spin up simple frontends for demoing or internal uses. When the goal is to expose a feature-rich product for a marketplace, Streamlit is not the tool I’d usually reach for. Additionally, Snowflake Native overrides or simply does not support many of Streamlit’s features. Some are documented, and some aren’t. The documented list is linked at the end of the article. It’s worth reviewing before writing any code, as it will likely change the approach to the application or even affect its feasibility entirely.
A few of the undocumented issues I encountered:
Background and element styling: Background styling just does not work. I assume this is due to Snowflake’s own internal light/dark theming. Styling on other elements may or may not stick. Lastly, text color is something that shouldn’t be touched because Snowflake’s light/dark themes include appropriate contrasting text colors. If they’re changed, those colors will be static and must be tested on both light and dark backgrounds for visibility.
Streamlit’s st.navigation doesn’t work, so for a multipage app the old /pages approach must be used. This means nested, folder-based navigation is not possible.
Backend Development
The current backend for Twing Pulse is a set of queries set within the app/code_artifacts/Streamlit/libraries directory. To leverage these queries to return data, we use Snowpark sessions, which can be created/retrieved like this:
The session returned from this function is used throughout the app to run queries and return data in pandas dataframes like this:
The current version of Twing Pulse gets slightly more complex from here with query parsing and recursive traversal logic used for query pattern detection, for example. But much deeper complexity is available, from native user defined functions (UDFs) and stored procedures, all the way to external functions, which are developed, maintained, stored, and executed outside of Snowflake’s environment. We’re early in our journey in Snowflake Native, and our exploration is ongoing. Perhaps as Native is developed further and we continue to explore, we will someday use external functions to natively connect our fully-featured Twing Data product for select clients.
Publishing
Once the app’s ready for the world, it’s time to publish. If the app is for internal use only, publishing offers systematic versioning and streamlined distribution throughout the org, as well as simpler access control. And, obviously, if it’s destined for the public marketplace, publishing is required. Public publishing has some additional steps, so I’ll cover this pathway here.
If you recall from the Mental Model section, it was shown that the application package, and not the application itself, is what is distributed to consumers. The downstream application that was used for development is just for that, and has nothing to do with the publishing process. From here we’ll only be referring to the application package itself.
The next step is to set package distribution to external:
‘ALTER APPLICATION PACKAGE <package_name> SET DISTRIBUTION = EXTERNAL;’
This will likely prompt you to accept some terms of service. Going forward, with this setting, every time the application package is updated, an automatic security scan kicks off. Completion on our app typically took several seconds to several minutes. Security scan status is located in the review_status column in the results from the query:
‘SHOW VERSIONS IN APPLICATION PACKAGE <package_name>;’
If the current version/patch shows ‘REJECTED’, you have some work to do. Ours failed many times during development, and it was never clear why or where the error logs were located. Luckily removing a few lines from the startup_script.sql file corrected the issue for us. Worst case scenario, the docs say that upon failure of the automatic review, the package goes into manual review, which can take days to complete.
When the review_status column starts saying, ‘ACCEPTED’, you’re in the clear to proceed.
Next, you must set default and, if necessary, custom release directives for the app package. A release directive represents the version of the application that’s available to consumers. Custom release directives can be assigned per consumer if desired. You’ll find those queries at the bottom of this useful worksheet screenshot showing all the queries you’ll need work through this process:
All versions to publish, defined by release directives, must pass the entire review process. In addition to the security scan, this involves an in-depth manual review upon publishing.
Now that we’re ready to submit for manual review, in the Snowsight UI, in the lefthand navbar, go to Data Products -> Provider Studio. In the top right you’ll see a blue button that says ‘Create Listing’. Here you can select whether to publish to the Internal or Public Marketplaces or for another specific consumer. Then name your app and select the type and availability:
Next, follow the prompts, assigning the intended application package from the list of packages with release directives, then filling in the details in the UI as required.
When all required sections and fields are filled in, the publish button at the top will become activated. Clicking that button will turn it yellow with a message signifying that it is under review.
Congratulations! The app is submitted!
The Review Process
The initial feedback took approximately 5 days to arrive. The review made clear that Snowflake is taking the process seriously. The issues they surfaced covered code, privilege permissions, and UI. While the message included many directives, there were also questions that indicated that they respected that the context affects the appropriateness of certain decisions.
Snowflake is prompt with replies after the process is started, and requires by policy that the publisher is also. I found the Snowflake contact to be warm, engaging, informative, and collaborative. A great first experience, to be sure!
That’s a Wrap
Developing for the Snowflake Native ecosystem makes a whole lot of sense for the right application. For instance, building a product destined to run within the isolated consumer account alleviates the need to worry about infrastructure. On the other hand, its intentionally-restrictive nature greatly limits the types of solutions possible, trading instead for greater security/governance.
We have a ways to go in exploring the full capabilities of the framework. While we’ve found it takes some rethinking and compromising to fit our existing approaches into this new model, we believe Snowflake Native is likely a simple way to natively expose our features to clients while protecting our IP. Twing Pulse in its current state is merely a test. As this thesis is validated, much more of our full product features will be on the way. If you’re a Snowflake user and are interested in deeper observability to connect how your system is actually used with Snowflake’s execution and pricing behaviors, give Twing Pulse a download. It’s free! All we ask is that you send some feedback to us so we can better prioritize the highest-impact improvements and feature migrations.
For an idea of what’s to come to Twing Pulse, reach out to us for demo links to our full platform at twingdata.com.
Have you developed for the Snowflake Native Marketplace? What was your experience? Let us know below!
Some Useful Docs Sources:
Snowflake unsupported Streamlit features: https://docs.Snowflake.com/en/developer-guide/native-apps/adding-Streamlit
https://docs.Snowflake.com/en/developer-guide/Streamlit/limitations#unsupported-Streamlit-features
Privileges:
https://docs.Snowflake.com/en/user-guide/security-access-control-privileges
Functions and Procedures:
https://docs.Snowflake.com/en/developer-guide/extensibility
External Functions:
https://docs.Snowflake.com/en/sql-reference/external-functions-introduction