Building for AWS Marketplace
Twing Data is now available in the AWS Marketplace!
One of our recent blog posts detailed how and why we built a Snowflake marketplace application — but that wasn’t the only marketplace we’ve been building for. Twing Data has also made its way into the AWS Marketplace! This article outlines our experience with getting this integration set up, the technical approach we took, and technical challenges that we came across.
What is the AWS Marketplace?
Like the Apple Store or Google Play Store, The AWS Marketplace is a curated digital marketplace that allows third-party developers to sell different types of software and services such as interesting data sets, development containers, machine learning models, and in our case — SaaS.
The SaaS Marketplace operates similarly to other ecosystems that enable third-party developers to integrate payments and authentication through the platform, while increasing product visibility with a page highlighting the features of the product.
For the buyer, AWS Marketplace offers important advantages, like flexible pricing and contract options that allow for pre-defined terms (with different pricing models) to avoid surprises. There are also options to create private contracts that can be fine-tuned further for specific buyers. Another bonus is that billing is consolidated through AWS. Finally, there’s a lot of value in knowing that the application has been through AWS’s curation and quality assurance process, and abides by their terms. Different organizations have different compliance and billing processes, so being vetted by and billed through AWS helps streamline adoption.
Twing Data AWS Marketplace App: Technical Details
For those interested in the technical implementation of this integration, here’s an overview of how it works.
Our SaaS offering is classified as “Contract with Consumption” meaning that AWS facilitates a contract between Twing Data and a buyer, and Twing Data is responsible for tracking usage (specifically, number of Redshift queries analyzed) and reporting it to AWS for billing, in case of overages. A buyer can update their contract to increase their usage caps (but not decrease).
The first step to get started is to create the listing, following the official “Lab: Create a SaaS listing” guide.
The "Lab: Integrate your SaaS" guide was an invaluable technical resource when designing and building this out, but assumes your application runs on AWS Lambda. In our use case we had to adapt the guide to work with our Remix application (soon to be React Router) and asynchronous analysis (in Python).
The linked guide is organized into three parts:
Integrate with buyer onboarding
Get entitlement updates
Get subscription updates and report usage
Let’s go into some details on each one.
Integrate with buyer onboarding
The first part of the guide acted as a template for linking the buyer’s AWS account and their Twing Data account. More directly, AWS redirects the buyer to the landing page URL which makes a POST request to a new Twing Data app route. This request includes a “x-amzn-marketplace-token” which we can use to get the buyer’s AWS account information for later.
In classic frontend development fashion, there are a few factors to consider: Does the buyer already have a Twing Data account? Is the buyer logged in? If they’re not logged in, how do we make this link?
These questions are answered by using the token header to get the AWS information, and store it in our own database. If the user is logged in, we can link their AWS account to their Twing Data account immediately, otherwise we redirect the user login/signup and make the link after. In each case, we also initialize information about their contracts’ limitations and usages so far.
Get entitlement updates
For the Contract with Consumption pricing model, it’s necessary to listen for events from AWS about contract terms changing; for example when a subscription is created or updated with increased limits.
The guide walks you through setting up an email subscription for the SNS topic, but the production implementation is effectively left as an exercise for the reader. It can be difficult to debug so the less unknowns there are, the better. We created a new route in our Remix app to handle this event and set the new URL as the subscription endpoint (making sure to use the correct protocol and “www” subdomain if required – otherwise your endpoint won’t get messages!)
HTTPS subscriptions need to be “Confirmed” which is a one-time process that tells AWS the endpoint is set up properly. Since it’s a one-time thing, we simply did the “Manual confirmation” that the SNS documentation describes.
Ultimately, this endpoint receives the “entitlement-updated” notification, and uses the provided customer-identifier and product-code to see what entitlements were created or changed. Again, the SaaS guide walks through a manual workflow using the AWS cli, but it was straightforward to adapt to use the @aws-sdk/client-marketplace-entitlement-service nodejs package.
Get subscription updates
The last part of the SaaS guide, “Report usage to charge the buyer” also walks you through creating another subscription, for listening to the aws-mp-subscription-notification. It’s important enough on its own so I decided to treat it as its own section.
For the consumption-based pricing models (“Contract with Consumption” and “Subscription (pay-as-you-go)”, consumption is tracked while the subscription is active, so we need to track not only usage (which we’ll go through below) but also changes to the subscription status. For this, we created a new route for the aws-mp-subscription-notification topic, and followed a similar process as the previous section to update an AWS user’s subscription on our side.
Report usage
Lastly, we need to track usage (number of queries analyzed) so we can report usage to AWS for billing.
Since usage reports are expected to be no more frequent than hourly, (“Timestamps that fall into the same hour are truncated to that hour.”), the logic between “tracking” and “reporting” are decoupled – we track number of queries as they’re analyzed, and then send consumption reports as needed to make sure they’re accurate and avoid any getting dropped.
End-to-end testing
Testing this flow turned out to be a bit cumbersome due to having to “test in production” – the seller and buyer account is the same during testing so to test a new contract, for example, the previous one has to be canceled. As of this writing, this is only done through contacting the AWS Marketplace support team.
To alleviate this friction, the different flows are covered by our automated testing suite that runs on each pull request to ensure there are no regressions to this critical feature. Of course, automated testing is a good practice regardless, but having to wait for canceling test contracts takes time so this is practically a requirement to shorten this feedback loop (and as a bonus, alleviate the AWS Marketplace support team’s workload).
Since you’re “testing in prod”, you end up being billed, but the price can be set low (and is said to be refundable):
Give our App a Try
Similar to our case for building a Snowflake Native application, we want to meet our users where they are — Twing Data already supports Redshift, so the next step is to get that integration set up as easily and securely as possible.
Being on the AWS Marketplace simplifies procurement, accelerates deployment, and aligns with the compliance, billing, and security standards you already trust. If your team is looking for a faster, easier way to integrate Twing Data into your Redshift warehouse, the AWS Marketplace is a direct path forward.
Helpful Links
What is AWS Marketplace? (AWS article)
What is AWS Marketplace? (video)
"Lab: Integrate your SaaS" guide (primary guide referenced above)
AWS Marketplace - Twing Data for Redshift: Query Analysis & Optimization (our live marketplace listing)