Integrate Stripe and Kafka Using Webhook API

In this post, we will show how to push Stripe events to Apache Kafka using Upstash Webhook API. Once data is in Kafka there are many use cases how you can use it:

• Processing payment events (with Apache Flink or Spark) to notify your business and sales team.
• Trigger processes such as sending Slack messages or email if a payment fails.
• Using a Kafka connector to move the data to a database or data warehouse to feed reporting and analytics applications.
• Feeding new payment activity data to your CRM.

Upstash Webhook API

Upstash Kafka has an HTTP based Webhook API which pushes the incoming requests’ payload to the Kafka. Thanks to the Webhook API, you do not need to write code to move data and you do not need a separate connector setup.

Step 1: Kafka Setup

If you don’t already have an Upstash account, you can sign up here. You’ll see the Upstash console afterward. Select the Kafka tab at the top. Now, you need a Kafka cluster. Click the Create Cluster:

Create a topic in the second page of the form:

You will see the Kafka Details page:

Scroll down to the Webhook section. You will need the URL in the next step.

Step 2: Stripe Webhook Setup

Sign in to your Stripe account, then click the Stripe Webhook page (Developers > Webhooks). Click the Add an endpoint button.

Now, you need to enter the webhook URL copied from Upstash Console. The url should have topic, user and pass parameters. You may want to change the topic parameter if you have more than one topic.

You can select which events to be sent. Then click on Add endpoint and you are done.

Step 3: Testing the Webhook

Now we have the pipeline, we can test it. Copy the curl command for Consumer REST API from the Upstash console. Run it on your terminal, it should output an empty message. Then add a new credit card to your Stripe account (or delete one) and run the curl command again. You should see the Stripe event is saved to Kafka topic.

curl https://full-mantis-14289-us1-rest-kafka.upstash.io/consume/GROUP_NAME/GROUP_INSTANCE_NAME/stripe-events -u \
  ZnVsbC1tYW50aXMtMTQyODkMiW63hFljqUrrq5_yAq4TPGd9c6JbqfQ==
[ {
  "topic" : "github-events",
  "partition" : 0,
  "offset" : 6,
  "timestamp" : 1643147778252,
  "key" : "",
  "value" : "{YOUR_STRIPE_EVENT}",
  "headers" : [ {
    "key" : "Stripe-Signature",
    "value" : "t=16477,v1=2457820fdebbb77e5fc8563"
  } ]

curl https://full-mantis-14289-us1-rest-kafka.upstash.io/consume/GROUP_NAME/GROUP_INSTANCE_NAME/stripe-events -u \
  ZnVsbC1tYW50aXMtMTQyODkMiW63hFljqUrrq5_yAq4TPGd9c6JbqfQ==
[ {
  "topic" : "github-events",
  "partition" : 0,
  "offset" : 6,
  "timestamp" : 1643147778252,
  "key" : "",
  "value" : "{YOUR_STRIPE_EVENT}",
  "headers" : [ {
    "key" : "Stripe-Signature",
    "value" : "t=16477,v1=2457820fdebbb77e5fc8563"
  } ]

Conclusion

In this post, we managed to integrate Stripe to Upstash Kafka without writing code and running a connector. This is possible for all services which have a webhook support to notify external services.