How Guesty Uses Svix To Power Your Next Vacation

Guesty is the world's leading holiday property management platform. Used by their customers to manage short-term rentals on platforms such as Airbnb and Booking.com.

Guesty operates in over 80 countries, and has over 800 employees worldwide spread across 15 offices. Guesty's extensive product suite helps their customers manage booking, listings, invoicing, communication, and more; for customers managing anywhere from a single listing to thousands of listings.

We sat down with Aram Ben Shushan Ehrlich, the engineering group manager leading the webhooks project at Guesty, to talk about their experience moving from their homegrown system to Svix.

Guesty is sending tens of millions of webhooks a month with Svix, and having a robust webhooks has enabled their customers building powerful systems on top of their product, increased customer satisfaction, reduced support burden, and saved their best engineers a lot of time that they can now focus on the business instead.

Webhooks drive enterprise value

Guesty uses webhooks to programmatically notify their customers about every change that happens on the system. They send an event when a property is booked, a message is sent, invoice is paid, and hundreds of other types of events.

These events are then used by their customers to build automations or integrate with internal systems. While Every segment of their customer base benefits from the webhooks, the larger enterprise customers rely on and require it.

Our customers who use webhooks are those that bring in the money. They are often your best customers. This means that having great webhooks is especially important, and any friction in the onboarding or issues in production can have significant adverse impact on the business.

Guesty's webhooks are also used by all of their ecosystem partners to build on top of the Guesty platform. Webhooks are how these integrations work, making the Guesty platform more extensible and powerful for their customers, and enabling more business to build on top of Guesty.

Our large enterprise customers use our webhooks to ensure that both they and their system stay up to date about the status of their properties.

Guesty were wasting valuable resources on their homegrown webhooks system

Guesty understood the importance of webhooks from early on and have been offering webhooks to their customers since the beginning. They have invested significant time and resources maintaining their webhook system over the years, with some of the companies' best engineers working on their webhook system.

Or as Aram put it:

We wasted a lot of the time our best engineers, tech leads, and managers; thinking about, improving, and solving issues and challenges with our webhooks system.

Though even with all of the time and effort invested in their homegrown webhook system they still had a variety of gaps that made them spend a lot of time on it.

The problem was that you'd have random alerts and issues coming from your webhook system that would distract our best engineers and waste a lot of their valuable time. It didn't happen every week, but when it did happen it was very painful.

Aram continued:

Ever since we added Svix we are all at ease. No more retries, no more endpoint failures, no random webhook errors. We had our best engineers working on it, and they are now free to do other things.

Svix enabled Guesty to stop worrying about webhooks altogether by turning webhook delivery to a simple API and shifting the complexity to Svix.

Managing our own webhook system was a nightmare I'm glad to have left behind. As the manager of the team responsible for webhooks, this has been my personal source of pain and suffering.

Webhooks are harder than they seem

Guesty also experienced first-hand the same realization we hear from many of our customers: webhooks are harder than they seem.

Or as Aram put it:

Part of the challenge with webhooks is that you start small with something that looks achievable and you have a good understanding of. For example, you may think you can just setup a queue and start sending. This works well at first, but the problems start appearing once customers start adopting it.

He then continued:

All of a sudden you'll start getting traffic spikes which can lead to the system getting backed up, then you'll start experiencing noisy neighbor where one noisy customer is affecting the delivery of another. You'll also want to add retries at some point, but you'll sooner or later face a thundering herd problem, random HTTP issues, very slow servers that are blocking your workers and much more.

At Svix, we hear the same story time and time again. A customer builds a simple system initially, whether it's due to deadlines, lack of time to invest in the webhooks, or unfamiliarity with the long list of challenges; and then they discover all the hard lessons in production which starts an endless cycle of issues with the webhooks system and constantly patching it.

This means you constantly have to add new monitoring, new mechanisms, and new solutions for problems. This is especially true as you grow your traffic, there are just a lot of challenges that will catch you by surprise that you may not be aware of ahead of time.

It's just annoying and exhausting to see yet another random webhook lag, error, or support ticket pop up.

Additionally, these challenges are made much worse because most companies don't have a dedicated webhooks team that just works on the webhooks system 24/7. This means that the team suffers from constant whiplash getting reintroduced to the webhooks code and challenges every time a new issue or customer request arises.

This is something Aram and his team also noticed with their homegrown solution:

What is especially annoying about webhooks is that you sometimes get weird issues. For example, "Why is my queue 15x larger or slower all of a sudden?" and it's just very hard to always see exactly what's going on and which exact customer or what exact issue is causing this.

At some point you just don't want to have to deal with queues backing up, lags, HTTP errors, retries, and all of that. Retries specifically are a big pain as you can easily overload your system if you don't have the right mechanisms in place. There are just a lot of things you need to build that your webhook v1, v2 and even v3 just won't have; which you'll constantly have to deal with.

Aram also shared that the extra annoying silent killer is timeouts. The problem is that webhooks receivers vary greatly in quality and latency. This means that oftentimes webhook calls would take a very long time. These long requests would then block delivery workers which can cause delayed delivery and the queues to get congested. Though what's even worse, is that oftentimes customers could take 60s to respond, at which point Guesty's request would already abort due to a timeout (thus be considered a failure) but the receiver would think it was successful (as their logs show a 200 response after 60s). This discrepancy would lead to a lot of confusion as Guesty would retry and the customer wouldn't understand why.

If we had switched to Svix a year ago, we would have saved ourselves a lot of pain and would have gained a lot more peace of mind. There's no question about it.

Another challenge was what Aram referred to as death by a thousand paper cuts. Even if you solve all of the main challenges above, there are still a long list of smaller webhook related annoyances you'll have to deal with on an ongoing basis (see the Svix blog about HTTP quirks for some examples).

There is also a long list of features and functionality that you would probably never build if you're building your own system that really makes the difference for your customers, such as: a UI for inspecting all webhook delivery logs, manually redriving messages, payload transformations, and libraries for webhook signature verification. These are things that your customers will greatly benefit from that Svix just offers out of the box.

Now that we have used Svix, I can't even imagine considering building a webhook system myself ever again. There's just no way this is ever going to happen.

Svix has been transformational for both Guesty

This is what Aram had to say about switching to Svix:

The experience with Svix was so good, that it made us start considering using vendors for other parts of the stack. If we can replicate the Svix experience there it would be well worth it.

One of the benefits that Aram highlighted is that the engineering team just doesn't have to think about webhooks anymore. They just work.

The thing is, we have so much we want to do with our own system, and it's just annoying and exhausting to see yet another random webhook lag, error, or support ticket pop up.

Now, I don't even think about it anymore, as it's all things Svix handles for us, and Svix is spending all their time on making sure it works well. We make API calls to Svix, and Svix just returns a 2xx. We don't need to think or do anything about webhooks anymore.

Our systems don't need to worry about retries, status codes, failures, etc., which also translated to us being able to process many more events than we did before with lower resources and without any issue. It's just a simple pipeline on our end. So even if we have a sudden spike in traffic: we don't care, and we don't even notice these anymore. We just send Svix the events, and Svix takes care of all it of for us.

We make API calls to Svix, and Svix just returns a 2xx. We don't need to think or do anything about webhooks anymore.

Another benefit that Aram highlighted was the added control and reliability they got from using Svix:

Being able to fully control traffic and have customers separate to their own separate delivery pipelines (in Svix). For example easily manage accounts and webhooks if needed. We really appreciate that. Saves us from having to start building complex routing logic, sharding, filtering rules, and other mechanisms to avoid noisy neighbor and other woes; which would otherwise be a pain to build and add more complexity to the system.

Aram also mentioned that they really appreciate that the Svix product keeps on getting better and we get more capabilities for our customers without having to do any additional work. Which he said was especially true for webhooks, because there's no way in the world we'll be able to spend the time it takes to build all of these things.

Guesty's customers love the new webhooks system

We asked Aram what their customers thought about the change, this is what he had to say:

We got amazing feedback from our customers about our new webhook system. They feel like it's a solid system that they can easily build on top of. This enables them to build large and significant systems on top of our webhooks that increase how deeply they integrate with the Guesty product.

Because of this, we now have the stability and confidence to push the webhooks to more of our customers so that they can build the tools they need to enable their business.

Specifically, one of our largest customers came to me and told me how much they are enjoying the new developer experience and visibility they have into their webhooks now.

The observability that we now have for our customers, our support teams, and our internal development team has been a major unlock for us. We knew we couldn't dedicate the engineering time required to build this at the level we required, but now that we have it we can't live without it.

Aram also mentioned that Svix's ultra low latency, no matter the scale and spikes meant that they could trust the data would be delivered in real-time. Being real-time is the difference between their customers being able to build real-time workflows on top of Guesty's systems and not. He further added that having delays can have real world implications to a business like ours. Think about it, a customer can go to their accommodation and not be able to enter the premises as the notification about the code rotation never arrived. Svix solved that for them.

Aram also called out the difference they've noticed when it comes to supporting their customers as they onboard and use the webhooks:

The difference in how we support our customers with their webhooks has been transformational. First of all there's much less of it, as the customers can self serve, but also our ability to help them greatly increased. Not to mention that less technical people can now provide support as the application portal is so easy to navigate.

This means that we can provide a much faster and better experience for our customers which leads to easier onboarding and increased trust in the platform.

It both makes us look better to our customers, and decreases the support burden. It helps them understand that Guesty is actually sending the webhooks, and there are no issues with the Guesty systems, but rather there's an issue with some specific endpoint. Completely transforming how they see us. Without the proper observability it's hard to always know where the problem actually is, and it's even harder to relay that in a customer in a way that is fully clear.

Our support around webhooks changed from helping people investigate issues in their webhooks, to people asking us to add more types of events because they want to use webhooks more.

Closing words

Aram summarized their experience by saying that switching to Svix is a no-brainer.

We asked Aram how long it took them to migrate to Svix

I think the full integration took 2-3 weeks total. Building the initial Svix integration was very fast, and adding the application portal to our dashboard was another 1-2 days. What slowed us down is that this work exposed other things we wanted to improve in our system.

Because a lot of their questions rely on their webhooks for critical business operations, they took an additional few weeks to gradually migrate production customers to the new system, while monitoring and ensuring everything works as expected.

Aram also mentioned they especially appreciated how reliable the Svix systems are:

I really appreciate that you guarantee 99.999% of uptime which is unusual, most vendors only offer 99.99%. We have our own monitoring in place for both uptime and latency, so we can also see that it's not just a number on your website, but you actually deliver.

As for support:

Support was great. We rarely needed to contact you as everything worked very well, but for the few times we did it was very easy. You responded promptly, were super helpful, and helped us get our question resolved quickly.

The experience with Svix was so good, that it made us start considering using vendors for other parts of the stack.

Aram also highlighted a surprising benefit they got from switching to Svix:

We have a lot of different teams at Guesty sending webhooks through this system, and the additional visibility enabled us to easily identify when a specific set of events are too large or misbehaving so that we can go to the right team and fix it.

Switching to Svix has made it easier for us to talk and reason about webhooks internally. The quality of the internal discourse significantly increased.