How our engineering team prepared for Black Friday

By Dave Marsland, 30 Nov 2017, in people, engineering

Black Friday is a very important day for us in Engineering. We see more traffic than any other day in the year and it’s a real test of our ability to stay reliable at scale, so we started to plan months in advance to ensure that it was a quiet night for us, and we could sit back and watch the graphs with a beer!

Now that we’re out the other-side, we’d like to share with you what we did, what we learnt and how we made sure the night was a success for us and our vendors.

Load testing

The great thing about predictable big days of load is just that: they’re predictable with no surprises… in theory. We knew that our platform could comfortably deal with the load it had been under in the past but Black Friday would be our highest volume day by far, so we wanted to make sure the platform had been under such load in controlled conditions before the big night.

To do this, we used a combination of JMeter and Locust.io to create realistic fake load against the checkout and our order processing systems, so we could stress them upto and beyond the levels we’d expect to see.

We already had NewRelic and Cloudwatch in place to tell us if we saw any slowdown, so once we had the tests set up we ramped up the load against our production-like test environment and watched closely…

Checkout load

After some scaling of our infrastructure in AWS, we could see that the core Checkout and Order Processing parts of the platform performed well under load beyond what we’d expect on Black Friday, so we looked for any opportunities to optimise further.

Optimisations

Although the platform coped well under extreme load, we saw that there were some reasonably simple changes we could make to increase our order and license processing throughput.

When we’re at our most busy it’s vital that we continue to deliver a great experience to our vendors and their customers, so we focused on reducing the time taken for our Workers to go through their queue of things to process.

By doing this we were able to re-run the same test again, and see that we’d dramatically improved our ability to process orders from the queue, so we were confident that we’d have a calm and peaceful evening.

Order queue

People

Our Engineers at Paddle are passionate that they should support the code that they write, so with only a small amount of convincing via free pizza everyone agreed that they’d stay around on Friday night to watch the platform, and address any unexpected issues.

Due to all the hard work that we put in during the preceding weeks we were able to sit together and watch our platform process huge amounts of load without breaking a sweat, delivering a very successful night for our vendors. A great achievement!

In summary

By eliminating all the predictable problems when it comes to high traffic situations, we were able to confidently go into our busiest weekend ever, and now know in detail where our Platform flies and where there is room for improvement. Bring on next year…

We think you may also like..

Announcing our $12.5m investment to continue our mission to change the way software is sold

By Christian Owens, 13 Dec 2017, in announcements

We're excited to announce that we’ve raised $12.5m in additional funding. This is just the beginning: this will allow us to double down on our ambition to build the best way to sell software and become the operating system for software companies.
Read more

How software sellers used coupons to boost their checkout conversion on Black Friday

By Anna Wittrup, 13 Dec 2017, in guide, Black Friday

Black Friday is great for maxing out your checkout conversion, with a huge supply of customers in the market primed and ready to buy. But how do you stand out from the crowd, promote your deals and capture valuable customer interest during this busy time of the year?
Read more