Have you ever struggled with slow or unresponsive reports in your company? Have data silos stopped you from solving problems? You might find the following success story compelling.
Turfsport is one of South Africa's leading wagering software providers. They process 2 billion+ bets a year. Now that's a lot of data.
Imagine for a second management makes the following request: "Please give me the total bet payout across this quarter and all customers." Well, at 2 billion bets a year, you're asking a question across 500 million bets. Are your reports working across 500 million users, bookings, bets, transactions or whatever your business revolves around? If they are, amazing! If they're not, you might be sitting with a big problem. How do you report to auditors or other high-risk parties when you can't even get metrics on your quarterly (or even monthly) data?
Turfsport approached Symbyte to solve this exact problem.
3 months into the project, Turfsport was able to pull internal reports on their data across the lifetime of all data. We had set up a data replication process on AWS replicating their databases to a serverless reporting engine that could easily query 100's of millions to billions of records within seconds. See our sister article for the underlying technology this is built on.
Now we changed the game, but we were not out of the woods yet.
We had to 1) get customers querying their own lifetime data and 2) setup the replication to be redundant, self-healing and communicate failures.
Number 1) is achieved through connecting Turfsport's self-service and integrated reporting module with AWS Redshift (a data warehouse).
Number 2) is achieved through AWS Step Functions (an automation tool). We use Step Functions and Lambda to extract, transform and load data, heal any issues encountered on the data and report on failures (observability).
Now customers are querying their lifetime data in seconds, with a 15min-60min age delay at maximum.
Mission accomplished 🫡
In Part 2 we are going to unpack how we phased out the existing integrated reporting module with CubeJS.