Overview
Affordable Car Loans (“ACL”) is a prominent provider of car finance solutions in Australia, committed to offering competitive rates and excellent customer service.
Challenge
As ACL continues to grow, their existing ETL (Extract, Transform, Load) and data warehouse solution has struggled to keep up with the growing volume of data and the need for timely reporting. They faced challenges in:
Scalability: Their current infrastructure could not efficiently handle the increasing data load.
Timeliness: Delays in data processing and reporting were impacting decision-making and operational efficiency.
Reliability: Issues with data integrity and system stability were becoming more frequent.
Cost: The cost of their existing ETL process had grown 10x and was continuing to grow at an exponential rate (representing significant dollar cost increases).
Solution
Report Simple implemented a comprehensive data analytics solution using Microsoft Fabric:
Review & Optimise Existing ETL: identified and revised costly legacy data processing scripts, ensuring timely cost savings during the architecture and implementation of the Microsoft Fabric solution.
End-to-End Solution: Implemented Microsoft Fabric for ETL, data warehouse, and reporting, leveraging:
Data Pipelines: Automated data extraction, transformation, and loading processes.
Dataflow Gen 2: Efficient data ingestion and management in Azure Data Lake Storage Gen2.
Spark SQL Notebooks: Advanced data processing using Spark SQL for scalable analytics.
Lakehouses: Integrated data lake and warehouse architecture for unified data management.
Reporting in Power BI: Utilised Power BI for intuitive and interactive reporting dashboards, enabling Affordable Car Loans to derive actionable insights quickly.
Best Practices and Data Security: Implemented industry best practices to ensure data security and compliance, particularly around handling Personal Identifiable Information (PII).
Review of Existing ETL: Identify troublesome processes and scripts causing cost blow out.
Initial Planning: The team held several planning sessions to map out the solution architecture and implementation process. A detailed timeline of 3 months was created, outlining each phase of the deployment.
Technology Stack: Microsoft Fabric F4 was utilised leveraging its integrated data and analytics capabilities.
Roles and Responsibilities: Report Simple was responsible for project management and implementation. ACL was responsible for project coordination.
Development Phase:
The development phase began with implementing code improvements to existing ETL which resulted in an immediate and ongoing cost saving of 80 – 90%. These cost savings effectively paid for the development of new Fabric solution.
Unpicked the existing ETL processes and underlying logic, which were scattered throughout the system.
After thoroughly understanding the current system, we implemented a best practice Medallion architecture, recommended by Microsoft, providing a structured and scalable data processing framework.
The best practice Medallion architecture allowed us to streamline data workflows and ensure data quality at each stage (bronze, silver, gold).
Testing Phase:
The testing phase involved reconciling the new Fabric Gold Lakehouse reporting layer with the existing ETL processes. This was crucial to ensure consistency and accuracy in reporting.
We scheduled regular refreshes of the respective data pipelines and monitored their robustness over a period of two weeks. This monitoring period helped us identify and address any issues in the data flow and processing.
Deployment Phase:
Deployment involved repointing existing Power BI reports to the new data source. This ensured that all reports were up-to-date and reflected the new architecture.
Clear communication was maintained with the broader business regarding deployment timelines. This included informing stakeholders about the schedule and any expected downtime.
A clear line of communication was established for feedback and bug reporting, ensuring any issues could be quickly addressed post-deployment.
Result
Significantly improved ETL robustness and reliability, resulting in a ~90% cost saving compared to the original ETL and ~50% cost savings compared to the original optimised ETL process and architecture. Because the solution is not as technically complex, the new process is easier to monitor and manage by different employees within the company.
A scalable platform to grow with ACL as more data sources are required to be reported on.
Conclusion
By collaborating with Report Simple, ACL transformed its reporting process from fragmented and expensive to a streamlined, data-driven leader. With continuous cost savings and enhanced capabilities in advanced analytics and machine learning, ACL is now better equipped to maintain its industry leadership.
To learn how Report Simple can transform your business’s data analytics, contact us today!
Our collaboration with Report Simple on this data analytics project has been invaluable. The insights provided have transformed our approach to obtaining data from our main data sources, resulting in tangible improvements in business intelligence reporting and access to our data. We are impressed by their expertise, dedication, actionable recommendations and delivery in line with communicated timelines.
Isaac Legge | General Manager