Transforming Data Management for Catholic Schools New South Wales with Microsoft Power Portal and Azure Databricks

Data Addiction collaborated with Catholic Schools New South Wales (CSNSW), the approved authority for Catholic schools in NSW, to modernise their data management system using Microsoft Power Portal and Azure Databricks. Supporting the Bishops of NSW in bringing to life the Church’s evangelising mission through Catholic schools, CSNSW is the peak body for 550 schools across the dioceses and manages the collection and dissemination of key datasets such as census data used by the Department of Education for government funding distribution. This project aimed to replace an outdated and costly census platform with a modern, scalable solution.

Having previously invested in the Databricks platform, CSNSW chose to leverage this infrastructure for their transformation. After seeking recommendations for suitable partners through Databricks, Data Addiction was selected due to their expertise in Microsoft technologies. Data Addiction’s swift response and ability to utilise Microsoft’s funding programs were also crucial factors in this decision.

A Call for Change

CSNSW faced challenges with their legacy system, which was used for collecting and managing census data from their schools across NSW. The system was manual, costly, and nearing end-of-life. They needed a solution that could scale up during data collection periods and scale down when not in use, reducing ongoing costs. Some of the key issues included:

Manual Data Handling: Schools had to upload CSV files manually, which were then validated and consolidated manually by CSNSW. This process was time-consuming and prone to errors.

Costly Licensing: CSNSW’s previous platform required a costly annual licence, even though it was only used twice a year for data collection.

Data Validation: The manual validation process was onerous and inefficient, leading to delays and potential inaccuracies in the final data submitted to the Department of Education.

The Path to Transformation

To meet CSNSW's needs, Data Addiction developed a comprehensive solution leveraging the full suite of Power Platform capabilities integrated with Azure Databricks. This approach ensured efficiency, scalability, and alignment with CSNSW's existing infrastructure investments. The solution comprised:

Microsoft Power Portal: Serving as the user-friendly front-end interface, Power Portal allows schools to easily upload their CSV files. Its simplicity and accessibility were key benefits.

Power Automate: Power Automate streamlines the data validation process by automatically checking uploaded data for accuracy and completeness. Issues are promptly flagged and returned to schools for correction, ensuring data integrity from the start.

Azure Databricks: After validation, the data is transferred to Azure Databricks for efficient consolidation and analysis. This platform provided CSNSW with a streamlined process to aggregate and organise data, preparing it effectively for detailed reporting.

Power BI: The final stage in the journey involved leveraging Power BI’s reporting capabilities. CSNSW is now able to generate detailed reports that provide critical insights into key metrics such as student numbers and additional needs. This capability has been instrumental in supporting CSNSW's planning and decision-making processes.

By integrating the Power Platform suite with Azure Databricks, Data Addiction delivered a robust and scalable solution, modernising CSNSW's data management and enhancing their reporting capabilities, paving the way for future improvements.

Entering a New Era

While still in the early stages, the new system has already demonstrated benefits.

Cost Savings: Eliminated the need for a costly annual licence. As Mark Cheeseman, Head of Data and Information Systems, CSNSW noted, “The ongoing costs are lower for us. We don’t have this annual licence fee for a product that we’re only using for a small fraction of the year. The Microsoft Stack’s licensing is included within our corporate A5 licensing, saving us money.”

Efficiency: Automation through Power Platform and Azure Databricks improved accuracy and processing speed. Mark reported, “The reduction in help desk requests was significant, indicating the new system’s efficiency.”

Scalability: The system scales easily up and down based on data collection periods, ensuring resource efficiency.

User Satisfaction: User satisfaction improved with the new system. The reduction in help desk issues indicated that the system was easier to use compared to its predecessor. Mark observed, “People have found the new solution easier to use than the one it was replacing.”

Additionally, CSNSW observed that “The Census process has been smooth this cycle and overall it was efficient from a data collection point of view for our users. Data Addiction were quick to jump on and fix any bugs we found.”

The project was characterised by high levels of cooperation and problem-solving. Mark praised the teamwork, stating, “The sense of collaboration and problem-solving was outstanding. It was always a can-do, how do we solve this problem together kind of attitude.”

Embracing Tomorrow's Possibilities

The partnership between Data Addiction and CSNSW has delivered a modern, scalable, and efficient data management system for Catholic schools across NSW. Leveraging Microsoft Power Portal, Power Automate, Azure Databricks, and Power BI, the solution addressed immediate challenges while laying the groundwork for future growth.

The new system has streamlined data collection, improved accuracy, and increased operational efficiency. Despite tight timelines and the challenge of large datasets, strong collaboration ensured the project’s success. CSNSW praised the smooth implementation, noting Data Addiction's responsiveness to issues.

To ensure ongoing success, Data Addiction and CSNSW have entered into a managed service agreement, providing continuous support and optimisation. This project has empowered CSNSW to make data-driven decisions confidently, reinforcing their mission to support Catholic schools across NSW and setting the stage for continued excellence.