In this article we will discuss on how ABN AMRO bank uses data mesh architecture on Microsoft Azure for faster data insights & business decisions while governing & securing the data.
Why ABN AMRO choose Microsoft Azure?
- Bank need to processes diverse & complex type of data daily in variety of application used in whole banking system.
- Migration to Azure Cloud because of greater scalability, ease of access using suite of Azure services to empower both customers and functional teams alike.
“We embraced a data mesh architecture on Azure, where you see a lot of forward thinking regarding data ownership and security,” — ABN AMRO
Design consideration of new data architecture (data mesh architecture azure):
- Readability of data is most importance to data architecture.
- Data read-versus-write ratio required in data analytics like analytical models that are constantly retrained read huge volumes of data.
- Migrating to Azure allowed the bank to capitalize on cloud-native services designed to drive both growth and digital transformation.
- Azure Service like Azure Data Factory is fully managed, serverless data integration service has automated processes
What is data mesh architecture?
A data mesh architecture embraces distributed data using domain-driven design principles. It supports data democratization and “data as a product,” with specific domains managing their own data pipeline.
How this migration to Azure helps Bank?
- Migration from the bank’s tightly coupled on-premises systems to a cloud-based platform that decoupled compute from storage.
- Decoupling of compute from storage improves scalability of compute and storage independently.
- Faster processing of data: Processes that once kept on-premises servers busy for months now take mere days in the cloud.
“Azure interoperates well with legacy systems, with an emphasis on security that’s really nice. Azure is really leapfrogging the competition, one or two years ahead. ” — ABN AMRO
What patterns are used in data mesh architecture & application integration?
- At the edges of data mesh, distributed domains are built on Domain-Driven Design and respective domain boundaries where each team responsible for their applications and the data that comes with it. data product owner is to provide high quality of the data.
- Mesh model is central place to decouple and control center to distribute data to any location. Domains can access each other’s datasets using mesh to distribute data implementing service-oriented architecture (SOA- API patterns), DIAL patterns & event-driven architecture (EDA) and CQRS together.
Why ABN AMRO chooses data mesh architecture (Data Mesh in Azure)?
- A data mesh opens up new possibilities for analysts looking to not just store data, but to use it to drive better business decisions among different teams with different needs.
- a cloud-distributed data mesh offers even more flexibility when managing the needs of cross-functional teams.
- When data is coming from multiple sources, in multiple formats, having one logical place to store and distribute that data is the starting point to develop a more responsive, customized solution.
- Azure Data Lake Storage is an optimal solution for storing both structured and unstructured data.
- It saved data in Parquet file format in Azure Data Lake Storage so that data is readily available to data team & a wide range of Azure services, including Azure Databricks and Azure Synapse & without having to process it into different formats.
Data Architecture Components:
- Azure Data Factory: for orchestrating all the data i.e. ingesting, preparing, and transforming data at scale
- Azure Databricks: for processing data , analytics workflows by enabling collaboration, AI insights, and advanced, automated machine learning capabilities; Shared workspace and inter-operation with Azure Synapse Analytics for high-performance data warehousing make for unparalleled levels of performance and scalability.
functional teams able to process data more efficiently & to use that data easily. - Azure Data Lake Storage: one logical place to store and distribute data.
- Azure Synapse : As data warehouse for consuming the data, query data on their own terms, using either serverless or dedicated resources — at scale.
- Azure API management to sync hybrid and multi-cloud platforms so bank can retain some data on-premises and interacts with other cloud platforms.
Governing data faster meeting regulatory compliance obligation
- ABN AMRO used a data mesh architecture on Azure to model and govern data faster and more accurately.
- Since this data includes the sensitive financial information of millions of customers, the systems managing it must absolutely meet the most rigorous security standards
- Data governance body within our company, and no data is allowed to be distributed or consumed without clear ownership
Data security- a game-changer in data security for ABN AMRO:
- Azure-native security features, including Azure Active Directory provides Single Sign-On & Multi Factor Authentication.
- This was complemented on the analytics side by Azure Synapse, with a robust set of security features that include dynamic data masking and row-level security.authentication
How Azure Synapse Analytics is useful for analytics for ABN AMRO
ABN AMRO’s digital transformation started with Ingesting and managing data at scale.
- Azure Synapse Analytics, a data integration, enterprise data warehousing, and big data analytic used as tools to see the forest through the trees.
- After business transformation, data insights of stored data is available through Azure Synapse Analytics.
- Analysts can now query data on their own terms, using either serverless or dedicated resources — at scale.
“All these Azure products are designed to seamlessly work together in precisely the way ABN AMRO needs them to. The company’s engineers and diverse user groups have been duly impressed by the integrated power and future-proofed innovations of Azure offerings.” — ABN AMRO
“I foresee a future where we have a much smaller on-premises footprint. There’s so much data processing and analytics we can do, making predictions, doing all kinds of complex calculations, and developing entirely new, cutting-edge use cases, for example with Azure Machine Learning Services. All of it can be done on Azure.” — ABN AMRO
Thanks for reading the article. Please feel free to comment below in case of query/thoughts.