Introduction
With the exponential growth of data volumes and increasing diversity of data sources and technologies, organisations face challenges in managing and making sense of their data to generate action-oriented insights. Therefore, organisations need to have a deep understanding of their needs and use cases to take appropriate data-driven decisions and design a data architecture that not only empowers the teams but also works across the ecosystem. They need to address the challenges associated with data management in complex and distributed environments including data integration, data governance, AI governance, and data science and machine learning operations (MLOps).
Topic of the month
The legacy approaches for data management often encounter challenges, as outlined below:
- Data proliferation and complexity
- Lack of agility
- Lack of trust in data
- Lack of collaboration
Data proliferation and complexity
Increasing data sources, data consumers and data domains
Lack of agility
Difficulties in meeting timely data and business demands
Lack of trust in data
Several copies of the same data
Lack of collaboration
Isolated developers from different domains
Therefore, there is a need to find a flexible and scalable way to maximise the value of data, quickly respond and adapt to changing business demands, and provide solutions to counter the complexities and limitations of traditional centralised methods.
Data fabric and data mesh are concepts that have emerged recently to manage and organise data within an organisation. Although both concepts aim to address the challenges of data management and create more unified, agile and distributed data ecosystems, allowing organisations to effectively leverage their data assets and empower domain-specific teams, they both take different approaches. Data fabric provides a unified view and integration layer for data, while data mesh promotes decentralised data ownership and governance. Organisations may choose to adopt one or both approaches based on their specific needs and requirements.
To manage data efficiently, various type of data management solutions are implemented by organisations. Different types of widely used data management solutions are as follows:
A data warehouse enables you to translate meaningful data into information, which in turn helps you plan your business strategies. It mainly deals with structured data.
Pros:
- Centralised data repository from multiple source systems
- Quick and fast access to data
Cons:
- Not suited for the all types of data
- Lack of flexibility due to centralised administration and rigid data transformation paths
- High storage and operational costs
- Limited support for analytical use cases
Data lakes, a next-generation data storage and management solution, were developed to meet the ever-evolving needs of business users.
Data lakes are built with the following characteristics:
- Volume: Vast amount of data generated through large-scale ratification and digitisation of information
- Variety: Different types and forms of data, including large amounts of unstructured data
- Veracity: Level of quality, accuracy and uncertainty of data and data sources
- Velocity: Speed at which data are generated and analysed
- Value: The relative importance of different data to the decision-making process and potential of big data for socio-economic development
Pros
- Centralised data repository
- Flexible enough to incorporate multiple storage types and formats
- Cost-efficient
Cons
- More suited for analytical use cases and offers less performance for BI workload as compared to data warehouse.
- Requires more effort on governance processes
Data lakehouses combine the flexibility and cost efficiency of a data lake with the uniform and proven processes of a data warehouse.
Pros
- Supports both traditional BI and advanced analytical workloads
- Supports huge volume and variety of data
Cons
- Requires complex and effort-intensive metadata and governance layer
- More difficult to set up due to its hybrid nature
Data fabric is a centralised approach to data management that focuses on creating a unified and integrated view of data across various sources and systems within an organisation. It aims to provide a single access point for data consumers, ensuring consistent and reliable access to the data they need. Data fabric is an innovative approach that addresses the challenges faced in big data projects by connecting diverse data tools through metadata-driven methods. It aims to provide a seamless and user-friendly experience by creating a unified management layer over distributed data.
By leveraging existing metadata resources, data fabric enables efficient design, deployment and optimal utilisation of data across multiple platforms and environments. Its architecture facilitates the seamless integration of different data pipelines and cloud environments through intelligent automation systems, ensuring end-to-end connectivity and enhanced efficiency.
Data fabric is an architecture that facilitates the end-to-end integration of various data pipelines and cloud environments using intelligent and automated systems.
Let’s have a look at the architecture layers that define a data fabric approach:
Data from various sources are integrated with the platform. The sources can be of different types, such as structured and unstructured data, logs and files, and integrated via various methods like batch, real time and APIs.
All types of metadata are collected and stored into data catalogues and business glossaries. These are further enhanced by constructing knowledge graphs for more efficient use of data. Data provenance is used extensively when visualising data lineage, recommending data sourcing as well as supporting AI/ML-based relationship inference engines.
Reliable and robust data integration pipelines, well-defined data transformation and cleansing routines make it possible to deliver high quality data to users in a quick and seamless manner. Virtualisation is a critical component that enables users to access a single version of data while abstracting the underlying data source and type complexities.
Data can easily be consumed in the form of BI dashboards, downstream applications, operational KPIs, etc. Exhaustive data catalogues enable business users to find data on their own. In addition to this, recommendation engines suggest data points depending on the type of user and purpose.
All types of users in an organisation can leverage data fabric to consume data of their choice. End users can quickly identify and access the information they need from a centralised platform without worrying about duplication or relying on other teams to derive insights and action upon them.
Data mesh is a decentralised approach to data architecture that aims to empower individual teams or domains within an organisation to own and manage their own data products. It recognises that data is a product and promotes the idea that each team should have control over the data they generate, ensuring data ownership and accountability. Data mesh targets to manage data through several groups of teams as per need, with some common governance provisions. The core principle driving data mesh is rectifying the incongruence between the data lake (storing largely unstructured data that’s used by data scientists for building predictive ML models) and data warehouse (storing largely structured data that’s used by data analysts for SQL analytics).
Let us understand the data mesh approach in detail by looking at the core principles.
- Domain-driven data ownership: This aims at defining the boundaries so that one of the critical aspects of design, i.e. ownership, can be addressed. There is no standard or uniform strategy to define the domain. It could be grouping based on similar functionalities, use case requirements, department/LOBs, etc.
- Data as a product: There is no uniform definition for data products. Some consider it as data + metadata and many other consider data product as data + metadata + infrastructure + business transformations (code repository). Domain owners should take ownership of delivering high-quality data products by using their domain knowledge to clean, transform and enhance their products.
- Self-serve infrastructure platform: The necessary technology to make the development of specialised data products accessible to developers could be made available as a part of a platform through centrally managed infrastructure.
- Federated governance: Global standards should be put in place to ensure interoperability and accessibility of data products across domains.
- Cross-domain interoperability: It’s a mechanism by which data products can be shared with other domains to help reduce data redundancy. The integration patterns could be batch, micro batch or real-time processing, depending on the requirements. To facilitate this, it is important to standardise the acquisition, transformation and data sharing, otherwise each domain team will make set different standards which may not be compatible with those of the other teams.
Below is a depiction of the above principles in a bank:
| Data fabric | Data mesh | Areas |
| Data fabric integrates data from diverse sources, such as databases, data warehouses, data lakes, APIs, and external systems, and provides a consistent view of the data. | Each team or domain takes responsibility for their own data products, inclauding data quality, governance, and life cycle management. | Implementation |
| Data fabric is low-code, no-code, which means that the API integration is happening inside the fabric without leveraging it directly. | A data mesh is basically an API-driven [solution] for developers where we can write code for the user interface. | Code usage |
| Data fabric enforces data governance policies, including data access controls, data quality monitoring and data lineage tracking. | Governance processes and policies are established across teams to ensure data interoperability, security, privacy, compliance, healthy and interoperable ecosystem. | Governance |
| A data fabric is an architectural approach, primarily focused on technology, which handles the complexity of data and metadata intelligently. | A data mesh is more focused on people and process rather than architecture in order to drive the requirements for data products upfront. | Underlying approach |
Pros:
|
Pros:
|
|
Cons:
|
Cons:
|
Data fabric
- Unified customer view: Data fabric integrates customer data from diverse systems, enabling a unified view. This allows relationship managers and customer service representatives to gain comprehensive customer information for enhanced engagement and personalised experiences.
- Enhanced risk management: Data fabric integrates risk data from various systems, providing a holistic view of risks. Risk managers can conduct thorough assessments and make informed decisions for effective risk mitigation.
- Fraud detection and prevention: By integrating data from multiple systems, including transactions and customer information, data fabric detects patterns and anomalies indicative of fraud. Its holistic view facilitates advanced analytics and ML algorithms to identify and prevent fraudulent transactions.
- Integration of data sources: Implementing a data fabric can help integrate disparate data sources that NBFCs deal with, such as customer information, financial transactions, credit scores, risk data and market data, providing a unified and consistent view of data across the organisation.
- Real-time trading and analytics: NBFCs often require timely insights into customer behaviour, creditworthiness, risk assessment and market trends. Data fabric can enable real-time data processing and analytics, providing the necessary information to make quick informed business decisions.
- Data governance and compliance: NBFCs operate in a highly regulated environment and must adhere to various data governance and compliance requirements. A data fabric approach can help implement robust data governance practices.
- Integrated policy management: Data fabric integrates policy data from various systems, streamlining administration and providing consistent information to agents and customers.
- Fraud detection and prevention: Data fabric integrates diverse data sources, aiding fraud detection efforts by identifying suspicious patterns and mitigating fraud risks.
- Actuarial analysis and pricing: Data fabric combines historical claims, market and demographic data for accurate risk assessments, pricing models and optimised insurance products.
Data mesh
- Customer data management: Banking units (e.g. retail, corporate, wealth management) develop dedicated data products for customer data. This enables personalised experiences, tailored banking products and efficient management of customer information.
- Automated virtual assistants: Chatbots are commonly used to support call centres and customer helpdesk teams. The FAQs usually touch on various datasets. Therefore, a distributed data architecture which ensures that virtual agent systems have access to the required data assets would be helpful. This allows the system to better understand the users and their queries and provide satisfactory responses.
- Regulatory compliance: Compliance teams leverage dedicated data products to ensure adherence to regulatory requirements (e.g. anti-money laundering, know your customer [KYC]). Efficient management of compliance data enables audits and streamlined regulatory reporting.
- Data collaboration and APIs: Implementing data mesh promotes the use of self-describing APIs to expose and consume data as services. This allows seamless data sharing and integration with other financial institutions, FinTech companies or credit bureaus.
- Customer analytics: Customer-facing teams own customer data products for analysing behaviour, preferences and demographics. This leads to personalised offers, improved satisfaction and optimised marketing campaigns.
- Data productisation: Each domain team can develop and maintain their data products tailored to their specific needs. For example, credit underwriting teams can develop credit risk models, lending teams can build loan approval engines and marketing teams can create customer segmentation models.
- Underwriting and risk assessment: Underwriting teams own data products for specific business lines or customer segments, enabling customised risk assessment and tailored underwriting models.
- Claims management: Dedicated data products help claims departments in the efficient management of claims data, streamlined processing, fraud detection and enhanced settlement efficiency.
- Customer experience and personalisation: Customerfacing teams utilise their own data products to analyse customer data, track interactions and personalise experiences. This leads to a better understanding of customer needs, tailored insurance offerings and proactive customer support.
A leading European fashion e-commerce company has implemented data mesh architecture to enable data autonomy and improve data discoverability. It has built a self-serve platform that provides data products owned by individual teams. The platform acts as a centralised catalogue for data discovery, allowing teams to find and access relevant data products.
A popular music streaming platform has embraced the data mesh approach to empower autonomous teams and promote data ownership. It has decentralised its data infrastructure and established cross-functional squads responsible for specific data domains. Each squad has ownership of its data products, including data quality, governance and analytics.
A global financial services firm has implemented a data fabric to integrate and manage vast amounts of data across different systems and business units. Their data fabric infrastructure provides a unified data layer, enabling consistent access to data for various purposes such as risk management, compliance and customer analytics.
A multinational insurance company has implemented a data fabric to address data integration challenges and enable real-time analytics. They have built a data fabric that integrates data from multiple sources, including policy systems, claims systems and external data feeds. This allows them to have a comprehensive view of risk, enhance underwriting decisions and improve claims management.
Assess the current data warehouse environment and identify its limitations and challenges.
Define a roadmap for implementing data fabric and data mesh strategies based on cloud or on-premises requirements.
Integrate data sources into the chosen implementation, considering data virtualisation and APIs for domain-driven integration.
Establish robust data governance practices within the selected approach to maintain data quality, security and compliance.
Gradually introduce data mesh or data fabric principles to empower domain teams and foster collaboration.
Transitioning from a traditional data warehouse to a data fabric and data mesh approach involves the following steps:
Data fabric
- Deploy data integration, virtualisation and governance tools onsite.
- Centralise data management and governance within the organisation’s data centre.
Data mesh
- Empower domain teams with ownership of their data assets.
- Allow domain teams to develop data products using on-premises infrastructure.
Data fabric
- Implement cloud-based data fabric solutions.
- Leverage managedservices for data integration, virtualisation and governance.
- Connect on-premises data sources to the cloud-based data fabric for unified data access.
Data mesh
- Utilise cloud-native technologies to implement a cloudbased data mesh.
- Domain teams develop data products using cloud services, fostering data collaboration and sharing.
Data fabric
- Combine on-premises and cloud components.
- Host data integration and governance layers on-premise for compliance reasons.
- Utilise cloud services for specific data products.
Data mesh
- Implement both onpremise and cloudbased data mesh.
- Allow domain teams to use on-premise infrastructure or cloud services for data productdevelopment.
Successful transition from a traditional data warehouse involves well-thought-out strategy, collaboration across teams, careful planning, integration and continuous improvement to modernise data management processes effectively.
The cost of moving from a traditional data warehouse to a data fabric or data mesh approach can vary significantly depending on several factors, including the size of the organisation, complexity of the existing data infrastructure, scope of the transformation, and chosen implementation approach (on-premises, cloud-based or hybrid). Some key cost considerations would be investing in new technologies and tools, upgrading the existing infrastructure, upgrades to the existing infrastructure for data migration, investing in training or hiring specialised talent to support the implementation, licencing or subscription fees, and ongoing maintenance and support.
While there are costs associated with the transition, it’s crucial to consider the potential benefits and returns on investment (ROI) that a data fabric or data mesh can bring. These benefits may include improved data accessibility, enhanced data-driven decision-making, faster data product development and increased agility.
Data fabric and data mesh can co-exist and complement each other. While their approach for data management varies, they address distinct challenges and can be applied in different contexts – such as in large organisations like investment management firms that must manage data locally within regions. The company’s overall data strategy and data governance principles are managed centrally via data fabric, whereas data retention and processing are carried out as per regulations in the respective countries via data mesh.
Organisations can adopt a hybrid approach where data fabric and data mesh principles are applied in different segments of the organisation based on specific needs. For example, certain domains may benefit from a data mesh approach to enable domainspecific autonomy, while other teams may benefit from a centralised data fabric for governance of critical data.
Industry news
1. Unified Payments Interface (UPI) now, pay later
The Reserve Bank of India (RBI) has announced a new UPI feature enabling transfers from presanctioned credit lines through the UPI network. Banks have to first take consent from the user to extend the credit line. Some banks might charge interest amounts while others may offer creditfree periods.
2. RBI approves Poonawalla Fincorp’s co-branded credit card with IndusInd Bank
Poonawalla Fincorp, an NBFC associated with the Cyrus Poonawalla group, has been granted approval by the RBI to introduce a co-branded credit card in collaboration with IndusInd Bank. The company intends to release this credit card within the next three months as announced by Poonawalla Fincorp in an official statement. This collaboration will pave the way for the company in the age of retail credit.
3. Innovative banking solution for Indian micro, small and medium enterprises (MSMEs): Axis Bank unveils ‘NEO for business’.
Axis Bank has launched ‘NEO for business’, a ground-breaking transaction banking platform tailored specifically for Indian MSMEs. This innovative digital solution is designed to meet the ever-evolving transaction banking needs of MSMEs, marking it a unique offering. The platform provides various capabilities such as seamless digital onboarding, efficient handling of bulk payments, generating GST-compliant invoices and integrating payment gateways
4. Indian Bank’s ‘IB SAATHI’ initiative: Expanding financial access
Indian Bank’s ‘IB SAATHI’ initiative seeks to create an integrated financial ecosystem through business correspondents, aiming to streamline services for stakeholders with 5,000 correspondents planned by March 2024, expanding accessibility.
5. Pilot launch of e-Rupee by RBI
The RBI announced that it will initiate a pilot launch of the e-Rupee for specific purposes as part of its digital currency testing in India. The central bank will provide updates on the features and advantages of the e-Rupee as these pilot launches expand
6. Yes Bank introduces revolutionary ‘ONDC Network Gift Card’ to transform shopping in India
On 4 October, YES Bank became the first bank in India to introduce the ‘ONDC Network Gift Card’ in collaboration with the Open Network for Digital Commerce (ONDC). This innovative gift card will revolutionise the shopping experience by allowing customers to shop from various brands and sellers across different product segments like food, fashion and electronics making digital commerce more accessible to Indian consumers.
7. SBI launches ‘mobile handheld device’ to enhance financial inclusion
On 4 October, SBI introduced a ‘mobile handheld device’ to improve accessibility and convenience for its financial inclusion customers. SBI Chairman Dinesh Khara stated that the goal of this initiative is to empower financial inclusion and make essential banking services more accessible to a wider audience.
8. Sumsub redefines user onboarding with nondocument verification using bank data
Sumsub collaborated with OneID to introduce a non-document verification solution, enabling swift and compliant user onboarding using bank official records in the UK. This approach will save time by eliminating the need for document uploading and will simplify the user experience and business compliance by leveraging financial data.
9. Indian NBFCs embrace digital solutions and AI/ML
The RBI is promoting the adoption of digital solutions and AI/ML tools among Indian NBFCs. To further enhance credit risk assessment and digital capabilities, RBI mandated NBFCs with ten or more service delivery units to implement the Core Financial Service Solution (CFSS) by 30 September 2025.
10. 3S Money and KYC modernised know your business (KYB) process
3S Money partnered with KYC to revolutionise KYB processes. This collaboration is a result of a LinkedIn challenge to ‘showcase your data sourcing capabilities’ posted by 3S Money’s CEO Ivan Zhiznevsky. 3S Money offers a global payment platform while KYC provides a SaaS platform and API streamlined for corporate onboarding and KYB compliance.
11. RegTech firm, TSS Consultancy, secured funding from WestBridge Capital
TSS consultancy has secured funding from WestBridge Capital at a valuation of INR 900 crore. The company plans to utilise the funds for research and development, with particular focus on AI, in order to expand its SaaS offerings and launch new products.
12. BNP Paribas embraces technology with cloud and AI focus in 2023
BNP Paribas has prioritised technology with its GTS strategy, focusing on cloud infrastructure and AI implementation with the aim to increase cloud adoption and have over 1,500 employees certified in cloud technology. It is actively recruiting for AI-related roles and has become the top European bank and sixth in terms of globally for hiring AI talent. It has made significant strides in AI, with over 700 AI use cases and approximately 300 more in development, including 26 utilising generative AI.
13. Unique Identification Authority of India (UIDAI) implemented AI-based liveness detection to boost security in Aadhaarenabled payments
The (UIDAI) is rolling out an AI-powered liveness detection feature to tackle fraudulent Aadhaarenabled payments (AePS) in the FinTech industry. This update aims to improve the security of transactions by verifying the authenticity of biometric data, which reduces the risk of fraud. FinTech companies offering AePS services have faced fraud attacks and anticipate that this solution will bolster transaction security. The feature’s software update is underway and will be implemented across biometric devices soon.
14. Major Indian banks embrace private LLMs to transform customer experience and operations
Indian banks like HDFC Bank and Axis Bank are embracing private large language models (LLMs) to revolutionise customer experiences and streamline internal operations. These LLMs, trained on proprietary data, empower banks to offer personalised, intuitive user interfaces. For instance, HDFC Bank is creating an LLMdriven website for customer interactions, while Axis Bank plans to deploy generative AI-powered virtual assistants and automate banking operations. These private LLMs have the potential to optimise decision making, content generation and cost-efficiency across the banking industry, with experts foreseeing widespread adoption in various banking functions in the near future.
15. Bank of England’s data revolution: Adapting to changing standards
The Bank of England recognises the importance of data in central banking and has made modernising its data-related capabilities a strategic priority. An evaluation shows that while progress has been made in enhancing data and analytics, the bank needs to invest more to keep up with best practices and ensure alignment with its mission. Recommendations include a clear vision, effective governance and trained staff
16. Insurance Regulatory and Development Authority (IRDAI) is all set to bring in two new features in the health insurance sector
The IRDAI is working towards a fully cashless settlement system for health insurance policies and making all hospitals across the country get on board with the same. This initiative of having an industry-wide and nationwide common cashless unified hospital network will reduce cases of fraud and provide ease of settlement to customers as well as more hospitals to choose from.
17. Star Health’s UPI QR code-based payment
Star Health has launched UPI QR-code based payment for smooth purchases and renewals. This ease of payment will lead to better customer experience. This feature is enabled by the National Payments Corporation of India (NPCI)
18. IRDAI: Coverage to employees traveling in employer’s vehicle mandatory
Insurance companies providing motor insurance have to include a feature to provide coverage to employees traveling in the employer’s vehicle in their third party policy. No extra premium will be charged until further notice.
19. Fujitsu begins trials with Hokuhoku Financial Group to probe new use cases leveraging generative AI in banking operations
The joint trials with two subsidiaries of Hokuhoku Financial Group – Hokuriku Bank and Hokkaido Bank would aim to leverage Fujitsu’s conversational AI offering via its Fujitsu Kozuchi AI Platform. By deploying generative AI into the banking operations of both these subsidiaries, the trial focuses on accuracy and efficiency in creating programs, test data, fixing bugs, clarifying internal inquiries about business and policies, and generating various business documents. The highlight of the program is to obtain promising results without having any separate testing environment.
20. GoTyme Bank joins NICE Actimize to prevent financial crime and fraud
GoTyme Bank, a digital bank based out of Philippines and joint venture between Tyme and the Gokongwei Group of companies, has partnered with Nice Actimize to fight against digital financial crimes by utilising latter’s AI and analytics solutions.
Acknowledgements
This newsletter has been researched and authored by Antima Garg, Arpita Shrivastava, Bhavika Tahiliani, Garima Yadav, Raghav Sharma, Samir Shah, Smruti Bal, Snehal Nandagawli, Soumodeep Mukherjee, Sourabh Shegaonkar and Srijita Dey
About PwC
At PwC, our purpose is to build trust in society and solve important problems. We’re a network of firms in 152 countries with over 328,000 people who are committed to delivering quality in assurance, advisory and tax services. Find out more and tell us what matters to you by visiting us at www.pwc.com.
PwC refers to the PwC network and/or one or more of its member firms, each of which is a separate legal entity. Please see www.pwc.com/structure for further details.
© 2023 PwC. All rights reserved.
