Data mesh may not be for all organizations....
All organizations will have enterprise-wide corporate systems (such as CRM, HCM) and operational systems (to manage products, services, inventory etc.). The operational systems powers the organization through transactional activities while the data lake & analytics systems power the organization through insights and data intelligence. I think the question of whether to data mesh or not, depends on your organization's IT organizational structure. If all the IT teams across the corporate systems, operational systems, analytics platforms are organizationally aligned and has the autonomy to drive the architecture and workflows as a network of interconnected workflows, then data mesh solution can be valuable. I'll go deep into the strategy, building your data architecture for operational systems, manage data access and failure points.
Here are some of the common enterprise software systems used by companies:
CRM - Customer Relationship Management
Popular systems: Salesforce, Microsoft Dynamics, Oracle Siebel, SAP C/4HANA
HCM - Human Capital Management
Popular systems: Workday, Oracle HCM Cloud, SAP SuccessFactors, ADP
ERP - Enterprise Resource Planning
Popular systems: SAP, Oracle E-Business Suite, Microsoft Dynamics 365, Infor
SCM - Supply Chain Management
Popular systems: SAP Ariba, Oracle SCM, Blue Yonder, E2open
PLM - Product Lifecycle Management
Popular systems: PTC Windchill, Siemens PLM, Oracle Agile, SAP PLM
BI - Business Intelligence
Popular systems: Tableau, Microsoft Power BI, Qlik Sense, Looker
HRIS - Human Resource Information System
Popular systems: Workday, Oracle HCM, SAP SuccessFactors, UKG Pro
Some other common enterprise systems are LMS (Learning Management), POS (Point of Sale), PPM (Project Portfolio Management), EAM (Enterprise Asset Management) among others. Most companies use a mix of the major enterprise software vendors like SAP, Oracle, Microsoft and niche players for their systems.
Your organization will have dedicated IT to support these enterprise and operational system. For instance, HR-IT, Finance-IT are departments in the organizations I've worked for. For instance, Finance-IT will service the departmental needs (both internal and external) by managing IT platforms and building their own data warehouse.
A financial data warehouse (FDW) can be used to support a wide range of use cases, including:
Financial reporting: FDWs can be used to generate accurate and timely financial reports, such as balance sheets, income statements, and cash flow statements. This information can be used to make informed decisions about the financial health of the organization and to track progress towards financial goals.
Financial planning and analysis (FP&A): FDWs can be used to perform FP&A tasks, such as budgeting, forecasting, and scenario planning. This information can be used to develop financial strategies and to make informed decisions about how to allocate resources.
Risk management: FDWs can be used to identify and assess financial risks, such as credit risk, market risk, and operational risk. This information can be used to develop mitigation strategies and to reduce the likelihood of financial losses.
Compliance: FDWs can be used to ensure compliance with financial regulations, such as those issued by the Securities and Exchange Commission (SEC) and the Financial Industry Regulatory Authority (FINRA). This information can be used to avoid costly fines and penalties.
How financial departments use FDWs for internal and external needs
Internal needs: Financial departments use FDWs to generate reports for internal stakeholders, such as the board of directors, senior management, and line-of-business managers. These reports are used to make informed decisions about the financial health of the organization, to track progress towards financial goals, and to identify and address financial risks.
External needs: Financial departments also use FDWs to generate reports for external stakeholders, such as investors, creditors, and regulators. These reports are used to meet financial reporting requirements, to disclose financial information to the public, and to demonstrate compliance with financial regulations.
Here are some specific examples of how financial departments use FDWs:
A bank uses an FDW to generate daily reports on its loan portfolio. This information is used to identify and manage credit risk.
An investment firm uses an FDW to generate weekly reports on its investment portfolio. This information is used to track performance and to make investment decisions.
A manufacturing company uses an FDW to generate monthly reports on its financial performance. This information is used to make decisions about resource allocation and to track progress towards financial goals.
A public company uses an FDW to generate quarterly and annual financial reports for investors. This information is used to disclose financial information to the public and to meet financial reporting requirements.
From a strategic perspective, we need to ensure that all cross-functional areas and stakeholders external to the finance department that use the financial data have data access, quality data, and the use cases satisfied. To achieve that, I established a data governance program with assigned roles and responsibilities for the finance business area, including Finance-IT. They were responsible for meeting the external business use cases besides their regular duties. For example, the financial identifiers (bill code, accounting codes), customer identifiers, reference data standardization (such as currency, country), and data category standardization (for e.g. list of values that affects cross-functional areas and external stakeholders).
The data architecture ensured efficient data integrations and data movements within the financial system landscape, that reduces point-to-point integration, automates to avoid manual entries, standardizes data visualization (e.g. PowerBI) and integrates with other systems that affects downstream systems. With data governance and data architecture, the financial systems integrated with the MDM system to ensure critical entities (such as customer and customer hierarchy) are synchronized. Our centralized data lake received the slice of the FDW data that the cross-functional teams needed (such as accounting codes, bill codes, certain financial aggregates like total outstanding receivables).
Each departmental IT team had to be accountable for data governance and had to fulfill use cases beyond their departmental needs. This enabled domain specific decentralization so that the subject matter experts manage their own data while complying with the overall business needs. This allowed us to establish a data architecture that was scalable and collaborative. The aim was to enable teams to work independently and efficiently with data, improving their decision-making and product development.
To enhance data discoverability and foster cross-functional collaboration, we need to build additional important components, such as a published data catalog that includes information about the data sets managed within the Finance area, a data dictionary, a glossary, metadata, etc. and the data lineage (data movements, transformations involved, integrations involved).
We integrated FDW with MDM and moved financial data to our centralized data lake using a standardized approach. This helps with downstream workflows, analytics, and data visualization. Your data architecture can balance decentralization and centralization without needing to productize the finance data.
We made our operational systems more scalable, fault tolerant, and have independent roadmaps thanks to microservice architecture.
In summary, ensure you have a strategy and an architecture to support your corporate-level systems and the operational systems. I do not recommend using an extreme centralized model for the strategy, where the management of all the IT needs and data warehouse needs are performed by a centralized IT team. At the operational-level, only 'productize' areas that have significant value. Centralizing data lakes provided a better solution for our ‘productization’ needs. The data mesh was not a good option for us due to lack of ownership of respective business areas (for instance, Finance alone could make a decision on their platform choice etc. They do comply and adhere by regulatory, data security and other data policies). This is likely the case for most organizations. I'll share how we addressed this, and also addressing the challenges that one would face in a data lake centralized models.
Ownership: We addressed the lack of ownership in the centralized data lake by ensuring subject matter experts are involved in solving the downstream use cases. For instance, the Finance-IT assisted in the data aggregation and data insights to ensure that the analytics and data visualization performed at the centralized warehouse level were meaningful.
Scalability: We ensured that the architecture allows for independent scaling at the workflow, operational and analytics layers.
Cost: The value provided by the centralized models not only balanced the cost of the decentralize-only model but allowed us to establish new revenue opportunities. The data marketplace, enterprise-view of our entire data landscape, external data exchange, cross-functional data aggregation and insights would not have been possible for us without the centralized model.
Data Governance: Without proper data governance, centralized models can quickly become disorganized, making it challenging for users to find the data they need. I continue to stress the important of the enterprise-wide data governance program to ensure a business driven journey progresses across business areas (thus data silos) over time and to ensure the organization has an enterprise-wide view of the data and ways (policies, guidelines, enforcement) to put all the data together for better data accessibility and with data quality.
Sash Barige
Nov/16/2022
Comments