01
What is big data architecture?
Big data architecture refers to the logical and physical structure of big data ingestion, processing, storage, access, and management. It includes the hardware, software, and storage components, along with information flow and security layers.
02
What are the big data architecture layers?
- Big data sources layer: Big data arrives from internal and external sources, public and social media, business interactions and transactions, and increasingly from machines. The big data environment can ingest data by both batch processing or real-time processing.
- Data management and storage layer: This layer receives data from all the sources, converts it into a compatible format for the data analytics tools, and stores the data according to formats. It may use hybrid storage to efficiently manage the different formats.
- Analysis Layer: The analytics tools access the stored big data to extract business intelligence. Multiple tools may be used on structured and unstructured data, leveraging advanced technologies for predictive analytics and forecasting.
- Consumption Layer: This layer receives the analytical results and presents them to the relevant output layer for consumption, to humans, applications, or business processes.
03
What are the big data architecture challenges?
- Data Quality: Insights from data can be trusted only if the data powering them is trusted. Comprehensive data governance capabilities are required to assure data quality and reliability.
- Scaling: By definition, big data volume is enormous and growing by the minute. The architecture must support the scaling needs to ensure consistent performance.
- Security: The architecture must support the needs of protecting sensitive data and compliance with data privacy regulations.
