I’ve worked quite a bit with Microsoft’s unified analytics platform “Fabric” in Azure and I’ll say it definitely helped make things easier to integrate all the different tools like python notebooks, Data Factory (ADF), Power BI and more. This is because Fabric is a unified tool that combines all these components making authentication seamless using the managed identity from Fabric itself. Is Fabric really necessary? It integrates data integration, data engineering, data warehousing, data science, real-time analytics, and business intelligence into a single SaaS offering with a unified storage layer called OneLake. Sounds amazing right?
Let’s say you aren’t familiar with Azure or are migrating away from Microsoft’s platform to the market share leader AWS or GCP or dare I say OCI (Oracle Cloud). Is there a comparable toolset in these clouds?
The short answer is no. These take AWS, GCP, and OCI all take a modular approach to data engineering. This means that AWS, GCP, and OCI do not have a single, direct, “one-to-one” equivalent to Microsoft Fabric because their approach is offering a collection of specialized services that, when combined, can achieve similar capabilities. To build a “Fabric-like” environment for data engineers in these cloud platforms, you would typically use a combination of these services:
AWS (AMAZON WEB SERVICES)
Let’s go over the components you’d need to have all the functionality of Microsoft Fabric.
1. Data Lake Storage (OneLake equivalent):
- Amazon S3 (Simple Storage Service): This is the foundational object storage service in AWS and serves as the primary building block for a data lake. It’s highly scalable, durable, and cost-effective for storing raw and processed data.
2. Data Integration & ETL (Data Factory equivalent):
- AWS Glue: This is a serverless ETL (Extract, Transform, Load) service that can visually develop, run, and monitor ETL workflows. It includes a Data Catalog (a metadata repository) and Glue Studio for building visual ETL jobs.
- AWS Data Pipeline: A web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
- AWS Step Functions: A serverless orchestration service that allows you to define workflows for complex ETL processes, integrating various AWS services.
- Amazon Managed Workflows for Apache Airflow (MWAA): A managed service for Apache Airflow, a popular open-source orchestration tool for programmatically authoring, scheduling, and monitoring data workflows (DAGs).
3. Data Engineering / Big Data Processing (Synapse Spark / Databricks Notebooks equivalent):
- Amazon EMR (Elastic MapReduce): A managed cluster platform that simplifies running big data frameworks like Apache Spark, Hadoop, Hive, Presto, and Flink on AWS. It’s used for large-scale data processing and analytics.
- AWS Glue (with Spark): As mentioned, Glue uses Apache Spark under the hood for its ETL jobs, providing serverless Spark execution.
- Amazon Athena: A serverless interactive query service that allows you to analyze data directly in Amazon S3 using standard SQL. It’s often used for ad-hoc analysis on data lakes.
- AWS Lake Formation: A service that helps you build, secure, and manage data lakes by simplifying the process of collecting, cleaning, and cataloging data, and centralizing security and access control.
- Databricks on AWS: While not a native AWS service, Databricks runs seamlessly on AWS, offering a unified platform for data engineering, data science, and machine learning, built on Apache Spark and Delta Lake. This is often chosen when users want a more managed Spark experience with the benefits of Delta Lake for a data lakehouse.
4. Data Warehousing (Synapse Data Warehouse equivalent):
- Amazon Redshift: A petabyte-scale cloud data warehouse service optimized for analytical workloads, providing fast query performance. It uses massively parallel processing (MPP) for fast query performance. Redshift also offers AI-driven scaling.
- Redshift Spectrum: Allows Redshift to query data directly from S3, enabling a data lakehouse pattern.
5. Real-time Analytics (Synapse Real-time Analytics equivalent):
- Amazon Kinesis: A family of services for real-time streaming data, including Kinesis Data Streams (for collecting and processing large streams of data records), Kinesis Data Firehose (for delivering streaming data to destinations), and Kinesis Data Analytics (for processing streaming data with SQL or Apache Flink).
- Amazon Managed Streaming for Apache Kafka (MSK): A fully managed service for Apache Kafka, allowing you to build and run applications that use Kafka for streaming data.
6. Business Intelligence (Power BI equivalent):
- Amazon QuickSight: A cloud-native business intelligence service that allows you to create interactive dashboards and reports.
7. Data Governance & Security:
- AWS Lake Formation: Centralizes security and access control for your data lake.
- AWS Identity and Access Management (IAM): Manages user access and permissions across all AWS services.
- Amazon Macie: Uses machine learning to discover and protect sensitive data in S3.
Summary:
AWS, offers a modular and highly customizable approach. While it requires you to integrate multiple services, this provides immense flexibility and allows organizations to build highly specialized data architectures tailored to their exact needs. AWS also has a broader global footprint and more mature multi-region support.
GCP (GOOGLE CLOUD PLATFORM)
Like AWS, Google Cloud Platform (GCP) also takes a modular approach rather than having a single, unified “Microsoft Fabric” equivalent. For data engineers, you would combine several services to achieve similar capabilities for data integration, processing, warehousing, and analytics.
Here’s the breakdown of GCP services that collectively form a robust data engineering and analytics platform:
1. Data Lake Storage (OneLake equivalent):
- Cloud Storage: GCP’s highly scalable and durable object storage. It serves as the primary data lake for raw and processed data, similar to S3 in AWS. You can configure buckets with different storage classes (Standard, Nearline, Coldline, Archive) to optimize costs.
2. Data Integration & ETL (Data Factory equivalent):
- Cloud Data Fusion: A fully managed, cloud-native data integration service built on the open-source Cloud Data Fusion. It provides a visual, code-optional interface for building and managing ETL/ELT pipelines, with pre-built connectors and transformations.
- Cloud Dataflow: A fully managed service for executing Apache Beam pipelines. It’s excellent for both batch and stream processing, making it suitable for complex transformations and large-scale data movement. It’s highly scalable and serverless.
- Cloud Composer: A managed Apache Airflow service. This is ideal for orchestrating complex, multi-service data workflows (DAGs) across different GCP services and external systems.
- BigQuery Data Transfer Service: A service specifically designed for automating data transfers from various SaaS applications (like Google Ads, YouTube Analytics, Salesforce) and other cloud storage services directly into BigQuery.
3. Data Engineering / Big Data Processing (Synapse Spark / Databricks Notebooks equivalent):
- Dataproc: A fully managed service for running Apache Spark, Hadoop, Hive, Presto, and other open-source big data frameworks. It allows you to create clusters quickly and run large-scale processing jobs.
- BigQuery: While primarily a data warehouse, BigQuery has significant data engineering capabilities. Its SQL interface allows for complex transformations (
CREATE TABLE AS SELECT,MERGE, etc.) on massive datasets directly within the data warehouse. It’s serverless and highly scalable. - Databricks on GCP: Similar to AWS, Databricks also offers its platform on GCP, providing a unified workspace for data engineering, data science, and machine learning, leveraging Apache Spark and Delta Lake on GCP’s infrastructure.
4. Data Warehousing (Synapse Data Warehouse equivalent):
- BigQuery: GCP’s flagship serverless, highly scalable, and cost-effective data warehouse. It’s designed for petabyte-scale analytics and offers features like columnar storage, automatic scaling, and machine learning capabilities directly within SQL. It’s a key component of the “Lakehouse” architecture on GCP.
5. Real-time Analytics (Synapse Real-time Analytics equivalent):
- Cloud Pub/Sub: A fully managed, real-time messaging service that allows you to ingest and deliver high volumes of events and data streams. It’s often used as the backbone for streaming analytics.
- Cloud Dataflow (Streaming mode): As mentioned, Dataflow can process real-time streams using Apache Beam, enabling continuous ETL and analytics on incoming data.
- BigQuery (Streaming Inserts): BigQuery supports streaming data inserts, allowing you to append new records to tables in near real-time for immediate querying.
6. Business Intelligence (Power BI equivalent):
- Looker (Google Cloud’s BI Platform): Acquired by Google, Looker is a powerful BI and data analytics platform that allows users to explore data, build dashboards, and create data applications. It integrates natively with BigQuery.
- Google Data Studio (Looker Studio): A free tool for creating interactive dashboards and reports from various data sources.
7. Data Governance & Security:
- Cloud Data Catalog: A fully managed metadata management service that helps discover, understand, and manage all your data assets across GCP and hybrid environments.
- Identity and Access Management (IAM): Manages user access and permissions across all GCP services.
- Data Loss Prevention (DLP): Helps discover, classify, and protect sensitive data across your GCP services.
Summary:
GCP’s strength lies in its serverless offerings (BigQuery, Dataflow, Pub/Sub), which significantly reduce operational overhead and allow data engineers to focus more on data logic rather than infrastructure management. BigQuery is a standout feature, offering both data warehousing and powerful data engineering capabilities within a single, highly performant service.
While AWS provides deep customization, GCP often offers a slightly more integrated feel among its separate services, especially with BigQuery serving as a central hub for many analytics workflows. Both clouds provide robust ecosystems for building sophisticated data platforms.
OCI (Oracle Cloud Infrastructure)
When considering Oracle Cloud Infrastructure (OCI) for data engineering and a “Microsoft Fabric-like” environment, it’s important to note that OCI’s approach, like AWS and GCP, is also modular. Oracle leverages its deep expertise in databases and enterprise applications to offer a suite of services for data ingestion, processing, warehousing, and analytics.
Here’s a breakdown of OCI services that, when combined, can create a comprehensive data platform for data engineers:
1. Data Lake Storage (OneLake equivalent):
- Oracle Cloud Infrastructure Object Storage: This is OCI’s highly scalable, durable, and cost-effective object storage service. It serves as the foundation for your data lake, storing raw and processed data in various formats. It’s comparable to Amazon S3 or Google Cloud Storage.
2. Data Integration & ETL (Data Factory equivalent):
- Oracle Cloud Infrastructure Data Integration: A fully managed, serverless ETL service that allows you to design, deploy, and manage data integration processes. It offers both a visual interface and code-based transformations, with connectors to various data sources, both on-premises and in the cloud.
- Oracle GoldenGate: A comprehensive software package for real-time data integration and replication. It’s often used for Change Data Capture (CDC) to ingest transactional data into a data lake or data warehouse in near real-time.
- Oracle Data Safe: While primarily a security service, it has features for data masking and discovery that can be part of an ETL pipeline for sensitive data.
3. Data Engineering / Big Data Processing (Synapse Spark / Databricks Notebooks equivalent):
- Oracle Cloud Infrastructure Data Flow: A fully managed Apache Spark service within OCI. It allows data engineers to run Spark applications on demand without managing the underlying infrastructure, ideal for large-scale data processing and transformations.
- Oracle Cloud Infrastructure Big Data Service: A managed service for Apache Hadoop, Spark, Kafka, and other big data components. This provides more control over the big data ecosystem for those who need it.
- Databricks on OCI: Databricks also has a partnership with Oracle, offering its unified analytics platform (including Apache Spark and Delta Lake) to run on OCI’s infrastructure. This provides the familiar Databricks experience within the Oracle Cloud.
4. Data Warehousing (Synapse Data Warehouse equivalent):
- Oracle Autonomous Data Warehouse (ADW): A fully managed, self-driving data warehouse service. It’s optimized for analytical workloads, scales automatically, and handles patching, backups, and tuning without manual intervention. It’s built on Oracle’s strong database foundation.
- Oracle Exadata Database Service: While not solely a data warehouse, for very large and demanding on-premises or cloud-based data warehouse deployments, Exadata offers extreme performance and scalability.
5. Real-time Analytics (Synapse Real-time Analytics equivalent):
- Oracle Cloud Infrastructure Streaming: A fully managed, scalable, and durable streaming service compatible with Apache Kafka APIs. It’s used for ingesting and processing high-volume, real-time data streams.
- Oracle GoldenGate Stream Analytics: Provides real-time analytics on streaming data, allowing for immediate insights and actions.
6. Business Intelligence (Power BI equivalent):
- Oracle Analytics Cloud (OAC): A comprehensive cloud-based platform that provides self-service data visualization, enterprise reporting, and advanced analytics capabilities. It integrates well with Oracle databases and other data sources.
7. Data Governance & Security:
- Oracle Cloud Infrastructure Data Catalog: A metadata management service that helps discover, organize, and govern data assets across OCI and other data sources.
- Oracle Cloud Infrastructure Identity and Access Management (IAM): Manages user access and permissions across all OCI services.
- Oracle Data Safe: Provides data security, including auditing, data masking, and sensitive data discovery.
Summary:
OCI’s strength for data engineers lies in its strong database heritage, particularly with Autonomous Data Warehouse, which offers a robust and easy-to-manage solution for analytical workloads. OCI also provides a good set of managed services for Spark and data integration.
While OCI has been growing its cloud offerings, it generally has a smaller market share compared to AWS, Azure, and GCP, which might mean a smaller community or fewer third-party integrations for some niche tools. However, for organizations already heavily invested in Oracle technologies or looking for a potentially more cost-effective cloud solution for specific workloads, OCI presents a viable and increasingly competitive option.
What’s next
Over the next few weeks we’ll be diving into AWS, creating sample data engineering projects and integrating them with AWS hosted AI solutions in Bedrock. Be sure to like and subscribe to the blog for the latest updates!
Like the content? Donations are welcome.
