C2C
Anywhere
Posted 8 hours ago

Job Title: Data Engineer 1

Location: San Francisco, CA – Onsite (MUST be local to San Francisco Bay Area)

Duration: 2 years

Project Scope

Provide professional services for transition from Oracle Business Intelligence to Microsoft Power BI for interactive dashboards and enterprise reporting.
Provide professional services to modernize the data architecture to align with cloud-first strategies and leverage cloud technologies like Snowflake, Microsoft Synapse, or Microsoft Fabric.
Perform Data Model/Metadata model conversion from Oracle Business Enterprise Edition RPD to Power BI data model
Rebuild OBIEE Dashboards reports to interactive Power BI Dashboards/Reports.
Perform the transition from Oracle Data Integrator (ODI) to Azure Data Factory (ADF) by rebuilding and modernizing Extract Load Transform (ELT)/Extract Transform Load (ETL) workflows.

Mandatory Qualifications / Skills:

Converts ODI Mappings to ADF workflows, implementing incremental loads, Change Data Capture (CDC), and performance tuning strategies.
Design and develop scalable ETL/ELT pipelines in Azure Data Factory (ADF) to migrate data from Oracle to cloud data warehouse.
Work closely with BI Solution Architect and BI Data Engineer to structure fact-dimension models for analytics and reporting in Power BI.
Troubleshoot and optimize SQL queries, dataflows, and ETL performance, ensuring efficient and cost-effective cloud data processing.

Roles/Responsibilities:

Analyze Current Environment A thorough assessment of the existing Oracle Data Integrator, Oracle Database, and OBIEE Rapid File Database (RPD) environment will be conducted, including:

Detailed plan with conversion steps, dependencies, and validation strategy.
OBIEE RPD Assessment – Identifying subject areas, data source mappings, and conversion complexity.
Data Models – Analyzing table relationships, fact/dimension models.
Security & Access – Identifying existing security roles, policies, and groups.
Performance Metrics – Evaluating pipeline execution efficiency and bottlenecks.
ETL/ELT Pipelines – Identifying dependencies and process flow.
Data Sources & Integrations – Documenting relational/non-relational databases.

Migrate data from Oracle to cloud data warehouse

Data Migration Process:

Extract historical and incremental data from Oracle.
Transform data using Azure Data Factory (ADF) or equivalent orchestration tools.
Load data into cloud data warehouse using bulk data movement strategies.
Validate migrated data against Oracle to ensure accuracy and completeness.
Optimize partitioning, indexing, and clustering in the target data warehouse.

Data Warehouse Design:

Implement staging, raw, and transformed layers for optimized data storage.
Design and transition of Oracle schemas and tables to cloud data warehouse

Convert PL/SQL transformation logic to cloud data warehouse and leverage built-in streaming and automation capabilities.

Migration from ODI to Azure Data Factory (ADF)

Topology – document Physical Architecture, Contexts and Logical Architecture etc.
Convert ODI workflows to ADF Pipelines with equivalent or enhanced functionality.
Rebuild parameterized pipelines to accommodate reusable ETL logic.
Configure linked services to integrate data sources/destinations cloud data warehouse
Develop logging, error handling, and monitoring mechanisms using Azure Monitor and Log Analytics.
Optimize workflows for scalability and cost-efficiency (parallelism, trigger scheduling).
Implement Change Data Capture (CDC) for incremental loads where applicable.

OBIEE RPD to Power BI Data Model Migration

Extract and document RPD metadata; including logical models, subject areas, and presentation layers.
Map OBIEE RPD relationships to Power BI datasets and dataflows.
Convert logical models to physical models where applicable for optimized performance.
Ensure role-based access security mappings are transitioned correctly.
Validate Power BI data models against OBIEE reports for accuracy.

OBIEE Dashboards to Power BI Dashboards Migration

Recreate OBIEE dashboards and reports in Power BI with modern interactive capabilities.
Implement row-level security (RLS) and access control policies in Power BI.
Enhance report performance using Power BI Direct Query and Import modes.
Optimize layouts, interactivity, and navigation for a seamless end-user experience.
Train business users on self-service analytics and report customization.

Knowledge Transfer

Deliver Comprehensive Knowledge Transfer sessions on:

Developing Power BI dashboards and reports.
Managing and maintaining the new post-deployment system.
Post-implementation support for issue resolution and system enhancements.

Tasks

OBIEE Expertise

Proficient in OBIEE architecture and components, including:
1. 1. Repository (RPD) design: physical, business model, and presentation layers.
  2. OBIEE dashboards and reports development.
Understanding of OBIEE security (users, roles, and object-level security).
Ability to analyze OBIEE RPD and XML data for metadata migration.

Power BI Expertise

Strong experience in:

Designing Power BI reports and dashboards.
Data modeling in Power BI, including relationships, calculated columns, and measures using DAX (Data Analysis Expressions).
Power Query for data transformation and cleansing.
Configuring Power BI gateways for on-premises data access.
Managing Power BI Service, including workspace setup, role-based security, and publishing reports.

Data Engineering & ETL Development

Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
Implement data ingestion, transformation, and orchestration workflows from multiple sources (SQL, NoSQL, APIs, on-premises, cloud).
Optimize ADF data flows for performance, cost, and scalability.
Design and implement data warehousing solutions.

Performance Tuning & Optimization

Monitor and optimize ADF pipeline performance (troubleshooting failures, improving execution times, reducing costs).
Optimize Synapse performance through indexing, distribution strategies, and caching techniques.
Implement best practices for data partitioning, compression, and parallel processing.

Integration with Azure Ecosystem

Automate data pipelines with Azure Functions, Logic Apps, and Power Automate.
Leverage Azure Monitor, Log Analytics, and Application Insights for observability and troubleshooting.