GCP Lead Data Engineer
San Jose, CA
Position: Data Engineer with extensive experience in building data integration pipelines in CI/CD model
Experience: Lead 12 + years , Sub-Leads 7+ years of experience
• Ability to design and develop a high-performance data pipeline framework from scratch
o Data ingestion across systemso Data quality and curationo Data transformation and efficient data storage
o Data reconciliation, monitoring and controls
• Support reporting model and other downstream application needs
o Skill in technical design documentation, data modeling and performance tuning applications
o Lead and manage a team of data engineers, contribute towards code reviews, and guide the team in designing and developing convoluted data pipelines adhering to the defined standards.
o Be hands on, performs POCs on the open source/licensed tools in the market and share recommendations.
o Provide technical leadership and contribute to the definition, development, integration, test, documentation and support across multiple platforms (GCP, Python, HANA)
o Establish a consistent project management framework and develop processes to deliver high quality software, in rapid iterations, for the business partners in multiple geographies
o Participate in a team that designs, develops, troubleshoots, and debugs software programs for databases, applications, tools etc.
o Experience in balancing production platform stability, feature delivery and reduction of technical debt across a broad landscape of technologies.
o Skill in the following platform, tools and technologies
• GCP cloud platform – GCS, Big Query, Streaming (pub/sub), data proc and data flow
• Python, PYSpark, Kafka, SQL, shell scripting & Stored procs
• Data warehouse, distributed data platforms and data lake
• Database definition, schema design, Looker Views, Models
• CI/CD pipelineo Proven track record in scripting code in Python, PySpark and SQL
o Excellent structured thinking skills, with the ability to break down multi-dimensional problems
o Ability to navigate ambiguity and work in a fast-moving environment with multiple stakeholders
o Good communication skills and ability to coordinate and work with cross functional teams.
From:
Rajesh,
iTech US Inc
rajesh.p@itechus.net
Reply to: rajesh.p@itechus.net