Automation Test Engineer (Must Have:- Databricks, AWS Glue, S3, Spark, Airflow)

Automation Test Engineer

• Qualifications: 
• The Sr. Test Automation Engineer typically requires five or more years of test automation, data science, or software engineering experience.
• The Lead Test Automation Engineer typically requires seven or more years of relevant technical or business work experience and five or more years of development or test automation experience. 
• Some domain knowledge of the Supervision and Regulation function and the associated systems.
• Experience with data engineering including knowledge of Big Data ecosystem preferred.
• Strong desire and motivation to take up potential challenges and opportunities as part of this exciting large and complex program with system wide visibility.
• Understanding of the automation strategies & techniques available for testing data, application, and CI/CD infrastructure.
• 2+ years of experience with a modern data stack (Object stores like S3, Spark, Airflow, Lakehouse architectures, real-time databases)
• Experience with Python code
• 2+ years of experience with test tools (Great Expectations, Lambda test, Selenium, Cucumber)

Responsibilities:

• As a member of the Agile delivery squad, actively participate in the various Agile rituals following the Scaled Agile processes and tools set forth by the CDP Program team.
• Determine, coordinate, and create necessary tests to ensure data and pipeline quality.
• Lead design, build and testing of automated frameworks including the coding of reusable components and functions, and integration of test frameworks.
• Lead automated test strategies and guide decisions to successful implementation and completion.
• Provide traceability from business requirements and technical specifications to test conditions and defect reports.
• Collect metrics to measure test progress and quality outcomes.
• Coach and mentor others on test criteria, functional test, test automation and quality engineering methods and techniques.

Key Activities

• Design, develop and maintain robust, and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted and quality data from disparate data sources into Common Data Platform and making it available for comprehensive business analysis.
• Good understanding of data architecture, information security, data governance, develops processes to enforce standards.
• Proactively identifying and resolving issues with data pipelines and analytical data stores
• Deploying, monitoring and alerting for the data pipelines, data stores and implementing auto remediation wherever possible to ensure the availability and reliability of data systems.
• Employing security, testing and automation first strategy and adhering to data engineering best practices.
• Collaborating with cross-functional teams, including product mgmt., data scientists, analysts, and business stakeholders, to understand their data requirements and provide them with the necessary infrastructure and tools.
• Keeping up with the latest trends and technologies, which include evaluating and recommending new tools, frameworks, and technologies to improve data engineering processes and efficiencies.
• Lead the team through continuous improvement of production operations.
• Offer technical support where needed and developing automation software to speed incident resolution.
• Stay current with industry trends and source new ways for our business to improve.
• Implement, configure, and operate tools and products in the DevOps and DevSecOps Toolchain.
• Building and maintaining tools, services, and automations associated with deployment and our operations platform, ensuring that all meet our customer service standards and reduce errors.
• Continually evaluate our systems and tooling, so that they can accommodate growth and continually changing requirements.
• Actively troubleshoot any issues that arise in production.
• Update our processes/documentation and design new tools and processes as needed.
• Deploy product updates as required while implementing integrations when they arise.
• Automate our operational processes as needed, with accuracy, and compliant with security standards