Senior Data Engineer

Senior Data Engineer

Essential Duties and Responsibilities:

\n
• \n
  \n
  • Actively work with business and other BI stakeholders to understand their needs and solution enhancements to the data warehouse
  \n
  • Be available for addressing issues or bugs in the present implementation
  \n
  • Design, build, and maintain data infrastructure, ETL processes, and data pipelines.
  \n
  • Design, build, and maintain data pipelines using Databricks
  \n
  • Configure and implement data sourcing events on Kafka
  \n
  • Build and optimize python/Pyspark framework
  \n
  • Maintain Snowflake and AWS data infrastructure
  \n
  • Work with Unstructured/Semi-Structured data and build reporting tables
  \n
  • Assist the BI Analyst to manage client reporting requirements
  \n
  • Build Data models and assist the BI Analyst to offer data-driven insights
  \n
  • Design data models and automate manual processes.
  \n
  • Create and execute a test strategy to ensure robustness of data pipelines
  \n
  • Ensure effectiveness in infrastructure consumption by implementing solutions that are optimized and scalable
  \n
  • Foster an environment that emphasizes trust, open communication, creative thinking, and cohesive team effort, across the business, IT and vendor teams
  \n
\n

Deep Expertise in Databricks Spark:

\n
• \n
  \n
  • Proficient in Databricks Spark: Exceptional skills in Databricks Spark for sophisticated data processing. Proven experience in leveraging Spark for complex ETL tasks, surpassing traditional data processing methods.
  \n
  • ETL Pipeline Mastery: Demonstrated excellence in designing and implementing ETL pipelines specifically within Databricks. Ability to utilize Spark's full capabilities to create efficient, scalable data pipelines.
  \n
  • Data Transformation and Analysis: Expert in data transformation using Databricks Spark, skilled in performing advanced data analytics and processing large datasets with high efficiency.
  \n
  • Optimization Techniques: Deep understanding of optimizing Databricks Spark applications for maximum performance, including fine-tuning Spark configurations, and memory management.
  \n
\n

Integration with Confluent Kafka:

\n
• \n
  \n
  • Kafka Exposure: Solid background in working with Confluent Kafka, particularly in integrating it with Spark-based systems for real-time data streaming and processing.
  \n
  • Efficient Data Pipelines: Proficiency in creating and managing data pipelines that seamlessly integrate Kafka with Databricks Spark, ensuring efficient data flow and processing.
  \n
  • Supporting integrations with business applications, using Kafka
  \n
\n

DataOps and Agile Methodologies:

\n
• \n
  \n
  • DataOps Principles: Strong grasp of DataOps methodologies, with a focus on improving the efficiency and quality of data analytics via automation, collaboration, and process optimization.
  \n
  • Agile Development: Experienced in Agile software development practices, adept at implementing Agile methodologies like Scrum or Kanban in data-centric projects for improved collaboration and rapid delivery.
  \n
\n

Proficiency in SQL and Platform Integrations:

\n
• \n
  \n
  • Advanced SQL Skills: Expertise in SQL, particularly for querying and managing tables/data warehouses within Snowflake. Ability to seamlessly integrate these with Databricks Spark.
  \n
  • Platform Adaptability: Skilled in adapting to and integrating various data platforms and technologies, aligning them with strategic organizational goals.
  \n
\n

Collaborative and Best Practice-Oriented:

\n
• \n
  \n
  • Adherence to High Standards: Committed to maintaining high standards in code quality, documentation, and adhering to DataOps and Agile best practices.
  \n
  • Team Collaboration and Leadership: Ability to work collaboratively in a team, fostering a culture of continuous learning and improvement.
  \n
\n

Qualifications:

\n
• \n
  \n
  • Bachelor's or master's degree in computer science, statistics, or analytics.
  \n
  • Over 8 years of experience in the field of data engineering with capabilities on working through the entire development lifecycle
  \n
  • Ability to work with senior business stakeholders to be able to ascertain data needs out of business opportunities or challenges
  \n
  • Minimum of 5 years of cloud experience on any of the major cloud platforms preferably AWS
  \n
  • Hands-on Knowledge of Data Modeling, Warehousing and Power BI (or equivalent) as a Reporting Tool
  \n
  • Advanced proficiency in SQL with the ability to read and write queries is required
  \n
  • Demonstrable experience with Databricks, Snowflake, Python, and PySpark
  \n
  • Preferably to have experience with Kafka and Power BI
  \n
\n