We currently seeking a Senior Data Engineer with 5-7 years’ experience. The ideal candidate would have the ability to work independently within an AGILE working environment and have experience working with cloud infrastructure leveraging tools such as Apache Airflow, Databricks, DBT and Snowflake. A familiarity with real-time data processing and AI implementation is advantageous.
n
Responsibilities:
- Design, build, and maintain scalable and robust data pipelines to support analytics and machine learning models, ensuring high data quality and reliability for both batch & real-time use cases.
- Design, maintain, optimize data models and data structures in tooling such as Snowflake and Databricks.
- Leverage Databricks for big data processing, ensuring efficient management of Spark jobs and seamless integration with other data services.
- Utilize PySpark and/or Ray to build and scale distributed computing tasks, enhancing the performance of machine learning model training and inference processes.
- Monitor, troubleshoot, and resolve issues within data pipelines and infrastructure, implementing best practices for data engineering and continuous improvement.
- Diagrammatically document data engineering workflows.
- Collaborate with other Data Engineers, Product Owners, Software Developers and Machine Learning Engineers to implement new product features by understanding their needs and delivery timeously.
