AI Big Data Engineer
1301 West Long Lake Road Troy, MI 48098 US
Job Description
NO C2C
Main Responsibilities
• Build automated pipelines to extract and process data from a variety of legacy platforms (predominantly SQL Server), e.g., in stored procedures, Glue processing, etc.
• Implement data-related business logic on modern data platforms, such as AWS Glue, Databricks, and Azure using best practices and industry standards.
• Create vector databases, data marts and the data models to support them
• Optimize and monitor the performance, reliability, and security of data systems and processes.
• Integrate and transform data from (or to) various sources and formats, such as structured, unstructured, streaming, and batch.
• Develop and maintain data quality checks, tests, and documentation.
• Support data analysis, reporting, and visualization using tools such as SQL, Python, Tableau and Quicksight
• Research and evaluate new data technologies and trends to improve data solutions and existing capabilities.
Qualifications And Skills
• Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related field • At least 5 years of experience in data engineering or a similar role (previous DBA experience is a plus)
• Experience with big data frameworks and tools, such as Spark, Hadoop, Kafka and Hive
• Expert in SQL, including a knowledge of efficient query and schema design, DDL, data modeling and use of stored procedures
• Proficient in at least one programming language, such as Python, Go or Java • Experience with CI/CD, containerization (ex: docker, K8s) and orchestration (ex: Airflow)
• Experience building production systems with more modern ETL, ELT and data systems, such as AWS Glue, Databricks, Snowflake, Elastic, and Azure Cognitive Search
• Experience deploying data infrastructure on cloud platforms (AWS, Azure, or GCP)
• Strong knowledge of data quality, data governance, and data security principles and practices
• Excellent communication, collaboration, and problem-solving