The data engineer will require high level competence in Data & IT development and have ability to strategize among proven technologies to meet critical business needs.
To be considered for this high-skilled data engineer role you will need to demonstrate.
Create and maintain optimal data pipeline architecture. Creating and operating in Hive Metastores; parquet, XML etc.
Knowledge with data ingestion & data preparation via Apache Nifi / HDF.
Data administration, auditing and security on Hadoop platform
Development of custom Nifi processors in JAVA.
Modelling for HDP and Hadoop search platforms.
Data modelling for HDP and elastic search platforms.
Writing SQL queries for hive and spark.
Apache Storm and Kafta.
Apache Spark and resilient distributed datasets (RDD) for in-memory processing.
Data transformation, enrichment in Nifi & HDF