Role: Data Engineer
- Understand & provide innovative solutions to business and product requirements using Big Data Architecture
- Take ownership of end-to-end data-pipeline including system design and integrating required
- Big Data tools & frameworks
- Implementing ETL processes and constructing data warehouse (HDFS, S3, etc.) at scale
- Developing highly performant Spark jobs for deriving data insights and building user preference
- Developing required querying and reporting tools for various business teams
- Working knowledge of Linux systems and distributed systems is a must.
- Knowledge of scripting language like python/scala
- Hands-on experience with writing Spark or Map-Reduce jobs and proficient understanding of distributed computing principles.
- Experience with integration of data from multiple data sources
- Experience with various messaging systems, such as Kafka
- Preferred:Understanding of Spark and HDFS internals.
- Experience with Lambda Architecture and building required infrastructure.
- Experience with Big Data querying tools, such as Hive, Pig, and Impala
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Experience with resource managers such as YARN, Mesos
- Experience with stream-processing systems such as Storm, Spark-Streaming, etc.
- Knowledge of Lucene, SOLR, Elastic Search or any other similar technology is a plus.
- Prior experience in developing segmentation and recommendation systems is recommended.
- Experience with reporting tools like Apache Zeppelin.