![]() Sound knowledge on Spark, HDFS/HIVE/HBASE, Shell Scripting, and Spark Streaming 8. Apache Kafka is a very popular system for message delivery and subscription, and provides a number of extensions that increase its versatility and power. Excellent knowledge of data backup, recovery, security and integrity 7. Hands on experience of working on databases such as Sql Servers, PostgreSql, Cloud infrastructure, etc. ![]() Good knowledge of SQL / NoSQL databases and data warehouse concepts 5. Combining Flume and Kafka allows Kafka to avoid custom coding and take advantage of Flumeās battle-tested sources and. Update to Kafka 2.0 client - Use slf4j in every component Bug - Flume 1.x build fails on Maven 2 - Add findbugs to Flume. Apache Flume 1.9.0 is production-ready software. Exposure to any or all latest data engineering ecosystem platforms such as AWS, Azure, GCP, Cloudera and Data bricks 3. Flafka includes a Kafka source, Kafka channel, and Kafka sink. Apache Flume 1.9.0 is the eleventh release of Flume as an Apache top-level project (TLP). ![]() 3 - 5 years relevant experience in data engineering 2. Drive customer communication during critical events and participate/lead various operational improvement initiatives Desired Candidate Profile : 1. Implement machine learning models on real time input data stream 10. Setup, administer, monitor, tune, optimize and govern large scale implementations 9. Design, implement, test and document performance benchmarking strategy for platforms as well as for different use cases 8. Telegraf is most likely a better option for this particular use-case since it nativly supports MQTT, Kafka and InfluxDB. ![]() Monitor job performances, manage file system/disk-space, cluster & database connectivity, log files, manage backup/security and troubleshoot various user issues 7. If you write some messages and press ENTER between, the messages should show up in your Kafka topic. Work with various teams to setup and manage users, secure and govern platforms and data and maintain business continuity through contingency plans (data archiving etc.) 6. Collaborate with various cross-functional teams: infrastructure, network and database 5. Knowledge of new components and various emerging technologies in on-premises and Cloud (AWS/Azure/Google) 4. Integrate multiple data sources to create data lake/data mart Perform data ingestion and ETL processes using SQL, Scoop, Spark or Hive 3. Design and implement data engineering projects. Intellectual Property Rights (IPR) Jobs. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |