Job Description

We are hiring a Spark Developer to design, develop, and maintain large-scale data processing systems using Apache Spark . This remote role provides the opportunity to work on cutting-edge big data technologies, optimizing and processing massive datasets to support analytics and business decisions.

• Develop, test, and maintain scalable data pipelines using Apache Spark for batch and real-time data processing.
• Work with large datasets, ensuring they are efficiently processed and made available for analytics and machine learning models.
• Integrate Spark with other big data technologies like Hadoop , Kafka , and Hive to build robust data processing solutions.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and ensure the successful implementation of data pipelines.
• Write efficient, high-performance code to process large amounts of data in real-time or batch mode.
• Ensure the performance, scalability, and security of data pipelines.
• Perform data transformations, aggregations, and analysis using Spark SQL and DataFrames .
• Optimize and troubleshoot Spark applications for performance and resource efficiency.
• Document and maintain technical specifications and workflows.

• Strong proficiency in Apache Spark (with PySpark , Spark SQL , and Spark Streaming ).
• Experience with Hadoop ecosystem components such as HDFS , Hive , and YARN .
• Expertise in SQL and NoSQL databases like MongoDB , Cassandra , or HBase .
• Proficiency in programming languages like Java , Scala , or Python .
• Hands-on experience with Kafka , AWS , Azure , or Google Cloud (for cloud-based data processing).
• Strong knowledge of data warehousing concepts, ETL processes, and data modeling.
• Experience in performance tuning of Spark applications.
• Ability to work with distributed systems and big data technologies.
• Excellent problem-solving and debugging skills.
• Bachelor's degree in Computer Science, Engineering, or a related field.
• 2+ years of experience in Spark-based data development and processing.

Job Tags

Similar Jobs