Data Analyst

**Key Responsibilities:** - Design and develop scalable batch and real-time data pipelines using Spark - Build and maintain distributed streaming systems using Kafka - Develop ETL/ELT workflows for large-scale structured and unstructured datasets - Optimize data processing jobs for performance - Implement data ingestion frameworks for high-volume data sources - Ensure data quality, integrity, and reliability across pipelines - Collaborate with cross-functional teams to define data architecture and solutions - Monitor, troubleshoot, and resolve data pipeline issues - Implement data security and governance standards **Required Skills & Qualifications** - Bachelors/Masters degree in Computer Science, Engineering, or related field - 4+ years of experience in data engineering or big data development - Strong hands-on experience with Apache Spark (Core, SQL, Structured Streaming) - Experience with Apache Kafka, Kafka Streams, and connectors - Proficiency in Python - Experience with distributed data processing and parallel computing - Strong knowledge of data structures and algorithms - Experience with ETL tools and workflow orchestration - Familiarity with data lake and data warehouse architectures - Experience with SQL and NoSQL databases **Preferred Qualifications** - Knowledge of modern data formats (Parquet, Avro, ORC) - Experience with CI/CD pipelines and DevOps practices - Familiarity with data governance and metadata management tools - Real-time analytics and event-driven architecture experience - Experience working in Agile/Scrum environments **Key Competencies** - Strong problem-solving and analytical skills - Excellent communication and teamwork abilities - Ability to work with large, complex datasets - Strong attention to detail - Self-motivated with ability to work independently Nice to Have - Experience with machine learning pipelines - Knowledge of data visualization tools - Certification in big data technologies Note: This is a third party job (Aggregated by careeruplift.ai). Shortlisting and Final hiring decision & process is handled by the company.