Software Data Engineer (Pyspark exp required)

Job: Full Stack Software Data Engineer

Location: Atlanta, GA (relocation assistance offered)

Work Authorization Requirements:

• U.S. Citizenship or U.S. Lawful Permanent Residency (Green Card) is strongly preferred.
• Candidates on H-1B visa may be considered if they have an approved I-140 and are eligible for extensions beyond the 6-year limit.
• We are not able to sponsor new H-1B petitions at this time.

How you will create impact:

As the Software Data Engineer Senior Developer for GenAI, you will lead the design, development, and operationalization of cutting-edge cloud-native big data solutions that enable innovative AI models and analytics. You’ll collaborate across teams to deliver data solutions that drive business insights and value, all while ensuring scalability and sustainability. This is an exciting opportunity to shape the future of data engineering at our organization, guiding teams through complex data challenges and enabling AI-powered decision-making that will fuel business growth.

Your responsibilities include:

• Collaborating with stakeholders to identify and deliver innovative data solutions that meet the evolving needs of the business.
• Building and maintaining cloud-native big data solutions, advanced analytics products and GenAI models.
• Partnering with data architects and senior engineers to implement best practices, ensuring sustainable and scalable technical designs.
• Working closely with analytics, application development, and data science teams to foster collaboration and drive innovation.
• Delivering on-time, high-quality results on fast-paced schedules while maintaining rigorous standards for development and code management.

What is needed to be successful:

• Design, develop and implement data processing pipelines using Python and Spark.
• Minimum of 4 years of professional software development experience.
• Strong proficiency in Python, hands-on experience with Apache Spark and PySpark for large-scale data processing.
• Write clean, efficient, and well-documented Python and TypeScript code.
• Experience with testing frameworks (pytest, unittest).
• You are proficient in cloud-native platforms and tools, including Hadoop, HDFS, Spark, Kubernetes, and ElasticSearch, to deliver high-performance data solutions.
• Experience with containerization technologies (Docker) and orchestration tools (Kubernetes, Docker Compose) to enable parallel executions of tasks.
• You have a strong background in deploying and maintaining AI/ML models, with hands-on experience using platforms like Palantir Foundry, Snowflake, and AWS SageMaker.
• Build and maintain scalable and reliable APIs to expose data and functionalities to other services and applications.
• Identify and implement performance optimization for existing code and data processing workflows, including leveraging containerization technologies for parallel execution.
• Contribute to the design and architecture of our data platform and applications.
• Participate in code reviews to ensure code quality and adherence to best practices.
• Experience with CI/CD pipeline, including containerized application, code versioning capabilities such as GitHub and Jenkins.
• Experience setting up E2E Engineering pipeline, monitoring, and troubleshooting.

Ready to take your career to the next level? We would love to hear from you.