Senior Data Engineer (Equity - Only)

Job description

Job Title: Senior Data Engineer

About Luupli:

Luupli is a social media app that has equity, diversity, and equality at its heart. We believe that social media can be a force for good, and we are committed to creating a platform that maximise the value that creators and businesses can gain from it, while making a positive impact on society and the planet. Luupli started internal testing since June 2024 and getting ready for a commercial BETA testing from December 2024, with the hope of launching fully summer of 2025.Our team is made up of passionate and dedicated individuals who are committed to making Luupli a success.

About the Role:

We’re seeking a skilled Senior Data Engineer with experience in recommendation systems to join our team. This role is pivotal in enhancing our backend systems, managing databases, and optimizing recommendation algorithms. You’ll play a critical role in the architecture and performance of our recommendation infrastructure, ensuring accurate, relevant, and efficient recommendations for our users.

Responsibilities:

Data and Database Management:

• Design, optimize, and maintain tables and data structures to support recommendation and trending content data.

• Work with structured data storage solutions, including PostgreSQL and JSONB, to manage recommendation and interaction data.

Recommendation Systems:

• Implement and refine recommendation algorithms (e.g., collaborative filtering, content-based, and hybrid approaches) to enhance relevancy.

• Use similarity search libraries like Annoy or Faiss to optimize recommendation speed and accuracy.

• Continuously evaluate recommendation logic to better serve user preferences, ensuring real-time delivery.

Data Aggregation and Analysis:

• Aggregate, analyze, and process user interaction data to support recommendations and trending content.

• Design efficient queries and implement aggregation methods to capture relevant data and insights for recommendations.

Debugging and Optimisation:

• Identify, troubleshoot, and resolve data handling issues to ensure accurate recommendation delivery.

• Optimise queries, processing workflows, and containerised services for high performance and scalability within AWS ECS.

Requirements:

• Proven experience as a Data Engineer or Backend Engineer, with a focus on recommendation systems.

• Proficient in SQL and database management, especially with PostgreSQL and JSONB for structured data handling.

• Solid understanding of recommendation algorithms (collaborative filtering, content-based, hybrid approaches).

• Experience with similarity search libraries such as Annoy or Faiss.

• Strong programming skills in Python, with experience in building backend logic for data-intensive applications in a containerised environment.

• Familiarity with AWS ECS for container management, including task scheduling and scaling.

• Experience using AWS EventBridge to trigger workflows or automate tasks in response to application events.

• Analytical skills for data aggregation, querying, and insights generation.

• Strong debugging and optimisation skills for handling large-scale data processing in cloud-based environments.

Preferred Qualifications:

• Knowledge of data aggregation pipelines, ETL processes, and data handling at scale.

• Familiarity with additional AWS services (e.g., S3, Lambda) for data storage and event-driven architectures.

• Experience with machine learning libraries or tools used in recommendation systems.

Compensation:

This is an equity-only position, offering a unique opportunity to gain a stake in a rapidly growing company and contribute directly to its success.