QA Consultant

Unlock yourself. Take your career to the next level.

At Atrium, we live and deliver at the intersection of industry strategy, intelligent platforms, and data science — empowering our customers to maximize the power of their data to solve their most complex challenges. We have a unique understanding of the role data plays in the world today and serve as market leaders in intelligent solutions. Our data-driven, industry-specific approach to business transformation for our customers places us uniquely in the market.

Who are you?

You are smart, collaborative and take ownership to get things done. You love to learn and are intellectually curious in business and technology tools, platforms and languages. You are energized by solving complex problems and bored when you don’t have something to do. You love working in teams, and are passionate about pulling your weight to make sure the team succeeds.

What will you be doing at Atrium?

We are seeking a highly skilled AI QA Consultant to ensure the quality, functionality, and performance of our AI solutions. This role is responsible for testing across the Salesforce platform with a specialised focus on validating AI-powered features, agentic workflows, and integrations with Large Language Models (LLMs). This position bridges traditional Salesforce QA with modern AI testing principles, requiring a deep understanding of Salesforce best practices, a keen eye for detail, and the technical acumen to test complex, non-deterministic systems. You will work closely with AI developers, Salesforce administrators, and business stakeholders to deliver robust and reliable AI-driven user experiences.

The QA Consultant Will
• Conduct thorough manual and exploratory testing of AI features on the Salesforce platform. You will identify, document, and track defects in Jira, clearly distinguishing between standard configuration bugs and nuanced, probabilistic issues inherent in AI model outputs
• Take ownership of data quality for the entire AI fine-tuning lifecycle. You will be responsible for validating and helping prepare the high-quality, unbiased datasets that are essential for building reliable and effective enterprise AI
• Develop detailed test plans, test cases, and test scripts specifically designed to evaluate the accuracy, relevance, safety, and performance of LLM responses and AI-driven logic
• Collaborate with product owners and AI developers to understand requirements and design comprehensive test strategies for Salesforce solutions that incorporate AI components like intelligent chatbots, RAG-based knowledge retrieval, automated case summarization, and agentic process automation
• Create scenarios to test for prompt injection vulnerabilities, model hallucinations, data grounding issues, and bias in AI outputs
• Assess the performance, latency, and resource consumption of AI features to ensure they meet user experience benchmarks
• Facilitate User Acceptance Testing (UAT) for AI-powered features. Guides business users on how to evaluate and provide feedback on intelligent systems
• Stay updated with the latest advancements in LLMs, prompt engineering techniques, and AI testing methodologies, and champion the adoption of best practices

In This Role, You Will Have
• Strong foundation with 2-3 years of dedicated QA experience on the Salesforce platform. It is crucial for validating how AI features interact with core Salesforce objects, data models, and business workflows
• A solid understanding of modern AI/NLP concepts, including Retrieval-Augmented Generation (RAG), model fine-tuning, and the principles of creating feedback loops for reinforcement learning (e.g., RLHF)
• Proficiency in Python, used for writing test scripts, data manipulation, and interacting with AI APIs
• Hands-on experience testing applications built with LLM APIs (like OpenAI, Gemini), frameworks such as LangChain/LlamaIndex, and various Vector Databases
• Working knowledge of SQL, NoSQL, and vector databases
• Excellent analytical and communication skills, with experience thriving in an Agile/Scrum environment