ICF
United States
Quality Assurance Analyst (Salesforce) - Remote
- Confluence
- Jira
- Prototypes
- Saas
C
AI SDET (Software Development Engineer in Test) - Quality Engineering
Capital District
Full Remote
AI Summary
Caseware is seeking an AI SDET to pioneer and scale AI-driven testing practices for their cloud-native SaaS platform. This role involves developing AI-first quality strategies, designing AI-focused test automation, and ensuring the reliability, safety, and performance of intelligent features. The ideal candidate has extensive experience in quality engineering with a focus on AI/ML/LLM systems.
Key Highlights
Pioneer and scale AI-driven testing practices from the ground up.
Develop AI-first quality strategy for cloud-native SaaS architecture.
Design and implement AI-focused test automation for LLM-based and agentic systems.
Key Responsibilities
Evolve a modern, AI-first quality strategy for our fast-scaling SaaS architecture, including foundational infrastructure and emerging agentic/intelligent systems.
Integrate AI enhancements into CI/CD pipelines (e.g., predictive flakiness detection, automated test generation, self-healing scripts) to improve isolation, data setup, & execution reliability using existing/suggesting tools.
Establish scalable testing practices that support hyper-growth and petabyte-scale AI data pipelines.
Design deterministic and statistical testing approaches for non-deterministic LLM-based and agentic systems, addressing hallucinations, prompt injection, bias, drift, and safety risks.
Build automated evaluation pipelines and harnesses for correctness, faithfulness, retrieval quality, generation accuracy, tool-calling, planning sequences, and multi-agent flows.
Execute/Develop test frameworks for the full AI lifecycle: prompts, datasets, embeddings, model versions, RAG pipelines (end-to-end validation), and guardrails.
Implement red-teaming, bias/fairness checks, and compliance mechanisms; leverage in trend frameworks for metrics and observability.
Integrate AI-specific quality signals into CI/CD for automated gating and continuous monitoring.
Partner closely with product, data science, AI engineering, and dev teams to test AI features, conduct multi-agent simulations, and ensure high-quality roadmap delivery.
Facilitate knowledge sharing and upskilling on AI testing best practices across the Quality Function.
Drive core metrics (DORA, test coverage/effectiveness) plus AI-specific indicators (e.g., hallucination rate, context precision, drift detection).
Build real-time dashboards and support A/B testing of models with post-deployment monitoring.
Champion a quality-first, ethical AI mindset organization-wide.
Mentor SDET’s, lead workshops on AI risks/validation, and influence design/deploy/incident processes.
Define roadmaps and best practices for sustainable AI quality assurance.
Interested in remote work opportunities in QA & Testing? Discover QA & Testing Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.
Technical Skills Required
Benefits & Perks
Competitive compensation
Competitive salary
Comprehensive benefits
Health insurance
Retirement plans
Flexible work options
Remote opportunities
Generous time-off policies
Work-life balance
Global opportunities
Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.
Nice to Have
Knowledge of performance, Stress and Load testing tools like K6, JMeter, Blazemeter
Knowledge of observability (NewRelic), statistical testing methods, red-teaming, and ethical AI practices.
Certifications (e.g., ISTQB AI Testing)
Job Description
Caseware is one of Canada's original Fintech companies, having led the global audit and accounting software industry for over 30 years, with more than 500,000 users across 130 countries and available in 16 different languages. While you might not have heard of us (yet) over 36,000 accounting and audit professionals list Caseware as a skill on their LinkedIn profiles!
Want the full job description?
Read the complete details on LinkedIn, the original posting.
Continue on LinkedIn
This is a short excerpt. All rights to the full description belong to its original publisher.
Keep exploring
Similar Jobs
Explore other opportunities that match your interests
Talenthop
United States
Test Automation Engineering Lead (Fully Remote)
- AI
- Ai Tools
- Cloud
- Cloud Native
Itrex
Poland
QA Engineer
- AI
- Ai Tools
- Android
- Blockchain
Remote Work Guides
Practical guides for finding and thriving in remote roles.
How to Get a Remote Job with No Experience
Getting a remote job with no experience is possible if the job seeker knows where to look and how to present their skills. Entry‑l...
Read Article
Do Remote Jobs Provide Equipment?
You might be wondering if your new employer will pay for your home office setup or if you’ll have to cover the costs yourself. The...
Read Article
What Are the Highest-Paying Remote Jobs?
If you’re looking for a career that offers a great income and the ability to work from anywhere, you’re not the only one. The job...
Read Article
Which Companies Offer Remote Jobs?
Advances in technology have made remote jobs more popular than before. Many companies now allow employees to work from home either...
Read ArticleSubscribe to our newsletter
New visa sponsorship, relocation and remote tech jobs, delivered to your inbox.