AlgoShack
AI Test Engineer
Location unstated
hirly Pro
Get in front of the people hiring at AlgoShack
- Reach the hiring manager at AlgoShack on LinkedIn, from your own account
- hirly applies to this job for you, with unlimited applications
- A resume and cover letter written for this job, unlimited
- The hiring manager and recruiter at AlgoShack, with contact details
$49 a month · ₹1,999 for 30 days in India · cancel any time
Get past the screening software and onto a recruiter's desk
hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.
- Keywords matched to this posting
- Fit score before you apply
- Cover letter included
Matched against 2M live jobs from 210,000+ employers in 190+ countries.
Tailor my resume for this job →Apply from your AI assistant
Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.
Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.
hirly's read of this role
- Role family
- Engineering
- Seniority
- Mid level
- Country
- IN
- Work mode
- On-site / unstated
- First seen by hirly
- 18 Nov 2026
Derived automatically from the posting. Upload your resume above to see how the role scores against it.
the posting
AI Test Engineer:
The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change , and who can build that testing setup alone.
What this person will actually do
- Build automated accuracy checks for the AI
- Make those checks a gate in the CI pipeline ,]
- Build and maintain test datasets
- Test the whole pipeline end to end :
- Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
- Run safety and robustness testing : prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
- Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
- Strong Python test automation : pytest, API testing, CI pipelines (GitLab).
- LLM evaluation hands-on : has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
- Experience with at least one eval or tracing tool : LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
- Understands non-deterministic testing : how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
- Document or vision AI exposure : OCR or extraction from PDFs and images is a big plus for us.
- Solo-capable : has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
- LangChain / LangGraph. Useful, but not essential : our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
- Google Cloud / Vertex AI, Django APIs, Playwright.
- Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
- Performance and cost testing of LLM calls (latency, tokens).
Listed on hirly, a job board. hirly is not the employer: AlgoShack is hiring for this role.
Similar jobs
- Automation Test Engineer (2-4 yrs exp in Selenium & Playwright)Gartner · IndiaFirst seen today
- Solution Test Engineer - 8yrs - Python, PyATS, Switching/Routing/Data CenterCisco · Bangalore, Karnataka, IndiaFirst seen today
- Software Test Engineer – IP Routing & AI/Data Center NetworksNokia Corporation · IndiaFirst seen today
- Software Test EngineerValtech · India - RemoteFirst seen todayremote
- VoIP Test EngineerVonage · Bangalore, IndiaFirst seen todayremote
Browse similar roles
Want this one?
Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.
Tailor my resume for this job