hirly

AlgoShack

AI Test Engineer

Location unstated

hirly Pro

Get in front of the people hiring at AlgoShack

  • Reach the hiring manager at AlgoShack on LinkedIn, from your own account
  • hirly applies to this job for you, with unlimited applications
  • A resume and cover letter written for this job, unlimited
  • The hiring manager and recruiter at AlgoShack, with contact details
Get Pro for this job →

$49 a month · ₹1,999 for 30 days in India · cancel any time

See how you match this job — and similar ones. Free.

Upload your resume and hirly scores it against this role at AlgoShack first, then against similar open jobs, and shows where you fit and why.

PDF or DOCX, up to 12MB. No sign-up to see your matches.

Get past the screening software and onto a recruiter's desk

hirly rewrites your resume for this job — matching the keywords and skills in the posting, moving your most relevant experience to the top, and writing a cover letter to fit. About 30 seconds.

  • Keywords matched to this posting
  • Fit score before you apply
  • Cover letter included

Matched against 2M live jobs from 210,000+ employers in 190+ countries.

Tailor my resume for this job →

Apply from your AI assistant

Connect hirly to Claude or ChatGPT and ask it to apply to this job. hirly tailors your resume, fills the employer’s form and asks before sending.

Some employer sites stop an application at a CAPTCHA or sign-in and hand it back with a link. Applying needs a paid plan. Works with any assistant that supports MCP.

hirly's read of this role

Role family
Engineering
Seniority
Mid level
Country
IN
Work mode
On-site / unstated
First seen by hirly
18 Nov 2026

Derived automatically from the posting. Upload your resume above to see how the role scores against it.

the posting

AI Test Engineer:

The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change , and who can build that testing setup alone.

What this person will actually do

  • Build automated accuracy checks for the AI
  • Make those checks a gate in the CI pipeline ,]
  • Build and maintain test datasets
  • Test the whole pipeline end to end :
  • Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
  • Run safety and robustness testing : prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
  • Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).

Must-have skills (screen hard on these)

  • Strong Python test automation : pytest, API testing, CI pipelines (GitLab).
  • LLM evaluation hands-on : has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
  • Experience with at least one eval or tracing tool : LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
  • Understands non-deterministic testing : how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
  • Document or vision AI exposure : OCR or extraction from PDFs and images is a big plus for us.
  • Solo-capable : has set up a test approach from scratch, not only executed someone else’s.

Good-to-have

  • LangChain / LangGraph. Useful, but not essential : our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
  • Google Cloud / Vertex AI, Django APIs, Playwright.
  • Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
  • Performance and cost testing of LLM calls (latency, tokens).
Original posting on AlgoShack's site ↗

Listed on hirly, a job board. hirly is not the employer: AlgoShack is hiring for this role.

Browse similar roles

Want this one?

Upload your resume and hirly rewrites it for this job and writes the cover letter — in about thirty seconds, before you sign up.

Tailor my resume for this job