Prompt Engineer & AI QA Analyst

Overview

Our client is seeking a Prompt Engineer & AI QA Analyst to help optimize and evaluate AI-generated content across a groundbreaking emotionally intelligent AI creativity platform. This role focuses on refining prompts, selecting language models, and using structured evaluations to ensure high-quality, coherent, and brand-aligned outputs. The position sits at the intersection of creativity and AI innovation, requiring strong analytical skills, writing ability, and a deep understanding of LLMs and prompt engineering workflows.

Challenges you would face

Evaluate AI outputs for clarity, relevance, and alignment with the brand’s voice
Refine and optimize prompts across various use cases to improve response quality and reduce costs
Collaborate with developers to assess and improve model performance
Select and compare language models that meet quality and performance standards
Develop structured metrics to track output effectiveness over time
Automate content evaluation using Python-based tools and scripts
Maintain clear documentation of prompt iterations, performance insights, and learnings
Design and implement AI agents leveraging prompt-based workflows
Build and maintain versioning systems for prompts
Integrate prompts and agents into broader product applications

The ideal candidate

Fluent in English with outstanding writing and editing skills
Detail-oriented and highly organized, capable of managing multiple prompt evaluations simultaneously
Familiarity with LLMOps tools such as Langsmith, Langfuse, and Opik
Experience working with generative AI models (GPT, Claude, Gemini, etc.) and prompt engineering
Strong Python skills for automation and evaluation scripting
Analytical mindset with experience in assessing AI output quality
Knowledge of model evaluation methods and selection processes
Understanding of reproducible AI/ML workflows, version control, and dataset tracking
Ability to work cross-functionally with engineers, product teams, and content stakeholders

Plus

Background in content moderation, editorial work, or NLP-based quality assurance

AI Prompt Engineer