Medior AI Engineer

Mission Overview

Keystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site. You will be part of the AI Center of Excellence, focusing on the Run phase of AI services in production. This role involves ensuring the reliability, security, and cost-efficiency of AI services built on a hybrid operating model, utilizing both managed Azure services and self-hosted open-source components.

Key Responsibilities

• Operate the AI services in production: monitor availability, latency, and errors, handle incidents and service requests, analyze root causes, then fix or escalate.
• Keep answer quality under control: run and maintain evaluation sets with business content owners, track answer quality, non-answer rate, and user feedback, and detect and investigate quality regressions.
• Operate the ingestion pipelines: manage incremental refresh, failed documents, index health, and onboarding of new content sources or business domains.
• Operate agent integrations: monitor tool and integration calls to business systems, handle failures and timeouts, and review action logs and exceptions.
• Manage change safely: model version upgrades and retirements, prompt and configuration changes, and dependency and security updates, validated against evaluation sets before release.
• Follow run cost: monitor token consumption and cost per service, alert on drift, and propose optimizations.
• Maintain Terraform, Helm, and Azure DevOps artifacts, ensuring consistency across DEV, TST, ACC, and PROD environments.
• Apply data protection rules: manage trace retention, erasure, and access restrictions as defined in each service’s data protection impact assessment.
• Maintain runbooks and documentation, and report service health to service owners and the AI CoE regularly.
• Deliver small enhancements and fixes; structural changes and new features will be handled by the Build team.

Why Join Keystone Solutions?

As a consultant with Keystone Solutions, you will experience:

• Consultancy Nature: Engage in on-site work, embodying the consultant identity.
• Dynamic Projects: Tackle diverse challenges across various client environments.
• Turbo-Charged Learning: Benefit from professional development and broad learning experiences.
• Skyrocketing Ambition: Enjoy a commitment to career growth within our framework.
• Values: Being a "K-Stone" means bringing our core values to every engagement.

Duration: 04/01/2027 - 31/12/2027 12 months • (full time)

Skills required

• Azure DevOpsIntermédiaire · pratiqué cette année
• Azure Kubernetes Service - AKSIntermédiaire · pratiqué cette année
• Azure OpenAI ServiceIntermédiaire · pratiqué cette année
• LangChainIntermédiaire
• LLM / Large Language ModelsIntermédiaire · pratiqué cette année
• PythonIntermédiaire · pratiqué cette année
• RAG (Retrieval-Augmented Generation)Intermédiaire · pratiqué cette année
• TerraformIntermédiaire · pratiqué cette année
• Vector databasesIntermédiaire · pratiqué cette année

Language requirements

• Néerlandais / FrançaisNative or bilingual proficiency
• AnglaisCompétence professionnelle complète