Skip to main content
POST
/
v2
/
evaluators
/
custom-evaluator
Create a custom LLM-as-a-judge evaluator
curl --request POST \
  --url https://api.traceloop.com/v2/evaluators/custom-evaluator \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "input_schema": [
    {
      "name": "<string>",
      "type": "string",
      "description": "<string>",
      "enum_values": [
        "<string>"
      ],
      "label": "<string>"
    }
  ],
  "messages": [
    {
      "content": "<string>",
      "role": "<string>",
      "tool_call_id": "<string>"
    }
  ],
  "model": "<string>",
  "name": "<string>",
  "output_schema": [
    {
      "name": "<string>",
      "type": "string",
      "description": "<string>",
      "enum_values": [
        "<string>"
      ],
      "label": "<string>"
    }
  ],
  "provider": "<string>",
  "description": "<string>",
  "frequency_penalty": 123,
  "logprobs": true,
  "max_tokens": 123,
  "mode": "<string>",
  "presence_penalty": 123,
  "slug": "<string>",
  "stop": [
    "<string>"
  ],
  "temperature": 123,
  "top_p": 123
}
'
{
  "bindings": [
    "<string>"
  ],
  "evaluator_id": "<string>",
  "slug": "<string>",
  "type": "AGENT_GOAL_ACCURACY",
  "version_id": "<string>"
}

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Custom evaluator creation input

input_schema
object[]
required

Schemas

messages
object[]
required

LLM Message configuration

model
string
required

LLM parameters (from BaseLLMConfig)

name
string
required

Basic fields

output_schema
object[]
required
provider
string
required

Provider configuration

description
string
frequency_penalty
number
logprobs
boolean
max_tokens
integer
mode
string
presence_penalty
number
slug
string
stop
string[]
temperature
number
top_p
number

Response

OK

bindings
string[]
evaluator_id
string
slug
string
type
enum<string>
Available options:
AGENT_GOAL_ACCURACY,
CHAR_COUNT,
CHAR_COUNT_RATIO,
ANSWER_COMPLETENESS,
FAITHFULNESS,
JSON_VALIDATOR,
PERPLEXITY,
PII_DETECTOR,
PLACEHOLDER_REGEX,
PROFANITY_DETECTOR,
SEXISM_DETECTOR,
PROMPT_INJECTION,
TONE_DETECTION,
PROMPT_PERPLEXITY,
TOXICITY_DETECTOR,
ANSWER_RELEVANCY,
ANSWER_CORRECTNESS,
REGEX_VALIDATOR,
SECRETS_DETECTOR,
SQL_VALIDATOR,
SEMANTIC_SIMILARITY,
TOPIC_ADHERENCE,
UNCERTAINTY_DETECTOR,
WORD_COUNT,
WORD_COUNT_RATIO,
INSTRUCTION_ADHERENCE,
AGENT_FLOW_QUALITY,
AGENT_EFFICIENCY,
AGENT_GOAL_COMPLETENESS,
CONVERSATION_QUALITY,
INTENT_CHANGE,
CONTEXT_RELEVANCE,
AGENT_TOOL_ERROR_DETECTOR,
AGENT_TOOL_TRAJECTORY,
HTML_COMPARISON,
LLM_AS_A_JUDGE
version_id
string