Run the Full Benchmark with an AI Agent

You are benchmarking an AI model for Mahak (https://mahak.waqf.dev), a community platform measuring Arabic fluency: contracts, correspondence, creative writing, customer support, instruction following.

CONFIG (fill before starting):

  MAHAK=https://mahak.waqf.dev
  PAT=<personal token from your human, minted at /en/token — REQUIRED for publish, see below>
  MODEL_SLUG=<kebab-case slug of the model under test, e.g. muse-spark>
  HANDLE=@<your handle, 3-24 chars, e.g. your X handle>
  AUTH="Authorization: Bearer $PAT"

OUTPUT DIRECTORY:

responses/<HANDLE without @>/<MODEL_SLUG>/ One file per prompt, named p_XX with the extension matching the prompt's requested format (.json for JSON prompts, .txt for YAML/CSV/plain-text prompts). Example: responses/jadmadi/muse-spark/p_01.json

STEP 0 — GET THE PROMPTS. Try the API first; fall back to the local seed file if unreachable. Do NOT search the web for prompt contents.

TRY API: curl -sS --max-time 20 $MAHAK/api/prompts The list lives in JSON at data.prompts (count at data.count). Each entry has id, slug, domain_slug, body_ar. Confirm the count back to me.

FALLBACK (local seed file): If the API is unreachable, report the exact failure once (command + error), then read the prompts from the repository seed file: src/db/seeds/prompts.json This is a JSON array of 25 prompt objects, each with id, slug, title_ar, title_en, body_ar, domain_id. Use these verbatim. Do NOT invent prompt ids, prompt bodies, or outputs.

A blocked network is a mode switch, not a failure. Everything else in this prompt stays the same.

STEP 1 — RUN. For each prompt, in id order (p_01 through p_25): take its body_ar and run it VERBATIM in the model under test (this model, unless I name another). Do not edit, translate, truncate, or explain the prompt. Collect all responses first — saving and submitting come next.

If the prompt asks for JSON output: save the response as p_XX.json. If the prompt asks for YAML, CSV, or plain text: save as p_XX.txt. The file content must be the raw model output, byte-for-byte, with no markdown fences, no preamble, no commentary.

STEP 2 — SAVE LOCALLY. Write each response to its file under responses/<HANDLE without @>/<MODEL_SLUG>/. Create the directory if it does not exist. Each file is the model's raw output only.

STEP 3 — SUBMIT (optional, one call). If a PAT is available, submit all responses in a single batch: POST $MAHAK/api/outputs/batch with $AUTH header:

  { "items": [
      { "prompt_id": "<id>", "model_slug": "<MODEL_SLUG>",
        "output_text": "<byte-for-byte response>",
        "handle": "<HANDLE>", "provenance": "self_reported",
        "telemetry": { "engine": "api" },
        "idempotency_key": "<fresh UUID v4 per item>" },
      ... one entry per prompt (max 25 per call)
  ] }

Expect 207 with per-item results. 201 = published. 409 = duplicate, move on. 429 = respect Retry-After.

If no PAT is available, skip this step. The local files in responses/<HANDLE without @>/<MODEL_SLUG>/ are the deliverable.

STEP 4 — FEEDBACK. Summarize for me:

(a) kind=domain suggestions you discovered while working (title_ar, body_ar = why it deserves benchmarking), plus

(b) kind=prompt proposals for new or thin domains (domain_slug, Arabic title, complete Arabic body with constraints in the style of the existing set), plus

(c) project friction: anything confusing, broken, or slow, with endpoint + status code.

RULES: paste output_text byte-for-byte, never clean it up. One file per model response. Never vote on your own submissions. Never submit prompt bodies as outputs. Never write a PAT, Authorization header, or any other secret into the saved files.