Hello, AI Hack Lab (@ai_hack_lab) here.
In our previously published implementation of the 【Python × Claude API】“Prompt Automatic Generation and Quality Validation Pipeline,” which is ready for immediate use in prompt sales and business automation, we explained a mechanism for automatically generating prompts and validating the output.
This article is the practical follow-up. We take the generated prompts and autonomously refine them through an evaluation → improvement → re-evaluation loop until they reach the target score, then publish the pipeline along with the full copy-and-paste runnable code.
In business automation and AI side gigs (selling prompts and creating them on commission), the most time-consuming and labor-intensive task is prompt quality validation and improvement (refactoring).
If you manually inspect each prompt you create to check whether it delivers the expected accuracy, and then make manual corrections, that alone can eat up hours.
Therefore, in this article, we explain the complete implementation of a system that uses Anthropic's Claude 3.5 Sonnet API to automatically evaluate prompt output (LLM-as-a-Judge) and continuously polish the prompt itself through a self-improvement loop until it reaches a target score (for example, 80 points or higher), known as the “Prompt Auto-Optimization Pipeline”.
The full Python script, which runs as-is simply by copying and pasting, is also included, so please feel free to use it to improve business efficiency and develop commercial offerings.
1. Overall System Configuration (Self-Improvement Loop)
This pipeline uses an architecture that separates and connects the following three roles (agents).
-
Target Agent (Execution Role): Uses the current prompt to generate responses to test cases.
-
Evaluator Agent (Evaluation Role): Scores generated responses on a 100-point scale based on predefined evaluation criteria (logic, comprehensiveness, specificity, tone, etc.), and outputs reasons for point deductions and suggestions for improvement.
-
Optimizer Agent (Improvement Role): Receives feedback from the Evaluator Agent and automatically revises the prompt instructions so that they can achieve a higher score.
This loop automatically repeats until the target score is reached or the maximum number of attempts (max_iterations) is reached.
[Initial Prompt]
│
▼
┌──────────────┐
│ Target Agent │ ── (Response Generation) ──► ┌─────────────────┐
└──────────────┘ │ Evaluator Agent │
▲ └─────────────────┘
│ │
(Prompt Update) (Score & Improvements)
│ ▼
┌─────────────────┐ [Score >= Target Value?]
│ Optimizer Agent │ ◄── (Improvement Instructions) ── No (Retry)
└─────────────────┘ │
Yes
▼
【Optimization Complete!】
2. Implementation Code (Python × anthropic SDK)
The following code works with Python 3.10 or later and the anthropic package.
pip install anthropic pydantic
import os
import json
from typing import Dict, Any, List
from pydantic import BaseModel, Field
import anthropicclient = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
MODEL_NAME = "claude-3-5-sonnet-20241022"
class EvaluationResult(BaseModel):
score: int = Field(description="Quality score on a 100-point scale (0–100)")
strengths: List[str] = Field(description="Strengths")
weaknesses: List[str] = Field(description="Weaknesses and deficiencies that should be improved")feedback: str = Field(description="Specific advice for improving the prompt")
def run_target_prompt(system_prompt: str, user_input: str) -> str:
"""Generate output using the current prompt"""
message = client.messages.create(
model=MODEL_NAME,
max_tokens=1500,
system=system_prompt,
messages=[{"role": "user", "content": user_input}]
)return message.content[0].text
def evaluate_output(criteria: str, user_input: str, output: str) -> EvaluationResult:
"""Strictly score and evaluate the generated output (LLM-as-a-Judge)"""
eval_system = f"""
You are a strict prompt quality evaluator.
Based on the 【Evaluation Criteria】 below, strictly and objectively score the output in response to the input on a 100-point scale.
【Evaluation Criteria】
{criteria}
Always return only the following JSON format.
{{
"score": an integer from 0 to 100,
"strengths": ["Strength 1", "Strength 2"],
"weaknesses": ["Weakness 1", "Weakness 2"],"feedback": "Specific instructions on how to revise the prompt to get closer to a perfect score"
}}
"""
prompt = f"【User input】:\n{user_input}\n\n【Output being evaluated】:\n{output}"
response = client.messages.create(
model=MODEL_NAME,
max_tokens=1000,
temperature=0.0,
system=eval_system,
messages=[{"role": "user", "content": prompt}]
) raw_text = response.content[0].text
# JSON parsing (handling Markdown fences)
clean_json = raw_text.strip().removeprefix("```json").removeprefix("```").removesuffix("```").strip()
data = json.loads(clean_json)
return EvaluationResult(**data))
def optimize_prompt(current_prompt: str, criteria: str, eval_result: EvaluationResult) -> str:
"""Improve the prompt based on the evaluation results"""opt_system = """
You are a world-class expert in prompt engineering.
Take the existing prompt and evaluation results, and refine the instructions, constraints, and output format so that a full score can be achieved under the evaluation criteria.
Return only the “text of the improved prompt.” Do not include any extra explanation whatsoever.
"""
prompt = f"""
【Current Prompt】:
{current_prompt}
【Evaluation Criteria to Be Achieved】:
{criteria}
【Previous Evaluation Score】: {eval_result.score} points
【Weaknesses and Shortcomings】:
{json.dumps(eval_result.weaknesses, ensure_ascii=False)}
【Improvement Feedback】:
{eval_result.feedback}
"""
response = client.messages.create(
model=MODEL_NAME,
max_tokens=2000,temperature=0.3,
system=opt_system,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text.strip()
def run_prompt_optimization_pipeline(
initial_prompt: str,
test_input: str,
criteria: str,
target_score: int = 85,
max_loops: int = 3) -> Dict[str, Any]:
"""Main function that executes the optimization loop"""
current_prompt = initial_prompt
history = []
print("🚀 Starting the automatic prompt optimization pipeline...")
for i in range(1, max_loops + 1):
print(f"\n--- [Iteration {i}/{max_loops}] ---")
# 1. Execution
output = run_target_prompt(current_prompt, test_input)
# 2. Evaluation eval_res = evaluate_output(criteria, test_input, output)
print(f"📊 Evaluation score: {eval_res.score} points")
print(f"💡 Areas for improvement: {', '.join(eval_res.weaknesses)}")
history.append({
"iteration": i,
"prompt": current_prompt,
"score": eval_res.score, "output": output,
"feedback": eval_res.feedback
})
# Goal achievement check
if eval_res.score >= target_score:
print(f"🎉 Target score ({target_score} points) achieved! Ending optimization.")
break
if i < max_loops:
print("🔄 Self-improving the prompt...")
current_prompt = optimize_prompt(current_prompt, criteria, eval_res)
return { "final_prompt": current_prompt,
"final_score": history[-1]["score"],
"iterations": len(history),
"history": history
}
3. Example of Practical Application (Improving a Business Email Writing Prompt)
For example, an initial prompt written carelessly:
You are a successful businessperson. Please write a polite reply email to an inquiry.
If you feed this into the pipeline and have it evaluated based on "accuracy of polite language," "empathy for the other party's intent," "clarification of the next action (deadline and person responsible)," and "conciseness," it will automatically evolve into a highly refined prompt such as the following in just two iterations.
You are a customer success lead at a publicly listed company.
Please create a reply email that maximizes the customer's sense of trust and enables the next action to proceed smoothly.
[Mandatory Requirements]
1. At the beginning, include one sentence thanking the customer for their inquiry and expressing empathy for their situation.
2. State the conclusion first, and organize the reasons for the answer in bullet points (no more than three).
3. Eliminate ambiguous expressions (such as "as soon as possible") and present a specific deadline and point of contact.
4. Keep the entire response within 300–400 characters and minimize the reading burden on the recipient.
4. Summary: Maximizing the ROI of AI Side Jobs and Practical Work Through Automation
By automating the creation and validation of prompts, you can ensure the quality of content production and commissioned projects while significantly reducing the time spent working.
Moving from the stage of using AI as "merely a chat tool" to the stage of "integrating it into a system and allowing it to operate autonomously" is the key to achieving overwhelming results in the coming AI era.
📚 Reference links / Official materials
Our lab (AI Hack Lab) publishes practical know-how on AI automation and prompt monetization across various media.
