For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Evaluation datasets contain the prompts you use to test how a candidate responds during an evaluation. A candidate can be an agent, model, inference router, or dedicated inference deployment, depending on the evaluation type.

For example, if you want to test the factual accuracy of a marketing agent or model, you could add a prompt to your dataset that asks, “What is the difference between a Droplet and a virtual machine?” Developing evaluation datasets with specific goals helps you measure and refine candidate performance more effectively.

Format Evaluation Datasets

Evaluation datasets are test files that contain prompts for the candidate to respond to during an evaluation.

Each dataset must include a query column. The query column contains the prompt or input you want the candidate to answer.

query
"What makes DigitalOcean different from other cloud providers?"
"Explain the benefits of using DigitalOcean for startups."
...

Add Ground Truth Responses

Some metrics require ground truth values. Ground truth values are expected responses that the judge model uses to compare against the candidate’s response.

For a ground truth dataset, include an additional expected_response column. This column contains the expected answer for each query.

query,expected_response
"What is a Droplet?","A Droplet is a virtual machine that runs on DigitalOcean’s cloud infrastructure."
"How do I add a member to my Team account?","To add a member to your DigitalOcean Team account, go to the DigitalOcean Control Panel, open your team settings, click the Members tab, and then click Invite Team Member."
"What is the pricing for Droplets?","Droplets have flexible pricing based on instance type."

Use ground truth responses when you want to evaluate whether a candidate’s answer matches a known correct answer. For example, ground truth datasets are useful for support questions, policy answers, factual product documentation, and other cases where the expected response is known.

Follow Dataset Formatting Requirements

When creating an evaluation dataset, follow these requirements:

• Include a query column.
• Include an expected_response column if you plan to use metrics that require ground truth.
• Use UTF-8 encoding.
• Review exported spreadsheet files for invalid or non-UTF-8 characters before uploading them.
• Remove empty rows when possible. Empty rows are skipped during scoring, but any tokens used to process them still count toward evaluation cost.
• Make sure the file is readable and correctly formatted. If required columns are missing, or your file is malformed or unreadable, the dataset upload fails.

We recommend using 50 to 100 queries for faster feedback and lower evaluation costs. You can upload datasets with more than 500 queries, but only the first 500 are used for evaluation.

Develop Your Dataset

Developing your dataset depends on the goals you have for your evaluation. Start by identifying what you want to measure, then write prompts that test that behavior directly.

For example, if you want to test factual accuracy, create queries that ask for specific information about your product, service, or policy. If you want to test safety and harmful content, create queries that test whether the candidate avoids unsafe, sensitive, or inappropriate responses.

While you may have one specific goal in mind, we recommend developing multiple datasets that test different aspects of candidate performance, such as:

• Factual accuracy: Whether the candidate provides correct information.
• Safety and harmful content: Whether the candidate avoids harmful, unsafe, or sensitive content.
• Instruction following: Whether the candidate follows user instructions and returns the requested format or behavior.
• Robustness and reliability: Whether the candidate handles ambiguous, incomplete, or unexpected queries without producing incorrect or harmful responses.
• Context use: Whether the candidate uses retrieved context, knowledge base content, or other provided information accurately.

This approach lets you evaluate performance across a range of scenarios and identify specific areas for improvement.

Write Focused Queries

Write each query to test one behavior or scenario. Focused queries make it easier to understand why a candidate passed or failed a metric.

For example, instead of writing:

query
"Tell me about DigitalOcean pricing and how to invite team members."

Split the query into separate rows:

query
"What affects Droplet pricing?"
"How do I invite a team member to my DigitalOcean team?"

Include Realistic User Language

Use prompts that reflect how your users actually ask questions. Include common phrasing, product names, abbreviations, and terminology from your support tickets, documentation searches, sales calls, or application logs.

For example, a realistic dataset might include both formal and informal versions of the same intent:

query
"How do I resize a Droplet?"
"Can I make my Droplet bigger without deleting it?"

Cover Expected and Edge Cases

A strong dataset includes both common requests and edge cases. This helps you evaluate whether the candidate performs well in normal situations and handles difficult cases safely.

Include prompts that test:

• Common user questions.
• Ambiguous or incomplete questions.
• Questions with incorrect assumptions.
• Requests that require the candidate to refuse or redirect.
• Requests that require retrieved context or a knowledge base.
• Questions where the candidate should say it does not know.

Use Ground Truth When You Know the Correct Answer

Use an expected_response column when there is a clear correct answer. The expected response should be accurate, concise, and specific enough for the judge model to compare against the candidate response.

Avoid making ground truth responses too narrow unless exact wording matters. If multiple answers could be correct, write the expected response to describe the required facts instead of requiring a specific phrase.

For example:

query,expected_response
"Can I recover a destroyed Droplet?","Destroyed Droplets cannot be recovered. Users should restore from a backup or snapshot if one is available."

Create Separate Datasets for Separate Goals

Use separate datasets for different evaluation goals. This makes results easier to interpret and compare over time.

For example, you might create separate datasets for:

• Billing accuracy.
• Support policy compliance.
• Safety and refusal behavior.
• Knowledge base retrieval quality.
• Instruction following.
• Regression testing after prompt or model changes.

Use Open-Source Datasets as References

You can explore the following open-source datasets to help you develop your own datasets: