How to evaluate headless site hosting

Evaluate the work that an agent can complete, and not only the presence of an API. Start with a real task, such as publishing a three-file site and correcting one file. Use the same task when you compare hosts.

Setup and credentials

Count the steps that a person must do by hand before the first deploy.

Check whether the agent can start the account itself, or whether a person must create a token in a dashboard and paste it into a config file.

Check how a lost key is replaced and whether keys can expire.

Check what the person approves, and whether that approval is a clear payment or consent step.

The deploy path

Ask the agent to publish the site through the documented interface. Record the number of requests and the time until the URL responds.

Check whether the host needs a repository, a manifest, or a build. Check which of these the agent would have to create.

Check the limits on file count, payload size, and deploys per hour.

Ask the agent to change one file. Check that the URL stays the same and that unchanged files do not need to be sent again.

Send a request that must fail. Check that the error names the cause and the fix, so that the agent can recover without a person.

Lifecycle and cost

Check what happens to a site that nobody manages. It can expire, stay up forever, or continue to cost money.

Check whether the agent can set an expiry, keep a site permanently, and delete it.

Work out the cost of one site for one day and for one month. Do the same for twenty short-lived sites.

Check for a free tier and its conditions. A free tier lowers cost. A paid-only host can state its limits and abuse controls more simply.

Check for charges on bandwidth, requests, or seats.

Domains

Ask the agent to attach a domain that you own. Record which steps it completes and which it hands to you.

Check whether HTTPS certificates are issued automatically.

If the host sells domains, check the price against the registry's cost, the renewal price, and how to move the domain elsewhere.

Permissions and safety

Check whether a key can be limited to deploys. An agent that only publishes should not be able to make purchases.

Check which actions cost money, and whether the host's documentation tells agents to confirm them with the person.

Check whether a site can be private, for example behind a password.

Read the acceptable use policy. Sites published by agents are still the account holder's responsibility.

What is served

Compare the served files with the files that were sent. Check for injected banners, badges, or scripts.

Check the routing behavior for single-page apps and for paths without a file extension.

Check where the site is served from and whether a request limit applies.

A useful evaluation states what the agent completed, what a person had to do, and what failed.

Applying these tests

shipsite.sh publishes its limits, prices, and error codes in a single text file written for agents. You can apply every test on this page to it, and to any other host, with the same task.