Internet-Draft Crawler Traffic Verification Levels November 2026
Arsentev Expires 19 May 2027 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-arsentev-crawler-traffic-reporting-00
Published:
Intended Status:
Informational
Expires:
Author:
E. Arsentev
Independent

Reporting Crawler Traffic by Verification Level

Abstract

The User-Agent header is written by the client. It is a claim, not an identity. Reports of crawler and AI crawler traffic that count requests by User-Agent token alone present claims as activity. This document defines four counts a crawler traffic report should give per crawler token: declared, route-consistent, verified, and unverifiable. It states which verification check was possible at the vantage point where the log was taken. It also defines a small JSON report object.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 19 May 2027.

▲

Table of Contents

1. Introduction

Site operators, CDNs, analytics vendors and researchers publish counts of "AI crawler traffic". They produce these counts by matching the User-Agent string (Section 10.1.5 of [RFC9110]) against lists of crawler tokens. The User-Agent value is chosen by the client.

Crawler operators increasingly publish IP address ranges for their crawlers, and some offer reverse DNS checks. Work on cryptographic bot authentication with HTTP Message Signatures ([RFC9421]) is under way in the IETF (webbotauth).

Whether a check can be applied depends on where the log is recorded. An origin server behind a CDN usually sees the CDN's address, not the client's address. A count that mixes checked and uncheckable requests cannot be compared across sites.

The goal of this document is to make the verification level of every reported number explicit.

This document does not define how to verify a crawler. It does not define access policy, which is the subject of robots.txt ([RFC9309]) and of AI preference work. It does not say whether unverified traffic should be blocked.

2. Conventions and Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

crawler token:
a product token a crawler operator documents for its crawler's User-Agent.
vantage point:
the component where the log record is produced (origin server, reverse proxy, CDN edge).
expected route:
the network path a legitimate request to the site takes given the site's deployment, for example through the site's CDN.
published address range:
a set of IP prefixes a crawler operator publishes as the source of its crawler.

3. The Four Counts

For each crawler token and reporting period, the report gives the following counts.

declared:
requests whose User-Agent contains the token. This count is always available.
route-consistent:
declared requests that arrived by the expected route. A declared request that reached the origin directly, while the site is served through a CDN, is not route-consistent.
verified:
declared requests whose source was confirmed by a check the report names. The check is a published address range match, forward-confirmed reverse DNS, or a verified HTTP message signature.
unverifiable:
declared requests for which no check was possible at the vantage point, for example because the logged address is the CDN's address.

The report MUST give the declared count together with at least one of the other three counts. A report MUST NOT present the declared count alone as crawler activity.

Requests that were checked and failed are the declared count minus the verified count minus the unverifiable count. The report SHOULD state that number.

When the operator of a token publishes no ranges and no other check exists, all requests under that token are unverifiable. The report says so instead of reporting zero verified.

4. Stating the Vantage Point and the Checks

The report names the vantage point, the period, and the token list with its version or date. For each token, the report states which checks were available: address range, reverse DNS, message signature, or none.

A count from a CDN edge and a count from an origin are different measurements. They are not added together.

If address ranges were fetched, the report gives the date they were fetched, because ranges change.

5. Report Object

The report is carried as a JSON object ([RFC8259]). The object carries one entry per crawler token with the four counts, the checks performed, and the vantage point. The example in the appendix uses illustrative numbers, not measurements.

The report is a JSON object [RFC8259] with the following members.

vantage_point
String, REQUIRED. One of "origin", "reverse-proxy", "cdn-edge", "other".
period
Object, REQUIRED, with members "start" and "end", each a date-time as defined in [RFC3339].
token_list
Object, REQUIRED, with members "source" (string naming the list of crawler tokens used) and "as_of" (full-date as defined in [RFC3339]).
tokens
Array, REQUIRED, one object per crawler token, with the members below.
tokens[].token
String, REQUIRED. The crawler token as matched.
tokens[].declared
Non-negative integer, REQUIRED.
tokens[].route_consistent
Non-negative integer, or null when the expected route could not be determined at the vantage point.
tokens[].verified
Non-negative integer, or null when no check was available.
tokens[].unverifiable
Non-negative integer, REQUIRED.
tokens[].failed
Non-negative integer, RECOMMENDED. Equal to declared minus verified minus unverifiable.
tokens[].checks
Array of strings, REQUIRED. Each one of "address-range", "reverse-dns", "http-signature", "none".
tokens[].ranges_fetched
full-date as defined in [RFC3339]. REQUIRED when "checks" contains "address-range".

A report MUST satisfy, for every entry, verified + unverifiable + failed = declared when all three are present. Members not defined here MAY be added; consumers MUST ignore members they do not understand.

6. Operational Experience

This section reports one case. The data come from one origin server behind a CDN, with full logs for 15 days. The server logged 345,808 requests in total. Of these, 46,155 carried one of 22 published crawler tokens.

Of those declared requests, 35.7% reached the origin directly, bypassing the CDN. None of the 11,276 direct requests under tokens whose operators publish address ranges came from those ranges. In other words, these requests were not confirmed by the published ranges.

The direct traffic never requested robots.txt or the sitemap. It used about 3.5 times more distinct User-Agent strings per request.

A token that Google states it never sends as a User-Agent arrived through the CDN 293 times.

On this origin, no request could be verified by address, because CDN traffic is logged with the CDN's address. A declared-only report from this server would have shown 46,155 crawler requests. The four counts show that none of them was verified.

This is one server, and the result is not a statement about the web in general. The analysis is published as a preprint on Qeios, DOI 10.32388/SRD4TB [QEIOS].

7. Security Considerations

Verification results can themselves be wrong. Published ranges change. Shared cloud addresses may host other clients. Reverse DNS needs forward confirmation. A report that names its checks lets readers judge these limits.

Publishing per-token counts may help an attacker learn which tokens pass a site's filters. Operators MAY delay or aggregate reports.

8. Privacy Considerations

The report carries counts only. It MUST NOT carry client IP addresses, full User-Agent strings of individual requests, request paths with query strings, or visitor identifiers.

Small counts for rare tokens can still point to individuals or organisations. Operators SHOULD suppress or round small cells.

9. IANA Considerations

This document has no IANA actions.

10. References

10.1. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC3339]
Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, , <https://www.rfc-editor.org/info/rfc3339>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.
[RFC8259]
Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, , <https://www.rfc-editor.org/info/rfc8259>.
[RFC9110]
Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, , <https://www.rfc-editor.org/info/rfc9110>.

10.2. Informative References

[QEIOS]
Arsentev, E., "Report AI Crawler Traffic in Two Columns, Declared and Verified", Qeios, preprint, DOI 10.32388/SRD4TB, , <https://doi.org/10.32388/SRD4TB>.
[RFC9309]
Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, , <https://www.rfc-editor.org/info/rfc9309>.
[RFC9421]
Backman, A., Ed., Richer, J., Ed., and M. Sporny, "HTTP Message Signatures", RFC 9421, DOI 10.17487/RFC9421, , <https://www.rfc-editor.org/info/rfc9421>.

Appendix A. Example Report

The numbers in this example are illustrative and are not measurements.

{
  "vantage_point": "origin",
  "period": {"start": "2026-11-01T00:00:00Z",
             "end":   "2026-11-15T00:00:00Z"},
  "token_list": {"source": "example-token-list",
                 "as_of": "2026-11-01"},
  "tokens": [
    {"token": "ExampleBot",
     "declared": 1200, "route_consistent": 800,
     "verified": 0, "unverifiable": 800, "failed": 400,
     "checks": ["address-range"], "ranges_fetched": "2026-11-01"},
    {"token": "OtherBot",
     "declared": 300, "route_consistent": 300,
     "verified": null, "unverifiable": 300, "failed": 0,
     "checks": ["none"]}
  ]
}

In the first entry, 800 requests arrived through the CDN and are logged with the CDN's address, so they cannot be checked at the origin; 400 arrived directly and none of them came from the published ranges. In the second entry the operator publishes no ranges, so every request is unverifiable.

Author's Address

Evgenii Arsentev
Independent