ThrottleBot

Identity

Every request ThrottleBot makes carries this user agent:

ThrottleBot/2.0 (+https://throttle.toys/bot; [email protected])

We also send a From: [email protected] header. If you see traffic that claims to be ThrottleBot but does not resolve back to this page, it is not us — tell us and we will say so in writing.

What we fetch

• robots.txt, first, before anything else, and again periodically.
• Your inventory index pages and your XML sitemap, where you publish one.
• Individual vehicle detail pages: year, make, model, price, VIN or HIN, stock number, specs, and the URLs of your photos.
• One published size of each photo we read from a listing (usually only the first photo, at most the first three), for our own category model (see below). Never a car or race-car photo.

Photos on Throttle are hotlinked from your server: the buyer's browser loads them from you and we never show a copy, so removing a photo on your side removes it from our site. We do not submit forms, create accounts, add to carts, or touch anything behind a login.

Photos, and our own category model

Effective 13 October 2026, 18:00 UTC. From then on, we read photos of dealer listings on our crawl server, one published size of each (usually only a listing's first photo, at most its first three), to train and check our own category model, so that a trailer, a part or an implement is not filed as a race car or a motorcycle.

• Each photo request follows the same rules as a page: our user agent, our crawl server's own address, your robots.txt for the photo's own path and for every redirect, your Crawl-delay and our per-site floors (never faster than the larger of the two, and never under 2.5 seconds), and one request at a time to your server, shared with the page crawl.
• A 401, 403 or 429, or a bot-challenge page, on your image server stops us there. We never retry around a block, change identity or address, or get past it any other way.
• We keep two small downscaled copies (256 and 128 pixels), a numeric fingerprint and the numeric representations (embeddings) our own model computes from them, on our own machines only: our crawl server and our training machine. We never display, serve or redistribute them: this site always shows your own photo, from your server. The full-size photo is never stored; it is discarded once it is downscaled.
• We do not read car or race-car photos (we are waiting on legal advice), or photos on marketplaces, classifieds or auction sites.
• Image servers of the Dealer.com platform are not read for now.

The model is ours. Our category model is our own, trained from scratch on our servers, on downscaled copies of dealer photos, each labelled with the category our own rules already gave its listing. Our staff's labels are used only to test it, never to train it. No third-party AI service sees any of it. Nothing else we fetch, your pages, descriptions, specifications or prices, is put in a training set; all they give the model is that category label. The model never decides a listing's category on its own.

Content-Signal. We record a site's Content-Signal on every photo we keep. search=no is honoured. ai-train=no and ai-input=no are recorded and those hosts are read last; we can remove everything from such a host on request or by our own policy, by host, at any time.

Removing the copies. Copies are deleted when a dealer sends a DMCA notice, a verified cease-and-desist or an opt-out, and every later model is trained without them. Otherwise they are kept until one of those, or a policy line of our own, removes them; there is no other expiry. What removal leaves behind is a record of which photo was read, from where and when, and when and why it was removed, never the image.

How politely

• One request at a time per site — never parallel connections to one host.
• At least 2.5 seconds between requests, or your Crawl-delay if it is longer.
• Typically once a day for powersports inventory, every three days for marine, weekly for aircraft.
• If we get a 429, a 403 or a bot-challenge page, we stop and back off. We do not change user agent or address to get around a block — a challenge is a hard stop, and we treat it as an answer.
• Crawling runs from a separate pool of addresses to the website you are reading now, so a crawler problem can never take the site down and vice versa.

robots.txt

We obey it. To block us entirely, add:

User-agent: ThrottleBot
Disallow: /

We re-read robots.txt regularly, so a block takes effect on the next visit without you having to contact anyone.

Contact pages, and a second user agent

ThrottleOutreach (ThrottleOutreach/1.0 (+https://throttle.toys/outreach; [email protected])) is our second user agent: it reads only a dealer's home, contact, about and terms (or privacy) pages, never inventory. What it fetches and keeps, and how to block it, are at /outreach. ThrottleOutreach has not made any request yet.

Opting out, and takedowns

Email [email protected] from an address at your dealership's domain and say which site to remove. We will stop crawling and remove your listings within 24 hours, and we will confirm in writing when it is done. No forms, no retention offer, no arguing.

You can also ask for a correction instead of a removal — a wrong price or a mis-categorised unit is usually our parser's fault, and we would rather fix it than lose the store.

Effective 13 October 2026, 18:00 UTC: we read dealer listings' photos (usually only the first photo, at most the first three) to train our own category model. What we keep, what it is never used for, and how it is removed are under Photos, and our own category model.

A feed is better than a crawl

If you would rather push than be pulled, send us a CSV, XML or JSON inventory feed and we will stop crawling you entirely. Feeds are more accurate, cheaper for both of us, and they let you control exactly what appears. Email the same address.

What we do with it

Every listing links back to your page, and we record when a price changes and when a listing leaves, for buyers' price history.

We never resell your inventory; we license only aggregated market data in which your store is not identified, and the opt-out above removes you from both.

What we never crawl

• Any marketplace, classifieds site or aggregator that is not in our source list (see “Marketplaces, classifieds and aggregators”), including when a dealer's page redirects to one, and any page a site's robots.txt closes.
• The one exception to the line above: the dealer-profile pages of the dealer directories listed under “Dealer census”, read for a dealer's name, address and own website. Never their listings.
• Anything behind a login, a paywall or a CAPTCHA.
• Any site that has opted out, at any level, ever.
• Contact forms are never submitted.

Marketplaces, classifieds and aggregators

ThrottleBot reads public pages on dealer websites and on public marketplaces and classifieds sites, where that site's robots.txt allows it, under the same rules as everywhere else: an honest user agent, robots.txt obeyed path by path, the site's Crawl-delay, one request at a time, and a full stop at the first challenge or block.

What we read on a marketplace. Pages anyone can open without an account: listing pages, category pages and the sitemaps the site publishes. We reach listings through the site's own sitemap or its category pages, never through a search or filter address that its robots.txt closes.

What we never do, on any marketplace:

• sign in, register, accept terms, or read anything behind a login, a paywall or an app;
• solve, avoid or retry around a challenge, a CAPTCHA or a block. We do not change our user agent or our network address to get past one. A site that challenges ThrottleBot is a site we do not read;
• submit a form, open a seller-contact or “reveal phone number” link, or send a message;
• store a private seller's name, username, email address, phone number, street address or exact location. Nor do we contact a seller who has not listed with Throttle;
• copy a seller's description. We record facts (make, model, year, price and what kind of price it is, location to city or state, category, and specifications stated in the listing) and link to the original;
• store a photo. Photos stay on the site that published them. Where a site's terms bar linking to its images, we show no photo and link to the listing;
• change our network address to reach a site again. Each marketplace is always read from the same address, listed on this page, chosen before our first scheduled visit. The one-time test reads of September 2026 (see “Test reads before we started” below) came from our development connection, whose public address was not recorded; neither site blocked it. If a site blocks our address, we stop reading the site;
• use a site's pages to train AI models where its robots.txt or its terms say no;
• buy, accept or use anyone else's copy of a site.

Sources. Every marketplace ThrottleBot reads is listed on this page, with its operator and the date we started.

A site that has an official API or data feed we can use on acceptable terms is read through that, not crawled. eBay, including eBay Motors, is never crawled; if eBay approves us, its listings will appear only through its official API, in a separate labelled section.

If you run one of these sites and want us to stop, see “If you operate a marketplace” below. We stop within 24 hours.

From marketplaces and classifieds we store the listing's facts, its address on the source site, when we saw it, and the kind of price shown (an asking price, “price on application”, a current bid, a hammer price, or a total that includes fees). We never store a private seller's identity or contact details, and we do not keep the seller's description.

Marketplace photos are never copied or stored; where the source's terms bar linking to its images, the listing shows no photo.

Private sellers on other sites

Many marketplace listings are placed by private individuals. For those listings we keep only the vehicle's facts and the town or state it is in. Where the listing prints the vehicle's registration or serial number (an aircraft's N-number, a VIN), we keep it to avoid storing the same vehicle twice; it is never shown or shared. We never keep the person's name, username, email, phone number or street address, we never contact them, and their listings are never available through our API, exports or any data licence. If you placed a listing on another site and want it removed from Throttle, email [email protected] with the link; we remove it from our site within 24 hours, and from our archive within 24 hours unless a legal matter about that site requires us to keep records unchanged, in which case it stays hidden from everyone and is removed from the archive as soon as that matter ends; we do not re-add it.

If you operate a marketplace

Email [email protected] from an address at your company's domain and name the sites you operate. We stop reading every site you name, and every other site our records show you operate, within 24 hours, and we remove their listings from the public site within 24 hours. A legal notice follows “Legal notices” below instead. If you would rather send us a feed than be crawled, “A feed is better than a crawl” applies to you as well: we will stop crawling as soon as the feed runs.

Test reads before we started

To build and test that work, in September 2026 we fetched a handful of public pages from two sites: from barnstormers.com its legal notices page and one page of recent ads; from racecarsforyou.com its terms of use, its sitemaps and five listing pages (one of which redirected to the site's search page, which was read as a result). Each of those pages was fetched once, with the ThrottleBot user agent and obeying robots.txt, and our requests to each site were at least 2.5 seconds apart; the one redirect was followed immediately, as part of the same request. We also read each site's robots.txt before every page: six times in all for barnstormers.com, whose robots.txt address answers with an ordinary web page rather than rules, and eleven times for racecarsforyou.com, which has none. We keep those pages only as test material, with any personal details in them replaced, and we copied no photos.

Dealer census

To find dealers whose own websites we can read, ThrottleBot reads the public dealer-profile pages of the dealer directories below. A profile names the dealer's own website; we then read that website first-hand, under every rule on this page, and nothing else from the directory.

• CarGurus (www.cargurus.com): its robots.txt, its sitemap, and each dealer's profile page with no query string. Read since 2026-10-01.

What we keep: the dealer's name, postal address, map location, its own website and how many vehicles its profile says it lists (used only to decide which dealer to read first, never shown as inventory). What we never read there: listings or their pages, reviews or reviewers' names, ratings, logos, photos, opening hours or phone numbers.

The same user agent, the same robots.txt rules, one request at a time, at least 3 seconds apart (or the site's Crawl-delay if longer), from one fixed address. A challenge or a refusal ends the census for that site; we do not retry it any other way.

Online auction sites

ThrottleBot reads public pages on some online car and motorcycle auction sites, where that site's robots.txt allows it, to show live auctions and their results on Throttle, labelled as auctions and never as a dealer's price. A separate crawler does this reading, under every rule on this page: the same honest user agent, robots.txt obeyed path by path, the site's Crawl-delay or 2.5 seconds, whichever is longer, one request at a time per site, and a full stop of the primary pass at the first challenge or block. Decision 0084 permits only the one delayed, bounded secondary render described in §11.2; a second refusal ends it.

What we read, and how often. The site's public list of live auctions, about every 15 minutes; each auction's own page a few times while it runs; and its result page twice, shortly after the auction ends and again about a day later (a third time, about three days after the end, when the seller's reserve was not met, because a sale can still be agreed).

What we never do on an auction site:

• sign in, register, bid, or read a member, account, message or bidding page, or the site's private API;
• bid, place a watch or send a reminder on anyone's behalf. Throttle is not a party to any auction, and every auction on Throttle links to the site's own page;
• download, copy or store a photo. Photos stay on the auction site: an auction on Throttle shows its photos the way a dealer listing does, loaded by your browser from the auction site itself, and every auction links to the site's own page;
• store a bidder's, seller's or commenter's name, username or message. We keep each bid's amount and time, and nothing about who placed it;
• add a buyer's fee to a price, or work one out. We record the prices the site prints, each labelled the way the site labels it;
• rotate our network address until a site answers. Each auction site's primary reads always use the same address, listed on this page. The one bounded recovery of §11.2 uses one different static address and never changes that primary pin.

Sources. Every auction site ThrottleBot reads is listed on this page, with its operator, the date we started and the primary address it is read from. A site we have stopped reading for now says “paused” instead of an address: nothing is read from it until it says an address again.

Cars & Bids and Collecting Cars are not started: each answered our one test visit with a challenge. Cars & Bids has a declared public-page adapter and may start only after its own gate passes; recovery does not start a source or waive that gate.

If you run one of these sites and want us to stop, email [email protected] from an address at your company's domain and name the sites you operate. We stop within 24 hours.

Auction test reads before we started

In September 2026, before any auction site was read on a schedule, we read a few public pages from six online auction sites to build test material: bringatrailer.com 7, pcarmarket.com 7, hagerty.com 7 (its robots.txt, its marketplace terms, reached through one redirect, and four pages under /marketplace), broadarrowauctions.com 5, and carsandbids.com and collectingcars.com 2 each, where the second request met a challenge and we stopped. Each page was read once, with the ThrottleBot user agent, obeying robots.txt and at least 2.5 seconds apart (10 seconds on broadarrowauctions.com, as its robots.txt asks). We keep those pages only as test material, with the names, handles and contact details of the people in them replaced, and we copied no photos.

Cars

ThrottleBot crawls 799 car dealer websites on a schedule.

From a car dealer's website, ThrottleBot fetches:

• robots.txt, first, and obeys it path by path;
• the inventory pages robots.txt allows, or the sitemap the site advertises;
• the vehicle pages those list: year, make, model, price, VIN, specs and photo URLs.

It does not fetch search pages with filters in their address, machine-readable inventory endpoints that robots.txt disallows (RSS inventory feeds, AJAX vehicle endpoints, searchable-data JSON), window stickers, or anything behind a login. Apart from the separately declared, disabled index-only Porsche Finder source described below, we read a dealer's stock from the dealer's own site rather than a manufacturer portal.

ThrottleBot has tested readers for car inventory on these dealer-website platforms: DealerOn and Dealer.com. It crawls car inventory on DealerOn today.

One request at a time per site, and never faster than the site's Crawl-delay or our own floor, whichever is longer: at least 10 seconds between requests on DealerOn sites (which publish Crawl-delay: 10), at least 20 seconds on Dealer.com sites, and at least 2.5 seconds anywhere.

On a source configured for the default-off recovery lane, a Cloudflare, WAF or transient block may trigger a bounded secondary visitor-browser attempt with the same honest ThrottleBot identity. It never solves a CAPTCHA, replays a private API, falsifies identity or overrides robots, a legal halt, an opt-out, SSRF protection, a per-host lease or a request budget. An unresolved challenge stops the attempt, and the outcome is recorded honestly.

Car listings show the dealer's photos, loaded by the buyer's browser from the dealer's own website, as for every other category. Car photos are never copied, cached or resized, and are not read for training. No manufacturer logo is shown.

The opt-out, the robots.txt rules and the legal-notice routes on this page apply to car dealer websites unchanged.

ThrottleBot has a declared, index-only Porsche Finder source. It is disabled and not started, so it reads no Finder pages on a schedule today. Its declaration covers only the bare US public search index and canonical numeric pagination, never a detail page, /api path or position parameter; it is not complete Porsche inventory coverage. The genuine 2026-11-02 search capture succeeded, and a later detail-page challenge remains recorded with detail fetching off.

On 2026-09-24 we fetched robots.txt, one page listing the inventory (an inventory page, a sitemap or the home page) and up to three vehicle pages from each of 18 car dealer websites (10 of them Porsche websites in California) as test pages for our software, one request at a time per site and at or above each site's Crawl-delay. No listing from those test pages is shown on Throttle.

On 2026-09-26 we fetched robots.txt, one page listing the inventory (an inventory page, a sitemap or the home page) and up to three vehicle pages from each of 5 car dealer websites as test pages for our software, and robots.txt alone from 3 more to check whether they allow an identified crawler, one request at a time per site and at or above each site's Crawl-delay. No listing from those test pages is shown on Throttle.

On 2026-09-22 we made a one-time reachability measurement of California car dealer websites under this user agent: robots.txt first, one request at a time per site, stopping on any challenge. It told us which sites allow an identified crawler. No listing from it is shown on Throttle.

Legal notices

The quickest way to send one is the form at /legal/notice. You sign in with your email address so we can write back to you, choose the kind of notice, and submit it, and you get a case number by email. Nobody has to read it first:

• A notice to stop crawling — a cease-and-desist letter, or a demand from a website's owner or someone acting for it — stops crawling of every domain it names the moment you submit it.
• A copyright notice takes the photos it identifies off Throttle within 24 hours. It does not stop crawling; see "Copyright notices" below.

You can also email [email protected]. An emailed notice is read by our founder and answered with a case number. A copyright notice takes the photos it identifies off Throttle within 24 hours, and any other legal notice stops crawling of every domain it mentions within 24 hours. We stop first; we do not wait for advice. A notice sent by post or any other way is handled the same way from the day it reaches us.

What proving it is yours changes. A stop on crawling starts before we know who sent the notice. If you show us that you control the website — with a DNS record or a file we give you, an email address at the website's own domain, or business documents we check by hand — the stop is permanent until you withdraw it. If a notice about crawling a site comes from someone who does not show that they own or act for that site within 14 days, we tell them, and crawling resumes; the site's own robots.txt, which we always obey, and the opt-out above remain open to its owner at any time. Someone who says they are a law firm, a website platform or a manufacturer gets 30 days instead, to send a letter of authority, and after that the same rule applies unless we have confirmed their authority. A new notice about the same site from someone whose earlier one closed without proof must be backed up within 72 hours. A notice that reached us by email or post holds until a person reviews it — unless the message passed neither of the standard email authentication checks (SPF and DKIM), or its sender has already sent us several this month, in which case the stop lapses after 14 days unless we confirm it.

Copyright notices. A copyright notice does not by itself stop crawling. Within 24 hours of a notice that identifies the work and where it appears, we stop showing the photos of the listings it names — or, if it names a dealer's website rather than pages, the photos of every listing from that website — and the description text as well if that is the work it identifies. We keep the facts of each listing: price, VIN, year, make, model, trim, specifications, features, mileage and location. They come from the dealer's own public advertisement, and facts are not protected by copyright (17 U.S.C. §102(b); Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340 (1991)). We reply to say what we took down and how to raise a further claim. The photos and text stay down whatever the result of any check, until the notice is withdrawn or we have a reason we record to show them again, such as the dealer showing they were taken down by mistake. Anyone who knowingly and materially misrepresents that material infringes may be liable for the damages, including costs and attorneys' fees, that the misrepresentation causes (17 U.S.C. §512(f)).

Checking a claim. When someone who claims a website asks us to check the file we gave them, we request https://<that domain>/.well-known/throttle-verify.txt — and, only if that fails, the same file on www.<that domain> — with the user agent ThrottleBot-Verify/1.0 (+https://throttle.toys/bot#legal-notices), and nothing else. It follows at most two redirects, and only within the same site. It does not read robots.txt first: it is a single request the claimant asked for, not crawling. We may also look up the domain's public registration record at its registry; that lookup never reaches the website.

This is separate from the opt-out above, which is a courtesy we extend to any dealer for any reason.

We have not yet registered a DMCA designated agent with the U.S. Copyright Office. Until we do and publish the agent here, send copyright notices through the form or to [email protected]; they are handled the same way.