How GrovlixScanner reads your store
GrovlixScanner is the scanner NoFlag uses to check an online store when someone asks for a scan. It reads public pages only, and slowly: plain requests go one at a time, and the browser starts one page load at a time per store. If robots.txt disallows a page for it, it does not read that page at all.
Last updated:
What GrovlixScanner is
NoFlag, a Grovlix product, compares online stores with the published policies of ad platforms and payment providers. When a person enters a store address, GrovlixScanner reads that one store and NoFlag writes a report. It does not crawl the web or follow links to other sites. Besides the store's own pages, it loads the scripts and tag containers those pages load. In browser mode it loads everything a page asks for, from any host (see Browser mode).
What it fetches
For one scan it requests:
- the address that was entered, and the pages it redirects to on the same site. The first request for the home page may also follow a redirect to another site, to find where the store really lives. When another page of the store redirects to another site, it reads the robots.txt of each site the redirect chain passes through, and requests each address there that robots.txt allows (at most 5 requests), only to see where the chain leads. If robots.txt refuses one, or the chain cannot be followed to its end, that page is not read at all;
- the home page, a few product pages (at most 5 pages) and the policy and contact pages: returns, shipping, privacy, terms, contact, about, FAQ, legal notice, complaints;
- robots.txt (a redirect of robots.txt is followed, even to another site) and the sitemaps it points to;
- public data the store platform offers without a login, such as Shopify's
/products.jsonor Salla's public store API; - the scripts and tag containers (analytics, advertising) that those pages load, to see which tags are installed;
- links to other pages of the same site, to check that they work: HEAD requests, and a GET if the store refuses HEAD, at most 20 requests in total per scan;
- the http:// address of the home page (a HEAD request, to see whether it redirects to https) and a TLS connection to read the certificate.
Plain requests use only GET and HEAD. In browser mode the store's own page scripts run as they would for a visitor, including any requests they make (their background requests can be POSTs). The scanner itself submits no forms, logs in nowhere, adds nothing to a cart and buys nothing.
How it behaves
- User agent. Plain requests send
GrovlixScanner/1.0 (+https://getnoflag.com/bot), which links to this page. - robots.txt. It follows the rules for the token
GrovlixScanner, or for*if robots.txt has no group for it. If robots.txt answers 404 or another 4xx error, everything is treated as allowed. The exceptions are a 429 or a challenge page on robots.txt itself, which count as disallowed. If robots.txt cannot be reached (a 5xx error or a timeout), the whole site is treated as off limits, as RFC 9309 says. - Pace. Plain requests: one request at a time to a store, and at least 1 second between the end of a request and the start of the next (at most 1 request per second per store). Stores on Shopify also share one pacing limit across all Shopify stores, because Shopify limits by address across all its stores: a request may wait longer, but a single store is never read faster than that.
- Retries. Plain requests: after a timeout or a server error it retries once. After a 429 (too many requests) it retries only if the store asks for a wait of 10 seconds or less in Retry-After; otherwise it stops asking the store for that page. In browser mode a page load is never retried.
- Bot protection. It never solves a challenge page, such as a Cloudflare check, and never retries one with tricks. The report lists what could not be read and why. A robots.txt refusal ends the read of a page, and so does a 404 or 410 answer (the store says the page does not exist). When a direct read fails for any other reason, for example 403, 429, a 5xx error, a timeout or a challenge page, the scan may ask Google's PageSpeed Insights instead, which loads the page from Google's side. When it is configured, PageSpeed Insights also runs the speed and SEO audits. The scan also reads copies of pages from archive.org, read only: it never asks archive.org to save a page (no Save Page Now).
- What it keeps. Evidence is the snippet or screenshot needed to show a finding, deleted after 90 days (30 days if it may contain personal data). Raw pages fetched for a scan are kept for 2 days: the day of the scan and the day after.
Browser mode
Some checks need a page the way a visitor sees it: scripts that run, cookie banners, layout. Scans can open those pages in a headless Chromium browser.
- Its user agent is Chromium's own, not the GrovlixScanner one, because it is a real browser. We do not disguise it in any way. It runs without stealth tools and keeps Chromium's own fingerprint. To keep it out, use the robots.txt rule below. The browser checks robots.txt for
GrovlixScannerbefore each page and each redirect. Before each browser page load, plain requests with the GrovlixScanner user agent first follow the redirect chain, each hop with its own request. - It starts one page load at a time per store, with at least 5 seconds between two page loads of the same store. An earlier page can still be loading when the next one starts. The scripts, styles, images and requests a page makes by itself while it loads are not paced.
- It opens only the store's own pages. Within a page, the browser loads everything the page asks for, from any host, including iframe documents (at most 30 documents per page load). It never submits a form, cancels downloads and opens no pop-ups. The one thing it clicks is the accept button of a cookie banner, to see what loads after consent.
- A challenge page, or a 429 answer to the page itself, ends that page load. It is not retried and no challenge is solved.
- After a 429 from your store, or a 503 on a page that is not a challenge page, the browser loads no more pages from your store for the rest of that scan. This covers a 429 on a page (a 429 challenge page too) or on anything the page loads from your store's own host. A 429 from another company's host, such as a CDN or an analytics service, does not count.
www.and the bare host count as one.
Opt out and removal
Block it with robots.txt
Add this to the robots.txt of your store:
User-agent: GrovlixScanner
Disallow: /GrovlixScanner reads robots.txt once per scan and checks it before every page, so this applies from the next scan, for plain requests and for browser mode. If robots.txt disallows a page for GrovlixScanner, or for * when it has no group for GrovlixScanner, we do not read it in any way: not directly, not through Google PageSpeed Insights, not from archive.org. The report then says what it could not read, and does not guess.
Ask us to remove a store
Use our opt-out form: you prove that you control the domain with a DNS record or a file on the site, and the domain goes on our opt-out list. The scanner then refuses to scan that domain and its subdomains, including when another address redirects to one of them. You can also write to us with the store's domain name, from an address at that domain or with something that shows you run the store. Deleting the reports and evidence we already hold for it is still a manual job: we do it when you ask, and it is not instant.
Contact
Questions, opt-out and removal requests: [email protected].
About the data we hold
Grovlix runs NoFlag. The public pages of a store can show personal data of the people behind it, for example the name, phone number or address of a sole trader. We keep the snippet or screenshot needed to show a finding, with its source and date, and the report itself, whose text can quote what a page shows, such as a business name or an email address. Raw pages fetched for a scan are kept for 2 days. To do its checks, the scanner may send the address of a store page to Google (PageSpeed Insights) and to archive.org, and, for Salla stores when the screenshot reader is switched on, the PageSpeed Insights screenshot of the home page to Anthropic (a screenshot can show contact details). Our basis is legitimate interest: helping store owners fix problems before an ad platform suspends them. Evidence is deleted after 90 days, or after 30 days when it may contain personal data. Reports have no set deletion date; we delete them when you ask. You can ask for access, correction, erasure or to object, at the address above.
This site counts page views with Cloudflare Web Analytics, which Cloudflare says does not track individual visitors across sites.
Frequently asked questions
Why is GrovlixScanner in my server logs?
Someone entered your store's address in a NoFlag scan, usually you or someone who works for you. It reads a limited, listed set of pages and files for that scan (see What it fetches) and then stops.
How do I block it?
Add a robots.txt rule for GrovlixScanner (above). If robots.txt disallows a page for GrovlixScanner, or for * when it has no group for GrovlixScanner, we do not read it in any way: not directly, not through Google PageSpeed Insights, not from archive.org. Or write to us and we will add your domain to our opt-out list: the scanner then refuses to scan the domain and its subdomains, even when another address redirects to them. Reports we already hold are deleted by hand, on request.
Does it change anything on my store?
No. The scanner itself never logs in, submits a form or places an order. Plain requests are only GET and HEAD. In browser mode your store's own page scripts run as they would for a visitor, including any requests they make.
Why does it use a browser user agent in browser mode?
Browser mode runs a real Chromium browser, which sends Chromium's own identity. It still follows your robots.txt rule for GrovlixScanner: a page that robots.txt disallows for GrovlixScanner (or for *, when there is no group for GrovlixScanner) is not read in any way, browser included.