Repository navigation
feat(devices): name the crawlers production reads generically - #2
Merged
Merged
Conversation
…ic of 2026-10-12 A dump of redirection.io's traffic of 2026-10-12 holds 150 user agents production answers with `Generic Bot`, or with the string itself. Thirty-seven of them name something whose operator says what it is: the search crawlers of Sofya and Nikysearch, Lyrenth's AI index, ByteDance's Dola, BeansLLM, Amazon Quick and Parallel's on-demand fetcher; the Checkmk, Distill, HetrixTools, PingBreak, ipMonitor and SolarWinds Observability monitors; Jetpack's image CDN, Stripe and Adyen webhooks, Sendcloud, Shopify and cert-manager outside its challenges; the Overture Maps and EPiServer link checkers; and the crawlers of Conductor, SEOJuice, SEO Gets, Oseox, Peec AI, Sender, Audigent, Captify, Gnowit, PressEngine, Cookie Information, Wine-Searcher, guenstiger.de, CardPulse and Rajahinta. feedparser 6, which names itself in lower case, now reads as matomo's UniversalFeedParser. The rest stays generic: a scraper someone wrote for themselves, a customer's own front end, or a token nothing on the web owns. Seven of those that look like something they may not be are pinned as generic beside the ones of last month. SofyaBot and NikyBot feed an index of their own and take the Search bot category, which moves their rows into the log injector's search-bot column, so `tests/pinned/traffic/false-negatives.yml` says why production's answer stands.
…2026-10-12 Five more strings production read as nothing: ORPI's property search application, PowerPoint importing from a URL with the year where the version goes, and three libraries that send their own name and little else -- R's default HTTPUserAgent, Deluge's downloader and the Jersey client on its HttpUrlConnection connector. `Deluge` and `R (...)` are short enough to read inside anything, so both are anchored on the whole string rather than searched for.
Ten mobile applications and two mail clients matomo added, Keeper's Android build, ocaml-cohttp, and the renames: `iOS Application` is now `Apple Core Media` and reads any AppleCoreMedia, RoboForm is `RoboForm Passwort Manager`. Iron asks for `Iron Safari` and Chrome Webview reads `Ojxvms/` as it reads `Version/`. matomo's `aria2(?!App)` is said as an ordering, Aria2App placed after Aria2 in the promoted clients, and `BlackBerry(?!-WorkConnect)` needs nothing: BlackBerry Work is a PIM, which already sits above the browsers. The two `^TPA` podcast rules are anchored and stay out, as the generator leaves every anchored rule out.
YodaOS-Master and YaOS, both in the Android family; SerenityOS no longer claims Ladybird, which runs elsewhere; and a Mac's Darwin may now be written `Darwin;` or `Darwin ` after any token rather than after CFNetwork alone.
Searched anywhere, the Internet Archive's crawler took Blackview's Zeno tablets with it, which matomo 6.5.2 now has fixtures for.
matomo 6.5.2 writes its Safari catch-all as `\bMacintosh`, so the `Power%20Macintosh` an old CFNetwork string carries for its architecture names no browser.
The fixtures move to e83ce7a, matomo's 6.5.2: 38 301 cases, 271 more than before. Every new case has its entry, one per user agent shape, in the file named after its fixture, tv-6 bringing a file of its own. What changes for an existing entry is matomo's own renaming: Macs named by their chip (`MacBook Air M1 13" (2020)`) and the Mac mini in lower case, Fairphones by their number, `Shift Phones` as `SHIFT`, Apple TVs, NextBook and Vivo models, Yandex's televisions on YaOS, and CHCNAV moved from the phones to the peripherals. The reduced user agent matomo now also reads, `Android 1x; <ten characters>; U;`, is answered from the client hints like the `K` one.
… belong matomo calls Bytespider and MJ12bot search bots and Meta-ExternalAgent an AI assistant. Meta documents its agent as crawling to train foundation models, Majestic's crawler feeds a backlink index and no search engine, and every tracker reads Bytespider as ByteDance's training-data collector. They are now AI Data Scraper, Crawler and AI Data Scraper. That contradicts the fixtures, so `tests/corrections.yml` says so: a bot, the category it gets instead, and the page that says why. The fixture loader applies it to every case naming that bot, `report` now reads the fixtures through the same loader, and the corpus fails on a correction no case needs any more, so it goes once matomo agrees. The two that left the search-bot column move to the log injector's tool one, which `tests/pinned/traffic/false-negatives.yml` excuses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Names crawlers, monitors, agents and applications that the detector only read as
Generic Bot, or not at all. Each one was looked up on the web. An entry was written where an operator or a project says what it is. Everything else stays generic.Two commits, one per typology, so a regression bisects to a file.
Bots —
src/devices/bots.ymlSofyaBot/1.0 (+https://sofya.co/bot)NikyBot/0.1 (prototipo NikySearch; +https://nikysearch.it/nikybot.html)AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)mozilla/5.0 (linux; android 14; pixel 8 pro) applewebkit/537.36 (khtml, like gecko) chrome/149.0.0.0 mobile safari/537.36 (compatible; dola-searchbot; dola-searchbot-feedback@bytedance.com)BeansLLM-CorpusBot/1.1 (+https://scrape.beansllm.ai/bot; respects robots.txt)amazon-Quick-on-behalf-of-ded61f40amazon-Quick-on-behalf-of-ddbedd22amazon-Quick-on-behalf-of-0374f8e5amazon-Quick-on-behalf-of-5fdf3a2camazon-Quick-on-behalf-of-93f852fbMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Shap-User/0.1.0checkmk-active-httpv2/2.5.0mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) distillwebmonitor/4.1.1-beta chrome/140.0.7339.249 electron/38.5.0 safari/537.36mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, like gecko) distillwebmonitor/4.1.1-beta chrome/140.0.7339.249 electron/38.5.0 safari/537.36HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.htmlMozilla/5.0 (compatible; PingBreak Free monitoring; +https://pingbreak.com/)Mozilla/4.0 (compatible; ipMonitor 11.1.1)solarwinds/1.0 (www.solarwinds.com/solarwinds-observability)Photon/1.0Stripe/1.0 (+https://stripe.com/docs/webhooks)Adyen HttpClient 1.0sendcloud/1.0 (magento_v2; https://www.sendcloud.sc)Shopify Webfile Service. http://www.shopify.com/contactcert-manager/v1.7.1 (clean)Overture-Maps-LinkChecker/1.0 (+https://overturemaps.org)EPiServer Link CheckerCaliperbot/1.0 (+http://www.conductor.com/caliperbot)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/botMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SEOGetsBot/1.0; +https://seogets.com/bot)oseox/1.0PeecAudit/1.0 (+https://peec.ai)Mozilla/5.0 (compatible; SenderBot/1.0; Sender Online Marketing GmbH; +https://sender-marketing.de/bot)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; AudigentAdBot; +http://www.audigent.com/bot.html) Chrome/W.X.Y.Z Safari/537.36captify-crawlerJava 1.7 Apache HttpClient (Linux x86_64) / GnowitNewsbot / Contact information at http://www.gnowit.comPressEngineBot (+http://pressengine.net/crawl-policy)Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/131.0.0.0 Safari/537.36 CookieInformationScannerwine-searcherbot (+https://www.wine-searcher.com/technology; crawler@wine-searcher.com)Mozilla/5.0 (compatible; BOT guenstiger.de)cardpulsebot/0.1 (+https://cardpulse.nl)rajahinta-crawler/1.0 (+https://rajahinta.fi)feedparser/6.0.8 +https://github.com/kurtmckee/feedparser/SofyaBot and NikyBot feed an index of their own, so they take the Search bot category.
Clients —
src/devices/clients.ymlOrpiaroundme/1 CFNetwork/3860.700.2 Darwin/25.6.0Orpiaroundme/1 CFNetwork/3860.700.1 Darwin/25.6.0microsoft office powerpoint 2014R (4.5.2 x86_64-apple-darwin20 x86_64 darwin20)DelugeJersey/2.35 (HttpUrlConnection 11.0.32)Left alone
main: Edge Health Probe, Amazon CloudFront, YisouSpider, NodePing, Zabbix, SEBot-WA, Chrome Privacy Preserving Prefetch Proxy, and the libraries Symfony BrowserKit, rss-parser, MWFeedParser, curl and undici.Mozilla/5.0strings, andMozilla/5.0 (compatible; crawler).node, which is in matomo's own generic list.lookuptoken, which needs its own change.Proof
cargo test --release --no-default-featuresandcargo test --release: every corpus case, the unit tests and the pinned tests pass.tests/pinned/production.rspins one case per entry, and each case fails against the database ofmain. Seven of the strings left alone are pinned as generic.Matomo 6.5.2
The fixtures move from
5a6f5d0toe83ce7a(matomo 6.5.2): 38 301 cases, 271 more than before. One commit per typology:google-xrawler, plusZenoread only as the whole string (matomo's^Zeno$), which the new Blackview Zeno fixtures would otherwise read as the crawler.iOS ApplicationbecomesApple Core Media, RoboForm becomesRoboForm Passwort Manager, Iron needsIron Safari, Chrome Webview also readsOjxvms/, and the Safari catch-all needsMacintoshto start a word.aria2(?!App)is written as an ordering instead.Darwin;orDarwinafter any token.SHIFT, Apple TV, NextBook and Vivo models, Yandex TVs on YaOS, and CHCNAV moving from phones to peripherals.cargo test --releaseandcargo test --release --no-default-featuresboth pass: 38 301 corpus cases, the unit tests and the pinned tests.Bot categories
Bytespider becomes AI Data Scraper, MJ12 Bot becomes Crawler and Meta-ExternalAgent becomes AI Data Scraper. Matomo's fixtures say Search bot, Search bot and AI Assistant.
tests/corrections.ymlrecords each disagreement along with its source, and the corpus fails once a correction no longer applies to any case.