Crawler
The affilinode crawler
Found this user agent in your logs? Then affilinode has read a page or a data sheet from your website. Here is why, which rules it follows and how to stop it.
User agent
affilinode-Anreicherung/1.0 (+https://affilinode.com/crawler)Why we read pages
affilinode is a product database for affiliate publishers. To give product boxes and comparison tables complete technical data, the enrichment fills in missing product details – such as dimensions, battery capacity or ports.
For this it reads product data sheets (including PDF), product pages of manufacturers and retailers, and results of a web search for the product. It evaluates the product details: structured data (JSON-LD), specification tables and the text of data sheets.
How often
Only when needed: a few pages per product, and again only while details are missing. Overall, only a few requests run at the same time.
robots.txt
- Before every page and every document we check the host’s robots.txt (following RFC 9309).
- The group for the product token “affilinode” applies; if there is none, the group “*”.
- The longest matching rule wins; for equal length, “Allow” wins. The wildcards “*” and “$” are supported.
- We cache robots.txt for 24 hours per host, so changes take effect within a day.
- Without a robots.txt (4xx response) everything is allowed. If the server returns an error (5xx) or does not respond, we fetch nothing for now and try again after an hour.
Some pages are rendered with a browser so their product data can be read. These requests may carry a common browser user agent; the robots.txt rules for “affilinode” apply to them as well.
How to block the crawler
The whole website:
User-agent: affilinode
Disallow: /Only one section:
User-agent: affilinode
Disallow: /intern/Questions or problems
Write to us at support@affilinode.com. Please include the host and roughly when the requests happened, so we can find them quickly.