How it identifies itself
Every request it makes, robots.txt included, carries this user agent, with the two exceptions below:
NewHomeSearchBot/1.0 (+https://newhomesearch.co.uk/bot)It is run by HomePortfolio HQ Limited, which trades as NewHomeSearch, from the pipelines of HomePortfolio, the company's data platform.
The exceptions are one housebuilder's website, which shows this crawler a challenge page but opens its pages to an ordinary browser, and one mortgage lender's published rates, which answer only a request that comes from the lender's own rates page. There alone it visits as a browser would: through a residential internet connection, presenting itself as an ordinary browser in place of the user agent above, and keeping the cookies a site sets for that one night's visit only. Everything else stays as it is everywhere: it follows that site's robots.txt for NewHomeSearchBot, path by path, asks for at most one page every 5 seconds and only for what changed (the lender's rates once a night), stops at a fixed amount of data each night, and stops for the night at the first challenge or refusal. It never solves a challenge.
What it reads
What housebuilders publish about their own new homes, and what mortgage lenders publish about their own rates, and nothing else:
- A builder's own website: its developments and homes for sale, with their prices, plots, house types, sales status, offers, descriptions and the addresses of their pictures.
- The brochures, floor plans and specifications those pages link to: the files themselves (PDFs and floor plan images), so that their text can be read and searched.
- A builder's logo and award badges, so the site can show them beside its homes, and a copy of some of its pictures where its website cannot be read from where the site runs.
- A mortgage lender's own published rates for buying a home: each product's rate, the rate after it, its fees, early repayment charges and the loan to value it is for, so the mortgage rates comparison can show them with a link to the lender's page.
It only reads: it never submits a form, and never reads anything behind a login.
How often
- One request to a site at a time, and at most one request to a site every 5 seconds, or less often when your robots.txt asks for a longer Crawl-delay.
- A lender's rates are asked for once a night: one request to each lender, besides its robots.txt, and never again that night, whatever happens after. A lender whose own rate finder answers one buyer at a time is asked once for each of twelve example buyers (six deposit sizes, for a first-time buyer and for a home mover), 5 seconds or more apart.
- Collection runs at night, UK time. NewHomeSearch also looks at a builder's logo or badge about once a day, to choose how to show it.
- A page or document it has read before is asked for with a conditional request, so an unchanged one costs a short "not modified" reply.
robots.txt
It reads your robots.txt before it asks for anything else on your site, and follows it, path by path, as the Robots Exclusion Protocol (RFC 9309) sets out, with one exception: a small number of specific public documents, which it fetches under a recorded decision although the robots.txt of the site holding them disallows them. Each decision names the documents it covers and the exact robots.txt rules, as published on the day of the decision, that it overrides. It lapses as soon as that robots.txt says anything more about those documents: a new rule, or a group naming NewHomeSearchBot.
To stop it everywhere on your site, those documents included, add:
User-agent: NewHomeSearchBot
Disallow: /A group in your robots.txt naming NewHomeSearchBot always decides what it may fetch, with no exception. To keep it away from some paths only, list them in that group as Disallow lines.
If your site refuses it
An answer of 401, 403 or 429, or a challenge page, stops it asking your site for anything more that night, and nothing is retried within it. It never solves a challenge.
Contact
To ask it to stop, to slow it down, or to report a problem, write to [email protected]. Please include the address and time of any request you are asking about.