LewisBot
LewisBot reads listing pages for one agent at a time.
When a real estate agent asks Lewis what is for sale in a town, Lewis reads the public listing pages for that town on their behalf. It says who it is on every request, and it takes no for an answer.
How to recognise it
Every request carries this user agent:
Mozilla/5.0 (compatible; LewisBot/1.0; +https://www.uselewis.com/bot)
There is no other name. LewisBot never presents itself as a person’s browser.
What it does
- Reads robots.txt first and follows the rules written for LewisBot, or for every bot when LewisBot is not named. A Crawl-delay is kept.
- One page per question. An agent’s search reads the town’s listing page on each site once, with at least a second between requests to the same site.
- Reads, never acts. It does not sign in, fill in forms, contact anyone, or save anything.
- Stops when told. A refusal, a rate limit or a bot check ends the request. LewisBot does not retry it, solve it, or route around it, and it leaves that site alone for a while afterwards.
How to turn it away
Add this to your robots.txt:
```
User-agent: LewisBot
Disallow: /
```
LewisBot reads robots.txt again at least every six hours, so the change holds from then on. You can also refuse its user agent at your firewall, and it will stop at the first refusal.
Questions
If LewisBot is reading something it should not, or you would like it to read your listings differently, tell us.