For website owners
Our web crawler
File 990 Now helps small nonprofits file their annual IRS return. Our crawler visits the websites that nonprofits list in their public IRS filings, or that we find for them, to read the contact details they publish: an organization’s own contact address, or its treasurer’s or president’s.
How it behaves
- It identifies itself as
File990NowDataEngine/1.0 (+https://file990now.com/bot), withbot@file990now.comin the From header. - It reads and obeys
robots.txt, including aCrawl-delay. - It reads a handful of pages per site at most (the home page, contact, about, board and policy pages), one at a time, with a pause of at least ten seconds between requests.
- It visits each site about twice a year, overnight in the site’s time zone.
- It stops when a site asks it to slow down, and it never tries to get past a block, a login or a challenge page.
- If your site says its email addresses must not be collected, or that it will not share them, the crawler stores nothing from it and deletes anything it already had.
What it does not do
It does not fill in forms, sign in, or read anything a visitor could not. It does not send email: addresses it finds are recorded so we know where they came from, and our mail system refuses to send to them.
Block it with robots.txt
To keep the crawler off your whole site, add this to the robots.txt at your site’s root:
User-agent: File990NowDataEngine Disallow: /
It checks robots.txt before every visit, so this takes effect the next time it comes by.
Opt out
Enter your website to have us delete what we hold from it and never visit it again, or an email address to have that one address removed and blocked. It takes effect immediately.
Or email bot@file990now.com with your website or the address you want removed, and we will reply when it is done.