Crawler
UnodosNewsBot — the unodos.news crawler
UnodosNewsBot is the collection robot for unodos.news. It reads articles that Mexican news outlets publish publicly to build an internal news corpus for research and analysis. We do not republish anyone's articles.
What it is and what it is for
UnodosNewsBot is the collection robot for unodos.news. It reads articles that outlets publish openly and gathers them into an internal corpus we use for research and analysis. The corpus is not public: we do not republish anyone's articles, and we do not collect personal data.
How it behaves
- It respects robots.txt, including wildcard rules. If your site disallows access, we do not visit it.
- It makes one request at a time per site.
- It leaves at least 1 second between requests, and more when your crawl-delay asks for it.
- It keeps to a daily request cap per site.
- It identifies itself honestly: the User-Agent on every request starts with UnodosNewsBot/0.1 and includes this contact.
- It does not evade blocks, log in, or submit forms.
How we cite
Any article we publish cites and links its original sources. We credit the outlet and its authors, and we never publish someone else's work as our own.
How to opt your site out
We respect your robots.txt: if you disallow us, we do not visit. If you would rather we leave your site out entirely, write to [email protected] and we will exclude it.
How to reach a person
Write to [email protected]. A person reads that mailbox and replies.