Indexing and crawling
We are using OnionBot crawler software to build the full text search.
Furthermore, we are gathering a list of known .onion sites and information about the sites. We try in the following order fill as much as much information about the sites:
- Downloading the official /description.json
- Parse information from the HTML of the site
- Let the users wrote the information
We are honouring robots.txt files. This means that you can prevent our crawler from indexing your page by using robots.txt rules. This way your page is not shown in the search results. On the other hand, your page still shows in the list of all known hidden services that are online.