For university and college websites
GyaanVidhyaBot
The crawler that reads public facts from official institution websites, and how you stay in control of it.
- User-Agent
GyaanVidhyaBot/1.0- robots.txt token
GyaanVidhyaBot
On this page 7 sections
What GyaanVidhyaBot is
GyaanVidhyaBot is the crawler Gyaan Vidhya uses to gather public, factual information about Indian universities and colleges from each institution’s own official website. Students use these facts to find and compare institutions.
Every record it creates is marked unverified until our team reviews it, and each one keeps the address of the page it came from and the date it was fetched. Facts an editor has corrected by hand are never overwritten by a later crawl.
How to recognise it
Requests carry a User-Agent that starts with GyaanVidhyaBot/1.0, usually followed by a link back to this page. In robots.txt it answers to the product token GyaanVidhyaBot.
User-Agent: GyaanVidhyaBot/1.0 (+<our site>/bot)What it collects
Short factual data points and links, nothing more:
- Institution name and other names it uses
- Address, PIN code, city and state
- Year of establishment
- Accreditation and approvals (such as NAAC grade or NIRF rank, when published)
- Streams and programme names
- Links to the official admissions page and logo (the logo itself is not downloaded)
- A short extract of the institution’s own description (up to 600 characters)
- Institutional contact details: role mailboxes on your own domain (admissions@, info@ …) and listed phone numbers
What it leaves alone
The crawler is built to avoid personal data and private areas of your site:
- Lines that name a person (Dr., Prof., Mr., Ms., Shri, Smt …)
- Staff, faculty-directory, alumni, careers, login and portal pages
- Images: nothing is downloaded or re-hosted
- Pages on other domains, including redirects that leave your domain
How it behaves
It reads your robots.txt before fetching anything else and follows its Allow, Disallow and Crawl-delay rules. A Crawl-delay longer than the default gap wins; a site that asks for more than 60 seconds between requests is skipped rather than crawled slowly. Default limits per visit:
- crawl per site at a time
- 1
- minimum gap between requests to one domain
- 3 s
- pages per visit, home page included
- 8
- read per page at most
- 2 MB
Each request times out after 15 seconds, follows at most 5 redirects within your own domain, and a failed visit is retried later with growing gaps, never in a tight loop.
Opt out with robots.txt
To stop GyaanVidhyaBot from crawling your site, add this group to the robots.txt at the root of your domain:
User-agent: GyaanVidhyaBot
Disallow: /Or keep it out of some sections and slow it down:
User-agent: GyaanVidhyaBot
Disallow: /admissions/drafts/
Crawl-delay: 10If no group names GyaanVidhyaBot, it follows your User-agent: * rules. robots.txt stops future crawling; to remove information already collected, write to us as described below.
Corrections and removal
You can ask us to correct a fact, remove your institution’s record, or stop crawling your site altogether. On a request we block the site from future crawls (later list uploads do not switch it back on), correct or hide the record, and keep a short log of what was asked and done.
Corrections and takedown requestsNot active yet
support@gyaanvidhya.example
This address is a placeholder for the preview and is not monitored yet. The contact page shows when support goes live.
Please include the institution’s name and official website, the Gyaan Vidhya page concerned, what should change, and an official source where one exists.
Continue reading