What is Web Crawler and How Does It Work?
Do you ever wonder what makes the search engines go around?
It’s fascinating, isn’t it?
The way some mechanism can systematically browse the World Wide Web for web indexing or web spidering…
Yes, you are right.
A #webcrawler is precisely what we are talking about!!!
It’s an #internetbot,
also known as #webspider,
#automaticindexer,
#webrobot or simply #crawler…
Guess what else, web crawlers can do?
Don’t be surprised,
By the functions it performs:
1. update their web content or indices of others sites' web content
2. copy all the visited pages for subsequent processing by a search engine which will index the downloaded pages to provide lightning fast searches
3. automate maintenance tasks on a website, such as checking links or validating HTML code…
Do you know the popular open source web crawlers?
Here’s the list:
1. Scrapy
2. Apache Nutch
3. Heritrix
4. HTTack
How the web crawler works :
Enough of the theory, let’s jump right into
How a web crawler works:
1. Select a starting seed URL or URLs
2. Add it to the frontier
3. Now pick the URL from the frontier
4. Fetch the web-page corresponding to that URL
5. Parse that web-page to find new URL links
6. Add all the newly found URLs into the frontier
7. Go to step 3 and reiterate till the frontier is empty
Did you notice what’s happening here?
Wonderful, isn’t it?
It’s amazing how search engines work,
websites update their content or
do their maintenance tasks
But…
What’s more amazing is the way web crawlers make it happen…
Don’t you think so???
for more info about web crawler : http://www.prowebscraper.com/blog/50-...
Follow us on Twitter!
/ prowebscraper
Nesta página do site você pode assistir ao vídeo on-line What is Web Crawler and How Does It Work? duração hora minuto segundo em boa qualidade , que foi baixado pelo usuário ProWebScraper 10 Outubro 2018, compartilhe o link com seus amigos e conhecidos, no youtube este vídeo já foi visto 85,680 vezes e gostou 648 espectadores. Boa visualização!
