Link Crawling

Introduction

Link crawling is the process of capturing all the pages (URLs/links) present on a website. It helps us know how many and which pages are there in our website. Site owners can also crosscheck whether these page are legitimate or not.

MTvScan Technique

We crawl all the pages which are present on a website by actually visiting the pages and links which are present in the view source of the page. In crawling we capture URLs/Pages by following methods:

Page Visit: We visit each and every page and links present on them via this recursive process. If we get 200, 403 response then we add this in URL file.

Admin Buster: We append admin page keywords such as index.EXT, admin.EXT, login.htm, login/, login.EXT, login/login.EXT, adm/, siteadmin.EXT, etc. After scanning the URL for e.g. http://demotest.com/admin.php if we get 200, 403 response then we add this in URL file.

Directory Buster: We append admin page keywords such as index, images, wimages, imgs, img, iconset, icons, home, etc. After scanning the URL for e.g. http://demotest.com/admin if we get 200, 403 response then we add this in URL file.


1 Comment

Viruses, Malwares & Protection of Systems & Web Assets | | · July 6, 2018 at 5:24 am

[…] Equipped deep and proof-based scanning, the MTvScan software performs activities like robust link crawling, banner grabbing, CMS detection, Malware Scan including page defacement, JS Codes, Iframe check, […]

Leave a Reply

Your email address will not be published. Required fields are marked *