Being An Severe case in point, the following robots.txt entry restricts all crawlers from accessing any A part of your web site: Determining An important web pages on your web site might be instead difficult. Talk to your self: which web pages would I wish to show new customers? https://www.youtube.com/watch?v=EVeTP0aV8gY