SEOClerks

Clear
Sign In Join

What type of Robots.txt file allow all bot to visit my site, my blog and make index it?

  • citadelsystems Level 1
    • 2

    Write the reason you're deleting this FAQ

    What type of Robots.txt file allow all bot to visit my site, my blog and make index it?

    Hello everyone...
    I want to know that what extension helps to allow all bot to index our sites and crawl them.
    thanks

    #Google #Crawl SEO
    Share post:

    Comments

    Please login or sign up to leave a comment

    Join
    rannt

    rannt

    Put this into your robot.txt file:

    User-agent: *
    Disallow:

    Thank you
    Gašper
    12 years ago
    1970-01-01T00:00:00+0000

    Are you sure you want to delete this post?

    Corzhens

    Corzhens

    From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.
    6 years ago
    1970-01-01T00:00:00+0000

    Are you sure you want to delete this post?