Page 1 of 3 123 LastLast
Results 1 to 10 of 28
Like Tree3Likes

Thread: robots.txt

  1. #1
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default robots.txt

    Hi,

    I like some information about the working of the robots.txt file on the pianetadonna platform. There are two things I notice that makes me wonder.

    The first thing is that the entries in my robots.txt folder do not seem to work all that well. The entry "Disallow: /search*" should prevent the indexing of search-URL's but according to the Google search console the Googlebot still tries to index these search-URL's. Another similar example would be the entry "Disallow: /*feed*" which should prevent the indexing of all feed-URL's.

    I wonder if this could be due to a syntax error in my entries. Alternatively the my question would be if the robots.txt file is working properly.

    The second thing I noted is that I get a lot (thousands) of URL exclusions reported by the Google search console. The reason for these exclusions is "Blocked by robots.txt". If I try to inquire further the Google search console states that I do not have a robots.txt. I do have a robots.txt. However, in this robots.txt I do not exclude any pictures.

    I wonder if this could be an error in the Google search console reporting. Meaning, maybe the indexing of the pictures is blocked by something else then the robots.txt while the search console reports it as a robots.txt block anyway. Alternatively I wonder if there is a robots.txt file on a higher level on the pianetadonna platform that blocks the indexing of pictures.

    I hope you can help me out.

    Thanx,
    Gert
    laravista likes this.

  2. #2
    alemoppo is offline AlterVista Staff
    Join Date
    Feb 2010
    Location
    IT
    Posts
    397

    Default

    Hello, can you provide an example URL of picture blocked by the robots.txt?

    Bye!

  3. #3
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

  4. #4
    alemoppo is offline AlterVista Staff
    Join Date
    Feb 2010
    Location
    IT
    Posts
    397

    Default

    The robots.txt file is ignored in the subdirectories . Your robots.txt file is not used by Google to index content. Can you provide a screenshot with error messages to better understand the problem?

    Bye!

  5. #5
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default

    Do I understand correctly that you setup the pianetadonna platform in such a way that the websites running on this platform are positioned as a subdirectory instead of as a root directory? And that as a direct result of this setup users of the pianetadonna platform (and other Altervista platforms) cannot use the robots.txt file to direct the search engine indexing crawlers? This could potentially harm search engine positioning and with that the revenues from advertisements. Do you have any workarounds for this little but quite annoying problem?

    As regard to the second part of my question: http://tinyurl.com/us8o5fw

    Thanx,
    Gert

  6. #6
    alemoppo is offline AlterVista Staff
    Join Date
    Feb 2010
    Location
    IT
    Posts
    397

    Default

    Quote Originally Posted by msp View Post
    Do I understand correctly that you setup the pianetadonna platform in such a way that the websites running on this platform are positioned as a subdirectory instead of as a root directory?
    Your site is: https://blog.pianetadonna.it/msp/

    Quote Originally Posted by msp View Post
    And that as a direct result of this setup users of the pianetadonna platform (and other Altervista platforms) cannot use the robots.txt file to direct the search engine indexing crawlers? This could potentially harm search engine positioning and with that the revenues from advertisements. Do you have any workarounds for this little but quite annoying problem?
    The robots.txt file is not used to index the site. How it could potentially harm search engine positioning and the revenues? Please read this page.
    In the "AlterVista platforms" you can use the robots.txt because the url is like "yoursite.altervista.org".

    The second thing I noted is that I get a lot (thousands) of URL exclusions reported by the Google search console. The reason for these exclusions is "Blocked by robots.txt".
    As regard to the second part of my question: http://tinyurl.com/us8o5fw
    I'm sorry but i didn't see the "Blocked by robots.txt" string in your image.
    I don't recommend using wordfence or other security plugins because they can damage your blog and are useless on AlterVista.

    p.s: i just noticed that your site is in italian, why you don't ask on the italian forum?

    Bye!
    Last edited by alemoppo; 02-07-2020 at 09:05 PM.

  7. #7
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default

    is this any better?

    http://tinyurl.com/srklav5

  8. #8
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default

    Do I understand correctly that you setup the pianetadonna platform in such a way that the websites running on this platform are positioned as a subdirectory instead of as a root directory? And that as a direct result of this setup users of the pianetadonna platform (and other Altervista platforms) cannot use the robots.txt file to direct the search engine indexing crawlers?
    So this statement seems to be correct.

    Which leaves the question:

    Do you have any workarounds for this little but quite annoying problem?
    Last edited by msp; 02-08-2020 at 08:51 AM.

  9. #9
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default

    The robots.txt file is not used to index the site.
    Really? I thought that the robots.txt tells a search engine indexing crawler which URL's on a site it is allowed to crawl and which one it is not. With a working robots.txt you could block crawlers from indexing your site altogether. If the crawler is not disregarding the robots.txt of course. Or so I thought. You think this is not true?

  10. #10
    msp
    msp is offline AlterBlog User
    Join Date
    Jul 2012
    Posts
    29

    Default

    p.s. Do you think I could ask questions on the Italian forum in English? My Italian is limited to "mama mia" and some poetic swearing so that would hamper the Italian conversation quite a bit.
    Last edited by msp; 02-08-2020 at 08:53 AM.
    laravista likes this.

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO