Xibo

robots.txt

Asked by X Xiao on 2013-11-28

I just noticed under sessions there are some strange info:
2013-11-27 06:14:08 login 109.68.190.145 masscan/1.0 (https://github.com/robertdavidgraham/masscan)
2013-11-27 09:36:23 login 109.68.191.26 masscan/1.0 (https://github.com/robertdavidgraham/masscan)
2013-11-27 09:49:40 login 42.156.136.54 YisouSpider

not sure what they're but I'm guessing that since I don't have a robots.txt under http://my.xibo.site which probably allows the search engine get in? what are these?

Question information

Language:: English Edit question

Status:: Solved

For:: Xibo Edit question

Assignee:: No assignee Edit question

Solved by:: Alex Harrington

Solved:: 2013-11-29

Last query:: 2013-11-29

Last reply:: 2013-11-29

Link existing bug

Revision history for this message

Dan Garner (dangarner) said on 2013-11-28:

As you say, they look like crawlers that you could prevent with a robots.text

They would not get past the login page.

Revision history for this message

X Xiao (austinxxh-linux) said on 2013-11-28:

just curious why xibo did not come with a default robots.txt? thanks.

Revision history for this message

Alex Harrington (alexharrington) said on 2013-11-28:

Because the search engine can only look at the login page and there's no
particular reason to want to stop that happening for all users. It's a
personal choice if you would prefer to exclude your site.

Revision history for this message

X Xiao (austinxxh-linux) said on 2013-11-29:

as my initial report shows, it does say the search engine is 'login', and I
could not log it out from admin panel, which made me wonder what 'login'
really means for xibo when robots.txt is not deployed.

On Thu, Nov 28, 2013 at 8:41 AM, Alex Harrington <
<email address hidden>> wrote:

> Your question #239947 on Xibo changed:
> https://answers.launchpad.net/xibo/+question/239947
>
> Alex Harrington proposed the following answer:
> Because the search engine can only look at the login page and there's no
> particular reason to want to stop that happening for all users. It's a
> personal choice if you would prefer to exclude your site.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/xibo/+question/239947/+confirm?answer_id=2
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/xibo/+question/239947
>
> You received this question notification because you asked the question.
>