Comments on: Page-store.com http://www.csamuel.org/2007/07/22/page-storecom The Thoughts and Feelings of a Melbourne Person Fri, 05 Mar 2010 20:34:51 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Toppy http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-16179 Toppy Tue, 15 Jan 2008 07:59:26 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-16179 Our site www.kantelpunten.com is currently being visisted by this mistery page-store. Unfortunately I cannot change the firewall settings. It does not seem to look at any rules, because it is also following links with rel="nofollow". Our site is completely dynamic and there are a lot of those links which should not be cached at all (It is not looping but they will be busy for a long while). Currently I am retruning them a page not to crawl us (so it will not find new links anymore). But i'm thinking of making a script which jumps to other generated pages for ever and fill up their sellable database with bogus. I assume they are not crawling just for fun. So does that mean that somebody wants to buy our content and make a similar site? Robots/crawler for search engine are welcome. But i want them to follow the robots.txt and other directives otherwise they will be banned from my site. There is one other thing which I noticed. This crawler will remember cookie information and use it in their requests. Does anyone know what they are doing with the crawled data? For example what is the selling price? Our site http://www.kantelpunten.com is currently being visisted by this mistery page-store. Unfortunately I cannot change the firewall settings. It does not seem to look at any rules, because it is also following links with rel=”nofollow”. Our site is completely dynamic and there are a lot of those links which should not be cached at all (It is not looping but they will be busy for a long while). Currently I am retruning them a page not to crawl us (so it will not find new links anymore). But i’m thinking of making a script which jumps to other generated pages for ever and fill up their sellable database with bogus.

I assume they are not crawling just for fun. So does that mean that somebody wants to buy our content and make a similar site?

Robots/crawler for search engine are welcome. But i want them to follow the robots.txt and other directives otherwise they will be banned from my site.

There is one other thing which I noticed. This crawler will remember cookie information and use it in their requests.

Does anyone know what they are doing with the crawled data? For example what is the selling price?

]]> By: Jack http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-16162 Jack Sun, 13 Jan 2008 18:41:16 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-16162 when you get one, try this URL http://XXX.XXX.XXX.XXX:8080/. replace the XXX with the ip of the crawler. Don't forget the ending DOT. Maybe you can shut it down. if you figure out how, please post it here when you get one, try this URL

http://XXX.XXX.XXX.XXX:8080/.

replace the XXX with the ip of the crawler. Don’t forget the ending DOT.

Maybe you can shut it down. if you figure out how, please post it here

]]> By: Phil Tillman http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-8334 Phil Tillman Mon, 06 Aug 2007 00:10:39 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-8334 Same story on a site that I manage. It tried to browse a directory protected by sessions and cookies; my site just directed it to a login page. Curious thing though is that it tried to access pages that are not linked to anything and don't have particularly obvious names. I wonder how it knew the pages were there? My logs are not public, but perhaps some of my users have left there tracks somewhere. Same story on a site that I manage. It tried to browse a directory protected by sessions and cookies; my site just directed it to a login page.

Curious thing though is that it tried to access pages that are not linked to anything and don’t have particularly obvious names. I wonder how it knew the pages were there? My logs are not public, but perhaps some of my users have left there tracks somewhere.

]]> By: Sugerør » Blog Archive » Who is page-store.com? http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-8080 Sugerør » Blog Archive » Who is page-store.com? Fri, 27 Jul 2007 15:34:38 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-8080 [...] I am not the only one who is being visited, it seems … [...] [...] I am not the only one who is being visited, it seems … [...]

]]> By: Robert Sweetnam http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-8060 Robert Sweetnam Thu, 26 Jul 2007 00:12:43 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-8060 As of tonight (26 July 07) I noticed the exact same spider crawling my site for the first time. It was following every link and ignoring my robots.txt so the easiest remedy was to block the address on my firewall. I also sent an e-mail to amazon but I don't expect a reply. As of tonight (26 July 07) I noticed the exact same spider crawling my site for the first time.

It was following every link and ignoring my robots.txt so the easiest remedy was to block the address on my firewall. I also sent an e-mail to amazon but I don’t expect a reply.

]]> By: TimC http://www.csamuel.org/2007/07/22/page-storecom/comment-page-1#comment-8047 TimC Mon, 23 Jul 2007 22:26:26 +0000 http://www.csamuel.org/2007/07/22/page-storecom/#comment-8047 And I notice they don't say what user agent they honour in robots.txt. Might have to ban them at the firewall instead. And I notice they don’t say what user agent they honour in robots.txt.

Might have to ban them at the firewall instead.

]]>