Log files - why you should care

In these competitive times - keeping an eye on your log files becomes increasingly important. Google-wanna-be's and other bigger sites simply attempting to lift off your hard work and use it at their will is happening every day - not just on an international scale - but also right here in the region.

As crawlers are becoming a $20 investment - anyone with limited or no skills at all can pretty much set up their own little spider and start harvesting content off your site and do what ever they like with it.

The nature of the web is that the content you publish is available for users to read and consequently also for crawlers to eat their way through - but just because content is available online it does not mean it is free for anyone to rip it off you.

So get your legal terms in place - ensure users acknowledge your terms of service when signing up and ensure that you include a well structured robots.txt on your server preventing "good" crawlers from getting to areas you want them to stay out of. Keep in mind that "evil" crawlers will ignore your robots.txt - hence watch your log files for spamming activity!