Why cry so loudly?  Don’t they all know that because of search engines, there will be more traffic for their online news websites?  Search engines are where everyone nowadays goes to for exploring the web, and without search engines, it will be very hard for everyone to find new websites and information.  Yet, Rupert Murdoch acts as if he doesn’t need a search engine to bring web traffic to his online news websites.  Perhaps, Mr. Rupert Murdoch doesn’t need, but all other small online magazine websites are surely do need those search engines’ web traffics.

Microsoft is striking a deal with Rupert Murdoch to buy exclusive right for indexing Rupert Murdoch’s online contents in an attempt to boast Bing’s online present over Google.  It’s hard to say even talking to many others well known brands of newspapers and magazines besides Rupert Murdoch, can Bing really do some hurt to Google?  The Google brand is so strong that its loyal users are in a habit of open up Google’s search box even though they know how to type a website’s address directly into a browser’s address bar.  It shows that Google is a very strong brand that will be around for years to come, and search engine technology is keep on getting better.  Whether Mr. Rupert Murdoch knows or not, his online news contents are just a grain of sand among a dune of sand, and perhaps, many other online news magazines and news organizations love to take over Mr. Rupert Murdoch’s spot within Google’s search results.

For your information, legitimate search engines allow website owners to use robot.txt file to code in a way that tell what contents on their website are allowed to be indexed.  Some not so well-known search engines do not follow the robot.txt’s rules, and so these crawl for everything on all websites.  Many website owners do not like this, and so they block’s the IP addresses of these unorthodox search engines.  The search engines that aren’t follow the robot.txt’s rules can sometimes create a denial of service attack effect and bandwidth overload effect.

Whether Mr. Rupert Murdoch likes it or not, his websites will get crawl by the search engines that do not follow the rules within robot.txt file.  It’s hard to know it’s legal or illegal, and probably it’s perfectly legal to link to another website without the need of asking for a permission of a website owner. People can always shortly refer to the contents on Mr. Rupert Murdoch’s news websites, and linking to Mr. Rupert Murdoch’s news websites as the sources; Google then has the ability to crawl for those referrer contents, and in a way this is another way for Google to index Rupert Murdoch’s contents.  Exclusive contents do not belong on the web!  Unfortunately for Mr. Rupert Murdoch, nowadays, the web carries the information that people spend most of their times on.  Source.