Why Don’t We See Any New Search Engines?
|
| |
 | |
Has anyone else noticed that there are no “up and coming” search engines any more?
Yeah, there’s Gigablast. But they don’t appear to be making substantial progress. Yandex picked up a lot of ground in Russia. But by and large, we haven’t seen any new search engines of any consequence in years.
A long, long time ago….
There were a lot of search engines. Quite awhile ago. Lycos, HotBot, Altavista, Mamma, DogPile, Yahoo, Google, Search Cactus, NorthernLight, MSN, AOL, AllTheWeb. Then you had the meta crawlers, which all searched through those in a different way! Twas a marvelous(albeit hectic) time.
Nowadays….
They’re all gone, or out of sight and mind. Lycos? No traffic. Hotbot? Merely a meta search of Ask+MSN. Altavista? Almost identical to Yahoo. Mamma? Meta search. DogPile? No traffic. Search Cactus? No traffic. NorthernLight? God knows. AOL? Google clone.
They all condensed to feed off of maybe 3-4 larger search engines.
So Why are There No new Competitors Now?
- Startup Costs - The days of the .com bubble are gone, and venture capitalists are no longer willing to dump millions of dollars into a venture that probably will not work out.
- Firmly Entrenched Competition - The search engines have become household names. It’s not just a question of ranking anymore(which is hard enough), but penetration into the public eye. For god’s sakes, Google has become a verb!
- Suspicion of New Crawlers - This one is very important. People don’t like seeing new crawlers. Random adult-filtering companies, intellectual protection companies, and a gang of other places started crawling with poorly configured/written web crawlers, draining bandwidth for no reward. So people began blocking them. This is a tremendous problem for any new search engine.
- Long Time Between Investment, and Reward - It takes a long time to crawl enough sites to be useful. Even longer to get advertisers. It actually might never happen. This is a HUGE problem for potential investors.
- The Brain Drain - It has been often commented on that Google itself has drained all of the smartest people from the industry, leaving little for the rest of the companies, and in doing so, hogging innovation. When you add in Yahoo and MSN, it’s almost impossible for a new search engine to find decent engineers with any kind of experience in search related technologies to head up it’s project.
How I Realized This Problem
Awhile ago(in my early SEO days), I ran an RSS syndicator. It scraped RSS feeds, categorized them, and sourced them. It allowed Google to crawl it’s categories/posts though. Bear in mind, my crawler announced itself, followed robots.txt, and the site itself had an opt-out mechanism.
Well, the site got several legal threats, and got the hosting suspended 3 times. This basic distrust and suspicion has fueled problems.
So What do you all think? Will we be seeing any new search engines in the future? Or is this what we’re stuck with for the next several decades? If there are no more, we might truly become Google’s bitch some day.
December 31st, 2007 at 1:57 pm
What about Wikia, Wikipedia’s new search engine that will be out in a month or two, or Mahalo? I think there are more out there you never hear about. I just don’t think yet another search engine is newsworthy enough to spread.
On a side note, I’m buying xrumer today, could you email me your recommended hosting company you mentioned in your xrumer article? It would be greatly appreciated.
December 31st, 2007 at 2:42 pm
Sure. I’m going to be out and about, but I’ll try and get a second to drop an e-mail.
And yeah, that’s just the problem. There’s niche engines, but nothing you ever hear about. There’s no real contender trying to break into the general search market.
December 31st, 2007 at 3:23 pm
I’m sure ’scraping’ was the keyword there not bandwidth, distrust etc..
December 31st, 2007 at 3:25 pm
You have to do two things.
1) Provide results that are dramatically better than the competition. Thats what Google did, and all anyone has been able to do is try to keep up with them for quality of results.
2) Hook up with a traffic source that can send you searchers. Google and Yahoo aren’t going to give you any of theirs, both work hard to filter search result pages out of their index.
Ask gave a good attempt at the above, plus major media advertising and it didnt work. A9 tried for the above two factors also, and it didnt work. M$ is trying as hard as it can to leverage its platform and its not working.
I don’t even think A9 and Ask were dramatically better than G in terms of quality but they at least were dong the Universal Search format ahead of Google.
Whenever something shows potential to be a traffic source that could use search to get advertisers. Google or Yahoo just buys it anyway.
Mad props for naming Yandex (dont forget rambler.ru either). Russia is one of the few places where Google is competing that they aren’t winning.
PS Mahalo is a piece of poop.
December 31st, 2007 at 8:05 pm
This search engine is going to be cool. http://www.powerset.com/ It’s a natural language search engine. Hopefully it will actually work the way it says it’s going to.
Look at its founders and investors.http://www.powerset.com/team/directors One of them is the guy who co-founded Paypal. Maybe, just maybe, it can actually give google a run for it’s money. But it has to actually do what it says it’s going to for that happen.
December 31st, 2007 at 8:14 pm
Got pointed at assista.com this morning.
I think they need to work on the interface. It doesn’t grab.
January 1st, 2008 at 1:06 am
Baidu is yet another search engine that beat Google in China, and is the 3rd biggest se on the world base on the amount of search queey ,I am expecting to see baidu’s international expansion.
January 1st, 2008 at 1:07 am
Baidu is yet another search engine that beats Google in China, and is the 3rd biggest se on the world base on the amount of search query ,I am expecting to see baidu’s international expansion.
January 1st, 2008 at 1:16 am
@terman: I doubt it. Baidu concentrates on crawling chinese pages. So sites without chinese backlinks are out of luck, and sites inside of the chinese niche are ensured success. Baidu’s entire success has been based on their limited crawling.
Although I guess time will tell.
January 1st, 2008 at 2:01 am
I do not think we will see any new search engine come along mainly for the amount of money that it would take to market the name to the general public. The time it would take and the millions of dollars in offline advertising would scare away many investors. There is also the thought of if investors did step fourth with the attempt, how bad would it fail. We have the big 3 that people use and probably many have as there home page. Be better off to start a new search engine and build new computers as well to set the homepage for you. lol
January 1st, 2008 at 8:42 am
To xmcp:
I dont think Baidu’s success is mainly for their focus on Chinese pages.Google has their own Google.cn,which is specifically for Chinese and whilst over 4 other China local search engines are competing as well.Their CEO originally worked for infoseek.I do believe Baidu is ambitious of English web world,though not now immediately ,we will see.
January 1st, 2008 at 12:17 pm
That’s because you don’t hang out at the right places. My forums have discussed a few new ones in beta over the months. None of which have been mentioned in this thread. Some of them “very” promising.
And Slightly; you know what forums I’m talking about.
If you would step outside your little “spin” thang you would find out lots of good info.
January 1st, 2008 at 12:36 pm
@Dough
I can assure you I’m outside of sphinn a fair amount. It’s a nice community, but not all there is.
And I’ll dig around your forum a bit more to look for a few of these. I’m sure you understand. It was kind of overload there for a bit. But whether or not we agree on everything(or anything) you’ve got some damn smart people over there.
January 1st, 2008 at 2:35 pm
Question though Doug is who are they “very” promising to? You, your forum members, the internet marketing community or the general public? If it is very promising to the general public and they have the finances to back them up then the search engine can compete. If you do not have those two then forget about it, no matter how promising it is they might as well remain at the bottom.