There were some great insights at last night's Xconomy Forum on the Future of Search and Information Discovery, and we'll get to those in a moment. But first, the primary challenge facing any company thinking about getting into the business was summarized succinctly in a zinger from panelist Harry Shum, the Microsoft engineer and corporate vice president in charge of core search development.

"Google is like smoking cigarettes," Shum said. "It’s a habit that’s going to be difficult to give up."

Oren Etzioni, the University of Washington professor and Farecast founder, joked in response: "It is a known fact that more Google users die every year than users of all other search engine combined."

Google would no doubt pick a different analogy, but its status as the search engine of choice is one of the first things people need to think about if they're thinking about competing in the search business. Finding opportunities around the edges was one of the themes of the forum, moderated by UW computer-science professor Ed Lazowska.

Read on for highlights. These are notes, not a transcript, but they capture the key comments and sentiments expressed by the panel.

Lazowska: There’s a trade-off between general search and focused search. Possible that a startup may introduce some new way of doing search, in a vertical, that people will use instead of general search.

Brian Bershad, Google Seattle site director: We’ve always tried to make search work generally. Until we feel comfortable that a given piece of search technology can generally work and be plugged into google.com search bar we tend not to promote it as a search property. There are many other ways of getting to intent other than a vertical search. More personal, understanding user's interest, social network, region, etc., all proven effective in making the Google.com one-box search work as well in many cases as a vertical.

Shum: I don’t think of Google as one box. Even Google has vertical search. It’s really a degree -- how fine-tuned you want the user to reveal the intent. Two questions: What types of tasks people have in mind. The second thing is discoverability. If you have thousands of applications, how can users find those important applications?

Steve Hall, Vulcan Capital: It is a real trend, exemplified by the convenience of apps on iphone and other mobile OSes. How many people type in the New York Times into search box? We all do, it's just easy and convenient. But on my iPhone I'm going to click the app. Bypassing search engine to go directly where I want for a small handful of apps. That's not the high value search. Not where these guys make their money. Even if they reduce search market share for navigational queries, not that big of a deal.

Etzioni: Old debate. Similar to the question of how many devices you want in your pocket. People argued that there wouldn't be a Swiss Army Knife approach until the iPhone came out. I would argue that people prefer a single solution so long as it works well. Once you have something that addresses a lot of your questions effectively, people will gravitate toward that.

Lazowska: What has changed in search in the past year?

Bershad: Couple interesting things. Voice search has finally come into its own. I now use it and don’t scream at it. Works mostly well enough most of the time. This is something we in computer science have been waiting 30 years for. Seeing a sizeable number of search queries coming in from Android and other devices where we released search by voice. Another change is social networking search, where queries go against not only general corpus but against social network. Can opt in to be able to search information made available by people in your contacts. Documents they've made available, etc. Third, starting to see rising real-time plus search. Both of our companies recently signed agreement with Twitter. At the very beginning of making useful information out of those streams. I don’t think it’s going to happen overnight. Finally, searching across documents written in other languages without you knowing document is in other language. Statistical translation systems are now working more reliably. Opens up whole new countries and regions.

Shum: Agree with Brian. In the least year real-time search, Twitter, getting to be very interesting. I think we’re only just seeing the beginning of that. Next couple years will be even more interesting. Bing trying to take search to next level, beyond 10 blue links. Trying to minimize the time you need to accomplish a task. It’s really about how to take structured knowledge and information into the corpus.

Lazowska: Hardest thing about teaching is understanding the misperception that underlies a malformed question. Students ask question, and there's some syntactic sense to the question, but the question often, if you're listening carefully reveals some underlying goal of the student, and the trick is to figure out what they're actually driving at, what they misunderstand. That's pretty clearly where search engines need to go. What I'm trying to accomplish as a searcher may be different than what I said.

Shum: Everybody is trying hard to understand user intent. Search engine has this advantage that people are willing to give you queries, not just single search. One power of search engines will be the ability to take the whole session of queries to figure out what the user intent is more precisely. It’s not about the query itself.

Lazowska: How much "crowd intent" is there? Must be a lot of instances in which suddenly a lot of people are groping for the same thing because it was on TV or in weather.

Bershad: Spelling correction is one example of crowdsourcing. Over time, you see that there aren’t realy an infinite number of ways to spell Britney Spears’ name, there’s really only 10. The crowd basically taught us the right spelling. Several years ago Google learned in Katrina that it had to get more serious about freshness. People searching for Katrina getting results about Russian czarinas instead. Pretty clear we were giving the wrong answer. Since that time we and everyone have been investing heavily in freshness. Stories of people updating blog post and then typing in search and seeing blog post updated within minute or so. That kind of freshness, you’re taught that you need to have it by the behavoir of the crowds.

Lazowska: How do you decide what the crawl frequency can be?

Bershad: You crawl as frequently as you possibly can.

Lazowska: Let's switch to semantic search. Understanding the semantics of material on the web. For example, Oren has done work trying to synthesize answers to questions where web pages don’t explicitly answer the question. Other extreme is semantics of user and user intent.

Etzioni: Most people don’t search for fun. There’s a task in mind. Even though we have great search engines, there's a bunch of tasks take way too long. Planning a vacation. Hours and days to sort through reviews, find availability. Check price on meta-search engine. I can imagine a day in the not too distant future, something that takes hours can happen in minutes. Have an engine that understands semantics, including what's being said in reviews -- positive review, negative? Rather than talking about semantics about an end, the computer understanding the web, you can think of it as a way to spend a lot more time.

Hall: We've got a few companies in the search space that use semantics to improve on keyword search. Keyword is great when you know what you’re looking for. Research Oren, type in his name, get some hits, great match. What if don't know what looking for? Show me news in general about computer science professors. Can't do that on Bing or Google. If you have a system that understands that Oren is a computer science professor, then he's going to show up in a semantic query that's going to get you that. "Show me Democratic senators that support carbon tax." Queries like that aren't really feasible with keyword syntax. One of our approaches is to invest in a startup you’re not going to go head to head against Bing and Google. Work around the edges. Applications, use cases that reduce the need to search in the first place. Automate content discovery. Understand intelligently the category of things I'm interested in, bring that to me on an automated base. Broad range of uses of semantics, just in the early days.

Lazowska: I wonder at some level what Bing can do. (To Shum) You're an engineer, a great engineer, but it's not just engineering. Most assessments say Bing is doing as well as Google in its general search results. But market share is still around 10% vs. 65%. It’s clearly not enough to do as well or slightly better. Bing has different approach, page jumps at me, Google looks like Russia in 1940s or something. What's the play, Harry?

Shum: First of all, huge respect for Google engineers. Fun to compete with Google in terms of technology and engineering. Why should an engineer work at Google, not many more problems for them to solve anymore. We really try hard to look at what Google is good at -- comprehensiveness of index, faster response. As an engineer, there are three problems in search: relevance, performance, user experience. A lot of those areas, Google has been leading, and we have been trying to catch up. With the Bing launch, I would challenge anyone in audience, most of the time if you can find something on Google’s first page you can find it on the first page of Bing. Google is like smoking cigarettes, it’s a habit that’s going to be difficult to give up. So what can you do? You have to think about the problem space. Google’s approach is to get people in and out of search engine quickly with their result. Not the right way to think about it. Right way to think about it is to think about minimizising time of completing a task, not minimize the amount of time to match a query with a url.

Etzioni: It is a known fact that more Google users die every year than users of all other search engine combined.

Bershad: Plenty of problems left to solve. I have a long list on my white board. Opening question is an example. No shortage of hard problems in search. The hardest problem, you left off your list of three, and that's comprehensiveness. Search needs to work across Web pages, newspapers, geoinformation, medical, newspapers, etc., all information for it to be ultimately useful. Comprehensiveness is actually what makes it challenging in terms of relevance, latency and user experience. Would be easy to get those things right if not concerned about user experience.

Lazowska: What could search engines do to keep news organizations in business in some way?

Bershad: Google sends about 100,000 clicks per minute to newspapers. That's a sizable contribution to their overall traffic. It's not clear that sending them more will help them monetize more given their same models. Not clear that Google or Microsoft or anybody but the newspaper industry can decide how they want to make money going forward. Craigslist is big issue. Not clear that the Googles and Microsofts of the world can do anything direct for them.

[Editor's Note: Microsoft's Shum didn't address this question. I approached him afterward to ask about reports that Microsoft is considering paying newspaper publishers for exclusive indexing rights in Bing. He declined to comment, deffering to the people on the business side of the Bing team.]

Lazowska: My sense is that there's less of adifferentiation between sponsor links and non-sponsor links nowadays. Seems to be blurring. Or a misperception on my part?

Shum: I think it's still pretty clear.

Bershad: Two sides of organization are separate (sponsored and algorithmic). Overall I think we're all working to make sponsor links more actionable. It’s not because we’re doing anything on non-sponsored side. Because we're trying to make sponsor links more actionable.

Shum: If you do it right, the sponsor links on the east side really add a lot of value. A lot of people actually use a search engine because they find relevant ads.

Lazowska: How many people here use a search engine to find relevant ads? Two people. I try not to click on sponsor links like I went 20 years without visiting Bellevue Square.

Shum: Technology behind sponsor links getting better. No paid inclusion in U.S.-based search engines, not true necessarily internationally.

Lazowska: Entrepreneurial opportunities. What are startup opportunities in next couple years for modest-scale startup, around edge of Bing and Google, small team could do something interesting in generally search-related area? What are the Pacific Northwest advantage/opportunities in search space? Google presence and enormous Microsoft presence, good startup community.

Hall: When we look at competing in search, wouldn't try to go head-to-head with keyword search. Three criteria: Can you do something that would prevent the need to search in the first place. (Gist one example among Vulcan portfolio, auomated intelligent discovery.) Second, can you be a better destination so that vertical search can be managed in a more discrete way. (Redfin gets massive amount of traffic from people searching address in Google or Bing, then gives them real estate structured vertical search on site). Third can you do something in semantics, differentiated from keyword search. (Evri is innovating around semantics.) Huge opportunity if you can intelligently categorize text and understand meaning and not just the syntax. Autocategorizing the web semantically is a huge opportunity.

Etzioni: Entrepreneurs should think about what is something that they would use if available? And others, too? That’s the first thought. Really want it. So badly that I’m going to spend years building. For example, searching for particular episode of Seinfeld or other TV show that I missed on the web. Example is clicker.com, attempts to solve that problem. Search is much broader than documents. Attempt to organize world’s information. We’re very far from end game on search on devices. Location-based search is one example. Lots of opportunities.

Bershad: Geo and search and mobile are three things that don’t work well together today. Phones can do neat tricks but how many people had to print out a map to figure out how to get here tonight? And I used to work here. Phone should just text with directions. GPS and voice directions should be integrated into the machine and background noise in my life. Opposed to Steve, I would actually not encourage small companies to go after anything in search having to do with text, because I think we’re going to get there. There are other areas where it’s an even playing field, and that’s where the real opportunity is. Geo-search mobile is an even playing field, and there are probably two or three more like that. Would look for those sorts of next-gen user interface issues.

Shum: Mobile is getting very interesting, with local. Open field, lots of opportunities for local. Two companies that we acquired that will hopefully serve as reference. Farecast and Powerset.com. Vertical travel search can help people complete tasks faster by understanding tasks better. Other is Powerset, natural language/semantic search. We believe in extracting semantic knowledge and using that to help search get better.

Audience question: As an entrepreneur, I've patented algorithm and turned that into a piece of software that turns six clicks into two clicks. Covers all three of Steve’s points. Unfortunately deals with what Brian said I should stay away from, which is text. What advice could you give me to successfully launching my application?

Bershad: You should get Steve to fund you.

Etzioni: To be very blunt, keep your day job because I’m not sure that turning six clicks into two is sufficient to get people to switch from Google and Bing.

Audience member: Doesn’t take over search engine, makes it easier.

New Audience Q: What has Google done that’s really an order of magnitude change over the past 10 years in search?

Bershad: In the area of comprehensiveness, corpuses have increased by more than order of magnitude. Speed, there's a definite order of magnitude improvement in terms of latency. Freshness it’s orders of magnitude. Probably biggest difference is in comprehensiveness. Ten years ago you could get most of the web on a couple of PCs. The short answer is that search is just as good but the web’s a lot bigger.

New Audience Q: The web is still visualized as flat in search engines. Why not better visualization, richer interfaces, self-organized maps, dimensionality.

Etzioni: Really fancy visualizations don’t work. Nobody wants to use them.

Bershad: We do have lots of visualization, not just links. Geo is an example. Google Earth. Present tremendous amount of rich information. All of that is not text. To Oren’s point, lots of stuff is going on in the lab, and you should be happy that it hasn’t launched.

Audience Q: When will search start synthesizing the target as opposed to being a navigation device? When will the result be the magazine or the movie you want to see?

Shum: That's pretty much where we see the future of search. It's really not just navigation. The direction is pretty clear -- you organize the information in a way that synthesizes the task that you want to accomplish. It is the direction that we're going in.

Bershad: Agree, you should eventually get the one thing you're looking for in the form you're looking for them. Three examples currently: Enter directions, you get a map. Enter lyrics from song, get click-through to actual song. Third is video and images, those come through. But it's all about delivering what the user is after as fast as possible.

Audience Q: Five years out. Any disruptive innovations in the field that will add new competitive advantage or depth to search?

Bershad: I really want to hear what’s coming out of Harry’s group.

Shum: Goes beyond 10 blue links and websites that we can find today. Goes back to understanding of user intent, and tasks they have in mind. The challenge is that there are millions and millions of tasks, which ones should a search engine company work on. Search entrepreneurs can participate by focusing on specific tasks?

Audience Q: How can you have a deeper relationship to help user with tasks?

Etzioni: Problem isn't technical, it's privacy concerns, how much people want search engines to know about them.

Shum: When you think about tasks, it's actually very difficult to describe. What do you really want to do? Not just plan vacation but buy tickets. Search engine provides a single place that you can hopefully do a lot of different tasks, starting from there, and then branch out.

Audience Q: Interested in the emerging conversations that are happening in social networks. People are talking about world’s challenges. I want to have access and insights into emerging conversations. How can I access some of the metadata that could be emerging through social networks. I want not just a piece of information. I want to know what conversation is happening about a certain topic that will actually move a conversation forward.

Bershad: Are you describing blog search for Facebook?

Audience member: No, I want to search for the juicy conversations, the ones that could be making a difference in our social systems.

Hall: Techmeme is one example. One of my favorite sites, because I don't have time to read 50 different blogs to find out what's happening. If you imagine the notion of “anything” meme -- any topic as granular as you want to get. This is what’s happening right now about this very high-fidelity topic. Not search but stitching together snippets of related info on web and turning it into high-value content.

Audience Q: How do you draw the line between sending people to a site for information and just delivering it within the search results page.

Bershad: I don't think there's a concious process there. This goes back a little bit to why I made the comment earlier about avoiding text if you're a small company going into the search field. There’s a long list, literally, a list of thousands of things that we would like to do in the search engine. As we work down that list, there may be some companies that are affected by that. We’re saddened by that, because it causes damage in the ecosystem, but I think that's what happens. Microsoft is in a very similar situation with boxed software. It's kind of the nature of the beast

Audience Q: What cultural and organizational weaknesses of Google and Microsoft could startups and other competitors exploit?

Hall: Mobile is huge. Apple is the big fish at the moment. Android coming on strong. Won't hold my breath on Microsoft. There's an exposure there. On the Microsoft side, they are still the large player in corporate email, Outlook, and that's an area where there hasn't been much innovation in a very long time. Their flagship Office suite is exposed quite a bit.

Etzioni: For a while, it did seem like there was, on the surface, little innovation coming out of Google's flagship. Good reasons, not to mess with success, simple and clean. Just recently I saw Google Sidewicki, which I think is an incredible innovation. The ability to write public notes on the side of a page. Makes web more informative and Google more sticky. Worry about Google would be inertia. As you become more successful you become more conservative, and I think that Google has done things to combat that in the past. But that would be one worry.

Shum: Google is a great organization. Competition is fun, and everyone is beatable. The great opportunity will come with the right timing. From a product point of view, it is a consumer product, sometimes people use a product a long time and they get tired of it. A lot of companies fall because the business changes, and it’s a matter of waiting for that.

Bershad: Two things which potentially threaten us. As we become bigger and older, it could become more difficult for Google to innovate. It’s very clear within the organization when there’s a piece that isn’t innovating. Like antibodies it gets surrounded and shaken up. Then they go back to hopefully an innovative state. I worry, I guess, that those antibodies would disappear. Also worry about diminishment of sense of entrepreneurship. It really still is the case that most teams are under 10 people, most things are started by a couple of engineers. Worry that would go away. That is really what I worry about. I don’t see them happening, but that’s what I would worry about.