Now that all the work has been done on identifying user requirements and reviewing the performance of the current search application perhaps the time has arrived to specify, select and install a new enterprise search application. In many respects the processes for selecting an enterprise search application is just the same as for other enterprise applications but there are some differences.
These include:
The company will not have undertaken the procurement of an enterprise search application before so there is no prior experience to go on.
Even if there are existing search applications the level of knowledge inside the IT department about how they work and how to evaluate an enterprise application is likely to be low.
There is probably no single business owner of search and yet it is because of the poor performance of existing search applications that the company is now undertaking the selection of a new application.
Most of the companies in the business are totally unknown to either the IT department or to procurement.
It is not easy for a company to understand the differences between the applications offered by the vendors and by open-source developers
If the procurement doesn’t deliver significantly better search performance the failure will be immediately apparent to most employees on the day it is launched.
This chapter provides advice on how to undertake the specification and selection of an enterprise search application.
Surely it should be ‘project team’? Not for an enterprise search application. There are three projects and a programme to be considered:
The project to write the specification for the enterprise search application
The project to select the vendor and/or integrator/developer
The project to install the application
The programme to manage the implementation
It is far too easy to see the enterprise search project as coming to a halt when the software is installed. One of the key messages of this book is that the hard work really starts once the software has been installed.
This chapter covers the management of the projects for writing the specification and selecting the vendor. Chapter 9 covers the installation and initial implementation and Chapter 10 provides advice on the on-going management of the application.
It is advisable to work out who is going to be on the three project teams and the search support team (see Chapter 10) right at the outset of the entire project. It may be that there are some gaps for which currently there are not people with the appropriate skills and experience, or perhaps the time to work on the project. These will need to be identified and filled ahead of the appropriate stage of the project. The gap that cannot exist is that of the Search Manager, the person who will ultimately have the responsibility for making sure that the search application meets the needs of the business and every individual employee who uses it. This position is so important that if a Search Manager cannot be identified and appointed then it would be advisable to delay the start of even writing the specification until the position is filled. The blend of skills in the three project teams will be different but the common factor acting as the glue to bind them together has to be the Search Manager.
So who should be on the project teams? There has to be a strong business focus to this team to ensure that the search engine delivers the information needed by the business and is not just another server in the data center. To keep the numbers down consider a three level approach.
The first level attend the project meetings. They need to have the time available not only to attend the meetings but to work on sub-projects between meetings. The second level consists of managers who may have some important contributions to make to the development of the specification, but perhaps only on specific topics. These might include managers with security management responsibilities and the managers of other applications that have embedded search functionality. There should be a mechanism to have excellent two-way communications with these managers, and perhaps they might participate in one or more of the project meetings as appropriate.
The third level has managers who need to be kept informed about the progress of the project, for example other business units, HR (as there could be recruitment or change-of-role decisions) and finance. They need to know about whether the project is on schedule because for example they may need to plan ahead to make space available for the implementation team. Everyone on the three levels needs to know who is on each level. Transparency is a major element of good project governance.
This team will be preparing the business case and the specification. There needs to be good representation of the business units that have most to gain from the new search application. In addition there will ideally be an enterprise architecture specialist, the intranet manager (who will not only know about search but will be in a position to support the communications plan for the project), someone with a good knowledge of the network architecture of the company and someone with experience of writing a reasonably complex specification. Add in a project manager, the Search Manager and the project sponsor and the result is a specification project team of perhaps eight to ten people.
If the Specification team have done their job well the selection team can have more of an IT focus. Security management is a very important element of a successful implementation and the project team needs to have expertise in identity management and/or systems security. At this stage obviously the procurement department needs to be closely involved. Again the Search Manager and project sponsor will be on the team along with the project manager.
The membership of this team is discussed in more detail in Chapter 9, but is included here because this team needs to be set up from the outset of the project.
The majority of enterprise search implementations will be across more than one country, and probably more than one language. Right from the outset the implications of a trans-country implementation need to be taken into account in setting up the project team that will take responsibility for preparing the specification, undertaking the selection and then managing the implementation. It might be the case that the initial implementation is in the USA and then the application will be rolled out more widely across the company. Even if this wider implementation is not going to be carried out for a year or more all the business and IT units involved need to have some degree of representation on the project team at one of the three levels set out above.
Even a few pages in to this Chapter you would be excused for wondering if all these project teams are really necessary. The answer is YES! Enterprise search is a very high touch application and if successfully implemented and managed will make a significant difference to the working lives of most employees. The risks of buying the wrong solution or not ensuring that there is an appropriate level of resource is very high indeed.
Whatever the project management approach being used a risk management section is very important. Risks are commonly scored by the impact on the project and the probability of the risk event arising. In the case of implementing an enterprise search application there will be not prior experience in the company, and using probabilities from other enterprise application implementations is not a sensible way to proceed. The approach to take is to weight the score heavily in favour of the impact on the project. Just multiplying a three level impact (Low, Medium, High) and a three level probability (Low, Medium, High) gives equal weight to both parameters.
For a search implementation a five level impact score is advisable that relates to the completion date of the project:
Will extend the project by no more than one month
Will extend the project by two months
Will extend the project by three – four months
Will require the project to be halted while a new project plan is developed
Will require the project to be terminated
The reason for taking this approach is that if the due diligence has been carried out on user requirements then the only risk that cannot be quantified is if the vendor either goes out of business or is acquired and there is an inevitable period of re-assessment and re-negotiation of the contract.
Other risks that need to be taken into account include:
A change in the project sponsor
An inability to manage secure access to confidential information
The loss of the Search Manager
Lack of internal IT resources because other enterprise implementations have been given a higher priority
At the Proof of Concept stage none of the vendors meet the core requirements
Poor quality content and/or metadata means that there is little perceived improvement in search quality
Substantial business change, such as a merger or acquisition
Risk scores are usually the product of impact and probability. Because there will be little or no previous search procurement experience it may be better to set out the impact and then the expertise available to address the issue should it arise.
Selecting and implementing an enterprise search application can take some time to bring to completion. The typical steps and the time that should be allocated to each step are set out in Table 8-1.
Table 8-1. Typical steps
Step | Duration |
|---|---|
Determine and document user requirements Develop a short list of vendors/developers/implementation partners Visit other companies who have recently implemented an enterprise search application | 2 months |
Prepare the Request for Proposal | 1 month |
Circulate to the short list of vendors and allow time for the response | 1 month |
Assess the proposals | 1 month |
Set up the Proof of Concept Invite vendors to participate | 1 month |
Undertake Proof of Concept evaluation and select a preferred vendor | 2 months |
Negotiate a contract | 1 month |
Prepare for installation | 1 month |
Implementation and acceptance testing | 1 month |
Initial roll-out and assessment of user experience | 2 months |
Total time | 13 months |
Adding in a Request for Information round will extend this project schedule by a month, and there could be additional time needed for the proposal preparation and evaluation stages if there are some regulatory requirements for public and Government procurement schedules.
It could be that in the case of one or more of these stages some reduction in time could be achieved, but it would be wise to work on the basis that from the time the decision is taken to implement a new search application to the time that users begin to use the fully-configured application is likely to be at least a year. During this time the business may have changed its objectives and/or there could have been changes to other enterprise applications, and it is advisable to have a major project review before the proposals are assessed to ensure that the specification is still valid.
Of course it will be different for your organization; it always is! You will look at this schedule and change the word ‘month’ into ‘week’ for many of the project stages. There are three possible outcomes:
You will reduce the elapsed project time but the search application will not deliver to requirements and expectations.
Costs and resources will be based on a shorter project time, but if the schedule cannot be met then there will be cost over-runs and members of the project team may have to go off to other projects.
The choice is yours.
The first decision to be made is whether to go straight to a Request for Proposal (RFP), also known as an Invitation to Tender (ITT) or to prepare a Request for Information (RFI) as a means of reducing the number of proposals received. The decision depends on the level of expertise that the company has, and also whether there is an intention to go down an open-source route. In the case of open-source applications there are an increasing number of development companies, and a preliminary shortlisting can be valuable. The route taken will also depend on the procurement policy of the organization, especially in the European Union where there are some EU-wide public sector procurement rules.
Most companies will have a preferred way of writing a specification. In this section the information that vendors will expect to see in the specification is set out. The order in which is appears in the specification is unimportant. With every one of these sections it is important to present not only the current state of affairs but what the expectations are going to be over the three years of operation after implementation.
Some background on the decision to go out to tender for an enterprise search application should be included. Vendors appreciate a high level of honesty at this stage because it enables them to judge what their approach should be in presenting the benefits of their solution.
This section should go into some detail about the volumes of content to be indexed, in terms of both raw storage but also an indication of the number of documents, the rate of addition of new content and how quickly this new content needs to be able to be searched. Also in this section the main file formats need to be listed, and of course any languages that need to be indexed and/or supported with language-specific interfaces. Even if the initial implementation is going to be for text-based content almost certainly the development of content analytics and search-based applications is going to create new opportunities and requirements in the not too-distant future.
The word ‘expectations’ is used rather than requirements because this section needs to cover the expectations of all the stake-holders. These expectations cover not just the search experience but the time-scale for the implementation and the ability to customize the search application without any further support from the vendor. If these are not set out in the specification they will do so at the selection meetings and can easily derail the entire process.
Hardware requirements can be a significant cost element, especially storage and network bandwidth so the current information systems architecture does need to be clearly set out. Again future intentions need to be clearly signaled, especially intentions to move towards cloud-based applications.
Many companies already have long-term contracts with systems integrators and have out-sourced development to companies based in India and some other countries. For a vendor some of these partnerships could be advantageous and others may present challenges because they may themselves have agreements with other search vendors. Included in these partnerships should be information about enterprise contracts with major suppliers of software and services, notably IBM, Oracle, Microsoft and HP, all of whom have significant interests in search technology.
This is especially important in the case of open-source projects. Vendors want to get a sense of who they will be dealing with at the installation and implementation stages, and it is advisable to present the corporate expertise as short profiles of individual members of staff. Almost certainly there will be other enterprise development projects taking place at the same time as the search implementation so there could be some potential issues about availability over the six-nine month period that it might take to fully deploy the search application.
The corporate approach to identity management and security management, both of access to the search system and also at a document level should be set out in detail. If there some current issues, or changes are going to made to system security (perhaps to accommodate access over mobile devices) then these need to be highlighted in this section.
The vision for enterprise search is that it will be able to search across multiple repositories and applications and present a ranked list of relevant information. This is certainly possible, but the costs and other implications are considerable. Although there may be some fine print in the functional requirements the expectations for federated search should be highlighted in the initial section of the specification. Also in this section should be a list of all current search applications (for example SharePoint 2010) and what the current plans are for upgrading these applications.
One of the most valuable benefits of enterprise search is being able to find individuals by name and by experience and expertise. In this section details of any HR databases should be given, together with the extent to which the company requires the enterprise search application to meet the requirements of national and EU-level data privacy legislation.
The first piece of advice is not to write a detailed functional specification that ends up with perhaps 500 individual functional requirements. This was certainly the case in a specification produced by a large financial institution a few years ago. There are a number of reasons why preparing a very detailed functional specification is not going to improve the chance of finding the best fit.
Most search vendors do not have a large team of people waiting around to prepare responses to RFPs which drop into their email boxes. They will look at the time it is going to take them to respond and they may well decide that filling in a response to 200 or more boxes is not a good return on their investment. Either they will not reply or they will just go through the motions and cut and paste content from the last such RFP they received
Probably at least 80% of all requirements can be met, to some extent, by all search vendors. The important functionalities will then be lost in the noise of the common features. There is no point asking for a list of all the file formats that the vendor is able to support when what is needed is absolute confirmation that the file formats that are important to the company can be handled with certainty.
The time it is going to take the project team to review each proposal is going to be considerable. By the time the last of ten proposals have been reviewed the team has almost lost the will to live and is not looking carefully enough at the proposals.
The approach that should be taken is to focus in on some areas where there are some differences of approach between vendors (and this includes open-source suppliers) and which therefore enable the team to come up with a well-considered short list for more detailed review.
If the search application needs to be able to index content from all required repositories or provide federated searches across applications then any concerns about the capability of the vendor to achieve this need to be identified at the outset. As with so many elements of search it is not just whether the connectors and APIs are available but the extent to which they have been deployed successfully in other clients.
Even if the connector technology is available just how the results presented on the user interface is equally important. There are a number of different approaches on offer and this is where you should be expecting to see some detailed screen shots of installations and ideally even a video demonstration of real interfaces.
The challenge for all search vendors is to be able to update the index with new content in a time that matches the requirement by the customer to be able to find recently indexed content. As with federated search there are various approached to index updating.
Most vendors now offer filters and facets to help users drill down into a set of results. It is important to check the extent to which these filters and facets can be modified by the Search Support Team without the need to involve the support team from the vendor.
Most companies will have some form of taxonomy, even if it is just a list of controlled terms or a list of approved abbreviations. Integrating these into the search application can be a challenge and understanding the way in which this can be achieved is important to clarify right from the outset.
If you cannot measure something then you cannot manage it, and that is certainly the case with search logs and system logs. Many vendors will have some standard search logs which are a good place to start, but creating just the views needed to manage your implementation could be a step too far and result in the need for the vendor to develop some customized reports
This topic has been covered in some detail in Chapter 6. Some vendors buy in third-party products from companies such as Teragram and Basis Technologies and others have developed their own entity extraction algorithms. As is so often the case with enterprise search it is not what is supplied with the initial install that matters as the ease with which changes can be made to the rules and algorithms on the basis of the experience gained following the initial implementation.
So far we have covered the base-line information that vendors will value and some of the functional requirements that need to be clarified. In addition there are some questions that need to be asked of vendors, the replies to which can be very valuable input into the evaluation process.
Any enterprise search implementation involves risk. The organization itself has never implemented search on this scale before and so has little idea of the specific risks there may be with the implementation of the search application from a specific vendor. However the vendor will have carried out a substantial number of implementations and should have a good idea of the risks and issues involved. It can be very illuminating to ask what the vendor sees as the main risks to a successful implementation based on the specification provided, and how it will work with the project team to ameliorate these risks?
It is not easy for the vendor to provide a definitive answer to the question about the duration of the implementation project, but it should certainly know how long it took in the case of other clients. Ask the vendor to provide a case study of a similar implementation, setting out the timetable from the time of starting the proof of concept and including information on the resources that both the vendor and the client contributed to the implementation.
The company may have its own approach to managing projects, often based on PRINCE2. A clear statement of the methodology used by the vendor will enable potential project management and communications issues to be identified at the earliest possible opportunity. Of particular importance is how red flag issues will be identified and dealt with. It is reasonable to ask the vendor to include some typical project management forms and procedures in the response to the proposal. Search implementation is quite complicated, as you will see from the next chapter, so the project management approach is a critical success factor.
It is useful to gain some understanding of the application development road-map of the search vendor. The search business is highly competitive and there is an understandable reluctance to go too public with a product road-map. Nevertheless there should be some element of comfort in seeing what the product roadmap might be and so assess the impact the future development possibilities.
The challenge with enterprise search is providing installation and implementation across multiple countries. The level of representation of search vendors globally is very variable. US vendors may have a representative in Europe but their task is pre-sales and some customer support. The technical teams are back at headquarters and that could be many time zones away. Global support issues need to be identified at an early opportunity. There is no point in acquiring a complex search application if the technical support cannot be effectively managed by the vendor or the vendor help-desk is based in the USA and so there is only a small time slot open for EU-based search operations.
Perhaps surprisingly currently there are not many vendor user groups. One of the understandable reasons for this is that the installed base in any single country is too small to support a national user group. With the technology now available there should at least be a regular virtual user group. The point here is to be certain about the quality of the dialogue between the vendor and the customer. You want to know about upgrades and bug fixes as soon as possible and also to feel that you have an influence on the way in which the search application is developed.
Even in quite large search vendors there are some employees engaged on either development or on installation that have accumulated a significant amount of expertise. Asking about whether the search vendor has a key employee strategy in place so that if one of these employees leaves the company is exposed in either development or in implementation expertise. Of the two skill sets implementation expertise is more important because it could directly affect the schedule and the quality of the implementation.
In the initial proposal it is unlikely that there will be more than an indication of the full cost of the implementation. Expect for Google and some other appliance vendors the algorithm for calculating the final amount payable will be difficult to unravel. Fixed cost contracts are very rare as there are so many unknowns from the viewpoint of the vendor at the stage of preparing the proposal. The only way to gain at least some sense of the final cost is to set out some scenarios for expansion routes and get at least some estimates from the vendors, but these will all be couched in very vague terms. It is not that they are being difficult, just cautious about committing themselves to a set of unknowns.
This is always a sensitive subject. The number of variables is such that assessing the extent to which an implementation and search satisfaction at one customer is going to provide any degree of insight into the performance in your case is asking far too much. What is worth exploring is the way in which the vendor and the reference customer worked together. Were there good channels of communication and were promises made that were not kept?
Some training can be delivered on the vendor premises on a test server but there will be a need for hands-on training during the installation and implementation stages. There should be a clear statement of how this is going to be carried out, and the prior experience that is expected of the staff being trained. Training employees up to be trainers themselves seems to be a smart idea but it is very important to make sure that the staff have the skills and time and incentives to be trainers of others,
As the list of vendors in Appendix B of this book illustrates there are around 80 search software vendors, but in reality the list of potential suppliers is a lot smaller. This is because many of the vendors are not in a position to support a multi-country implementation, or even a large-scale implementation in a single country which is many miles and time zones away. Many of these companies will have partnerships with sales and possibly implementation companies in other countries but often only limited business is conducted through these channels.
At the other end of the scale IBM, Oracle, HP and SAP have global sales, implementation and support networks, so in theory they should be in an ideal position to provide a multinational offering. The reality can be different. In 2012 and well into 2013 these companies are and will be digesting the acquisitions they made in 2011 and 2012, and it may not be clear exactly what is on offer at what price. If your company already has enterprise agreements with these companies then it would be foolish not to consider what they have to offer.
A good starting point is the Enterprise Search Report published by the Real Story Group, based in Washington DC. This report, one of many published by RSG, provides detailed profiles of the leading enterprise search vendors.
The 2012 report has profiles of:
Adobe
Apache
Coveo
Exalead
HP-Autonomy
IBM
Lexmark - Isys-Search
Microsoft
Oracle
Recommind
SAP
This is not a complete list of ‘enterprise search vendors’ but the merit of the report is that the detailed profiles highlight some of the issues that need to be taken into consideration when selecting a search application.
The search industry is also covered by a number of other consulting companies, notably The 451 Group, Forrester, Gartner and International Data Corporation. Many consulting companies try to summarize the leaders and laggards in a graphical format, but using these as the basis for a shortlist is not a good idea because search applications are far too complex to reduce to a two-dimensional diagram.
Another useful approach is to post a request for advice to the Linkedin Enterprise Search Professionals Group. There are currently over 7000 members. Many of the replies maycome via the private response route!
Many companies score the proposals they receive to help develop a short list to move onto the next stage. In the case of enterprise search the complexity of the functionality means that the differences between vendors in terms of delivering functionality are going to be small. You may end up with scores of 235, 246, 297 and 299. Dropping off the two low scores does not make sense, as each member of the team will be scoring the proposals with little previous experience of enterprise search.
This is why a multi-stage approach is the best option, with an initial Request for Information to come up with a list of perhaps six vendors to whom the Request for Proposal is then sent out. The aim should be to end up with no more than three vendors for the Proof of Concept stage.
There are consultants who maintain a strictly vendor-neutral approach to vendor selection projects, offering services from the user requirements work right up to supporting the project team in advising on the selection of a search application. These consultants also provide ongoing support post-implementation but very rarely get involved in the implementation work. One of them is writing this book!
As described in Chapter 7 many of the larger systems integration companies offer search implementation services. In addition there is an increasing number of specialized search implementation companies, some of whom will provide development services for open-source search applications. Most of these companies will be in a position to do everything from the user research through to implementation. It may seem a very convenient way to manage a search project but there is a substantial risk of ending up with an installed application and no knowledge of how it works and how it should be managed. Using an implementation partner should indeed be a partnership and organizations should be well aware of the benefits and risks of any IT implementation partnership.
If using a partner is attractive in terms of speed of implementation and overcoming a lack of internal expertise the first two questions to answer are how important speed of implementation really is and will the lack of internal expertise have a negative impact on the long-term management of the search application. It can be instructive to review other implementation partnerships that the organization has established and take the lessons learned into selecting and working with a search implementation partner. If there is no prior experience then the project risk increases substantially.
The big decision is whether the search application is chosen first and then an implementation partner is appointed or the partner is asked to advise on the selection of the search application. Most integration companies work with a small number of search applications with which they have partnership contracts and good access to expertise within the search vendor. This is especially the case with the larger general systems integration companies where search implementation has not been a major business for them in the past. One result of this is that they may not have much experience with the current version of the search software.
Many organizations have an incumbent systems integration partnership, perhaps as a result of outsourcing some elements of IT service provision. Care needs to be taken that the partner concerned has an appropriate level of knowledge of search technology and implementation. In addition if a second partner is appointed just to support the search implementation there is then a triangle of relationships between the organization, the incumbent systems integrator and the search integrator which could be a challenging test the skills of the project manager. This situation is especially likely to arise when the search vendor does not provide implementation services but works through a local partner.
There is no ‘best solution’ to partner selection. The decision will have both benefits and risks that are dependent on the organizational context, and these need to be worked through in detail before any decision is made.
In the final analysis you need to be certain that the implementation partner has your interests at heart and will be prepared to face up to a vendor problem from your perspective. If the integration partner has a commercial relationship with the vendor then there could be some conflicts of interest. These need to be surfaced, assessed and managed from the outset of the project, not when they arise during the project.
So far this chapter has been concerned with commercial software applications though many of the principles are common to open source software selection. However there are some important differences and these are the subject of this section.
The relationship with a commercial search vendor is very much about buying a software product with some consultancy services provided to assist with customization and support. The chances that there will ever be a meeting with the team who have written the application are very small indeed. With open-source software the business model is all about buying consulting services and it is very likely that you will be meeting developers who will go back to the office after the meeting and start writing code. It is possible to carry out the entire development operation in-house, especially if the organization has a strong Java development team, but the missing element will almost certainly be enough understanding how search works to build an application which meets not only current requirements but also future requirements. In-house development can also be hindered by the IT department missing out the user requirements and Statement of Requirements steps and just asking the development team to get on with it. Which they will until a ‘more important’ project comes along and the priorities of the development team are changed over-night.
Finding potential developers is not difficult. There is a list on the Apache Software Foundation site of people who have contributed code to Lucene and Solr, but many of these may work for large IT companies and will not be available for commercial development work. Using Google and Bing will also result in a list of potential developers, but there are probably fewer developers around than might be expected given the high visibility of Lucene and Solr.
Before beginning to approach potential developers it is important that all relevant managers have signed off on the use of open-source search software. Probably the only other example of open-source software in the organization is going to be a content management application, and these are far less complex than open-source search. With a few exceptions open-source search developers either work for small companies or as members of a virtual team. The statutory accounts of these companies may cause some problems for procurement departments more used to working with large multi-national IT companies.
Another aspect of working with small companies and virtual teams is that they may not be able to start work at once on a project. Indeed if they can it is worth finding out why as competent open-source search developers are in short supply.
There is no point in sending off a highly detailed Statement of Requirements at this early stage. The engagement has to be about both sides building a confidence in each other, and the road to defining the requirements is much more of a collaborative process than might be the case with a commercial vendor. The initial discussions should focus on understanding who the members of the development team would be and what their role would be in the project. The development team will certainly be focusing on what the milestones would be for the project, as it will certainly not be a turn-key development approach with the team going off for a few months and returning with the finished software. The milestones are needed to keep the project on track and also to define payment points. Some developers may work on a fixed-fee basis for a small project but for anything approaching an enterprise development the contract will be on a time and expenses basis. At the beginning of the engagement it might be quite difficult for the development team to give more than a broad estimate of the total cost of development.
The development team will largely work off-site and will need good access to the content that needs to be indexed. This can be a procedural challenge for an organization worried about the leakage of confidential data. It has to be recognized that any transgression on the part of the development team would be immensely painful to the company, to them personally and indeed to the open-source community. The fewer non-disclosure agreements and complex firewall protocols the better. Open-source development works best when all concerned see it as a win-win partnership. This win-win extends to the developers being able to share innovative code with the community.
As well as small independent development teams there are many larger companies, notably LucidWorks, who also offer open-source software development services. The business model may be different but the fundamental elements of a shared commitment to development success remain.
An organization may feel that it wants to hedge its bets and go out to tender to both commercial and open-source solutions. The problem with this approach is that at the evaluation stage the choice will be between apples and oranges. Using a productized open-source solution will make the process a little easier but at all costs resist the temptation to choose between a commercial solution and a bespoke open-source development.
This is sometimes referred to as a ‘bake-off’ and is a very important part of the overall selection process. The objective is to give the potential vendors the opportunity to demonstrate how well their technology works on real corporate information repositories and applications. Preparing for Proof of Concept tests is quite time consuming, and at the stage of writing the proposal the objectives of the PoC tests needs to be set out.
Two test collections should be developed. One of these should be a collection of perhaps 5000 documents against which a number of representative use cases can be run. This collection will enable some key performance parameters to be verified, such as speed of indexing, speed of updating the index, server performance, and the default search user interface. This is sometimes referred to as the Gold or Golden Collection. A proportion of this collection, say 20%, needs to be kept on one side for use in evaluating how the index is updated.
The second should be a collection of documents in every file format that has been identified in the content audit, the objective being to evaluate the performance of the document filters in indexing and in presenting documents in these formats.
Of major importance in a Proof of Concept is how security will be managed. Real ACL lists need to be presented and tested and setting these up can be time-consuming.
An even bigger challenge is to set up a federated search PoC, because this will require the vendor to have the appropriate connectors available. It is probably not worth the effort and a more pragmatic approach would be to assess federated search capabilities at some reference sites. Take careful note of relevance ranking issues.
Carrying out a Proof of Concept tests is probably the most time-consuming difficult and essential element of the selection process. A balance needs to be set between a reasonable level of investment on the part of the vendor on the tests and what the reasonable expectations of the company are for the outcomes of the tests. Typically a PoC may take a week to set up, test, run and evaluate. The conditions for all the vendors need to be the same and the project team from the company needs to be consistent. Servers need to be provisioned and a set of ACLs developed to assess security handling. The vendor team needs somewhere to work that can be kept secure; a desk in an open-desk area is not suitable. The tests may require the participation of IT staff and users in other countries and their availability has to be factored in to the schedule. This is why in Table X the duration of the PoC tests is shown as two months.
It will not be until the contract documents arrive will the full cost of the project become apparent. The vendor will have learned a lot during the PoC tests and will have factored in the impact of discoveries made during the process, especially about the skills of the team that will be responsible for supporting the implementation. It is advisable not to focus just on the cost of the initial installation and implementation. Enterprise search applications are scalable but the costs of scalability can be quite substantial, for example in terms of the costs of developing connectors for specific applications and repositories.
A key factor in the cost structure will be the extent to which the vendor regards the prospective customer as a reference site. If the customer is the first in the sector to invest in a search application then the potential business that could accrue from being able to make a lot of publicity from the win is worth quite a reasonable discount.
An element of the license cost that often only becomes visible at contract stage is the number of servers that are required to establish test, development and production environments and to be able to scale as more content and applications are indexed. Another factor that is often overlooked is the costs involved through an acquisition or through a divestment. Although it is not possible to foretell the future if the organization has growth through acquisition or divestment some scenarios should be discussed at the contract stage that take examples of both into account. This is not just the case of arriving at a ‘cost per user’ number but about understanding changes in support contracts. Search vendors will always be interested in increasing income but far less so if the acquisition or divestment means a reduction in support income.
It is not uncommon for the contract negotiations to take some time to conclude. Very rarely will a vendor table a fixed price contract. In the table at the beginning of this chapter a period of one month for these negotiations is suggested. That could be optimistic, and it would be wise to build in a float at this point.
Although organizations will usually have previous experience of selecting and purchasing enterprise-level solutions little of this experience will prepare them for an enterprise search project. The project could easily extend over a period of more than a year from the time of the initial decision to upgrade search capabilities to the day of launch. Bringing in specialist expertise from consultants and from systems integrators will reduce many of the risks but probably not the overall timetable. It is very important to specific what the search application needs to achieve in terms of business impact and not to provide vendors with a long list of features derived from a cut and paste of product documents downloaded from vendor web sites. Undertaking a Proof of Concept is essential. Working out the implications of the lessons learned from the PoC and the complexities of negotiating a contract can take a substantial amount of time and effort.
You'll find some additional information regarding the subject matter of this chapter in the Further Reading section in Appendix A.