Base class for Locking implementation. {@link Directory} uses instances of this class to implement locking.

Note that there are some useful tools to verify that your LockFactory is working correctly: {@link VerifyingLockFactory}, {@link LockStressTest}, {@link LockVerifyServer}.

Typical use might look like:

            new Lock.With(directory.makeLock("my.lock")) {
            public Object doBody() {
            ... code to execute while locked ...
            }
            }.run();
            

This query uses the {@link MultiTermQuery#CONSTANT_SCORE_AUTO_REWRITE_DEFAULT} rewrite method.

This query cannot be used directly; you must subclass it and define {@link #getEnum} to provide a {@link FilteredTermEnum} that iterates through the terms to be matched.

NOTE: if {@link #setRewriteMethod} is either {@link #CONSTANT_SCORE_BOOLEAN_QUERY_REWRITE} or {@link #SCORING_BOOLEAN_QUERY_REWRITE}, you may encounter a {@link BooleanQuery.TooManyClauses} exception during searching, which happens when the number of terms to be searched exceeds {@link BooleanQuery#GetMaxClauseCount()}. Setting {@link #setRewriteMethod} to {@link #CONSTANT_SCORE_FILTER_REWRITE} prevents this.

The recommended rewrite method is {@link #CONSTANT_SCORE_AUTO_REWRITE_DEFAULT}: it doesn't spend CPU computing unhelpful scores, and it tries to pick the most performant rewrite method given the query. Note that {@link QueryParser} produces MultiTermQueries using {@link #CONSTANT_SCORE_AUTO_REWRITE_DEFAULT} by default.

Instantiable subclasses are:

• {@link TermQuery}
• {@link MultiTermQuery}
• {@link BooleanQuery}
• {@link WildcardQuery}
• {@link PhraseQuery}
• {@link PrefixQuery}
• {@link MultiPhraseQuery}
• {@link FuzzyQuery}
• {@link TermRangeQuery}
• {@link NumericRangeQuery}
• {@link Lucene.Net.Search.Spans.SpanQuery}

A parser for queries is contained in:

• {@link Lucene.Net.QueryParsers.QueryParser QueryParser}

The representation used is one that is supposed to be readable by {@link Lucene.Net.QueryParsers.QueryParser QueryParser}. However, there are the following limitations:

• If the query was created by the parser, the printed representation may not be exactly what was parsed. For example, characters that need to be escaped will be represented without the required backslash.
• Some of the more complicated queries (e.g. span queries) don't have a representation that can be parsed by QueryParser.

Only implemented by primitive queries, which re-write to themselves.

A utility for use by {@link #Combine(Query[])} implementations.

This method is faster than the BooleanQuery rewrite methods when the number of matched terms or matched documents is non-trivial. Also, it will never hit an errant {@link BooleanQuery.TooManyClauses} exception.

NOTE: This rewrite method will hit {@link BooleanQuery.TooManyClauses} if the number of terms exceeds {@link BooleanQuery#getMaxClauseCount}.

NOTE: This rewrite method will hit {@link BooleanQuery.TooManyClauses} if the number of terms exceeds {@link BooleanQuery#getMaxClauseCount}.

This method is not thread safe, be sure to only call it when no query is running! If you re-use the same query instance for another search, be sure to first reset the term counter with {@link #clearTotalNumberOfTerms}.

On optimized indexes / no MultiReaders, you get the correct number of unique terms for the whole index. Use this number to compare different queries. For non-optimized indexes this number can also be achived in non-constant-score mode. In constant-score mode you get the total number of terms seeked for all segments / sub-readers.

Note: In Lucene 3.0 {@link #Bits(IndexReader)} will be removed and {@link #GetDocIdSet(IndexReader)} will be defined as abstract. All implementing classes must therefore implement {@link #GetDocIdSet(IndexReader)} in order to work with Lucene 3.0.

Note: This method will be called once per segment in the index during searching. The returned {@link DocIdSet} must refer to document IDs for that segment, not for the top-level reader. @param reader a {@link IndexReader} instance opened on the index currently searched on. Note, it is likely that the provided reader does not represent the whole underlying index i.e. if the index has more than one segment the given reader only represents a single segment.

A Scorer iterates over documents matching a query in increasing order of doc Id.

Document scores are computed using a given Similarity implementation.

NOTE: The values Float.Nan, Float.NEGATIVE_INFINITY and Float.POSITIVE_INFINITY are not valid scores. Certain collectors (eg {@link TopScoreDocCollector}) will not properly collect hits with these scores.

• -1 or {@link #NO_MORE_DOCS} if {@link #NextDoc()} or {@link #Advance(int)} were not called yet.
• {@link #NO_MORE_DOCS} if the iterator has exhausted.
• Otherwise it should return the doc ID it is currently on.

NOTE: in 3.0, this method will become abstract.

            public int nextDoc() throws IOException {
            return next() ? doc() : NO_MORE_DOCS;
            }
            

Behaves as if written:

            int advance(int target) {
            int doc;
            while ((doc = nextDoc()) < target) {
            }
            return doc;
            }
            

NOTE: certain implemenations may return a different value (each time) if called several times in a row with the same target.

NOTE: this method may be called with {@value #NO_MORE_DOCS} for efficiency by some Scorers. If your implementation cannot efficiently determine that it should exhaust, it is recommended that you check for that value in each call to this method.

NOTE: after the iterator has exhausted you should not call this method, as it may result in unpredicted behavior.

NOTE: in 3.0 this method will become abstract, following the removal of {@link #SkipTo(int)}.

Is thread safe and completely reusable.

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

At its default/simplest form, values - one per doc - are used as the score of that doc.

Values are instantiated as {@link Lucene.Net.Search.Function.DocValues DocValues} for a particular reader.

ValueSource implementations differ in RAM requirements: it would always be a factor of the number of documents, but for each document the number of bytes can be 1, 2, 4, or 8.

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

Mandatory: every DocValues implementation must implement at least this method.

Optional: DocValues implementation can (but don't have to) override this method.

Optional: DocValues implementation can (but don't have to) override this method.

Optional: DocValues implementation can (but don't have to) override this method.

Optional: DocValues implementation can (but don't have to) override this method.

Allows tests to verify that loaded values are:

1. indeed cached/reused.
2. stored in the expected size/type (byte/short/int/float).

This operation is optional

This operation is optional

This operation is optional

The query matches all documents, and scores each document according to the numeric value of that field.

It is assumed, and expected, that:

• The field used here is indexed, and has exactly one token in every scored document.
• Best if this field is un_tokenized.
• That token is parsable to the selected type.

Combining this query in a FunctionQuery allows much freedom in affecting document scores. Note, that with this freedom comes responsibility: it is more than likely that the default Lucene scoring is superior in quality to scoring modified as explained here. However, in some cases, and certainly for research experiments, this capability may turn useful.

When contructing this query, select the appropriate type. That type should match the data stored in the field. So in fact the "right" type should be selected before indexing. Type selection has effect on the RAM usage:

• {@link Type#BYTE} consumes 1 * maxDocs bytes.
• {@link Type#SHORT} consumes 2 * maxDocs bytes.
• {@link Type#INT} consumes 4 * maxDocs bytes.
• {@link Type#FLOAT} consumes 8 * maxDocs bytes.

Caching: Values for the numeric field are loaded once and cached in memory for further use with the same IndexReader. To take advantage of this, it is extremely important to reuse index-readers or index-searchers, otherwise, for instance if for each query a new index reader is opened, large penalties would be paid for loading the field values into memory over and over again!

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

This query provides a score for each and every undeleted document in the index.

The value source can be based on a (cached) value of an indexed field, but it can also be based on an external source, e.g. values read from an external database.

Score is set as: Score(doc,query) = query.getBoost()² * valueSource(doc).

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

The purpose of {@link Weight} is to ensure searching does not modify a {@link Query}, so that a {@link Query} instance can be reused.
{@link Searcher} dependent state of the query should reside in the {@link Weight}.
{@link IndexReader} dependent state should reside in the {@link Scorer}.

A Weight is used in the following way:

1. A Weight is constructed by a top-level query, given a Searcher ({@link Query#CreateWeight(Searcher)}).
2. The {@link #SumOfSquaredWeights()} method is called on the Weight to compute the query normalization factor {@link Similarity#QueryNorm(float)} of the query clauses contained in the query.
3. The query normalization factor is passed to {@link #Normalize(float)}. At this point the weighting is complete.
4. A Scorer is constructed by {@link #Scorer(IndexReader,boolean,boolean)}.

NOTE: even if scoreDocsInOrder is false, it is recommended to check whether the returned Scorer indeed scores documents out of order (i.e., call {@link #ScoresDocsOutOfOrder()}), as some Scorer implementations will always return documents in-order.
NOTE: null can be returned if no documents will be scored by this query.

NOTE: the default implementation returns false, i.e. the Scorer scores documents in-order.

The type param tells how to parse the field string values into a numeric score value.

The type selected at search search time should match the data stored in the field. Different types have different RAM requirements:

• {@link #BYTE} consumes 1 * maxDocs bytes.
• {@link #SHORT} consumes 2 * maxDocs bytes.
• {@link #INT} consumes 4 * maxDocs bytes.
• {@link #FLOAT} consumes 8 * maxDocs bytes.

This is the same functionality as TermsFilter (from contrib/queries), except this filter requires that the field contains only a single term for all documents. Because of drastically different implementations, they also have different performance characteristics, as described below.

The first invocation of this filter on a given field will be slower, since a {@link FieldCache.StringIndex} must be created. Subsequent invocations using the same field will re-use this cache. However, as with all functionality based on {@link FieldCache}, persistent RAM is consumed to hold the cache, and is not freed until the {@link IndexReader} is closed. In contrast, TermsFilter has no persistent RAM consumption.

With each search, this filter translates the specified set of Terms into a private {@link OpenBitSet} keyed by term number per unique {@link IndexReader} (normally one reader per segment). Then, during matching, the term number for each docID is retrieved from the cache and then checked for inclusion using the {@link OpenBitSet}. Since all testing is done using RAM resident data structures, performance should be very fast, most likely fast enough to not require further caching of the DocIdSet for each possible combination of terms. However, because docIDs are simply scanned linearly, an index with a great many small documents may find this linear scan too costly.

In contrast, TermsFilter builds up an {@link OpenBitSet}, keyed by docID, every time it's created, by enumerating through all matching docs using {@link TermDocs} to seek and scan through each term's docID list. While there is no linear scan of all docIDs, besides the allocation of the underlying array in the {@link OpenBitSet}, this approach requires a number of "disk seeks" in proportion to the number of terms, which can be exceptionally costly when there are cache misses in the OS's IO cache.

Generally, this filter will be slower on the first invocation for a given field, but subsequent invocations, even if you change the allowed set of Terms, should be faster than TermsFilter, especially as the number of Terms being matched increases. If you are matching only a very small number of terms, and those terms in turn match a very small number of documents, TermsFilter may perform faster.

Which filter is best is very application dependent.

Note that this does not do line-number counting, but instead keeps track of the character position of the token in the input, as required by Lucene's {@link Lucene.Net.Analysis.Token} API.

It is up to the implementation to make sure it is thread-safe.

Default implementation IGNORES the document number. Override if your implementation needs the document number.

NOTE: Document numbers are internal to Lucene and subject to change depending on indexing operations.

The document portion names each document containing the term. Documents are indicated by number. The frequency portion gives the number of times the term occurred in each document.

The pairs are ordered by document number.

This is invalid until {@link #Next()} is called for the first time.

This is invalid until {@link #Next()} is called for the first time.

Returns true iff there is such a next pair in the enumeration.

Returns the number of entries read. Zero is only returned when the stream has been exhausted.

Returns true iff there is such an entry.

Behaves as if written:

            boolean skipTo(int target) {
            do {
            if (!next())
            return false;
            } while (target > doc());
            return true;
            }
            

This class implements a {@link MergePolicy} that tries to merge segments into levels of exponentially increasing size, where each level has fewer segments than the value of the merge factor. Whenever extra segments (beyond the merge factor upper bound) are encountered, all segments within the level are merged. You can get or set the merge factor using {@link #GetMergeFactor()} and {@link #SetMergeFactor(int)} respectively.

This class is abstract and requires a subclass to define the {@link #size} method which specifies how a segment's size is determined. {@link LogDocMergePolicy} is one subclass that measures size by document count in the segment. {@link LogByteSizeMergePolicy} is another subclass that measures size as the total byte size of the file(s) for the segment.

Expert: a MergePolicy determines the sequence of primitive merge operations to be used for overall merge and optimize operations.

Whenever the segments in an index have been altered by {@link IndexWriter}, either the addition of a newly flushed segment, addition of many segments from addIndexes* calls, or a previous merge that may now need to cascade, {@link IndexWriter} invokes {@link #findMerges} to give the MergePolicy a chance to pick merges that are now required. This method returns a {@link MergeSpecification} instance describing the set of merges that should be done, or null if no merges are necessary. When IndexWriter.optimize is called, it calls {@link #findMergesForOptimize} and the MergePolicy should then return the necessary merges.

Note that the policy can return more than one merge at a time. In this case, if the writer is using {@link SerialMergeScheduler}, the merges will be run sequentially but if it is using {@link ConcurrentMergeScheduler} they will be run concurrently.

The default MergePolicy is {@link LogByteSizeMergePolicy}.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)

NOTE: This class typically requires access to package-private APIs (e.g. SegmentInfos) to do its job; if you implement your own MergePolicy, you'll need to put it in package Lucene.Net.Index in order to use these APIs.

Returns the number of segments that are merged at once and also controls the total number of segments allowed to accumulate in the index.

Determines the largest segment (measured by document count) that may be merged with other segments. Small values (e.g., less than 10,000) are best for interactive indexing, as this limits the length of pauses while indexing to a few seconds. Larger values are best for batched indexing and speedier searches.

The default value is {@link Integer#MAX_VALUE}.

The default merge policy ({@link LogByteSizeMergePolicy}) also allows you to set this limit by net size (in MB) of the segment, using {@link LogByteSizeMergePolicy#setMaxMergeMB}.

Upon calling this, the writer is notified that this commit point should be deleted.

Decision that a commit-point should be deleted is taken by the {@link IndexDeletionPolicy} in effect and therefore this should only be called by its {@link IndexDeletionPolicy#onInit onInit()} or {@link IndexDeletionPolicy#onCommit onCommit()} methods.

Clones of a stream access the same data, and are positioned at the same point as the stream they were cloned from.

Expert: Subclasses must ensure that clones may be positioned at different points in the input from each other and from the stream they were cloned from.

{@link Document#Add(Fieldable)} should be called by the Reader.

{@link Document#Add(Fieldable)} should be called by the Reader.

{@link Document#Add(Fieldable)} should not be called by the Reader.

{@link Document#Add(Fieldable)} should be called by the Reader.

{@link Document#Add(Fieldable)} should be called by the Reader.

This is an abstract class; subclasses must override {@link #IncrementToken()}

NOTE: Subclasses overriding {@link #next(Token)} must call {@link AttributeSource#ClearAttributes()} before setting attributes. Subclasses overriding {@link #IncrementToken()} must call {@link Token#Clear()} before setting Token attributes.

This is an abstract class. Concrete subclasses are:

• {@link Tokenizer}, a TokenStream whose input is a Reader; and
• {@link TokenFilter}, a TokenStream whose input is another TokenStream.

TokenStream now extends {@link AttributeSource}, which provides access to all of the token {@link Attribute}s for the TokenStream. Note that only one instance per {@link AttributeImpl} is created and reused for every token. This approach reduces object creation and allows local caching of references to the {@link AttributeImpl}s. See {@link #IncrementToken()} for further details.

The workflow of the new TokenStream API is as follows:

1. Instantiation of TokenStream/{@link TokenFilter}s which add/get attributes to/from the {@link AttributeSource}.
2. The consumer calls {@link TokenStream#Reset()}.
3. The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access
4. The consumer calls {@link #IncrementToken()} until it returns false and consumes the attributes after each call.
5. The consumer calls {@link #End()} so that any end-of-stream operations can be performed.
6. The consumer calls {@link #Close()} to release any resource when finished using the TokenStream

You can find some example code for the new API in the analysis package level Javadoc.

Sometimes it is desirable to capture a current state of a TokenStream , e. g. for buffering purposes (see {@link CachingTokenFilter}, {@link TeeSinkTokenFilter}). For this usecase {@link AttributeSource#CaptureState} and {@link AttributeSource#RestoreState} can be used.

Signature for Java 1.5: public AttributeImpl createAttributeInstance(Class%lt;? extends Attribute> attClass)

This setting only affects TokenStreams instantiated after this call. All TokenStreams already created use the other setting.

All core {@link Analyzer}s are compatible with this setting, if you have your own TokenStreams that are also compatible, you should enable this.

When enabled, tokenization may throw {@link UnsupportedOperationException} s, if the whole tokenizer chain is not compatible eg one of the TokenStreams does not implement the new TokenStream API.

The default is false, so there is the fallback to the old API available.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a {@link WhitespaceTokenizer} was used.

• A consumer must fully consume the previously returned {@link Token} before calling this method again.
• A producer must call {@link Token#Clear()} before setting the fields in it and returning it

This is an abstract class; subclasses must override {@link #IncrementToken()}.

NOTE: Be sure to call super.end() first when overriding this method.

Removes 's from the end of words.

Removes dots from acronyms.

Note that for simple usage, {@link NumericField} is recommended. {@link NumericField} disables norms and term freqs, as they are not usually needed during searching. If you need to change these settings, you should use this class.

See {@link NumericField} for capabilities of fields indexed numerically.

Here's an example usage, for an int field:

             Field field = new Field(name, new NumericTokenStream(precisionStep).setIntValue(value));
             field.setOmitNorms(true);
             field.setOmitTermFreqAndPositions(true);
             document.add(field);
            

For optimal performance, re-use the TokenStream and Field instance for more than one document:

             NumericTokenStream stream = new NumericTokenStream(precisionStep);
             Field field = new Field(name, stream);
             field.setOmitNorms(true);
             field.setOmitTermFreqAndPositions(true);
             Document document = new Document();
             document.add(field);
            
             for(all documents) {
               stream.setIntValue(value)
               writer.addDocument(document);
             }
            

This stream is not intended to be used in analyzers; it's more for iterating the different precisions during indexing a specific numeric value.

NOTE: as token streams are only consumed once the document is added to the index, if you index more than one numeric field, use a separate NumericTokenStream instance for each.

See {@link NumericRangeQuery} for more details on the precisionStep parameter as well as how numeric fields work under the hood.

NOTE: This API is experimental and might change in incompatible ways in the next release.

Searchable is the abstract network protocol for searching. Implementations provide search over a single index, over multiple indices, and over indices on remote servers.

Queries, filters and sort criteria are designed to be compact so that they may be efficiently passed to a remote index, with only the top-scoring hits being returned, rather than every matching hit. NOTE: this interface is kept public for convenience. Since it is not expected to be implemented directly, it may be changed unexpectedly between releases.

{@link HitCollector#Collect(int,float)} is called for every non-zero scoring document.
HitCollector-based access to remote indexes is discouraged.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query)}) is usually more efficient, as it skips non-high-scoring hits.

{@link Collector#Collect(int)} is called for every document.
Collector-based access to remote indexes is discouraged.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query)}) is usually more efficient, as it skips non-high-scoring hits.

Called by {@link Hits}.

Applications should usually call {@link Searcher#Search(Query)} or {@link Searcher#Search(Query,Filter)} instead.

This is intended to be used in developing Similarity implementations, and, for good performance, should not be displayed with every hit. Computing an explanation is as expensive as executing the query over the entire index.

Applications should call {@link Searcher#Explain(Query, int)}.

Applications should usually call {@link Searcher#Search(Query,Filter,int,Sort)} instead.

This filter matches the documents looking for terms that fall into the supplied range according to {@link String#compareTo(String)}. It is not intended for numerical ranges, use {@link NumericRangeFilter} instead.

If you construct a large number of range filters with different ranges but on the same field, {@link FieldCacheRangeFilter} may have significantly better performance.

MultiTermQueryWrapperFilter is not designed to be used by itself. Normally you subclass it to provide a Filter counterpart for a {@link MultiTermQuery} subclass.

For example, {@link TermRangeFilter} and {@link PrefixFilter} extend MultiTermQueryWrapperFilter. This class also provides the functionality behind {@link MultiTermQuery#CONSTANT_SCORE_FILTER_REWRITE}; this is why it is not abstract.

This method is not thread safe, be sure to only call it when no filter is running! If you re-use the same filter instance for another search, be sure to first reset the term counter with {@link #clearTotalNumberOfTerms}.

NOTE: This class pre-allocates a full array of length maxSize+1, in {@link #initialize}.

            // extends getSentinelObject() to return a non-null value.
            PriorityQueue pq = new MyQueue(numHits);
            // save the 'top' element, which is guaranteed to not be null.
            MyObject pqTop = (MyObject) pq.top();
            <...>
            // now in order to add a new element, which is 'better' than top (after 
            // you've verified it is better), it is as simple as:
            pqTop.change().
            pqTop = pq.updateTop();
            

            pq.top().change();
            pq.adjustTop();
            

            o = pq.pop();
            o.change();
            pq.push(o);
            

            pq.top().change();
            pq.updateTop();
            

            o = pq.pop();
            o.change();
            pq.push(o);
            

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

Attributes are used to add data in a dynamic, yet type-safe way to a source of usually streamed objects, e. g. a {@link Lucene.Net.Analysis.TokenStream}.

            public String toString() {
            return "start=" + startOffset + ",end=" + endOffset;
            }
            

            public int hashCode() {
            int code = startOffset;
            code = code * 31 + endOffset;
            return code;
            }
            

The default value is one.

Some common uses for this are:

• Set it to zero to put multiple terms in the same position. This is useful if, e.g., a word has multiple stems. Searches for phrases including either stem will match. In this case, all but the first stem's increment should be set to zero: the increment of the first instance should be one. Repeating a token with an increment of zero can also be used to boost the scores of matches on that token.
• Set it to values greater than one to inhibit exact phrase matches. If, for example, one does not want phrases to match across removed stop words, then one could build a stop word filter that removes stop words and also sets the increment to the number of stop words removed before each non-stop word. Then exact phrase queries will only match when the terms occur with no intervening stop words.

Get the bitset for any bits that have been set. This is completely distinct from {@link TypeAttribute#Type()}, although they do share similar purposes. The flags can be used to encode information about the token for use by other {@link Lucene.Net.Analysis.TokenFilter}s.

The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.

The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

A Token can optionally have metadata (a.k.a. Payload) in the form of a variable length byte array. Use {@link TermPositions#GetPayloadLength()} and {@link TermPositions#GetPayload(byte[], int)} to retrieve the payloads from the index.

NOTE: As of 2.9, Token implements all {@link Attribute} interfaces that are part of core Lucene and can be found in the {@code tokenattributes} subpackage. Even though it is not necessary to use Token anymore, with the new TokenStream API it can be used as convenience class that implements all {@link Attribute}s, which is especially useful to easily switch from the old to the new TokenStream API.

NOTE: As of 2.3, Token stores the term text internally as a malleable char[] termBuffer instead of String termText. The indexing code and core tokenizers have been changed to re-use a single Token instance, changing its buffer and other fields in-place as the Token is processed. This provides substantially better indexing performance as it saves the GC cost of new'ing a Token and String for every term. The APIs that accept String termText are still available but a warning about the associated performance cost has been added (below). The {@link #TermText()} method has been deprecated.

Tokenizers and TokenFilters should try to re-use a Token instance when possible for best performance, by implementing the {@link TokenStream#IncrementToken()} API. Failing that, to create a new Token you should first use one of the constructors that starts with null text. To load the token from a char[] use {@link #SetTermBuffer(char[], int, int)}. To load from a String use {@link #SetTermBuffer(String)} or {@link #SetTermBuffer(String, int, int)}. Alternatively you can get the Token's termBuffer by calling either {@link #TermBuffer()}, if you know that your text is shorter than the capacity of the termBuffer or {@link #ResizeTermBuffer(int)}, if there is any possibility that you may need to grow the buffer. Fill in the characters of your term into this buffer, with {@link String#getChars(int, int, char[], int)} if loading from a string, or with {@link System#arraycopy(Object, int, Object, int, int)}, and finally call {@link #SetTermLength(int)} to set the length of the term text. See LUCENE-969 for details.

Typical Token reuse patterns:

• Copying text from a string (type is reset to {@link #DEFAULT_TYPE} if not specified):
  

              return reusableToken.reinit(string, startOffset, endOffset[, type]);
              

• Copying some text from a string (type is reset to {@link #DEFAULT_TYPE} if not specified):
  

              return reusableToken.reinit(string, 0, string.length(), startOffset, endOffset[, type]);
              

• Copying text from char[] buffer (type is reset to {@link #DEFAULT_TYPE} if not specified):
  

              return reusableToken.reinit(buffer, 0, buffer.length, startOffset, endOffset[, type]);
              

• Copying some text from a char[] buffer (type is reset to {@link #DEFAULT_TYPE} if not specified):
  

              return reusableToken.reinit(buffer, start, end - start, startOffset, endOffset[, type]);
              

• Copying from one one Token to another (type is reset to {@link #DEFAULT_TYPE} if not specified):
  

              return reusableToken.reinit(source.termBuffer(), 0, source.termLength(), source.startOffset(), source.endOffset()[, source.type()]);
              

• clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
• Because TokenStreams can be chained, one cannot assume that the Token's current type is correct.
• The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
• When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.

The default value is one.

Some common uses for this are:

• Set it to zero to put multiple terms in the same position. This is useful if, e.g., a word has multiple stems. Searches for phrases including either stem will match. In this case, all but the first stem's increment should be set to zero: the increment of the first instance should be one. Repeating a token with an increment of zero can also be used to boost the scores of matches on that token.
• Set it to values greater than one to inhibit exact phrase matches. If, for example, one does not want phrases to match across removed stop words, then one could build a stop word filter that removes stop words and also sets the increment to the number of stop words removed before each non-stop word. Then exact phrase queries will only match when the terms occur with no intervening stop words.

Get the bitset for any bits that have been set. This is completely distinct from {@link #Type()}, although they do share similar purposes. The flags can be used to encode information about the token for use by other {@link Lucene.Net.Analysis.TokenFilter}s.

• {@link SimpleFSDirectory} is a straightforward implementation using java.io.RandomAccessFile. However, it has poor concurrent performance (multiple threads will bottleneck) as it synchronizes when multiple threads read from the same file.
• {@link NIOFSDirectory} uses java.nio's FileChannel's positional io when reading to avoid synchronization when reading from the same file. Unfortunately, due to a Windows-only Sun JRE bug this is a poor choice for Windows, but on all other platforms this is the preferred choice.
• {@link MMapDirectory} uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. Due to this bug in Sun's JRE, MMapDirectory's {@link IndexInput#close} is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs.

The locking implementation is by default {@link NativeFSLockFactory}, but can be changed by passing in a custom {@link LockFactory} instance. The deprecated getDirectory methods default to use {@link SimpleFSLockFactory} for backwards compatibility. The system properties org.apache.lucene.store.FSDirectoryLockFactoryClass and org.apache.lucene.FSDirectory.class are deprecated and only used by the deprecated getDirectory methods. The system property org.apache.lucene.lockDir is ignored completely, If you really want to store locks elsewhere, you can create your own {@link SimpleFSLockFactory} (or {@link NativeFSLockFactory}, etc.) passing in your preferred lock directory.

In 3.0 this class will become abstract.

Java's i/o APIs not used directly, but rather all i/o is through this API. This permits things such as:

• implementation of RAM-based indices;
• implementation indices stored in a database, via JDBC;
• implementation of an index as a single file;

NOTE: the source directory cannot change while this method is running. Otherwise the results are undefined and you could easily hit a FileNotFoundException.

NOTE: this method only copies files that look like index files (ie, have extensions matching the known extensions of index files).

Currently this returns {@link SimpleFSDirectory} as NIOFSDirectory is currently not supported.

Currently this returns {@link SimpleFSDirectory} as NIOFSDirectory is currently not supported.

NOTE: this method may suddenly change which implementation is returned from release to release, in the event that higher performance defaults become possible; if the precise implementation is important to your application, please instantiate it directly, instead. On 64 bit systems, it may also good to return {@link MMapDirectory}, but this is disabled because of officially missing unmap support in Java. For optimal performance you should consider using this implementation on 64 bit JVMs.

See above

Currently this returns {@link SimpleFSDirectory} as NIOFSDirectory is currently not supported.

NOTE: this method may suddenly change which implementation is returned from release to release, in the event that higher performance defaults become possible; if the precise implementation is important to your application, please instantiate it directly, instead. On 64 bit systems, it may also good to return {@link MMapDirectory}, but this is disabled because of officially missing unmap support in Java. For optimal performance you should consider using this implementation on 64 bit JVMs.

See above

This was introduced due to Sun JVM Bug 6478546, which throws an incorrect OutOfMemoryError when attempting to read too many bytes at once. It only happens on 32bit JVMs with a large maximum heap size.

Changes to this value will not impact any already-opened {@link IndexInput}s. You should call this before attempting to open an index on the directory.

NOTE: This value should be as large as possible to reduce any possible performance impact. If you still encounter an incorrect OutOfMemoryError, trying lowering the chunk size.

Expert: Collectors are primarily meant to be used to gather raw results from a search, and implement sorting or custom result filtering, collation, etc.

As of 2.9, this class replaces the deprecated HitCollector, and offers an API for efficient collection of hits across sequential {@link IndexReader}s. {@link IndexSearcher} advances the collector through each of the sub readers, in an arbitrary order. This results in a higher performance means of collection.

Lucene's core collectors are derived from Collector. Likely your application can use one of these classes, or subclass {@link TopDocsCollector}, instead of implementing Collector directly:

• {@link TopDocsCollector} is an abstract base class that assumes you will retrieve the top N docs, according to some criteria, after collection is done.
• {@link TopScoreDocCollector} is a concrete subclass {@link TopDocsCollector} and sorts according to score + docID. This is used internally by the {@link IndexSearcher} search methods that do not take an explicit {@link Sort}. It is likely the most frequently used collector.
• {@link TopFieldCollector} subclasses {@link TopDocsCollector} and sorts according to a specified {@link Sort} object (sort by field). This is used internally by the {@link IndexSearcher} search methods that take an explicit {@link Sort}.
• {@link TimeLimitingCollector}, which wraps any other Collector and aborts the search if it's taken too much time, will subclass Collector in 3.0 (presently it subclasses the deprecated HitCollector).
• {@link PositiveScoresOnlyCollector} wraps any other Collector and prevents collection of hits whose score is <= 0.0

Collector decouples the score from the collected doc: the score computation is skipped entirely if it's not needed. Collectors that do need the score should implement the {@link #setScorer} method, to hold onto the passed {@link Scorer} instance, and call {@link Scorer#Score()} within the collect method to compute the current hit's score. If your collector may request the score for a single hit multiple times, you should use {@link ScoreCachingWrappingScorer}.

NOTE: The doc that is passed to the collect method is relative to the current reader. If your collector needs to resolve this to the docID space of the Multi*Reader, you must re-base it by recording the docBase from the most recent setNextReader call. Here's a simple example showing how to collect docIDs into a BitSet:

            Searcher searcher = new IndexSearcher(indexReader);
            final BitSet bits = new BitSet(indexReader.maxDoc());
            searcher.search(query, new Collector() {
            private int docBase;
            
            // ignore scorer
            public void setScorer(Scorer scorer) {
            }
            
            // accept docs out of order (for a BitSet it doesn't matter)
            public boolean acceptsDocsOutOfOrder() {
            return true;
            }
            
            public void collect(int doc) {
            bits.set(doc + docBase);
            }
            
            public void setNextReader(IndexReader reader, int docBase) {
            this.docBase = docBase;
            }
            });
            

Not all collectors will need to rebase the docID. For example, a collector that simply counts the total number of hits would skip it.

NOTE: Prior to 2.9, Lucene silently filtered out hits with score <= 0. As of 2.9, the core Collectors no longer do that. It's very unusual to have such hits (a negative query boost, or function query returning negative custom scores, could cause it to happen). If you need that behavior, use {@link PositiveScoresOnlyCollector}.

NOTE: This API is experimental and might change in incompatible ways in the next release.

Note: This is called in an inner search loop. For good search performance, implementations of this method should not call {@link Searcher#Doc(int)} or {@link Lucene.Net.Index.IndexReader#Document(int)} on every hit. Doing so can slow searches by an order of magnitude or more.

Most Lucene Query implementations will visit * matching docIDs in order. However, some queries * (currently limited to certain cases of {@link * BooleanQuery}) can achieve faster searching if the * Collector allows them to deliver the * docIDs out of order. * *

Many collectors don't mind getting docIDs out of * order, so it's important to return true * here. *

• Finer (smaller) resolution is more accurate but less efficient.
• Setting resolution to less than 5 milliseconds will be silently modified to 5 milliseconds.
• Setting resolution smaller than current resolution might take effect only after current resolution. (Assume current resolution of 20 milliseconds is modified to 5 milliseconds, then it can take up to 20 milliseconds for the change to have effect.

Created: Feb 11, 2004 1:25:29 PM

In order to take advantage of this, you must override {@link Lucene.Net.Search.Similarity#ScorePayload(String, byte[],int,int)} which returns 1 by default.

Payload scores are averaged across term occurrences in the document.

In order to take advantage of this, you must override {@link Lucene.Net.Search.Similarity#ScorePayload(String, byte[],int,int)} which returns 1 by default.

Payload scores are aggregated using a pluggable {@link PayloadFunction}.

Should not be overriden without good cause!

Concrete subclasses of IndexReader are usually constructed with a call to one of the static open() methods, e.g. {@link #Open(String, boolean)}.

For efficiency, in this API documents are often referred to via document numbers, non-negative integers which each name a unique document in the index. These document numbers are ephemeral--they may change as documents are added to and deleted from an index. Clients should thus not rely on a given document having the same number between sessions.

An IndexReader can be opened on a directory for which an IndexWriter is opened already, but it cannot be used to delete documents from the index then.

NOTE: for backwards API compatibility, several methods are not listed as abstract, but have no useful implementations in this base class and instead always throw UnsupportedOperationException. Subclasses are strongly encouraged to override these methods, but in many cases may not need to.

NOTE: as of 2.4, it's possible to open a read-only IndexReader using one of the static open methods that accepts the boolean readOnly parameter. Such a reader has better concurrency as it's not necessary to synchronize on the isDeleted method. Currently the default for readOnly is false, meaning if not specified you will get a read/write IndexReader. But in 3.0 this default will change to true, meaning you must explicitly specify false if you want to make changes with the resulting IndexReader.

NOTE: {@link IndexReader} instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexReader instance; use your own (non-Lucene) objects instead.

This Constructor should not be used, it exists for backwards compatibility only to support legacy subclasses that did not "own" a specific directory, but needed to specify something to be returned by the directory() method. Future subclasses should delegate to the no arg constructor and implement the directory() method as appropriate.

Opening an IndexReader is an expensive operation. This method can be used to refresh an existing IndexReader to reduce these costs. This method tries to only load segments that have changed or were created after the IndexReader was (re)opened.

If the index has not changed since this instance was (re)opened, then this call is a NOOP and returns this instance. Otherwise, a new instance is returned. The old instance is not closed and remains usable.

If the reader is reopened, even though they share resources internally, it's safe to make changes (deletions, norms) with the new reader. All shared mutable state obeys "copy on write" semantics to ensure the changes are not seen by other readers.

You can determine whether a reader was actually reopened by comparing the old instance with the instance returned by this method:

            IndexReader reader = ... 
            ...
            IndexReader newReader = r.reopen();
            if (newReader != reader) {
            ...     // reader was reopened
            reader.close(); 
            }
            reader = newReader;
            ...
            

NOTE: If this reader is a near real-time reader (obtained from {@link IndexWriter#GetReader()}, reopen() will simply call writer.getReader() again for you, though this may change in the future.

On cloning a reader with pending changes (deletions, norms), the original reader transfers its write lock to the cloned reader. This means only the cloned reader may make further changes to the index, and commit the changes to the index on close, but the old reader still reflects all changes made up until it was cloned.

Like {@link #Reopen()}, it's safe to make changes to either the original or the cloned reader: all shared mutable state obeys "copy on write" semantics to ensure the changes are not seen by other readers.

If this reader is based on a Directory (ie, was created by calling {@link #Open}, or {@link #Reopen} on a reader based on a Directory), then this method returns the version recorded in the commit that the reader opened. This version is advanced every time {@link IndexWriter#Commit} is called.

If instead this reader is a near real-time reader (ie, obtained by a call to {@link IndexWriter#GetReader}, or by calling {@link #Reopen} on a near real-time reader), then this method returns the version of the last commit done by the writer. Note that even as further changes are made with the writer, the version will not changed until a commit is completed. Thus, you should not rely on this method to determine when a near real-time reader should be opened. Use {@link #IsCurrent} instead.

For IndexReader implementations that use TermInfosReader to read terms, this sets the indexDivisor to subsample the number of indexed terms loaded into memory. This has the same effect as {@link IndexWriter#setTermIndexInterval} except that setting must be done at indexing time while this setting can be set per reader. When set to N, then one in every N*termIndexInterval terms in the index is loaded into memory. By setting this to a value > 1 you can reduce memory usage, at the expense of higher latency when loading a TermInfo. The default value is 1.

NOTE: you must call this before the term index is loaded. If the index is already loaded, an IllegalStateException is thrown.

For IndexReader implementations that use TermInfosReader to read terms, this returns the current indexDivisor as specified when the reader was opened.

If this reader is based on a Directory (ie, was created by calling {@link #open}, or {@link #reopen} on a reader based on a Directory), then this method checks if any further commits (see {@link IndexWriter#commit} have occurred in that directory).

If instead this reader is a near real-time reader (ie, obtained by a call to {@link IndexWriter#getReader}, or by calling {@link #reopen} on a near real-time reader), then this method checks if either a new commmit has occurred, or any new uncommitted changes have taken place via the writer. Note that even if the writer has only performed merging, this method will still return false.

In any event, if this returns false, you should call {@link #reopen} to get a new reader that sees the changes.

NOTE: for performance reasons, this method does not check if the requested document is deleted, and therefore asking for a deleted document may yield unspecified results. Usually this is not required, however you can call {@link #IsDeleted(int)} with the requested document ID to verify the document is not deleted.

NOTE: for performance reasons, this method does not check if the requested document is deleted, and therefore asking for a deleted document may yield unspecified results. Usually this is not required, however you can call {@link #IsDeleted(int)} with the requested document ID to verify the document is not deleted.

Term    =>    <docNum, freq>^*

The enumeration is ordered by document number. Each document number is greater than all that precede it in the enumeration.

Term    =>    <docNum, freq, <pos₁, pos₂, ... pos_freq-1> >^*

This positional information facilitates phrase and proximity searching.

The enumeration is ordered by document number. Each document number is greater than all that precede it in the enumeration.

Caution: this should only be used by failure recovery code, when it is known that no other process nor thread is in fact currently accessing this index.

WARNING: this API is new and experimental and may suddenly change.

NOTE: You should not try using sub-readers returned by this method to make any changes (setNorm, deleteDocument, etc.). While this might succeed for one composite reader (like MultiReader), it will most likely lead to index corruption for other readers (like DirectoryReader obtained through {@link #open}. Use the parent reader directly.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)

Construct a MultiReader aggregating the named set of (sub)readers. Directory locking for delete, undeleteAll, and setNorm operations is left to the subreaders.

Note that all subreaders are closed if this Multireader is closed.

Construct a MultiReader aggregating the named set of (sub)readers. Directory locking for delete, undeleteAll, and setNorm operations is left to the subreaders.

A re-opened instance might share one or more subreaders with the old instance. Index modification operations result in undefined behavior when performed before the old instance is closed. (see {@link IndexReader#Reopen()}).

If subreaders are shared, then the reference count of those readers is increased to ensure that the subreaders remain open until the last referring reader is closed.

If subreaders are shared, then the reference count of those readers is increased to ensure that the subreaders remain open until the last referring reader is closed.

Expert: represents a single commit into an index as seen by the {@link IndexDeletionPolicy} or {@link IndexReader}.

Changes to the content of an index are made visible only after the writer who made that change commits by writing a new segments file (segments_N). This point in time, when the action of writing of a new segments file to the directory is completed, is an index commit.

Each index commit point has a unique segments file associated with it. The segments file associated with a later index commit point would have a larger N.

WARNING: This API is a new and experimental and may suddenly change.

Upon calling this, the writer is notified that this commit point should be deleted.

Decision that a commit-point should be deleted is taken by the {@link IndexDeletionPolicy} in effect and therefore this should only be called by its {@link IndexDeletionPolicy#onInit onInit()} or {@link IndexDeletionPolicy#onCommit onCommit()} methods.

• VInt fileCount
• {Directory} fileCount entries with the following structure:
• long dataOffset
• String fileName
• {File Data} fileCount entries with the raw data of the corresponding file

WARNING: This interface may change within minor versions, despite Lucene's backward compatibility requirements. This means new methods may be added from version to version. This change only affects the Fieldable API; other backwards compatibility promises remain intact. For example, Lucene can still read and write indices created within the same major version.

The boost is multiplied by {@link Lucene.Net.Documents.Document#GetBoost()} of the document containing this field. If a document has multiple fields with the same name, all such values are multiplied together. This product is then used to compute the norm factor for the field. By default, in the {@link Lucene.Net.Search.Similarity#ComputeNorm(String, FieldInvertState)} method, the boost value is multiplied by the {@link Lucene.Net.Search.Similarity#LengthNorm(String, int)} and then rounded by {@link Lucene.Net.Search.Similarity#EncodeNorm(float)} before it is stored in the index. One should attempt to ensure that this product does not overflow the range of that encoding.

The default value is 1.0.

Note: this value is not stored directly with the document in the index. Documents returned from {@link Lucene.Net.Index.IndexReader#Document(int)} and {@link Lucene.Net.Search.Hits#Doc(int)} may thus not have the same value present as when this field was indexed.

For indexing, if isStored()==true, the stringValue() will be used as the stored field value unless isBinary()==true, in which case binaryValue() will be used. If isIndexed()==true and isTokenized()==false, this String value will be indexed as a single token. If isIndexed()==true and isTokenized()==true, then tokenStreamValue() will be used to generate indexed tokens if not null, else readerValue() will be used to generate indexed tokens if not null, else stringValue() will be used to generate tokens.

About reuse: if you pass in the result byte[] and it is used, likely the underlying implementation will hold onto this byte[] and return it in future calls to {@link #BinaryValue()} or {@link #GetBinaryValue()}. So if you subsequently re-use the same byte[] elsewhere it will alter this Fieldable's value.

To use this filter with other analyzers, you'll want to write an Analyzer class that sets up the TokenStream chain as you want it. To use this with LowerCaseTokenizer, for example, you'd write an analyzer like this:

            class MyAnalyzer extends Analyzer {
            public final TokenStream tokenStream(String fieldName, Reader reader) {
            return new PorterStemFilter(new LowerCaseTokenizer(reader));
            }
            }
            

NOTE: This class is not particularly efficient. For example, a new class instance is created for every call to {@link #addOffCorrectMap}, which is then appended to a private list.

CachingTokenFilter implements the optional method {@link TokenStream#Reset()}, which repositions the stream to the first Token.

Created: Apr 21, 2004 3:49:28 PM This class will be used as part of a key to a FieldCache value. You must implement hashCode and equals to avoid an explosion in RAM usage if you use instances that are not the same instance. If you are searching using the Remote contrib, the same instance of this class on the client will be a new instance on every call to the server, so hashCode/equals is very important in that situation.

This query may be combined with other terms or queries with a {@link BooleanQuery}.

The slop is in fact an edit-distance, where the units correspond to moves of terms in the query phrase out of position. For example, to switch the order of two words requires two moves (the first move places the words atop one another), so to permit re-orderings of phrases, the slop must be at least two.

More exact matches are scored higher than sloppier matches, thus search results are sorted by exactness.

The slop is zero by default, requiring exact matches.

Applications usually need only call the inherited {@link #Search(Query)} or {@link #Search(Query,Filter)} methods. For performance reasons it is recommended to open only one IndexSearcher and use it for all of your searches.

Note that you can only access Hits from an IndexSearcher as long as it is not yet closed, otherwise an IOException will be thrown.

NOTE: {@link IndexSearcher} instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexSearcher instance; use your own (non-Lucene) objects instead.

Note that you can only access hits from a Searcher as long as it is not yet closed, otherwise an IOException will be thrown.

NOTE: this does not compute scores by default; use {@link IndexSearcher#setDefaultFieldSortScoring} to enable scoring.

{@link HitCollector#Collect(int,float)} is called for every matching document.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query)}) is usually more efficient, as it skips non-high-scoring hits.

Note: The score passed to this method is a raw score. In other words, the score will not necessarily be a float whose value is between 0 and 1.

{@link Collector#Collect(int)} is called for every matching document.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query, int)} ) is usually more efficient, as it skips non-high-scoring hits.

Note: The score passed to this method is a raw score. In other words, the score will not necessarily be a float whose value is between 0 and 1.

{@link HitCollector#Collect(int,float)} is called for every matching document.
HitCollector-based access to remote indexes is discouraged.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query, Filter, int)}) is usually more efficient, as it skips non-high-scoring hits.

{@link Collector#Collect(int)} is called for every matching document.
Collector-based access to remote indexes is discouraged.

Applications should only use this if they need all of the matching documents. The high-level search API ({@link Searcher#Search(Query, Filter, int)}) is usually more efficient, as it skips non-high-scoring hits.

This is intended to be used in developing Similarity implementations, and, for good performance, should not be displayed with every hit. Computing an explanation is as expensive as executing the query over the entire index.

This defaults to the current value of {@link Similarity#GetDefault()}.

NOTE: this does not compute scores by default. If you need scores, create a {@link TopFieldCollector} instance by calling {@link TopFieldCollector#create} and then pass that to {@link #Search(Weight, Filter, Collector)}.

The native lucene index order is used to assign an ordinal value for each field value.

Field values (terms) are lexicographically ordered by unicode value, and numbered starting at 1.

Example:
If there were only three field values: "apple","banana","pear"
then ord("apple")=1, ord("banana")=2, ord("pear")=3

WARNING: ord() depends on the position in an index and can thus change when other documents are inserted or deleted, or if a MultiSearcher is used.

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

NOTE: with the switch in 2.9 to segment-based searching, if {@link #getValues} is invoked with a composite (multi-segment) reader, this can easily cause double RAM usage for the values in the FieldCache. It's best to switch your application to pass only atomic (single segment) readers to this API. Alternatively, for a short-term fix, you could wrap your ValueSource using {@link MultiValueSource}, which costs more CPU per lookup but will not consume double the FieldCache RAM.

It does not have an upper bound on the number of clauses covered in the range.

This query matches the documents looking for terms that fall into the supplied range according to {@link String#compareTo(String)}. It is not intended for numerical ranges, use {@link NumericRangeQuery} instead.

This query is hardwired to {@link MultiTermQuery#CONSTANT_SCORE_AUTO_REWRITE_DEFAULT}. If you want to change this, use {@link TermRangeQuery} instead.

This query matches the documents looking for terms that fall into the supplied range according to {@link String#compareTo(String)}. It is not intended for numerical ranges, use {@link NumericRangeQuery} instead.

This query uses the {@link MultiTermQuery#CONSTANT_SCORE_AUTO_REWRITE_DEFAULT} rewrite method.

If an endpoint is null, it is said to be "open". Either or both endpoints may be open. Open endpoints may not be exclusive (you can't select all but the first or last term without explicitly specifying the term to exclude.)

If an endpoint is null, it is said to be "open". Either or both endpoints may be open. Open endpoints may not be exclusive (you can't select all but the first or last term without explicitly specifying the term to exclude.)

If collator is not null, it will be used to decide whether index terms are within the given range, rather than using the Unicode code point order in which index terms are stored.

WARNING: Using this constructor and supplying a non-null value in the collator parameter will cause every single index Term in the Field referenced by lowerTerm and/or upperTerm to be examined. Depending on the number of index Terms in this Field, the operation could be very slow.

The document and frequency are the same as for a TermDocs. The positions portion lists the ordinal positions of each occurrence of a term in a document.

This is invalid until {@link #Next()} is called for the first time.

Payloads can only be loaded once per call to {@link #NextPosition()}.

• C1 Controls and Latin-1 Supplement: http://www.unicode.org/charts/PDF/U0080.pdf
• Latin Extended-A: http://www.unicode.org/charts/PDF/U0100.pdf
• Latin Extended-B: http://www.unicode.org/charts/PDF/U0180.pdf
• Latin Extended Additional: http://www.unicode.org/charts/PDF/U1E00.pdf
• Latin Extended-C: http://www.unicode.org/charts/PDF/U2C60.pdf
• Latin Extended-D: http://www.unicode.org/charts/PDF/UA720.pdf
• IPA Extensions: http://www.unicode.org/charts/PDF/U0250.pdf
• Phonetic Extensions: http://www.unicode.org/charts/PDF/U1D00.pdf
• Phonetic Extensions Supplement: http://www.unicode.org/charts/PDF/U1D80.pdf
• General Punctuation: http://www.unicode.org/charts/PDF/U2000.pdf
• Superscripts and Subscripts: http://www.unicode.org/charts/PDF/U2070.pdf
• Enclosed Alphanumerics: http://www.unicode.org/charts/PDF/U2460.pdf
• Dingbats: http://www.unicode.org/charts/PDF/U2700.pdf
• Supplemental Punctuation: http://www.unicode.org/charts/PDF/U2E00.pdf
• Alphabetic Presentation Forms: http://www.unicode.org/charts/PDF/UFB00.pdf
• Halfwidth and Fullwidth Forms: http://www.unicode.org/charts/PDF/UFF00.pdf

The fields used to determine sort order must be carefully chosen. Documents must contain a single term in such a field, and the value of the term should indicate the document's relative position in a given sort order. The field must be indexed, but should not be tokenized, and does not need to be stored (unless you happen to want it back with the rest of your document data). In other words:

document.add (new Field ("byNumber", Integer.toString(x), Field.Store.NO, Field.Index.NOT_ANALYZED));

Valid Types of Values

There are four possible kinds of term values which may be put into sorting fields: Integers, Longs, Floats, or Strings. Unless {@link SortField SortField} objects are specified, the type of value in the field is determined by parsing the first term in the field.

Integer term values should contain only digits and an optional preceding negative sign. Values must be base 10 and in the range Integer.MIN_VALUE and Integer.MAX_VALUE inclusive. Documents which should appear first in the sort should have low value integers, later documents high values (i.e. the documents should be numbered 1..n where 1 is the first and n the last).

Long term values should contain only digits and an optional preceding negative sign. Values must be base 10 and in the range Long.MIN_VALUE and Long.MAX_VALUE inclusive. Documents which should appear first in the sort should have low value integers, later documents high values.

Float term values should conform to values accepted by {@link Float Float.valueOf(String)} (except that NaN and Infinity are not supported). Documents which should appear first in the sort should have low values, later documents high values.

String term values can contain any valid String, but should not be tokenized. The values are sorted according to their {@link Comparable natural order}. Note that using this type of term value has higher memory requirements than the other two types.

Object Reuse

One of these objects can be used multiple times and the sort order changed between usages.

This class is thread safe.

Memory Usage

Sorting uses of caches of term values maintained by the internal HitQueue(s). The cache is static and contains an integer or float array of length IndexReader.maxDoc() for each field name for which a sort is performed. In other words, the size of the cache in bytes is:

4 * IndexReader.maxDoc() * (# of different fields actually used to sort)

For String fields, the cache is larger: in addition to the above array, the value of every term in the field is kept in memory. If there are many unique terms in the field, this could be quite large.

Note that the size of the cache is not affected by how many fields are in the index and might be used to sort - only by the ones actually used to sort a result set.

Created: Feb 12, 2004 10:53:57 AM

Is thread safe and completely reusable.

WARNING: The status of the Search.Function package is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

Note that a null field or null text value results in undefined behavior for most Lucene APIs that accept a Term parameter.

See {@link FieldSelectorResult#LOAD_AND_BREAK}

• a count() method, which efficiently computes the number of one bits;
• optimized read from and write to disk;
• inlinable get() method;
• store and load, as bit set or d-gaps, depending on sparseness;

See the {@link #create(Lucene.Net.Search.Sort, int, boolean, boolean, boolean, boolean)} method for instantiating a TopFieldCollector.

NOTE: This API is experimental and might change in incompatible ways in the next release.

NOTE: The instances returned by this method pre-allocate a full array of length numHits.

This class should only be used if the other SortField types (SCORE, DOC, STRING, INT, FLOAT) do not provide an adequate sorting. It maintains an internal cache of values which could be quite large. The cache is an array of Comparable, one for each document in the index. There is a distinct Comparable for each unique term in the field - if some documents have the same term in the field, the cache array will have entries which reference the same Comparable. This class will be used as part of a key to a FieldCache value. You must implement hashCode and equals to avoid an explosion in RAM usage if you use instances that are not the same instance. If you are searching using the Remote contrib, the same instance of this class on the client will be a new instance on every call to the server, so hashCode/equals is very important in that situation.

Created: Apr 21, 2004 5:08:38 PM

For example, if the Terms contained integer values, this method would return new Integer(termtext). Note that this might not always be the most efficient implementation - for this particular example, a better implementation might be to make a ScoreDocLookupComparator that uses an internal lookup table of int.

Caution: Iterate only over the hits needed. Iterating over all hits is generally not desirable and may be the source of performance issues. If you need to iterate over many or all hits, consider using the search method that takes a {@link HitCollector}.

Note: Deleting matching documents concurrently with traversing the hits, might, when deleting hits that were not yet retrieved, decrease {@link #Length()}. In such case, {@link java.util.ConcurrentModificationException ConcurrentModificationException} is thrown when accessing hit n > current_{@link #Length()} (but n < {@link #Length()} _at_start).

Documents are cached, so that repeated requests for the same element may return the same Document object.

Caution: Iterate only over the hits needed. Iterating over all hits is generally not desirable and may be the source of performance issues. If you need to iterate over many or all hits, consider using a search method that takes a {@link HitCollector}.

Created: Dec 8, 2003 12:56:03 PM

Created: May 19, 2004 4:40:36 PM

NOTE: These CacheEntry objects maintain a strong refrence to the Cached Values. Maintaining refrences to a CacheEntry the IndexReader associated with it has garbage collected will prevent the Value itself from being garbage collected when the Cache drops the WeakRefrence.

EXPERIMENTAL API: This API is considered extremely advanced and experimental. It may be removed or altered w/o warning in future releases of Lucene.

EXPERT: Instructs the FieldCache to forcibly expunge all entries from the underlying caches. This is intended only to be used for test methods as a way to ensure a known base state of the Cache (with out needing to rely on GC to free WeakReferences). It should not be relied on for "Cache maintenance" in general application code.

EXPERIMENTAL API: This API is considered extremely advanced and experimental. It may be removed or altered w/o warning in future releases of Lucene.

EXPERIMENTAL API: This API is considered extremely advanced and experimental. It may be removed or altered w/o warning in future releases of Lucene.

This API is designed to achieve high performance sorting, by exposing a tight interaction with {@link FieldValueHitQueue} as it visits hits. Whenever a hit is competitive, it's enrolled into a virtual slot, which is an int ranging from 0 to numHits-1. The {@link FieldComparator} is made aware of segment transitions during searching in case any internal state it's tracking needs to be recomputed during these transitions.

A comparator must define these functions:

• {@link #compare} Compare a hit at 'slot a' with hit 'slot b'.
• {@link #setBottom} This method is called by {@link FieldValueHitQueue} to notify the FieldComparator of the current weakest ("bottom") slot. Note that this slot may not hold the weakest value according to your comparator, in cases where your comparator is not the primary one (ie, is only used to break ties from the comparators before it).
• {@link #compareBottom} Compare a new hit (docID) against the "weakest" (bottom) entry in the queue.
• {@link #copy} Installs a new hit into the priority queue. The {@link FieldValueHitQueue} calls this method when a new hit is competitive.
• {@link #setNextReader} Invoked when the search is switching to the next segment. You may need to update internal state of the comparator, for example retrieving new values from the {@link FieldCache}.
• {@link #value} Return the sort value stored in the specified slot. This is only called at the end of the search, in order to populate {@link FieldDoc#fields} when returning the top results.

For a search that hits many results, this method will be the hotspot (invoked by far the most frequently).

FieldCacheRangeFilter builds a single cache for the field the first time it is used. Each subsequent FieldCacheRangeFilter on the same field then reuses this cache, even if the range itself changes.

This means that FieldCacheRangeFilter is much faster (sometimes more than 100x as fast) as building a {@link TermRangeFilter} (or {@link ConstantScoreRangeQuery} on a {@link TermRangeFilter}) for each query, if using a {@link #newStringRange}. However, if the range never changes it is slower (around 2x as slow) than building a CachingWrapperFilter on top of a single TermRangeFilter. For numeric data types, this filter may be significantly faster than {@link NumericRangeFilter}. Furthermore, it does not need the numeric values encoded by {@link NumericField}. But it has the problem that it only works with exact one value/document (see below).

As with all {@link FieldCache} based functionality, FieldCacheRangeFilter is only valid for fields which exact one term for each document (except for {@link #newStringRange} where 0 terms are also allowed). Due to a restriction of {@link FieldCache}, for numeric ranges all terms that do not have a numeric value, 0 is assumed.

Thus it works on dates, prices and other single value fields but will not work on regular text fields. It is preferable to use a NOT_ANALYZED field to ensure that there is only a single term.

This class does not have an constructor, use one of the static factory methods available, that create a correct instance for different data types supported by {@link FieldCache}.

By default, an Explanation represents a "match" if the value is positive.

After each to call to next() or skipTo() currentSumScore is the total score of the current matching doc, nrMatchers is the number of matching scorers, and all scorers are after the matching doc, or are exhausted.

• a plus (+) or a minus (-) sign, indicating that the clause is required or prohibited respectively; or
• a term followed by a colon, indicating the field to be searched. This enables one to construct queries which search multiple fields.

• a term, indicating all the documents that contain this term; or
• a nested query, enclosed in parentheses. Note that this may be used with a +/- prefix to require any of a set of terms.

            Query  ::= ( Clause )*
            Clause ::= ["+", "-"] [<TERM> ":"] ( <TERM> | "(" Query ")" )
            

Examples of appropriately formatted queries can be found in the query syntax documentation.

In {@link TermRangeQuery}s, QueryParser tries to detect date values, e.g. date:[6/1/2005 TO 6/4/2005] produces a range query that searches for "date" fields between 2005-06-01 and 2005-06-04. Note that the format of the accepted input depends on {@link #SetLocale(Locale) the locale}. By default a date is converted into a search term using the deprecated {@link DateField} for compatibility reasons. To use the new {@link DateTools} to convert dates, a {@link Lucene.Net.Documents.DateTools.Resolution} has to be set.

The date resolution that shall be used for RangeQueries can be set using {@link #SetDateResolution(DateTools.Resolution)} or {@link #SetDateResolution(String, DateTools.Resolution)}. The former sets the default date resolution for all fields, whereas the latter can be used to set field specific date resolutions. Field specific date resolutions take, if set, precedence over the default date resolution.

If you use neither {@link DateField} nor {@link DateTools} in your index, you can create your own query parser that inherits QueryParser and overwrites {@link #GetRangeQuery(String, String, String, boolean)} to use a different method for date conversion.

Note that QueryParser is not thread-safe.

NOTE: there is a new QueryParser in contrib, which matches the same syntax as this class, but is more modular, enabling substantial customization to how a query is created.

NOTE: there is a new QueryParser in contrib, which matches the same syntax as this class, but is more modular, enabling substantial customization to how a query is created. NOTE: You must specify the required {@link Version} compatibility when creating QueryParser:

• As of 2.9, {@link #SetEnablePositionIncrements} is true by default.

When set, * or ? are allowed as the first character of a PrefixQuery and WildcardQuery. Note that this can produce very slow queries on big indexes.

Default: false.

When set, result phrase and multi-phrase queries will be aware of position increments. Useful when e.g. a StopFilter increases the position increment of the token that follows an omitted token.

Default: false.

WARNING: Setting the rangeCollator to a non-null collator using this method will cause every single index Term in the Field referenced by lowerTerm and/or upperTerm to be examined. Depending on the number of index Terms in this Field, the operation could be very slow.

Depending on settings, prefix term may be lower-cased automatically. It will not go through the default Analyzer, however, since normal Analyzers are unlikely to work properly with wildcard templates.

Can be overridden by extending classes, to provide custom handling for wildcard queries, which may be necessary due to missing analyzer calls.

Depending on settings, a prefix term may be lower-cased automatically. It will not go through the default Analyzer, however, since normal Analyzers are unlikely to work properly with wildcard templates.

Can be overridden by extending classes, to provide custom handling for wild card queries, which may be necessary due to missing analyzer calls.

It will, when parse(String query) is called, construct a query like this (assuming the query consists of two terms and you specify the two fields title and body):

(title:term1 body:term1) (title:term2 body:term2)

When setDefaultOperator(AND_OPERATOR) is set, the result will be:

+(title:term1 body:term1) +(title:term2 body:term2)

When you pass a boost (title=>5 body=>10) you can get

+(title:term1^5.0 body:term1^10.0) +(title:term2^5.0 body:term2^10.0)

In other words, all the query's terms must appear, but it doesn't matter in what fields they appear.

It will, when parse(String query) is called, construct a query like this (assuming the query consists of two terms and you specify the two fields title and body):

(title:term1 body:term1) (title:term2 body:term2)

When setDefaultOperator(AND_OPERATOR) is set, the result will be:

+(title:term1 body:term1) +(title:term2 body:term2)

When you pass a boost (title=>5 body=>10) you can get

+(title:term1^5.0 body:term1^10.0) +(title:term2^5.0 body:term2^10.0)

In other words, all the query's terms must appear, but it doesn't matter in what fields they appear.

It will, when parse(String query) is called, construct a query like this (assuming the query consists of two terms and you specify the two fields title and body):

(title:term1 body:term1) (title:term2 body:term2)

When setDefaultOperator(AND_OPERATOR) is set, the result will be:

+(title:term1 body:term1) +(title:term2 body:term2)

In other words, all the query's terms must appear, but it doesn't matter in what fields they appear.

It will, when parse(String query) is called, construct a query like this (assuming the query consists of two terms and you specify the two fields title and body):

(title:term1 body:term1) (title:term2 body:term2)

When setDefaultOperator(AND_OPERATOR) is set, the result will be:

+(title:term1 body:term1) +(title:term2 body:term2)

In other words, all the query's terms must appear, but it doesn't matter in what fields they appear.

If x fields are specified, this effectively constructs:

            <code>
            (field1:query1) (field2:query2) (field3:query3)...(fieldx:queryx)
            </code>
            

If x fields are specified, this effectively constructs:

            <code>
            (field1:query1) (field2:query2) (field3:query3)...(fieldx:queryx)
            </code>
            

            Usage:
            
            String[] fields = {"filename", "contents", "description"};
            BooleanClause.Occur[] flags = {BooleanClause.Occur.SHOULD,
            BooleanClause.Occur.MUST,
            BooleanClause.Occur.MUST_NOT};
            MultiFieldQueryParser.parse("query", fields, flags, analyzer);
            
            

The code above would construct a query:

            
            (filename:query) +(contents:query) -(description:query)
            
            

            Usage:
            <code>
            String[] fields = {"filename", "contents", "description"};
            BooleanClause.Occur[] flags = {BooleanClause.Occur.SHOULD,
            BooleanClause.Occur.MUST,
            BooleanClause.Occur.MUST_NOT};
            MultiFieldQueryParser.parse("query", fields, flags, analyzer);
            </code>
            

The code above would construct a query:

            <code>
            (filename:query) +(contents:query) -(description:query)
            </code>
            

            Usage:
            
            String[] query = {"query1", "query2", "query3"};
            String[] fields = {"filename", "contents", "description"};
            BooleanClause.Occur[] flags = {BooleanClause.Occur.SHOULD,
            BooleanClause.Occur.MUST,
            BooleanClause.Occur.MUST_NOT};
            MultiFieldQueryParser.parse(query, fields, flags, analyzer);
            
            

The code above would construct a query:

            
            (filename:query1) +(contents:query2) -(description:query3)
            
            

            Usage:
            <code>
            String[] query = {"query1", "query2", "query3"};
            String[] fields = {"filename", "contents", "description"};
            BooleanClause.Occur[] flags = {BooleanClause.Occur.SHOULD,
            BooleanClause.Occur.MUST,
            BooleanClause.Occur.MUST_NOT};
            MultiFieldQueryParser.parse(query, fields, flags, analyzer);
            </code>
            

The code above would construct a query:

            <code>
            (filename:query1) +(contents:query2) -(description:query3)
            </code>
            

Expert: policy for deletion of stale {@link IndexCommit index commits}.

Implement this interface, and pass it to one of the {@link IndexWriter} or {@link IndexReader} constructors, to customize when older {@link IndexCommit point-in-time commits} are deleted from the index directory. The default deletion policy is {@link KeepOnlyLastCommitDeletionPolicy}, which always removes old commits as soon as a new commit is done (this matches the behavior before 2.2).

One expected use case for this (and the reason why it was first created) is to work around problems with an index directory accessed via filesystems like NFS because NFS does not provide the "delete on last close" semantics that Lucene's "point in time" search normally relies on. By implementing a custom deletion policy, such as "a commit is only removed once it has been stale for more than X minutes", you can give your readers time to refresh to the new commit before {@link IndexWriter} removes the old commits. Note that doing so will increase the storage requirements of the index. See LUCENE-710 for details.

This is called once when a writer is first instantiated to give the policy a chance to remove old commit points.

The writer locates all index commits present in the index directory and calls this method. The policy may choose to delete some of the commit points, doing so by calling method {@link IndexCommit#delete delete()} of {@link IndexCommit}.

Note: the last CommitPoint is the most recent one, i.e. the "front index state". Be careful not to delete it, unless you know for sure what you are doing, and unless you can afford to lose the index content while doing that.

This is called each time the writer completed a commit. This gives the policy a chance to remove old commit points with each commit.

The policy may now choose to delete old commit points by calling method {@link IndexCommit#delete delete()} of {@link IndexCommit}.

If writer has autoCommit = true then this method will in general be called many times during one instance of {@link IndexWriter}. If autoCommit = false then this method is only called once when {@link IndexWriter#close} is called, or not at all if the {@link IndexWriter#abort} is called.

Note: the last CommitPoint is the most recent one, i.e. the "front index state". Be careful not to delete it, unless you know for sure what you are doing, and unless you can afford to lose the index content while doing that.

NOTE: this class is meant only to be used internally by Lucene; it's only public so it can be shared across packages. This means the API is freely subject to change, and, the class could be removed entirely, in any Lucene release. Use directly at your own risk! */

For instance, 'À' will be replaced by 'a'.

NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult {@link #setMaxChunkSize} if you have problems with mmap failing because of fragmented address space. If you get an OutOfMemoryException, it is recommened to reduce the chunk size, until it works.

Due to this bug in Sun's JRE, MMapDirectory's {@link IndexInput#close} is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed.

This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of.

This class supplies the workaround mentioned in the bug report (disabled by default, see {@link #setUseUnmap}), which may fail on non-Sun JVMs. It forcefully unmaps the buffer on close by using an undocumented internal cleanup functionality. {@link #UNMAP_SUPPORTED} is true, if the workaround can be enabled (with no guarantees).

NOTE: Enabling this is completely unsupported by Java and may lead to JVM crashs if IndexInput is closed while another thread is still accessing it (SIGSEGV).

The TimeLimitedCollector is used to timeout search requests that take longer than the maximum allowed search time limit. After this time is exceeded, the search thread is stopped by throwing a TimeExceeded Exception.

If, for example, an application wished to collect all of the hits for a query in a BitSet, then it might:

            Searcher searcher = new IndexSearcher(indexReader);
            final BitSet bits = new BitSet(indexReader.maxDoc());
            searcher.search(query, new HitCollector() {
            public void collect(int doc, float score) {
            bits.set(doc);
            }
            });
            

Note: This is called in an inner search loop. For good search performance, implementations of this method should not call {@link Searcher#Doc(int)} or {@link Lucene.Net.Index.IndexReader#Document(int)} on every document number encountered. Doing so can slow searches by an order of magnitude or more.

Note: The score passed to this method is a raw score. In other words, the score will not necessarily be a float whose value is between 0 and 1.

• Finer (smaller) resolution is more accurate but less efficient.
• Setting resolution to less than 5 milliseconds will be silently modified to 5 milliseconds.
• Setting resolution smaller than current resolution might take effect only after current resolution. (Assume current resolution of 20 milliseconds is modified to 5 milliseconds, then it can take up to 20 milliseconds for the change to have effect.

Wrapper to allow {@link SpanQuery} objects participate in composite single-field SpanQueries by 'lying' about their search field. That is, the masked SpanQuery will function as normal, but {@link SpanQuery#GetField()} simply hands back the value supplied in this class's constructor.

This can be used to support Queries like {@link SpanNearQuery} or {@link SpanOrQuery} across different fields, which is not ordinarily permitted.

This can be useful for denormalized relational data: for example, when indexing a document with conceptually many 'children':

            teacherid: 1
            studentfirstname: james
            studentsurname: jones
            
            teacherid: 2
            studenfirstname: james
            studentsurname: smith
            studentfirstname: sally
            studentsurname: jones
            

a SpanNearQuery with a slop of 0 can be applied across two {@link SpanTermQuery} objects as follows:

            SpanQuery q1  = new SpanTermQuery(new Term("studentfirstname", "james"));
            SpanQuery q2  = new SpanTermQuery(new Term("studentsurname", "jones"));
            SpanQuery q2m new FieldMaskingSpanQuery(q2, "studentfirstname");
            Query q = new SpanNearQuery(new SpanQuery[]{q1, q2m}, -1, false);
            

Note: as {@link #GetField()} returns the masked field, scoring will be done using the norms of the field name supplied. This may lead to unexpected scoring behaviour.

            PriorityQueue pq = new HitQueue(10, true); // pre-populate.
            ScoreDoc top = pq.top();
            
            // Add/Update one element.
            top.score = 1.0f;
            top.doc = 0;
            top = (ScoreDoc) pq.updateTop();
            int totalHits = 1;
            
            // Now pop only the elements that were *truly* inserted.
            // First, pop all the sentinel elements (there are pq.size() - totalHits).
            for (int i = pq.size() - totalHits; i > 0; i--) pq.pop();
            
            // Now pop the truly added elements.
            ScoreDoc[] results = new ScoreDoc[totalHits];
            for (int i = totalHits - 1; i >= 0; i--) {
            results[i] = (ScoreDoc) pq.pop();
            }
            

NOTE: This class pre-allocate a full array of length size.

WARNING: This API is a new and experimental and may suddenly change.

To store payloads in the index a {@link TokenStream} has to be used that produces payload data.

Use {@link TermPositions#GetPayloadLength()} and {@link TermPositions#GetPayload(byte[], int)} to retrieve the payloads from the index.

WARNING: This API is new and experimental, and may suddenly change.

That is, if l1 is less than l2 for any two longs l1 and l2, then NumberTools.longToString(l1) is lexicographically less than NumberTools.longToString(l2). (Similarly for "greater than" and "equals".)

This class handles all long values (unlike {@link Lucene.Net.Documents.DateField}).

Get the bitset for any bits that have been set. This is completely distinct from {@link TypeAttribute#Type()}, although they do share similar purposes. The flags can be used to encode information about the token for use by other {@link Lucene.Net.Analysis.TokenFilter}s.

It is also useful for doing things like entity extraction or proper noun analysis as part of the analysis workflow and saving off those tokens for use in another field.

            TeeSinkTokenFilter source1 = new TeeSinkTokenFilter(new WhitespaceTokenizer(reader1));
            TeeSinkTokenFilter.SinkTokenStream sink1 = source1.newSinkTokenStream();
            TeeSinkTokenFilter.SinkTokenStream sink2 = source1.newSinkTokenStream();
            TeeSinkTokenFilter source2 = new TeeSinkTokenFilter(new WhitespaceTokenizer(reader2));
            source2.addSinkTokenStream(sink1);
            source2.addSinkTokenStream(sink2);
            TokenStream final1 = new LowerCaseFilter(source1);
            TokenStream final2 = source2;
            TokenStream final3 = new EntityDetect(sink1);
            TokenStream final4 = new URLDetect(sink2);
            d.add(new Field("f1", final1));
            d.add(new Field("f2", final2));
            d.add(new Field("f3", final3));
            d.add(new Field("f4", final4));
            

            ...
            TokenStream final1 = new LowerCaseFilter(source1.newSinkTokenStream());
            TokenStream final2 = source2.newSinkTokenStream();
            sink1.consumeAllTokens();
            sink2.consumeAllTokens();
            ...
            

Note, the EntityDetect and URLDetect TokenStreams are for the example and do not currently exist in Lucene.

This should be a good tokenizer for most European-language documents:

• Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
• Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
• Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

You must specify the required {@link Version} compatibility when creating StandardAnalyzer:

• As of 2.4, Tokens incorrectly identified as acronyms are corrected (see LUCENE-1608

See http://issues.apache.org/jira/browse/LUCENE-1068

This should be used only with indices that can fit into memory.

Note that the resulting RAMDirectory instance is fully independent from the original Directory (it is a complete copy). Any subsequent changes to the original Directory will not be visible in the RAMDirectory instance.

Implements {@link LockFactory} using native OS file locks. Note that because this LockFactory relies on java.nio.* APIs for locking, any problems with those APIs will cause locking to fail. Specifically, on certain NFS environments the java.nio.* locks will fail (the lock can incorrectly be double acquired) whereas {@link SimpleFSLockFactory} worked perfectly in those same environments. For NFS based access to an index, it's recommended that you try {@link SimpleFSLockFactory} first and work around the one limitation that a lock file could be left when the JVM exits abnormally.

The primary benefit of {@link NativeFSLockFactory} is that lock files will be properly removed (by the OS) if the JVM has an abnormal exit.

Note that, unlike {@link SimpleFSLockFactory}, the existence of leftover lock files in the filesystem on exiting the JVM is fine because the OS will free the locks held against these files even though the files still remain.

If you suspect that this or any other LockFactory is not working properly in your environment, you can easily test it by using {@link VerifyingLockFactory}, {@link LockVerifyServer} and {@link LockStressTest}.

Returns true iff there is such a match.

Behaves as if written:

            boolean skipTo(int target) {
            do {
            if (!next())
            return false;
            } while (target > doc());
            return true;
            }
            

WARNING: The status of the Payloads feature is experimental. The APIs introduced here might change in the future and will not be supported anymore in such a case.

Payloads can only be loaded once per call to {@link #Next()}.

You create a new NumericRangeFilter with the static factory methods, eg:

            Filter f = NumericRangeFilter.newFloatRange("weight",
            new Float(0.3f), new Float(0.10f),
            true, true);
            

NOTE: This API is experimental and might change in incompatible ways in the next release.

Created: Feb 11, 2004 1:23:38 PM

Expert: {@link IndexWriter} uses an instance implementing this interface to execute the merges selected by a {@link MergePolicy}. The default MergeScheduler is {@link ConcurrentMergeScheduler}.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)

NOTE: This class typically requires access to package-private APIs (eg, SegmentInfos) to do its job; if you implement your own MergePolicy, you'll need to put it in package Lucene.Net.Index in order to use these APIs.

NOTE: This API is new and still experimental (subject to change suddenly in the next release)

It uses <segment>.fdt and <segment>.fdx; files.

The boost is multiplied by {@link Lucene.Net.Documents.Document#GetBoost()} of the document containing this field. If a document has multiple fields with the same name, all such values are multiplied together. This product is then used to compute the norm factor for the field. By default, in the {@link Lucene.Net.Search.Similarity#ComputeNorm(String, FieldInvertState)} method, the boost value is multipled by the {@link Lucene.Net.Search.Similarity#LengthNorm(String, int)} and then rounded by {@link Lucene.Net.Search.Similarity#EncodeNorm(float)} before it is stored in the index. One should attempt to ensure that this product does not overflow the range of that encoding.

The default value is 1.0.

Note: this value is not stored directly with the document in the index. Documents returned from {@link Lucene.Net.Index.IndexReader#Document(int)} and {@link Lucene.Net.Search.Hits#Doc(int)} may thus not have the same value present as when this field was indexed.

NOTE: While this option reduces storage space required in the index, it also means any query requiring positional information, such as {@link PhraseQuery} or {@link SpanQuery} subclasses will silently fail to find results.

NOTE: This API is still experimental and subject to change.

Created: Feb 3, 2004 9:00:16 AM

This query uses the {@link MultiTermQuery#CONSTANT_SCORE_AUTO_REWRITE_DEFAULT} rewrite method.

NOTE: The instances returned by this method pre-allocate a full array of length numHits.

If the compoundFile flag is set, then the segments will be merged into a compound file.

Construct a FilterIndexReader based on the specified base reader. Directory locking for delete, undeleteAll, and setNorm operations is left to the base reader.

Note that base reader is closed if this FilterIndexReader is closed.

Term enumerations are always ordered by Term.compareTo(). Each term in the enumeration is greater than all that precede it.

Returns true iff there is such an entry.

Behaves as if written:

            public boolean skipTo(Term target) {
            do {
            if (!next())
            return false;
            } while (target > term());
            return true;
            }
            

WARNING: {@link TeeTokenFilter} and {@link SinkTokenizer} only work with the old TokenStream API. If you switch to the new API, you need to use {@link TeeSinkTokenFilter} instead, which offers the same functionality.

WARNING: Adding tokens to this list requires the {@link #Reset()} method to be called in order for them to be made available. Also, this Tokenizer does nothing to protect against {@link java.util.ConcurrentModificationException}s in the case of adds happening while {@link #Next(Lucene.Net.Analysis.Token)} is being called.

WARNING: Since this SinkTokenizer can be reset and the cached tokens made available again, do not modify them. Modify clones instead.

Example usage:

            PerFieldAnalyzerWrapper aWrapper =
            new PerFieldAnalyzerWrapper(new StandardAnalyzer());
            aWrapper.addAnalyzer("firstname", new KeywordAnalyzer());
            aWrapper.addAnalyzer("lastname", new KeywordAnalyzer());
            

In this example, StandardAnalyzer will be used for all fields except "firstname" and "lastname", for which KeywordAnalyzer will be used.

A PerFieldAnalyzerWrapper can be used like any other analyzer, for both indexing and query parsing.

Subclasses implement search scoring.

The score of query q for document d correlates to the cosine-distance or dot-product between document and query vectors in a Vector Space Model (VSM) of Information Retrieval. A document whose vector is closer to the query vector in that model is scored higher. The score is computed as follows:

score(q,d)   =   coord(q,d)  ·  queryNorm(q)  ·  ∑ ( tf(t in d)  ·  idf(t)2  ·  t.getBoost() ·  norm(t,d) ) t in q

where

1. tf(t in d) correlates to the term's frequency, defined as the number of times term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score. The default computation for tf(t in d) in {@link Lucene.Net.Search.DefaultSimilarity#Tf(float) DefaultSimilarity} is:
    
   

   {@link Lucene.Net.Search.DefaultSimilarity#Tf(float) tf(t in d)}   =

   frequency½

   
    
   
2. idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the term t appears). This means rarer terms give higher contribution to the total score. The default computation for idf(t) in {@link Lucene.Net.Search.DefaultSimilarity#Idf(int, int) DefaultSimilarity} is:
    
   

   {@link Lucene.Net.Search.DefaultSimilarity#Idf(int, int) idf(t)}  =

   1 + log (

   numDocs ––––––––– docFreq+1

   )

   
    
   
3. coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in {@link #Coord(int, int) coord(q,d)} by the Similarity in effect at search time.
    
   
4. queryNorm(q) is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation in {@link Lucene.Net.Search.DefaultSimilarity#QueryNorm(float) DefaultSimilarity} is:
    
   

   queryNorm(q)   =   {@link Lucene.Net.Search.DefaultSimilarity#QueryNorm(float) queryNorm(sumOfSquaredWeights)}   =

   1 –––––––––––––– sumOfSquaredWeights½

   
    
   The sum of squared weights (of the query terms) is computed by the query {@link Lucene.Net.Search.Weight} object. For example, a {@link Lucene.Net.Search.BooleanQuery boolean query} computes this value as:
    
   

   {@link Lucene.Net.Search.Weight#SumOfSquaredWeights() sumOfSquaredWeights}   =   {@link Lucene.Net.Search.Query#GetBoost() q.getBoost()} 2  ·	∑	( idf(t)  ·  t.getBoost() ) 2
   	t in q

   
    
   
5. t.getBoost() is a search time boost of term t in the query q as specified in the query text (see query syntax), or as set by application calls to {@link Lucene.Net.Search.Query#SetBoost(float) setBoost()}. Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi {@link Lucene.Net.Search.TermQuery TermQuery} objects, and so the boost of a term in the query is accessible by calling the sub-query {@link Lucene.Net.Search.Query#GetBoost() getBoost()}.
    
   
6. norm(t,d) encapsulates a few (indexing time) boost and length factors:
   • Document boost - set by calling {@link Lucene.Net.Documents.Document#SetBoost(float) doc.setBoost()} before adding the document to the index.
   • Field boost - set by calling {@link Lucene.Net.Documents.Fieldable#SetBoost(float) field.setBoost()} before adding the field to a document.
   • {@link #LengthNorm(String, int) lengthNorm(field)} - computed when the document is added to the index in accordance with the number of tokens of this field in the document, so that shorter fields contribute more to the score. LengthNorm is computed by the Similarity class in effect at indexing.

   When a document is added to the index, all the above factors are multiplied. If the document has multiple fields with the same name, all their boosts are multiplied together:
    
   

   norm(t,d)   =   {@link Lucene.Net.Documents.Document#GetBoost() doc.getBoost()}  ·  {@link #LengthNorm(String, int) lengthNorm(field)}  ·	∏	{@link Lucene.Net.Documents.Fieldable#GetBoost() f.getBoost}()
   	field f in d named as t

   
    
   However the resulted norm value is {@link #EncodeNorm(float) encoded} as a single byte before being stored. At search time, the norm byte value is read from the index {@link Lucene.Net.Store.Directory directory} and {@link #DecodeNorm(byte) decoded} back to a float norm value. This encoding/decoding, while reducing index size, comes with the price of precision loss - it is not guaranteed that decode(encode(x)) = x. For instance, decode(encode(0.89)) = 0.75. Also notice that search time is too late to modify this norm part of scoring, e.g. by using a different {@link Similarity} for search.
    
   

This is initially an instance of {@link DefaultSimilarity}.

Implementations should calculate a float value based on the field state and then return that value.

For backward compatibility this method by default calls {@link #LengthNorm(String, int)} passing {@link FieldInvertState#GetLength()} as the second argument, and then multiplies this value by {@link FieldInvertState#GetBoost()}.

WARNING: This API is new and experimental and may suddenly change.

Matches in longer fields are less precise, so implementations of this method usually return smaller values when numTokens is large, and larger values when numTokens is small.

Note that the return values are computed under {@link Lucene.Net.Index.IndexWriter#AddDocument(Lucene.Net.Documents.Document)} and then stored using {@link #EncodeNorm(float)}. Thus they have limited precision, and documents must be re-indexed if this method is altered.

This does not affect ranking, but rather just attempts to make scores from different queries comparable.

The encoding uses a three-bit mantissa, a five-bit exponent, and the zero-exponent point at 15, thus representing values from around 7x10^9 to 2x10^-9 with about one significant decimal digit of accuracy. Zero is also represented. Negative numbers are rounded up to zero. Values too large to represent are rounded down to the largest representable value. Positive values too small to represent are rounded up to the smallest positive representable value.

Terms and phrases repeated in a document indicate the topic of the document, so implementations of this method usually return larger values when freq is large, and smaller values when freq is small.

The default implementation calls {@link #Tf(float)}.

A phrase match with a small edit distance to a document passage more closely matches the document, so implementations of this method usually return larger values when the edit distance is small and smaller values when it is large.

Terms and phrases repeated in a document indicate the topic of the document, so implementations of this method usually return larger values when freq is large, and smaller values when freq is small.

The default implementation is:

            return idf(searcher.docFreq(term), searcher.maxDoc());
            

The default implementation uses:

            idf(searcher.docFreq(term), searcher.maxDoc());
            

The default implementation sums the {@link #Idf(Term,Searcher)} factor for each term in the phrase.

The default implementation sums the idf factor for each term in the phrase.

Terms that occur in fewer documents are better indicators of topic, so implementations of this method usually return larger values for rare terms, and smaller values for common terms.

The presence of a large portion of the query terms indicates a better match with the query, so implementations of this method usually return larger values when the ratio between these parameters is large and smaller values when the ratio between them is small.

The default implementation returns 1.

The default implementation returns 1.

• reqScorer != null,
• exclScorer != null,
• reqScorer was advanced once via next() or skipTo() and reqScorer.doc() may still be excluded.

Term enumerations are always ordered by Term.compareTo(). Each term in the enumeration is greater than all that precede it.