Lucene scoring: get cosine similarity as scores - elasticsearch

Question

I'm trying to solve nearest neighbor search problem.
Here is my code:
// Indexing
val analyzer = new StandardAnalyzer()
val directory = new RAMDirectory()
val config = new IndexWriterConfig(analyzer)
val iwriter = new IndexWriter(directory, config)
val queryField = "fieldname"
stringData.foreach { str =>
val doc = new Document()
doc.add(new TextField(queryField, str, Field.Store.YES))
iwriter.addDocument(doc)
}
iwriter.close()
// Searching
val ireader = DirectoryReader.open(directory)
val isearcher = new IndexSearcher(ireader)
val parser = new QueryParser(queryField, analyzer)
val query = parser.parse("Some text for testing")
val hits = isearcher.search(query, 10).scoreDocs
When I look on the value hits I see scores more then 1.
As far as I know, lucene scoring formula is:
score(q,d) = coord-factor(q,d) · query-boost(q) · cosSim(q,d) · doc-len-norm(d) · doc-boost(d)
But I want to get only cosine similarity in range[0,1] between query and document instead of coord-factor, doc-len-norm and so on.
What is a possible way to achieve it?

Answer 1

If you have gone through this official documentation, you would realize that the rest of the terms in the score expression is important and makes the scoring process more logical and coherent.
But still if you want to achieve a scoring process using only Cosine Similaity, then you can write your custom similarity class. I have used different types of similarity method for document retrieval in my class assignment. So, in short you can write your own similarity method and assign it to the Lucene's index searcher. I am giving an example here which you modify to accomplish what you want.
Write your custom class (you just need to override one method in your class).
import org.apache.lucene.search.similarities.BasicStats;
import org.apache.lucene.search.similarities.SimilarityBase;
public class MySimilarity extends SimilarityBase {
#Override
protected float score(BasicStats stats, float termFreq, float docLength) {
double tf = 1 + (Math.log(termFreq) / Math.log(2));
double idf = Math.log((stats.getNumberOfDocuments() + 1) / stats.getDocFreq()) / Math.log(2);
float dotProduct = (float) (tf * idf);
return dotProduct;
}
}
Then assign your implemented method to index searcher for relevance calculation as below.
IndexReader reader = DirectoryReader.open(FSDirectory.open(new File(indexPath)));
IndexSearcher indexSearcher = new IndexSearcher(reader);
indexSearcher.setSimilarity(new MySimilarity());
Here, i am using tf-idf dot product to compute similarity between query and documents. Formula is,
Two things need to be mentioned here are:
stats.getNumberOfDocuments() returns total number documents in the index.
stats.getDocFreq() returns document frequency for a term appeared in both query and document.
Lucene will now call the score() method that you have implemented to compute relevance score for each of the matched terms; terms that appeare both in query and documents.
This is not an straight forward answer to your question i know but you can use the approach i mentioned above in anyway you want. I implemented 6 different scoring technique in my homework assignment. I hope it will help you too.