I was trying to read document(json data stored in ES) from elasticsearch using alpakka.
I got this alpakka-Elasticsearch.
Here it says that you can stream messages from or to Elasticsearch using the
ElasticsearchSource, ElasticsearchFlow or the ElasticsearchSink.
I tried to impliment ElasticsearchSource method. So my code looks like this
val url = "http://localhost:9200"
val connectionSettings = ElasticsearchConnectionSettings(url)
val sourceSettings = ElasticsearchSourceSettings(connectionSettings)
val elasticsearchParamsV7 = ElasticsearchParams.V7("category_index")
val copy = ElasticsearchSource
.typed[CategoryData](
elasticsearchParamsV7,
query = query,
sourceSettings
).map { message: ReadResult[CategoryData] =>
println("Inside message==================> "+message)
WriteMessage.createIndexMessage(message.id, message.source)
} .runWith(
ElasticsearchSink.create[CategoryData](
elasticsearchParamsV7,ElasticsearchWriteSettings(connectionSettings)
)
)
println("Final data==============>. "+copy)
At the end, copy value returning Future[Done].
But I was not able to read data from ES.
Is there Something I missing?
And also is there any other way using akka http client api to do the same?
What is preferred way to use ES in akka?
To read data from Elasticsearch, sth like this should be enough:
val matchAllQuery = """{"match_all": {}}"""
val result = ElasticsearchSource
.typed[CategoryData](
elasticsearchParamsV7,
query = matchAllQuery,
sourceSettings
).map { message: ReadResult[CategoryData] =>
println("Read message==================> "+message)
}.runWith(Sink.seq)
result.onComplete(res => res.foreach(col => println(s"Read: ${col.size} records")))
If the type CategoryData does not match correctly to what is stored in the index, the query may not return results.
If in doubt, it's possible to read raw JSON:
val elasticsearchSourceRaw = ElasticsearchSource
.create(
elasticsearchParamsV7,
query = matchAllQuery,
settings = sourceSettings
)
Related
Any ideas why I get this error when making a query:
org.springframework.data.mongodb.InvalidMongoDbApiUsageException: Due to limitations of the com.mongodb.BasicDocument, you can't add a second 'id' criteria. Query already contains '{ "id" : "123"}'
I'm using Spring Boot and Mongo:
fun subGenreNames(subGenreIds: List<String>?): List<String> {
val results = mutableListOf<String>()
var query = Query()
subGenreIds!!.forEach{
query.addCriteria(Criteria.where("id").`is`(it))
var subGenreName = mongoTemplate.findById(it, SubGenre::class.java)
results.add(subGenreName!!.name)
}
return results
}
I have the class SubGenre set with:
#Document(collection = "subgenres")
data class SubGenre(
#Field("id")
val id: String,
val name: String
)
Thanks
Based on your code, you need to use either
query.addCriteria(Criteria.where("id").`is`(it))
var subGenreName = mongoTemplate.find(query, SubGenre::class.java)
or
var subGenreName = mongoTemplate.findById(it, SubGenre::class.java)
but not both.
I'm making application that fetches data from elasticsearch and sends it to kafka. But producer.send() function does not work inside of map, however outside of it, everything works perfectly
val f1 = ElasticsearchSource
.create(
indexName = "products",
typeName = "product",
query = """{"match_all": {}}"""
)
.map { message: OutgoingMessage[spray.json.JsObject] =>
val product = message.source
producer.send(new ProducerRecord("test", product))
println("publishing message ")
IncomingMessage(Some(message.id), message.source)
}
.runWith(Sink.seq)
What might be the cause of it?
I want to send a Nest delete request to elasticsearch without specifying the object which I don't have. I've seen solutions like:
var response = elasticClient.DeleteByQuery<MyClass>(q => q
.Match(m => m.OnField(f => f.Guid).Equals(someObject.Guid))
);
From: DeleteByQuery using NEST and ElasticSearch
As I'm just reading plain text from a queue I don't have access to the MyClass object to use with the delete request. Basically I just want to delete all documents in an index (whose name I know) where a variable matches for example ordId = 1234. Something like:
var response = client.DeleteByQuery<string>( q => q
.Index(indexName)
.AllTypes()
.Routing(route)
.Query(rq => rq
.Term("orgId", "1234"))
);
I see that the nest IElasticClient interface does have a DeleteByQuery method that doesn't require the mapping object but just not sure how to implement it.
You can just specify object as the document type T for DeleteByQuery<T> - just be sure to explicitly provide the index name and type name to target in this case. T is used to provide strongly type access within the body of the request only. For example,
var client = new ElasticClient();
var deleteByQueryResponse = client.DeleteByQuery<object>(d => d
.Index("index-name")
.Type("type-name")
.Query(q => q
.Term("orgId", "1234")
)
);
Will generate the following query
POST http://localhost:9200/index-name/type-name/_delete_by_query
{
"query": {
"term": {
"orgId": {
"value": "1234"
}
}
}
}
Replace _delete_by_query with _search in the URI first, to ensure you're targeting the expected documents :)
Just find out about nest. I already insert some number of document in Elastic Search. Right now I want to search the data based on my type, subcriberId. I did run through curl and it works just fine. But when I tried using nest, no result found.
My curl which work:
http://localhost:9200/20160902/_search?q=subscribeId:aca0ca1a-c96a-4534-ab0e-f844b81499b7
My NEST code:
var local = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(local);
var elastic = new ElasticClient(settings);
var response = elastic.Search<IntegrationLog>(s => s
.Index(DateTime.Now.ToString("yyyyMMdd"))
.Type("integrationlog")
.Query(q => q
.Term(p => p.SubscribeId, new Guid("aca0ca1a-c96a-4534-ab0e-f844b81499b7"))
)
);
Can someone point what I did wrong?
A key difference between your curl request and your NEST query is that the former is using a query_string query and the latter, a term query. A query_string query input undergoes analysis at query time whilst a term query input does not so depending on how subscribeId is analyzed (or not), you may see different results. Additionally, your curl request is searching across all document types within the index 20160902.
To perform the exact same query in NEST as your curl request would be
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool)
// set up NEST with the convention to use the type name
// "integrationlog" for the IntegrationLog
// POCO type
.InferMappingFor<IntegrationLog>(m => m
.TypeName("integrationlog")
);
var client = new ElasticClient(connectionSettings);
var searchResponse = client.Search<IntegrationLog>(s => s
.Index("20160902")
// search across all types. Note that documents found
// will be deserialized into instances of the
// IntegrationLog type
.AllTypes()
.Query(q => q
// use query_string query
.QueryString(qs => qs
.Fields(f => f
.Field(ff => ff.SubscribeId)
)
.Query("aca0ca1a-c96a-4534-ab0e-f844b81499b7")
)
)
);
}
public class IntegrationLog
{
public Guid SubscribeId { get; set; }
}
This yields
POST http://localhost:9200/20160902/_search
{
"query": {
"query_string": {
"query": "aca0ca1a-c96a-4534-ab0e-f844b81499b7",
"fields": [
"subscribeId"
]
}
}
}
this specifies the query_string query in the body of the request which is analogous to using the q query string parameter to specify the query.
I am trying to do a search in ElasticSearch using Nest. I want to use the object initializer syntax because I need to build the parts of the search dynamically. I have figured out how to build much of the request, but am not clear how I would initialize a Raw Query. The OIS doesn't seem to have QueryRaw as a parameter to the request.
Code that I have now:
var searchResults = client.Search<dynamic>(s => s
.Index("myIndex"),
.Type("myType),
.Aggregations(a => a
.DateHistogram("my_date_histogram", h => h
.Field("DateField")
.Interval("day")
)
)
.QueryRaw(queryText)
)
Code that I am trying to create:
var request = new SearchRequest<dynamic>
{
Index = "MyIndex",
Type = "MyType",
QueryRaw = <doesn't exist>
};
You can do this by
var searchResponse = client.Search<dynamic>(new SearchRequest
{
Query = new RawQuery(yourquery)
});
Tested with NEST 2.0.0.alpha2 and ES 2.1.0
Here's how to do raw queries using the new object structure:
var response = client.Search<dynamic>(s => s
.Query(qry => qry
.Raw(yourRawQueryStringHere)
)
);