I'm making application that fetches data from elasticsearch and sends it to kafka. But producer.send() function does not work inside of map, however outside of it, everything works perfectly
val f1 = ElasticsearchSource
.create(
indexName = "products",
typeName = "product",
query = """{"match_all": {}}"""
)
.map { message: OutgoingMessage[spray.json.JsObject] =>
val product = message.source
producer.send(new ProducerRecord("test", product))
println("publishing message ")
IncomingMessage(Some(message.id), message.source)
}
.runWith(Sink.seq)
What might be the cause of it?
Related
I was trying to read document(json data stored in ES) from elasticsearch using alpakka.
I got this alpakka-Elasticsearch.
Here it says that you can stream messages from or to Elasticsearch using the
ElasticsearchSource, ElasticsearchFlow or the ElasticsearchSink.
I tried to impliment ElasticsearchSource method. So my code looks like this
val url = "http://localhost:9200"
val connectionSettings = ElasticsearchConnectionSettings(url)
val sourceSettings = ElasticsearchSourceSettings(connectionSettings)
val elasticsearchParamsV7 = ElasticsearchParams.V7("category_index")
val copy = ElasticsearchSource
.typed[CategoryData](
elasticsearchParamsV7,
query = query,
sourceSettings
).map { message: ReadResult[CategoryData] =>
println("Inside message==================> "+message)
WriteMessage.createIndexMessage(message.id, message.source)
} .runWith(
ElasticsearchSink.create[CategoryData](
elasticsearchParamsV7,ElasticsearchWriteSettings(connectionSettings)
)
)
println("Final data==============>. "+copy)
At the end, copy value returning Future[Done].
But I was not able to read data from ES.
Is there Something I missing?
And also is there any other way using akka http client api to do the same?
What is preferred way to use ES in akka?
To read data from Elasticsearch, sth like this should be enough:
val matchAllQuery = """{"match_all": {}}"""
val result = ElasticsearchSource
.typed[CategoryData](
elasticsearchParamsV7,
query = matchAllQuery,
sourceSettings
).map { message: ReadResult[CategoryData] =>
println("Read message==================> "+message)
}.runWith(Sink.seq)
result.onComplete(res => res.foreach(col => println(s"Read: ${col.size} records")))
If the type CategoryData does not match correctly to what is stored in the index, the query may not return results.
If in doubt, it's possible to read raw JSON:
val elasticsearchSourceRaw = ElasticsearchSource
.create(
elasticsearchParamsV7,
query = matchAllQuery,
settings = sourceSettings
)
I am trying to create an Elasticsearch MLT query using NEST's object initializer syntax. However, the final query when serialized, is ONLY missing the MLT part of it. Every other query is present though.
When inspecting the query object, the MLT is present. It's just not getting serialized.
I wonder what I may be doing wrong.
I also noticed that when I add Fields it works. But I don't believe fields is a mandatory property here that when it is not set, then the MLT query is ignored.
The MLT query is initialized like this;
new MoreLikeThisQuery
{
Like = new[]
{
new Like(new MLTDocProvider
{
Id = parameters.Id
}),
}
}
MLTDocProvider implements the ILikeDocument interface.
I expect the serialized query to contain the MLT part, but it is the only part that is missing.
This looks like a bug in the conditionless behaviour of more like this query in NEST; I've opened an issue to address. In the meantime, you can get the desired behaviour by marking the MoreLikeThisQuery as verbatim, which will override NEST's conditionless behaviour
var client = new ElasticClient();
var parameters = new
{
Id = 1
};
var searchRequest = new SearchRequest<Document>
{
Query = new MoreLikeThisQuery
{
Like = new[]
{
new Like(new MLTDocProvider
{
Id = parameters.Id
}),
},
IsVerbatim = true
}
};
var searchResponse = client.Search<Document>(searchRequest);
which serializes as
{
"query": {
"more_like_this": {
"like": [
{
"_id": 1
}
]
}
}
}
I am trying to do a search in ElasticSearch using Nest. I want to use the object initializer syntax because I need to build the parts of the search dynamically. I have figured out how to build much of the request, but am not clear how I would initialize a Raw Query. The OIS doesn't seem to have QueryRaw as a parameter to the request.
Code that I have now:
var searchResults = client.Search<dynamic>(s => s
.Index("myIndex"),
.Type("myType),
.Aggregations(a => a
.DateHistogram("my_date_histogram", h => h
.Field("DateField")
.Interval("day")
)
)
.QueryRaw(queryText)
)
Code that I am trying to create:
var request = new SearchRequest<dynamic>
{
Index = "MyIndex",
Type = "MyType",
QueryRaw = <doesn't exist>
};
You can do this by
var searchResponse = client.Search<dynamic>(new SearchRequest
{
Query = new RawQuery(yourquery)
});
Tested with NEST 2.0.0.alpha2 and ES 2.1.0
Here's how to do raw queries using the new object structure:
var response = client.Search<dynamic>(s => s
.Query(qry => qry
.Raw(yourRawQueryStringHere)
)
);
I can create the following string saved in a Java String object called updates.
{ "update":{ "_index":"myindex", "_type":"order", "_id":"1"} }
{ "doc":{"field1" : "aaa", "field2" : "value2" }}
{ "update":{ "_index":"myindex", "_type":"order", "_id":"2"} }
{ "doc":{"field1" : "bbb", "field2" : "value2" }}
{ "update":{ "_index":"myindex", "_type":"order", "_id":"3"} }
{ "doc":{"field1" : "ccc", "field2" : "value2" }}
Now I want to do bullk update within a Java program:
Client client = getClient(); //TransportClient
BulkRequestBuilder bulkRequest = client.prepareBulk();
//?? how to attach updates variable to bulkRequest?
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
I am unable to find a way to attach the above updates variable to bulkRequest before execute.
I notice that I am able to add UpdateRequest object to bulkRequest, but it seems to add only one document one time. As indicated above, I have multiple to-be-updated document in one string.
Can someone enlighten me on this? I have a gut feel that I may do things wrong way.
Thanks and regards.
The following code should work fine for you.
For each document updation , you need to create a separate update request as below and keep on adding it to the bulk requests.
Once the bulk requests is ready , execute a get on it.
JSONObject obj = new JSONObject();
obj.put("field1" , "value1");
obj.put("field2" , "value2");
UpdateRequest updateRequest = new UpdateRequest(index, indexType, id1).doc(obj.toString());
BulkRequestBuilder bulkRequest = client.prepareBulk();
bulkRequest.add(updateRequest);
obj = new JSONObject();
obj.put("fieldX" , "value1");
obj.put("fieldY" , "value2");
updateRequest = new UpdateRequest(index, indexType, id2).doc(obj.toString());
bulkRequest = client.prepareBulk();
bulkRequest.add(updateRequest);
bulkRequest.execute().actionGet();
I ran into the same problem where only 1 document get updated in my program. Then I found the following way which worked perfectly fine. This uses spring java client. I have also listed the the dependencies I used in the code.
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.index.query.QueryBuilder;
import org.springframework.data.elasticsearch.core.query.UpdateQuery;
import org.springframework.data.elasticsearch.core.query.UpdateQueryBuilder;
private UpdateQuery updateExistingDocument(String Id) {
// Add updatedDateTime, CreatedDateTime, CreateBy, UpdatedBy field in existing documents in Elastic Search Engine
UpdateRequest updateRequest = new UpdateRequest().doc("UpdatedDateTime", new Date(), "CreatedDateTime", new Date(), "CreatedBy", "admin", "UpdatedBy", "admin");
// Create updateQuery
UpdateQuery updateQuery = new UpdateQueryBuilder().withId(Id).withClass(ElasticSearchDocument.class).build();
updateQuery.setUpdateRequest(updateRequest);
// Execute update
elasticsearchTemplate.update(updateQuery);
}
See the function 'postRequest' in this UserModel class I have defined. Well the function is not used anymore anywhere and when I try to remove it XCode syntax highlighting crashes and my program will not compile with a segmentation fault error 11 which points to nowhere.
I moved the function into a struct called SharedModel. So this function is not called anywhere! Why can I not remove it from the file? It makes no sense, I just want to remove an unused function. But Xcode keeps crashing and it makes me keep it! What is going on.
import Foundation
import SwiftHTTP
import SwiftyJSON
public enum UserProperties : String {
case user_id = "user_id", single_access_token = "single_access_token", first_name = "first_name",
last_name = "last_name", role = "role", email = "email", username = "username", barn_id = "barn_id",
location_id = "location_id", farm_id = "farm_id"
static let allValues = [user_id, single_access_token, first_name, last_name, role, email, username, barn_id, location_id, farm_id]
}
class UserModel {
var userPropertiesJSON: JSON?
init () {
}
func postRequest(url: String, params: Dictionary<String,AnyObject>, completionHandler: (JSON?) -> ()) {
var request = HTTPTask()
request.requestSerializer = JSONRequestSerializer()
//The expected response will be JSON and be converted to an object return by NSJSONSerialization instead of a NSData.
request.responseSerializer = JSONResponseSerializer()
//we have to add the explicit type, else the wrong type is inferred. See the vluxe.io article for more info.
//let params: Dictionary<String,AnyObject> = ["username": username, "password": password]
request.POST(url, parameters: params, success: {(response: HTTPResponse) in
println(response.responseObject!)
let json = JSON(response.responseObject!)
completionHandler(json)
},failure: {(error: NSError, response: HTTPResponse?) in
completionHandler(false)
})
}
func login(username: String, password: String, completionHandler: (Bool) -> ()) {
let params: Dictionary<String,AnyObject> = ["username": username, "password": password]
SharedModel.postRequest("http://farmcentral.softimum.com/user_sessions.json", params: params) { jsonOrError in
if let json: JSON = jsonOrError {
self.userPropertiesJSON = json
SharedModel.userPropertiesJSON = json
completionHandler(true)
} else {
completionHandler(false)
}
}
}
}
UPDATE
Still and issue but I have narrowed it down to the exact line of code that if removed causes xcode to crash and causes the build to stop compiling due to a segment fault 11 error.
This is the offending line: let json = JSON(response.responseObject!)
This line is in the function inside the closure. When I remove this line Xcode crashes. I am running cocoapods beta version, this might be a bug.
This was a bug with the framework I was using called SwiftyJSON. The solution was to import SwiftyJSON.swift file directly into your project.
Here is the Github issue: https://github.com/SwiftyJSON/SwiftyJSON/issues/67