Unable to create an index with nested json using Jest elastic client - elasticsearch

I am using java and Jest (https://github.com/searchbox-io/Jest) as a client for elastic search cluster.
Trying to create and index with the below structure in http://localhost:9200/myindex/mytype
{
"doc": {
"_content": "mycontent",
"_name": "mypdf.pdf",
"_content_type": "application/pdf"
}
}
And below is my java code,
XContentBuilder docObject = jsonBuilder().startObject().field("_content", doc).field("_name", name).field("_content_type", contentType)
.field("title", title).field("author", author).endObject();
index1 = new Index.Builder(docObject).build();
source = jsonBuilder()
.startObject()
.field("doc", docObject.string())
.endObject().string();
Index index = new Index.Builder(source).index(baseIndexName).type(ElasticConstants.BASE_TYPE).build();
But when this gets executed , the source is not passed as a Json and value for the key "doc" is passed as a string literal because of which the index is not getting created . How do I pass nested json objects to the Index.Builder using Jest ?

//Instantiate JEST Client
JestClientFactory factory = new JestClientFactory();
HttpClientConfig config = new HttpClientConfig.
Builder("http://localhost:9201")
.multiThreaded(true).build();
factory.setHttpClientConfig(config);
JestClient jestClient = factory.getObject();
//create index
jestClient.execute(new CreateIndex.Builder("so4index").build());
//create index mapping
RootObjectMapper.Builder rootObjMapBuilder1 = new RootObjectMapper.Builder("so1type");
RootObjectMapper.Builder rootObjMapBuilder = new RootObjectMapper.Builder("doc");
rootObjMapBuilder.add(new StringFieldMapper.Builder("_content").store(true));
rootObjMapBuilder.add(new StringFieldMapper.Builder("_name").store(true));
rootObjMapBuilder.add(new StringFieldMapper.Builder("_content_type").store(true));
rootObjMapBuilder1.add(rootObjMapBuilder);
DocumentMapper docMapper = new DocumentMapper.Builder("so4index", null, rootObjMapBuilder1).build(null);
String source = docMapper.mappingSource().toString();
PutMapping putMapping = new PutMapping.Builder("so4index", "so1type", source).build();
JestResult mappingResult = jestClient.execute(putMapping);
//Insert Document Content
String doc="mycontent";
String name="mypdf.pdf";
String contentType = "application/pdf";
XContentBuilder docObject = jsonBuilder()
.startObject()
.startObject("doc").field("_content", doc)
.field("_name", name)
.field("_content_type", contentType)
.endObject()
.endObject();
Index index = new Index.Builder(docObject.string()).index("so4index").type("so1type").build();
JestResult jestResult= jestClient.execute(index);
jestClient.shutdownClient();

Related

GraphQL on LinkedHashMap Data with schema and query

I have a requirement to transform the source JSON to a target JSON structure; for this, I am experimenting with GraphQL.
Below is my Java Code
String schema = "type Query{AuthToken: String}";
SchemaParser schemaParser = new SchemaParser();
TypeDefinitionRegistry typeDefinitionRegistry = schemaParser.parse(schema);
DataFetcher<LinkedHashMap<Object, Object>> dataFetcher = environment -> {
String json = "{\"AuthToken\":\"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJjdXJyZW50QmF0Y2hOdW1iZXIiOiIxODk3MTczIiwidGVybWluYWxJZCI6InhucFByb2R1Y3QtT25saW5lIiwidXNlck5hbWUiOiJ4bnBVc2VyMSIsInBhc3N3b3JkIjoiQ24zdDNXaDlUZjlHYXBkT2RCamdNQTgrM1NZPSIsImVuYyI6InRydWUiLCJhdXRoVHlwZSI6IkJBU0lDIiwidG9rZW5UeXBlIjoieG5wX2F1dGhfdG9rZW4iLCJuYmYiOjE2NjI2NTIzMDMsImV4cCI6MTY2MjY1MjkwMywiaWF0IjoxNjYyNjUyMzAzLCJpc3MiOiJodHRwczovL3F3aWtjaWx2ZXIuY29tLyJ9.tiYe88waCpiWKAmYCkFyxxBCmuyFgLas_XsobFWxDrqfI9jzbrqQ1ZNiPdsovULd6xk1_0553aVOPJap15GE6Q\",\"ResponseMessage\":\"Transactionsuccessful.\"}";
TypeReference<LinkedHashMap<Object, Object>> typeReference = new TypeReference<>() {
};
return mapper.readValue(json, typeReference);
};
RuntimeWiring runtimeWiring = newRuntimeWiring()
.type("Query", builder -> builder.defaultDataFetcher(dataFetcher))
.build();
SchemaGenerator schemaGenerator = new SchemaGenerator();
GraphQLSchema graphQLSchema = schemaGenerator.makeExecutableSchema(typeDefinitionRegistry, runtimeWiring);
GraphQL build = GraphQL.newGraphQL(graphQLSchema).build();
ExecutionResult executionResult = build.execute("{token: AuthToken}");
log.info("GraphQL response {}", executionResult.getData().toString());
However, when this code is executed, GraphQL returns the complete JSON and not just the property AuthToken.
Also, the expected response is of structure
{token: 'the token from the input json' }
however, below response is received
{token={AuthToken=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJjdXJyZW50QmF0Y2hOdW1iZXIiOiIxODk3MTczIiwidGVybWluYWxJZCI6InhucFByb2R1Y3QtT25saW5lIiwidXNlck5hbWUiOiJ4bnBVc2VyMSIsInBhc3N3b3JkIjoiQ24zdDNXaDlUZjlHYXBkT2RCamdNQTgrM1NZPSIsImVuYyI6InRydWUiLCJhdXRoVHlwZSI6IkJBU0lDIiwidG9rZW5UeXBlIjoieG5wX2F1dGhfdG9rZW4iLCJuYmYiOjE2NjI2NTIzMDMsImV4cCI6MTY2MjY1MjkwMywiaWF0IjoxNjYyNjUyMzAzLCJpc3MiOiJodHRwczovL3F3aWtjaWx2ZXIuY29tLyJ9.tiYe88waCpiWKAmYCkFyxxBCmuyFgLas_XsobFWxDrqfI9jzbrqQ1ZNiPdsovULd6xk1_0553aVOPJap15GE6Q, ResponseMessage=Transactionsuccessful.}}

Scroll API misses some documents

I am trying to get all the documents from multiple indexes with Scroll Api but it doesn't return all of them. I found a similar question but op was obviously missing first set of documents. Link to the question: Elasticsearch Search Scroll API doesn't retrieve all the documents from an index
Here is my code:
//Code to get indexes
for (String indexName : indexNames) {
final Scroll scroll = new Scroll(TimeValue.timeValueSeconds(45L));
SearchRequest searchRequest = new SearchRequest(indexName);
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
QueryBuilder query = QueryBuilders.boolQuery()
.filter(QueryBuilders.termQuery(sourceId, 2))
.filter(QueryBuilders.rangeQuery(date).gte(01-05-2021).lte(31-05-2021));
searchSourceBuilder.query(query);
searchSourceBuilder.size(10000);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
List<Model> model = new ArrayList<>();
while(searchHits != null && searchHits.length > 0) {
for (SearchHit document : searchHits){
//add document to model list created above
} //end of for loop
// insert model list to database
SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(scroll);
searchResponse = client.scroll(searchScrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();
} //end of while loop
ClearScrollRequest clear = new ClearScrollRequest();
clear.addScrollId(scrollId);
} //end of for loop at the top
Total number of documents I should get is 115 millions but I am missing more than 2 millions documents. I repeatedly checked my code but no idea what I am missing.

How to Insert data in Elastic Search

In my dotnet core project i need to insert data in elastic search. I am using bellow code for insert.
List<Patient> employeeData = null;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://localhost:5001/api/Employee/GetAll");
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
var data= reader.ReadToEnd();
reader.Close();
dataStream.Close();
employeeData = JsonConvert.DeserializeObject<List<Employee>>(data);
}
var lst = employeeData;
int count = 0;
foreach (var obj in lst)
{
count ++;
this.client.Index(obj, i => i
.Index("employee")
.Type("myEmployee")
.Id(count)
// .Refresh()
);
}
After Execute the above code i am using bellow url for check the inserted data
localhost:9200/emp
I am getting following output.
{"emp":{"aliases":{},"mappings":{"myEmpl":{"properties":{"firstName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"gEmailId":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"gMobile":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"wmployeeID":{"type":"long"},"registrationNo":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1547020635852","number_of_shards":"5","number_of_replicas":"1","uuid":"6kle4jzMQDSICnPsmATbDw","version":{"created":"6050499"},"provided_name":"emp"}}}}
I am not able to see any of my data. What is the problem in this.
localhost:9200/emp returns the settings and mappings for index 'emp'. For the content, try localhost:9200/emp/_search.

How to create and retrieve using the Java API an Elasticsearch field with a complex type?

I have the following:
final duration = (jsonBuilder()
.startObject()
.field('start', new DateTime(testResult.startTime, dateTimeZone))
.field('end', new DateTime(testResult.endTime, dateTimeZone))
.endObject())
client.prepareIndex('builds', 'test')
.setSource(jsonBuilder()
.startObject()
.field("duration", duration)
.endObject())
SearchResponse searchResponse = client.prepareSearch('builds')
.setQuery(boolQuery()
.must(termQuery('_type', 'test')))
.execute()
.actionGet()
final source = searchResponse.hits.hits[0].source as Map<String, Object>
How do I retrieve the values of duration.start and duration.end from here?
Try 1..!
SearchHit[] searchHits = searchResponse.getHits().getHits();
Map<String, Object> s=searchHits[0].sourceAsMap();
Map<String, Date> duration=(Map<String, Date>) s.get("duration");
Date start=duration.get("start");
Date end=duration.get("end");
Try 2..!
SearchHit[] searchHits = searchResponse.getHits().getHits();
StringBuilder builder = new StringBuilder();
int length = searchHits.length;
builder.append("[");
for (int i = 0; i < length; i++) {
if (i == length - 1) {
builder.append(searchHits[i].getSourceAsString());
} else {
builder.append(searchHits[i].getSourceAsString());
builder.append(",");
}
}
builder.append("]");
String result= builder.toString();
it will return a string and its a valid JSON array use JSON parser and fetch values as normal json process..!
The problem is that field() doesn't recognize XContentBuilder as a value despite what http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/index_.html implies. From the source code for XContentBuilder, it's unclear to me how to use field with XContentBuilder.
It's easy enough to use a Map as a value, though.
final duration = [
'start': new DateTime(testResult.startTime, dateTimeZone),
'end': new DateTime(testResult.endTime, dateTimeZone)]
client.prepareIndex('builds', 'test')
.setSource(jsonBuilder()
.startObject()
.field("duration", duration)
.endObject())
SearchResponse searchResponse = client.prepareSearch('builds')
.setQuery(boolQuery()
.must(termQuery('_type', 'test')))
.execute()
.actionGet()
final source = searchResponse.hits.hits[0].source
assertThat(source.duration.start, equalTo('1970-01-01T00:00:00.001Z'))
assertThat(source.duration.end, equalTo('1970-01-01T00:00:00.002Z'))

How to Get the Array of data from json array using WEB API in mvc3?

I have create web services using Web Api in mvc3,in this i want get the data from json. Json Result like this
{"order": {
"locationid": "1",
"deviceidentifier": "XXXXXXXXXXXXXXXXXXXXXXX",
"ordercontactname": "XXXXXXXXXXXXXXXXXXXXXXX",
"ordercontactphone": "XXXXXXXXXXXXXXXXXXXXXXX",
"ordercontactemail": "XXXXXXXXXXXXXXXXXXXXXXX",
"shipaddress1": "17 Brookfield",
"shipaddress2": "Suite 17",
"shipcity": "Irvine",
"shipstate": "CA",
"shipzipcode": "92604",
"shipphonenumber": "9493742114",
"shipemail": "Info#mail.com",
"shipmethod": "pickup",
"billingfirstname":"Tester",
"billinglastname":"test",
"billingmiddleinitial":"S",
"billingaddress":"field",
"billingcity":"Sample",
"billingstate":"SM",
"billingzipcode":"523201",
"billingphonenumber": "1234567891",
"billingemail": "",
"paytype":"creditcard",
"amount"="10.50",
"acctno"="123456789987",
"exproute"="0114",
"coupon"="X2323",
"notes"="",
"items": [
{"itemid":"1","quantity":"1","price":"2.5","notes":"make it spicy"},
{"itemid":"4","quantity":"2","price":"4.5","notes":""},
{"itemid":"3","quantity":"1","price":"1.5","notes":""}
]
}}
for this i have create Poco class and i get Order data using poco class, but i can't get the items array data how can i get items data
Here is my code
public List<Message> PlaceOrder(PlaceOrder order)
{
// List<PlaceOrder> entities = (List<PlaceOrder>)JavaScriptConvert.DeserializeObject(json, typeof(List<PlaceOrder>));
int PayOrderID = 0;
List<Message> Result;
Result = new List<Message>();
try
{
Order objOrder = new Order();
PayHist objPayhis = new PayHist();
objOrder.LocationId = order.LocationId;
objOrder.DeviceIdentifier = order.DeviceIdentifier;
objOrder.OrderContactName = order.OrderContactName;
objOrder.OrderContactPhone = order.OrderContactPhone;
objOrder.OrderContactEmail = order.OrderContactEmail;
string guid = Guid.NewGuid().ToString();
objOrder.ShipMethod = order.ShipMethod;
objOrder.ShippingDate = Convert.ToDateTime(order.PickupDate);
objOrder.OrderGuid = guid;
entities.AddObject("Orders", objOrder);
entities.SaveChanges();
int orderId = objOrder.OrderId;
PayOrderID = orderId;
objPayhis.OrderId = orderId;
objPayhis.PayType = order.ShipMethod;
objPayhis.Amount = float.Parse(order.Amount);
entities.AddObject("PayHists", objPayhis);
entities.SaveChanges();
JavaScriptSerializer ser = new JavaScriptSerializer();
// Order foo = ser.Deserialize<Order>(json);
Message objMessage = new Message();
objMessage.Messagevalue = "Sucess";
Result.Add(objMessage);
return Result;
}
Please help me..
Try this (you need to fix your Json by replacing the "=" signs by ":" signs" before):
[WebInvoke(Method = "POST", UriTemplate = "")]
public HttpResponseMessage Add(JsonValue json) {
JsonValue order = json["order"];
JsonArray items = (JsonArray) order["items"];
JsonValue item1 = items[0];
var notes1 = item1["notes"];
return new HttpResponseMessage(HttpStatusCode.OK);
}

Resources