Elasticsearch query gives different results from cURL and Kibana - elasticsearch

I am a newbie in Elasticsearch and kibana and am trying to connect elasticsearch with PHP to generate some reports.
I am trying to run a query, which runs perfectly fine in kibana sense but surprisingly gives different results whenever I cURL it.
GET /_search
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*",
"analyze_wildcard": true
}
},
"filter": {
"bool": {
"must": [
{
"query": {
"query_string": {
"analyze_wildcard": true,
"query": "*"
}
}
},
{
"range": {
"start_time": {
"gte": 1532889000000,
"lte": 1532975399999,
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
}
}
},
"size": 0,
"aggs": {
"2": {
"terms": {
"field": "layers",
"size": 50,
"order": {
"_count": "desc"
}
}
}
}
}
When I cURL this query, it returns documents instead of aggregate results.
Where am I going wrong?
TIA
EDIT: I am working on Elasticsearch 2.3
The PHP Code I am running to retrieve the results:
$jarray=array (
'query' =>
array (
'filtered' =>
array (
'query' =>
array (
'query_string' =>
array (
'query' => '*',
'analyze_wildcard' => true,
),
),
'filter' =>
array (
'bool' =>
array (
'must' =>
array (
0 =>
array (
'query' =>
array (
'query_string' =>
array (
'analyze_wildcard' => true,
'query' => '*',
),
),
),
1 =>
array (
'range' =>
array (
'start_time' =>
array (
'gte' => 1532889000000,
'lte' => 1532975399999,
'format' => 'epoch_millis',
),
),
),
),
'must_not' =>
array (
),
),
),
),
),
'size' => 0,
'aggs' =>
array (
2 =>
array (
'terms' =>
array (
'field' => 'layer_count',
'size' => 50,
'order' =>
array (
'_count' => 'desc',
),
),
),
),
);
$jdata=json_encode($jarray);
$url = '10.10.113.97:9200/my_index/_search';
echo $url.'<br><br><br><br><br><br>';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_GETFIELDS, json_encode($jdata));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$return = curl_exec($ch) or die(curl_error());
curl_close($ch);
print_r(stripcslashes($return));

In your cURL code, you need to change the field you use in your aggregation section:
'aggs' =>
array (
2 =>
array (
'terms' =>
array (
'field' => 'layers', <--- change this line
'size' => 50,
'order' =>
array (
'_count' => 'desc',
),
),
),
),
Also you need to send your request like this. My worry is that you're sending a GET without any request body and ES will simply ignore your query and perform a generic search.
$query = json_encode($jdata);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($query))
);

Related

Curl didn't work PHP, always return empty esponse

I always get the return empty when I call my api with this code, but it work on my Insomnia, and on my shell
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $this->headers);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_URL, $this->url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($this->body));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$return_data = curl_exec($ch);
curl_close($ch);
if ($return_data) return $return_data;
else return "empty";
my curl_getinfo return :
[content_type] => text/html;charset=UTF-8
[http_code] => 500
[header_size] => 543
[request_size] => 164
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.269685
[namelookup_time] => 0.002741
[connect_time] => 0.024587
[pretransfer_time] => 0.157608
[size_upload] => 636
[size_download] => 0
[speed_download] => 0
[speed_upload] => 2364
[download_content_length] => -1
[upload_content_length] => 636
[starttransfer_time] => 0.157693
[redirect_time] => 0
I fix my problem, thanks to insomnia who can generate PHP code for cURL,
I put the answer in case someone need it
$curl = curl_init();
curl_setopt_array($curl, [
CURLOPT_URL => $this->url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => json_encode($body),
CURLOPT_HTTPHEADER => [
"Accept: application/json",
"Accept-Charset: UTF-8",
"Content-Type: application/json"
],
]);
$response = json_decode(curl_exec($curl));

Tried to made an POST request using Guzzle

Im so new using API in my project, so i tried to make an POST request using Guzzle in Laravel, but i really dont know to do it, i've been seraching on the internet how to but i can't find the answer, here's what i've been tried:
$headers = [
'Content-Type' => 'application/json',
'signature' => '73ceef837b9be3cf098eca4a4697bd6a36718b64b0cf407c4324415941ff9780',
'va' => '0000002298436631',
'timestamp', '20191209155701'
];
$body = '{
"name": "Dudy",
"phone": "082298436631",
"email": "muhammadmaududy4#gmail.com",
"amount": "10000",
"notifyUrl": "https://mywebsite.com",
"expired": "24",
"expiredType": "hours",
"comments": "Catatan",
"referenceId": "1",
"paymentMethod": "qris",
"paymentChannel": "qris"
}';
$request = new Request('POST', 'https://sandbox.ipaymu.com/api/v2/payment/direct',
$headers, $body);
And i tried to do it on postman and its work perfectly, here's the setting on my postman:
You can use Laravel's Http Facade to deal with requests (easier to test later, if needed):
$headers = [];
$body = [];
$request = Http::send(
method: 'POST',
url: 'https://your.url/',
options: [
'headers' => $headers,
'form_params' => $body,
]
);
$response = Http::withHeaders([
'X-First' => 'foo',
'X-Second' => 'bar'
])->post('http://example.com/users', [
'name' => 'Taylor',
]);
I show a example code on top. I provide your code below.
$response = Http::withHeaders ([
'Content-Type' => 'application/json',
'signature' => '73ceef837b9be3cf098eca4a4697bd6a36718b64b0cf407c4324415941ff9780',
'va' => '0000002298436631',
'timestamp', '20191209155701'
])->post('https://sandbox.ipaymu.com/api/v2/payment/direct',[
"name": "Dudy",
"phone": "082298436631",
"email": "muhammadmaududy4#gmail.com",
"amount": "10000",
"notifyUrl": "https://mywebsite.com",
"expired": "24",
"expiredType": "hours",
"comments": "Catatan",
"referenceId": "1",
"paymentMethod": "qris",
"paymentChannel": "qris"
]);
Try this
$client = new \GuzzleHttp\Client();
$response = $client->post(
'https://sandbox.ipaymu.com/api/v2/payment/direct',
[
'form_params' => [
"name" => "Dudy",
"phone" => "082298436631",
"email" => "muhammadmaududy4#gmail.com",
"amount" => "10000",
"notifyUrl" => "https://mywebsite.com",
"expired" => "24",
"expiredType" => "hours",
"comments" => "Catatan",
"referenceId" => "1",
"paymentMethod" => "qris",
"paymentChannel" => "qris"
]
],
[
'headers' => [
'Content-Type' => 'application/json',
'signature' => '73ceef837b9be3cf098eca4a4697bd6a36718b64b0cf407c4324415941ff9780',
'va' => '0000002298436631',
'timestamp', '20191209155701'
]
]
);
dd($response) //response data

Laravel 8 - Get value from api json response

I am trying to get the value from 'title' in an API JSON response in Laravel 8.
{
"kind": "search#companies",
"page_number": 1,
"items_per_page": 20,
"total_results": 1,
"items": [
{
"kind": "searchresults#company",
"links": {
"self": "/company/13082103"
},
"title": "BITPIX LIMITED",
}
],
"start_index": 0
}
The call is setup and it works as expected but I am unsure how to get the variable.
The call will only ever return 1 response within items[] as the search is based on id.
Here is my method
$company = Auth::user()->companies()->first();
$apiKey = env("COMPANIES_HOUSE");
$query = $request->validate([
'query' => 'required',
]);
$response = Http::withHeaders([
'Authorization' => 'Basic ' . $apiKey,
'Connection' => 'keep-alive',
'Accept-Encoding' => 'gzip, deflate, br',
'Accept' => '*/*'
])->get('https://api.companieshouse.gov.uk/search/companies?q=' . $query['query']);
$data = json_decode($response->body());
//dd($data);
$provider = $company->providers()->create([
'name' => $data['items'][0]->title
]);
which produces this error:
Cannot use object of type stdClass as array relating to the ->create line
Many thanks for any help you can provide.
you have to decode the json you have recieved then extract the data you want:
$value = '{ "kind": "search#companies", "page_number": 1, "items_per_page": 20, "total_results": 1, "items": [{ "kind": "searchresults#company", "links": { "self": "/company/13082103" }, "title": "BITPIX LIMITED" }], "start_index": 0 }';
$decodeValue = json_decode($value);
$firstTitle = $decodeValue->items[0]->title; // first title is BITPIX LIMITED

Guzzle POST request always returns 400 Bad Request

I have been trying to make a simple POST request to an endpoint with a payload using Guzzle but I always get 400 Bad Request returned.
I can make the same request in Postman and it works. Also, If I make the request using cURL it works.
Can anyone tell from my code what I am doing wrong?
Here's the original cURL request:
curl "https://endpoint.com/" \
-H "Authorization: ApiKey pp_test_*********" \
--data '{
"shipping_address": {
"recipient_name": "Deon Botha",
"address_line_1": "Eastcastle House",
"address_line_2": "27-28 Eastcastle Street",
"city": "London",
"county_state": "Greater London",
"postcode": "W1W 8DH",
"country_code": "GBR"
},
"customer_email": "email£example.com",
"customer_phone": "123455677",
"customer_payment": {
"amount": 29.99,
"currency": "USD"
},
"jobs": [{
"assets": ["http://psps.s3.amazonaws.com/sdk_static/1.jpg"],
"template_id": "i6_case"
}, {
"assets": ["http://psps.s3.amazonaws.com/sdk_static/2.jpg"],
"template_id": "a1_poster"
}]
}'
And the Postman PHPHttp Request which works too.
<?php
$request = new HttpRequest();
$request->setUrl('https://endpoint.com/');
$request->setMethod(HTTP_METH_POST);
$request->setHeaders(array(
'cache-control' => 'no-cache',
'Connection' => 'keep-alive',
'Content-Length' => '678',
'Accept-Encoding' => 'gzip, deflate',
'Host' => 'api.kite.ly',
'Cache-Control' => 'no-cache',
'Accept' => '*/*',
'User-Agent' => 'PostmanRuntime/7.19.0',
'Content-Type' => 'text/plain',
'Authorization' => 'ApiKey pk_test_*******'
));
$request->setBody(' {
"shipping_address": {
"recipient_name": "Deon Botha",
"address_line_1": "Eastcastle House",
"address_line_2": "27-28 Eastcastle Street",
"city": "London",
"county_state": "Greater London",
"postcode": "W1W 8DH",
"country_code": "GBR"
},
"customer_email": "email#example.com",
"customer_phone": "12345667",
"customer_payment": {
"amount": 29.99,
"currency": "USD"
},
"jobs": [{
"assets": ["http://psps.s3.amazonaws.com/sdk_static/1.jpg"],
"template_id": "i6_case"
}, {
"assets": ["http://psps.s3.amazonaws.com/sdk_static/2.jpg"],
"template_id": "a1_poster"
}]
}');
try {
$response = $request->send();
echo $response->getBody();
} catch (HttpException $ex) {
echo $ex;
}
But when I try to send make the same request with Guzzle it fails with 400 Bad Request and I can't understand why.
$data = [
"shipping_address" => [
"recipient_name" => "Deon Botha",
"address_line_1" => "Eastcastle House",
"address_line_2" => "27-28 Eastcastle Street",
"city" => "London",
"county_state" => "Greater London",
"postcode" => "W1W 8DH",
"country_code" => "GBR"
],
"customer_email" => "example#email.com",
"customer_phone" => "+44 (0)784297 1234",
"customer_payment" => [
"amount" => 29.99,
"currency" => "USD"
],
"jobs" =>
[
"assets" => ["http://psps.s3.amazonaws.com/sdk_static/1.jpg"],
"template_id" => "i6_case"
]
];
$options = json_encode($data);
$response = $client->request('POST', config('services.endpoint.com'),
['headers' => ["Authorization" => config('services.endpoint.com.public_key'),
'Content-Type' => "application/json"], $options]);
If anyone can help me to debug this I'd be really grateful.
If you're using Guzzle 6 (and you probably should be), you're actually constructing in a more complex way than you need to, such that the endpoint is not receiving the expected JSON. Try this instead:
$client = new Client([
'base_uri' => 'https://my.endpoint.com/api',
'headers' => [
'Accept' => 'application/json',
...other headers...
]
]);
$data = [...your big slab of data...];
$response = $client->post('/kitely/path', ['json' => $data]);
// a string containing the results, which will depend on the endpoint
// the Accept header says we will accept json if it is available
// then we can use json_decode on the result
$result = $response->getBody()->getContents();
I'm using Unirest to make any sort of HTTP requests. I tried using Guzzle, but was facing the same issue as you are.
So what I did was install Unirest in my project, all the process are given in details in their documentation. This works perfectly fine for me.

How do i combine stopword and remove space filters in elasticsearch?

If i have to remove certain keywords and then remove all spaces in the string during index analysis, using :
'analysis' => array(
'filter' => array(
'whitespace_remove' => array(
'type' => 'pattern_replace',
'pattern' => ' ',
'replacement' => ''
),
'my_stop' => array(
'type' => 'stop',
'stopwords' => array('bad', 'horrible', 'useless')
),
'edge' => array(
'type' => 'edge_ngram',
'min_gram' => '1',
'max_gram' => '5'
)
),
and the analyzer with
'keyword_space_ngram' => array(
'type' => 'custom',
'tokenizer' => 'keyword',
'filter' => array(
'lowercase',
'my_stop',
'whitespace_remove',
'edge'
)
)
How do i ensure that i apply the filters in this order, that is convert to lowercase, remove keywords , remove spaces and then perform ngram analysis?
You can remove stopwords and white_spaces with custom char_filter at index time:
{
"analysis": {
"char_filter": {
"whitespace_remove": {
"type": "pattern_replace",
"pattern": "\\s+",
"replacement": ""
},
"custom_stop_words_char_filter": {
"type": "mapping",
"mappings": [
"bad => ",
"horrible => ",
"useless => "
]
}
},
"analyzer": {
"custom_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": ["lowercase", "asciifolding"],
"char_filter": ["custom_stop_words_char_filter", "whitespace_remove"]
}
}
}
}
This will transform bad angry man to angryman, for example
For adding your edge_ngram filter just add edge at the end of your filter array
Note: your stop words will only be substituted if they are lowercase

Resources