I'm struggling to figure out how to include multiple indexes in a search using the Elastic low level client.
My understanding (right or wrong) is that I should be able to include multiple indexes by separation of commas, this doesn't work for me though. In the code example below, I find that the first index specified is still working and returning results, but the second one is ignored. Any ideas?
Appsettings.json file:
// System settings configured here for the WebApp. Applicable to all users.
"SystemSettings": {
// Sets the maximum number of distinct values returned by Elastic for a log property
"_distinctPropertyValuesLimit": 1000, // See LogPropertiesController.cs
// String for the list of Elastic Search indexes that are searched by default.
"indexesToSearch": "webapp-razor-*, systemconfig-api-*"
}
Query class:
_indexesToSearch = configuration.GetSection("SystemSettings").GetSection("indexesToSearch").Value;
var searchResponse = await _elasticLowLevelClient.SearchAsync<StringResponse>(_indexesToSearch, #"
{
""from"": """ + fromParameter + #""",
""size"": """ + rowsPerPage + #""",
""query"": {
""match"": {
""" + searchColumn + #""": {
""query"": """ + searchString + #"""
}
}
},
""sort"": [
{
""#timestamp"": {
""order"": ""desc""
}
}
]
}
");
Turns out that there must not be any spaces between the index names when multiple values are provided, see below:
Related
In my product index I have 60K records, I'm running a match query with from, size params. by default, ES support a 10k record search so I have increased it to 60k.
My issue is I'm already passing size param from API & I want to search in 60K records with the same pagination size param.
So how I can match 60k records with my pagination size param.
Here is my code:
const query = {
query: {
match: {
name: {
query: req.text
}
}
}
}
const { body: { hits } } = await esclient.search({
from: req.skip || 0,
size: req.offset || 50,
index: productsIndex,
type: productsType,
body: query,
});
here code with 60K size:
const query = {
query: {
match: {
name: {
query: req.text
}
}
},
size:60000
}
in my query, if I use size=60K, I'm getting 20 records (without that I'm getting 12 records) but I can't use pagination params.
If I get your question correctly,
size parameter is the number of hits/documents you want in response. It's not that elasticsearch will process your query only in first size number of documents. 10K is the maximum number of documents you can get in response. If you specify size more than 10K, elastic search will give you an error like this.
ES error
Without worring about the above problem, use from and size as the parameters for your pagination.
I am creating a watcher in elasticsearch that reports when we havent had new entry or events in the index for 10 minutes this is further split out by looking at the source field in the entry.
I am only getting the last 10 mins of the index and seeing which source is not present in the buckets.
to do this I am first creating a list of all the source types we receive then creating a list from the bucket keys returned. Then I want to compare the lists to see which one is missing to then pass this into the message.
I am getting a generic error for the for loop. Any feedback is helpful quite new to elastic and painless so could be something simple I've missed.
"transform": {
"script": {
"source": """String vMessage = 'Clickstream data has been loaded although there are no iovation records from the following source in the last 10 mins:
';if(ctx.payload.clickstream.hits.total > 0 && ctx.payload.iovation.aggregations.source.buckets.size() < 3) { source_list = ['wintech', 'login', 'clickstream']; source_array = new String[] for (source in ctx.payload.iovation.aggregations.source.buckets){ source_array.add(source.key); } for (key in source_list){ if (!source_array.contains(key){ vMessage += '<ul><li>' + key + '</li></ul>';} } }return [ 'message': vMessage ];""",
"lang": "painless"
}
},
So I figured it out after digging through more documentation.
I was declaring my lists incorrectly. To declare a list it needs to be in format as below.
List new_list = new ArrayList();
This solved my issue and now the transform script works as expected.
"""String vMessage = 'Clickstream data has been loaded although there are no iovation records from the following source in the last 10 mins:
';if(ctx.payload.clickstream.hits.total > 0 && ctx.payload.iovation.aggregations.source.buckets.size() < 3) { List source_list = new ArrayList(['wintech', 'login', 'clickstream']); List source_array = new ArrayList(); for (source in ctx.payload.iovation.aggregations.source.buckets){ source_array.add(source.key); } for (key in source_list){ if (!source_array.contains(key)){ vMessage += '<ul><li>' + key + '</li></ul>';} } }return [ 'message': vMessage ];""",
I want to limit the number of posts fetched on my index page. Currently, the number of pages is hard-coded into the GraphQL query.
query {
allMarkdownRemark(limit: 5, sort: { fields: [frontmatter___date], order: DESC }) {
totalCount
edges {
node {
...
}
}
}
}
I want to replace "5" with the value of a variable. String interpolation will not work with the graphql function, so I have to use another method.
Is there a way to achieve this and pass variables into a GraphQL query in GatsbyJS?
You can only pass variables to GraphQL query via context since string interpolation doesn't work in that way. In page query (rather than static queries) you can pass a variable using the context object as an argument of createPage API. So, you'll need to add this page creation to your gatsby-node.js and use something like:
const limit = 10;
page.forEach(({ node }, index) => {
createPage({
path: node.fields.slug,
component: path.resolve(`./src/pages/index.js`), // your index path
// values in the context object are passed in as variables to page queries
context: {
limit: limit,
},
})
})
Now, you have in your context object a limit value with all the required logic behind (now it's a simple number but you can add there some calculations). In your index.js:
query yourQuery($limit: String) {
allMarkdownRemark(limit: $limit, sort: { fields: [frontmatter___date], order: DESC }) {
totalCount
edges {
node {
...
}
}
}
}
I need to search across multiple indices using OIS(Object Initializer Syntax).
I have seen examples of executing search across multiple indices with Fluent DSL, but I still do not know how to execute an equivalent search with OIS.
Here is my OIS search(Only searching against one index) :
var searchResult =
await _client.LowLevel.SearchAsync<string>(ApplicationsIndexName, "application", new SearchRequest()
{
From = (query.PageSize * query.PageNumber) - query.PageSize,
Size = query.PageSize,
Query = GetQuery(query),
Aggregations = GetAggregations()
});
Which modifications can be done, so I can search across multiple indices?
After some research, I found out how to search across multiple indices:
var searchResult =
await _client.LowLevel.SearchAsync<string>(new SearchRequest()
{
IndicesBoost = new Dictionary<IndexName, double>
{
{ "applications", 1.4 },
{ "attachments", 1.4 }
},
From = (query.PageSize * query.PageNumber) - query.PageSize,
Size = query.PageSize,
Query = GetQuery(query),
Aggregations = GetAggregations()
});
I'm trying to insert an alert in elasticsearch from bosun but I don't know how to fill the variable $timestamp (Have a look at my example) with the present time. Can I use functions in bosun.conf? I'd like something like now().
Can anybody help me, please?
This is an extract of an example configuration:
macro m1
{
$timestamp = **???**
}
notification http_crit
{
macro = m1
post = http://xxxxxxx:9200/alerts/http/
body = {"#timestamp":$timestamp,"level":"critical","alert_name":"my_alert"}
next = http_crit
timeout = 1m
}
alert http
{
template = elastic
$testHTTP = lscount("logstash", "", "_type:stat_http,http_response:200", "1m", "5m", "")
$testAvgHTTP = avg($testHTTP)
crit = $testAvgHTTP < 100
critNotification = http_crit
}
We use .State.Touched.Format which was recently renamed to .Last.Time.Format in the master branch. The format string is a go time format, and you would have to get it to print the correct format that elastic is expecting.
template elastic {
subject = `Time: {{.State.Touched.Format "15:04:05UTC"}}`
}
//Changed on 2016 Feb 01 to
template elastic {
subject = `Time: {{.Last.Time.Format "15:04:05UTC"}}`
}
Which when rendered would look like:
Time: 01:30:13UTC