Find elasticsearch indices with no aliases - elasticsearch

I have an elasticsearch cluster whose indices I need to investigate. Specifically, I want to find all indices that do not have aliases.
Is this even possible? If so, how?
(Using sense notation for this question.)
I know I can get all indices with their aliases, even if the alias field is empty:
GET _aliases
And I know I can get all indices that have aliases:
GET /*/_alias/*
But can I get all indices without aliases?
Clearly, I can just get all indices, and then use some tool like awk or whatever to do the work for me, but my naïve guess is that it's most efficient for elasticsearch to do all this work at once.

Short answer: There is no easy api for you to find out which indices have not been linked to an alias.
Long answer: You could try the '_cluster/state' endpoint like this:
GET _cluster/state?filter_path=metadata.indices.test.aliases
Where test is the name of your index. It gives the following result:
{
"metadata": {
"indices": {
"test": {
"aliases": []
}
}
}
}
Now if I try it with an index that is actually linked:
GET _cluster/state?filter_path=metadata.indices.test_with_alias.aliases
I get the following result:
{
"metadata": {
"indices": {
"test_with_alias": {
"aliases": [
"new_alias"
]
}
}
}
}
It's not the prettiest way to do this, but it's possible :)
Hope this helps!

This shell script will help you
#!/bin/bash
ES="https://localhost:9316"
INDICES=`curl -XGET ${ES}/_cat/indices -s --insecure | awk '{print $3}'`
META=`curl -XGET "${ES}/_cluster/state?filter_path=metadata.indices.*.aliases" -s --insecure`
echo ${#META}
for INDEX in $INDICES
do
if [[ ${INDEX} != \.* ]]
then
RES=`jq ".metadata.indices.${INDEX}.aliases" <<< ${META}`
if [[ ${#RES} == 2 ]]
then
RES="NOALIAS"
fi
echo ${INDEX} $RES
fi
done

You can determine this as follows via the Java REST High Level Client:
val indexResponse = client.indices().get(GetIndexRequest("*"), RequestOptions.DEFAULT)
val aliases = indexResponse.aliases
aliases is of type Map<String, List<AliasMetadata>>. Every Map entry with a empty list (without alias metadata) is a index without an alias.
With this in mind, here's what you need to do to get all the indices without aliases.
val indicesWithoutAliases = indexResponse.aliases.filterValues { it.isEmpty() }

Related

How to update value in key using jq inside a file in shell script?

I have file like this(exact format):
{ "test": "10" }
{ "games": "30" }
{ "REPOSITORY": "5"}
I want to update the value of key. For example games to 66. So the new file would be:
{ "test": "10" }
{ "games": "66" }
{ "REPOSITORY":"5"}
I used jq with combination of a temporary file
tmp=$(mktemp);
jq '.games = "66"' pair.json > "$tmp" && mv "$tmp" change.json
but got this kind of output.
{
"test": "10",
"games": "66"
}
{
"games": "66"
}
{
"REPOSITORY": "5",
"games": "66"
}
where games is written all 3 whereas I just want it be updated only at its position.
Furthermore i want the value to be updated using bash variable but that I can try to figure out using --arg after I resolve this issue. Pls assist here.
Your input is a stream of objects. As such your filter expression, .games = "66" asserts that the record games is present in each of the object present in the stream.
To update the specific object alone, select it first and assign the required value or update it.
jq -c 'select(has("games")).games = "66"'
To answer OP's clarification on how to do the same with variables, do
jq -c --arg key "games" 'select(has($key))[$key] = "555"'
A simple solution is using select with assignment. select will return the special value empty if the condition does not match and the input value if it does match.
select(.games).games = "66"
jq -c to generate compact format, i.e. one JSON object/document per line.
To handle arbitrary names, change the syntax slightly:
select(.["games"])["games"] = "66"
and used with arguments passed via CLI:
jq -r --arg key "$var" 'select(.[$key])[$key] = "66"'

Kibana visualize use wild card in search bar

Is it possible to use wild card in Kibana visualize search bar.
Tried to use it like below, but did not work.
operation: "Revers" NOT file:"*Test.Revers"
This returns 2 because there are two Revers terms ("Revers", "/test/count/Test.Revers" ) even though only one data entry is in the stats data.
The following also returns the same value as 2.
operation: "Revers"
Stat data sample is as below.
"_source": {
"status": 0,
"trstime": 1819,
"username": "test",
"operation": "Revers",
"file": "/test/count/Test.Revers"
}
I have tested it in ES 7.10 as you not mentioned ES version.
Answer to your question is YES, you can use wildcrad in Kibana visualize search bar but value should be without double quotes. Because if you give value in doble quotes it will consider as text and search for it.
You can try below query and it will give you your expected output:
operation: Revers AND NOT file.keyword: *Test.Revers
The result given for the below query as 1 without double quotes.
operation: Revers AND NOT file: *Test.Revers

C like preprocessor macros in Bash

I'm not really sure a good name for this question so please rename it if you can think of a better one.
In Bash I have a function that I am using to store certain functions. Consider:
menv_funcs=()
function menv_function {
menv_funcs+=($1)
}
I am then using it in this manner:
menv_function fetch
function fetch {
...
}
I would like to use it like this though:
menv_function fetch {
...
}
Essentially I'm looking for something like the preprocessor macros in C would do but I have been unable to find a way. Any ideas?
As far as I'm aware, you can't directly achieve this. However, I can think of two solutions that may be of interest to you.
First of all, you could just declare the functions as usual, and then obtain the list of declared functions through declare -F. This could be done like:
function fetch {
:
}
menv_funcs=()
while IFS=$"\n" read l; do
menv_funcs+=${l#declare -f }
done < <(declare -F)
Which will cause menv_funcs[#] to list all the functions declared at the point of calling the snippet. Of course, this may catch unwanted functions as well.
To avoid this, you may add some prefix to function names and filter the list:
function menv_fetch {
:
}
menv_funcs=()
while IFS=$"\n" read l; do
if [[ ${l} == 'declare -f menv_'* ]]; then
menv_funcs+=${l#declare -f menv_}
fi
done < <(declare -F)
And if you really want to achieve something like macros, you may try to play with eval:
menv_funcs=()
function menv_function {
local name=${1}
local body=${2}
menv_funcs+=( ${name} )
eval "function ${name} ${body}"
}
menv_function fetch "{
:
}"
But note that you will actually need to quote the whole function body and escape everything appropriately.

Bash: Reading from array

I'm writing a script in bash, and I'm trying to read from an array. When I iterate through the array with the code below:
for item in "${!functionDict[#]}"
do
echo "{ $item : ${functionDict[$item]} }" >> array.txt
done
it outputs (in "array.txt"):
{ month_start_date($year_selected, $month_selected) : return $date; }
{ logWarning($message) : return logEvent($message, $c_event_warning); }
{ daysPastLastQuarterX($curYear, $curMonth, $curDay, $selected_year, $selected_quarter, $nDays) : return false;:return false;:return false;:return false;:return true;:return $delta > $nDays; }
{ setExcelLabelCell($sheet, $cell, $label, $width) : }
{ asCurrencyString($value) : return formatCurrency($value); }
{ getNumericMonthName($m) : return $numericMonth; }
{ normalize_for_PDF(&$text) : }
However, I'm having trouble querying individual elements from the array.
I've tried:
string='month_start_date($year_selected, $month_selected)'
echo "test_output: ${functionDict[$string]}"
but I get
test_output: <blank>
I've also tried inserting some RegEx wildcards, in case there is some whitespace around the key.
echo 'size of array: '"${#functionDict[#]}"
echo "TEST: functDict[logWarning] = ${functionDict[.*'logWarning($message)'.*]}"
I get
size of array: 157 //I didn't copy/paste all the elements in the array in this post
TEST: functDict[logWarning] = <blank>
Alas, I'm stuck. The content that I'm trying to get back are the "return _" items, or just a "blank" for the keys that don't have any "return" items.
Credits to answer goes to #gniourf_gniourf. (See comments)
There was an extra leading space in all of the keys, which I hadn't included in my test queries.
(Providing an answer here so that people know this question has been solved. Hopefully this is okay practice on SO)

CouchDB Filtered Replication

Trying out filters for replication, I stumbled upon a problem.
While my filter is working as an entry in the _replicator database, I doesn't when using cURL.
The filter in the design document is:
{
"_id": "_design/partial",
"filters": {
"mobile": "function(doc, req) {
if (doc._attachments) {
var result = new Boolean(true);
for (attachment in doc._attachments) {
if (attachment.content_type == 'image/jpeg') {
return true;
}
if (doc._attachments.length > 1024) {
result = false;
}
}
return result;
} else {
return true;
}
}"
}
}
The cURL line:
curl -X POST http://admin:pass#192.168.178.13:5985/_replicate -d '{\"source\":\"http://admin:pass#192.168.2:5984/docs2\",\"target\":\"docs2_partial\",\"filter\":\"partial/mobile\",\"create_target\":true}' -H "Content-Type: application/json"
I created _design/partial document on both target and source, but all documents are being replicated. Even the one with an attached binary bigger than 1 MB.
Any help is appreciated!
The cURL reply is:
{"ok":true,"session_id":"833ff96d21278a24532d116f57c45f31","source_last_seq":32,"replication_id_version":2,"history":[{"session_id":"833ff96d21278a24532d116f57c45f31","start_time":"Wed, 17 Aug 2011 21:43:46 GMT","end_time":"Wed, 17 Aug 2011 21:44:22 GMT","start_last_seq":0,"end_last_seq":32,"recorded_seq":32,"missing_checked":0,"missing_found":28,"docs_read":28,"docs_written":28,"doc_write_failures":0}]}
Using either " instead of \" or " instead of ' the result is:
{"error":"bad_request","reason":"invalid UTF-8 JSON: [...]}
Now I think perhaps the logic of your filter function simply has a bug. Here is how I read your filter policy:
All docs that have no attachments pass
All docs that have an image/jpeg attachment pass
Docs with more than 1,024 attachments fail
In any other case, the docs pass
That sounds like perhaps an incorrect policy. Another way to restate this policy is "Docs with more than 1024 attachments fail, everything else passes." However since you wrote so much code, I suspect my summary is not the true policy.
Another quick note, on what looks like a bug. Given:
for (attachment in doc._attachments) { /* ... */ }
The attachment variable will be things like "index.html" or "me.jpeg", i.e. filenames. To get the attachment content-type, you need:
var type;
// This is WRONG
type = attachment.content_type; // type set to undefined
// This is RIGHT
type = doc._attachments[attachment].content_type; // type set to "text/html" etc.
To avoid this bug, you could change your code to make things more clear:
for (attachment_filename in doc._attachments) { /* ... */ }
Next, doc._attachments.length will tell you the number of attachments in the document, not for example the length of the current attachment. It is odd that you test for that inside the loop, because the expression will never change. Are you trying to test for attachment size instead?
What is the output from curl (i.e. from CouchDB)?
From your example, my first guess is that you have a quoting error. Inside single-quotes, you do not need to escape the double-quotes. Try removing all those backslashes. What happens?
If you are on Windows, the single quote is not valid in the shell. In that case, keep the backslashes and just change the single-quote to a double-quote.

Resources