I am connecting to Elasticsearch cluster using Nest client and I'm getting this issue while adding Index records.
An i missing something?
Nest v 7.2.1
Elastic version 7.2.1
Exception
Failed to create DB Index for Alarm Event.Invalid NEST response built from a unsuccessful (401) low level call on HEAD: /
Audit trail of this API call
[1] BadResponse: Node: http://192.168.0.4:9200/ Exception: PipelineException Took: 00:00:00.0588591
var addresses = new[] {
new Uri("http://192.168.0.4:9200/"),
new Uri("http://192.168.0.5:9200/"),
new Uri("http://192.168.0.6:9200/"),
new Uri("http://192.168.0.7:9200/"),
};
connectionPool = new StaticConnectionPool(addresses);
connectionSettings = new ConnectionSettings(connectionPool).BasicAuthentication("testuser", "Something_123").DisablePing();
ElasticSearchClient = new ElasticClient(connectionSettings);
var indexes = GetAsxIndexes();
var response = ElasticSearchClient.Ping().DebugInformation;
Related
Im using Java High level rest client version 7.17.5. Im applying the same but it doesnt work. Please find the below code
SearchSourceBuilder metaDataSearch = new SearchSourceBuilder();
org.elasticsearch.action.search.SearchRequest metaSearch = new org.elasticsearch.action.search.SearchRequest().source(metaDataSearch).indices(indexName);
SearchSourceBuilder keywordExactRequestBuilder = new SearchSourceBuilder().size(10);
keywordExactRequestBuilder.query(QueryBuilders.matchAllQuery());
org.elasticsearch.action.search.SearchRequest keywordExactRequestSearch = new org.elasticsearch.action.search.SearchRequest().source(keywordExactRequestBuilder).indices(indexName);
MultiSearchRequest multiSearchRequest = new MultiSearchRequest()
.add(metaSearch)
.add(keywordExactRequestSearch);
MultiSearchResponse multiSearchResponse = null;
try {
multiSearchResponse = elasticClient.msearch(multiSearchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
e.printStackTrace();
}
Im facing the below exception
org.jboss.resteasy.spi.UnhandledException: ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception, reason=key [types] is not supported in the metadata section]]
.......
Caused by:
ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception, reason=key [types] is not supported in the metadata section]]
......
Im enabled API compatibility mode. Im using elasticsearch version 8.3.2
Thanks
I am trying to log into Elasticsearch in one of my ASP.NET Web API project using Serilog, but unfortunately, I can't find the logs in Kibana.
public class Logger
{
private readonly ILogger _localLogger;
public Logger()
{
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 0
};
_localLogger = new LoggerConfiguration()
.MinimumLevel.Information()
.WriteTo.File(HttpContext.Current.Server.MapPath("~/logs/log-.txt"), rollingInterval: RollingInterval.Day)
.WriteTo.Elasticsearch(options)
.CreateLogger();
}
public void LogError(string error)
{
_localLogger.Error(error);
}
public void LogInformation(string information)
{
_localLogger.Information(information);
}
}
I can see the logs in the file specified above, just not in Elasticsearch. So, I am wondering is there is any way I can debug why it failed to log into Elasticsearch? I am also open to using other logging framework to log into Elasticsearch.
*The credentials and url for Elasticsearch are valid as I have implemented this in my other AWS Lambda project (.net core).
To see exactly what went wrong, the easiest way is to write into console, and in case of ASP.NET project, it will be Debug.WriteLine. So the code to see what went wrong would be
Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));
ElasticsearchSinkOptions options = new ElasticsearchSinkOptions(new Uri("xxx"))
{
IndexFormat = "log-myservice-dev",
AutoRegisterTemplate = true,
ModifyConnectionSettings = (c) => c.BasicAuthentication("yyy", "zzz"),
NumberOfShards = 2,
NumberOfReplicas = 1,
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog,
MinimumLogEventLevel = Serilog.Events.LogEventLevel.Information
};
The following error message was retrieved from the output console.
Failed to create the template.
Elasticsearch.Net.ElasticsearchClientException: The request was
aborted: Could not create SSL/TLS secure channel.. Call: Status code
unknown from: HEAD /_template/serilog-events-template --->
System.Net.WebException: The request was aborted: Could not create
SSL/TLS secure channel.
The issue is quite clear cut. Added the following in my logger class constructor helped with the issue.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Hope it helps others that encounter issue trying to use Serilog to log into Elasticsearch for .Net Framework.
We are migrating from Transport Client to High Level Rest Client. How do we reconcile the difference in the mappings field of GetIndexResponse object returned from Transport client vs Rest High Level Client?
For transport client, we use this code to obtain the index information:
GetIndexResponse response = client.get()
.admin()
.indices()
.prepareGetIndex()
.setFeatures(GetIndexRequest.Feature.MAPPINGS, GetIndexRequest.Feature.SETTINGS)
.setIndices(indices)
.get();
The response mappings field is a ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>>
But for rest high level client, the mappings field is just a Map<String, MappingMetadata>.
Here is the code for High Level Rest Client:
GetIndexRequest request = new GetIndexRequest(indices);
request.addFeatures(GetIndexRequest.Feature.MAPPINGS, GetIndexRequest.Feature.SETTINGS);
try {
GetIndexResponse response = esClient.indices().get(request, RequestOptions.DEFAULT);
return
} catch (IOException ex) {
throw new SafeRuntimeException(ex);
}
}
These are the response object classes:
Transport Client: org.elasticsearch.action.admin.indices.get.GetIndexResponse
Rest High Level Client: org.elasticsearch.client.indices.GetIndexResponse
for mappings use the GetMappings request:
ImmutableOpenMap<String, ?> mappings = esclient.admin().indices().getMappings(new GetMappingsRequest()).actionGet().getMappings();
and you can also specify the indices you want.
EDIT: ah i see, you mean this:
client.indices().getMapping(new GetMappingsRequest().indices("index_name1", "index_name2"), RequestOptions.DEFAULT);
client.indices().getSettings(new GetSettingsRequest().indices("index_name1", "index_name2"), RequestOptions.DEFAULT);
when "client" is an instance of the High level REST client.
Below is the code for serilog elastic search.It's working perfectly when elastic service is running but fails when the server is not available
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200/")) // humio elastic bulk endpoint
{
AutoRegisterTemplate = true,
OverwriteTemplate = true,
DetectElasticsearchVersion = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
NumberOfReplicas = 1,
NumberOfShards = 2,
RegisterTemplateFailure = RegisterTemplateRecovery.FailSink,
FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
EmitEventFailureHandling.WriteToFailureSink |
EmitEventFailureHandling.RaiseCallback,
//FailureSink = new FileSink("./fail-{Date}.txt", new JsonFormatter(), null, null)
//ModifyConnectionSettings = x => x.BasicAuthentication(username: "", password: "88WS28JEuB0G2WC3cYufdgTINAxOGQvliBrqH5Vqutjb") // password is ingest token from humio
})
exception is occuring when elastic server is missing
Elasticsearch.Net.ElasticsearchClientException: 'No connection could be made because the target machine actively refused it.. Call: Status code unknown from: GET /_cat/nodes?h=v'
HttpRequestException: No connection could be made because the target machine actively refused it.
SocketException: No connection could be made because the target machine actively refused it.
This exception was originally thrown at this call stack:
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
System.Net.Http.ConnectHelper.ConnectAsync(string, int, System.Threading.CancellationToken)
If elasticsearch runs on single mode, I can easily establish the RestHighLevel connection with this line of code:
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("localhost", 9200, "http"),
new HttpHost("localhost", 9201, "http")));
But if my elastic cluster has 3 machines, e.g., "host1", "host2", "host3", how to create the rest high level client in cluster mode ?
Thanks
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("host1", 9200, "http"),
new HttpHost("host2", 9200, "http"),
new HttpHost("host2", 9200, "http")
)
);
As the doc it looks like you were referencing states, RestClient.builder accepts an array of HttpHosts to connect to. The client (which under the hood is the ES low-level REST client) will round-robin requests to these hosts. See also the Javadoc.
As per the Elasticsearch docs you can pass multiple Elasticsearch hosts in RestClient.builder().
The better solution is to load the Elasticsearch hosts from configuration(application.conf in case of Scala-based application) instead of hardcoding it in the codebase.
Here is the Scala-based solution using Java Varargs(:_*).
application.conf
es_hosts = ["x.x.x.x","x.x.x.x","x.x.x.x"] // You can even use service-name/service-discovery
es_port = 9200
es_scheme = "http"
Code snippet
import collection.JavaConverters._
import com.typesafe.config.ConfigFactory
import org.apache.http.HttpHost
import org.elasticsearch.client.{RestClient, RestHighLevelClient}
val config = ConfigFactory.load()
val port = config.getInt(ES_PORT)
val scheme = config.getString(ES_SCHEME)
val es_hosts = config.getStringList(ES_HOSTS).asScala
val httpHosts = es_hosts.map(host => new HttpHost(host, port, scheme))
val low_level_client = RestClient.builder(httpHosts:_*)
val high_level_client: RestHighLevelClient = new RestHighLevelClient(low_level_client)
To create High level REST client using multiple hosts, you can do something like following:
String[] esHosts = new String[]{"node1-example.com:9200", "node2-example.com:9200",
"node3-example.com:9200"};
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(esHosts)
.build();
RestHighLevelClient restClient = RestClients.create(clientConfiguration).rest();
// Hostnames used for building client can be verified as following
List<Node> nodes = restClient.getLowLevelClient().getNodes();
nodes.forEach(node -> System.out.println(node.toString()));
References:
Docs for High Level REST Client
Source code for ClientConfigurationBuilder
I am creating my elastic REST client with below steps:
RestHighLevelClient client=null;
List<HttpHost> hostList = new ArrayList<>();
for (String host : hosts) {
String[] hostDetails = host.split("\\:");hostList.add(new
HttpHost(hostDetails[0],Integer.parseInt(hostDetails[1]),https));
}
try(RestHighLevelClient client1 = new RestHighLevelClient(
RestClient.builder(hostList.toArray(new
HttpHost[hostList.size()]))
.setHttpClientConfigCallback(
httpClientBuilder ->
// to do this only if auth is enabled
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))))
{
client = client1;
}
} catch (IOException e) {
log.error("exception occurred while setting elastic client");
}