I implemented to setup elasticsearch Backup and Restore From Azure Blob Storage but When i am running the below query I am getting the response as "acknowledged" but when i check the storage account, I dont see the backups
PUT _snapshot/azure-kibana
{
"type": "azure",
"settings": {
"container": "elasticsearch-snapshots",
"chunk_size": "32MB",
"compress": true
}
}
output
#! Deprecation: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
{
"acknowledged" : true
}
GET /_snapshot/_all
#! Deprecation: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
{
"my_backup" : {
"type" : "fs",
"settings" : {
"location" : "/home/ubuntu/mount/backup"
}
},
"azure-kibana" : {
"type" : "azure",
"settings" : {
"container" : "elasticsearch-snapshots",
"compress" : "true",
"chunk_size" : "32MB"
}
},
"azure-kibana2" : {
"type" : "azure",
"settings" : {
"container" : "test",
"compress" : "true"
}
}
}
I reinstalled the plugin and did the setup again, now my snapshots are visible in the storage account
Related
I wand to add big inline list of rules for stem_override filter (see https://www.elastic.co/guide/en/elasticsearch/reference/5.6/analysis-stemmer-override-tokenfilter.html).
My index settings looks like this:
{
"settings": {
"analysis" : {
"analyzer" : {
"my_analyzer" : {
"tokenizer" : "standard",
"filter" : ["lowercase", "custom_stems", "porter_stem"]
}
},
"filter" : {
"custom_stems" : {
"type" : "stemmer_override",
"rules" : [
"running => run",
"stemmer => stemmer"
... //200 000 rules
]
}
}
}
}
When I send this request to ES it runs so long that I don't receive any response and no index eventually created.
There is solution with rules stored in file in ES config folder. Then filter configuration is:
"filter" : {
"custom_stems" : {
"type" : "stemmer_override",
"rules_path" : "analysis/stemmer_override.txt"
}
}
In this case everything works fine. But this is not what I need. I will not have access to ES filesystem. I need to have possibility create new settings only via request.
Are there any solutions how to make ES process such huge requests(around 4Mb) quickly?
Thanks
We have a few external applications in cloud (IBM Bluemix) which logs its application syslogs in the bluemix logmet service which internally uses the ELK stack.
Now on a periodic basis, we would like to download the logs from the cloud and upload it into a local Elastic/Kibana instance. This is because storing logs in cloud services incurs cost and additional cost if we want to search the same by Kibana. The local elastic instance can delete/flush old logs which we don't need.
The downloaded logs will look like this
{"instance_id_str":"0","source_id_str":"APP/PROC/WEB","app_name_str":"ABC","message":"Hello","type":"syslog","event_uuid":"474b78aa-6012-44f3-8692-09bd667c5822","origin_str":"rep","ALCH_TENANT_ID":"3213cd20-63cc-4592-b3ee-6a204769ce16","logmet_cluster":"topic3-elasticsearch_3","org_name_str":"123","#timestamp":"2017-09-29T02:30:15.598Z","message_type_str":"OUT","#version":"1","space_name_str":"prod","application_id_str":"3104b522-aba8-48e0-aef6-6291fc6f9250","ALCH_ACCOUNT_ID_str":"","org_id_str":"d728d5da-5346-4614-b092-e17be0f9b820","timestamp":"2017-09-29T02:30:15.598Z"}
{"instance_id_str":"0","source_id_str":"APP/PROC/WEB","app_name_str":"ABC","message":"EFG","type":"syslog","event_uuid":"d902dddb-afb7-4f55-b472-211f1d370837","origin_str":"rep","ALCH_TENANT_ID":"3213cd20-63cc-4592-b3ee-6a204769ce16","logmet_cluster":"topic3-elasticsearch_3","org_name_str":"123","#timestamp":"2017-09-29T02:30:28.636Z","message_type_str":"OUT","#version":"1","space_name_str":"prod","application_id_str":"dcd9f975-3be3-4451-a9db-6bed1d906ae8","ALCH_ACCOUNT_ID_str":"","org_id_str":"d728d5da-5346-4614-b092-e17be0f9b820","timestamp":"2017-09-29T02:30:28.636Z"}
I have created an index in our local elasticsearch as
curl -XPUT 'localhost:9200/commslog?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"logs" : {
"properties" : {
"instance_id_str" : { "type" : "text" },
"source_id_str" : { "type" : "text" },
"app_name_str" : { "type" : "text" },
"message" : { "type" : "text" },
"type" : { "type" : "text" },
"event_uuid" : { "type" : "text" },
"ALCH_TENANT_ID" : { "type" : "text" },
"logmet_cluster" : { "type" : "text" },
"org_name_str" : { "type" : "text" },
"#timestamp" : { "type" : "date" },
"message_type_str" : { "type" : "text" },
"#version" : { "type" : "text" },
"space_name_str" : { "type" : "text" },
"application_id_str" : { "type" : "text" },
"ALCH_ACCOUNT_ID_str" : { "type" : "text" },
"org_id_str" : { "type" : "text" },
"timestamp" : { "type" : "date" }
}
}
}
}'
Now to bulk upload the file, used the command
curl -XPOST -H 'Content-Type: application/x-ndjson' http://localhost:9200/commslog/logs/_bulk --data-binary '#commslogs.json'
The above command throws an error
Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]
The solution is to follow the rules for bulk upload as per
https://discuss.elastic.co/t/bulk-insert-file-having-many-json-entries-into-elasticsearch/46470/2
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
So i manually changed few of the log statements by adding action before every line
{ "index" : { "_index" : "commslog", "_type" : "logs" } }
This works!!.
Another option was to call the curl command, providing the _idex and _type in the path
curl -XPOST -H 'Content-Type: application/x-ndjson' http://localhost:9200/commslog/logs/_bulk --data-binary '#commslogs.json'
but without the action, this too throws the same error
The problem is we cannot do this for thousands of log records we get. Is there an option where once we download the log files from Bluemix and upload the files without adding the action.
NOTE We are not using logstash at the moment, but
is it possible to use logstash and just use grok to transform the
logs and add the necessary entries?
How can we bulk upload documents via Logstash?
Is logstash the ideal solution or we can just write a program to
transform and do that
Thanks
As #Alain Collins said, you should be able to use filebeat directly.
For logstash:
it should be possible to use logstash, but rather than using grok, you should use the json codec/filter, it would be much easier.
You can use the file input with logstash to process many files and wait for it to finish (to know when it's finished, use a file/stdout, possibly with the dot codec, and wait for it to stop writing).
Instead of just transforming the files with logstash, you should directly upload to elasticsearch (with the elasticsearch output).
As for your problem, I think it will be much easier to just use a small program to add the missing action line or use filebeat, unless you are experimented enough with logstash config to write and logstash config quicker than a program adding one line everywhere in the document.
I have added the index creation date in my index setting as
below
"settings" :{
"index" :{
"provided_name":"test",
"creation_Date":"1493750591836",
"number_of_shards" : "1",
"number_of_replicas" : "0"
}
}
But when i try to post the _template am getting error as below
"unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
Does it means the creation time setting is not available, could any please clarify. Am not able to find more details on this in
https://www.elastic.co/guide/en/elasticsearch/reference/1.4/indices-update-settings.html
The version used is 5.1
You're not allowed to set that setting, only read it. However, what you can do is to use the mappings._meta section for that in order to store your custom index creation date:
PUT my_index
{
"settings" :{
"index" :{
"number_of_shards" : "1",
"number_of_replicas" : "0"
}
},
"mappings": {
"test": {
"_meta": {
"creation_date":"1493750591836"
}
}
}
}
I use elasticsearch for logs, I don't want to use daily index to delete them with a cron job but with the TTL. I 've actived and set TTL with the value: 30s. I have a succesfull answer when I send this operation and I can see the TTL value(in milliseconds) when I do the mapping request.
All seems good but documents are not be deleted...
_mapping :
{
"logs" : {
"webservers" : {
"_ttl" : {
"default" : 30000
},
"properties" : {
#timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
}
}
}
I guess you just need to enable _ttl for your type, which is disabled by default. Have a look here.
{
"webservers" : {
"_ttl" : { "enabled" : true, "default" : "30s" }
}
}
I'm trying to upload my web app to an EC2 instance through the .NET SDK using the "Publish to AWS" feature. For my other web application, this worked perfectly! The only difference between the two that I can see is this web app has a connection string to an attached RDS instance that is already live.
When the publish does go through, the instance launches and seems to be running fine. Within about ten minutes, the instance rolls back and terminates itself. After unchecking the terminate on fail checkbox the instance won't terminate, however I'm not able to access my Default.aspx page. I haven't been able to find anything helpful in the logs. I'm thinking it must have something to do with the connection string and attached volume.
I added my security group to the RDS instance that I use to launch the instance, however it still returns the same error.
Publish Info:
Instance Size: Micro
SDK Version: 1.5.10
AMI: ami-10ec6520
Region: US West 2
Error:
WaitCondition timed out. Received 0 conditions when expecting 1.
Stack Template
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "This will launch a single EC2 instance and deploy your application to it. **WARNING** This template creates one or more Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.",
"Parameters" : {
"InstanceType" : {
"Type" : "String",
"Default" : "t1.micro",
"Description" : "EC2 instance type."
},
"KeyPair" : {
"Type" : "String",
"Description" : "EC2 Key Pair."
},
"SecurityGroup" : {
"Type" : "String",
"Description" : "EC2 Security Group"
},
"BucketName" : {
"Type" : "String",
"Description" : "[Hidden]S3 Bucket for deployment."
},
"ConfigFile" : {
"Type" : "String",
"Description" : "[Hidden]Deployment Configuration File."
},
"AmazonMachineImage" : {
"Type" : "String",
"Default" : "ami-10ec6520",
"Description" : "AMI to launch."
},
"UserData" : {
"Type" : "String",
"Description" : "[Hidden]Base64-Encoded user data."
}
},
"Resources" : {
"DeployedApplicationWaitHandle" : {
"Type" : "AWS::CloudFormation::WaitConditionHandle",
"Properties" : {
}
},
"DeployedApplicationWaitCondition" : {
"Type" : "AWS::CloudFormation::WaitCondition",
"DependsOn" : "Ec2Instance",
"Properties" : {
"Handle" : { "Ref" : "DeployedApplicationWaitHandle" },
"Timeout" : "900"
}
},
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : { "Ref" : "AmazonMachineImage" },
"KeyName" : { "Ref" : "KeyPair" },
"InstanceType" : { "Ref" : "InstanceType" },
"SecurityGroups" : [{ "Ref" : "SecurityGroup" }],
"UserData" : { "Fn::Base64" : {"Fn::Join" : [ "", ["[", { "Ref" : "UserData" }, "]", "[", { "Ref" : "DeployedApplicationWaitHandle" }, "]"] ]}}
}
}
},
"Outputs" : {
"URL" : {
"Description": "URL of the deployed application",
"Value" : { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }]]}
},
"Bucket" : {
"Description" : "The S3 Bucket where the Web Deploy archive and configuration file are uploaded",
"Value" : { "Ref" : "BucketName" }
},
"ConfigFile" : {
"Description" : "The deployment configuration for the application",
"Value" : { "Ref" : "ConfigFile" }
},
"VSToolkitDeployed" : {
"Description" : "A flag indicating that the stack was created via VSToolkit Deploy wizard",
"Value" : "True"
}
}
}
UPDATE 1/18/2013
I finally got an instance launched that let me RDP into it. Below is the error that stuck out.
Error:
Microsoft.Web.Deployment.DeploymentFatalException: The SQL provider cannot run because of a missing dependency. Please make sure that Microsoft SQL Server Management Objects (Version 10 or higher) is installed. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
Any help would be greatly appreciated! If there's any more information that would be helpful to provide, just let me know.
Turns out the answer was within my deployment settings in Visual Studio. Very amateur mistake on my part. In Project Properties => Package/Publish Web I needed to uncheck "Include all databases configured in Package/Publish SQL tab"
Since my database was already out and configured on RDS, all I needed to do was include the correct connection string in the Web.config file.
Thanks everyone for the help!