According to http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html# the new EMR release, 4.0.0, allows for a configuration json file to configure components. Also looking at http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html, I have the json structured:
[
{
"Classification": "mapred-site",
"Properties": {
"mapreduce.map.memory.mb": "4096",
"mapreduce.map.java.opts": "-Xmx4096M"
}
},
{
"Classification": "hadoop-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"HADOOP_CLIENT_OPTS": "-Xmx4096M"
}
}
]
}
]
The Hadoop env property works, but the mapred-site properties are not reflected when a create an EMR cluster. I can always set these properties from Hive, but any ideas how to make it work with configuration file?
I experienced a similar issue, but with the "hadoop-env" classification.
From this post: https://github.com/aws/aws-cli/issues/1502
I found that making the keywords (Classification, Properties, ...) lower case solved the problem.
Note: I am using AWS console, not CLI.
Related
I find the documentation at - https://devcenter.heroku.com/articles/app-json-schema#scripts - is poor. It doesn't list all of the different events which can be hooked into to run scripts. I want to run a script for when the app is first created and another with each release. The first is objective is already achieved with a script with runs on the "postdeploy" event. I can't see anything though to help with the second. If using a procfile I know I just specify the "release" process type but I'm trying to do this with an app.json file so I believe the procfile is not relevant.
Does anybody know how I can do what I want? Even better the whereabouts of this explained on Heroku.
At the moment I have this for my app.json -
{
"name": "accounts",
"success_url": "/users/signup",
"addons": [
{
"plan": "heroku-postgresql:hobby-dev"
}
],
"env": {
"NEW_USERS_ARE_SUPERUSERS": "0"
},
"formation": {
"web": {
"quantity": 1,
"size": "free",
"command": "gunicorn proj.wsgi"
}
},
"buildpacks": [
{
"url": "heroku/python"
}
],
"scripts": {
"postdeploy": "psql -c \"create extension pg_trgm\" $DATABASE_URL && ./bin/setup.sh"
}
}
Can I just add a "release" key to the "scripts" dictionary? I saw somewhere somebody had "heroky-postbuild" but that didn't seem to work when I tried it.
My objective is to write a script that promotes a "stage" application in our Heroku pipeline to production, but only if there are any changes to promote.
I can promote without issues by using:
// POST /pipeline-promotions
{
"pipeline": {
"id": "<pipeline-id>"
},
"source": {
"app": {
"id": "<stage-app-id>"
}
},
"targets": [
{
"app": {
"id": "<production-id>"
}
}
]
}
My issues is that if I execute this, without any changes are present, the release actions are still started on the production application.
In other words, how can I determine if any changes are available for promotion - just like heroku does in their GUI?
Thanks to Heroku support, I managed to figure this out.
It can be done by comparing the slug id's when looking at pipelines/<pipeline-id>/latest-releases.
If the slug-ids are the same, there are no changes to promote.
I have indexes around 250 GB all-together in 3 host i.e. 750 GB data in ELK cluster.
So how can I rotate ELK logs to keep three months data in my ELK cluster and older logs should be pushed some other place.
You could create your index using "indexname-%{+YYYY.MM}" naming format. This will create a distinct index every month.
You could then filter this index, based on timestamp, using a plugin like curator.
The curator could help you set up a CRON job to purge those older indexes or back them up on some s3 repository.
Reference - Backup or Restore using curator
Moreover, you could even restore these backup indexes whenever needed directly from s3 repo for historical analysis.
Answer by dexter_ is correct, but as the answer is old, a better answer would be:
version 7.x of elastic stack provides a index life cycle management policies, which can be easily managed with kibana GUI and is native to elk stack.
PS, you still have to manage the indices like "indexname-%{+YYYY.MM}" as suggested dexter_
elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html
It took me a while to figure out exact syntax and rules, so I'll post the final policy I used to remove old indexes (it's based on the example from https://aws.amazon.com/blogs/big-data/automating-index-state-management-for-amazon-opensearch-service-successor-to-amazon-elasticsearch-service/):
{
"policy": {
"description": "Removes old indexes",
"default_state": "active",
"states": [
{
"name": "active",
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "14d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": {
"index_patterns": [
"mylogs-*"
]
}
}
}
It will automatically apply the policy for any new mylogs-* indexes, but you'll need to apply it manually for existing ones (under "Index Management" -> "Indices").
Using the Serilog.Settings.AppSettings project I am having difficulty setting a filter to exclude certain namespaces from a configured sink.
In code I would do something like this:
[...].Filter.ByExcluding(Matching.FromSource<MyNameSpace>())
However I don't seem able to do it using the app settings.
Is this supported and if so how can I achieve this using configuration?
Thanks
Vincent
Make sure you have the package.
Install-Package Serilog.Filters.Expressions
Then follow the example here: https://github.com/serilog/serilog-filters-expressions and https://github.com/serilog/serilog-settings-configuration/blob/dev/sample/Sample/appsettings.json#L64
"Using": ["Serilog.Settings.Configuration"],
"Filter": [
{
"Name": "ByIncludingOnly",
"Args": {
"expression": "SourceContext = 'MyNameSpace'"
}
}
]
I'm using Serverless to deploy a couple of functions written in C# to AWS.
While deploying a message duplicated mapping key in "...\serverless.yml" is thrown.
Separately, both functions get deployed but when put together the said error message is shown.
What am I missing?
{
"service": "serverlessquick",
"provider": {
"name": "aws",
"runtime": "nodejs4.3"
},
"functions": {
"hello": {
"handler": "handler.hello",
"events": [
{
"http": {
"path": "hello",
"method": "get"
}
}
]
}
}
}
Make your yaml code into json this way any mistakes are obvious, and will be more likely to be picked up by the parser. If anyone comes across this, use the referenced config above.
I got a similar error but in my case it was indentation problem. Go through your YML file and see if the changes you have made have the correct identation. A json parser as mention by others can help narrow down where in the YML file the problem exists