In a Logstash pipeline, I'm trying to verify if a nested object exists. If it doesn't, I want to create it and initialized its fields.
Here is my pipeline output:
output {
elasticsearch {
hosts => ["${ELASTICSEARCH_HOST:localhost:9200}"]
user => "${ELASTICSEARCH_USERNAME:elastic}"
password => "${ELASTICSEARCH_PASSWORD:password}"
index => "shape_idx"
action => "update"
document_id => "%{shapeId}"
script_type => "inline"
scripted_upsert => true
document_type => "shape"
script => '
if (ctx._source["realViews"]) {
ctx._source.realViews.all=0;
ctx._source.realViews.desktop=0;
ctx._source.realViews.phone=0;
ctx._source.realViews.tablet=0;
}
ctx._source.realViews.all = (ctx._source.realViews.all ?:0) + params.event.deviceCounter.all;
ctx._source.realViews.desktop = (ctx._source.realViews.desktop ?:0) + params.event.deviceCounter.desktop;
ctx._source.realViews.phone = (ctx._source.realViews.phone ?:0) + params.event.deviceCounter.phone;
ctx._source.realViews.tablet = (ctx._source.realViews.tablet ?:0) + params.event.deviceCounter.tablet;
'
}
}
Unfortunately, my pipeline fails silently, any idea ?
For the record, I'm using Logstashg 7.1.1 and Elasticsearch 5.6.16.
Mutations should happen in a mutate filter, just add a check there and add the field when needed within your filter.
Try to remind yourself that you have input / filter / output and make each part do what it is supposed to do.
Let the input just input data in a pipeline, maybe add a field if you really want to (useful if you use multiple inputs in the same pipeline running through the same filter).
Let the filter do whatever manipulation you want to do, this includes checking if fields exist and then acting upon it.
Let the output just output (if statements to have conditional outputs can be part of it, but no further processing).
I've created some custom product attributes with REST API, correctly populated them and assigned values to products.
Everything works fine in dev environment whereas I got difficulties in staging environment, even though values and settings are the same, even in the DB.
If, for example, I want to update the value of "is_filterable" option in an existing attribute from the backend, Magento says it saved the attribute correctly, but actually no changes are made.
I've tried to update with a PUT call to REST API with correct values, but I'm able to update only the attribute labels,
This is the json i'm passing to "products/attributes/my_attr"
{"attribute":{"attribute_code":"my_attr","attribute_id":174,"scope":"Global","default_frontend_label":"my_attr","frontend_labels":
[{"store_id":1,"label":"my_attr"}],"frontend_input":"select","is_searchable":true,
"is_visible_in_advanced_search":true,"is_comparable":true,"is_filterable":true}}
And this is the Json response, where you can see it is totally ignoring the values for is_filterable and scope.
{"is_wysiwyg_enabled":false,"is_html_allowed_on_front":false,"used_for_sort_by":
true,"is_filterable":false,"is_filterable_in_search":false,"is_used_in_grid":
true,"is_visible_in_grid":true,"is_filterable_in_grid":true,"position":0,
"apply_to"[], "is_searchable":"0","is_visible_in_advanced_search":"0",
"is_comparable":"0","is_used_for_promo_rules":"0","is_visible_on_front":"0",
"used_in_product_listing":"0","is_visible":true,"scope":"global",
"attribute_id":174,"attribute_code":"taglia","frontend_input":"select",
"entity_type_id":"4","is_required":false,"....
Obviously I've reindexed and cleared caches continuously.
The only way I managed to change those values is directly from the DB.
I'm going to try with an extension but I'm not sure to succeed.
I've seen there was an issue that should have been fixed in older Magento versions and actually other attributes work fine.
Any clue?
EDIT
With a different set of parameters in the call, I managed to change the attribute values, but still no luck from the backend.
This is the updated request (array version)
$data = [
"attribute" => [
"attribute_code" => $my_opt,
"attribute_id" => 173,
"entity_type_id" => 4,
"scope" => "global",
"default_frontend_label" => $my_opt,
"frontend_labels" => [
["store_id" => 1,
"label" => $my_opt ]
],
"frontend_input" => "select",
"is_wysiwyg_enabled" => true,
"is_html_allowed_on_front" => true,
"used_for_sort_by" => true,
"is_filterable" => true,
"is_filterable_in_search" => true,
"is_used_in_grid" => true,
"is_visible_in_grid" => true,
"is_filterable_in_grid" => true,
"position" => 1,
"is_searchable" => true,
"is_visible_in_advanced_search" => true,
"is_comparable" => true,
"is_used_for_promo_rules" => true,
"is_visible_on_front" => true,
"used_in_product_listing" => true,
"is_visible" => true
]
];
EDIT2: Moreover, if I try to update some info from the backend, Magento resets the values of the default view store label to empty and the "use in layered navigation" to "No". Stranger at any try.
EDIT 3
seems like the most possible issue can be caused by duplicated values in these dropdown attributes, but I carefully removed all of them from
eav_attribute_option
eav_attribute_option_value
in the db, but actually removing options only, works from the backend too, whereas the creation of new options is not working
This is not a definitive solution, for which I will create a new question to which someone hopefully will be able to answer, but I wanted to post at least what the true problem is.
Actually, I observed that everything was working until a certain amount of options, and after that, no matter what the values were, it quit working.
Actually I experienced it also in the dev environment, for which the limit was higher, but for an attribute with more than 1300 options, nothing was working as well.
So looking around I found out this is a php memory problem and that you have to increase the value of
max_input_vars=100000
whereas by default is only 1000.
someone suggest also other parameters, but this is actually the most used.
The problem that remains, is that I still have not found the good values to have my attributes with many options, to work.
Trying to implement Factual API with provided Ruby wrapper. Looking to return all Bars within certain number of meters of a geo point. My query looks like this:
factual.table("places").search("category_id"=>"312").geo("$circle" => {"$center" => [40.7811, -73.98], "$meters" => 10000}).rows
This returns a 200:OK response, but 0 records (even though that location is in Manhattan). I'm pretty sure there's a problem with the way I'm passing in the category info.
Per the API documentation, I've also tried passing the category data like this, which returns a syntax error:
factual.table("places").filters("category_ids" => {"category_ids":["6"]}).geo("$circle" => {"$center" => [40.7811, -73.98], "$meters" => 10000}).rows
and this, which returns the Factual error *references unknown field category_ids*:
factual.table("places").filters("category_ids" => {"$in" => ["312", "338"]}).geo("$circle" => {"$center" => [40.7811, -73.98], "$meters" => 10000}).rows
I'm following the documentation samples here and using v3: https://github.com/Factual/factual-ruby-driver/wiki/Read-API and here: http://developer.factual.com/display/docs/Places+API+-+Global+Place+Attributes
EDIT:
I've also tried changing the filters method to search like this:
factual.table("places").search("category_ids" => {"$in" => ["312", "338"]}).geo("$circle" => {"$center" => [40.7811, -73.98], "$meters" => 10000}).rows
This returns records with 338 in the address, irrespective of category. Very strange. I've been trying different things for hours. I'm pretty sure it's an issue with how I'm passing in category information. I'm following the docs as closely as I can, but I can't get it to work.
Try this
factual.table("places").filters("category" => "Bars")
.geo("$circle" => {"$center" =>[40.7811, -73.98] , "$meters" => 1000}).rows
I am new to mongodb/mongomapper and can't find an answer to this.
I have a mongomapper class with the following fields
key :author_id, Integer
key :partecipant_ids, Array
Let's say I have a "record" with the following attributes:
{ :author_id => 10, :partecipant_ids => [10,15,201] }
I want to retrieve all the objects where the partecipant with id 15 is involved.
I did not find any mention in the documentation.
The strange thing is that previously I was doing this query
MessageThread.where :partecipant_ids => [15]
which worked, but after (maybe) some change in the gem/mongodb version it stopped working.
Unfortunately I don't know which version of mongodb and mongomapper I was using before.
In the current versions of MongoMapper, this will work:
MessageThread.where(:partecipant_ids => 15)
And this should work as well...
MessageThread.where(:partecipant_ids => [15])
...because plucky autoexpands that to:
MessageThread.where(:partecipant_ids => { :$in => [15] })
(see https://github.com/jnunemaker/plucky/blob/master/lib/plucky/criteria_hash.rb#L121)
I'd say take a look at your data and try out queries in the Mongo console to make sure you have a working query. MongoDB queries translate directly to MM queries except for the above (and a few other minor) caveats. See http://www.mongodb.org/display/DOCS/Querying
I'm using the following with datamapper to create/get a new user from my db:
user = User.first_or_create({:id => data['id']})
This gets the user with id = data['id'] or creates it if it doesn't already exist.
I want to know how to set other attributes/fields of the returned object regardless of whether it is a new record or existing?
Is the only way to do this to then call user.update({:field => value ...}) or is there a better way to achieve this?
Well, you could write it as one line:
(User.first_or_create(:id => data['id'])).update(:field => value)
with hashes for the parameters if you wish (or if you need to specify more than one); however, it's worth noting that this will only work if the model as specified by the first_or_create is valid. If :name were a required field, for instance, then this wouldn't work:
(User.first_or_create({:id => data['id'], :name => "Morse"})).update(:name => "Lewis")
as the creation in the first part would fail.
You could get around this by specifying the parameters needed for a new record with something like
(User.first_or_create({:id => data['id'], :name => "Morse"}, {:name => "Lewis"})).update(:name => "Lewis")
but this is unusually painful, and is difficult to read.
Also note that using first_or_create with an :id will attempt to create a model with that specific :id, if such a record doesn't exist. This might not be what you want.
Alternatively, you can use first_or_new. You can't call update on an object created using this, however, as the record won't exist (although I believe this might have worked in previous versions of DataMapper).
Just for anyone coming across this answer, User.first_or_create({:id => data['id']}) does NOT "get the user with id = data['id'] or creates it if it doesn't already exist." It actually gets the first record in the table and changes its id t0 data['id'].
To actually get the first record with that id, or create it if it doesn't exist, you need to use a where clause:
User.where(id: data['id]).first_or_create