How does Phoenix conglomerate cookie data? - session

I'm attempting to store some data into the session storage and I'm getting the same cookie error as this guy, the cookie is over the system byte limit of 4096.
This seems pretty straight forward, don't attempt to store more than the system limit in the session. Right, but I'm not attempting to do that. Clearly, the cookie is over 4096 bytes and my additions have caused it to overflow, but that doesn't explain where the data is.
The data I'm attempting to store is only 1500 bytes. In fact, the entire session that is being saved is 1500 bytes (the errored session). Thats nowhere near the overflow limit. So that means one thing for certain: The data stored in :plug_session inside of conn is not the only data being stored inside of the session cookie.
This is the session that's throwing the CookieOverflowError:
:plug_session => %{
"_csrf_token" => "XmE4kgdxk4D0NwwlfTL77Ic62t123123sdfh1s",
"page_trail" => [{"/", "Catalog"}, {'/', "Catalog"}],
"shopping_cart_changeset" => #Ecto.Changeset<
action: nil,
changes: %{
order: #Ecto.Changeset<
action: :insert,
changes: %{
address: #Ecto.Changeset<
action: :insert,
changes: %{
address_one: "800 Arola Drive, apt 321, apt 321",
address_two: "apt 321",
city: "Wooster",
company: "Thomas",
country: "US",
name: "user one",
phone: "3305551111",
state: "WV",
zip_code: "44691"
},
errors: [],
data: #FulfillmentCart.Addresses.Address<>,
valid?: true
>,
priority: false,
shipping_method: #Ecto.Changeset<
action: :insert,
changes: %{id: 2, is_priority?: false, name: "3 Day Select"},
errors: [],
data: #FulfillmentCart.ShippingMethods.ShippingMethod<>,
valid?: true
>
},
errors: [],
data: #FulfillmentCart.Orders.Order<>,
valid?: true
>
},
errors: [],
data: #FulfillmentCart.ShoppingCarts.ShoppingCart<>,
valid?: true
>,
"user_id" => 8
},
I actually followed this guide on decoding a phoenix session cookie, and I get the session before the error.
Which gives me:
iex(8)> [_, payload, _] = String.split(cookie, ".", parts: 3)
["SFMyNTY",
"g3QAAAADbQAAAAtfY3NyZl90b2tlbm0AAAAYWU92dkRfVDh5UXlRTUh4TGlpRTQxOFREbQAAAApwYWdlX3RyYWlsbAAAAAJoAm0AAAABL20AAAAHQ2F0YWxvZ2gCawABL20AAAAHQ2F0YWxvZ2ptAAAAB3VzZXJfaWRhCA",
"Ytg5oklzyWMvtu1vyXVvQ2xBzdtMnS9zVth7LIRALsU"]
iex(9)> {:ok, encoded_term } = Base.url_decode64(payload, padding: false)
{:ok,
<<131, 116, 0, 0, 0, 3, 109, 0, 0, 0, 11, 95, 99, 115, 114, 102, 95, 116, 111,
107, 101, 110, 109, 0, 0, 0, 24, 89, 79, 118, 118, 68, 95, 84, 56, 121, 81,
121, 81, 77, 72, 120, 76, 105, 105, 69, 52, 49, ...>>}
iex(10)> :erlang.binary_to_term(encoded_term)
%{
"_csrf_token" => "YOvvD_T8yQyQMHxLiiE418TD",
"page_trail" => [{"/", "Catalog"}, {'/', "Catalog"}],
"user_id" => 8
}
iex(11)>
This is 127 bytes, so the addition of the 1500 bytes isn't the problem. It's the other allocation of storage that isn't represented inside of the session. What is that?

My assumption of the byte size of the text itself in :plug_session is correct, but the reason the cookie is overflowing is not because the byte size of the decoded text in :plug_session is too big but that the encoded version of the :plug_session is too big. I figured this out by creating multiple cookies and looking at the byte_size of the data.
Save a new cookie
conn = put_resp_cookie(conn, "address",
changeset.changes.order.changes.address.changes, sign: true)
Get a saved cookie
def get_resp_cookie(conn, attribute) do
cookie = conn.req_cookies[attribute]
case cookie != nil do
false ->
{:invalid, %{}}
true ->
[_, payload, _] = String.split(cookie, ".", parts: 3)
{:ok, encoded_term } = Base.url_decode64(payload, padding: false)
{val, max, max_age} = :erlang.binary_to_term(encoded_term)
{:valid, val}
end
end
get_resp_cookie/2 pattern matching
address_map = case Connection.get_resp_cookie(conn, "address") do
{:invalid, val} -> IO.puts("Unable to find cookie.");val
{:valid, val} -> val
end
I made a few changes to the way I save the data from when I posted this. Namely I am now storing a map of changes, not the actual changeset...which means that the session most likely would've worked for me all along.
I think the answer to this issue was that the encoded %Ecto.Changeset{} was too big for the cookie to hold.
If you use this solution then be wary, you have to manage the newly created cookies yourself.

Related

Boto3 Amplify list apps

I have a lot of amplify apps which I want to manage via Lambdas. What is the equivalent of the cli command aws amplify list-apps in boto3, I had multiple attempts, but none worked out for me.
My bit of code that was using nextToken looked like this:
amplify = boto3.client('amplify')
apps = amplify.list_apps()
print(apps)
print('First token is: ', apps['nextToken'])
while 'nextToken' in apps:
apps = amplify.list_apps(nextToken=apps['nextToken'])
print('=====NEW APP=====')
print(apps)
print('=================')
Then I tried to use paginators like:
paginator = amplify.get_paginator('list_apps')
response_iterator = paginator.paginate(
PaginationConfig={
'MaxItems': 100,
'PageSize': 100
}
)
for i in response_iterator:
print(i)
Both of the attempts were throwing inconsistent output. The first one was printing first token and second entry but nothing more. The second one gives only the first entry.
Edit with more attemptsinfo + output. Bellow piece of code:
apps = amplify.list_apps()
print(apps)
print('---------------')
new_app = amplify.list_apps(nextToken=apps['nextToken'], maxResults=100)
print(new_app)
print('---------------')```
Returns (some sensitive output bits were removed):
EVG_long_token_x4gbDGaAWGPGOASRtJPSI='}
---------------
{'ResponseMetadata': {'RequestId': 'f6...e9eb', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/json', 'content-length': ...}, 'RetryAttempts': 0}, 'apps': [{'appId': 'dym7444jed2kq', 'appArn': 'arn:aws:amplify:us-east-2:763175725735:apps/dym7444jed2kq', 'name': 'vesting-interface', 'tags': {}, 'repository': 'https://github.com/...interface', 'platform': 'WEB', 'createTime': datetime.datetime(2021, 5, 4, 3, 41, 34, 717000, tzinfo=tzlocal()), 'updateTime': datetime.datetime(2021, 5, 4, 3, 41, 34, 717000, tzinfo=tzlocal()), 'environmentVariables': {}, 'defaultDomain': 'dym7444jed2kq.amplifyapp.com', 'customRules': _rules_, 'productionBranch': {'lastDeployTime': datetime.datetime(2021, 5, 26, 15, 10, 7, 694000, tzinfo=tzlocal()), 'status': 'SUCCEED', 'thumbnailUrl': 'https://aws-amplify-', 'branchName': 'main'}, - yarn install\n build:\n commands:\n - yarn run build\n artifacts:\n baseDirectory: build\n files:\n - '**/*'\n cache:\n paths:\n - node_modules/**/*\n", 'customHeaders': '', 'enableAutoBranchCreation': False}]}
---------------
I am very confused, why next iteration doesn't has nextToken and how can I get to the next appId.
import boto3
import json
session=boto3.session.Session(profile_name='<Profile_Name>')
amplify_client=session.client('amplify',region_name='ap-south-1')
output=amplify_client.list_apps()
print(output['apps'])

K-fold cross validation - save folds for different models

I am trying to train my models and validate them using sklearn's cross validation. What I want to do is use the same folds across all of my models (which will be running from different python scripts).
How can I do this? Should I save them to a file? or should I save the kfold model? or should I use the same seed?
kfold = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=seed)
Well the easiest way I found to save the folds was to simply get them from the stratified k fold split method by looping over it. Then storing it to a json file:
kfold = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=seed)
folds = {}
count = 1
for train, test in kfold.split(np.zeros(len(y)), y.argmax(1)):
folds['fold_{}'.format(count)] = {}
folds['fold_{}'.format(count)]['train'] = train.tolist()
folds['fold_{}'.format(count)]['test'] = test.tolist()
count += 1
print(len(folds) == n_splits)#assert we have the same number of splits
#dump folds to json
import json
with open('folds.json', 'w') as fp:
json.dump(folds, fp)
Note 1: Argmax here is used because my y values are one hot variables so we need to get the class that is predicted/ground truth.
Now to load it from any other script:
#load to dict to be used
with open('folds.json') as f:
kfolds = json.load(f)
From here we can easily just loop over the elements in the dict:
for key, val in kfolds.items():
print(key)
train = val['train']
test = val['test']
Our json file looks like so:
{"fold_1": {"train": [193, 2405, 2895, 565, 1215, 274, 2839, 1735, 2536, 1196, 40, 2541, 980,...SNIP...830, 1032], "test": [1, 5, 6, 7, 10, 15, 20, 26, 37, 45, 52, 54, 55, 59, 60, 64, 65, 68, 74, 76, 78, 90, 100, 106, 107, 113, 122, 124, 132, 135, 141, 146,...SNIP...]}

Same string but different bytes codes

I have two strings:
a = 'hà nội'
b = 'hà nội'
When I compare them with a == b, it returns false.
I checked the byte codes:
a.bytes = [104, 97, 204, 128, 32, 110, 195, 180, 204, 163, 105]
b.bytes = [104, 195, 160, 32, 110, 225, 187, 153, 105]
What is the cause? How can I fix it so that a == b returns true?
This is an issue with Unicode equivalence.
In order to compare these strings you need to normalize them, so that they both use the same byte sequences for these types of characters.
a.unicode_normalize == b.unicode_normalize
unicode_normalize(form=:nfc) [link]
Returns a normalized form of str, using Unicode normalizations NFC,
NFD, NFKC, or NFKD. The normalization form used is determined by form,
which is any of the four values :nfc, :nfd, :nfkc, or :nfkd. The
default is :nfc.
If the string is not in a Unicode Encoding, then an Exception is
raised. In this context, 'Unicode Encoding' means any of UTF-8,
UTF-16BE/LE, and UTF-32BE/LE, as well as GB18030, UCS_2BE, and
UCS_4BE. Anything else than UTF-8 is implemented by converting to
UTF-8, which makes it slower than UTF-8.

Extract ruby hash element value from an array of objects

I've got the following array
[#<Attachment id: 73, container_id: 1, container_type: "Project", filename: "Eumna.zip", disk_filename: "140307233750_Eumna.zip", filesize: 235303, content_type: nil, digest: "9a10843635b9e9ad4241c96b90f4d331", downloads: 0, author_id: 1, created_on: "2014-03-07 17:37:50", description: "", disk_directory: "2014/03">, #<Attachment id: 74, container_id: 1, container_type: "Project", filename: "MainApp.cs", disk_filename: "140307233750_MainApp.cs", filesize: 1160, content_type: nil, digest: "6b985033e19c5a88bb5ac4e87ba4c4c2", downloads: 0, author_id: 1, created_on: "2014-03-07 17:37:50", description: "", disk_directory: "2014/03">]
I need to extract the value 73 and 74 from this string which is Attachment id.
is there any way to extract this value
just in case author meant he has an actual String instance:
string = '[#<Attachment id: 73, container_id: 1, container_type: "Project", filename: "Eumna.zip", disk_filename: "140307233750_Eumna.zip", filesize: 235303, content_type: nil, digest: "9a10843635b9e9ad4241c96b90f4d331", downloads: 0, author_id: 1, created_on: "2014-03-07 17:37:50", description: "", disk_directory: "2014/03">, #<Attachment id: 74, container_id: 1, container_type: "Project", filename: "MainApp.cs", disk_filename: "140307233750_MainApp.cs", filesize: 1160, content_type: nil, digest: "6b985033e19c5a88bb5ac4e87ba4c4c2", downloads: 0, author_id: 1, created_on: "2014-03-07 17:37:50", description: "", disk_directory: "2014/03">]'
string.scan(/\sid: (\d+)/).flatten
=> ["73", "74"]
Do as below using Array#collect:
array.collect(&:id)
In case it is a string use JSON::parse to get the array back from the string first, then use Array#collect method as below :
require 'json'
array = JSON.parse(string)
array.collect(&:id)
The elements of the array (I'll call it a) look like instances of the class Attachment (not strings). You can confirm that by executing e.class in IRB, where e is any element a (e.g., a.first). My assumption is correct if it returns Attachment. The following assumes that is the case.
#Arup shows how to retrieve the values of the instance variable #id when it has an accessor (for reading):
a.map(&:id)
(aka collect). You can see if #id has an accessor by executing
e.instance_methods(false)
for any element e of a. This returns an array which contains all the instance methods defined for the class Attachment. (The argument false causes Ruby's built-in methods to be excluded.) If #id does not have an accessor, you will need to use Object#instance_variable_get:
a.map { |e| e.instance_variable_get(:#id) }
(You could alternatively write the argument as a string: "#id").
If
s = '[#<Attachment id: 73, container_id: 1,..]'
in fact a string, but you neglected to enclose it in (single) quotes, then you must execute
a = eval(s)
to convert it to an array of instances of Attachment before you can extract the values of :#a.
Hear that 'click'? That was me starting my stop watch. I want to see how long it will take for a comment to appear that scolds me for suggesting the use of (the much-maligned) eval.
Two suggestions: shorten code to the essentials and avoid the need for readers to scroll horizontally to read it. Here, for example, you could have written this:
a = [#<Attachment id: 73, container_id: 1>, #<Attachment id: 74, container_id: 1>]
All the instance variables I've removed are irrelevant to the question.
If that had been too long to fit on one lines (without scrolling horizontally, write it as:
a = [#<Attachment id: 73, container_id: 1>,
#<Attachment id: 74, container_id: 1>]
Lastly, being new to SO, have a look at this guide.

magento - Update product default image to first image in image gallery

I have an import script that imports well over 2000+ products including their images. I run this script via CLI because I feel that this is the best way to go speed-wise even though I have the same import script available and executable at the magento admin as an extension. The script runs pretty well. Almost perfect! However, sometimes the addToImageGallery somehow malfunctions and results into some images having No Image as the default product image and the only other image as not selected as defaults at all. How do I mass-update all products to set the first image in the media gallery for the product to the default 'base', 'image' and 'thumbnail' image(s)?
I found a couple of tricks on doing this (and more) on this link:
http://www.magentocommerce.com/boards/viewthread/59440/ (Thanks transio!)
Although, for Magento 1.6.2.0 (which I use), the first SQL trick there (Trick 1 - Auto-set default base, thumb, small image to first image.) needs a bit of modification.
On the second-to-the last-line there is a AND ev.attribute_id IN (70, 71, 72) part. This should point to attribute ID's which will probably not be relevant in Magento 1.6.2.0 anymore. To fix this, using any MySQL query tool (PHPMyAdmin or MySQL Query Browser), I took a look at the catalog_product_entity_varchar table. There should be entries like:
value_id, entity_type_id, attribute_id, store_id, entity_id, value
..
146649, 4, 116, 0, 1, '2'
146650, 4, 76, 0, 1, ''
146651, 4, 78, 0, 1, ''
146652, 4, 79, 0, 1, '/B/0/B05-01.jpg'
146653, 4, 80, 0, 1, '/B/0/B05-01.jpg'
146654, 4, 81, 0, 1, '/B/0/B05-01.jpg'
146655, 4, 96, 0, 1, ''
146656, 4, 100, 0, 1, ''
146657, 4, 102, 0, 1, 'container2'
..
My money was on the group of three image paths as possible replacements. So the resulting SQL now should be:
UPDATE catalog_product_entity_media_gallery AS mg,
catalog_product_entity_media_gallery_value AS mgv,
catalog_product_entity_varchar AS ev
SET ev.value = mg.value
WHERE mg.value_id = mgv.value_id
AND mg.entity_id = ev.entity_id
AND ev.attribute_id IN (79, 80, 81) # <-- attribute IDs updated here
AND mgv.position = 1;
So I committed to it, ran it and.. presto! All fixed! You might also want to encapsulate this in a transaction if you want. But this is out of this question's scope.
Well, this is the fix that worked for me so far! If there are any more out there, please share!
There was:
146652, 4, 79, 0, 1, '/B/0/B05-01.jpg'
146653, 4, 80, 0, 1, '/B/0/B05-01.jpg'
146654, 4, 81, 0, 1, '/B/0/B05-01.jpg'
So it should be:
AND ev.attribute_id IN (79, 80, 81) # <-- attribute IDs updated here
instead of:
AND ev.attribute_id IN (78, 80, 81) # <-- attribute IDs updated here
Is looking for something similar.
UPDATE catalog_product_entity_media_gallery AS mg,
catalog_product_entity_media_gallery_value AS mgv,
catalog_product_entity_varchar AS ev
SET ev.value = mg.value
WHERE mg.value_id = mgv.value_id
AND mg.entity_id = ev.entity_id
AND ev.attribute_id IN (79, 80, 81) # <-- attribute IDs updated here
AND mgv.position = 1;

Resources