When trying to create a job in the AWS Transcoder pipeline, I'm getting
Aws::ElasticTranscoder::Errors::BadRequest. I'm pretty sure I have the params right, but I'm still getting this ambiguous error. Has anyone encountered that?
My call looks like this
client = Aws::ElasticTranscoder::Client.new(
access_key_id: Rails.application.credentials.aws_s3_access_key_id,
secret_access_key: Rails.application.credentials.aws_s3_secret_access_key,
region: Rails.application.credentials.aws_s3_region,
endpoint: 'https://s3.us-west-2.amazonaws.com'
)
params = {:pipeline_id=>"xxxxxxxxxxxxx-xxxxxx",
:inputs=>
[{:key=>"test/folder1/0.mp4", :frame_rate=>"auto", :resolution=>"auto", :aspect_ratio=>"auto", :interlaced=>"auto", :container=>"auto"},
{:key=>"test/folder1/1.mp4", :frame_rate=>"auto", :resolution=>"auto", :aspect_ratio=>"auto", :interlaced=>"auto", :container=>"auto"}],
:output=>{:key=>"final.mp4", :preset_id=>"1351620000001-000010"}}
client.create_job(params)
Pipeline is active and it's in the same region. So is s3 bucket.
Turns out it was the endpoint param in the client that made this mess... Just remove it. Hope this will hint someone and spare a few hours of debugging.
Related
I have trained a CNN using Tensorflow/Keras and successfully deployed it to Sagemaker using the saved_model format. It answers pings and the dashboard shows it is running.
I now need to be able to send it images and get back inferences. I have already successfully deployed an ANN to Sagemaker and gotten predictions back, so most of the "plumbing" is already working.
The Ruby performing the request is as follows:
def predict
sagemaker = Aws::SageMakerRuntime::Client.new(
access_key_id: Settings.sagemaker_key_id,
secret_access_key: Settings.sagemaker_secret,
region: Settings.sagemaker_aws_region
)
response = sagemaker.invoke_endpoint(endpoint_name: Settings.sagemaker_endpoint_name,
content_type: 'application/x-image',
body: File.open('developer/ai/caox_test_128.jpg', 'rb'))
return response[:body].string
end
(For now, I simply hardcoded a known file for testing.)
When I fire this, I get back this error: Aws::SageMakerRuntime::Errors::ModelError: Received client error (400) from model with message "{ "error": "JSON Parse error: Invalid value. at offset: 0" }"
It's almost as if the model is expecting more in the body than just the image, but I can't tell what. AWS's documentation has an example for Python using boto:
import boto3
import json
endpoint = '<insert name of your endpoint here>'
runtime = boto3.Session().client('sagemaker-runtime')
# Read image into memory
with open(image, 'rb') as f:
payload = f.read()
# Send image via InvokeEndpoint API
response = runtime.invoke_endpoint(EndpointName=endpoint, ContentType='application/x-image', Body=payload)
# Unpack response
result = json.loads(response['Body'].read().decode())
As far as I can tell, they are simply opening a file and sending it directly to sagemaker with no additional pre-processing. And, insofar as I can tell, I'm doing exactly what they are doing in Ruby, just using 'aws-sdk'.
I've looked through Amazon's documentation, and for examples on Google, but there is scant mention of doing anything special before sending the file, so I'm scratching my head.
What else do I need to consider when sending a file to a Sagemaker endpoint running a TensorFlow/Keras CNN to get it to respond with a prediction?
Using the aws-sdk-s3 gem, I am currently able to upload items to buckets and create signed URLs, and am trying to determine if an object exists in a bucket or not. All the documentation I see says client.bucket('bucketname') should work, but in my case it is not. I've tried:
client.bucket('bucketname')
client.bucket['bucketname']
client.buckets('bucketname')
client.buckets['bucketname']
but none work. This suggestion using head_object is a possibility (https://github.com/cloudyr/aws.s3/issues/160), but I'm still curious why bucket isn't working.
DOCS:
https://gist.github.com/hartfordfive/19097441d3803d9aa75ffe5ecf0696da
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/index.html#Resource_Interfaces
You should call bucket or buckets on Aws::S3::Resource instance and not on Aws::S3::Client instance as an error states.
And the links you provided as well as docs show that:
s3 = Aws::S3::Resource.new(
region: 'us-east-1',
credentials: Aws::InstanceProfileCredentials.new()
)
bucket = s3.bucket('my-daily-backups')
how can I get ALL records from route53?
referring code snippet here, which seemed to work for someone, however not clear to me: https://github.com/aws/aws-sdk-ruby/issues/620
Trying to get all (I have about ~7000 records) via resource record sets but can't seem to get the pagination to work with list_resource_record_sets. Here's what I have:
route53 = Aws::Route53::Client.new
response = route53.list_resource_record_sets({
start_record_name: fqdn(name),
start_record_type: type,
max_items: 100, # fyi - aws api maximum is 100 so we'll need to page
})
response.last_page?
response = response.next_page until response.last_page?
I verified I'm hooked into right region, I see the record I'm trying to get (so I can delete later) in aws console, but can't seem to get it through the api. I used this: https://github.com/aws/aws-sdk-ruby/issues/620 as a starting point.
Any ideas on what I'm doing wrong? Or is there an easier way, perhaps another method in the api I'm not finding, for me to get just the record I need given the hosted_zone_id, type and name?
The issue you linked is for the Ruby AWS SDK v2, but the latest is v3. It also looks like things may have changed around a bit since 2014, as I'm not seeing the #next_page or #last_page? methods in the v2 API or the v3 API.
Consider using the #next_record_name and #next_record_type from the response when #is_truncated is true. That's more consistent with how other paginations work in the Ruby AWS SDK, such as with DynamoDB scans for example.
Something like the following should work (though I don't have an AWS account with records to test it out):
route53 = Aws::Route53::Client.new
hosted_zone = ? # Required field according to the API docs
next_name = fqdn(name)
next_type = type
loop do
response = route53.list_resource_record_sets(
hosted_zone_id: hosted_zone,
start_record_name: next_name,
start_record_type: next_type,
max_items: 100, # fyi - aws api maximum is 100 so we'll need to page
)
records = response.resource_record_sets
# Break here if you find the record you want
# Also break if we've run out of pages
break unless response.is_truncated
next_name = response.next_record_name
next_type = response.next_record_type
end
trying to figure AWS S3 API and failing miserably...
I currently got a bucket which consist lots of videos.
I need to request all the videos as an object, that will have the video meta-data which I set once uploading, and the link to share the video.
Problem is I'm getting the object without any of the above...
What Iv'e got so far -
AWS.config.update({accessKeyId: 'id', secretAccessKey: 'key', region: 'eu-
west-1'});
var s3 = new AWS.S3();
var params = {
Bucket: 'Bucket-name',
Delimiter: '/',
Prefix: 'resource/folder-with-videos/'
}
s3.listObjects(params, function (err, data) {
if(err)throw err;
console.log(data);
});
Thanks for reading :)
UPDATE - found that when using getObject and adding ExposeHeader to the CORS setting I can indeed get the metadata I set.
problem is getObject only works on a specific Object (video in my case).
any Idea how I can get all the object like listObject and have values of each object like I do on getObject?
Only solution I can think of is doing listObject to get a list of all the objects, and then by this result to do for each object an getObject ajax?... rip UX
thanks :)
A couple of things to sort out first.
As per the documentation, http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property listObjects API returns exactly what is mentioned in the callback parameters. Using a delimiter causes the S3 to list objects only one level below the prefix given. It won't traverse all objects recursively.
In order to get the URL, you can use http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property. I am not really sure what you meant by meta-data.
I'm using splunk-client to extract results from splunk. Here's the code:
query = "sourcetype=collection #{order_id}"
search = #splunk_client.search(query)
search.wait
The search is happening fine, and it seems like I'm doing everything according to the example (https://github.com/cbrito/splunk-client), but I get this error on the 'search.wait' line:
Undefined namespace prefix: //s:key[#name='isDone']
Any ideas what could be going wrong? Running these commands in irb works fine. Is there some sort of blocking issue?
There is currently very little error checking which occurs within the gem itself. The reason for the error is that wait looks for the status of the isDone key to change to true.
Since your credentials were not properly setup in the first place, the gem creates a search object with an invalid session. The search does not initially fail, because enough response came back from Splunk that Nokogiri processes it into an object without a Splunk search sid.
In the future I should likely raise an exception if a proper sid is not returned to avoid confusion.
Source: I wrote the gem.
I found out the issue -- the splunk client wasn't authenticating properly, and so search was actually a broken SplunkJob object (with a nil username and authentication key). It's strange that there was no error raised until the wait command, but upon inspecting the search object, one of the fields stated that the object was malformed.