We have a lambda deployed to Localstack with an API gateway endpoint configured as AWS_PROXY. Invoking the Lambda with a POST works if the body is empty or contains JSON, however, sending XML in the body (regardless of the Content-Type) always returns a 502. The error in the Localstack log is:
localstack_1 | 2019-11-14T13:29:49:ERROR:localstack.services.generic_proxy: Error forwarding request: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last):
localstack_1 | File "/opt/code/localstack/localstack/services/generic_proxy.py", line 240, in forward
localstack_1 | path=path, data=data, headers=forward_headers)
localstack_1 | File "/opt/code/localstack/localstack/services/apigateway/apigateway_listener.py", line 35, in forward_request
localstack_1 | data = data and json.loads(to_str(data))
localstack_1 | File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
localstack_1 | return _default_decoder.decode(s)
localstack_1 | File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
localstack_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
localstack_1 | File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
localstack_1 | raise JSONDecodeError("Expecting value", s, err.value) from None
localstack_1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I thought AWS_PROXY integration was supposed to pass data direct to the Lambda without any attempt to decode or validate? Is there a way to set up the API gateway integration so that it will happily accept XML?
This is the function that we're using to set up API Gateway after the lambda has been deployed: https://gist.github.com/jgwconsulting/0c5597a661cbc6d29aaa346b2aed5020
This looks to be a problem with Localstack always assuming that POSTed data will be JSON and attempting to parse it to a Python dictionary.
I've created an issue to track the problem here and will attempt to make a PR to fix it if I get time: https://github.com/localstack/localstack/issues/1762
Related
I just upgraded from OSA-Rocky to OSA-Stein and everything seems to work fine except when I try and upload an image via the python-openstackclient CLI. We are running ceph nautilus as our storage. Running the --debug option on the command gives the following:
###Cut majority of debug messages###
Starting new HTTPS connection (2): xxx.xxx.xxx.xxx:9292
[> ] 1%https://xxx.xxx.xxx.xxx:9292 "PUT /v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file HTTP/1.1" 409 79
RESP: [409] Connection: close Content-Length: 79 Content-Type: text/plain; charset=UTF-8 x-openstack-request-id: req-bf01af04-aa24-4ba6-bf93-7f909f442f37
RESP BODY: Omitted, Content-Type is set to text/plain; charset=UTF-8. Only application/json responses have their bodies logged.
PUT call to image for https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file used request id req-bf01af04-aa24-4ba6-bf93-7f909f442f37
Deleting failed upload of image ubuntu-server-20.04-focal
REQ: curl -g -i --insecure -X DELETE https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5 -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.4.0 python-requests/2.26.0 CPython/3.6.8" -H "X-Auth-Token: {SHA256}7413f5abcbdc45993c0b64c560665f198387e1d3a4fa58549b55ba12dc25d03a"
Resetting dropped connection: xxx.xxx.xxx.xxx
https://xxx.xxx.xxx.xxx:9292 "DELETE /v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5 HTTP/1.1" 204 0
RESP: [204] Connection: close Content-Length: 0 Content-Type: text/html; charset=UTF-8 x-openstack-request-id: req-631ee055-77ce-4b6a-8374-d53e3ab0e984
RESP BODY: Omitted, Content-Type is set to text/html; charset=UTF-8. Only application/json responses have their bodies logged.
DELETE call to image for https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5 used request id req-631ee055-77ce-4b6a-8374-d53e3ab0e984
Image creation failed
Traceback (most recent call last):
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 235, in _upload_image
**kwargs)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 293, in _upload_image_put
exceptions.raise_from_response(response)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/exceptions.py", line 238, in raise_from_response
http_status=http_status, request_id=request_id
openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file, Conflict
ConflictException: 409: Client Error for url: https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file, Conflict
Traceback (most recent call last):
File "/home/rkruck/.local/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
result = cmd.run(parsed_args)
File "/home/rkruck/.local/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/home/rkruck/.local/lib/python3.6/site-packages/cliff/display.py", line 115, in run
column_names, data = self.take_action(parsed_args)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 502, in take_action
image = image_client.create_image(**kwargs)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/image/_base_proxy.py", line 213, in create_image
**image_kwargs)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 235, in _upload_image
**kwargs)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 293, in _upload_image_put
exceptions.raise_from_response(response)
File "/home/rkruck/.local/lib/python3.6/site-packages/openstack/exceptions.py", line 238, in raise_from_response
http_status=http_status, request_id=request_id
openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file, Conflict
clean_up CreateImage: ConflictException: 409: Client Error for url: https://xxx.xxx.xxx.xxx:9292/v2/images/395b1140-c0e3-48a9-872e-8b535cb237e5/file, Conflict
END return value: 1
Happy to provide anything else as I am stuck trying to get past this. If I log into Horizon I do see an attempt to save the image, so I don't feel it is an authentication/access issue.
I can't make a template.yaml with Mappings work when testing locally, although they will deploy and work as expected.
As an example, one can follow:
sam init --runtime python3.8
Insert Mappings such as the following in template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
Parameters:
Environment:
Description: 'Required. Environment'
Type: String
AllowedValues:
- test
- prod
Default: test
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
MemorySize: !FindInMap [Environments, !Ref Environment, MemorySize]
Mappings:
Environments:
test:
MemorySize: 512
prod:
MemorySize: 2048
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.8
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
sam build --parameter-overrides ParamaterKey=Environment,ParameterValue=test --use-container
sam local start-api --debug, and it will show unresolved references.
2021-03-06 00:30:42,394 | local start-api command is called
2021-03-06 00:30:42,399 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,417 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,417 | 2 resources found in the stack
2021-03-06 00:30:42,417 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,433 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,434 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,449 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,449 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2021-03-06 00:30:42,452 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,467 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,467 | 2 resources found in the template
2021-03-06 00:30:42,468 | Found '1' API Events in Serverless function with name 'HelloWorldFunction'
2021-03-06 00:30:42,468 | Detected Inline Swagger definition
2021-03-06 00:30:42,468 | Lambda function integration not found in Swagger document at path='/hello' method='get'
2021-03-06 00:30:42,468 | Found '0' APIs in resource 'ServerlessRestApi'
2021-03-06 00:30:42,468 | Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
2021-03-06 00:30:42,468 | 1 APIs found in the template
2021-03-06 00:30:42,471 | Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2021-03-06 00:30:42,471 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2021-03-06 00:30:42,471 | Localhost server is starting up. Multi-threading = True
2021-03-06 00:30:42 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Insist with curl http://127.0.0.1/hello and sam will show an error:
2021-03-06 00:33:25,099 | Found one Lambda function with name 'HelloWorldFunction'
2021-03-06 00:33:25,099 | Invoking app.lambda_handler (python3.8)
2021-03-06 00:33:25,099 | No environment variables found for function 'HelloWorldFunction'
2021-03-06 00:33:25,099 | Environment variables overrides data is standard format
2021-03-06 00:33:25,099 | Loading AWS credentials from session with profile 'None'
2021-03-06 00:33:25,113 | Resolving code path. Cwd=/home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build, CodeUri=/home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction
2021-03-06 00:33:25,113 | Resolved absolute path to code is /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction
2021-03-06 00:33:25,113 | Code /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction is not a zip/jar file
2021-03-06 00:33:25,127 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.19.1.
2021-03-06 00:33:25,127 | Mounting /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
2021-03-06 00:33:25,127 | Cleaning all decompressed code dirs
2021-03-06 00:33:25,128 | Exception on /hello [GET]
Traceback (most recent call last):
File "/home/ricardo/.local/lib/python3.8/site-packages/docker/utils/utils.py", line 415, in parse_bytes
digits = int(digits_part)
ValueError: invalid literal for int() with base 10: "OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])])"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/apigw/local_apigw_service.py", line 297, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream_writer, stderr=self.stderr)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/commands/local/lib/local_lambda.py", line 111, in invoke
self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 216, in wrapped_func
return_value = func(*args, **kwargs)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/lambdafn/runtime.py", line 148, in invoke
container = self.create(function_config, debug_context)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/lambdafn/runtime.py", line 82, in create
self._container_manager.create(container)
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 95, in create
container.create()
File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/docker/container.py", line 167, in create
real_container = self.docker_client.containers.create(self._image, **kwargs)
File "/home/ricardo/.local/lib/python3.8/site-packages/docker/models/containers.py", line 860, in create
create_kwargs = _create_container_args(kwargs)
File "/home/ricardo/.local/lib/python3.8/site-packages/docker/models/containers.py", line 1077, in _create_container_args
create_kwargs['host_config'] = HostConfig(**host_config_kwargs)
File "/home/ricardo/.local/lib/python3.8/site-packages/docker/types/containers.py", line 182, in __init__
self['Memory'] = parse_bytes(mem_limit)
File "/home/ricardo/.local/lib/python3.8/site-packages/docker/utils/utils.py", line 417, in parse_bytes
raise errors.DockerException(
docker.errors.DockerException: Failed converting the string value for memory (OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])])) to an integer.
sam --version gives SAM CLI, version 1.19.1.
Isn't FindInMap intrinsic still not implemented? What am I doing wrong?
I am installing openstack devstack deploying identity ldap
I followed these steps:
i followed the step of the quick start installation of devstack too i created a local.conf file with the following contents:
[[Local|localrc]]
ADMIN_PASSWORD=sesame
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
ENABLED_SERVICES=key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,mysql,rabbit,ldap
KEYSTONE_IDENTITY_BACKEND=ldap
KEYSTONE_CLEAR_LDAP=yes
LDAP_PASSWORD=sesame
then I started the installation ./stack.sh
I had the same problem after every installation attempt. please help me.
CRITICAL keystone [None req-eac86609-c531-420b-8f0c-c05dbb1ea404 None None] Unhandled error: Forbidden: You are not authorized to perform the requested action.
ERROR keystone Traceback (most recent call last):
ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
ERROR keystone sys.exit(main())
ERROR keystone File "/opt/stack/keystone/keystone/cmd/manage.py", line 45, in main
ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 1191, in main
ERROR keystone CONF.command.cmd_class.main()
ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 178, in main
ERROR keystone klass.do_bootstrap()
ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 169, in do_bootstrap
ERROR keystone self.bootstrapper.bootstrap()
ERROR keystone File "/opt/stack/keystone/keystone/cmd/bootstrap.py", line 63, in bootstrap
ERROR keystone self._bootstrap_admin_user()
ERROR keystone File "/opt/stack/keystone/keystone/cmd/bootstrap.py", line 207, in _bootstrap_admin_user
ERROR keystone 'password': self.admin_password
ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 116, in wrapped
ERROR keystone __ret_val = __f(*args, **kwargs)
ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 416, in wrapper
ERROR keystone return f(self, *args, **kwargs)
ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 426, in wrapper
ERROR keystone return f(self, *args, **kwargs)
ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 967, in create_user
ERROR keystone ref = driver.create_user(user['id'], user)
ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 160, in create_user
ERROR keystone self._disallow_write()
ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 157, in _disallow_write
ERROR keystone raise exception.Forbidden(READ_ONLY_LDAP_ERROR_MESSAGE)
ERROR keystone Forbidden: You are not authorized to perform the requested action.
ERROR keystone
+lib/keystone:bootstrap_keystone:1 exit_trap
+./stack.sh:exit_trap:515 local r=1
++./stack.sh:exit_trap:516 jobs -p
+./stack.sh:exit_trap:516 jobs=
+./stack.sh:exit_trap:519 [[ -n '' ]]
+./stack.sh:exit_trap:525 '[' -f /tmp/tmp.sbrPqeUJur ']'
+./stack.sh:exit_trap:526 rm /tmp/tmp.sbrPqeUJur
+./stack.sh:exit_trap:530 kill_spinner
+./stack.sh:kill_spinner:425 '[' '!' -z '' ']'
+./stack.sh:exit_trap:532 [[ 1 -ne 0 ]]
+./stack.sh:exit_trap:533 echo 'Error on exit'
Error on exit
+./stack.sh:exit_trap:535 type -p generate-subunit
+./stack.sh:exit_trap:536 generate-subunit 1530240644 2563 fail
+./stack.sh:exit_trap:538 [[ -z /opt/stack/logs ]]
+./stack.sh:exit_trap:541 /home/stack/devstack/tools/worlddump.py -d /opt/stack/logs
World dumping... see /opt/stack/logs/worlddump-2018-06-29-033327.txt for details
+./stack.sh:exit_trap:550 exit 1
I am about to code with SublimeText and I have installed the Ruby Debugger Plugin. Everything works fine, but if I want to debug some rspec's the debugger disconnects due to this error:
Debugger exception: [Errno 104] Connection reset by peer
StackTrace: Traceback (most recent call last):
File "/home/philipp/.config/sublime-text-3/Packages/Ruby Debugger/debugge/ruby_imp/ruby_debugger_connector.py", line 102, in reader_thread
bytes = self.client.recv(4096)
ConnectionResetError: [Errno 104] Connection reset by peer
the spec file I am debugging looks like this:
1 | require_relative '../spec_helper'
2 |
3 | describe Parser do
4 | before :each do
5 | #parser = Parser.new '../../../file.txt'
6 | end
7 |
8 | describe '#new' do
9 | it 'raises an error if no input file is given' do
10| lambda { Parser.new }.should raise_exception ArgumentError
11| end
12| end
13| end
I have set a breakpoint at line 5, but the debugger never overcomes line 3. What could that be?
I got this Error if i want to Setup the Datastax Cassandra Cluster (Community Edition), i have used this:
http://www.datastax.com/docs/1.2/install/install_ami
[EXEC] 06/05/13-10:52:05 sudo rm -rf /etc/motd
[EXEC] 06/05/13-10:52:05 sudo touch /etc/motd
[INFO] Started with user data set to:
[INFO] --clustername myDSCcluster --totalnodes 3 --version community
[INFO] Using instance type: m1.large
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 31, in initial_configurations
ds2_configure.run()
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 932, in run
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 125, in get_ec2_data
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
It seems as though you may have attempted to launch the cluster within a VPC, which isn't supported.
I did, however, notice that we don't specifically mention that VPC's aren't supported. I'll make sure we get that fixed.