I want to download an Amazon S3 folder, from within a bucket, via CLI. Since you cannot download sub-folders via. Console.
$ aws --version
aws-cli/1.20.30 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.30
Update: My RequestTime is 1 hour behind ServerTime.
RequestTime: 2021-08-25T20:17:12Z
ServerTime: 2021-08-26T12:13:31Z
How do I increase my RequestTime?
I've successfully placed my access key pair into aws configure command.
$ aws configure
AWS Access Key ID [None]: <access_key_id>
AWS Secret Access Key [None]: <secret_access_key>
Default region name [None]:
Default output format [None]:
I have an S3 URL, of this structure: s3://parent_filepath/foo/bar/folder_of_interest/
Execution with --debug:
danielbellio#LAPTOP-G1DAPU88:~$ aws s3 cp --recursive --debug s3://iotahoe-datascience/python_workers/dl_test/data_for_daniel/ .
2021-08-25 21:17:12,125 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29
2021-08-25 21:17:12,125 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', '--recursive', '--debug', 's3://iotahoe-datascience/python_workers/dl_test/data_for_daniel/', '.']
2021-08-25 21:17:12,126 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f1db9b1ce50>
2021-08-25 21:17:12,126 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f1dba3605e0>
2021-08-25 21:17:12,127 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f1dba2c24c0>
2021-08-25 21:17:12,131 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-08-25 21:17:12,134 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f1db9c57550>
2021-08-25 21:17:12,135 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f1db9b259d0>
2021-08-25 21:17:12,137 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,137 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x7f1db9b259d0>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,143 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,143 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,144 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,144 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,146 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,148 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,148 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,150 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,150 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,151 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,151 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,153 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,154 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,154 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,155 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,156 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,156 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,157 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,157 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,158 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,159 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,159 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,160 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,160 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,161 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-08-25 21:17:12,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-08-25 21:17:12,175 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2021-08-25 21:17:12,178 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/endpoints.json
2021-08-25 21:17:12,192 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,198 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/service-2.json
2021-08-25 21:17:12,241 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,242 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,246 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,250 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/_retry.json
2021-08-25 21:17:12,251 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,253 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,255 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,255 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,259 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,261 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,263 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,264 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,264 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,269 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,270 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,272 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2021-08-25 21:17:12,285 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7f1dba735b80>
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7f1dba7298b0>
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,289 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f1db9577700>>
2021-08-25 21:17:12,289 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7f1dba729700>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7f1dba729c10>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7f1dba735f70>
2021-08-25 21:17:12,291 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/iotahoe-datascience?list-type=2', 'query_string': {'prefix': 'python_workers/dl_test/data_for_daniel/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29'}, 'body': b'', 'url': 'https://s3.amazonaws.com/iotahoe-datascience?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f1db95bab20>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'iotahoe-datascience'}}}
2021-08-25 21:17:12,291 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1db95baa90>>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1db95e0d90>>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7f1dba7295e0>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f1db9577790>>
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/iotahoe-datascience?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - URI updated to: https://iotahoe-datascience.s3.amazonaws.com/?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
encoding-type=url&list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F
host:iotahoe-datascience.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210825T201712Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210825T201712Z
20210825/us-east-1/s3/aws4_request
e4f14fd5d86caf530371707ca7a13db8451306584538e8b48520e03be2d83ae0
2021-08-25 21:17:12,295 - MainThread - botocore.auth - DEBUG - Signature:
ff8a86a8720c3a2ae019399c78e6f6a2902d9323d462b57cc35ec06355f52163
2021-08-25 21:17:12,295 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://iotahoe-datascience.s3.amazonaws.com/?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29', 'X-Amz-Date': b'20210825T201712Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARAHBOZTNVW5ZZIGU/20210825/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ff8a86a8720c3a2ae019399c78e6f6a2902d9323d462b57cc35ec06355f52163'}>
2021-08-25 21:17:12,297 - MainThread - botocore.httpsession - DEBUG - Certificate path: /home/danielbellio/.local/lib/python3.8/site-packages/certifi/cacert.pem
2021-08-25 21:17:12,297 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): iotahoe-datascience.s3.amazonaws.com:443
2021-08-25 21:17:12,739 - MainThread - urllib3.connectionpool - DEBUG - https://iotahoe-datascience.s3.amazonaws.com:443 "GET /?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url HTTP/1.1" 403 None
2021-08-25 21:17:12,754 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'eu-central-1', 'x-amz-request-id': 'JQNECTZFFXWB7K1H', 'x-amz-id-2': 'SyNDbSFcc+D051KKti5p3A0hbpnrDinWH6v3XT3UUGXDr6OAt6VTxY0o+D8SRGzEDuhnBU93FYk=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 26 Aug 2021 12:13:29 GMT', 'Server': 'AmazonS3'}
2021-08-25 21:17:12,755 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><RequestTime>20210825T201712Z</RequestTime><ServerTime>2021-08-26T12:13:31Z</ServerTime><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>JQNECTZFFXWB7K1H</RequestId><HostId>SyNDbSFcc+D051KKti5p3A0hbpnrDinWH6v3XT3UUGXDr6OAt6VTxY0o+D8SRGzEDuhnBU93FYk=</HostId></Error>'
2021-08-25 21:17:12,763 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7f1db95775e0>
2021-08-25 21:17:12,763 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2021-08-25 21:17:12,764 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,764 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7f1dba735ca0>
2021-08-25 21:17:12,765 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7f1db9b1c8b0>
2021-08-25 21:17:12,766 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
Traceback (most recent call last):
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/s3handler.py", line 173, in call
for fileinfo in fileinfos:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
for file_base in files:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/filegenerator.py", line 142, in call
for src_path, extra_information in file_iterator:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/filegenerator.py", line 322, in list_objects
for key in lister.list_objects(bucket=bucket, prefix=prefix,
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/utils.py", line 416, in list_objects
for page in pages:
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/paginate.py", line 255, in __iter__
response = self._make_request(current_kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/paginate.py", line 332, in _make_request
return self._method(**current_kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
fatal error: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
2021-08-25 21:17:12,771 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
I followed this tutorial to ensure my Windows 10 machine has Date Time automatically synced.
New account...
Solution:
sudo apt install ntpdate
sudo ntpdate -sb time.nist.gov
Source
To confirm change:
curl http://s3.amazonaws.com -v
date -u
Both times should be the same :)
Related
I am trying to test the reconnection feature of RSocket but it is not working when the sever was brought down in between and restart.
Step 1: Started the client when the server is down
16:02:35.351 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.processId: 55104 (auto-detected)
16:02:35.515 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.machineId: 10:e7:c6:ff:fe:31:38:c0 (auto-detected)
16:02:35.560 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x7b01559f] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:35.569 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x7b01559f] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
16:02:36.619 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x8a39ea36] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:36.620 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x8a39ea36] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
16:02:37.625 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x9754e5dd] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:37.625 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x9754e5dd] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
Step 2: Server Started . Reconnection worked
16:08:31.359 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:08:31.359 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x67b9f3a1] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
16:08:31.862 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] Registering pool release on close event for channel
16:08:31.863 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] Channel connected, now 1 active connections and 0 inactive connections
16:08:31.863 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] onStateChange(PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999]}, [connected])
16:08:31.865 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] onStateChange(ChannelOperations{PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999]}}, [configured])
16:08:31.891 [reactor-tcp-nio-2] DEBUG io.rsocket.FrameLogger - sending ->
Frame => Stream ID: 0 Type: SETUP Flags: 0b0 Length: 75
Data:
16:08:31.957 [reactor-tcp-nio-2] DEBUG reactor.netty.channel.FluxReceive - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
16:08:31.969 [reactor-tcp-nio-2] DEBUG io.rsocket.FrameLogger - sending ->
Frame => Stream ID: 1 Type: REQUEST_STREAM Flags: 0b100000000 Length: 57 InitialRequestN: 9223372036854775807
Metadata:
Step 3: Brought down the server
16:10:10.993 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] Channel closed, now 0 active connections and 0 inactive connections
null
16:10:10.999 [reactor-tcp-nio-2] DEBUG org.springframework.core.codec.CharSequenceEncoder - Writing "ClientName:1607847010998"
16:10:11.008 [reactor-tcp-nio-2] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException: null
16:10:11.008 [reactor-tcp-nio-2] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException: null
16:10:11.010 [reactor-tcp-nio-2] DEBUG reactor.netty.ReactorNetty - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] Non Removed handler: RSocketLengthCodec, context: ChannelHandlerContext(RSocketLengthCodec, [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999]), pipeline: DefaultChannelPipeline{(RSocketLengthCodec = io.rsocket.transport.netty.RSocketLengthCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
16:10:11.010 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] onStateChange(ChannelOperations{PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999]}}, [disconnecting])
RSocket didn't reconnect after the above step. Below is my program. Can someone pls help to review and advice what is wrong with this.
RSocketStrategies strategies = RSocketStrategies.builder()
.encoders(e -> e.add(new Jackson2JsonEncoder()))
.decoders(e -> e.add(new Jackson2JsonDecoder()))
.build();
RSocketRequester r = RSocketRequester.builder()
.rsocketConnector(connector ->
connector.reconnect(Retry.indefinitely().doAfterRetry(e-> System.out.println("doAfterRetry ===>"+e.failure())))
).dataMimeType(MediaType.APPLICATION_JSON)
.rsocketStrategies(strategies)
.tcp("localhost", 7999);
Updated on 15/Dec
Following code sends request after the step 2. After the disconnection, it wasn't able resume the stream. I am sure I must be missing something in my code. Kindly help
requester.route("route_name")
.data("RequestData")
.retrieveFlux(MyResponse.class)
.doOnError(ex ->{
System.out.println("doOnError"+ex);
}).doOnCancel(()->{
System.out.println("doOnCancel");
}).doOnComplete(()-> {
System.out.println("doOnCancel");
})
.subscribe(result -> {
System.out.println("===>"+result);
});
The reconnect method has extensive Javadoc. The main purpose of this feature is to establish a single shared connection, no matter how many subscribers there may be at one time:
When this is enabled, the connect methods of this class return a
special Mono that maintains a single, shared RSocket
The Retry determines how long to keep trying to connect before giving up, but once it stops trying, or once the connection is established and then lost, it won't automatically try to connect again.
downstream subscribers for individual requests still need their own retry logic
to determine if or when failed requests should be retried which in turn
triggers the shared reconnect
So the needs of each individual request determine whether to even try to connect again while the Retry given to reconnect determines the retry logic for each shared reconnect.
Be sure to review the corresponding code snippets in the Javadoc as well.
I got the following exception during my Spring app.
I use Spring-boot-starter 2.2.1, with spring-boot-starer-reactor-netty 2.2.1. (Reactory netty 0.9.1 in it.)
I use spring-boot-admin-starter-client 2.2.0 within my application.
It is not thrown sequentially, and I can't see any pattern on the occurrences. I've just started my application as a unix service and after some time it threw. My idea is about the Spring admin and client, but I upgraded them to the latest but the exception still exists.
Can u give me ANY advice why is it throwing?
This exception is happened on thread or-http-epoll-1:
2019-12-12 17:24:46.756+0100 ERROR 15674 --- [or-http-epoll-1] i.n.u.ResourceLeakDetector
and on this thread everything seems alright earlier:
2019-12-12 17:23:16.763+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServer : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter#33681b9d
2019-12-12 17:23:16.763+0100 DEBUG 15674 --- [or-http-epoll-1] o.s.w.s.a.HttpWebHandlerAdapter : [967afa3f] HTTP GET "/actuator/health"
2019-12-12 17:23:16.764+0100 DEBUG 15674 --- [or-http-epoll-1] .b.a.e.w.r.WebFluxEndpointHandlerMapping : [967afa3f] Mapped to org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#6de54b40
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Decreasing pending responses, now 0
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] o.s.w.s.a.HttpWebHandlerAdapter : [967afa3f] Completed 200 OK
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP response frame
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP packet was sent, terminating the channel
2019-12-12 17:24:17.959+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Decreasing pending responses, now 0
2019-12-12 17:24:17.959+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP packet was sent, terminating the channel
2019-12-12 17:24:46.756+0100 ERROR 15674 --- [or-http-epoll-1] i.n.u.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.i o/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
#011io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
#011io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
#011io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
#011io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
#011io.netty.buffer.ByteBufUtil.readBytes(ByteBufUtil.java:445) #011io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:314)
#011io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
#011io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
#011io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
#011io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
#011io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
#011io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
#011io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
#011io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
#011io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) #011io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
#011io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
#011io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
#011java.lang.Thread.run(Thread.java:748)
SOLUTION: I found a message.getPayload() what was not released, i wrote that there and the problem is not occurred anymore.
I have this upstream Publisher that emits a number every second:
private fun counter(emissionIntervalMillis: Long) =
Flux.interval(Duration.ofMillis(emissionIntervalMillis))
.map { it }.log()
Consider this implementation in which a UnicastProcessor subscribes to the previous Flux. In addition there is a ConnectableFlux generated with processor.publish().autoConnect(). Finally I subscribe to this ConnectableFlux:
val latch = CountDownLatch(15)
val numberGenerator: Flux<Long> = counter(1000)
val processor = UnicastProcessor.create<Long>()
numberGenerator.subscribeWith(processor)
val connectableFlux = processor.doOnSubscribe { println("subscribed!") }.publish().autoConnect()
Thread.sleep(5000)
connectableFlux.subscribe {
logger.info("Element [{}]", it)
latch.countDown()
}
latch.await()
Logs:
15:58:26.941 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
15:58:26.967 [main] INFO reactor.Flux.Map.1 - onSubscribe(FluxMap.MapSubscriber)
15:58:26.969 [main] INFO reactor.Flux.Map.1 - request(unbounded)
15:58:27.973 [parallel-1] INFO reactor.Flux.Map.1 - onNext(0)
15:58:28.973 [parallel-1] INFO reactor.Flux.Map.1 - onNext(1)
15:58:29.975 [parallel-1] INFO reactor.Flux.Map.1 - onNext(2)
15:58:30.974 [parallel-1] INFO reactor.Flux.Map.1 - onNext(3)
15:58:31.974 [parallel-1] INFO reactor.Flux.Map.1 - onNext(4)
subscribed!
15:58:31.979 [main] INFO com.codependent.processors.Tests - Element [0]
15:58:31.980 [main] INFO com.codependent.processors.Tests - Element [1]
15:58:31.980 [main] INFO com.codependent.processors.Tests - Element [2]
15:58:31.980 [main] INFO com.codependent.processors.Tests - Element [3]
15:58:31.980 [main] INFO com.codependent.processors.Tests - Element [4]
15:58:32.972 [parallel-1] INFO reactor.Flux.Map.1 - onNext(5)
15:58:32.972 [parallel-1] INFO com.codependent.processors.Tests - Element [5]
As you see, when there is a subscriber to the connectableFlux, it gets the previously generated items which were cached by the UnicastProcessor. I guess this is the expected behaviour:
if you push any amount of data through it while its Subscriber has not
yet requested data, it will buffer all of the data.
Now, instead of using autoConnect I use connect():
val latch = CountDownLatch(15)
val numberGenerator: Flux<Long> = counter(1000)
val processor = UnicastProcessor.create<Long>()
numberGenerator.subscribeWith(processor)
val connectableFlux = processor.doOnSubscribe { println("subscribed!") }.publish()
connectableFlux.connect()
Thread.sleep(5000)
connectableFlux.subscribe {
logger.info("Element [{}]", it)
latch.countDown()
}
The result now quite different, the subscriber doesn't get the items that should've been cached by the UnicastProcessor. Can someone explain the difference?
16:08:44.299 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
16:08:44.324 [main] INFO reactor.Flux.Map.1 - onSubscribe(FluxMap.MapSubscriber)
16:08:44.326 [main] INFO reactor.Flux.Map.1 - request(unbounded)
subscribed!
16:08:45.330 [parallel-1] INFO reactor.Flux.Map.1 - onNext(0)
16:08:46.329 [parallel-1] INFO reactor.Flux.Map.1 - onNext(1)
16:08:47.329 [parallel-1] INFO reactor.Flux.Map.1 - onNext(2)
16:08:48.331 [parallel-1] INFO reactor.Flux.Map.1 - onNext(3)
16:08:49.330 [parallel-1] INFO reactor.Flux.Map.1 - onNext(4)
16:08:50.328 [parallel-1] INFO reactor.Flux.Map.1 - onNext(5)
16:08:50.328 [parallel-1] INFO com.codependent.processors.Tests - Element [5]
16:08:51.332 [parallel-1] INFO reactor.Flux.Map.1 - onNext(6)
16:08:51.332 [parallel-1] INFO com.codependent.processors.Tests - Element [6]
After rereading the docs I found that autoConnect() can pass the minimum number of subscribers necessary to subscribe to the upstream. Changing it to autoConnect(0) has the same effect as connect(), not passing the previous items to the subscriber:
val latch = CountDownLatch(15)
val numberGenerator: Flux<Long> = counter(1000)
val processor = UnicastProcessor.create<Long>()
numberGenerator.subscribeWith(processor)
val connectableFlux = processor.doOnSubscribe { println("subscribed!") }.log().publish().autoConnect(0)
Thread.sleep(5000)
connectableFlux.subscribe {
logger.info("Element [{}]", it)
latch.countDown()
}
latch.await()
It seems that since the connectableFlux is ready (connected), the processor gets the OnSubscribe signal, and as there aren't any actual subscribers for the connectableFlux, it discards the items.
Changing publish() to replay() would make the subscriber get the items from the beginning , as stated in the doc.
val connectableFlux = processor.doOnSubscribe { println("subscribed!") }.log().replay().autoConnect(0)
I've set up the API Manager and the IS as Key Manager. (2.1.0 and 5.3.0)
Furthermore I've set up a Primary and a Secondary Userstore.
We've created Users in the Secondary Userstore and used them to subscribe to Applications succesfully in the past, but suddenly, that does not work anymore.
When creating an application to subscribe APIs to and then generating the access tokens the AM and IS throw errors.
IS:
TID: [-1234] [] [2017-10-17 09:07:37,479] WARN {org.wso2.carbon.identity.application.mgt.ApplicationManagementServiceImpl} - Illegal Access! User EXTERNAL/test#test.com does not have access to the application EXTERNAL_test-AT-test.com_201710170906_SANDBOX
TID: [-1234] [] [2017-10-17 09:07:37,480] ERROR {org.wso2.carbon.apimgt.impl.utils.APIUtil} - Error occurred while creating ServiceProvider for app EXTERNAL_test-AT-test.com_201710170906_SANDBOX
org.wso2.carbon.identity.application.common.IdentityApplicationManagementException: Error occurred while updating the application: EXTERNAL_test-AT-test.com_201710170906_SANDBOX
at org.wso2.carbon.identity.application.mgt.ApplicationManagementServiceImpl.updateApplication(ApplicationManagementServiceImpl.java:300)
[...]
Caused by: org.wso2.carbon.identity.application.common.IdentityApplicationManagementException: User not authorized
at org.wso2.carbon.identity.application.mgt.ApplicationManagementServiceImpl.updateApplication(ApplicationManagementServiceImpl.java:277)
AM:
TID: [-1234] [] [2017-10-17 09:07:37,490] ERROR {org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl} - Can not create OAuth application : 201710170906_SANDBOX {org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl}
org.wso2.carbon.apimgt.keymgt.stub.subscriber.APIKeyMgtSubscriberServiceAPIManagementException: APIKeyMgtSubscriberServiceAPIManagementException
[...]
TID: [-1234] [] [2017-10-17 09:07:37,492] ERROR {org.wso2.carbon.apimgt.impl.utils.APIUtil} - Error occurred while executing SubscriberKeyMgtClient. {org.wso2.carbon.apimgt.impl.utils.APIUtil}
org.wso2.carbon.apimgt.api.APIManagementException: Can not create OAuth application : 201710170906_SANDBOX
[...]
Caused by: org.wso2.carbon.apimgt.keymgt.stub.subscriber.APIKeyMgtSubscriberServiceAPIManagementException: APIKeyMgtSubscriberServiceAPIManagementException
at sun.reflect.GeneratedConstructorAccessor339.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
I've tried and tested a couple things and verified in the database (Oracle) directly:
When I create an application it is created in the database. (key store)
When I subscribe to an api, the link between api and app is created in the database.
When I generate the access tokens the errors occur, the role for the user is created. (auth store)
"View Users" on the role shows the users having the role.
"View Roles" on the user does not show the user having the role.
SP_APP table in the database contains the newly created application, but without an access token.
When I delete the application
the SP_APP stays in the database.
the Role stays in the database.
But now for the tricky part:
Everything works fine for users in the Primary Userstore.
And the really tricky part:
Sometimes (like once every 15-20 times), the token generation for Secondary Users works.
[EDIT]
Further investigation lead me to the assumption, that something is wrong with the user/role caching/reading.
Logs from the key manager:
Questions:
Why does the org.apache.axis2.transport.http.AxisServlet throw an NPE?
Shouldn't the 2017-10-18 19:46:28,877 part work after it created the roles in the DB without a problem?
The full log:
[2017-10-18 19:46:28,732] DEBUG - Created singleton instance for org.wso2.carbon.identity.auth.service.handler.HandlerManager {org.wso2.carbon.identity.auth.service.handler.HandlerManager}
[2017-10-18 19:46:28,733] DEBUG - Get first priority handler for the given handler list. {org.wso2.carbon.identity.auth.service.handler.HandlerManager}
[2017-10-18 19:46:28,733] DEBUG - Get first priority handler : DefaultAuthenticationManager(org.wso2.carbon.identity.auth.service.AuthenticationManager) {org.wso2.carbon.identity.auth.service.handler.HandlerManager}
[2017-10-18 19:46:28,733] DEBUG - {org.apache.axis2.transport.http.AxisServlet}
java.lang.NullPointerException
at javax.servlet.GenericServlet.getServletContext(GenericServlet.java:123)
at org.apache.axis2.transport.http.AxisServlet.createMessageContext(AxisServlet.java:715)
at org.apache.axis2.transport.http.AxisServlet.createMessageContext(AxisServlet.java:741)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:138)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:72)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
[2017-10-18 19:46:28,734] DEBUG - getAction (null) from org.apache.axis2.client.Options#72ca18ef {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - Old SoapAction is (null) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,734] DEBUG - New SoapAction is (null) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,734] DEBUG - setAction Old action is (null) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - setAction New action is (null) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - Input contentType (application/soap+xml; charset=UTF-8; action="urn:createOAuthApplicationByApplicationInfo") {org.apache.axis2.builder.BuilderUtil}
[2017-10-18 19:46:28,734] DEBUG - CharSetEncoding from content-type (UTF-8) {org.apache.axis2.builder.BuilderUtil}
[2017-10-18 19:46:28,734] DEBUG - getAction (null) from org.apache.axis2.client.Options#72ca18ef {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - Old SoapAction is (null) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,734] DEBUG - New SoapAction is (urn:createOAuthApplicationByApplicationInfo) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,734] DEBUG - setAction Old action is (null) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - setAction New action is (urn:createOAuthApplicationByApplicationInfo) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - getAction (urn:createOAuthApplicationByApplicationInfo) from org.apache.axis2.client.Options#72ca18ef {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,734] DEBUG - SoapAction is (urn:createOAuthApplicationByApplicationInfo) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,734] DEBUG - createSOAPEnvelope using Builder (class org.apache.axis2.builder.SOAPBuilder) selected from type (application/soap+xml) {org.apache.axis2.transport.TransportUtils}
[2017-10-18 19:46:28,734] DEBUG - char set encoding set from default =UTF-8 {org.apache.axis2.builder.BuilderUtil}
[2017-10-18 19:46:28,734] DEBUG - [MessageContext: logID=62aa565ba9f5ad1bb8a9618af3ccb60006e855dbc7d5825a] Checking for Service using target endpoint address : https://wimtest.domain.com/services/APIKeyMgtSubscriberService {org.apache.axis2.dispatchers.RequestURIBasedServiceDispatcher}
[2017-10-18 19:46:28,735] DEBUG - org.apache.axis2.i18n.resource::handleGetObject(servicefound) {org.apache.axis2.i18n.ProjectResourceBundle}
[2017-10-18 19:46:28,735] DEBUG - [MessageContext: logID=62aa565ba9f5ad1bb8a9618af3ccb60006e855dbc7d5825a] Found AxisService : APIKeyMgtSubscriberService {org.apache.axis2.engine.AbstractDispatcher}
[2017-10-18 19:46:28,735] DEBUG - getAction (urn:createOAuthApplicationByApplicationInfo) from org.apache.axis2.client.Options#72ca18ef {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,735] DEBUG - SoapAction is (urn:createOAuthApplicationByApplicationInfo) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,735] DEBUG - [MessageContext: logID=62aa565ba9f5ad1bb8a9618af3ccb60006e855dbc7d5825a] Checking for Operation using Action : urn:createOAuthApplicationByApplicationInfo {org.apache.axis2.dispatchers.ActionBasedOperationDispatcher}
[2017-10-18 19:46:28,735] DEBUG - getOperationBySOAPAction: Operation (org.apache.axis2.description.InOutAxisOperation#5dd7070f,createOAuthApplicationByApplicationInfo) for soapAction: urn:createOAuthApplicationByApplicationInfo found in action map. {org.apache.axis2.description.AxisService}
[2017-10-18 19:46:28,735] DEBUG - org.apache.axis2.i18n.resource::handleGetObject(operationfound) {org.apache.axis2.i18n.ProjectResourceBundle}
[2017-10-18 19:46:28,735] DEBUG - [MessageContext: logID=62aa565ba9f5ad1bb8a9618af3ccb60006e855dbc7d5825a] Found AxisOperation : createOAuthApplicationByApplicationInfo {org.apache.axis2.engine.AbstractDispatcher}
[2017-10-18 19:46:28,735] DEBUG - getAddressingRequirementParemeterValue: value: 'null' {org.apache.axis2.addressing.AddressingHelper}
[2017-10-18 19:46:28,735] DEBUG - [MessageContext: logID=62aa565ba9f5ad1bb8a9618af3ccb60006e855dbc7d5825a] isReplyRedirected: ReplyTo is null. Returning false {org.apache.axis2.addressing.AddressingHelper}
[2017-10-18 19:46:28,735] DEBUG - Resolving tenant id from tenant domain {org.wso2.carbon.context.PrivilegedCarbonContext}
[2017-10-18 19:46:28,735] DEBUG - Authorization cache miss for username : admin resource /permission/admin/login action : ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,735] DEBUG - Roles which have permission for resource : /permission/admin/login action : ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,736] DEBUG - Role : admin {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,736] DEBUG - Role : INTERNAL/creator {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,736] DEBUG - Role : INTERNAL/subscriber {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,736] DEBUG - Role : INTERNAL/publisher {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,736] DEBUG - Retrieving internal roles for user name : admin and search filter * {org.wso2.carbon.user.core.common.AbstractUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Searching for user admin {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Replace escape characters configured to: true {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Replace escape characters configured to: true {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - value after escaping special characters in admin : admin {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - User: admin exist: true {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Replace escape characters configured to: true {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Reading roles with the memberOfProperty Property: memberOf {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,740] DEBUG - Result for searchBase: CN=admin,OU=EDV,DC=test,DC=com searchFilter: (&(|(objectClass=userProxyFull)(objectClass=user))(mail=admin)) property:memberOf appendDN: false {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,755] DEBUG - Replace escape characters configured to: true {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,757] DEBUG - Found user: CN=admin,OU=GROUPS,DC=test,DC=com {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,757] DEBUG - Result : CN=admin,OU=GROUPS,DC=test,DC=com {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,757] DEBUG - GetGroupNameAttributeValuesOfGroups with DN {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,773] DEBUG - Using DN: CN=admin,OU=GROUPS,DC=test,DC=com {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,773] DEBUG - Found role: admin {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager}
[2017-10-18 19:46:28,774] DEBUG - admin user has permitted resource : /permission/admin/login, action :ui.execute {org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager}
[2017-10-18 19:46:28,774] DEBUG - getAction (null) from org.apache.axis2.client.Options#7889e70c {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,774] DEBUG - Old WSAAction is (null) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,774] DEBUG - New WSAAction is (urn:createOAuthApplicationByApplicationInfoResponse) {org.apache.axis2.context.MessageContext}
[2017-10-18 19:46:28,774] DEBUG - setAction Old action is (null) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,774] DEBUG - setAction New action is (urn:createOAuthApplicationByApplicationInfoResponse) {org.apache.axis2.client.Options}
[2017-10-18 19:46:28,774] DEBUG - messageID is null. {org.apache.axis2.context.ConfigurationContext}
[2017-10-18 19:46:28,778] DEBUG - Resolving tenant id from tenant domain {org.wso2.carbon.context.PrivilegedCarbonContext}
[2017-10-18 19:46:28,778] DEBUG - Creating application role : Application/EXTERNAL_test-AT-test.com_201710181946_SANDBOX and assign the user : [EXTERNAL/test#test.com] to that role {org.wso2.carbon.identity.application.mgt.ApplicationMgtUtil}
[2017-10-18 19:46:28,779] DEBUG - Is roleName: Application/EXTERNAL_test-AT-test.com_201710181946_SANDBOX Exist: false TenantId: -1234 {org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager}
[2017-10-18 19:46:28,783] DEBUG - Is roleName: Application/EXTERNAL_test-AT-test.com_201710181946_SANDBOX Exist: false TenantId: -1234 {org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager}
[2017-10-18 19:46:28,792] DEBUG - Executed a batch update. Query is : INSERT INTO UM_HYBRID_USER_ROLE (UM_USER_NAME, UM_ROLE_ID, UM_TENANT_ID, UM_DOMAIN_ID) VALUES (?,(SELECT UM_ID FROM UM_HYBRID_ROLE WHERE UM_ROLE_NAME=? AND UM_TENANT_ID=?), ?, (SELECT UM_DOMAIN_ID FROM UM_DOMAIN WHERE UM_TENANT_ID=? AND UM_DOMAIN_NAME=?)): and result is[1] {org.wso2.carbon.user.core.util.DatabaseUtil}
[2017-10-18 19:46:28,812] DEBUG - Creating Application EXTERNAL_test-AT-test.com_201710181946_SANDBOX for user EXTERNAL/test#test.com {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,820] DEBUG - Application Stored successfully with application id 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,822] DEBUG - Loading Basic Application Data of EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,823] DEBUG - ApplicationID: 135 ApplicationName: EXTERNAL_test-AT-test.com_201710181946_SANDBOX UserName: test#test.com TenantDomain: carbon.super {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,823] DEBUG - Reading Clients of Application 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,825] DEBUG - Reading Steps of Application 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,831] DEBUG - Reading Claim Mappings of Application 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,834] DEBUG - Reading Role Mapping of Application 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,874] DEBUG - Clearing cache entry for EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.identity.provisioning.listener.ProvisioningApplicationMgtListener}
[2017-10-18 19:46:28,874] DEBUG - Provisioning cached entry not found for sp EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.identity.provisioning.listener.ProvisioningApplicationMgtListener}
[2017-10-18 19:46:28,874] DEBUG - Checking whether user has role : Application/EXTERNAL_test-AT-test.com_201710181946_SANDBOX by retrieving role list of user : EXTERNAL/test#test.com {org.wso2.carbon.identity.application.mgt.ApplicationMgtUtil}
[2017-10-18 19:46:28,874] DEBUG - Loading Application Name for ID: 135 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,876] DEBUG - ApplicationName : EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl}
[2017-10-18 19:46:28,877] DEBUG - Checking whether user has role : Application/EXTERNAL_test-AT-test.com_201710181946_SANDBOX by retrieving role list of user : EXTERNAL/test#test.com {org.wso2.carbon.identity.application.mgt.ApplicationMgtUtil}
[2017-10-18 19:46:28,877] WARN - Illegal Access! User EXTERNAL/test#test.com does not have access to the application EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.identity.application.mgt.ApplicationManagementServiceImpl}
[2017-10-18 19:46:28,877] ERROR - Error occurred while creating ServiceProvider for app EXTERNAL_test-AT-test.com_201710181946_SANDBOX {org.wso2.carbon.apimgt.impl.utils.APIUtil}
org.wso2.carbon.identity.application.common.IdentityApplicationManagementException: Error occurred while updating the application: EXTERNAL_test-AT-test.com_201710181946_SANDBOX
at org.wso2.carbon.identity.application.mgt.ApplicationManagementServiceImpl.updateApplication(ApplicationManagementServiceImpl.java:300)
Regards,
Mat
so, I found a fix to the current problem.
The caching configuration for the roles in the secondary userstore was somehow not working correctly. The UI configuration did not disable the cache, but showed it disabled. After I changed the configuration in the xml, it did disable the role caching and the problem is gone. (Now the UI works to en- and disable the cache).
However, as soon as I reactivate the caching, the problem is back again (still with a random small chance of running smoothly through the workflow).
The caching is on for the development and live environment, and we do not have problems there. So somehow, the test environment doesn't want to clear the cache/reload the data for the user roles... Strange...
So it is not ideal, but at least usable again. But if someone has a clue how to fix the caching issue, I would be glad to know!
Cheers,
Mat
What does "<domain>" refer to in the following example HTTP POST?
Surely the host should not literally be "kinesis.<region>.<domain>", but the article makes no mention of what region and domain are specifically.
Please provide a couple of literal examples for an actual "Host:" line.
POST / HTTP/1.1
Host: kinesis.<region>.<domain>
Content-Length:
User-Agent:
Content-Type: application/x-amz-json-1.1
Authorization:
Connection: Keep-Alive
X-Amz-Date:
X-Amz-Target: Kinesis_20131202.PutRecord
{
"StreamName": "exampleStreamName",
"Data": "XzxkYXRhPl8x",
"PartitionKey": "partitionKey"
}
A link to the entire article is here: http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#API_PutRecord_Examples
The example you are following is actual HTTP request being posted.
And domain simply is amazonaws.com, whereas regions could be us-west-1 or us-west-2 etc etc based on which region you are using.
So, the endpoint or what they call "host" would look something like kinesis.us-west-2.amazonaws.com.
Read moe at AWS Regions and Endpoints
Now, to put an event in the kinesis-stream, I use CLI which is cool and easy,
aws kinesis put-record --stream-name GregorSamsa --partition-key gregor-samsa --data "whatever" --region us-west-2 --profile aws-credentials
{
"ShardId": "shardId-000000000000",
"SequenceNumber": "49572068906441151838856607979148753861106406028656771074"
}
And too see what actual http request it posts run it in the --verbose mode,
$ aws kinesis put-record --stream-name GregorSamsa --partition-key gregor-samsa --data "whatever" --region us-west-2 --profile aws-credentials --debug
2017-04-16 19:48:34,993 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.66 Python/2.7.10 Darwin/15.6.0 botocore/1.5.29
2017-04-16 19:48:34,994 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['kinesis', 'put-record', '--stream-name', 'a0135-GregorSamsa', '--partition-key', 'gregor-samsa', '--data', 'whatever', '--region', 'us-west-2', '--profile', 'aws-credentials', '--debug']
2017-04-16 19:48:34,994 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x103039938>
2017-04-16 19:48:34,994 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x102d9ac08>
2017-04-16 19:48:34,994 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2017-04-16 19:48:34,994 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/service-2.json
2017-04-16 19:48:34,998 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.kinesis: calling handler <function register_retries_for_service at 0x1029992a8>
2017-04-16 19:48:34,998 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: kinesis
2017-04-16 19:48:34,999 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kinesis: calling handler <function add_waiters at 0x103043a28>
2017-04-16 19:48:35,002 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/waiters-2.json
2017-04-16 19:48:35,003 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'stream-name', <awscli.arguments.CLIArgument object at 0x1033c7110>), (u'data', <awscli.arguments.CLIArgument object at 0x1033c7150>), (u'partition-key', <awscli.arguments.CLIArgument object at 0x1033c7190>), (u'explicit-hash-key', <awscli.arguments.CLIArgument object at 0x1033c71d0>), (u'sequence-number-for-ordering', <awscli.arguments.CLIArgument object at 0x1033c7210>)])
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_streaming_output_arg at 0x103039b90>
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_cli_input_json at 0x102d9fa28>
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function unify_paging_params at 0x102fc52a8>
2017-04-16 19:48:35,005 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/paginators-1.json
2017-04-16 19:48:35,005 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_generate_skeleton at 0x102fb5668>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kinesis.put-record: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1033c7290>>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kinesis.put-record: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1033bae10>>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.stream-name: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'a0135-GregorSamsa' for parameter "stream_name": u'a0135-GregorSamsa'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.data: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'whatever' for parameter "data": u'whatever'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.partition-key: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'gregor-samsa' for parameter "partition_key": u'gregor-samsa'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.explicit-hash-key: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.sequence-number-for-ordering: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.cli-input-json: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.generate-cli-skeleton: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event calling-command.kinesis.put-record: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1033bae10>>
2017-04-16 19:48:35,008 - MainThread - botocore.hooks - DEBUG - Event calling-command.kinesis.put-record: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1033c7290>>
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2017-04-16 19:48:35,008 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/endpoints.json
2017-04-16 19:48:35,018 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: kinesis
2017-04-16 19:48:35,019 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.kinesis: calling handler <function add_generate_presigned_url at 0x10295cc80>
2017-04-16 19:48:35,019 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-04-16 19:48:35,022 - MainThread - botocore.endpoint - DEBUG - Setting kinesis timeout as (60, 60)
2017-04-16 19:48:35,022 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.kinesis.PutRecord: calling handler <function generate_idempotent_uuid at 0x102995c80>
2017-04-16 19:48:35,023 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=PutRecord) (verify_ssl=True) with params: {'body': '{"PartitionKey": "gregor-samsa", "Data": "d2hhdGV2ZXI=", "StreamName": "a0135-GregorSamsa"}', 'url': u'https://kinesis.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'aws-cli/1.11.66 Python/2.7.10 Darwin/15.6.0 botocore/1.5.29', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'Kinesis_20131202.PutRecord'}, 'context': {'client_region': u'us-west-2', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x103530690>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2017-04-16 19:48:35,023 - MainThread - botocore.hooks - DEBUG - Event request-created.kinesis.PutRecord: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x103530650>>
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:kinesis.us-west-2.amazonaws.com
x-amz-date:20170417T024835Z
x-amz-security-token:"FQoDYXdzECQaDHw6n1Hfn5egvEAxeyK1AgClSVMQWgZr9dr0bRI4Xo3ocilODVE0SEymvty4OP0lSKu19DiVbpg4t+p4PAaWVQ6fQGyu9DGi5JkQZBsyW5JlZDFE8nOmpqZOddhrrc1KRWTstqtTtwNiLL18FMxQ43S5wdveS1YMsgYhEcuQtwknbN4sdYwNrDETDF4Ies/DGQqVhM0m9YX2Zu6nYZiuA8RLqf0RLpbPHDEKpLFE/uXPo4qIaGez3IZWU6Spu59Wt52Z7yQMoSlV4vJVrCoYpM7x9Eai0zZkynGD1FSW5i3knF7V0Y3dBdRBDFDSF3UXCF45+3yRFjJSvd+Se323dTzKXoZPc9AgvNl1gBLGYMY3pY5v5bGjPK4OkVF+vQkStWKQpodNe1lTGgqkniZlpNqLefiUuhtUVSTQPVBuTV/N4VA6MiiT0dDHBQ=="
x-amz-target:Kinesis_20131202.PutRecord
content-type;host;x-amz-date;x-amz-security-token;x-amz-target
337fb90b0529a5bc7eb6274475923e82c948363ff492f56071c948b24bf7181a
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20170417T024835Z
20170417/us-west-2/kinesis/aws4_request
da8e3f33931941015cd88df2ef325da5488ff70fb93b55eaa46349cea9f547e0
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - Signature:
d7d071a403c97b018ba63477e91fe3799ae15ba895a6b216fa898ae2b19d3bdc
2017-04-16 19:48:35,025 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-04-16 19:48:35,026 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): kinesis.us-west-2.amazonaws.com
2017-04-16 19:48:35,566 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 200 110
2017-04-16 19:48:35,567 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-requestid': 'e56a1871-3e84-a549-b404-3732842305a5', 'content-length': '110', 'x-amz-id-2': 'wartiXlESG6jqLxOILWnDGjdKF6J4Q/EXwd05kGMYbWE3nDvmjJ+U+VPahhPPrn4nw7kYKoV1MC47y4NiGTJW/UV/kuGa5wh', 'server': 'Apache-Coyote/1.1', 'date': 'Mon, 17 Apr 2017 02:48:34 GMT', 'content-type': 'application/x-amz-json-1.1'}
2017-04-16 19:48:35,567 - MainThread - botocore.parsers - DEBUG - Response body:
{"SequenceNumber":"49572068906441151838856607983585511619092105132872040450","ShardId":"shardId-000000000000"}
2017-04-16 19:48:35,567 - MainThread - botocore.hooks - DEBUG - Event needs-retry.kinesis.PutRecord: calling handler <botocore.retryhandler.RetryHandler object at 0x103315510>
2017-04-16 19:48:35,568 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-04-16 19:48:35,568 - MainThread - awscli.formatter - DEBUG - RequestId: e56a1871-3e84-a549-b404-3732842305a5
{
"ShardId": "shardId-000000000000",
"SequenceNumber": "49572068906441151838856607983585511619092105132872040450"
}
AWS profile(--profile) is used for authenticating to kinesis stream, put in ~/.aws/credentials file
example,
[aws-credentials]
aws_access_key_id=ASIAI7QTMT4VDPQBZVSS
aws_secret_access_key=bsYA6yZGBIRTiqfJP6V2YPOl0qTvS/bLmFdZd6bG
aws_session_token="FQoDYXdzECQaDCBVBForaMuopUR+LyK1AhN4L2BZ4GCQ8prX/gtWN7vZI6pa+Ltv3xetSnDXxPCLeCAXTj12ENX/VldWfzvGPQ8NbpP2GGHPUHVNNU5bJyE6umLy/nBEDV7jJiSW2eBOD6X0d33GfJ2G893iIwFHu6IfrXiSwoQs8BuulG/ygOYZa6GWbYxagwy69HFMAtWTnGKs/L6dRikgpU6KoyVkBwfnEN0jKWcIQKd1VcED1+iIcbbDROl9P1NcxQIBnJDdV3qYk1+jLTu/hPoB8kwcKZOj/bZAnhGto3nPMb8vkQ/uLgZ5GA7e/JBhsnMLIyqrUVdd0txokHylVCajHPo//vYDaO7/NrkHeaizt2zCGkPNqm5XvCVrQvp2cJMFV850A2b7Ilq3N/KHr0iiVy5m4CHkC5BS9eqzAq8WuR3/D/mOy9slrCiH2tDHBQ=="
aws_security_token="FQoDYXdzECQaDCBVBForaMuopUR+LyK1AhN4L2BZ4GCQ8prX/gtWN7vZI6pa+Ltv3xetSnDXxPCLeCAXTj12ENX/VldWfzvGPQ8NbpP2GGHPKHVNNU5bJyE6umLy/nBEDV7jJiSW2eBOD6X0d33GfJ2G893iIwFHu6IfrXiSwoQ8pBuulG/ygOYZa6GWbYxagwy69HFMAtWTnGKs/L6dRikgpU6KoyVkBwfnEN0jKWcIQKd1VcED1+iIcbbDROl9P1NcxQIBnJDdV3qYk1+jLTu/hPoB8kwcKZOj/bZAnhGto3nPMb8vkQ/uGpZ5GA7e/JBhsnMLIyqrUVdd0txokHylVCajHPo//vYDaO7/NrkHeaizt2zCGkPNqm5XvCVrQvp2cJMFV850A2b7Ilq3N/KHr0iiVy5m4CHkC5BS9eqzAq8WuR3/D/mOy9slrCiH2tDHBQ=="