What does "<domain>" refer to in the following example HTTP POST? - http-post

What does "<domain>" refer to in the following example HTTP POST?
Surely the host should not literally be "kinesis.<region>.<domain>", but the article makes no mention of what region and domain are specifically.
Please provide a couple of literal examples for an actual "Host:" line.
POST / HTTP/1.1
Host: kinesis.<region>.<domain>
Content-Length:
User-Agent:
Content-Type: application/x-amz-json-1.1
Authorization:
Connection: Keep-Alive
X-Amz-Date:
X-Amz-Target: Kinesis_20131202.PutRecord
{
"StreamName": "exampleStreamName",
"Data": "XzxkYXRhPl8x",
"PartitionKey": "partitionKey"
}
A link to the entire article is here: http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#API_PutRecord_Examples

The example you are following is actual HTTP request being posted.
And domain simply is amazonaws.com, whereas regions could be us-west-1 or us-west-2 etc etc based on which region you are using.
So, the endpoint or what they call "host" would look something like kinesis.us-west-2.amazonaws.com.
Read moe at AWS Regions and Endpoints
Now, to put an event in the kinesis-stream, I use CLI which is cool and easy,
aws kinesis put-record --stream-name GregorSamsa --partition-key gregor-samsa --data "whatever" --region us-west-2 --profile aws-credentials
{
"ShardId": "shardId-000000000000",
"SequenceNumber": "49572068906441151838856607979148753861106406028656771074"
}
And too see what actual http request it posts run it in the --verbose mode,
$ aws kinesis put-record --stream-name GregorSamsa --partition-key gregor-samsa --data "whatever" --region us-west-2 --profile aws-credentials --debug
2017-04-16 19:48:34,993 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.66 Python/2.7.10 Darwin/15.6.0 botocore/1.5.29
2017-04-16 19:48:34,994 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['kinesis', 'put-record', '--stream-name', 'a0135-GregorSamsa', '--partition-key', 'gregor-samsa', '--data', 'whatever', '--region', 'us-west-2', '--profile', 'aws-credentials', '--debug']
2017-04-16 19:48:34,994 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x103039938>
2017-04-16 19:48:34,994 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x102d9ac08>
2017-04-16 19:48:34,994 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2017-04-16 19:48:34,994 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/service-2.json
2017-04-16 19:48:34,998 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.kinesis: calling handler <function register_retries_for_service at 0x1029992a8>
2017-04-16 19:48:34,998 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: kinesis
2017-04-16 19:48:34,999 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kinesis: calling handler <function add_waiters at 0x103043a28>
2017-04-16 19:48:35,002 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/waiters-2.json
2017-04-16 19:48:35,003 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'stream-name', <awscli.arguments.CLIArgument object at 0x1033c7110>), (u'data', <awscli.arguments.CLIArgument object at 0x1033c7150>), (u'partition-key', <awscli.arguments.CLIArgument object at 0x1033c7190>), (u'explicit-hash-key', <awscli.arguments.CLIArgument object at 0x1033c71d0>), (u'sequence-number-for-ordering', <awscli.arguments.CLIArgument object at 0x1033c7210>)])
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_streaming_output_arg at 0x103039b90>
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_cli_input_json at 0x102d9fa28>
2017-04-16 19:48:35,003 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function unify_paging_params at 0x102fc52a8>
2017-04-16 19:48:35,005 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/kinesis/2013-12-02/paginators-1.json
2017-04-16 19:48:35,005 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kinesis.put-record: calling handler <function add_generate_skeleton at 0x102fb5668>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kinesis.put-record: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1033c7290>>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kinesis.put-record: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1033bae10>>
2017-04-16 19:48:35,006 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.stream-name: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'a0135-GregorSamsa' for parameter "stream_name": u'a0135-GregorSamsa'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.data: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'whatever' for parameter "data": u'whatever'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.partition-key: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kinesis.put-record: calling handler <awscli.argprocess.ParamShorthandParser object at 0x103049110>
2017-04-16 19:48:35,007 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'gregor-samsa' for parameter "partition_key": u'gregor-samsa'
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.explicit-hash-key: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.sequence-number-for-ordering: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.cli-input-json: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kinesis.put-record.generate-cli-skeleton: calling handler <function uri_param at 0x102c37938>
2017-04-16 19:48:35,007 - MainThread - botocore.hooks - DEBUG - Event calling-command.kinesis.put-record: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x1033bae10>>
2017-04-16 19:48:35,008 - MainThread - botocore.hooks - DEBUG - Event calling-command.kinesis.put-record: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x1033c7290>>
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-04-16 19:48:35,008 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2017-04-16 19:48:35,008 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/as18/Library/Python/2.7/lib/python/site-packages/botocore/data/endpoints.json
2017-04-16 19:48:35,018 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: kinesis
2017-04-16 19:48:35,019 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.kinesis: calling handler <function add_generate_presigned_url at 0x10295cc80>
2017-04-16 19:48:35,019 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-04-16 19:48:35,022 - MainThread - botocore.endpoint - DEBUG - Setting kinesis timeout as (60, 60)
2017-04-16 19:48:35,022 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.kinesis.PutRecord: calling handler <function generate_idempotent_uuid at 0x102995c80>
2017-04-16 19:48:35,023 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=PutRecord) (verify_ssl=True) with params: {'body': '{"PartitionKey": "gregor-samsa", "Data": "d2hhdGV2ZXI=", "StreamName": "a0135-GregorSamsa"}', 'url': u'https://kinesis.us-west-2.amazonaws.com/', 'headers': {'User-Agent': 'aws-cli/1.11.66 Python/2.7.10 Darwin/15.6.0 botocore/1.5.29', 'Content-Type': u'application/x-amz-json-1.1', 'X-Amz-Target': u'Kinesis_20131202.PutRecord'}, 'context': {'client_region': u'us-west-2', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x103530690>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2017-04-16 19:48:35,023 - MainThread - botocore.hooks - DEBUG - Event request-created.kinesis.PutRecord: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x103530650>>
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:kinesis.us-west-2.amazonaws.com
x-amz-date:20170417T024835Z
x-amz-security-token:"FQoDYXdzECQaDHw6n1Hfn5egvEAxeyK1AgClSVMQWgZr9dr0bRI4Xo3ocilODVE0SEymvty4OP0lSKu19DiVbpg4t+p4PAaWVQ6fQGyu9DGi5JkQZBsyW5JlZDFE8nOmpqZOddhrrc1KRWTstqtTtwNiLL18FMxQ43S5wdveS1YMsgYhEcuQtwknbN4sdYwNrDETDF4Ies/DGQqVhM0m9YX2Zu6nYZiuA8RLqf0RLpbPHDEKpLFE/uXPo4qIaGez3IZWU6Spu59Wt52Z7yQMoSlV4vJVrCoYpM7x9Eai0zZkynGD1FSW5i3knF7V0Y3dBdRBDFDSF3UXCF45+3yRFjJSvd+Se323dTzKXoZPc9AgvNl1gBLGYMY3pY5v5bGjPK4OkVF+vQkStWKQpodNe1lTGgqkniZlpNqLefiUuhtUVSTQPVBuTV/N4VA6MiiT0dDHBQ=="
x-amz-target:Kinesis_20131202.PutRecord
content-type;host;x-amz-date;x-amz-security-token;x-amz-target
337fb90b0529a5bc7eb6274475923e82c948363ff492f56071c948b24bf7181a
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20170417T024835Z
20170417/us-west-2/kinesis/aws4_request
da8e3f33931941015cd88df2ef325da5488ff70fb93b55eaa46349cea9f547e0
2017-04-16 19:48:35,023 - MainThread - botocore.auth - DEBUG - Signature:
d7d071a403c97b018ba63477e91fe3799ae15ba895a6b216fa898ae2b19d3bdc
2017-04-16 19:48:35,025 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-04-16 19:48:35,026 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): kinesis.us-west-2.amazonaws.com
2017-04-16 19:48:35,566 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 200 110
2017-04-16 19:48:35,567 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-requestid': 'e56a1871-3e84-a549-b404-3732842305a5', 'content-length': '110', 'x-amz-id-2': 'wartiXlESG6jqLxOILWnDGjdKF6J4Q/EXwd05kGMYbWE3nDvmjJ+U+VPahhPPrn4nw7kYKoV1MC47y4NiGTJW/UV/kuGa5wh', 'server': 'Apache-Coyote/1.1', 'date': 'Mon, 17 Apr 2017 02:48:34 GMT', 'content-type': 'application/x-amz-json-1.1'}
2017-04-16 19:48:35,567 - MainThread - botocore.parsers - DEBUG - Response body:
{"SequenceNumber":"49572068906441151838856607983585511619092105132872040450","ShardId":"shardId-000000000000"}
2017-04-16 19:48:35,567 - MainThread - botocore.hooks - DEBUG - Event needs-retry.kinesis.PutRecord: calling handler <botocore.retryhandler.RetryHandler object at 0x103315510>
2017-04-16 19:48:35,568 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-04-16 19:48:35,568 - MainThread - awscli.formatter - DEBUG - RequestId: e56a1871-3e84-a549-b404-3732842305a5
{
"ShardId": "shardId-000000000000",
"SequenceNumber": "49572068906441151838856607983585511619092105132872040450"
}
AWS profile(--profile) is used for authenticating to kinesis stream, put in ~/.aws/credentials file
example,
[aws-credentials]
aws_access_key_id=ASIAI7QTMT4VDPQBZVSS
aws_secret_access_key=bsYA6yZGBIRTiqfJP6V2YPOl0qTvS/bLmFdZd6bG
aws_session_token="FQoDYXdzECQaDCBVBForaMuopUR+LyK1AhN4L2BZ4GCQ8prX/gtWN7vZI6pa+Ltv3xetSnDXxPCLeCAXTj12ENX/VldWfzvGPQ8NbpP2GGHPUHVNNU5bJyE6umLy/nBEDV7jJiSW2eBOD6X0d33GfJ2G893iIwFHu6IfrXiSwoQs8BuulG/ygOYZa6GWbYxagwy69HFMAtWTnGKs/L6dRikgpU6KoyVkBwfnEN0jKWcIQKd1VcED1+iIcbbDROl9P1NcxQIBnJDdV3qYk1+jLTu/hPoB8kwcKZOj/bZAnhGto3nPMb8vkQ/uLgZ5GA7e/JBhsnMLIyqrUVdd0txokHylVCajHPo//vYDaO7/NrkHeaizt2zCGkPNqm5XvCVrQvp2cJMFV850A2b7Ilq3N/KHr0iiVy5m4CHkC5BS9eqzAq8WuR3/D/mOy9slrCiH2tDHBQ=="
aws_security_token="FQoDYXdzECQaDCBVBForaMuopUR+LyK1AhN4L2BZ4GCQ8prX/gtWN7vZI6pa+Ltv3xetSnDXxPCLeCAXTj12ENX/VldWfzvGPQ8NbpP2GGHPKHVNNU5bJyE6umLy/nBEDV7jJiSW2eBOD6X0d33GfJ2G893iIwFHu6IfrXiSwoQ8pBuulG/ygOYZa6GWbYxagwy69HFMAtWTnGKs/L6dRikgpU6KoyVkBwfnEN0jKWcIQKd1VcED1+iIcbbDROl9P1NcxQIBnJDdV3qYk1+jLTu/hPoB8kwcKZOj/bZAnhGto3nPMb8vkQ/uGpZ5GA7e/JBhsnMLIyqrUVdd0txokHylVCajHPo//vYDaO7/NrkHeaizt2zCGkPNqm5XvCVrQvp2cJMFV850A2b7Ilq3N/KHr0iiVy5m4CHkC5BS9eqzAq8WuR3/D/mOy9slrCiH2tDHBQ=="

Related

Amazon S3 download fails: RequestTimeTooSkewed

I want to download an Amazon S3 folder, from within a bucket, via CLI. Since you cannot download sub-folders via. Console.
$ aws --version
aws-cli/1.20.30 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.30
Update: My RequestTime is 1 hour behind ServerTime.
RequestTime: 2021-08-25T20:17:12Z
ServerTime: 2021-08-26T12:13:31Z
How do I increase my RequestTime?
I've successfully placed my access key pair into aws configure command.
$ aws configure
AWS Access Key ID [None]: <access_key_id>
AWS Secret Access Key [None]: <secret_access_key>
Default region name [None]:
Default output format [None]:
I have an S3 URL, of this structure: s3://parent_filepath/foo/bar/folder_of_interest/
Execution with --debug:
danielbellio#LAPTOP-G1DAPU88:~$ aws s3 cp --recursive --debug s3://iotahoe-datascience/python_workers/dl_test/data_for_daniel/ .
2021-08-25 21:17:12,125 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29
2021-08-25 21:17:12,125 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', '--recursive', '--debug', 's3://iotahoe-datascience/python_workers/dl_test/data_for_daniel/', '.']
2021-08-25 21:17:12,126 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f1db9b1ce50>
2021-08-25 21:17:12,126 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f1dba3605e0>
2021-08-25 21:17:12,127 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f1dba2c24c0>
2021-08-25 21:17:12,131 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-08-25 21:17:12,134 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f1db9c57550>
2021-08-25 21:17:12,135 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f1db9b259d0>
2021-08-25 21:17:12,137 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,137 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x7f1db9b259d0>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,142 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,143 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,143 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,144 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,144 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,145 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,146 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,147 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,148 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,148 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,149 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,150 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,150 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,151 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,151 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,153 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,154 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,154 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,155 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,156 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,156 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,157 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,157 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,158 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f1dba2ff2b0>
2021-08-25 21:17:12,159 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,159 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,160 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,160 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,161 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f1db9a7f370>
2021-08-25 21:17:12,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-08-25 21:17:12,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-08-25 21:17:12,162 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-08-25 21:17:12,175 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2021-08-25 21:17:12,178 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/endpoints.json
2021-08-25 21:17:12,192 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,198 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/service-2.json
2021-08-25 21:17:12,241 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,242 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,246 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,250 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/_retry.json
2021-08-25 21:17:12,251 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,253 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,255 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,255 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,259 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,261 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,263 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f1dba785670>
2021-08-25 21:17:12,264 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f1dba7b0ca0>
2021-08-25 21:17:12,264 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f1dba7b0a60>
2021-08-25 21:17:12,269 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2021-08-25 21:17:12,270 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2021-08-25 21:17:12,272 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2021-08-25 21:17:12,285 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/danielbellio/.local/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7f1dba735b80>
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7f1dba7298b0>
2021-08-25 21:17:12,288 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,289 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f1db9577700>>
2021-08-25 21:17:12,289 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7f1dba729700>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7f1dba729c10>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,290 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7f1dba735f70>
2021-08-25 21:17:12,291 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/iotahoe-datascience?list-type=2', 'query_string': {'prefix': 'python_workers/dl_test/data_for_daniel/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29'}, 'body': b'', 'url': 'https://s3.amazonaws.com/iotahoe-datascience?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f1db95bab20>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'iotahoe-datascience'}}}
2021-08-25 21:17:12,291 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1db95baa90>>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1db95e0d90>>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7f1dba7295e0>
2021-08-25 21:17:12,292 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f1db9577790>>
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/iotahoe-datascience?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url
2021-08-25 21:17:12,293 - MainThread - botocore.utils - DEBUG - URI updated to: https://iotahoe-datascience.s3.amazonaws.com/?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
encoding-type=url&list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F
host:iotahoe-datascience.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210825T201712Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-08-25 21:17:12,294 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210825T201712Z
20210825/us-east-1/s3/aws4_request
e4f14fd5d86caf530371707ca7a13db8451306584538e8b48520e03be2d83ae0
2021-08-25 21:17:12,295 - MainThread - botocore.auth - DEBUG - Signature:
ff8a86a8720c3a2ae019399c78e6f6a2902d9323d462b57cc35ec06355f52163
2021-08-25 21:17:12,295 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://iotahoe-datascience.s3.amazonaws.com/?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/1.20.29 Python/3.8.5 Linux/5.4.72-microsoft-standard-WSL2 botocore/1.21.29', 'X-Amz-Date': b'20210825T201712Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARAHBOZTNVW5ZZIGU/20210825/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ff8a86a8720c3a2ae019399c78e6f6a2902d9323d462b57cc35ec06355f52163'}>
2021-08-25 21:17:12,297 - MainThread - botocore.httpsession - DEBUG - Certificate path: /home/danielbellio/.local/lib/python3.8/site-packages/certifi/cacert.pem
2021-08-25 21:17:12,297 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): iotahoe-datascience.s3.amazonaws.com:443
2021-08-25 21:17:12,739 - MainThread - urllib3.connectionpool - DEBUG - https://iotahoe-datascience.s3.amazonaws.com:443 "GET /?list-type=2&prefix=python_workers%2Fdl_test%2Fdata_for_daniel%2F&encoding-type=url HTTP/1.1" 403 None
2021-08-25 21:17:12,754 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'eu-central-1', 'x-amz-request-id': 'JQNECTZFFXWB7K1H', 'x-amz-id-2': 'SyNDbSFcc+D051KKti5p3A0hbpnrDinWH6v3XT3UUGXDr6OAt6VTxY0o+D8SRGzEDuhnBU93FYk=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 26 Aug 2021 12:13:29 GMT', 'Server': 'AmazonS3'}
2021-08-25 21:17:12,755 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><RequestTime>20210825T201712Z</RequestTime><ServerTime>2021-08-26T12:13:31Z</ServerTime><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>JQNECTZFFXWB7K1H</RequestId><HostId>SyNDbSFcc+D051KKti5p3A0hbpnrDinWH6v3XT3UUGXDr6OAt6VTxY0o+D8SRGzEDuhnBU93FYk=</HostId></Error>'
2021-08-25 21:17:12,763 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7f1db95775e0>
2021-08-25 21:17:12,763 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2021-08-25 21:17:12,764 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f1db9577640>>
2021-08-25 21:17:12,764 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7f1dba735ca0>
2021-08-25 21:17:12,765 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7f1db9b1c8b0>
2021-08-25 21:17:12,766 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
Traceback (most recent call last):
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/s3handler.py", line 173, in call
for fileinfo in fileinfos:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
for file_base in files:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/filegenerator.py", line 142, in call
for src_path, extra_information in file_iterator:
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/filegenerator.py", line 322, in list_objects
for key in lister.list_objects(bucket=bucket, prefix=prefix,
File "/home/danielbellio/.local/lib/python3.8/site-packages/awscli/customizations/s3/utils.py", line 416, in list_objects
for page in pages:
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/paginate.py", line 255, in __iter__
response = self._make_request(current_kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/paginate.py", line 332, in _make_request
return self._method(**current_kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/danielbellio/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
fatal error: An error occurred (RequestTimeTooSkewed) when calling the ListObjectsV2 operation: The difference between the request time and the current time is too large.
2021-08-25 21:17:12,771 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
I followed this tutorial to ensure my Windows 10 machine has Date Time automatically synced.
New account...
Solution:
sudo apt install ntpdate
sudo ntpdate -sb time.nist.gov
Source
To confirm change:
curl http://s3.amazonaws.com -v
date -u
Both times should be the same :)

RSocket reconnect is not working when the server is restarted in the mid way

I am trying to test the reconnection feature of RSocket but it is not working when the sever was brought down in between and restart.
Step 1: Started the client when the server is down
16:02:35.351 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.processId: 55104 (auto-detected)
16:02:35.515 [main] DEBUG io.netty.channel.DefaultChannelId - -Dio.netty.machineId: 10:e7:c6:ff:fe:31:38:c0 (auto-detected)
16:02:35.560 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x7b01559f] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:35.569 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x7b01559f] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
16:02:36.619 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x8a39ea36] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:36.620 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x8a39ea36] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
16:02:37.625 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x9754e5dd] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:02:37.625 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x9754e5dd] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
doAfterRetry ===>io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/127.0.0.1:7999
Step 2: Server Started . Reconnection worked
16:08:31.359 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1] Created a new pooled channel, now 1 active connections and 0 inactive connections
16:08:31.359 [reactor-tcp-nio-2] DEBUG reactor.netty.transport.TransportConfig - [id: 0x67b9f3a1] Initialized pipeline DefaultChannelPipeline{(reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
16:08:31.862 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] Registering pool release on close event for channel
16:08:31.863 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] Channel connected, now 1 active connections and 0 inactive connections
16:08:31.863 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] onStateChange(PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999]}, [connected])
16:08:31.865 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] onStateChange(ChannelOperations{PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999]}}, [configured])
16:08:31.891 [reactor-tcp-nio-2] DEBUG io.rsocket.FrameLogger - sending ->
Frame => Stream ID: 0 Type: SETUP Flags: 0b0 Length: 75
Data:
16:08:31.957 [reactor-tcp-nio-2] DEBUG reactor.netty.channel.FluxReceive - [id: 0x67b9f3a1, L:/127.0.0.1:57689 - R:localhost/127.0.0.1:7999] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
16:08:31.969 [reactor-tcp-nio-2] DEBUG io.rsocket.FrameLogger - sending ->
Frame => Stream ID: 1 Type: REQUEST_STREAM Flags: 0b100000000 Length: 57 InitialRequestN: 9223372036854775807
Metadata:
Step 3: Brought down the server
16:10:10.993 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] Channel closed, now 0 active connections and 0 inactive connections
null
16:10:10.999 [reactor-tcp-nio-2] DEBUG org.springframework.core.codec.CharSequenceEncoder - Writing "ClientName:1607847010998"
16:10:11.008 [reactor-tcp-nio-2] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException: null
16:10:11.008 [reactor-tcp-nio-2] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.nio.channels.ClosedChannelException
Caused by: java.nio.channels.ClosedChannelException: null
16:10:11.010 [reactor-tcp-nio-2] DEBUG reactor.netty.ReactorNetty - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] Non Removed handler: RSocketLengthCodec, context: ChannelHandlerContext(RSocketLengthCodec, [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999]), pipeline: DefaultChannelPipeline{(RSocketLengthCodec = io.rsocket.transport.netty.RSocketLengthCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
16:10:11.010 [reactor-tcp-nio-2] DEBUG reactor.netty.resources.DefaultPooledConnectionProvider - [id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999] onStateChange(ChannelOperations{PooledConnection{channel=[id: 0x67b9f3a1, L:/127.0.0.1:57689 ! R:localhost/127.0.0.1:7999]}}, [disconnecting])
RSocket didn't reconnect after the above step. Below is my program. Can someone pls help to review and advice what is wrong with this.
RSocketStrategies strategies = RSocketStrategies.builder()
.encoders(e -> e.add(new Jackson2JsonEncoder()))
.decoders(e -> e.add(new Jackson2JsonDecoder()))
.build();
RSocketRequester r = RSocketRequester.builder()
.rsocketConnector(connector ->
connector.reconnect(Retry.indefinitely().doAfterRetry(e-> System.out.println("doAfterRetry ===>"+e.failure())))
).dataMimeType(MediaType.APPLICATION_JSON)
.rsocketStrategies(strategies)
.tcp("localhost", 7999);
Updated on 15/Dec
Following code sends request after the step 2. After the disconnection, it wasn't able resume the stream. I am sure I must be missing something in my code. Kindly help
requester.route("route_name")
.data("RequestData")
.retrieveFlux(MyResponse.class)
.doOnError(ex ->{
System.out.println("doOnError"+ex);
}).doOnCancel(()->{
System.out.println("doOnCancel");
}).doOnComplete(()-> {
System.out.println("doOnCancel");
})
.subscribe(result -> {
System.out.println("===>"+result);
});
The reconnect method has extensive Javadoc. The main purpose of this feature is to establish a single shared connection, no matter how many subscribers there may be at one time:
When this is enabled, the connect methods of this class return a
special Mono that maintains a single, shared RSocket
The Retry determines how long to keep trying to connect before giving up, but once it stops trying, or once the connection is established and then lost, it won't automatically try to connect again.
downstream subscribers for individual requests still need their own retry logic
to determine if or when failed requests should be retried which in turn
triggers the shared reconnect
So the needs of each individual request determine whether to even try to connect again while the Retry given to reconnect determines the retry logic for each shared reconnect.
Be sure to review the corresponding code snippets in the Javadoc as well.

Why REST call with xml payload enclosed with "DeferredDocumentImpl" (spring resttemplate)

I am making a rest call using restemplate and payload is XML. (we are dealing with legacy servers) But When I debug XML at server which is a spring application (legacy, but here irrelevant) , I see that the xml payload enclosed with "DeferredDocumentImpl"
Client code:
This is a new piece which is being added and is straightforward (so may be issue is here) Also I am testing by faking xml payload jsut to check parsing:
.....
String xmlString ="<?xml version=\"1.0\" ?>\n"
+ "<myroot>\n"
//node values
+ "</myroot>";
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);
DocumentBuilder builder = factory.newDocumentBuilder();
Document document = builder.parse(new InputSource(new StringReader(xml);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.TEXT_XML);
HttpEntity<Document> request = new HttpEntity<>(document, headers);
final ResponseEntity<String> response = restTemplate.postForEntity(uri, request, String.class);
when I debugged outgoing call I see xml payload enclosed with "DeferredDocumentImpl"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /myuri HTTP/1.1[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: text/plain, application/xml, text/xml, application/json, application/*+xml, application/*+json, */*[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-Forwarded-For: value[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: text/xml[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 277[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localdev.stanfordhealthcare.org:8443[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.11 (Java/12.0.2)[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "<DeferredDocumentImpl><?xml version="1.0" encoding="UTF-16"?><myroot>[\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> " <notification>[\n]"
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> " <type>3</type>[\n]"
/////
22:17:02.879 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "</myroot></DeferredDocumentImpl>"
22:17:08.324 [poolScheduler1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Closing expired connections
Server code :
This is a working production legacy code and it does process xml payload properly .So I dont think there is issue here as it process same xml payload from other clients.
#SuppressWarnings("unchecked")
public static <T> T parseRequestObject(Class<T> pojoClass, HttpServletRequest request) {
T pojoObject = null;
String xml = IOUtils.toString(request.getReader());
...
}
So here When I debug xml I see the xml payload enclosed with "DeferredDocumentImpl"
"<DeferredDocumentImpl><?xml version="1.0" encoding="UTF-16"?><myroot> <..></DeferredDocumentImpl>"
Q:I am not sure from where this "DeferredDocumentImpl" is coming and why and how to fix it.
I solved it and the error is in a because of legacy handling of payload).
It is a REST call but content acceptable is plain text (as xml) not as XML document!
HttpHeaders headers = new HttpHeaders();
headers.add("X-Forwarded-For", "value");
headers.setContentType(MediaType.TEXT_XML);
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));//This is by default
HttpEntity<String> request = new HttpEntity<String>(xmlString, headers);

Exception: Leak detection on http-epoll-1 thread in Spring with Spring Client

I got the following exception during my Spring app.
I use Spring-boot-starter 2.2.1, with spring-boot-starer-reactor-netty 2.2.1. (Reactory netty 0.9.1 in it.)
I use spring-boot-admin-starter-client 2.2.0 within my application.
It is not thrown sequentially, and I can't see any pattern on the occurrences. I've just started my application as a unix service and after some time it threw. My idea is about the Spring admin and client, but I upgraded them to the latest but the exception still exists.
Can u give me ANY advice why is it throwing?
This exception is happened on thread or-http-epoll-1:
2019-12-12 17:24:46.756+0100 ERROR 15674 --- [or-http-epoll-1] i.n.u.ResourceLeakDetector
and on this thread everything seems alright earlier:
2019-12-12 17:23:16.763+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServer : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter#33681b9d
2019-12-12 17:23:16.763+0100 DEBUG 15674 --- [or-http-epoll-1] o.s.w.s.a.HttpWebHandlerAdapter : [967afa3f] HTTP GET "/actuator/health"
2019-12-12 17:23:16.764+0100 DEBUG 15674 --- [or-http-epoll-1] .b.a.e.w.r.WebFluxEndpointHandlerMapping : [967afa3f] Mapped to org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#6de54b40
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Decreasing pending responses, now 0
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] o.s.w.s.a.HttpWebHandlerAdapter : [967afa3f] Completed 200 OK
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP response frame
2019-12-12 17:23:17.768+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP packet was sent, terminating the channel
2019-12-12 17:24:17.959+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Decreasing pending responses, now 0
2019-12-12 17:24:17.959+0100 DEBUG 15674 --- [or-http-epoll-1] r.n.h.s.HttpServerOperations : [id: 0x967afa3f, L:/127.0.1.1:8080 - R:/127.0.0.1:54646] Last HTTP packet was sent, terminating the channel
2019-12-12 17:24:46.756+0100 ERROR 15674 --- [or-http-epoll-1] i.n.u.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.i o/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
#011io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
#011io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
#011io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
#011io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
#011io.netty.buffer.ByteBufUtil.readBytes(ByteBufUtil.java:445) #011io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:314)
#011io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
#011io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
#011io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
#011io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
#011io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
#011io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
#011io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
#011io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
#011io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
#011io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) #011io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
#011io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
#011io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
#011java.lang.Thread.run(Thread.java:748)
SOLUTION: I found a message.getPayload() what was not released, i wrote that there and the problem is not occurred anymore.

Pact Provider #State Test always returning 404

I am able to get a test running for a Spring Boot Project but I'm always getting a 404 on the #State test.
#TargetRequestFilter
public void exampleRequestFilter(HttpRequest request) {
System.out.println(request.toString());
request.addHeader("Authorization", JIMMY_CARTER_TOKEN);
}
#BeforeClass
public static void setupApplication() {
SpringApplication application = new SpringApplication(App.class);
application.setAdditionalProfiles("integration");
application.run("--server.port=9000");
}
#TestTarget
public final HttpTarget target = new HttpTarget("http", "127.0.0.1", 9000);
#State("user id") // Method will be run before testing interactions that require "default" or "no-data" state
public void toUserId() {
System.out.println("Test User Id");
}
What's strange is I can tell it's hitting the right endpoint by printing out the request information and the Authorization header. I put a debug statement in and verified that I can call with the same credentials and endpoint as the test. However the test is always failing with a 404. Is there something I'm missing in my setup?
"request": {
"method": "GET",
"path": "/api/user/XXXXXX"
},
"response": {
"status": 200,
"headers": {
"content-type": "application/vnd.api+json;charset=UTF-8"
},
"body": ...
},
"providerStates": [
{
"name": "user id"
}
]
}
You can see what requests are being made by enabling debug logging with the Apache HTTP Client and the pact-jvm libraries. For Apache HTTP Client, please refer to https://hc.apache.org/httpcomponents-client-ga/logging.html.
For an example of what the debug logs you are looking for, this is from the example ContractTest from pact-jvm (https://github.com/DiUS/pact-jvm/blob/master/pact-jvm-provider-junit/src/test/java/au/com/dius/pact/provider/junit/ContractTest.java):
13:09:20.012 [Test worker] DEBUG au.com.dius.pact.provider.ProviderClient - Making request for provider au.com.dius.pact.provider.ProviderInfo(http, localhost, 8332, /, myAwesomeService, null, null, au.com.dius.pact.provider.junit.target.HttpTarget$$Lambda$14/771479970#1dec1536, null, null, false, null, changeit, null, true, false, true, null, [], []):
13:09:20.018 [Test worker] DEBUG au.com.dius.pact.provider.ProviderClient - method: GET
path: /data
query: [:]
headers: [:]
matchers: MatchingRules(rules=[:])
generators: Generators(categories={})
body: OptionalBody(state=MISSING, value=null)
13:09:20.475 [Test worker] INFO au.com.dius.pact.provider.junit.ContractTest - exampleRequestFilter called: GET http://localhost:8332/data HTTP/1.1
13:09:20.537 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /data HTTP/1.1
13:09:20.538 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: localhost:8332
13:09:20.538 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
13:09:20.551 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_131)
13:09:20.553 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
13:09:20.553 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "GET /data HTTP/1.1[\r][\n]"
13:09:20.554 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localhost:8332[\r][\n]"
13:09:20.555 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
13:09:20.558 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_131)[\r][\n]"
13:09:20.559 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
13:09:20.560 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
13:09:20.774 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 204 No Content[\r][\n]"
13:09:20.775 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Sat, 23 Sep 2017 03:09:20 GMT[\r][\n]"
13:09:20.775 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: rest-client-driver(1.1.45)[\r][\n]"
13:09:20.779 [Test worker] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
13:09:20.784 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 204 No Content
13:09:20.785 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Sat, 23 Sep 2017 03:09:20 GMT
13:09:20.785 [Test worker] DEBUG org.apache.http.headers - http-outgoing-0 << Server: rest-client-driver(1.1.45)
13:09:20.842 [Test worker] DEBUG au.com.dius.pact.provider.ProviderClient - Received response: HTTP/1.1 204 No Content
13:09:20.867 [Test worker] DEBUG au.com.dius.pact.provider.ProviderClient - Response: [statusCode:204, headers:[Date:Sat, 23 Sep 2017 03:09:20 GMT, Server:rest-client-driver(1.1.45)]]
13:09:21.724 [Test worker] DEBUG au.com.dius.pact.model.Matching$ - Found a matcher for text/plain -> Some((text/plain,au.com.dius.pact.matchers.PlainTextBodyMatcher#29c3e77b))
returns a response which
has status code 204 (OK)
has a matching body (OK)

Resources