What policies are required when using hadoop command with S3 - hadoop

Question

I want to copy the data in on-pre to S3.
I tried to use the following command for that.
hadoop fs -Dfs.s3a.access.key=******* -Dfs.s3a.secret.key=******* -cp -f hdfs://on-pre/cluster/mydata/dt=20200601/ s3a://some-bucket/somewhere/
When I run this command, I get the following error (The path are all fakes):
cp: s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: getFileStatus on s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxx
I set S3 policies following.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9999999999:user/john"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::some-bucket/somewhere/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9999999999:user/john"
},
"Action": "s3:List*",
"Resource": "arn:aws:s3:::some-bucket",
"Condition": {
"StringLike": {
"s3:prefix": [
"somewhere/*"
]
}
}
}
]
}
What S3 policy should I set to use hadoop fs cp?

Answer 1

A ListBucket was needed for the subdirectory itself
The necessary permissions may change as noted in the comments.
However, here's the code that solved it for your reference
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9999999999:user/john"
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::some-bucket/somewhere/*"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9999999999:user/john"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::some-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"somewhere",
"somewhere/*"
]
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::9999999999:user/john"
},
"Action": [
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::some-bucket"
]
}
]
}
Note This IAM declaration worked on June 5 2020 with Hadoop 3.2.1 or earlier. Future Hadoop releases may change the rules as they or AWS change the connector's or S3's capabilities respectively.