Amazon S3 client connect through proxy - putObject getting NullPointerException - proxy

I've got this simple piece of code which is trying to upload a file to Amazon S3 via a proxy. This is the code:
BasicAWSCredentials basicCred = new BasicAWSCredentials("my_access_key", "my_secret_key");
ClientConfiguration clientCfg = new ClientConfiguration();
clientCfg.setProtocol(Protocol.HTTP);
//setup proxy connection:
clientCfg.setProxyHost("192.168.2.12");
clientCfg.setProxyPort(80);
AmazonS3 s3 = new AmazonS3Client(basicCred, clientCfg);
String bucketName = "mybucket";
String key = "/test/Capture.JPG";
File file = new File("d:/Test_Data/Capture.JPG");
System.out.println("Uploading a new object to S3 from a file");
s3.putObject(new PutObjectRequest(bucketName, key, file));
However this is the error I got from running the program:
Exception in thread "main" java.lang.NullPointerException
at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:69)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1066)
at javaapplication8.JavaApplication8.main(JavaApplication8.java:48)
Java Result: 1
I'm using the latest aws 1.3.8 sdk from amazon. The proxy is set up in another PC next to me and it's just a simple Javascript proxy (http://www.catonmat.net/http-proxy-in-nodejs/)
I can't figure it out why. Somebody can help me, please?

here is another approach,
ClientConfiguration config = new ClientConfiguration();
config.setProtocol(Protocol.HTTPS);
config.setProxyHost("YOUR_PROXY_IP");
config.setProxyPort(YOUR_PROXY_PORT);
BasicAWSCredentials creds = new BasicAWSCredentials("YOUR_KEY", "YOUR_SECRET");
s3Client = AmazonS3ClientBuilder.standard()
.withClientConfiguration(config)
.withRegion(Regions.US_EAST_2)
.withCredentials(new AWSStaticCredentialsProvider(creds))
.build();

I ran into this issue as well. I stepped through everything in a debugger and found that the cause of the problem is the Amazon S3 client code expecting a header tag called "ETag", but the response contains "Etag", with a lowercase "t".
The headers are put into a Map<String,String>, so when the Amazon S3 client code calls get("ETag"), it gets back null, which is then fed into the hexData() method and blows up.
This seems to happen to a number of people using non-Amazon S3 services or accessing Amazon's AWS S3 service through a web proxy.
I don't know of a fix other than grabbing the Amazon client source, changing the Headers.ETAG value to "Etag" and building the JAR yourself.

Change the protocol to HTTPS.
$clientCfg.setProtocol(Protocol.HTTPS);

Related

MinIO .Net client API failing with bogus General Exception

MinIO is returning a general exception when calling the API from a .net client. In the library parsing the xml fails telling me that "Client calls PutObjectAsync General Exception 'doctype' is an unexpected token. The expected token is 'DOCTYPE'" which is no help at all.
MinIO Version
2021-09-09T21:37:07Z
Uploading objects using the webconsole works as expected.
Can you share an example of the code you are using? Please make sure you are using the S3 endpoint (running on port 9000 by default) and not the console-ui endpoint (9090 by default).
Here is a simple example of how to use the .net library to connect to the MinIO running on https://play.min.io:9000.
using Minio;
// Initialize the client with access credentials.
private static MinioClient minio = new MinioClient("play.min.io",
"Q3AM3UQ867SPQQA43P2F",
"zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
).WithSSL();
// Create an async task for listing buckets.
var getListBucketsTask = minio.ListBucketsAsync();
// Iterate over the list of buckets.
foreach (Bucket bucket in getListBucketsTask.Result.Buckets)
{
Console.WriteLine(bucket.Name + " " + bucket.CreationDateDateTime);
}
Please read more at https://docs.min.io/docs/dotnet-client-quickstart-guide.html

SocketTimeOutException not thrown when making a rest-call inside AWS lamda (java)

In my current project we are using aws-lambda to make a rest call to external service and consume the response. Happy path works fine but when it comes to connection-timeout or socket-timeout it is not working as expected. Little more details below
When making a call to external system and if the read-timeout scenario happens (external system connection got established but did not receive any response from the system within 15 sec) the aws lambda keeps waiting for the response till lambda-timeout (25 sec) and returns error.
But I expect the rest-call code invoked within lamda to throw the SocketTimeOutException or related one which is not happening.
Same thing, when I tried using a sample java code (using apache's http-client implementation which is what I have used in lambda) it works perfectly fine and I am getting proper exception thrown.
Initially we tried with jersey implementation for making rest-call and thought this is having issue and changed to http-client implementation, but none of them thrown the exception as it does in sample java code.
Please let me know your suggestions or solutions if faced already.
Below is the code snippet that I use in both lambda as well as sample program for making the rest call. (this whole block is wrapped under try-catch)
HttpPost post = new HttpPost(URL);
RequestJSONObject request = new RequestJSONObject();
//setting required request payload
ObjectMapper mapper = new ObjectMapper();
String jsonStr = mapper.writeValueAsString(request);
post.setEntity(new StringEntity(jsonStr));
post.addHeader("content-type", "application/json");
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(1000)
.setSocketTimeout(3000).build();
CloseableHttpClient httpClient =
HttpClientBuilder.create().setDefaultRequestConfig(config).build();
CloseableHttpResponse response = httpClient.execute(post);
Thanks,
Ganesh Karthik C.

calling a https api from another web api

I am attaching the below piece of code which works perfectly fine in localhost but throws web exception/socket when hosted in IIS on another server.
System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 40.113.232.243:443
It was throwing the same error in local too, unless I added this line-
httpWebRequest.Proxy = WebRequest.GetSystemWebProxy();
yet it throws socketexception when hosted in iis server.
public async Task<string> Get()
{
try
{
string uri = "https://hp-reporting-*****.azurewebsites.net/********";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(uri);
httpWebRequest.Timeout = 600000;
httpWebRequest.Proxy = WebRequest.GetSystemWebProxy(); // adding this line resolved error in local but still same issue persists when hosted in iis in another server
httpWebRequest.Method = "GET";
HttpWebResponse httpResponse = (HttpWebResponse)await httpWebRequest.GetResponseAsync();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var response = streamReader.ReadToEnd();
// this is your code here...
System.Xml.Linq.XNode node = JsonConvert.DeserializeXNode(response, "Root");
return node.ToString();
}
well, look at what that line does : https://learn.microsoft.com/en-us/dotnet/api/system.net.webrequest.getsystemwebproxy?view=netframework-4.7.2
On your local machine, you have a web proxy defined in Internet Explorer which you use when making the call. On the deployed IIS you clearly don't have it.
So, either you setup the server exactly how you setup your local machine or find another way to solve this issue locally, without using that local proxy. When you get it working, then you deploy again and it will work.

Parse iOS SDK not sending application Id

I'm testing out deploying my own parse server following the steps in the Parse Server Guide. I've got the server up and running and have been able to create and fetch objects via curl. I built a simple iOS app using the Parse SDK (1.14.2). I've initialized the SDK with the app id and server url as described in the Parse Server Guide. When I try to make requests, I get back unauthorized from the server. Digging further, I noticed that the SDK is not sending the application id header to the server. I modified the SDK to send the application id header and everything works. Am I missing a configuration step somewhere?
This is because you are not passing the ClientKey. In swift 3 you would pass it like this in the didFinishLaunchingWithOptions.
// Init Parse
let configuration = ParseClientConfiguration {
$0.applicationId = PARSE_APP_KEY
$0.clientKey = PARSE_CLIENT_KEY
$0.server = PARSE_SERVER_URL
$0.isLocalDatastoreEnabled = true
}
Parse.initialize(with: configuration)
If you are falling when trying to test CloudCode, then its because your parse-server is not passing the Javascript key. So just make sure you initialize the server to do so if this issue is related to Parse.Cloud function.

Getting "Object is read only" error when setting ClientCredentials in WCF

I have a proxy object generated by Visual Studio (client side) named ServerClient. I am attempting to set ClientCredentials.UserName.UserName/Password before opening up a new connection using this code:
InstanceContext context = new InstanceContext(this);
m_client = new ServerClient(context);
m_client.ClientCredentials.UserName.UserName = "Sample";
As soon as the code hits the UserName line it fails with an "Object is read-only" error. I know this can happen if the connection is already open or faulted, but at this point I haven't called context.Open() yet.
I have configured the Bindings (which uses netTcpBinding) to use Message as it's security mode, and MessageClientCredentialType is set to UserName.
Any ideas?
I noticed that after creating an instance of the proxy class for the service, I can set the Username and Password once without errors and do a successful call to my webservice. When I then try to set the Username and Password again on the existing instance (unnecessary of course) I get the 'Object is Read-Only' error you mentioned. Setting the values once per instance lifetime worked for me.
It appears that you can only access these properties pretty early in the instanciation cycle. If I override the constructor in the proxy class (ServerClient), I'm able to set these properties:
base.ClientCredentials.UserName.UserName = "Sample";
I'm beginning to appreciate the people who suggest not using the automatically built proxies provided by VS.
here is the solution:
using SysSvcmod = System.ServiceModel.Description;
SysSvcmod.ClientCredentials clientCredentials = new SysSvcmod.ClientCredentials();
clientCredentials.UserName.UserName = "user_name";
clientCredentials.UserName.Password = "pass_word";
m_client.ChannelFactory.Endpoint.Behaviors.RemoveAt(1);
m_client.ChannelFactory.Endpoint.Behaviors.Add(clientCredentials);
I have similar code that's passing UserName fine:
FooServiceClient client = new FooServiceClient("BasicHttpBinding_IFooService");
client.ClientCredentials.UserName.UserName = "user";
client.ClientCredentials.UserName.Password = "password";
Try creating the proxy with binding name in app.config.
The correct syntax is:
// Remove the ClientCredentials behavior.
client.ChannelFactory.Endpoint.Behaviors.Remove<ClientCredentials>();
// Add a custom client credentials instance to the behaviors collection.
client.ChannelFactory.Endpoint.Behaviors.Add(new MyClientCredentials());
http://msdn.microsoft.com/en-us/library/ms730868.aspx
It worked for me.
I was facing same problem, my code started working when I changed my code i.e. assigning values to Client credential immediately after initializing Client object.
here is the solution ,
ProductClient Manager = new ProductClient();
Manager.ClientCredentials.UserName.UserName = txtUserName.Text;
Manager.ClientCredentials.UserName.Password = txtPassword.Text;
This will not happen if the service reference is added through -> Add service reference ->Advanced->Add Web Reference-> Url/wsdl (local disk file).
I was facing this issue where I was trying to create a generic method to create a clients for different end points.
Here how I achieved this.
public static T CreateClient<T>(string url) where T : class
{
EndpointAddress endPoint = new EndpointAddress(url);
CustomBinding binding = CreateCustomBinding();
T client = (T)Activator.CreateInstance(typeof(T), new object[] { binding, endPoint });
SetClientCredentials(client);
return client;
}
public static void SetClientCredentials(dynamic obj)
{
obj.ChannelFactory.Endpoint.Behaviors.Remove<ClientCredentials>();
obj.ChannelFactory.Endpoint.Behaviors.Add(new CustomCredentials());
obj.ClientCredentials.UserName.UserName = "UserId";
obj.ClientCredentials.UserName.Password = "Password";
}
I think your problem might be related to the use of the InstanceContext. I thought that was only needed for duplex communication channels from the server side.
I admit I'm not sure about this, but I think in this case you are telling the client to use an existing instance context so it thinks there is already a running service and will not allow changes.
What is driving the use of InstanceContext?
If using a duplex client, when you instantiate it the DuplexChannelFactory within the DuplexClientBase that your client is derived from is initialized with existing credentials so it can open the callback channel, which is why the credentials would be read only.
I second Mike's question and also ask why are you using NetTcpBinding if you are not going to use its inherent transport level security? Perhaps an HTTP based binding would be a better fit? That would allow you to use certificate based security which I believe can be modified after instantiation (http://msdn.microsoft.com/en-us/library/ms576164.aspx).
A shot in the dark but does netTcpBinding allow username and password validation? Try using application layer (SOAP) security using a http binding
or you could just simply check the Credentials
if (client.ClientCredentials.ClientCertificate.Certificate == null || string.IsNullOrEmpty(client.ClientCredentials.ClientCertificate.Certificate.Thumbprint))
{
client.ClientCredentials.ClientCertificate.SetCertificate(
StoreLocation.LocalMachine,
StoreName.My,
X509FindType.FindByThumbprint, ConfigurationManager.AppSettings.Get("CertificateThumbprint"));
}
In .NET 4.6 I couldn't remove the credentials using Fabienne's answer. Kept getting Compiler Error CS0308 in the Remove method. What worked for me was this:
Type endpointBehaviorType = serviceClient.ClientCredentials.GetType();
serviceClient.Endpoint.EndpointBehaviors.Remove(endpointBehaviorType);
ClientCredentials clientCredentials = new ClientCredentials();
clientCredentials.UserName.UserName = userName;
clientCredentials.UserName.Password = password;
serviceClient.Endpoint.EndpointBehaviors.Add(clientCredentials);

Resources