I want to connect two S3 Buckets from spring boot Application. I make two different beans with different credentials and make one #primary now my application runs properly but when I tried to access second bucket which is not #primary It gives me 403 Access Denied Exception
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
Below is my code any help will be highly appreciated
Thanks in Advance
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
#Configuration
public class AWSConfiguration {
#Value("${One.cloud.aws.credentials.accessKey}")
private String accessKeyOne;
#Value("${One.cloud.aws.credentials.secretKey}")
private String secretKeyOne;
#Value("${One.cloud.aws.region}")
private String regionOne;
#Value("${Two.bucket.accessKey}")
private String accessKeyTwo;
#Value("${Two.bucket.secretKey}")
private String secretKeyTwo;
#Value("${Two.bucket.region}")
private String regionTwo;
#Bean
#Primary
public BasicAWSCredentials basicAWSCredentialsOne() {
return new BasicAWSCredentials(accessKeyOne, secretKeyOne);
}
#Bean
#Primary
public AmazonS3 amazonS3ClientOne(AWSCredentials awsCredentials) {
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withCredentials(new AWSStaticCredentialsProvider(awsCredentials));
builder.setRegion(regionOne);
AmazonS3 amazonS3 = builder.build();
return amazonS3;
}
#Bean
public BasicAWSCredentials basicAWSCredentialsTwo() {
return new BasicAWSCredentials(accessKeyTwo, secretKeyTwo);
}
#Bean
public AmazonS3 amazonS3ClientTwo(AWSCredentials awsCredentials) {
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withCredentials(new AWSStaticCredentialsProvider(awsCredentials));
builder.setRegion(regionTwo);
AmazonS3 amazonS3 = builder.build();
return amazonS3;
}
}
Is it necessary to expose both BasicAWSCredentials as beans for your application? Can't you just inline the credentials like the following?
#Configuration
public class AWSConfiguration {
#Value("${One.cloud.aws.credentials.accessKey}")
private String accessKeyOne;
#Value("${One.cloud.aws.credentials.secretKey}")
private String secretKeyOne;
#Value("${One.cloud.aws.region}")
private String regionOne;
#Value("${Two.bucket.accessKey}")
private String accessKeyTwo;
#Value("${Two.bucket.secretKey}")
private String secretKeyTwo;
#Value("${Two.bucket.region}")
private String regionTwo;
#Bean
#Primary
public AmazonS3 amazonS3ClientOne(AWSCredentials awsCredentials) {
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyOne, secretKeyOne)));
builder.setRegion(regionOne);
AmazonS3 amazonS3 = builder.build();
return amazonS3;
}
#Bean
public AmazonS3 amazonS3ClientTwo(AWSCredentials awsCredentials) {
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyTwo, secretKeyTwo)));
builder.setRegion(regionTwo);
AmazonS3 amazonS3 = builder.build();
return amazonS3;
}
}
If you want to stick to your current approach with exposing both credentials as beans, you can have a look at the #Qualifier annotation to specify the correct Credentials/AmzonS3 bucket instance while injecting them, e.g.
#Bean
public AmazonS3 amazonS3ClientTwo(#Qualifier("basicAWSCredentialsTwo") AWSCredentials awsCredentials) {
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withCredentials(new AWSStaticCredentialsProvider(awsCredentials));
builder.setRegion(regionTwo);
AmazonS3 amazonS3 = builder.build();
return amazonS3;
}
There is a good tutorial on Baeldung available for this.
Related
I am exposing a SOAP web service using Spring Boot. This web service is secured using Web Service Security (WSS) which is configured with this security_policy.xml:
<xwss:SecurityConfiguration
xmlns:xwss="http://java.sun.com/xml/ns/xwss/config">
<xwss:RequireUsernameToken
passwordDigestRequired="true" nonceRequired="true" />
</xwss:SecurityConfiguration>
Until this point, the application is working just fine. It is able to authenticate successfully.
Now, I need to add a specific HTTP header based on the WSS username. It is, adds the HTTP header "x-auth-type" with the values:
"test-auth-type" when the username is "test"
"production-auth-type" when the username is "production"
"undefined-auth-type" otherwise
I thought it was easy to add an EndpointInterceptor in which I can set the HTTP header based on the user, but is not been possible to me until now.
My Web Service Configuration class looks like this:
package com.godev.soapwebserviceswithspring;
import java.util.Collections;
import java.util.List;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.ws.config.annotation.EnableWs;
import org.springframework.ws.config.annotation.WsConfigurerAdapter;
import org.springframework.ws.server.EndpointInterceptor;
import org.springframework.ws.server.endpoint.interceptor.PayloadLoggingInterceptor;
import org.springframework.ws.soap.security.xwss.XwsSecurityInterceptor;
import org.springframework.ws.soap.security.xwss.callback.SimplePasswordValidationCallbackHandler;
import org.springframework.ws.soap.server.endpoint.interceptor.PayloadValidatingInterceptor;
import org.springframework.ws.transport.http.MessageDispatcherServlet;
import org.springframework.ws.wsdl.wsdl11.DefaultWsdl11Definition;
import org.springframework.xml.xsd.SimpleXsdSchema;
import org.springframework.xml.xsd.XsdSchema;
#EnableWs
#Configuration
public class WebServiceConfig extends WsConfigurerAdapter {
private static final String WS_SCHEMA_PATH = "godev_contract.xsd";
private static final String NAMESPACE_URI = "http://godev.com/soap/webservices/demo";
#Bean
public ServletRegistrationBean<MessageDispatcherServlet> messageDispatcherServlet(
ApplicationContext applicationContext) {
MessageDispatcherServlet servlet = new MessageDispatcherServlet();
servlet.setApplicationContext(applicationContext);
servlet.setTransformWsdlLocations(true);
return new ServletRegistrationBean<>(servlet, "/ws/*");
}
#Bean(name = "xml_message")
public DefaultWsdl11Definition defaultWsdl11Definition(XsdSchema billsSchema) {
DefaultWsdl11Definition wsdl11Definition = new DefaultWsdl11Definition();
wsdl11Definition.setPortTypeName("XmlMessagePort");
wsdl11Definition.setLocationUri("/ws");
wsdl11Definition.setTargetNamespace(NAMESPACE_URI);
wsdl11Definition.setSchema(billsSchema);
return wsdl11Definition;
}
#Bean
public XsdSchema countriesSchema() {
return new SimpleXsdSchema(new ClassPathResource(WS_SCHEMA_PATH));
}
#Bean
PayloadLoggingInterceptor payloadLoggingInterceptor() {
return new PayloadLoggingInterceptor();
}
#Bean
PayloadValidatingInterceptor payloadValidatingInterceptor() {
final PayloadValidatingInterceptor payloadValidatingInterceptor = new PayloadValidatingInterceptor();
payloadValidatingInterceptor.setSchema(new ClassPathResource(WS_SCHEMA_PATH));
return payloadValidatingInterceptor;
}
#Bean
XwsSecurityInterceptor securityInterceptor() {
XwsSecurityInterceptor securityInterceptor = new XwsSecurityInterceptor();
securityInterceptor.setCallbackHandler(callbackHandler());
securityInterceptor.setPolicyConfiguration(new ClassPathResource("security_policy.xml"));
return securityInterceptor;
}
#Bean
SimplePasswordValidationCallbackHandler callbackHandler() {
SimplePasswordValidationCallbackHandler callbackHandler = new SimplePasswordValidationCallbackHandler();
callbackHandler.setUsersMap(Collections.singletonMap("admin", "pwd123"));
return callbackHandler;
}
#Override
public void addInterceptors(List<EndpointInterceptor> interceptors) {
interceptors.add(payloadLoggingInterceptor());
interceptors.add(payloadValidatingInterceptor());
interceptors.add(securityInterceptor());
}
}
My Web Service Endpoint class looks like this:
package com.godev.soapwebserviceswithspring;
import org.springframework.ws.server.endpoint.annotation.Endpoint;
import org.springframework.ws.server.endpoint.annotation.PayloadRoot;
import org.springframework.ws.server.endpoint.annotation.RequestPayload;
import org.springframework.ws.server.endpoint.annotation.ResponsePayload;
import com.godev.soap.webservices.demo.GetXmlMessageRequest;
import com.godev.soap.webservices.demo.GetXmlMessageResponse;
#Endpoint
public class XmlMessageEndpoint {
private static final String NAMESPACE_URI = "http://godev.com/soap/webservices/demo";
#PayloadRoot(namespace = NAMESPACE_URI, localPart = "getXmlMessageRequest")
#ResponsePayload
public GetXmlMessageResponse getXmlDocument(#RequestPayload GetXmlMessageRequest request) {
GetXmlMessageResponse response = new GetXmlMessageResponse();
response.setXmlMessage("<xml>empty document</xml>");
return response;
}
}
Any advice will be very appreciated!
It works for me:
Inject the Security element present in the SOAP header in the Endpoint:
#Endpoint
public class XmlMessageEndpoint {
private static final String NAMESPACE_URI = "http://godev.com/soap/webservices/demo";
#PayloadRoot(namespace = NAMESPACE_URI, localPart = "getXmlMessageRequest")
#ResponsePayload
public GetXmlMessageResponse getXmlDocument(#RequestPayload GetXmlMessageRequest request, #SoapHeader("{" + Security.SECURITY_NAMESPACE + "}Security") SoapHeaderElement securityHeader) {
GetXmlMessageResponse response = new GetXmlMessageResponse();
response.setXmlMessage("<xml>empty document</xml>");
return response;
}
In order to parse the securityHeader into something usable, you need to define a couple of POJOs. In my case, I only need the username
POJO for Security element:
#XmlAccessorType(XmlAccessType.FIELD)
#XmlRootElement(namespace = Security.SECURITY_NAMESPACE, name = "Security")
#Getter
#Setter
public class Security {
public static final String SECURITY_NAMESPACE = "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";
#XmlElement(namespace = Security.SECURITY_NAMESPACE, name = "UsernameToken")
private UsernameToken usernameToken;
}
POJO for UsernameToken element:
#XmlAccessorType(XmlAccessType.FIELD)
#XmlRootElement(namespace = Security.SECURITY_NAMESPACE, name = "UsernameToken")
#Getter
#Setter
public class UsernameToken {
#XmlElement(namespace = Security.SECURITY_NAMESPACE, name = "Username")
private String username;
}
And finally, you can parse the securityHeader using something like this:
public class SoapParser {
public static Security parseSecurityElement(SoapHeaderElement soapHeaderElement) {
Security securityElement = null;
try {
JAXBContext context = JAXBContext.newInstance(Security.class);
Unmarshaller unmarshaller = context.createUnmarshaller();
securityElement = (Security) unmarshaller.unmarshal(soapHeaderElement.getSource());
} catch (JAXBException e) {
e.printStackTrace();
}
return securityElement;
}
}
I hope, it helps!
I need to consume an API securised by OAuth2 with WebClient. I have configure the OAuth2AuthorizedClientManager to manage the access token and refresh it when it need to be.
However I encounter some issue, java.net.UnknownHostException. There is a proxy between my Application and the OAuth2 token ressource and I do not know how to configure it.
What I have try :
Test it in an other environment without proxy and it's work. My OAuth2AuthorizedClientManager configuration is correct.
System.setProperty(), not a solution, I have several proxy to manage.
maybe I am misunderstanding some OAuth2 notions
Here some code:
application.properties
spring.security.oauth2.client.registration.client.client-id=clientId
spring.security.oauth2.client.registration.client.authorization-grant-type=client_credentials
spring.security.oauth2.client.registration.client.client-secret=clientSecret
spring.security.oauth2.client.provider.client.token-uri=URI/oauth2/token
WebClientConfig
#Bean
public OAuth2AuthorizedClientManager authorizedClientManager(
ClientRegistrationRepository clientRegistrationRepository,
OAuth2AuthorizedClientService clientService)
{
OAuth2AuthorizedClientProvider authorizedClientProvider =
OAuth2AuthorizedClientProviderBuilder.builder()
.clientCredentials()
.build();
AuthorizedClientServiceOAuth2AuthorizedClientManager authorizedClientManager =
new AuthorizedClientServiceOAuth2AuthorizedClientManager(
clientRegistrationRepository, clientService);
authorizedClientManager.setAuthorizedClientProvider(authorizedClientProvider);
return authorizedClientManager;
}
#Bean
WebClient webClient(OAuth2AuthorizedClientManager authorizedClientManager) {
ServletOAuth2AuthorizedClientExchangeFilterFunction oauth2Client =
new ServletOAuth2AuthorizedClientExchangeFilterFunction(authorizedClientManager);
oauth2Client.setDefaultClientRegistrationId("client");
return WebClient.builder()
.baseUrl("URI")
.clientConnector(getReactorClientHttpConnector(url))
.apply(oauth2Client.oauth2Configuration())
.build();
}
My test
#Autowired
WebClient webClient;
public void test() {
RequestHeadersSpec<?> request = webClient.get()
.uri("/heartbeats");
}
Error
org.springframework.security.oauth2.core.OAuth2AuthorizationException: [invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: I/O error on POST request for "URI/oauth2/token": URI; nested exception is java.net.UnknownHostException: URI
My question is, How to configure a proxy for the OAuth2AuthorizedClientManager ?
Please feel free to ask for clarification.
Any help would be appreciated. Thanks
We had some similar problem in the past and solved it by following configuration.
#Configuration
public class AuthConfiguration {
#Bean
public JwtDecoderFactory<ClientRegistration> jwtDecoderFactory() {
return new CustomOidcIdTokenDecoderFactory(jwksRestTemplate());
}
#Bean
public DefaultAuthorizationCodeTokenResponseClient oAuth2AccessTokenResponseClient() {
var defaultAuthorizationCodeTokenResponseClient = new DefaultAuthorizationCodeTokenResponseClient();
defaultAuthorizationCodeTokenResponseClient.setRestOperations(tokenRestTemplate());
return defaultAuthorizationCodeTokenResponseClient;
}
#Bean
public RestTemplate jwksRestTemplate() {
return new RestTemplate(requestFactory());
}
#Bean
public RestTemplate tokenRestTemplate() {
// Copied from constructor of DefaultAuthorizationCodeTokenResponseClient
var restTemplate = new RestTemplate(Arrays.asList(
new FormHttpMessageConverter(), new OAuth2AccessTokenResponseHttpMessageConverter()));
restTemplate.setErrorHandler(new OAuth2ErrorResponseErrorHandler());
restTemplate.setRequestFactory(requestFactory());
return restTemplate;
}
private ClientHttpRequestFactory requestFactory() {
var requestFactory = new SimpleClientHttpRequestFactory();
var proxy = new Proxy(Type.HTTP, new InetSocketAddress("my.host.com", 8080));
requestFactory.setProxy(proxy);
return requestFactory;
}
}
Maybe this helps. Also the following class needs to be added because it is package private in Spring ;)
package org.springframework.security.oauth2.client.oidc.authentication;
import org.springframework.core.convert.converter.Converter;
import org.springframework.security.oauth2.client.oidc.authentication.DefaultOidcIdTokenValidatorFactory;
import org.springframework.security.oauth2.client.oidc.authentication.OidcIdTokenDecoderFactory;
import org.springframework.security.oauth2.client.registration.ClientRegistration;
import org.springframework.security.oauth2.core.OAuth2AuthenticationException;
import org.springframework.security.oauth2.core.OAuth2Error;
import org.springframework.security.oauth2.core.OAuth2TokenValidator;
import org.springframework.security.oauth2.core.converter.ClaimTypeConverter;
import org.springframework.security.oauth2.jose.jws.JwsAlgorithm;
import org.springframework.security.oauth2.jose.jws.SignatureAlgorithm;
import org.springframework.security.oauth2.jwt.Jwt;
import org.springframework.security.oauth2.jwt.JwtDecoder;
import org.springframework.security.oauth2.jwt.JwtDecoderFactory;
import org.springframework.security.oauth2.jwt.NimbusJwtDecoder;
import org.springframework.util.StringUtils;
import org.springframework.web.client.RestTemplate;
import java.util.Map;
import java.util.function.Function;
import static org.springframework.security.oauth2.jwt.NimbusJwtDecoder.withJwkSetUri;
/**
* extension for {#link OidcIdTokenDecoderFactory} to mock the JWKS request
*/
public class CustomOidcIdTokenDecoderFactory implements JwtDecoderFactory<ClientRegistration> {
private static final String MISSING_SIGNATURE_VERIFIER_ERROR_CODE = "missing_signature_verifier";
private static final Converter<Map<String, Object>, Map<String, Object>> DEFAULT_CLAIM_TYPE_CONVERTER =
new ClaimTypeConverter(OidcIdTokenDecoderFactory.createDefaultClaimTypeConverters());
private Function<ClientRegistration, JwsAlgorithm> jwsAlgorithmResolver = clientRegistration -> SignatureAlgorithm.RS256;
private Function<ClientRegistration, OAuth2TokenValidator<Jwt>> jwtValidatorFactory = new DefaultOidcIdTokenValidatorFactory();
private Function<ClientRegistration, Converter<Map<String, Object>, Map<String, Object>>> claimTypeConverterFactory =
clientRegistration -> DEFAULT_CLAIM_TYPE_CONVERTER;
private final RestTemplate restTemplate;
public CustomOidcIdTokenDecoderFactory(RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
#Override
public JwtDecoder createDecoder(ClientRegistration clientRegistration) {
NimbusJwtDecoder jwtDecoder = buildDecoder(clientRegistration);
jwtDecoder.setJwtValidator(this.jwtValidatorFactory.apply(clientRegistration));
Converter<Map<String, Object>, Map<String, Object>> claimTypeConverter =
this.claimTypeConverterFactory.apply(clientRegistration);
if (claimTypeConverter != null) {
jwtDecoder.setClaimSetConverter(claimTypeConverter);
}
return jwtDecoder;
}
private NimbusJwtDecoder buildDecoder(ClientRegistration clientRegistration) {
JwsAlgorithm jwsAlgorithm = this.jwsAlgorithmResolver.apply(clientRegistration);
String jwkSetUri = clientRegistration.getProviderDetails().getJwkSetUri();
if (!StringUtils.hasText(jwkSetUri)) {
OAuth2Error oauth2Error = new OAuth2Error(
MISSING_SIGNATURE_VERIFIER_ERROR_CODE,
"Failed to find a Signature Verifier for Client Registration: '" +
clientRegistration.getRegistrationId() +
"'. Check to ensure you have configured the JwkSet URI.",
null
);
throw new OAuth2AuthenticationException(oauth2Error, oauth2Error.toString());
}
return withJwkSetUri(jwkSetUri).jwsAlgorithm((SignatureAlgorithm) jwsAlgorithm).restOperations(restTemplate).build();
}
}
I've got an AWS S3 bucket, where I place on a weekly basis a new ZIP file.
I want to add a functionality to my existing Web Service, written with Spring Boot: synchronize the bucket locally and watch for changes.
For the time being, synchronization works well: whenever a new file is added to the bucket, it gets downloaded locally. However, I don't know to listen for file updates, this is, a method that fires when a new file is downloaded locally. Can it be done?
This is the piece of code I've:
# --------
# | AWS S3 |
# --------
s3.credentials-access-key=***
s3.credentials-secret-key=****
s3.bucket = my-bucket
s3.remote-dir = zips
s3.local-dir = D:/s3-bucket/
#Log4j2
#Configuration
public class S3Config {
public static final String OUT_CHANNEL_NAME = "s3filesChannel";
#Value("${s3.credentials-access-key}") private String accessKey;
#Value("${s3.credentials-secret-key}") private String secretKey;
#Value("${s3.remote-dir}") private String remoteDir;
#Value("${s3.bucket}") private String s3bucket;
#Value("${s3.local-dir}") private String localDir;
/*
* AWS S3
*/
#Bean
public AmazonS3 getAmazonS3(
){
BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1)
.withCredentials(new AWSStaticCredentialsProvider(creds))
.build();
return s3client;
}
#Bean
public S3SessionFactory s3SessionFactory(AmazonS3 pAmazonS3) {
return new S3SessionFactory(pAmazonS3);
}
#Bean
public S3InboundFileSynchronizer s3InboundFileSynchronizer(S3SessionFactory pS3SessionFactory) {
S3InboundFileSynchronizer sync = new S3InboundFileSynchronizer(pS3SessionFactory);
sync.setPreserveTimestamp(true);
sync.setDeleteRemoteFiles(false);
String fullRemotePath = s3bucket.concat("/").concat(remoteDir);
sync.setRemoteDirectory(fullRemotePath);
sync.setFilter(new S3RegexPatternFileListFilter(".*\\.zip$"));
return sync;
}
#Bean
#InboundChannelAdapter(value = OUT_CHANNEL_NAME, poller = #Poller(fixedDelay = "30"))
public S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource(
S3InboundFileSynchronizer pS3InboundFileSynchronizer
) {
S3InboundFileSynchronizingMessageSource messageSource = new S3InboundFileSynchronizingMessageSource(pS3InboundFileSynchronizer);
messageSource.setAutoCreateLocalDirectory(true);
messageSource.setLocalDirectory(new File(localDir));
messageSource.setLocalFilter(new AcceptOnceFileListFilter<File>());
return messageSource;
}
#Bean("s3filesChannel")
public PollableChannel s3FilesChannel() {
return new QueueChannel();
}
#Bean
public IntegrationFlow fileReadingFlow(
S3InboundFileSynchronizingMessageSource pS3InboundFileSynchronizingMessageSource,
GtfsBizkaibus pGtfsBizkaibus,
#Qualifier("fileProcessor") MessageHandler pMessageHandler) {
return IntegrationFlows
.from(pS3InboundFileSynchronizingMessageSource, e -> e.poller(p -> p.fixedDelay(5, TimeUnit.SECONDS)))
.handle(pMessageHandler)
.get();
}
#Bean("fileProcessor")
public MessageHandler fileProcessor() {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(localDir));
handler.setExpectReply(false); // end of pipeline, reply not needed
handler.setFileExistsMode(FileExistsMode.APPEND);
handler.setNewFileCallback((file, msg) -> {
log.debug("New file created... " + file.getAbsolutePath());
});
return handler;
}
You can use S3 Event Notifications and an SQS queue. Basically when an object is added to your bucket, S3 can publish an event to a registered SQS queue. You can then have your on-premise application long poll the queue for new events and process any events that are added.
See here for more information.
Actually, the S3InboundFileSynchronizingMessageSource does all the necessary work for you: when new file is added into a remote bucket, it is downloaded to local dir and produced as a payload in the message to be sent to the configured channel.
When remote file is modified, it is also downloaded to local dir.
Starting with version 5.0, the AbstractInboundFileSynchronizingMessageSource provides this option:
/**
* Switch the local {#link FileReadingMessageSource} to use its internal
* {#code FileReadingMessageSource.WatchServiceDirectoryScanner}.
* #param useWatchService the {#code boolean} flag to switch to
* {#code FileReadingMessageSource.WatchServiceDirectoryScanner} on {#code true}.
* #since 5.0
*/
public void setUseWatchService(boolean useWatchService) {
this.fileSource.setUseWatchService(useWatchService);
if (useWatchService) {
this.fileSource.setWatchEvents(
FileReadingMessageSource.WatchEventType.CREATE,
FileReadingMessageSource.WatchEventType.MODIFY,
FileReadingMessageSource.WatchEventType.DELETE);
}
}
If that makes some sense to you.
But yeah... with S3 to SQS notification it is also going to be a good solution. There is an SqsMessageDrivenChannelAdapter in Spring Integration AWS project.
Finally, as #Artem Bilian suggested, I've make use of #ServiceActivator annotation. Here it's the full example:
import java.io.File;
import java.util.concurrent.TimeUnit;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.annotation.Poller;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.aws.inbound.S3InboundFileSynchronizer;
import org.springframework.integration.aws.inbound.S3InboundFileSynchronizingMessageSource;
import org.springframework.integration.aws.support.S3SessionFactory;
import org.springframework.integration.aws.support.filters.S3RegexPatternFileListFilter;
import org.springframework.integration.channel.QueueChannel;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.file.FileWritingMessageHandler;
import org.springframework.integration.file.support.FileExistsMode;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.PollableChannel;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import lombok.extern.log4j.Log4j2;
#Log4j2
#Configuration
public class S3Config {
public static final String IN_CHANNEL_NAME = "s3filesChannel";
#Value("${s3.credentials-access-key}") private String accessKey;
#Value("${s3.credentials-secret-key}") private String secretKey;
#Value("${s3.remote-dir}") private String remoteDir;
#Value("${s3.bucket}") private String s3bucket;
#Value("${s3.local-dir}") private String localDir;
/*
* AWS S3
*/
#Bean
public AmazonS3 getAmazonS3(
){
BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3client = AmazonS3ClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1)
.withCredentials(new AWSStaticCredentialsProvider(creds))
.build();
return s3client;
}
#Bean
public S3SessionFactory s3SessionFactory(AmazonS3 pAmazonS3) {
return new S3SessionFactory(pAmazonS3);
}
#Bean
public S3InboundFileSynchronizer s3InboundFileSynchronizer(S3SessionFactory pS3SessionFactory) {
S3InboundFileSynchronizer sync = new S3InboundFileSynchronizer(pS3SessionFactory);
sync.setPreserveTimestamp(true);
sync.setDeleteRemoteFiles(false);
String fullRemotePath = s3bucket.concat("/").concat(remoteDir);
sync.setRemoteDirectory(fullRemotePath);
sync.setFilter(new S3RegexPatternFileListFilter(".*\\.zip$"));
return sync;
}
#Bean
#InboundChannelAdapter(value = IN_CHANNEL_NAME, poller = #Poller(fixedDelay = "30"))
public S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource(
S3InboundFileSynchronizer pS3InboundFileSynchronizer
) {
S3InboundFileSynchronizingMessageSource messageSource = new S3InboundFileSynchronizingMessageSource(pS3InboundFileSynchronizer);
messageSource.setAutoCreateLocalDirectory(true);
messageSource.setLocalDirectory(new File(localDir));
messageSource.setUseWatchService(true);
return messageSource;
}
#Bean("s3filesChannel")
public PollableChannel s3FilesChannel() {
return new QueueChannel();
}
#Bean
public IntegrationFlow fileReadingFlow(
S3InboundFileSynchronizingMessageSource pS3InboundFileSynchronizingMessageSource,
#Qualifier("fileProcessor") MessageHandler pMessageHandler) {
return IntegrationFlows
.from(pS3InboundFileSynchronizingMessageSource, e -> e.poller(p -> p.fixedDelay(5, TimeUnit.SECONDS)))
.handle(pMessageHandler)
.get();
}
#Bean("fileProcessor")
public MessageHandler fileProcessor() {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(localDir));
handler.setExpectReply(false); // end of pipeline, reply not needed
handler.setFileExistsMode(FileExistsMode.REPLACE);
return handler;
}
#ServiceActivator(inputChannel = IN_CHANNEL_NAME, poller = #Poller(fixedDelay = "30"))
public void asada(Message<?> message) {
// TODO
log.debug("<< New message!");
}
}
Note that I've replaced OUT_CHANNEL_NAME by IN_CHANNEL_NAME.
PS: I'm almost new to Spring Integration, so I'm still learning its concepts.
We are just starting on AWS and have requirement to use AWS ElasticCache with Redis-jedis with Spring.
Spring-data-redis 1.8.8.RELEASE
aws-java-sdk 1.11.228
Spring 4.2.9.RELEASE
jedis 2.9.0
I was able to connect and cache data to local redis with below code. I have tried making code changes as https://github.com/fishercoder1534/AmazonElastiCacheExample/tree/master/src/main/java , but not been successful. Would really appreciate some guidance and help with some sample code.
AWS ElasticCache is currently configured as option 1, but would also need to go to option 2 soon.
1. Non-replicated cluster - Redis cluster-disabled with no replicas
2. Replicated cluster - Redis cluster-enabled and Redis cluster disabled with read replicas.
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CachingConfigurerSupport;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.JdkSerializationRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import redis.clients.jedis.Jedis;
import org.springframework.cache.interceptor.KeyGenerator;
import java.lang.reflect.Method;
import java.util.List;
#Configuration
#EnableCaching
// #PropertySource("classpath:/redis.properties")
public class CacheConfig extends CachingConfigurerSupport {
// private #Value("${redis.host}") String redisHost;
// private #Value("${redis.port}") int redisPort;
//#Bean
public KeyGenerator keyGenerator() {
return new KeyGenerator() {
#Override
public Object generate(Object o, Method method, Object... objects) {
// This will generate a unique key of the class name, the method name, and all method parameters appended.
StringBuilder sb = new StringBuilder();
sb.append(o.getClass().getName());
sb.append(method.getName());
for (Object obj : objects) {
sb.append(obj.toString());
}
return sb.toString();
}
};
}
#Bean
public JedisConnectionFactory redisConnectionFactory() {
JedisConnectionFactory redisConnectionFactory = new JedisConnectionFactory();
// Defaults for redis running on Local Docker
redisConnectionFactory.setHostName("192.168.99.100");
redisConnectionFactory.setPort(6379);
return redisConnectionFactory;
}
#Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory cf) {
RedisTemplate<String, String> redisTemplate = new RedisTemplate<String, String>();
redisTemplate.setConnectionFactory(cf);
redisTemplate.setDefaultSerializer(new JdkSerializationRedisSerializer());
return redisTemplate;
}
#Bean
public CacheManager cacheManager(RedisTemplate<String, String> redisTemplate) {
RedisCacheManager cacheManager = new RedisCacheManager(redisTemplate);
// Number of seconds before expiration. Defaults to unlimited (0)
cacheManager.setDefaultExpiration(1200);
cacheManager.getCacheNames().forEach(cacheM-> {System.out.println(cacheM);});
return cacheManager;
}
}
Implemented Caching with AWS Elastic Cache + Lettuce (Redis java Client) + spring-data-redis. 3 master with 2 slaves and SSL using spring #Cachable and #CacheEvict annotation. Please provide any inputs if you see any issue or it can be done in a better way.
Spring 4.3.12.RELEASE
Spring-data-redis 1.8.8.RELEASE
aws-java-sdk 1.11.228
Lettuce (Redis java Client) 4.4.2.Final
#Configuration
#EnableCaching
public class CacheConfig extends CachingConfigurerSupport {
long expirationDate = 1200;
static AWSCredentials credentials = null;
static {
try {
//credentials = new ProfileCredentialsProvider("default").getCredentials();
credentials = new SystemPropertiesCredentialsProvider().getCredentials();
} catch (Exception e) {
System.out.println("Got exception..........");
throw new AmazonClientException("Cannot load the credentials from the credential profiles file. "
+ "Please make sure that your credentials file is at the correct "
+ "location (/Users/USERNAME/.aws/credentials), and is in valid format.", e);
}
}
#Bean
public LettuceConnectionFactory redisConnectionFactory() {
AmazonElastiCache elasticacheClient = AmazonElastiCacheClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).withRegion(Regions.US_EAST_1).build();
DescribeCacheClustersRequest dccRequest = new DescribeCacheClustersRequest();
dccRequest.setShowCacheNodeInfo(true);
DescribeCacheClustersResult clusterResult = elasticacheClient.describeCacheClusters(dccRequest);
List<CacheCluster> cacheClusters = clusterResult.getCacheClusters();
List<String> clusterNodes = new ArrayList <String> ();
try {
for (CacheCluster cacheCluster : cacheClusters) {
for (CacheNode cacheNode : cacheCluster.getCacheNodes()) {
String addr = cacheNode.getEndpoint().getAddress();
int port = cacheNode.getEndpoint().getPort();
String url = addr + ":" + port;
if(<CLUSTER NAME>.equalsIgnoreCase(cacheCluster.getReplicationGroupId()))
clusterNodes.add(url);
}
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
LettuceConnectionFactory redisConnectionFactory = new LettuceConnectionFactory(new RedisClusterConfiguration(clusterNodes));
redisConnectionFactory.setUseSsl(true);
redisConnectionFactory.afterPropertiesSet();
return redisConnectionFactory;
}
#Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory cf) {
RedisTemplate<String, String> redisTemplate = new RedisTemplate<String, String>();
redisTemplate.setConnectionFactory(cf);
redisTemplate.setDefaultSerializer(new JdkSerializationRedisSerializer());
return redisTemplate;
}
#Bean
public CacheManager cacheManager(RedisTemplate<String, String> redisTemplate) {
RedisCacheManager cacheManager = new RedisCacheManager(redisTemplate);
// Number of seconds before expiration. Defaults to unlimited (0)
cacheManager.setDefaultExpiration(expirationDate);
cacheManager.setLoadRemoteCachesOnStartup(true);
return cacheManager;
}
}
I am trying to test a Spring Web Service which is currently secured with Basic Authentication underneath. For these tests, I have written a Web Service client using Spring's WebServiceTemplate class.
My Web Service client calls to the Web Service work okay when I create the template's MessageSender as a org.springframework.ws.transport.http.CommonsHttpMessageSender object bean with org.apache.commons.httpclient.UsernamePasswordCredentials and, although the client works, the code has a warning highlighted saying that the CommonsHttpMessageSender class is now deprecated and that I should be using HttpComponentsMessageSender instead.
I have tried re-configuring the client's WebServiceTemplate to work using the newer HttpComponentsMessageSender class, but I am unable to have the basic auth part configured correctly with it. For the new HttpComponentsMessageSender class, I have created credentials using the org.apache.http.auth.UsernamePasswordCredentials class but, when I make a call to the Web Service, the credentials seem to not be available with the request? Is there a working example of a WebServiceTemplate client anywhere that uses these newer classes for authenticating requests, etc?
Jars that my working code with old deprecated classes uses: commons-httpclient-3.1, spring-ws-core-2.2.0.RELEASE.
Jars that my NON-working code with newer classes uses: httpclient-4.3.4, httpcore-4.3.2, spring-ws-core-2.2.0.RELEASE.
Test Configuration as it stands for NON-working code:
package com.company.service.a.ws.test.config;
import java.io.IOException;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.protocol.HTTP;
import org.apache.http.protocol.HttpContext;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.context.annotation.PropertySource;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import org.springframework.oxm.jaxb.Jaxb2Marshaller;
import org.springframework.ws.client.core.WebServiceTemplate;
import org.springframework.ws.soap.saaj.SaajSoapMessageFactory;
import org.springframework.ws.transport.http.HttpComponentsMessageSender;
#PropertySource("classpath:/${environment}-use-case-data.properties")
#ComponentScan(basePackages = "com.company.service.a.ws.test")
#Configuration
public class TestConfig {
#Value("${ws.url}")
private String wsUrl;
#Value("${ws.username}")
private String username;
#Value("${ws.password}")
private String password;
private static final Logger logger = LogManager.getLogger();
#Bean
public SaajSoapMessageFactory messageFactory() {
return new SaajSoapMessageFactory();
}
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setContextPath("com.company.service.a.ws.model.data");
return marshaller;
}
#Bean RequestConfig requestConfig() {
RequestConfig requestConfig = RequestConfig.custom()
.setAuthenticationEnabled(true)
.build();
return requestConfig;
}
#Bean
#DependsOn( value = "propertyConfigurer" )
public UsernamePasswordCredentials credentials() {
logger.debug("creating credentials for username: {} passowrd={}",
username, password);
UsernamePasswordCredentials credentials = new UsernamePasswordCredentials(
username, password);
return credentials;
}
#Bean
public CredentialsProvider credentialsProvider() {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, credentials());
return credentialsProvider;
}
private static class ContentLengthHeaderRemover implements HttpRequestInterceptor{
#Override
public void process(HttpRequest request, HttpContext context)
throws HttpException, IOException {
// fighting org.apache.http.protocol.RequestContent's
// ProtocolException("Content-Length header already present");
request.removeHeaders(HTTP.CONTENT_LEN);
}
}
#Bean
public HttpComponentsMessageSender messageSender() {
RequestConfig requestConfig = RequestConfig.custom()
.setAuthenticationEnabled(true)
.build();
HttpClientBuilder httpClientBuilder = HttpClients.custom();
HttpClient httpClient = httpClientBuilder
.addInterceptorFirst(new ContentLengthHeaderRemover())
.setDefaultRequestConfig(requestConfig)
.setDefaultCredentialsProvider(credentialsProvider())
.build();
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender(httpClient);
return messageSender;
}
#Bean( name = "propertyConfigurer" )
public static PropertySourcesPlaceholderConfigurer propertyConfigurer() {
PropertySourcesPlaceholderConfigurer configurer =
new PropertySourcesPlaceholderConfigurer();
return configurer;
}
#Bean
public WebServiceTemplate webServiceTemplate() {
logger.debug("creating webServiceTemplate to url: {}", wsUrl);
WebServiceTemplate webServiceTemplate = new WebServiceTemplate(messageFactory());
webServiceTemplate.setDefaultUri(wsUrl);
webServiceTemplate.setMarshaller(marshaller());
webServiceTemplate.setUnmarshaller(marshaller());
webServiceTemplate.setMessageSender(messageSender());
return webServiceTemplate;
}
}
Thanks in advance,
PM
Use HttpComponentsMessageSender with UsernamePasswordCredentials. Note that HttpComponentsMessageSender must be created as Spring bean or you must call afterPropertiesSet manually to be http client correctlly set up.
This works for me:
#Configuration
public class WsClientConfiguration {
#Bean
public ESignatureProcessorClient eSignatureProcessorClient() {
ESignatureProcessorClient client = new ESignatureProcessorClient();
client.setWebServiceTemplate(mwWebServiceTemplate());
return client;
}
#Bean
public WebServiceTemplate mwWebServiceTemplate() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setContextPath("cz.csas.services.esignatureprocessor.v02_02");
WebServiceTemplate template = new WebServiceTemplate(marshaller, marshaller);
template.setDefaultUri("https://osb-st2.vs.csin.cz:5001/CSMW/WS_MW_ESignatureProcessor_v02_02");
template.setMessageSender(defaultMwMessageSender());
return template;
}
#Bean
public HttpComponentsMessageSender defaultMwMessageSender() {
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender();
messageSender.setCredentials(new UsernamePasswordCredentials("user", "password"));
return messageSender;
}
}
This is workout for our project using org.apache.httpcomponents :
httpclient-4.5.3, httpcore-4.4.6
We create interceptor header RequestDefaultHeaders reqHeader = new RequestDefaultHeaders(headers) and then add to httpClient using .addInterceptorLast(reqHeader) when building CloseableHttpClient
Configuration class :
import org.apache.http.message.BasicHeader;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.Header;
import org.apache.http.client.protocol.RequestDefaultHeaders;
#Bean
HttpClient createHttpClient() {
List<Header> headers = new ArrayList<>();
BasicHeader authHeader = new BasicHeader("Authorization", "Basic " + base64authUserPassword());
headers.add(authHeader);
// add more header as more as needed
RequestDefaultHeaders reqHeader = new RequestDefaultHeaders(headers);
CloseableHttpClient httpClient =
HttpClients.custom()
.addInterceptorFirst(new HttpComponentsMessageSender.RemoveSoapHeadersInterceptor())
.addInterceptorLast(reqHeader)
.build();
return httpClient;
}
#Bean
public HttpComponentsMessageSender defaultMyMessageSender()
throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException {
HttpComponentsMessageSender messageSender = new HttpComponentsMessageSender(createHttpClient());
//messageSender.setCredentials(credentials());
return messageSender;
}
#Bean
WebServiceTemplate webServiceTemplate() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException{
WebServiceTemplate wsTemplate = new WebServiceTemplate();
wsTemplate.setDefaultUri(endpointURI);
wsTemplate.setMessageSender(defaultMyMessageSender());
return wsTemplate;
}
One solution I have used is to create a custom WebServiceMessageSender with a custom CredentialsProvider. This solution also sets a route planner that respects the default java proxy settings.
#Configuration
public class WebServiceConfiguration {
#Bean
public WebServiceMessageSender webServiceMessageSender(#Value("${endpoint.uri}") endpointUri,
#Value("${endpoint.username}") String username,
#Value("${endpoint.password}") String password) throws Exception {
SystemDefaultRoutePlanner routePlanner = new SystemDefaultRoutePlanner(
ProxySelector.getDefault());
BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(endpointUri.getHost(), endpointUri.getPort(), ANY_REALM, ANY_SCHEME), new UsernamePasswordCredentials(username, password););
CloseableHttpClient httpclient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.addInterceptorFirst(new HttpComponentsMessageSender.RemoveSoapHeadersInterceptor())
.setDefaultCredentialsProvider(credentialsProvider)
.build();
return new HttpComponentsMessageSender(httpclient);
}
}
In the end, to make Basic Authentication work with the Spring WebServiceTemplate in spring-ws-xxx.2.2.0.RELEASE using current httpclient-4.3.+, httpcore-4.3.+ classes, I've added a preemptive authentication interceptor to the HttpClient (as suggested by #Oliv in Preemptive Basic authentication with Apache HttpClient 4). Note that, as pointed out by #Oliv, this solution adds authentication to ALL requests made.
I am still not sure if this is the best way to configure the Spring WebServiceTemplate but it is the only way I have found (so far) of enabling preemptive authentication without direct access to the HttpClient's HttpClientContext object. Any simpler better answers I would very much welcome...
Interceptor code:
private static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context)
throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(
HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme is avaialble yet, initialize it preemptively
if ( authState.getAuthScheme() == null ) {
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(
HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(
HttpCoreContext.HTTP_TARGET_HOST);
Credentials creds = credsProvider.getCredentials(
new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if ( creds == null ) {
throw new HttpException("no credentials available for preemptive "
+ "authentication");
}
authState.update(new BasicScheme(), creds);
}
}
}
Thread is old but to summaries.
As per spring documentation:
UsernamePasswordCredentials and HttpComponentsMessageSender should be spring beans. So define beans and inject them. It should solve the problem.