I cannot connect to Amazon S3 through the IAM role. I have been told that I cannot use the secret-key or the access-key, but I can't find any way to do it without this.
Actually what I have is this:
public class S3Config {
#Autowired
Environment env;
#Value("${amazon.aws.accesskey}")
private String awsId;
#Value("${amazon.aws.secretkey}")
private String awsKey;
#Value("${amazon.aws.role}")
private String roleArn;
#Value("${amazon.aws.region}")
private String region;
#Bean
#Scope("prototype")
#Primary
public AmazonS3Client s3client() {
BasicAWSCredentials awsCreds = new BasicAWSCredentials("", "");
AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withRegion(Regions.fromName(region))
.build();
AssumeRoleRequest assumeRequest = new AssumeRoleRequest().withRoleArn(roleArn).withDurationSeconds(3600)
.withRoleSessionName("Test");
AssumeRoleResult roleResponse = stsClient.assumeRole(assumeRequest);
Credentials sessionCredentials = roleResponse.getCredentials();
BasicSessionCredentials awsCredentials = new BasicSessionCredentials(
sessionCredentials.getAccessKeyId(),
sessionCredentials.getSecretAccessKey(),
sessionCredentials.getSessionToken());
AmazonS3Client s3ClientRole = (AmazonS3Client) AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withRegion(Regions.fromName(region))
.build();
AmazonS3Client s3Client = s3ClientRole;
return s3Client;
}
}
If on the line where I create the awsCreds object, I fill it correctly with the values I get from the application.properties, if it does everything correctly. But if I leave it empty I never get to execute the line stsClient.assumeRole(assumeRequest); and it gives me connection time out error.
I think you should follow this doc: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-roles.html#default-provider-chain
It should follow you step bu step on resolving you problem.
Basically an IAM Role allows a specific "resource", such as an EC2 instance, to connect to the service and thus you configure your client to get temporary credentials just to "authenticate" and "authorize" it.
Related
Currently I have three roles, wherein each has the separate SQS access. I am connecting to AWS by using the roles.
For connecting with single role, this code is helpful.
#Value("${cloud.aws.assumeRoleARN:}")
private String assumeRoleARN;
#Autowired
private AWSCredentialsProvider awsCredentialsProvider;
#Bean
#Primary
public AWSCredentialsProvider awsCredentialsProvider() {
log.info("Assuming role {}",assumeRoleARN);
if (StringUtils.isNotEmpty(assumeRoleARN)) {
AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.standard()
.withClientConfiguration(clientConfiguration())
.withRegion("us-east-1")
.withCredentials(awsCredentialsProvider)
.build();
return new STSAssumeRoleSessionCredentialsProvider
.Builder(assumeRoleARN, "test")
.withStsClient(stsClient)
.build();
}
return awsCredentialsProvider;
}
But I need same credentials for all the three roles which I have. Is it possible to pass list of regions in the AWSSecurityTokenServiceClientBuilder?
Thanks in Advance!
I'm building a little microservice to access files from a SFTP file server. I decided to use Spring Integration SFTP to get the job done. I'm new to Spring Integration and confused with how it all works.
My goal is to get a list of files a directory on a SFTP server and present them to the user interface. From there a user will select a file for download and I'll use the filename to stream the file from the SFTP server to the user interface.
I'm using the following code which does work.
Entire class to handle SFTP with SSH
#Slf4j
#Configuration
public class SftpConfig {
#Value("${sftp.host}")
private String sftpHost;
#Value("${sftp.port:22}")
private int sftpPort;
#Value("${sftp.user}")
private String sftpUser;
#Value("${sftp.privateKey:#{null}}")
private Resource sftpPrivateKey;
#Value("${sftp.privateKeyPassphrase:}")
private String sftpPrivateKeyPassphrase;
#Value("${sftp.password:#{null}}")
private String sftpPasword;
#Value("${sftp.remote.directory:/}")
private String sftpRemoteDirectory;
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase);
} else {
factory.setPassword(sftpPasword);
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#ServiceActivator(inputChannel = "ftpLS")
#Bean
public SftpOutboundGateway getFtpLS() {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpSessionFactory(), "ls", "'" + sftpRemoteDirectory + "' + payload");
gateway.setOption(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY);
return gateway;
}
#ServiceActivator(inputChannel = "ftpGet")
#Bean
public SftpOutboundGateway getFtpGet() {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpSessionFactory(), "get", "'" + sftpRemoteDirectory + "' + payload");
gateway.setOption(AbstractRemoteFileOutboundGateway.Option.STREAM);
return gateway;
}
#MessagingGateway(defaultRequestChannel = "ftpLS")
public interface FtpLS {
List list(String directory);
}
#MessagingGateway(defaultRequestChannel = "ftpGet")
public interface FtpGet {
InputStream get(String fileName);
}
}
Run
#Bean
public ApplicationRunner runner(SftpConfig.FtpLS ftpLS, SftpConfig.FtpGet ftpGet) {
return args -> {
List<String> list = ftpLS.list("139");
System.out.println("Result:" + list);
InputStream is = ftpGet.get("139/" + list.get(0));
String theString = IOUtils.toString(is,"UTF-8");
System.out.println("Result:" + theString);
};
}
My number one question is this the correct approach?
Secondly, do I need two interfaces in order to use the two different SftpOutboundGateway's?
Lastly, is there a better way to pass in a dynamic directory name when doing a FtsGet? Right now I'm passing I'm concatenating 139 with the base directory in a string and passing it in through the interface.
is this the correct approach?
Yes, the approach is correct. Although I would suggest do not use isSharedSession for the gateway since it might be used from different threads by different users.
do I need two interfaces?
No, you really can have one #MessagingGateway, but with several methods marked with their own #Gateway annotations.
is there a better way to pass in a dynamic directory?
No, your approach is correct. There is no something like working directory to switch automatically, like we can do in FTP.
I'm pretty new to spring-integration anyway I'm using it in order to receive mails and elaborate them.
I used this spring configuration class:
#Configuration
#EnableIntegration
#PropertySource(value = { "classpath:configuration.properties" }, encoding = "UTF-8", ignoreResourceNotFound = false)
public class MailReceiverConfiguration {
private static final Log logger = LogFactory.getLog(MailReceiverConfiguration.class);
#Autowired
private EmailTransformerService emailTransformerService;
// Configurazione AE
#Bean
public MessageChannel inboundChannelAE() {
return new DirectChannel();
}
#Bean(name= {"aeProps"})
public Properties aeProps() {
Properties javaMailPropertiesAE = new Properties();
javaMailPropertiesAE.put("mail.store.protocol", "imap");
javaMailPropertiesAE.put("mail.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.auth.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.smtp.socketFactory.fallback", "false");
javaMailPropertiesAE.put("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
return javaMailPropertiesAE;
}
#Bean(name="mailReceiverAE")
public MailReceiver mailReceiverAE(#Autowired MailConfigurationBean mcb, #Autowired #Qualifier("aeProps") Properties javaMailPropertiesAE) throws Exception {
return ConfigurationUtil.getMailReceiver("imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX", new BigDecimal(2), javaMailPropertiesAE);
}
#Bean
#InboundChannelAdapter( autoStartup = "true",
channel = "inboundChannelAE",
poller = {#Poller(fixedRate = "${fixed.rate.ae}",
maxMessagesPerPoll = "${max.messages.per.poll.ae}") })
public MailReceivingMessageSource pollForEmailAE(#Autowired MailReceiver mailReceiverAE) {
MailReceivingMessageSource mrms = new MailReceivingMessageSource(mailReceiverAE);
return mrms;
}
#Transformer(inputChannel = "inboundChannelAE", outputChannel = "transformerChannelAE")
public MessageBean transformitAE( MimeMessage mailMessage ) throws Exception {
// amministratore email inbox
MessageBean messageBean = emailTransformerService.transformit(mailMessage);
return messageBean;
}
#Splitter(inputChannel = "transformerChannelAE", outputChannel = "nullChannel")
public List<Message<?>> splitIntoMessagesAE(final MessageBean mb) {
final List<Message<?>> messages = new ArrayList<Message<?>>();
for (EmailFragment emailFragment : mb.getEmailFragments()) {
Message<?> message = MessageBuilder.withPayload(emailFragment.getData())
.setHeader(FileHeaders.FILENAME, emailFragment.getFilename())
.setHeader("directory", emailFragment.getDirectory()).build();
messages.add(message);
}
return messages;
}
}
So far so good.... I start my micro-service and there is this component listening on the specified mail server and mails are downloaded.
Now I have this requirement: mail server configuration (I mean the string "imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX") must be taken from a database and it can be configurable. In any time a system administrator can change it and the mail receiver must use the new configuration.
As far as I understood I should create a new instance of MailReceiver when a new configuration is present and use it in the InboundChannelAdapter
Is there any best practice in order to do it? I found this solution: ImapMailReceiver NO STORE attempt on READ-ONLY folder (Failure) [THROTTLED];
In this solution I can inject the ThreadPoolTaskScheduler if I define it in my Configuration class; I can also inject the DirectChannel but every-time I should create a new MailReceiver and a new ImapIdleChannelAdapter without considering this WARN message I get when the
ImapIdleChannelAdapter starts:
java.lang.RuntimeException: No beanfactory at org.springframework.integration.expression.ExpressionUtils.createStandardEvaluationContext(ExpressionUtils.java:79) at org.springframework.integration.mail.AbstractMailReceiver.onInit(AbstractMailReceiver.java:403)
Is there a better way to satisfy my scenario?
Thank you
Angelo
The best way to do this is to use the Java DSL and dynamic flow registration.
Documentation here.
That way, you can unregister the old flow and register a new one, each time the configuration changes.
It will automatically handle injecting dependencies such as the bean factory.
I was trying to setup Feign to work with RibbonClient, something like MyService api = Feign.builder().client(RibbonClient.create()).target(MyService.class, "https://myAppProd");, where myAppProd is an application which I can see in Consul. Now, if I use Spring annotations for the Feign client (#FeignClient("myAppProd"), #RequestMapping), everything works as Spring Cloud module will take care of everything.
If I want to use Feign.builder() and #RequestLine, I get the error:
com.netflix.client.ClientException: Load balancer does not have available server for client: myAppProd.
My first initial thought was that Feign was built to work with Eureka and only Spring Cloud makes the integration with Consul, but I am unsure about this.
So, is there a way to make Feign work with Consul without Spring Cloud?
Thanks in advance.
In my opinion, it's not feign work with consul, its feign -> ribbon -> consul.
RibbonClient needs to find myAppProd's serverList from its LoadBalancer.
Without ServerList, error: 'does not have available server for client'.
This job has been done by SpringCloudConsul and SpringCloudRibbon project, of course you can write another adaptor, it's just some glue code. IMHO, you can import this spring dependency into your project, but use it in non-spring way . Demo code:
just write a new feign.ribbon.LBClientFactory, that generate LBClient with ConsulServerList(Spring's class).
public class ConsulLBFactory implements LBClientFactory {
private ConsulClient client;
private ConsulDiscoveryProperties properties;
public ConsulLBFactory(ConsulClient client, ConsulDiscoveryProperties consulDiscoveryProperties) {
this.client = client;
this.properties = consulDiscoveryProperties;
}
#Override
public LBClient create(String clientName) {
IClientConfig config =
ClientFactory.getNamedConfig(clientName, DisableAutoRetriesByDefaultClientConfig.class);
ConsulServerList consulServerList = new ConsulServerList(this.client, properties);
consulServerList.initWithNiwsConfig(config);
ZoneAwareLoadBalancer<ConsulServer> lb = new ZoneAwareLoadBalancer<>(config);
lb.setServersList(consulServerList.getInitialListOfServers());
lb.setServerListImpl(consulServerList);
return LBClient.create(lb, config);
}
}
and then use it in feign:
public class Demo {
public static void main(String[] args) {
ConsulLBFactory consulLBFactory = new ConsulLBFactory(
new ConsulClient(),
new ConsulDiscoveryProperties(new InetUtils(new InetUtilsProperties()))
);
RibbonClient ribbonClient = RibbonClient.builder()
.lbClientFactory(consulLBFactory)
.build();
GitHub github = Feign.builder()
.client(ribbonClient)
.decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
List<Contributor> contributors = github.contributors("OpenFeign", "feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
}
interface GitHub {
#RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(#Param("owner") String owner, #Param("repo") String repo);
}
public static class Contributor {
String login;
int contributions;
}
}
you can find this demo code here, add api.github.com to your local consul before running this demo.
Being new to Spring Boot I am wondering on how I can configure connection details for MongoDB.
I have tried the normal examples but none covers the connection details.
I want to specify the database that is going to be used and the url/port of the host that runs MongoDB.
Any hints or tips?
Just to quote Boot Docs:
You can set spring.data.mongodb.uri property to change the url, or alternatively specify a host/port. For example, you might declare the following in your application.properties:
spring.data.mongodb.host=mongoserver
spring.data.mongodb.port=27017
All available options for spring.data.mongodb prefix are fields of MongoProperties:
private String host;
private int port = DBPort.PORT;
private String uri = "mongodb://localhost/test";
private String database;
private String gridFsDatabase;
private String username;
private char[] password;
It's also important to note that MongoDB has the concept of "authentication database", which can be different than the database you are connecting to. For example, if you use the official Docker image for Mongo and specify the environment variables MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD, a user will be created on 'admin' database, which is probably not the database you want to use. In this case, you should specify parameters accordingly on your application.properties file using:
spring.data.mongodb.host=127.0.0.1
spring.data.mongodb.port=27017
spring.data.mongodb.authentication-database=admin
spring.data.mongodb.username=<username specified on MONGO_INITDB_ROOT_USERNAME>
spring.data.mongodb.password=<password specified on MONGO_INITDB_ROOT_PASSWORD>
spring.data.mongodb.database=<the db you want to use>
spring.data.mongodb.host and spring.data.mongodb.port are not supported if you’re using the Mongo 3.0 Java driver. In such cases, spring.data.mongodb.uri should be used to provide all of the configuration, like this:
spring.data.mongodb.uri=mongodb://user:secret#mongo1.example.com:12345
In a maven project create a file src/main/resources/application.yml with the following content:
spring.profiles: integration
# use local or embedded mongodb at localhost:27017
---
spring.profiles: production
spring.data.mongodb.uri: mongodb://<user>:<passwd>#<host>:<port>/<dbname>
Spring Boot will automatically use this file to configure your application. Then you can start your spring boot application either with the integration profile (and use your local MongoDB)
java -jar -Dspring.profiles.active=integration your-app.jar
or with the production profile (and use your production MongoDB)
java -jar -Dspring.profiles.active=production your-app.jar
You can define more details by extending AbstractMongoConfiguration.
#Configuration
#EnableMongoRepositories("demo.mongo.model")
public class SpringMongoConfig extends AbstractMongoConfiguration {
#Value("${spring.profiles.active}")
private String profileActive;
#Value("${spring.application.name}")
private String proAppName;
#Value("${spring.data.mongodb.host}")
private String mongoHost;
#Value("${spring.data.mongodb.port}")
private String mongoPort;
#Value("${spring.data.mongodb.database}")
private String mongoDB;
#Override
public MongoMappingContext mongoMappingContext()
throws ClassNotFoundException {
// TODO Auto-generated method stub
return super.mongoMappingContext();
}
#Override
#Bean
public Mongo mongo() throws Exception {
return new MongoClient(mongoHost + ":" + mongoPort);
}
#Override
protected String getDatabaseName() {
// TODO Auto-generated method stub
return mongoDB;
}
}
In case that somebody is trying to connect to a Atlas MongoDB Cluster in application.properties has to have a config like:
spring.data.mongodb.uri=mongodb+srv://databaseUsername:usernamePassword#cluster0.j4koa.mongodb.net/databaseUsername?retryWrites=true&w=majority
In my case I needed to set up MongoDB for integration tests using Testcontainers. Using properites file was not an option since Mongo port had to be specified during runtime. I wanted to preseve original MongoDB autoconfiguration provided by SpringBoot but override some of the properties. This can be achieved by defining a bean of type MongoClientSettingsBuilderCustomizer which can be used to customize mongo settings :
#Bean
public MongoClientSettingsBuilderCustomizer clientSettingsBuilderCustomizer(final GenericContainer<?> mongoDBContainer) {
String database = environment.getProperty("spring.data.mongodb.database");
ConnectionString connectionString = new ConnectionString(String.format("mongodb://localhost:%s/%s", mongoDBContainer.getFirstMappedPort(), database));
return settings -> settings.applyConnectionString(connectionString);
}
If you simply want to read a MongoDB connection string from an environment variable, one way is to set the following environment variable:
SPRING_DATA_MONGODB_URI=mongodb://localhost:27017/trying-mongo
This doesn't require any changes in the application.properties as the spring data mongo will read the value from the above environment variable by default.
Here is How you can do in Spring Boot 2.0 by creating custom MongoClient adding Providing more control for Connection ,
Please follow github Link for Full Source Code
#Configuration
#EnableMongoRepositories(basePackages = { "com.frugalis.repository" })
#ComponentScan(basePackages = { "com.frugalis.*" })
#PropertySource("classpath:application.properties")
public class MongoJPAConfig extends AbstractMongoConfiguration {
#Value("${com.frugalis.mongo.database}")
private String database;
#Value("${com.frugalis.mongo.server}")
private String host;
#Value("${com.frugalis.mongo.port}")
private String port;
#Value("${com.frugalis.mongo.username}")
private String username;
#Value("${com.frugalis.mongo.password}")
private String password;
#Override
protected String getDatabaseName() {
return database;
}
#Override
protected String getMappingBasePackage() {
return "com.frugalis.entity.mongo";
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
return new MongoTemplate(mongoClient(), getDatabaseName());
}
#Override
#Bean
public MongoClient mongoClient() {
List<MongoCredential> allCred = new ArrayList<MongoCredential>();
System.out.println("???????????????????"+username+" "+database+" "+password+" "+host+" "+port);
allCred.add(MongoCredential.createCredential(username, database, password.toCharArray()));
MongoClient client = new MongoClient((new ServerAddress(host, Integer.parseInt(port))), allCred);
client.setWriteConcern(WriteConcern.ACKNOWLEDGED);
return client;
}}