We have a set of microservices that share the configuration. The configuration is huge - for now it is about 500k entries and there is more to be included. Is doesn't contain just keys and values, but also objects (that we build from several entries).
For example:
we.list.key1.field1 = 1A
we.list.key1.field2 = 1B
we.list.key1.field3 = 1C
we.list.key2.field1 = 2A
we.list.key2.field2 = 2B
we.list.key2.field3 = 2C
we map to a bean as follows:
#Data
#RefreshScope
#Configuration
#ConfigurationProperties("we")
public class OurProperties {
private Map<String, OurBean> list;
}
where OurBean is defined like this:
#Data
public class OurBean {
private String field1;
private String field1;
private String field1;
}
We are trying to use Spring Cloud Config Server + Rabbit for it, but we face performance problems - too little memory, read timeouts, service start timeouts. We are trying to tune the resources to make it work.
The question is, if this is reasonable option at all. Is SCCS suitable for such amount of data?
Any other options, if we need to share the configuration data between 100+ microservices and refresh them at once, after the data changes? Data doesn't change very frequently - lets assume a few times per week.
Related
I have a spring app, that pushes data in an s3 bucket.
public class Ebook implements Serializable {
#Column(name= "cover_path", unique = true, nullable = true)
private String coverPath;
private String coverDownloadUrl;
#Value("${aws.cloudfront.region}")
private String awsCloudFrontDns;
#PostLoad
public void init(){
// I want to access the property here
System.out.println("PostConstruct");
String coverDownloadUrl = "https://"+awsCloudFrontDns+"/"+coverPath;
}
When a data is pushed, let's say my cover here, I get the key 1/test-folder/mycover.jpg which is the important part of the future http URL of the data.
When I read the data from database, I enter inside #PostLoad method and I want construct the complete URL using the cloudfront value. This value changes frequently so we don't want to save hardly in the database.
How could I do to construct my full path just after reading the data in database?
The only way to do this is to use a service that update the data after using repository to read it? For readbyId it can be a good solution, but for reading list or using other jpa methods, this solutions won't work because I have each time to create a dedicated service for the update.
It doesn't look good for Entity to depend on property.
How about EntityListener.
#Component
public class EbookEntityListener {
#Value("${aws.cloudfront.region}")
private String awsCloudFrontDns;
#PostLoad
void postload(Ebook entity) { entity.updateDns(awsCloudFrontDns); }
}
I recommend trying this way :)
In a regular entity for SDR, it takes care of all properties of an entity for you saving it to the database. But how do you handle files?
#Entity
public class User {
String name;
Set<File> myfiles; //how can I make this work?
}
#RepositoryRestResource
public interface UserRepository extends JpaRepository<User, Long> {}
How can I make it so that a User owns a list of files, can upload and download them?
This is not really possible with Spring Data/REST as it focusses on structured data; i.e. tables and associations, for the most part.
#Lob is problematic as it forces you to store your content in the database which isn't necessarily where you want to store it. The file-system or S3 might be better for example.
byte[] is also problematic if you have very large files as you will likely cause OutOfMemoryExceptions.
Instead, there is a community project called Spring Content that addresses exactly the problem you are trying to solve.
Spring Content provides the same programming paradigms as Spring Data/REST for unstructured data; i.e. images, documents, movies, etc. So, using this project you can associate one, or in your case, many "content" objects with your Spring Data entities and manage them over HTTP just like you do with your Spring Data Entities too.
Its pretty simple to add to your project, as follows:
pom.xml (boot starters also available)
<!-- Java API -->
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-fs</artifactId>
<version>1.0.0.M4</version>
</dependency>
<!-- REST API -->
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-rest</artifactId>
<version>1.0.0.M4</version>
</dependency>
Configuration
#Configuration
#EnableFilesystemStores
#Import("org.springframework.content.rest.config.RestConfiguration.class")
public class ContentConfig {
#Bean
FileSystemResourceLoader fileSystemResourceLoader() throws IOException {
return new FileSystemResourceLoader(new File("/path/to/uploaded/files").getAbsolutePath());
}
}
To associate content, modify your User entity as follows:
#Entity
public class User {
String name;
List<Image> images;
}
Add an Image entity:
#Entity
public class Image {
#ContentId
private String contentId;
#ContentLength
private long contentLength = 0L;
#MimeType
private String mimeType = "text/plain";
}
And to this add a "store" (the equivalent of a Repository but for content):
ImageStore.java
#StoreRestResource
public interface ImageStore extends FilesystemContentStore<Image, String> {}
This is all you need to create REST endpoints # /users/{userId}/images. When your application starts, Spring Content will look at your dependencies seeing Spring Content Filesystem, look at your ImageStore interface and inject a filesystem-based implementation of that interface. It will also see the Spring Content REST dependency and inject an #Controller implementation that forwards HTTP requests to your ImageStore. Just like Spring Data does for your UserRepository. This saves you having to implement any of this yourself which I think is what you are after.
So...
To manage content with the injected REST API:
curl -X POST /users/{userId}/images -F file=#/path/to/image.jpg
will store the image on the filesystem at `` and associate it with the user entity whose id is userId.
curl /users/{userId}/images/{contentId} -H "Accept: image/jpeg"
will fetch it again and so on...supports all CRUD methods and video streaming as well BTW!
There are a couple of getting started guides here. The reference guide for Spring Content Filesystem is here. And there is a tutorial video here. The coding bit starts about 1/2 way through.
A couple of additional points:
- if you use the Spring Boot Starters then you don't need the #Configuration for the most part.
- Just like Spring Data is an abstraction, so is Spring Content so you aren't limited to storing your images on the filesystem. You could store them as BLOBs in the database, or in cloud storage like S3.
HTH
I suggest you can use #Lob instead to save file data (fileData variable below)
#Entity
public class File {
#Id
#GeneratedValue(generator = "uuid")
#GenericGenerator(name = "uuid", strategy = "uuid2")
private String id;
private String fileName;
private String fileType;
#Lob
private byte[] fileData;
}
I've been wondering which kind of projections should I use, so I did a little test, which covered 5 types of projections (based on docs: https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#projections):
1. Entity projection
This is just a standard findAll() provided by Spring Data repository. Nothing fancy here.
Service:
List<SampleEntity> projections = sampleRepository.findAll();
Entity:
#Entity
#Table(name = "SAMPLE_ENTITIES")
public class SampleEntity {
#Id
private Long id;
private String name;
private String city;
private Integer age;
}
2. Constructor projection
Service:
List<NameOnlyDTO> projections = sampleRepository.findAllNameOnlyConstructorProjection();
Repository:
#Query("select new path.to.dto.NameOnlyDTO(e.name) from SampleEntity e")
List<NameOnlyDTO> findAllNameOnlyConstructorProjection();
Data transfer object:
#NoArgsConstructor
#AllArgsConstructor
public class NameOnlyDTO {
private String name;
}
3. Interface projection
Service:
List<NameOnly> projections = sampleRepository.findAllNameOnlyBy();
Repository:
List<NameOnly> findAllNameOnlyBy();
Interface:
public interface NameOnly {
String getName();
}
4. Tuple projection
Service:
List<Tuple> projections = sampleRepository.findAllNameOnlyTupleProjection();
Repository:
#Query("select e.name as name from SampleEntity e")
List<Tuple> findAllNameOnlyTupleProjection();
5. Dynamic projection
Service:
List<DynamicProjectionDTO> projections = sampleRepository.findAllBy(DynamicProjectionDTO.class);
Repository:
<T> List<T> findAllBy(Class<T> type);
Data transfer object:
public class DynamicProjectionDTO {
private String name;
public DynamicProjectionDTO(String name) {
this.name = name;
}
}
Some additional info:
The project was built using gradle spring boot plugin (version 2.0.4), which uses Spring 5.0.8 under the hood. Database: H2 in memory.
Results:
Entity projections took 161.61 ms on average out of 100 iterations.
Constructor projections took 24.84 ms on average out of 100 iterations.
Interface projections took 252.26 ms on average out of 100 iterations.
Tuple projections took 21.41 ms on average out of 100 iterations.
Dynamic projections took 23.62 ms on average out of 100 iterations.
-----------------------------------------------------------------------
One iteration retrieved (from DB) and projected 100 000 objects.
-----------------------------------------------------------------------
Notes:
It is understandable that retrieving entities takes some time. Hibernate tracks these objects for changes, lazy loading and so on.
Constructor projections are really fast and have no limitations on the DTO side, but require manual object creation in #Query annotation.
Interface projections turned out to be really slow. See question.
Tuple projections were the fastest, but are not the most convinient to play with. They need an alias in JPQL and the data has to be retrieved by calling .get("name") instead of .getName().
Dynamic projections look pretty cool and fast, but must have exactly one constructor. No more, no less. Otherwise Spring Data throws an exception, because it doesn't know which one to use (it takes constructor parameters to determine which data to retrieve from DB).
Question:
Why interface projections take longer than retrieving entities? Each interface projection returned is actually a proxy. Is it so expensive to create that proxy? If so, doesn't it defeat the main purpose of projections (since they are meant to be faster than entities)? Other projections look awesome tho. I would really love some insight on this. Thank you.
EDIT :
Here is the test repository: https://github.com/aurora-software-ks/spring-boot-projections-test in case you want to run it yourself. It is very easy to set up. Readme contains everything you need to know.
I experienced similar behavior with an older version of Spring Data and this was my take on it: https://arnoldgalovics.com/how-much-projections-can-help/
I had a talk with Oliver Gierke (Spring Data lead) and he made some improvements (that's why you get so "good" results :-) ) but basically there will be always a cost on having abstractions vs coding it manually.
This is a trade-off as everything else is. On one hand you got flexibility, easier development, less maintenance (hopefully), on the other hand you get full control, a bit uglier query model.
Each one has its Pros and Cons:
Interface projection :
Nested, dynamic and open projection allowed, but Spring generates proxy at runtime.
DTO projection :
Faster, but nested, dynamic and open projection not allowed.
I've set up a simple document model (below) along with a Spring Data repository
#Document(collection = "users")
public class UserDocument {
#Id
private String userId;
#Indexed(expireAfterSeconds=3600)
private LocalDateTime registeredDate;
}
This seems to work fine and deletes the user documents after whatever time I set expireAfterSeconds to. However, rather than typing in a value of 3600 I'd like to pull that number from a config .yml. The usual way of adding #Value("${config.file.path.of.expiry}") won't work because #Indexedrequires the value to be a runtime constant, which #Value isn't.
Is there any other way to set up document expiry dates in Spring/Mongodb that doesn't use #Indexed(expireAfterSeconds=x)?
Instead of using the Indexed annotation to declare an index which Spring Data Mongo will create on your behalf ... you could create the index explicitly using a MongoTemplate instance.
#Value("${config.file.path.of.expiry}")
long expireAfterSeconds;
...
mongoTemplate.indexOps("users").ensureIndex(
new Index().on("registeredDate", Order.ASCENDING)
.expire(expireAfterSeconds)
);
This would allow you to source the value for expireAfterSeconds from a configuration file at runtime rather than hardcoding it in an annotation.
I have a Spring Boot project with multiple databases of different years and these databases have same tables so the only difference is the year (..., DB2016, DB2017). In the controller of the application i need to return data that belong to "different" years. Moreover in future years other databases will be created (eg. in 2018 there's going to be a db named "DB2018"). So my problem is how to switch the connection among databases without creating a new datasource and a new repository every new year.
In an other question posted by me (Spring Boot - Same repository and same entity for different databases) the answer was to create different datasources and different repositories for every existing database, but in this case i want to return data from existing databases on the basis of the current year. More specifically:
SomeEntity.java
#Entity(name = "SOMETABLE")
public class SomeEntity implements Serializable {
#Id
#Column(name="ID", nullable=false)
private Integer id;
#Column(name="NAME")
private String name;
}
SomeRepository.java
public interface SomeRepository extends PagingAndSortingRepository<SomeEntity, Integer> {
#Query(nativeQuery= true, value = "SELECT * FROM SOMETABLE WHERE NAME = ?1")
List<SomeEntity> findByName(String name);
}
SomeController.java
#RequestMapping(value="/foo/{name}", method=RequestMethod.GET)
public ResponseEntity<List<SomeEntity>> findByName(#PathVariable("name") String name) {
List<SomeEntity> list = autowiredRepo.findByName(name);
return new ResponseEntity<List<SomeEntity>>(list,HttpStatus.OK);
}
application.properties
spring.datasource.url=jdbc:postgresql://localhost:5432/DB
spring.datasource.username=xxx
spring.datasource.password=xxx
So if the current year is 2017 i want something like this:
int currentyear = Calendar.getInstance().get(Calendar.YEAR);
int oldestDbYear = 2014;
List<SomeEntity> listToReturn = new LinkedList<SomeEntity>();
//the method getProperties is a custom method to get properties from a file
String url = getProperties("application.properties", "spring.datasource.url");
props.setProperty("user", getProperties("application.properties","spring.datasource.username"));
props.setProperty("password", getProperties("application.properties","spring.datasource.password"));
for (int i = currentYear, i>oldestDbYear, i--) {
//this is the connection that must be used by autowiredRepo Repository, but i don't know how to do this.
//So the repository uses different connection for every year.
Connection conn = getConnection(url+year,props);
List<SomeEntity> list_of_specific_year = autowiredRepo.findByName(name);
conn.close;
listToReturn.addAll(list_of_specific_year);
}
return listToReturn;
Hope everithing is clear
The thing that is probably most suitable to your needs here is Spring's AbstractRoutingDataSource. You do need to define multiple DataSources but you will only need a single repository. Multiple data sources is not an issue here as there is always a way to create the DataSource beans programatically at run time and register them with the application context.
How it works is you basically register a Map<Object, DataSource> inside your #Configuration class when creating your AbstractRoutingDataSource #Bean and in this case the lookup key would be the year.
Then you need create a class that implements AbstractRoutingDataSource and implement the determineCurrentLookupKey() method. Anytime a database call is made, this method is called in the current context to lookup which DataSource should be returned. In your case it sounds like you simply want to have the year as a #PathVariable in the URL and then as the implementation of determineCurrentLookupKey() grab that #PathVariable out of the URL (e.g in your controller you have mappings like #GetMapping("/{year}/foo/bar/baz")).
HttpServletRequest request = ((ServletRequestAttributes)RequestContextHolder
.getRequestAttributes()).getRequest();
HashMap templateVariables =
(HashMap)request.getAttribute(HandlerMapping.URI_TEMPLATE_VARIABLES_ATTRIBUTE);
return templateVariables.get("year");
I used this approach when writing a testing tool for a product where there were many instances running on multiple different servers and I wanted a unified programming model from my #Controllers but still wanted it to be hitting the right database for the server/deployment combination in the url. Worked like a charm.
The drawback if you are using Hibernate is that all connections will go through a single SessionFactory which will mean you can't take advantage of Hibernate's 2nd level caching as I understand it, but I guess that depends on your needs.