I recently discovered GridFS which I'd like to use for file storage with metadata. I just wondered if it's possible to use a MongoRepository to query GridFS? If yes, can someone give me an example?
I'd also take a solution using Hibernate, if there is some.
The reason is: My metadata contains a lot of different fields and it would be much easier to query a repository than to write some new Query(Criteria.where(...)) for each scenario. And I hopefully could also simply take a Java object and provide it via REST API without the file itself.
EDIT: I'm using
Spring 4 Beta
Spring Data Mongo 1.3.1
Hibernate 4.3 Beta
There is a way to solve this:
#Document(collection="fs.files")
public class MyGridFsFile {
#Id
private ObjectId id;
public ObjectId getId() { return id; }
private String filename;
public String getFilename() { return filename; }
private long length;
public long getLength() { return length; }
...
}
You can write a normal Spring Mongo Repo for that. Now you can at least query the fs.files collection using a Spring Data Repo. But: You cannot access the file contents this way.
For getting the file contents itself, you've got (at least) 2 options:
Use file = gridOperations.findOne(Query.query(Criteria.where("_id").is(id))); InputStream is = file.getInputStream();
Have a look at the source code of GridFSDBFile. There you can see, how it internally queries the fs.chunks collection and fills the InputStream.
(Option 2 is really low level, Option 1 is a lot easier and this code gets maintained by the MongoDB-Java-Driver devs, though Option 1 would be my choice).
Updating GridFS entries:
GridFS is not designed to update file content!
Though only updating the metadata field can be useful. The rest of the fields is kinda static.
You should be able to simply use your custom MyGridFsFileRepo's update method. I suggest to only create a setter for the metadata field.
Different metadata for different files:
I solved this using an abstract MyGridFsFile class with generic metadata, i.e.:
#Document(collection="fs.files")
public abstract class AbstractMyGridFsFile<M extends AbstractMetadata> {
...
private M metadata;
public M getMetadata() { return metadata; }
void setMetadata(M metadata) { this.metadata = metadata; }
}
And of course each impl has its own AbstractMetadata impl associated. What have I done? AbstractMetadata always has a field called type. This way I can find the right AbstractMyGridFsFile impl. Though I have also a generic abstract repository.
Btw: In the meantime I switched here from using Spring Repo, to use plain access via MongoTemplate, like:
protected List<A> findAll(Collection<ObjectId> ids) {
List<A> files = mongoTemplate.find(Query.query(Criteria
.where("_id").in(ids)
.and("metadata.type").is(type) // this is hardcoded for each repo impl
), typeClass); // this is the corresponding impl of AbstractMyGridFsFile
return files;
}
Hope this helps. I can write more, if you need more information about this. Just tell me.
You can create a GridFS object with the database from your MongoTemplate, and then interact with that:
MongoTemplate mongoTemplate = new MongoTemplate(new Mongo(), "GetTheTemplateFromSomewhere");
GridFS gridFS = new GridFS(mongoTemplate.getDb());
The GridFS object lets you create, delete and find etc.
Related
Accessing the DB repeatedly for individual entities is much slower than doing a bulk select. How do I cache a the result of a bulk select into a cache, and later access it individually?
For example, I have a Employee entity:
public class Employee {
private Integer id;
}
And I have repository that can access it either with bulk select, or individually by id:
public class EmployeeRepository {
public Map<Integer, Employee> retrieveByEmployeeIds(List<Integer> ids) {
// impl
}
public Employee retrieveByEmployeeId(Integer id) {
// impl
}
}
How do I implement it so that when retrieveByEmployeeId(Integer id) is called it will check the same cache as retrieveByEmployeeIds(List<Integer> ids), and if it doesn't exist it'll make a call to the DB, and also storing that cache with the id again?
I have answered similar questions before, for example see (Spring Cache with collection of items/entities).
Essentially, you must implement a custom CacheManager and Cache, the 2 primary interfaces that form the basis of Spring's Cache Abstraction, as described here. It may even extend or delegate to an existing caching provider, but you must "decorate" the existing functionality.
The link I referred to above also contains examples.
Hope this helps give you ideas on how to handle your particular UC.
I have a Spring boot application using an AWS DynamoDb table which contains a list of items as such:
#DynamoDBTable(tableName = MemberDbo.TABLENAME)
public class MemberDbo {
public static final String TABLENAME = "Member";
#NonNull
#DynamoDBHashKey
#DynamoDBAutoGeneratedKey
protected String id;
// some more parameters
#DynamoDBAttribute
private List<String> membergroupIds;
}
I would like to find all members belonging to one specific groupId. In best case I would like to use CrudRepository like this:
#EnableScan
public interface MemberRepository extends CrudRepository<MemberDbo, String> {
List<MemberDbo> findByMembergroupIdsContaining(String membergroupIds); // actually I want to filter by ONE groupId
}
Unfortunately the query above is not working (java.lang.String cannot be cast to java.util.List)
Any suggestions how to build a correct query with CrudRepository?
Any suggestions how to create a query with Amazon SDK or some other Springboot-compliant methods?
Alternatively can I create a dynamoDb index somehow and filter by that index?
Or do I need to create and maintain a new table programmatically containing the mapping between membergroupIds and members (which results in a lot of overhead in code and costs)?
A solution for CrudRepository is preferred since I may use Paging in future versions and CrudRepository easily supports paging.
If I have understood correctly this looks very easy. You using DynamoDBMapper for model persistence.
You have a member object, which contains a list of membergroupids, and all you want to do is retrieve this from the database. If so, using DynamoDBMapper you would do something like this:
AmazonDynamoDB dynamoDBClient = new AmazonDynamoDBClient();
DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient);
MemberDbo member = mapper.load(MemberDbo.class, hashKey, rangeKey);
member.getMembergroupIds();
Where you need to replace hashKey and rangeKey. You can omit rangeKey if you don't have one.
DynamoDBMapper also supports paging out of the box.
DynamoDBMapper is an excellent model persistence tool, it has strong features, its simple to use and because its written by AWS, it has seamless integration with DynamoDB. Its creators have also clearly been influenced by spring. In short, I would use DynamoDBMapper for model persistence and Spring Boot for model-controller stuff.
I want to create a multi field search in a Spring-Boot back-end. How to do this with a Specification<T> ?
Environment
Springboot
Hibernate
Gradle
Intellij
The UI in the front end is a Jquery Datatable. Each column allows a single string search term to be applied. The search terms across more than one column is joined by a and.
I have the filters coming from the front end already getting populated into a Java object.
Step 1
Extend JPA Specification executor
public interface SomeRepository extends JpaRepository<Some, Long>, PagingAndSortingRepository<Some, Long>, JpaSpecificationExecutor {
Step2
Create a new class SomeSpec
This is where I am lost as to what the code looks like it and how it works.
Do I need a method for each column?
What is Root and what is Criteria Builder?
What else is required?
I am rather new at JPA so while I don't need anyone to write the code for me a detailed explanation would be good.
UPDATE
It appears QueryDSL is the easier and better way to approach this. I am using Gradle. Do I need to change my build.gradle from this ?
If you don't want to use QueryDSL, you'll have to write your own specifications. First of all, you need to extend your repository from JpaSpecificationExecutor like you did. Make sure to add the generic though (JpaSpecificationExecutor<Some>).
After that you'll have to create three specifications (one for each column), in the Spring docs they define these specifications as static methods in a class. Basically, creating a specification means that you'll have to subclass Specification<Some>, which has only one method to implement, toPredicate(Root<Some>, CriteriaQuery<?>, CriteriaBuilder).
If you're using Java 8, you can use lambdas to create an anonymous inner class, eg.:
public class SomeSpecs {
public static Specification<Some> withAddress(String address) {
return (root, query, builder) -> {
// ...
};
}
}
For the actual implementation, you can use Root to get to a specific node, eg. root.get("address"). The CriteriaBuilder on the other hand is to define the where clause, eg. builder.equal(..., ...).
In your case you want something like this:
public class SomeSpecs {
public static Specification<Some> withAddress(String address) {
return (root, query, builder) -> builder.equal(root.get("address"), address);
}
}
Or alternatively if you want to use a LIKE query, you could use:
public class SomeSpecs {
public static Specification<Some> withAddress(String address) {
return (root, query, builder) -> builder.like(root.get("address"), "%" + address + "%");
}
}
Now you have to repeat this for the other fields you want to filter on. After that you'll have to use all specifications together (using and(), or(), ...). Then you can use the repository.findAll(Specification) method to query based on that specification, for example:
public List<Some> getSome(String address, String name, Date date) {
return repository.findAll(where(withAddress(address))
.and(withName(name))
.and(withDate(date));
}
You can use static imports to import withAddress(), withName() and withDate() to make it easier to read. The where() method can also be statically imported (comes from Specification.where()).
Be aware though that the method above may have to be tweaked since you don't want to filter on the address field if it's null. You could do this by returning null, for example:
public List<Some> getSome(String address, String name, Date date) {
return repository.findAll(where(address == null ? null : withAddress(address))
.and(name == null ? null : withName(name))
.and(date == null ? null : withDate(date));
}
You could consider using Spring Data's support for QueryDSL as you would get quite a lot without having to write very much code i.e. you would not actually have to write the specifictions.
See here for an overview:
https://spring.io/blog/2011/04/26/advanced-spring-data-jpa-specifications-and-querydsl/
Although this approach is really convenient (you don’t even have to
write a single line of implementation code to get the queries
executed) it has two drawbacks: first, the number of query methods
might grow for larger applications because of - and that’s the second
point - the queries define a fixed set of criterias. To avoid these
two drawbacks, wouldn’t it be cool if you could come up with a set of
atomic predicates that you could combine dynamically to build your
query?
So essentially your repository becomes:
public interface SomeRepository extends JpaRepository<Some, Long>,
PagingAndSortingRepository<Some, Long>, QueryDslPredicateExecutor<Some>{
}
You can also get request parameters automatically bound to a predicate in your Controller:
See here:
https://spring.io/blog/2015/09/04/what-s-new-in-spring-data-release-gosling#querydsl-web-support
SO your Controller would look like:
#Controller
class SomeController {
private final SomeRepository repository;
#RequestMapping(value = "/", method = RequestMethod.GET)
String index(Model model,
#QuerydslPredicate(root = Some.class) Predicate predicate,
Pageable pageable) {
model.addAttribute("data", repository.findAll(predicate, pageable));
return "index";
}
}
So with the above in place it is simply a Case of enabling QueryDSL on your project and the UI should now be able to filter, sort and page data by various combinations of criteria.
My question is about what is best way to inhibit an endpoint that is automatically provided by Olingo?
I am playing with a simple app based on Spring boot and using Apache Olingo.On short, this is my servlet registration:
#Configuration
public class CxfServletUtil{
#Bean
public ServletRegistrationBean getODataServletRegistrationBean() {
ServletRegistrationBean odataServletRegistrationBean = new ServletRegistrationBean(new CXFNonSpringJaxrsServlet(), "/user.svc/*");
Map<String, String> initParameters = new HashMap<String, String>();
initParameters.put("javax.ws.rs.Application", "org.apache.olingo.odata2.core.rest.app.ODataApplication");
initParameters.put("org.apache.olingo.odata2.service.factory", "com.olingotest.core.CustomODataJPAServiceFactory");
odataServletRegistrationBean.setInitParameters(initParameters);
return odataServletRegistrationBean;
} ...
where my ODataJPAServiceFactory is
#Component
public class CustomODataJPAServiceFactory extends ODataJPAServiceFactory implements ApplicationContextAware {
private static ApplicationContext context;
private static final String PERSISTENCE_UNIT_NAME = "myPersistenceUnit";
private static final String ENTITY_MANAGER_FACTORY_ID = "entityManagerFactory";
#Override
public ODataJPAContext initializeODataJPAContext()
throws ODataJPARuntimeException {
ODataJPAContext oDataJPAContext = this.getODataJPAContext();
try {
EntityManagerFactory emf = (EntityManagerFactory) context.getBean(ENTITY_MANAGER_FACTORY_ID);
oDataJPAContext.setEntityManagerFactory(emf);
oDataJPAContext.setPersistenceUnitName(PERSISTENCE_UNIT_NAME);
return oDataJPAContext;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
...
My entity is quite simple ...
#Entity
public class User {
#Id
private String id;
#Basic
private String firstName;
#Basic
private String lastName;
....
Olingo is doing its job perfectly and it helps me with the generation of all the endpoints around CRUD operations for my entity.
My question is : how can I "inhibit" some of them? Let's say for example that I don't want to enable the delete my entity.
I could try to use a Filter - but this seems a bit harsh. Are there any other, better ways to solve my problem?
Thanks for the help.
As you have said, you could use a filter, but then you are really coupled with the URI schema used by Olingo. Also, things will become complicated when you have multiple, related entity sets (because you could navigate from one to the other, making the URIs more complex).
There are two things that you can do, depending on what you want to achieve:
If you want to have a fined grained control on what operations are allowed or not, you can create a wrapper for the ODataSingleProcesor and throw ODataExceptions where you want to disallow an operation. You can either always throw exceptions (i.e. completely disabling an operation type) or you can use the URI info parameters to obtain the target entity set and decide if you should throw an exception or call the standard single processor. I have used this approach to create a read-only OData service here (basically, I just created a ODAtaSingleProcessor which delegates some calls to the standard one + overridden a method in the service factory to wrap the standard single processor in my wrapper).
If you want to completely un-expose / ignore a given entity or some properties, then you can use a JPA-EDM mapping model end exclude the desired components. You can find an example of such a mapping here: github. The mapping model is just an XML file which maps the JPA entities / properties to EDM entity type / properties. In order for olingo to pick it up, you can pass the name of the file to the setJPAEdmMappingModel method of the ODataJPAContext in your initialize method.
I'm using Jedis as the java client to connect to Redis Servers.
Question 1: It seems there is no method to get/set Object < ? extends Serializable> ? All the values must be String or byte[]?
Other clients like "JRedis" and Spymemcache(for memcached Server) could.
Question 2: If I use ShardedJedis, it cannot set auth/password? While Jedis class can (using auth(String password)).
Regard Question 1: Jedis won't handle POJOs. You should serialize to a string or byte[] and use jedis to do that, although I won't recommend to store your java objects serialized, as you won't be able to use all Redis cool features. A different approach would be to use something like a object-hash mapper, like JOhm.
Regard Question 2: ShardedJedis will only support commands that run on a single key. This is to guarantee atomicity. If you want to run a specific command on a specific redis you should use shardedJedis.getShard('someky') which will return a Jedis instance that you can use.
Another way to handle this, the recommended one, is to specify your password in the JedisShardInfo instances.
You can see and example of this in the tests.
Answer to question 1:
Redisson (Redis Java client) can work with POJO objects. And you don't need to serialize/deserialize object by yourself each time and work with connections (acquire/release). It's all done by the Redisson.
Here is an example:
RBucket<AnyObject> bucket = redisson.getBucket("anyObject");
// set an object
bucket.set(new AnyObject());
// get an object
AnyObject myObject = bucket.get();
or you can use LiveObjectService which stores POJO fields as keys in Redis hash object.
#REntity
public class MyObject {
#RId
private String id;
#RIndex
private String value;
private MyObject parent;
public MyObject(String id) {
this.id = id;
}
public MyObject() {
}
// getters and setters
}
Redisson supports many popular codecs like Jackson JSON, Avro, Smile, CBOR, MsgPack, Kryo, FST, LZ4, Snappy and JDK Serialization.