I want to create a capped collection from Java code. I found the syntax for creating it through JavaScript, but could not find an example for Java.
Mongo mongo = new Mongo("127.0.0.1");
DB db = mongo.getDB("mydbid");
DBCollection collection;
if (db.collectionExists("mycollection")) {
collection = db.getCollection("mycollection");
} else {
collection = /* ????? Create the collection ?????? */
}
}
Use the DB.createCollection operation and then specify a DBObject that has capped as a parameter. You can then specify size and max in order to control the byte size and the maximum number of documents. The MongoDB site has a tutorial on capped collections that explains all the options, but is missing an example for each driver.
Mongo mongo = new Mongo("127.0.0.1");
DB db = mongo.getDB("mydbid");
DBCollection collection;
if (db.collectionExists("mycollection")) {
collection = db.getCollection("mycollection");
} else {
DBObject options = BasicDBObjectBuilder.start().add("capped", true).add("size", 2000000000l).get();
collection = db.createCollection("mycollection", options);
}
}
With more recent java mongo driver (ie 3.4) the creation should slightly change:
CreateCollectionOptions opts = new CreateCollectionOptions().capped(true).sizeInBytes(1024*1024);
database.createCollection("test", opts);
Please, notice that the createCollection is not returning any value.
Related
The goal of my springBoot webflux r2dbc application is Controller accepts a Request including a list of DB UPDATE or INSERT details, and Response a result summary back.
I can write a ReactiveCrudRepository based repository to implement each DB operation. But I don't know how to write the Service to group the executions of the list of DB operations and compose a result summary response.
I am new to java reactive programing. Thanks for any suggestions and help.
Chen
I get the hint from here: https://www.vinsguru.com/spring-webflux-aggregation/ . Ideas are :
From request to create 3 Monos
Mono<List> monoEndDateSet -- DB Row ids of update operation;
Mono<List> monoCreateList -- DB Row ids of new inserted;
Mono monoRespFilled -- partly fill some known fields;
use Mono.zip aggregate the 3 monos, map and aggregate the Tuple3 to Mono to return.
Below are key part of codes:
public Mono<ChangeSupplyResponse> ChangeSupplies(ChangeSupplyRequest csr){
ChangeSupplyResponse resp = ChangeSupplyResponse.builder().build();
resp.setEventType(csr.getEventType());
resp.setSupplyOperationId(csr.getSupplyOperationId());
resp.setTeamMemberId(csr.getTeamMemberId());
resp.setRequestTimeStamp(csr.getTimestamp());
resp.setProcessStart(OffsetDateTime.now());
resp.setUserId(csr.getUserId());
Mono<List<Long>> monoEndDateSet = getEndDateIdList(csr);
Mono<List<Long>> monoCreateList = getNewSupplyEntityList(csr);
Mono<ChangeSupplyResponse> monoRespFilled = Mono.just(resp);
return Mono.zip(monoRespFilled, monoEndDateSet, monoCreateList).map(this::combine).as(operator::transactional);
}
private ChangeSupplyResponse combine(Tuple3<ChangeSupplyResponse, List<Long>, List<Long>> tuple){
ChangeSupplyResponse resp = tuple.getT1().toBuilder().build();
List<Long> endDateIds = tuple.getT2();
resp.setEndDatedDemandStreamSupplyIds(endDateIds);
List<Long> newIds = tuple.getT3();
resp.setNewCreatedDemandStreamSupplyIds(newIds);
resp.setSuccess(true);
Duration span = Duration.between(resp.getProcessStart(), OffsetDateTime.now());
resp.setProcessDurationMillis(span.toMillis());
return resp;
}
private Mono<List<Long>> getNewSupplyEntityList(ChangeSupplyRequest csr) {
Flux<DemandStreamSupplyEntity> fluxNewCreated = Flux.empty();
for (SrmOperation so : csr.getOperations()) {
if (so.getType() == SrmOperationType.createSupply) {
DemandStreamSupplyEntity e = buildEntity(so, csr);
fluxNewCreated = fluxNewCreated.mergeWith(this.demandStreamSupplyRepository.save(e));
}
}
return fluxNewCreated.map(e -> e.getDemandStreamSupplyId()).collectList();
}
...
I'm using spring data elastic search, Now my document do not have any static fields, and it is accumulated data per qtr, I will be getting ~6GB/qtr (we call them as versions). Lets say we get 5GB of data in Jan 2021 with 140 columns, in the next version I may get 130 / 120 columns, which we do not know, The end user requirement is to get the information from the database and show it in a tabular format, and he can filter the data. In MongoDB we have BasicDBObject, do we have anything in springboot elasticsearch
I can provide, let say 4-5 columns which are common in every version record and apart from that, I need to retrieve the data without mentioning the column names in the pojo, and I need to use filters on them just like I can do in MongoDB
List<BaseClass> getMultiSearch(#RequestBody Map<String, Object>[] attributes) {
Query orQuery = new Query();
Criteria orCriteria = new Criteria();
List<Criteria> orExpression = new ArrayList<>();
for (Map<String, Object> accounts : attributes) {
Criteria expression = new Criteria();
accounts.forEach((key, value) -> expression.and(key).is(value));
orExpression.add(expression);
}
orQuery.addCriteria(orCriteria.orOperator(orExpression.toArray(new Criteria[orExpression.size()])));
return mongoOperations.find(orQuery, BaseClass.class);
}
You can define an entity class for example like this:
public class GenericEntity extends LinkedHashMap<String, Object> {
}
To have that returned in your calling site:
public SearchHits<GenericEntity> allGeneric() {
var criteria = Criteria.where("fieldname").is("value");
Query query = new CriteriaQuery(criteria);
return operations.search(query, GenericEntity.class, IndexCoordinates.of("indexname"));
}
But notice: when writing data into Elasticsearch, the mapping for new fields/properties in that index will be dynamically updated. And there is a limit as to how man entries a mapping can have (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html). So take care not to run into that limit.
I implemented pageable functionality into Criteria API query and I noticed increased memory usage during query execution. I also used spring-data-jpa method query to return same result, but there memory is cleaned up after every batch is processed. I tried detaching, flushing, clearing objects from EntityManager, but memory use would keep going up, occasionally it will drop but not as much as with method queries. My question is what could cause this memory use if objects are detached and how to deal with it?
Memory usage with Criteria API pageable:
Memory usage with method query:
Code
Since I'm also updating entities retrieved from DB, I use approach where I save ID of last processed entity, so when entity gets updated query doesen't skip next selected page. Below I provide code example that is not from real app I'm working on, but it just recreation of the issue I'm having.
Repository code:
#Override
public Slice<Player> getPlayers(int lastId, Pageable pageable) {
List<Predicate> predicates = new ArrayList<>();
CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
CriteriaQuery<Player> criteriaQuery = criteriaBuilder.createQuery(Player.class);
Root<Player> root = criteriaQuery.from(Player.class);
predicates.add(criteriaBuilder.greaterThan(root.get("id"), lastId));
criteriaQuery.where(criteriaBuilder.and(predicates.toArray(Predicate[]::new)));
criteriaQuery.orderBy(criteriaBuilder.asc(root.get("id")));
var query = entityManager.createQuery(criteriaQuery);
if (pageable.isPaged()) {
int pageSize = pageable.getPageSize();
int offset = pageable.getPageNumber() > 0 ? pageable.getPageNumber() * pageSize : 0;
// Fetch additional element and skip it based on the pageSize to know hasNext value.
query.setMaxResults(pageSize + 1);
query.setFirstResult(offset);
var resultList = query.getResultList();
boolean hasNext = pageable.isPaged() && resultList.size() > pageSize;
return new SliceImpl<>(hasNext ? resultList.subList(0, pageSize) : resultList, pageable, hasNext);
} else {
return new SliceImpl<>(query.getResultList(), pageable, false);
}
}
Iterating through pageables:
#Override
public Slice<Player> getAllPlayersPageable() {
int lastId = 0;
boolean hasNext = false;
Pageable pageable = PageRequest.of(0, 200);
do {
var players = playerCriteriaRepository.getPlayers(lastId, pageable);
if(!players.isEmpty()){
lastId = players.getContent().get(players.getContent().size() - 1).getId();
for(var player : players){
System.out.println(player.getFirstName());
entityManager.detach(player);
}
}
hasNext = players.hasNext();
} while (hasNext);
return null;
}
I think you are running into a query plan cache issue here that is related to the use of the JPA Criteria API and how numeric values are handled. Hibernate will render all numeric values as literals into an intermediary HQL query string which is then compiled. As you can imagine, every "scroll" to the next page will be a new query string so you gradually fill up the query plan cache.
One possible solution is to use a library like Blaze-Persistence which has a custom JPA Criteria API implementation and a Spring Data integration that will avoid these issues and at the same time improve the performance of your queries due to a better pagination implementation.
All your code would stay the same, you just have to include the integration and configure it as documented in the setup section.
I am new to Web API, Entity Framework and OData. I asked a similar question in another forum but haven't gotten a relevant response.
We have a OData compliant web api service for use in Salesforce. We have a custom complex query in Oracle that we need to expose.
I am not sure how to use a custom query like we want to also allow for odata parameter filtering to occur? ($filter, $top, $skip, etc) For example, when a $filter is used i want to apply that filter to the custom query and then send it back to the database to have it return the result set. How can i do this?
The issue i seem to have is that I can see the parameters as they come in but they are not translating to the query being passed to oracle. It seems that it will fire the query returning the full result set and then apply the parameters. This is very slow as the result set is very large.
I am hoping 2 figure out 2 things
1. How can i use custom sql and apply odata parameters to the underlying query?
2. When using EF or a custom query, how can i apply odata parameters to the query so that when the query is sent to the database that the $filter parameter, for example, is included in the query? I don't want the full result returned then apply the filter.
Can anyone give me some pointers on how to make this happen?
private static ODataValidationSettings _validationSettings = new ODataValidationSettings();
//public IHttpActionResult GetName()
//{ }
// GET: odata/ShareData
[ODataRoute("Orders")]
[EnableQuery(PageSize = 50)]
public IHttpActionResult GetOrders(ODataQueryOptions<Orders> queryOptions)
{
// validate the query.
try
{
queryOptions.Validate(_validationSettings);
}
catch (ODataException ex)
{
return BadRequest(ex.Message);
}
try
{
string connectionString = ConfigurationManager.ConnectionStrings["DNATestConnectionString"].ConnectionString;
var items = GetDataItems(connectionString);
return Ok<IEnumerable<Orders>>(items);
}
catch (Exception ex)
{
return StatusCode(HttpStatusCode.InternalServerError);
}
}
#region Load Data Methods
private static List<Orders> GetDataItems(string connectionString)
{
List<Orders> items = new List<Orders>();
using (OracleConnection con = new OracleConnection(connectionString))
{
con.Open();
using (OracleCommand cmd = con.CreateCommand())
{
cmd.CommandText = "select po_header_id, segment1, vendor_id, vendor_site_id from po_headers_all where vendor_id=4993";
using (OracleDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
items.Add(ToOrders(rdr));
}
}
}
return items;
}
private static Orders ToOrders(OracleDataReader rdr)
{
Orders data = new Orders();
data.VENDOR_ID = ToInt32(rdr, "VENDOR_ID");
data.VENDOR_SITE_ID = ToInt32(rdr, "VENDOR_SITE_ID");
data.PO_HEADER_ID = ToInt32(rdr, "PO_HEADER_ID");
data.SEGMENT1 = Convert.ToString(rdr["SEGMENT1"]);
return data;
}
private static int ToInt32(OracleDataReader rdr, string name)
{
int index = rdr.GetOrdinal(name);
return rdr.IsDBNull(index) ? 0 : Convert.ToInt32(rdr[index]);
}
#endregion
I don't think this is possible.
How can i use custom sql and apply odata parameters to the underlying query?
As far as I'm aware, you can't. The whole point of the OData library is that it needs to work off an IQueryable. By using custom SQL in a string like you have in your example, you can't combine it with the OData parameters that are being passed in.
One approach would be to have your custom SQL in a SQL view, then add the SQL view to your EF model in the same way as you would add a table - it will be represented as a DbSet just like tables are.
You can then get an IQueryable to represent the dataset and then apply the OData parameters as follows:
public IHttpActionResult GetOrders(ODataQueryOptions<OrdersView> queryOptions)
{
IQueryable<OrdersView> allData = // ... get the DbSet from entity framework...
// this will apply the OData query to the data set and only pull the data you want from the database
var filteredResults = queryOptions.ApplyTo(allData) as IQueryable<OrdersView>;
return Ok<IQueryable<OrdersView>>(filteredResults);
}
I'm using Spring data MongoTemplate to manage mongo operations. I'm trying to save & update json full documents (using String.class in java).
Example:
String content = "{MyId": "1","code":"UG","variables":[1,2,3,4,5]}";
String updatedContent = "{MyId": "1","code":"XX","variables":[6,7,8,9,10]}";
I know that I can update code & variables independently using:
Query query = new Query(where("MyId").is("1"));
Update update1 = new Update().set("code", "XX");
getMongoTemplate().upsert(query, update1, collectionId);
Update update2 = new Update().set("variables", "[6,7,8,9,10]");
getMongoTemplate().upsert(query, update2, collectionId);
But due to our application architecture, it could be more useful for us to directly replace the full object. As I know:
getMongoTemplate().save(content,collectionId)
getMongoTemplate().save(updatedContent,collectionId)
implements saveOrUpdate functionality, but this creates two objects, do not update anything.
I'm missing something? Any approach? Thanks
You can use Following Code :
Query query = new Query();
query.addCriteria(Criteria.where("MyId").is("1"));
Update update = new Update();
Iterator<String> iterator = json.keys();
while(iterator.hasNext()) {
String key = iterator.next();
if(!key.equals("MyId")) {
Object value = json.get(key);
update.set(key, value);
}
}
mongoTemplate.updateFirst(query, update, entityClass);
There may be some other way to get keyset from json, you can use according to your convenience.
You can use BasicDbObject to get keyset.
you can get BasicDbObject using mongoTemplate.getConverter().