Hey i have the following query which i am executing via an autowired jdbctemplate in my spring-boot application:
query = "select fromfilepath from filecopyeventlog where copytype = 1 ORDER BY copytimestamp DESC LIMIT 1"
i execute the query via:
String result = jdbcTemplate.queryForObject(query, String.class);
since the result is just a single column defined as varchar(250).
Now the problem is, that this query is very slow when executed in my application - it takes many minutes..
when i execute the query direcly on the database webinterface (h2), i get the result in less then 900ms.
does someone have any idea? The result looks like:
/data/internal/s3/sensor-data/semiconductor_2019-07-12T01_17_11Z.mf4.gz
in the database are more then 500.000 rows.
I think the query is fine it seems like something in the result mapping? I thought maybe the backslashes are bad and changed them, to "_" but that resulted in no change.
hope someone can help :)
Related
I've got an employees collection that looks like:
{
_id
company_id
job_type
name
age
...
}
And a remove query that looks like:
val criteria = Criteria.where("company_id").isEqualTo(companyId).and("job_type").isEqualTo(jobType)
val query = Query(criteria).limit(3)
When I run it like:
mongoTemplate.remove(query, null, "collection_name")
It works great:)
Buy when I run it in Bulk Ops like:
val bulkOp = mongoTemplate.bulkOps(BulkOperations.BulkMode.UNORDERED, null, "collection_name")
bulkOp.remove(query)
...
bulkOp.execute()
It removes all of the documents for that company & job type while completely ignoring the limit=3
Is there a solution for the issue?
Bulk writes with cursor options is not supported and is opposite for what bulk is designed for - Cursor is used for processing results in iterative fashion and make multiple trips to the server. On the other hand bulk updates are done in batch on server side.
The reason remove works when using mongo template is behind the scene it makes multiple queries one for fetching all the matches for query and picking up ids followed by another call to remove those ids.
You can also do the same by collecting ids before running remove query in mongodb for bulk.
In my Spring Repository Class, I have the following query (kind of analytics query) running on a Postgresql 9.6 server :
#Query("SELECT d.id as departement_id, COUNT(m.id) as nbMateriel FROM Departement d LEFT JOIN d.sites s LEFT JOIN s.materiels m WHERE "
+ "(s.metier.id IN (:metier_id) OR :metier_id IS NULL) AND (s.entite.id IN (:entite_id) OR :entite_id IS NULL) "
+ "AND (m.materielType.id IN (:materielType_id) OR :materielType_id IS NULL) AND "
+ "(d.id= :departement_id OR :departement_id IS NULL) "
+ "AND m.dateLivraison is not null and (EXTRACT(YEAR FROM m.dateLivraison) < :date_id OR :date_id IS NULL) "
+ "AND ( m.estHISM =:estHISM OR :estHISM IS NULL OR m.estHISM IS NULL) "
+ "GROUP BY d.id")
List<Map<Long, Long>> countByDepartementWithFilter(#Param("metier_id") List<Long> metier_id,#Param("entite_id") List<Long> entite_id,#Param("materielType_id") List<Long> materielType_id,
#Param("departement_id") Long departement_id, #Param("date_id") Integer date_id,
#Param("estHISM") Boolean estHISM);
The problem is : this query is called several times with different combination of parameters, and after 5-6 calls, time execution go from 20 ms to 10 000 ms
From what I have read, what cause this is the use of prepared statements which is not suited to analytics queries, where there are number of parameters whose values can change a lot. And indeed, running the above query directly is always fast (20 ms).
Question 1 : How can I say to Spring JPA not to use prepared statements for this specific query ?
Question 2 : If Question 1 not possible, what workaround can I have ?
There are some tips in general to enhance query performance both from JPA / DB POV:
1- use #NamedQuery instead of #Query
2- For reporting queries, don't run it inside a transaction
3- You can set the flush mode to COMMIT if you don't need to flush the persistence context before the query runs
4- check the generated query, take it and run it on SQL developer od TOAD, check its cost and run strategy, you can also consult your DBA if you can enhance it with some native DB functions / provcedures , hence use a native query instead of JPQL query
5- if data returning is large, consider making this query a DB view or materialized view and calling it directly
6- make use of query hints to activate a certain index for example, note that indexes may be ignored in case of JPQL
7- you can use native query if the query hint didn't work on JPQL
8- While comparing the query on SQL Developer with that from the code make sure that you are comparing right , the query might run very quickly initially on DB directly but takes loong time to fetch all the data , and you might be comparing this initial short time with the application data fetch time
9- use fetch size hint according to your provider
10- According to my knowledge, you might escape prepared statement if you use native non parametrized query (thus using manual placeholders and replacing values manually) but generally this should be used with care and avoided as much as possible because of SQL injection vulnerabilities and also disallows the DB query engine from as well as the hibernate engine from precompiling the queries
I was having difficulty specifying the title for the question properly - essentially I am getting an error saying "ExecuteReader requires the command to have a transaction when the connection assigned to the command is in a pending local transaction. The Transaction property of the command has not been initialized." for a situation similar to this:
using (db = getDbContext())
{
var results = (from t in db.table
select t.column).SingleOrDefault();
}
As the error says, all this is already wrapped in another transaction, which I am trying to use for this query as well.
How can I specify the transaction using this query format?
I've tried creating an SqlCommand("select column from table", myconnection, mytransaction),
which works, but I'd much rather use the LINQ syntax for the intellisense (amongst other) benefits, as the actual query is rather more complex
Many thanks, this has been annoying me for hours.
Alex
You can set the transaction into the context itself:
db.Transaction = theTransaction;
I've the following problem with EJB and Oracle database.
I've some native SQL query deployed in Oc4j that returns more than 21k rows from Oracle DB. when I run the query against Oracle DB I get JOOM (out of memory) exception.
And because the requirements was to include pagination for the result set, so we decided to use em.setMaxResult, em.setFirstResult to return only 10 rows a time.
Using the EntityManager to implement the pagination put us in some problem As Later, it was required to sort the result returned, but the whole result not just the 10 rows returned by setMaxResult()! We found that, to put the clause ORDER BY xxxx in the native query makes the query performance became too bad.
So, we are considering doing the pagination in the Database layer (using Oracle rownum or any other technique).
Later, we recognized that, If we use em.clear() we might be able to avoid the JOOM exception by making something like:
define the result list
while database has more records
{
use entityManager get next 10 records and add them to the result list
entityManager.clear();
}
return result list
So, we could implement the paging on the Servlet side (using session.getAttribute("all_result").sublist(from, to)) and thus we can do the sort using Java as opposite to SQL sort.
Provided code for pagination, might help you.
em.createQuery("Select o from Entity o where o.id > :lastId order by o.id");
query.setParameter("lastId", previousResult.get(previousResult.size()-1).getId());
query.setMaxResults(10);
I'm trying to rewrite a direct (disconnected) DataSet.DataTable.Select to LINQ for a textual search:
string search = "ProductNo like '%" + searchpattern +
"%' OR ProductName like '%" + searchpattern +
"%' OR Description like '%" + searchpattern + "%'";
DataSetProducts.sk_productsRow[] dsp = (DataSetProducts.sk_productsRow[])dsProducts.sk_products.Select(search, sort);
This works really fast. However if I re-enact it in LINQ:
productlist = from prds in dsProducts.sk_products.AsEnumerable()
where (prds.Field<string>("productno").Contains(searchpattern) ||
prds.Field<string>("productname").Contains(searchpattern) ||
prds.Field<string>("description").Contains(searchpattern))
select prds;
It becomes really slow because of the text-operations, and will return far less rows.
How can I achieve same results with the same speed? I have only ~1000 records in the table.
I've never used it myself, but perhaps i4o - Indexed LINQ might help speed things up...
As far as why it returns fewer rows, is it a case-sensitivity issue? (i.e. "like" is not case-sensitive by default; "Contains" is, by default)
If you are running within a LINQ to SQL database context, use the System.Data.Linq.SqlClient.SqlMethods.Like operation instead of String.Contains.
The reason you are seeing the difference in speed when you execute the LINQ query versus the disconnected dataset is because the dataset is already in memory while the LINQ query is executed at time of use.
Other than that if you haven't installed the LINQ Visualizer which will show you the query that is actually being executed.
Dumb question, but is the .AsEnumerable() necessary for something? I'm only experienced so far with Linq2Sql, but I would normally treat things as AsQueryable as long as possible in order to avoid the "load dataset, then immediately filter it down again" scenario.