I want to build sync method, both from my local database to parse.com and from parse.com to my local database. For first case ( from local to parse.com, Alhamdulillah my script run well. But for second case, i need query between column last_update and last_sync (i do not use updatedAt caused by i can't control it, so i use last_update).
Please explain me how to get all data if last_update is greaterThan last_sync?
From https://www.parse.com/questions/querying-between-two-dates i got query between two dates, but it's from same column, the value of query is real value. But my case, the value of key = last_update is column name, last_sync.
Thank you...
You can't use a query constraint that compares two columns. You need to change your logic. Why do you store both a last_update and last_sync in a record?
Syncing is a complex subject and you can easily mess up the logic. I don't understand how you can store a last_sync date on every record, as this has to be different for every user. You need to store the last_sync value for each user, and use that to compare against the last_update column on all records.
Related
I am trying to delete a set of data in the target table based on a column (year) from the lookup in IICS (Informatica Cloud).
I want to solve this problem using pre/post sql commands but the constraint is I can't pass year column to my query.
I tried this:
delete from sample_db.tbl_emp where emp_year = {year}
I want to delete all the employees in a specific year i get from lookup return
For Ex:
I got year as '2019', all the records in table sample_db.tbl_emp containing emp_year=2019 must be deleted.
I am not sure how this works in informatica cloud.
Any leads would be helpful.
How are you getting the year value? A pre/post SQL may not be the way to go unless you need to do this as part of another transformation, i.e., before or after the transformation runs. Also, does your org only have ICDI, or also ICAI? ICAI may be a better solution depending on the value is being provided.
The following steps would help you achieve this.
Create an input-output parameter in your mapping.
Assign the result of your lookup in an expression transformation to the parameter using SetMaxVariable
Use the parameter in your target pre SQL as
delete from sample_db.tbl_emp where emp_year = $$parameter
Let me know if you have any further questions
First, I don't have unique value to directly point out a value I want to retrieve, so there for, I cannot use the WHERE Statement. The reason for that, because I have a database where I constantly updated or deleted, so if I use WHERE statement I have to edit the code again.
Then do apply a unique value - add a field of data type AutoNumber or you will never get out of this mess.
I'm try to understand the best way to design the key for my HBase Table.
My use case :
Structure right now
PersonID | BatchDate | PersonJSON
When some thing about the person is modified, a new PersonJSON and new a batchdate is inserted in to Hbase updating the old records. And every 4 hours a scan of all the people who are modified are then pushed to Hadoop for further processing.
If my key is just personID it great for updating the data. But my performance sucks because I have to add a filter on BatchData column to scan all the rows greater than a batch date.
If my key is a composite key like BatchDate|PersonID I could use startrow and endrow on the row key and get all the rows that have been modified. But then I would have lot of duplicated since the key is not unique and can no longer update a person.
Is bloom filter on row+col (personid+batchdate) an option ?
Any help is appreciated.
Thanks,
Abhishek
In addition to the table with PersonID as the rowkey, it sounds like you need a dual-write secondary index, with BatchDate as the rowkey.
Another option would be Apache Phoenix, which provides support for secondary indexes.
I usually do two steps:
Create table one just have key is commbine of BatchDate+PersonId, value could be empty.
Create table two just as normal you did. Key is PersonId Value is the whole data.
For date range query: query table one first to get the PersonIds, and then use Hbase batch get API to get the data by batch. it would be very fast.
I'm trying to use an incremental sqoop job across all tables in a database. Some of the databases only have string values in the columns. Is there a way to increment on a string value? There is a common string name across all tables.
After my initial comment I was thinking if the question you asked even made sense. It would if your database forced you to store either the record date or the incrementing number into a text column, but the odds of that is very slim.
If you have a date field you can actually use, you can just use 'lastmodified' mode instead of 'append' mode.
I have 3 columns in my mongodb named as days (long), startDate (java.util.Date), endDate (java.util.Date). What all I want to fetch the records between startDate and (endDate-days) OR (endDate-startDate) <= days.
Can you please let me know how could i achieve this using mongoTemplate spring.
I don't want to fetch all the records from table and then resolve this on java side since in future my table may have million of records.
Thanks
Jitender
There is no way to do this in the query on the DB side (the end minus start part). What I recommend if this is an important feature for your application is that you alter the schema to maintain in the document the delta between the two fields in the format you need it. Since you can update that field when you update endDate (or if you populate both dates at the same time you can just compute the field then).
If you receive this data in bulk from another source, or if you do multi-updates of the endDate then you will probably need another job to run and periodically compute the delta of the documents where it's not computed (then you can start with always setting delta to 99999 and update it in this job to accurate value once endDate is set).
While you can use $where clause, it will be a very slow full collection scan so I would not suggest its use - it's probably better to come up with a more performant alternative even if it requires altering the schema.
http://docs.mongodb.org/manual/reference/operator/where/