I have a task that I need to load to the database only new CSV files. if the file was already loaded to the database then this file should be ignored. Currently, I am loading all the CSV files regardless.
// daily/*.csv
#Value("${dailyExportDirPath}")
private Resource[] inputResources;
#Bean
public MultiResourceItemReader<DailyExport> multiResourceItemReader(FieldSetMapper<DailyExport> testClassRowMapper) {
MultiResourceItemReader<DailyExport> multiResourceItemReader = new MultiResourceItemReader<>();
multiResourceItemReader.setName("dailyExportMultiReader");
multiResourceItemReader.setDelegate(reader(testClassRowMapper));
multiResourceItemReader.setStrict(true);
multiResourceItemReader.setResources(inputResources);
return multiResourceItemReader;
}
I was thinking to create a table that will contain all the files that where already loaded. So instead of initiate a resources array with all the existing CSV’s from the directory, the job will create a resource array that will contain only the new reports.
all reports in directory – reports in database = new reports that need to load.
How can I optimize the resources array so they will contain only the relevant files meaning the new ones?
Thank you
Assuming you've one table for file metadata, and another for file content (with a FK back to file metadata table), you can simply use the current date for file id in the metadata table. Then database will fail any attempt to insert duplicate rows in the file metadata table, you catch it, and keep processing other files.
Related
i have costumer's every one i have to upload mullite files
so is there any way to get it?
i create table with
FILE_NAME
FILE_MINETYPE
FILE_CHARSET
i see many video's but they aren't useful for me
You need to create a Page Item with the type File Browse.... Then, under Storage Type, you can use your own custom table or you can use APEX_APPLICATION_TEMP_FILES.
If you use APEX_APPLICATION_TEMP_FILES it will automatically save information about the file type such as file name, mime type, etc. If you use a custom table, you will need to designate which column in the table should hold which file attribute.
You will also need a button or some way of submitting the page. Once the page is submitted, the files will be uploaded and saved to the database.
I have a Core Data entity that is used to store earthquake reports. I would like to store a bit of global data to keep track of when the data was last updated.
The date should be a single date common to all entity objects in the store.
Can I use the Entity userInfo for this? I can't determine if this is read-only or if it can be altered at runtime.
Is there a better way? I don't want to store it in UserDefaults because if the sql files are deleted I want the "date updated" data to go away as well.
Persistent stores can have metadata about the store in addition to the actual data. That would be a good place for this. Metadata is part of the NSPersistentStore, so you have to reach down into the NSPersistentContainer to get it and change it. Something like this
guard let persistentStore = persistentContainer.persistentStoreCoordinator.persistentStores.first else {
return
}
var metadata = persistentContainer.persistentStoreCoordinator.metadata(for: persistentStore)
metadata["updated"] = Date()
persistentContainer.persistentStoreCoordinator.setMetadata(metadata, for: persistentStore)
self.saveContext()
The last line is there because metadata doesn't get saved until you save changes, even though it's metadata instead of actual data.
It's best to read the existing metadata first because Core Data puts some of its own metadata in there. You probably wouldn't be able to corrupt that, but make sure by reading it first and leaving the existing values alone.
I am using Parse.File() in cloud code for storing the thumbnail(JPEG),So once user edits profile with new profile image I create a thumbnail with new Parse.File() and saves the data.
Now the case is each new Parse File object creates the new file and uploads on server creating a new file "NAME" in the database which is correct.
But my requirement is that I want to keep the name of the thumbnail same.
i.e
If current name is 1.JPEG then after uploading the image name should remain 1.JPEG.
and if the user is uploading the image first time then it will generate the unique value as per parse File object behavior.
Any updates on this would be very helpful. :)
I am new with Spring, my application, developed with Spring Roo has a Cron that every day download some files and update a database.
The update is done, after downloading and parsing the files, using merge(),
an Entity class Dataset has a list called resources, after the download I do:
dataset.setResources(resources);
dataset.merge();
and dataset.merge() does the following:
#Transactional
public Dataset Dataset.merge() {
if (this.entityManager == null) this.entityManager = entityManager();
Dataset merged = this.entityManager.merge(this);
this.entityManager.flush();
return merged;
}
I expect that doing dataset.setResources(resources); I would overwrite the filed resources, and so even the database entry would be overwritten.
But I get double entries in the database: every resource appear twice, with different IDs (incremental).
How can I succed in let my application doing updates and not insert? A naive solution would be delete manually the old resource and then call merge(); is this the way or is there some more smart solution?
This situation occurs when you use Hibernate as persistence engine and your entities have version field.
Normally the ID field is what we need for merging a detached object with its persistent state in the database, but Hibernate takes the version field in account and if you don't set it (it is null) Hibernate discards the value of ID field and creates a new object with new ID.
To know if you are affected by this strange feature of Hibernate, set a value in the version field, if an Exception is thrown you got it. In that case the best way to solve it is the data to parse contain the right value of version. Another ways are to disable version checking (see Hibernate ref guide to know about it) or load persistent state before merging.
How can I transfer the build folder to another site?
The problem is that the name of the database table prefix is sewn into the files and if I have a lot of sites I will have to manually change the data. Is it possible to automatically change the table names and prefixes?
For example:
abstract class BaseCategoriesPeer {
/** the default database name for this class */
const DATABASE_NAME = 'test';
/** the table name for this class */
const TABLE_NAME = 'test_123_categories';
For each new site I will have to change the data?
If I got it correctly, here's a possible solution:
You can check the directory in which your source file is, and make a switch to define a variable (let's say) $appStored.
Then in a class for database access definitions (it's more common than a table name schema that depends on the deploy site), you make another switch, this time switch ($appStored), and then define the names for each deploy configuration.