I'm trying to make the next implementation.
Due to size reasons, I have to split my output file in, for example, 10k row chunks.
So, I need to dump 10k in file "out1.csv", the next 10k in file "out2.csv", and so on.
With one output file, the schema batch:chunk with reader-processor-writer is easy and direct.
The output stream is opened in batch:streams XML section inside the chunk, so I avoid the
"Writer must be open before it can be written to" exception.
I want to make an implementation avoiding this strict and preset solution:
<batch:chunk reader="reader" writer="compositeWriter" commit-interval="10000" processor-transactional="false">
<batch:streams>
<batch:stream ref="writer1" />
<batch:stream ref="writer2" />
<batch:stream ref="writer3" />
.
.
.<batch:stream ref="writer20" />
</batch:streams>
</batch:chunk>
<bean id="writer1" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="out1.csv" />
...
</bean>
<bean id="writer2" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="out2.csv" />
...
</bean>
...
<!-- writer 20 -->
Supposing that 20 writers are quite enough.
I'm looking for a solution to create output writers dynamically (maybe programatically), open them and avoid the above exception.
Due to size reasons, I have to split my output file in, for example, 10k row chunks. So, I need to dump 10k in file "out1.csv", the next 10k in file "out2.csv", and so on.
You seem to be using a CompositeItemWriter, but this is not the way to go. What you need to use is the MultiResourceItemWriter which allows you to split the output by item count. In your case, you would need to configure a MultiResourceItemWriter and set the itemCountLimitPerResource to 10.000. You can also provide a ResourceSuffixCreator to customize the output file names like out1.csv, out2.csv, etc.
Related
i'm working with spring integration and i have the next case: i'm reading a XML file with an int-file:inbound-channel-adapter and i split the file with a int-xml:xpath-splitter, the thing is that i need to move the file after been splitted.
I want all features of int-xml:xpath-splitter plus moving the file, should i implement a custom splitter extending XPathMessageSplitter? or is there any other way to do that with an out-of-box components?
Thanks.
<int-xml:xpath-splitter id="salesTransSplitter"
input-channel="salesInputChannel"
output-channel="splitterOutChannel" order="1">
<int-xml:xpath-expression expression="/sales_transactions/trans"/>
</int-xml:xpath-splitter>
Something like this should work...
<int-file:inbound ... channel="foo" />
<int:publish-subscribe-channel id="foo" />
<int-xml:xpath-splitter input-channel="foo" ... order="1" />
<int-service-activator input-channel="foo" order="2"
expression="payload.renameTo(new java.io.File('/newDir/' + payload.name)" output-channel="nullChannel" />
If you want to test the rename was successful, send to some other channel other than nullChannel - boolean true means success.
EDIT
Sorry about that; order should be supported on every consuming endpoint, I will open a JIRA issue.
The order is not strictly necessary; if no order is present, the order they appear in the configuration will be used; I just prefer to make it explicit.
There are (at least) two work arounds:
Remvoe the order attribute from BOTH consumers and they will be invoked in the order they appear in the XML.
Configure the XPath splitter as a normal splitter, which does support order...
<int:splitter id="salesTransSplitter" order="1"
input-channel="salesInputChannel"
output-channel="splitterOutChannel" order="1">
<bean class="org.springframework.integration.xml.splitter.XPathMessageSplitter">
<constructor-arg value="/sales_transactions/trans" />
</bean>
</int-xml:xpath-splitter>
I tried to find a way to configure a Camel endpoint using a spring bean that is referenced from the endpoint declaration in route in a camel context, but it does not work.
For exemple, sometime defining an endpoint URI with many parameters is very horrible (!!), it would be lot more easier to configure the endpoint with a bean and its properties. (Or even better, when configuring a endpoint in XML, maybe the or elements should have sub-elements like a regular beans where we could configure the parameters of the endpoint).
The first approach below work well and is very standard and pretty simple. The second approach, is the one I would like to use instead, but it does not work. I tried with many variations, but without success! The third alternative below would just be an interesting proposal for Camel developers in fact, but it also illustrate my point.
In my example below, I only configured 3 parameters for the file endpoint, but imagine the URI with 10 parameters!! My question is how can I make my second approach working properly, I'm sure there is a simple solution!? I also tried using a factory-bean and a factory method, but it diid not work neither.
1) Standard way to configure a camel endpoint in XML (spring beans):
...
<camel:camelContext id="camelContext" >
<camel:route id="deviceDataLogsPoller" >
<camel:from uri="file://long/path/to/input?preMove=../inprogress&move=../done&moveFailed=../error" />
<camel:log message="Input device data file read from file in input folder {{im.filePoller.folder.input}}." loggingLevel="INFO" />
</camel:route>
</camel:camelContext>
2) Alternative that I expected to be valide but that does not work (for me!):
<bean id="filePoller" class="org.apache.camel.component.file.FileEndpoint" >
<property name="camelContext" ref="camelContext" />
<property name="localWorkDirectory" value="/long/path/to/input" />
<property name="preMove" value="../inprogress" />
<property name="move" value="../done" />
<property name="moveFailed" value="../error" />
...
</bean>
...
<camel:camelContext id="camelContext" >
<camel:route id="deviceDataLogsPoller" >
<camel:from ref="filePoller" />
<camel:log message="Input device data file read from file in input folder {{im.filePoller.folder.input}}." loggingLevel="INFO" />
</camel:route>
</camel:camelContext>
3) Alternative that would be interesting in the future (mixed between two alternatives above):
...
<camel:route id="deviceDataLogsPoller" >
<camel:from uri="file://long/path/to/input" >
<property name="preMove" value="../inprogress" />
<property name="move" value="../done" />
<property name="moveFailed" value="../error" />
...
</camel:from>
<camel:log message="Input device data file read from file in input folder {{im.filePoller.folder.input}}." loggingLevel="INFO" />
</camel:route>
</camel:camelContext>
What did not work for you exactly?
Following setup did the job as expected:
<bean id="filePoller" class="org.apache.camel.component.file.FileEndpoint">
<property name="file" value="src/data/bean-ref" />
<property name="move" ref="moveExpression"/>
</bean>
<bean id="moveExpression" class="org.apache.camel.model.language.SimpleExpression">
<constructor-arg value="${file:parent}/.done/${file:onlyname}" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext">
<route>
<from ref="filePoller" />
<log message="${body}" />
</route>
</camelContext>
Note:
The property file is mandatory
The properties move, moveFailed, and preMove are not of type java.lang.String but of type org.apache.camel.Expression and have to be initialized accordingly.
The property moveExpression needs a full file expression. If only .done is used instead of ${file:parent}/.done/${file:onlyname} then the file is renamed to .done and not moved to a directory named .done.
As stated in my last comment, I was able to make the bean configuration for an endpoint work (see comments above), but this approach is finally lot more complex and heavy than simply using the URIs after all!!
It would have been more interesting to have a way to configure the endpoints like I proposed in my 3rd alternative above. Maybe if I have time, I will try to create my own and tags that will wrap the existing ones by constructing the full URI from params elements...! I can also propose this to Camel developers.
See an example below of how it could be interesting to configure endpoints in the future (or with the XML wrapper I would like to code):
<camel:route id="deviceDataLogsPoller">
<camel:from uri="file://long/path/to/input" >
<param name="preMove" value="../inprogress" />
<param name="move" value="../done" />
<param name="moveFailed" value="../error" />
...
</camel:from>
...
</camel:route>
Unfortunately, the endpoint configuration as shown above is not possible for the moment, but it would be a nice to have I think! For the moment, the only way is either to specify all the parameters as params in a very long URI, or to configure the endpoint as a regular bean, with with all the complexity it implies (see comments above for details).
I have a Spring form wih some validations and all the personalized messeges from javax.validation.constraints appears to use the wrong encoding.
Lets take this as an example:
#NumberFormat(style=Style.NUMBER)
#NotNull
private BigDecimal maintenanceCosts;
With applicationContext.xml file containing
<bean id="messageSource" class="org.springframework.context.support.ReloadableResourceBundleMessageSource">
<property name="defaultEncoding" value="UTF-8" />
<property name="basenames">
<list>
<value>classpath:messages</value>
<value>classpath:ValidationMessages</value>
</list>
</property>
</bean>
<bean id="localeResolver" class="org.springframework.web.servlet.i18n.SessionLocaleResolver">
<property name="defaultLocale" value="en" />
</bean>
And a ValidationMessage_en.properties encoded in UTF-8 and marked so (in Eclipse with right click, properties) with the text:
javax.validation.constraints.NotNull.message=This field can't be empty
I want to show thouse messages in localized strings with the right codification so I added an UTF-8 file ValidationMessages_ru.properties with:
javax.validation.constraints.NotNull.message=Это поле не может быть
пустым
But the message shows this message:
Ðто поле не может быть пуÑтым
On the other hand I has able to customize the spring managed error messages with the right encoding. But JSR303 texts seems to behave differently.
I found that it was much more elegant to get rid of the ValidationMessages_XX.properties files. Just all the localized configuration in one file that supports UTF-8!
The answer is to set the ValidationMessageSource of your validator with your resourceBundle. For example:
<bean id="messageSource" class="org.springframework.context.support.ReloadableResourceBundleMessageSource"
p:defaultEncoding="UTF-8"
p:basenames ="classpath:messages"/>
<!-- JSR-303 validation-->
<bean id ="validator" class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean"
p:validationMessageSource-ref="messageSource"/>
All merit must be attributed to this blog post. Not being able to read thouse characters was annoying. No more: message=\u042D\u0442\u043E \u043F\u043E\u043B\u0435 \u043D\u0435 \u043C
Properties file in Java are per default ISO 8859-1. Characters which you cannot display this way need to use unicode escapes. To quote the javadocs of Properties:
...except the input/output stream is encoded in ISO 8859-1 character
encoding. Characters that cannot be directly represented in this
encoding can be written using Unicode escapes ; only a single 'u'
character is allowed in an escape sequence.
You will need to use unicode escapes. You can di it manually or use native2ascii.
I have the following core mongo options configuration in spring:
<mongo:mongo host="${db.hostname}" >
<mongo:options
connections-per-host="40"
threads-allowed-to-block-for-connection-multiplier="1500"
connect-timeout="15000"
auto-connect-retry="true"
socket-timeout="60000"
write-number="1"
write-fsync="false"/>
</mongo:mongo>
What I want to know is about different write-number options which is relevant to write concern like none, normal, safe etc.
Can I assume the mapping of write-number to writeconcern as below?
NONE: -1
NORMAL: 0
SAFE: 1 (default)
FSYNC_SAFE: 2
REPLICAS_SAFE: 3
JOURNAL_SAFE: 4
MAJORITY: 5
Following link has provided a good help to set mongo options in spring, but not specific enough for write-number values:
How to configure MongoDB Java driver MongoOptions for production use?
The write-concern number is the value of "w" which maps to the number of replicas that the write must propagate to before being considered successful when w > 1.
FSYNC_SAFE maps to setting write-fsync (true or false) and since JOURNAL_SAFE is also a boolean value, I suspect there is a similar boolean setting in Spring but I couldn't find it in any of their docs.
If you have everything installed to test this out empirically, just try several configurations and check the actual setting of the resultant write concern with something like:
WriteConcern wc = new WriteConcern(); // should get your default write concern
System.out.println(wc.getJ());
System.out.println(wc.getFsync());
System.out.println(wc.getW());
That should show you Journal setting, Fsync setting (both boolean), W (as an int).
You can confiture write-concern="ACKNOWLEDGED".
<mongo:mongo id="replicaSetMongo" replica-set="${mongo.replicaSetSevers}" />
<mongo:db-factory dbname="${mongo.dbname}" mongo-ref="replicaSetMongo" write-concern="ACKNOWLEDGED" />
<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
</bean>
Hope this can help.
I'm currently evaluating the Spring-db4o integration. I was impressed by the declarative transaction support as well as the ease to provide declarative configuration.
Unfortunately, I'm struggling to figure how to create an index on specific fields. Spring is preparing the db during the tomcat server startup. Here's my spring entry :
<bean id="objectContainer" class="org.springmodules.db4o.ObjectContainerFactoryBean">
<property name="configuration" ref="db4oConfiguration" />
<property name="databaseFile" value="/WEB-INF/repo/taxonomy.db4o" />
</bean>
<bean id="db4oConfiguration" class="org.springmodules.db4o.ConfigurationFactoryBean">
<property name="updateDepth" value="5" />
<property name="configurationCreationMode" value="NEW" />
</bean>
<bean id="db4otemplate" class="org.springmodules.db4o.Db4oTemplate">
<constructor-arg ref="objectContainer" />
</bean>
db4oConfiguration doesn't provide any means to specify the index. I wrote a simple ServiceServletListener to set the index. Here's the relevant code:
Db4o.configure().objectClass(com.test.Metadata.class).objectField("id").indexed(true);
Db4o.configure().objectClass(com.test.Metadata.class).objectField("value").indexed(true);
I inserted around 6000 rows in this table and then used a SODA query to retrieve a row based on the key. But the performance was pretty poor. To verify that indexes have been applied properly, I ran the following program:
private static void indexTest(ObjectContainer db){
for (StoredClass storedClass : db.ext().storedClasses()) {
for (StoredField field : storedClass.getStoredFields()) {
if(field.hasIndex()){
System.out.println("Field "+field.getName()+" is indexed! ");
}else{
System.out.println("Field "+field.getName()+" isn't indexed! ");
}
}
}
}
Unfortunately, the results show that no field is indexed.
On a similar context, in OME browser, I saw there's an option to create index on fields of each class. If I turn the index to true and save, it appears to be applying the change to db4o. But again, if run this sample test on the db4o file, it doesn't reveal any index.
Any pointers on this will be highly appreciated.
Unfortunately I don't know the spring extension for db4o that well.
However the Db4o.configure() stuff is deprecated and works differently than in earlier versions. In earlier versions there was a global db4o configuration. Not this configuration doesn't exist anymore. The Db4o.configure() call doesn't change the configuration for running object containers.
Now you could try to do this work around and a running container:
container.ext().configure().objectClass(com.test.Metadata.class).objectField("id").indexed(true);
This way you change the configuration of the running object container. Note that changing the configuration of a running object container can lead to dangerous side effect and should be only used as last resort.