I would like to delete a file after all the rows in the file have been processed.
My streams look like
source (file --fixedDelay=0 --outputType=text/plain --dir=XXX) |
splitter --expression=payload.split('\\n') |
transform -> filter -> sink
My files are stored in a directory, that is being watched by the file module. I would like that each file is deleted after it has been processed.
Thanks.
Indeed it is surprising that the file source does not have an option for deleting the file after processing, which can be confirmed by looking at the configuration file in xd/modules/source/file/config/file.xml (as of version 1.1.0).
While the file source does not have this option, the sftp source does have it. Hence you could use the sftp source. This will require an ssh server on the machine where spring xd is installed. Does this help?
You may also want to add your own custom source module by configuring a file transformer. File transformers support a delete-files="true" option:
http://docs.spring.io/spring-integration/reference/html/files.html
Related
I am trying to configure NXlog to work with AlienVault based on the guide here
I installed the custom config file from AlienVault and modified the destination IP only. When I did this I could not get the NXlog service to start - Then I reinstalled the default config but I still cannot get it to open.
I edited the file in notepad which I thing should be safe, however I have read here that it is possibly the UTF-8 BOM - I am not sure how to check if there is one but I do not believe there is because I only used notepad.
The first line in the config file looks like so:
define ROOT C:\Program Files (x86)\nxlog
The NXlog Log file with the errors is only displaying this error:
nxlog failed to start: Invalid keyword: define at C:\Program Files (x86)\nxlog\conf\nxlog.conf:1
Not very helpful - Seems to be choking on the very first word - Anyone seen this before???
I'm pretty sure that's caused by the UTF-8 BOM in your config file. I suggest using and checking with an editor that can handle this. In HEX mode you can confirm whether the file has a BOM or not.
The NXLog EE v4.0 can cope with the BOM properly BTW.
As B0ti mentioned, my problem was caused by the BOM - I couldn't figure out how to fix this on windows so I downloaded the file into a Linux environment and fixed it there. To do so follow these steps -
First I verified there was a BOM in place with the file command:
ex: file filename.txt -This will print information about the file - if there is a BOM you will see that.
Next I followed the answer here for removing the BOM:
Basically just do this in the Linux box - sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt
Then I transferred the new file back to the Windows box and all was right with the world!
I want to scan a text file such as how much line in a .sh file. In sonarqube6.7 unknown files = true option is missing. I was in sonar-project.properties file to configure sonar.import_unknown_files = true also was not used. Does a higher version support unknown files?
The property sonar.import_unknown_files has no effect with SQ 6.7. By default, all the files are indexed (SONAR-8623). Then SQ keeps only the files on which a given Sensor (a plugin) saved something on it (SONAR-8631).
For your case, you need a custom plugin that:
declares a new language dealing with .sh files (similar to: https://github.com/SonarSource/sonar-custom-plugin-example/blob/master/src/main/java/org/sonarsource/plugins/example/languages/FooLanguage.java)
implements a Sensor implementing some logic to raise issues/metrics on your .sh files
I have several configuration files in /etc/rsyslog.d, e.g., 01-templates.conf, 02-error-logs.conf, 03-system-logs.conf, and have the following line in my /etc/rsyslog.conf file
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
What order do the /etc/rsyslog.d/*.conf files get loaded in? Where is this documented? I read the Rsyslog configuration page but did not see it.
According to this and this post to the rsyslog-users mailing list, it's processed in alphabetical order.
But with the exception of some 7.2.x and 7.3.x versions where a bug caused them to be read in reverse order.
I have my camel configured to download all files from a specific FTP directory. Now this is all easy enough and everything seems to be working fine. However, I am running into errors when the files contain a space in their names such as File 123.csv. I know I could specifically target the files with an escape character. The only difficulty is that these files are dynamic in nature and change daily, so I will not know which files may or may not have spaces.
I figure I can just read all the file URI and make adjustments from there. But I was wondering if there is any Camel specific way to handle this.
Errors: java.lang.IllegalArgumentException: Illegal character in path at index 60: hdfs://test.net/user/CamelTests/File Layout.csv
GenericFileOnCompletion - Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy#fe8d1b for file: RemoteFile[File Layout.csv]
Camel Code
from("{{ftp.serverLP}}/Memo/Loss?username=ftp&password=pass")
.to("hdfs2://Test.net/user/CamelTests/?fileSystemType=HDFS")
.log("Downloaded file ${file:name} complete.");
Try changing the .to(..) to use a non-HDFS file system.
The error posted seems to indicate a problem with the destination to which the files are being copied (HDFS), not the FTP source.
I've set up a DSC PullServer on Server 2008 R2 and I've run into an issue deploying a custom module. The module on the PullServer is in C:\Program Files\WindowsPowerShell\Modules\NTFSPermission, I've zipped everything using Windows Explorer (and 7-zip) and placed the zip in C:\Program Files\WindowsPowerShell\DscService\Modules\NTFSPermission_1.0.zip, I create a checksum using NEW-DSCChecksum for the zip file and I've got Import-DSCResource NTFSPermission in my node configuration. When I run the Invoke-CimMethod to push the config to a node it errors out with:
Invoke-CimMethod : Failed to extract the module from zip file
C:\Windows\TEMP\\635291179507191263\NTFSPermission_1.0.zip downloaded by Download
Manager WebDownloadManager.
The node logs a 4104 error in the event viewer reading:
This event indicates that failure happens when LCM is trying to get the configuration from pull server using download manager WebDownloadManager. ErrorId is 0x1. ErrorDetail is Failed to extract the module from zip file C:\Windows\TEMP\\635291179507191263\NTFSPermission_1.0.zip downloaded by Download Manager WebDownloadManager.
Has anyone else run into this? I'm not sure if there's a config file somewhere that is literally piping in an extra backslash and causing an invalid download/extraction path or if there is something wrong with the zipped module. If I move the module over manually the config will be pushed and apply successfully, which leads me to believe it's not the module.
There is a known issue with extracting zip files created using .NET compression class in 4.5. The workaround is to use the shell method of compressing files. That is, send to compressed archive option in right-click context menu or using Shell COM object.
I tested one of the modules from the DSC Resource Kit and it extracted with no problems on the desired node. I tracked the problem down to the psd file that the author created for their DSC-Resource. I updated the CLRVersion to require 4.0 and removed RequiredModules, RequiredAssemblies, ScriptsToProcess, TypesToProcess,FormatsToProcess and NestedModules (all were empty values anyway), pushed the config without the workaround and it downloaded and extracted the resource.