Trying to set up simple file copy processin spring-xd:
stream create --name mystrea --definition "file --dir=/path/source
--fixedDelay=5 | sink:file --dir=/path/dest --binary=true
--name=headers['file_name']"
This seems to create and append fils to the file header['file_name'].out in the dest folder
Looking at sink:file definition
<file:outbound-channel-adapter id="files"
mode="${mode}"
charset="${charset}"
directory="${dir}"
filename-generator-expression="'${name}' + '${extensionWithDot}'"/>
I see it puts '' around the name which causes it not to be evaluated.
Any suggestions besides create new sink:simplefile module that would do what I am looking for ? Am I missing something
Yes, the standard sink is not designed to do what you are trying to do (pass in an expression for the filename).
We should add an alternative property --fileNameExpression=... or similar.
In the meantime, you are correct, you'll need a custom sink (or modify the standard one).
I created a JIRA Issue for this enhancement.
Related
I'm trying to create a temporary file in my pipeline, then use that file in another rule.
For example, I have two rules in a .smk file:
#Unzip adapter trimmed fastq file
rule unzip_fastq:
input:
'{sample}.adapterTrim.round2.fastq.gz',
output:
temp('{sample}.adapterTrim.round2.fastq')
conda:
'../envs/rep_element.yaml'
shell:
'gunzip -c {input[0]} > {output[0]}'
#Run bowtie2 to align to rep elements and parse output
rule parse_bowtie2_output_realtime:
input:
'{sample}.adapterTrim.round2.fastq'
output:
'rep_element_pipeline/{sample}.fastq.gz.mapped_vs_' + config["ref"]["bt2_index"] + '.sam'
params:
bt2=config["ref"]["bt2_index_path"], eid=config["ref"]["enst2id"]
conda:
'../envs/rep_element.yaml'
shell:
'perl ../scripts/parse_bowtie2_output_realtime_includemultifamily.pl '
'{input[0]} {params.bt2} {output[0]} {params.eid}'
{sample}.adapterTrim.round2.fastq is used once and should ultimately be deleted upon completion. However, I'm finding that this file is uploaded to Amazon S3, even with the addition of temp(). I'm also finding that this file is removed locally, but still persists on S3.
Am I doing this correctly? '{sample}.adapterTrim.round2.fastq' is not currently written in the rule-all of the Snakefile.
We ultimately need to prevent this file from being uploaded to S3, so if there is a way to specify not to upload this file in the rule, that would be useful.
It seems that the snippet in the question is not consistent with actual use, since for S3 files one would need to wrap file names in remote.
However, as a general solution, documentation contains the following:
The remote() wrapper is mutually-exclusive with the temp() and protected() wrappers.
Hence, if you intend to use a temp file, make sure it's not wrapped in remote, or explicitly wrap the file in local.
In windows 10, I followed the step-by-step MXnet tutorial to use im2rec.py to create a dataset. I created a image list file like this:
integer_image_index \t label_index \t path_to_image
Next, I modified .txt to .lst.
Finally, I executed the command:
python im2rec.py --exts '.jpg' --train-ratio 0.41 --test-ratio 0.49 --recursive=True --pack-label=True D:\CUB_200_2011\data\image_label.lst D:\CUB_200_2011\CUB_200_2011\image
It is shown that "read no error", but the files created by the command like .lst and .rec are 0K, there is empty. I don't know why.
Please tell me what mistakes I made.
im2rec.py will print
read none error:(filename)
for any file that it can't load for whatever reason. Maybe some of the files you list aren't there or are empty? Or maybe the base path you've specified is wrong -- I notice you have the folder name CUB_200_2011 twice.
qrouter [-c < config_name>] [options] < basename>
where <basename> is without an extension.
File <basename>.def is assumed to exist
and to define cell placement and netlist
information. File <config_name> is
assumed to exist and contains basic
routing parameters, or points to a LEF
file containing detailed routing parameters.
If this option is not specified, then the
default configuration file name of "route.cfg"
is used.
How to write a .cfg file that points to an existing file?
The commend is used to run an open source VLSI routing tool called Qrouter, an example .cfg file can be like below:
lef /usr/lib/ibm01.lef
Hi i'm using apache camel 2.9 with spring. My requirement is this.
camel looks for a file in a specific directory (e.g import) and the file format is this test_22-10-2015_p1.psv
After the file has been processed i need to rename the file to test_22-10-2015_p1_ACK.psv and keep it in the same folder without moving it to the .camel directory.
Is this possible
Thanks in advance
Yes read the documentation about the file file component and you can find the move option.
http://camel.apache.org/file2
You can use an expression to define the file name which uses the simple/file language
http://camel.apache.org/file-language.html
http://camel.apache.org/simple.html
So it would be something along the lines of
move=test_${file:name.noext}_ACK.${file:name.ext}
And then you need to make a exclude to skip files starting with test_ as you move the file to the same directory.
exclude=test_.*
I am creating files from within Ruby scripts and adding stuff to them. But where are these files stored that I am creating?
I'm very new to this, sorry!
The files are created at whatever location you specified. For instance:
f = File.new("another_test.txt","w+")
that will create the file in the current working directory. You specify the path along with the file name. For example:
f = File.new("~/Desktop/another_test.txt","w+") # will create the file on the desktop.
For more details, check the File documentation.
Updated:
Included mu is too short correction.