I have a rule whose output files are symbolic link files. Even though the link files are being made, Snakemake exits with a MissingOutputException and lists the output files as being missing. If instead of making a symlink with "ln -s" I copy the files with "cp -p" it works. I tried increasing the --latency-wait but it made no difference.
Sounds like you are using relative path for source file when symlinking. Use absolute path.
Snakemake sees broken symlinks as missing output.
Related
myProgram takes three files as inputs, like so:
$ myProgram inputA inputB inputC
And say these inputs themselves reside in their own respective directories w/ some additional files:
directoryA
inputA
inputA_helperfile1
inputA_helperfile2
directoryB
inputB
inputB_helperfile1
inputB_helperfile2
directoryC
inputC
inputC_helperfile1
inputC_helperfile2
myProgram will not run properly unless all three inputs as well as these additional files (dependencies? Is that the right term?) are in the same directory. But I do not want to put all these files into the same directory in order to execute myProgram. Is there a workaround for this scenario?
I am very new to bash (and programming/scripting in general), so please forgive me if this is a trivial question! (It is non-trivial to me, and I was unable to find an adequate answer by Googling for it.)
It might be easier to propose a good solution if you would explain what exactly myProgram is. Did you implement it?
Is it documented that it requires all files to be in one directory?
What happens if you call your program like this?
myProgram directoryA/inputA directoryB/inputB directoryC/inputC
If myProgram requires that all files are in the same directory, you could write a script that creates a temporary directory, changes the working directory into this temporary directory, copies all files there, executes myProgram inputA inputB inputC, leaves the temporary directory and removes it including all contents.
Instead of copying the files you can also create symbolic links in the temporary directory if your file system allows this.
You probably would implement your script to be called like this
myScript directoryA/inputA directoryB/inputB directoryC/inputC
You could use dirname and find to list all files from directory[ABC] if your program needs all files that reside in these directories. Otherwise you have to specify how to find out which of all the files are inputA_helperfile1 etc.
You may have to handle duplicate file names. If e.g. inputA_helperfile1 and inputB_helperfile1 would actually be the same file names with different content you cannot copy both files into the same directory.
I need my client to be able to see the file in the directory they are allowed on. So I soft link the directory they are allowed on but can't see the files inside even tho they have the right (rwx).
ex:
/home/user1/project1.link/(couple of files)**
/clients/client_shamwow/project1/(couples of files)
**: Can't see the files.
This is the line I used:
ln -s /clients/client_shamwow/projet_prod /home/user1/projet_prod
is there something wrong that I am doing so they can't see the files in project_prod or I should use something else?
Your command doesn't match your example, but I assume you mean /home/user1/project1.link is a soft (symbolic) link, and when you run ls it lists just that name, rather than the contents of the directory the link points to. If that's the case, add the -L option to your ls command.
ls -lL /home/user1/project1.link
The man page says:
-L, --dereference
when showing file information for a symbolic link, show information
for the file the link references rather than for the link itself
Another way is simply to append /. to the end of your command, as in
ls -l /home/user1/project1.link/.
If that doesn't answer your question, I think you need to be more clear, and perhaps clean up the inconsistencies in your question. Even show some real output and the commands you ran.
Solved. No idea what happend. I just recreated the link the exact same way I did before and now I am able to see AND modify the files as the user1 w/o him being able to go anywhere else than what is in the folder project_prod. Thx for your time :)
lftp can get the files which symbolic links point to, but can it get these files if they are in directories represented by symbolic links? For example, I am looking to get files at
ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Acinetobacter_nosocomialis/all_assembly_versions/GCA_000162375.2_Acin_sp_RUH2624_V1/
where /GCA_000162375.2_Acin_sp_RUH2624_V1/ is a symbolic link to a directory.
I tried adding set ftp:list-options "-La" to ~/.lftprc, ~.lftp/rc, and /etc/lftp.conf.
This is the command I am using:
lftp -c 'open -e "mirror -c -p --no-empty-dirs -I *.gz /genomes/genbank/bacteria/Acinetobacter_nosocomialis/ ~/ncbi_bacteria_mirror" ftp.ncbi.nlm.nih.gov'
This command DOES work on ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Pseudomonas_sp._URMO17WK12_I11/all_assembly_versions/
where /all_assembly_versions/ is not a symbolic link. It does not however recursively follow the symbolically linked directories contained within and get the files from those directories, which I would like for it to do if possible.
Resolving the file name to the file on disk and thereby resolving symbolic links is done at the server. Thus there is no need for the ftp client to resolve these links, which means it will not even attempt it by trying to guess the format of the directory listing.
I am not sure I understand the question, and the answer is hidden in it: the -L flag is what you want.
Quoting lftp's man page:
-L, --dereference download symbolic links as files
That's really it!
I want to copy all the contents from one directory to another (including broken symlinks) in my Ruby script. I am using FileUtils.cp_r 'src/.', 'dest' but it is complaining about the broken symlinks. Can someone please help me with this? It is a show-stopper for me right now.
FileUtils.cp_r internally copies the src folder recursively to dest. When it finds a symlink, it will create a symlink using File#symlink method (Refer line 1369 of fileutils.rb).
The documentation of File#symlink states that:
Creates a symbolic link called new_name for the existing file
old_name. Raises a NotImplemented exception on platforms that do not
support symbolic links.
So, it seems that it may not be possible to use FileUtils.cp_r to copy directories if the one of the symlinks in it is broken and pointing to a non-existing file.
Workaround
You can execute shell command cp -r command from your ruby script, it may not be platform-independent code and may not be easy to debug, but it will help you to get around the given scenario which you consider to be a show-stopper.
src = "/path/to/src/dir"
dest = "/path/to/dest/dir"
`cp -r #{src} #{dest}`
Should be an easy question for the gurus here, though it's hard to explain it in text so hopefully this is clear. I've got two directories on a box with some flavor of unix on it. I've got a script that I want to use to move all the files and directories from one location to another.
First, an example of how the directories look:
Directory A: final/results/2012/2012-02/2012-02-25/name/files
Directory B: test/results/2012/2012-02/2012-02-24/name/files
So you see they're very similar. What I want to do is move everything from the Directory B 2012 directory, recursively, to the same level of Directory A. So you'd end up with:
someproject/results/2012/2012-02/2012-02-25/name/files
someproject/results/2012/2012-02/2012-02-24/name/files
etc.
I want this script to be future proof though, meaning I don't want the 2012 hardcoded. Also, towards the end of a month you will potentially have data from two different months and both need to be copied into the 2012 directory. So here is the command I used in the shell script file:
CONS="/someproject";
ROOT="/test";
/bin/cp -r ${ROOT}/results/* ${CONS}/results/*
but this resulted in:
/final/results/2012/2012-02/2012-02-25/name/files
and
/final/results/2012/2012/2012-02/2012-02-24/name/files
So as I hope is clear, it started a level below where I wanted it too. Can anyone fill me in on what I'm doing wrong, if they can understand what I'm even trying to explain. My apologies if it's not clear. I'm sure this is a fairly simple fix but I'm not sure what to do. Shell scripting is not a strong point of mine.
One poster suggests rsync, which is overkill.
cp -rp will work fine. if you want to move the files, just mv the directory -- it and everything under it will move too.
The only real problem here is the use of terminating *'s in the command line in the original script. You don't need the *, you're just trying to pass directories to the cp command, you aren't trying to pass it the names of all the files already in the source (and more importantly, the destination).
You could also use a tool like rsync to make sure your source and target are synchronized.
rsync -av ${ROOT}/results/ ${CONS}/results/
You specified that you want to "move" the files, though. Which means deleting the originals after they're copied:
rsync -av --remove-source-files ${ROOT}/results/ ${CONS}/results/
If you start playing around with rsync, be sure to read the man page about how it treats trailing slashes.