I created this little Python script Essai_Bash.py to make some tests:
#!/usr/bin/python
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument('-i', action='store', dest='InputDir', help='Working Directory') # Empty folders for outputs.
parser.add_argument('--version', action='version', version='%(prog)s 0.1')
results = parser.parse_args()
print 'Current input directory =', results.InputDir
dir_path=str(os.path.abspath(results.InputDir)) # Retrieving an output folder name to use as species ID:
path,CodeSp = os.path.split(dir_path)
print "Currently working on species: "+str(CodeSp)
Back to my shell, I type the following command, expecting my script to run on each directory that is present in my "Essai_Bash" folder:
listdir='ls ../Main_folder/' # I first used backtips instead of simple quotes but it did not work.
for dir in $listdir; do ./Essai_Bash.py -i ../Main_folder/$dir; done
I am surely missing something obvious but it does not work. It seems like $listdir is considered as a characters strings and not a list of directories. However, just typing $listdir in my shell actually gives me this list!
Just use glob extension, parsing ls output is not safe.
Also dir variable already contains ../Main_folder/
listdir=( ../Main_folder/*/ )
for dir in "${listdir[#]}"; do ./Essai_Bash.py -i "$dir"; done
Related
I want to schedule my Perl code to be run every day at a specific time. so I put the below code in bash file:
Automate.sh
#!/bin/sh
perl /tmp/Taps/perl.pl
The schedule has been specified in below path:
10 17 * * * sh /tmp/Taps/Automate.sh > /tmp/Taps/result.log
When the time arrived to 17:10 the .sh file hasn't been running. however, when I run ./Automate.sh (manually) it is running and I see the result. I don't know what is the problem.
Perl Code
#!/usr/bin/perl -w
use strict;
use warnings;
use Data::Dumper;
use XML::Dumper;
use TAP3::Tap3edit;
$Data::Dumper::Indent=1;
$Data::Dumper::Useqq=1;
my $dump = new XML::Dumper;
use File::Basename;
my $perl='';
my $xml='';
my $tap3 = TAP3::Tap3edit->new();
foreach my $file(glob '/tmp/Taps/X*')
{
$files= basename($file);
$tap3->decode($files) || die $tap3->error;
}
my $filename=$files.".xml\n";
$perl = $tap3->structure;
$dump->pl2xml($perl, $filename);
print "Done \n";
error:
No such file or directory for file X94 at /tmp/Taps/perl.pl line 22.
X94.xml
foreach my $file(glob 'Taps/X*') -- when you're running from cron, your current directory is /. You'll want to provide the full path to that Taps directory. Also specify the output directory for Out.xml
Cron uses a minimal environment and a short $PATH, which may not necessarily include the expected path to perl. Try specifying this path fully. Or source your shell settings before running the script.
There are a lot of things that can go wrong here. The most obvious and certain one is that if you use a glob to find the file in directory "Taps", then remove the directory from the file name by using basename, then Perl cannot find the file. Not quite sure what you are trying to achieve there. The file names from the glob will be for example Taps/Xfoo, a relative path to the working directory. If you try to access Xfoo from the working directory, that file will not be found (or the wrong file will be found).
This should also (probably) lead to a fatal error, which should be reported in your error log. (Assuming that the decode function returns a false value upon error, which is not certain.) If no errors are reported in your error log, that is a sign the program does not run at all. Or it could be that decode does not return false on missing file, and the file is considered to be empty.
I assume that when you test the program, you cd to /tmp and run it, or your "Taps" directory is in your home directory. So you are making assumptions about where your program looks for the files. You should be certain where it looks for files, probably by using only absolute paths.
Another simple error might be that crontab does not have permission to execute the file, or no read access to "Taps".
Edit:
Other complications in your code:
You include Data::Dumper, but never actually use that module.
$xml variable is not used.
$files variable not declared (this code would never run with use strict)
Your $files variable is outside your foreach loop, which means it will only run once. Since you use glob I assumed you were reading more than one file, in which case this solution will probably not do what you want. It is also possible that you are using a glob because the file name can change, e.g. X93, X94, etc. In that case you will read the last file name returned by the glob. But this looks like a weak link in your logic.
You add a newline \n to a file name, which is strange.
I'm trying to write a file with a python program. When I perform all the actions command line, they all work fine. The file is created.
When I perform the actions in a python script, the file does not exist after the script terminates.
I created a small script that demonstrates the behavior.
import os
import os.path
current_dir = os.getcwd()
output_file = os.path.join(current_dir, "locations.js")
print output_file
f = open(output_file, "w")
f.write("var locations = [")
f.write("{lat: 55.978467, lng: 9.863467}")
f.write("]")
f.close()
if os.path.isfile(output_file):
print output_file + " exists"
exit()
Running the script from the command line, I get these results:
D:\Temp\GeoMap>python test.py
D:\Temp\GeoMap\locations.js
D:\Temp\GeoMap\locations.js exists
D:\Temp\GeoMap>dir locations.js
Volume in drive D is Data
Volume Serial Number is 0EBF-9720
Directory of D:\Temp\GeoMap
File Not Found
D:\Temp\GeoMap>
Hence the file is actually created, but removed when the script terminates.
What do I need to do the keep the file?
Problem was solved by changing firewall settings.
I am looking for a way to look through a list of thousands of file names, search for them within a certain set of subfolders on a server and to export a list of those files locations.
Example:
Saved in Textedit or Excel I have:
12345.psd
67890.psd
These need to be searched for here:
Servername:Image Bank:Photography:
I then need a list of the locations of these files exported into TextEdit or Excel to get a list such as:
Servername:Image Bank:Photography:Theme name:12345.psd
Servername:Image Bank:Photography:Theme name:67890.psd
Manually searching for individual files is taking an age. I am new to both Automator and Applescript, so am learning on the fly. So far I have scoured forums and tutorials, gone through it with friends, colleagues and IT and spent hours testing and trialing with no success.
Does this at least sound possible? If so, I will continue trying as this will be invaluable in future projects.
As an example and test...
I created a hierarchical folder structure at 'Servername:Image Bank:Photography' containing three 'Theme name' folders which between them contains 90,000 .psd files (30,000 files in each 'Theme name' folder).
I then created a list of 3,100 .psd filenames. This list contains only the filename and .psd extension and the filenames are spread out over the names of the 90,000 .psd files created for this test so as to have to progressively read through the entire master list, as necessary, that gets created to get the fully qualified path filenames.
I then wrote this bash script in Terminal:
touch getpfn
nano getpfn
In nano I typed the following:
#!/bin/bash
master_list="The_PSD_Path_Filenames_List_on_Servername_at_Image_Bank_Photography.txt"
get_list="The_Get_Path_Filenames_List.txt"
the_list="The_Wanted_Path_Filenames_List.txt"
psd_file_path='/Volumes/Image Bank/Photography'
find "$psd_file_path" -type f -iname '*.psd' > "$master_list"
while read -r line; do
grep -m 1 "$line" "$master_list" >> "$the_list"
done<"$get_list"
Saved the changes and exited nano by pressing the following key sequences: ControlXYEnter
Back in Terminal:
Made the script executable:
chmod u+x getpfn
Executed the script:
./getpfn
On my system it took under six minutes to get the 3,100 fully qualified path filenames of PSD files on the server against the list of 90,000 entries. Understandably so, YMWV based on all the relevant factors!
I have included a commented version of this bash script below:
#!/bin/bash
# Define Variables
# These filenames are arbitrary but should adequately
# differentiate their functionality within this script.
# Set them to whatever is appropriate for your usage.
# 'master_list' gets created by the output of the 'find'
# command. It will contain the fully qualified path filenames
# of all PSD files within Servername:Image Bank:Photography
# mounted at '/Volumes/Image Bank' using: 'psd_file_path'
master_list="The_PSD_Path_Filenames_List_on_Servername_at_Image_Bank_Photography.txt"
# 'get_list' Is a plain text file created from the spreadsheet.
# It contains one filename, e.g. 12345.psd, per line. Filenames
# can be numeric and or alphanumeric, doesn't matter as long as
# there is only one filename per line and is a plain text file.
get_list="The_Get_Path_Filenames_List.txt"
# 'the_list' gets created by the output of the 'grep' command.
# Each line of 'master_list' that contains the filename on
# the processed line of the 'get_list' is written to 'the_list'
# 'the_list' then contains the fully qualified pathnames
# of the filenames in 'get_list' based on 'psd_file_path'.
the_list="The_Wanted_Path_Filenames_List.txt"
# Note: This script assumes that 'Image Bank' is the mounted share
# of 'Servername' and is accessible at: '/Volumes/Image Bank'
psd_file_path='/Volumes/Image Bank/Photography'
# Create the Master PSD Path Filename List
find "$psd_file_path" -type f -iname '*.psd' > "$master_list"
# Loop through the e.g. 'Get_Path_Filenames_List.txt' against the e.g.
# 'The_PSD_Path_Filenames_List_on_Servername_at_Image_Bank_Photography.txt'
# while writing matches to e.g: The_Wanted_Path_Filenames_List.txt
while read -r line; do
grep -m 1 "$line" "$master_list" >> "$the_list"
done<"$get_list"
# Now finished, the e.g. 'The_Wanted_Path_Filenames_List.txt' file
# contains the mounted path filenames of all the filenames in
# the e.g. 'Get_Path_Filenames_List.txt' file.
#
# Depending on what will be done with this list, further
# processing can be done as needed and or wanted.
I'm not very good in shell scripting and would like to ask you some question about looping of files big dataset: in my example I have alot of files with the common .pdb extension in the work dir. I need to loop all of them and i) to print name (w.o pdb extension) of each looped file and make some operation after this. E.g I need to make new dir for EACH file outside of the workdir with the name of each file and copy this file to that dir. Below you can see example of my code which are not worked- it's didn't show me the name of the file and didn't create folder for each of them. Please correct it and show me where I was wrong
#!/bin/bash
# set the work dir
receptors=./Receptors
for pdb in $receptors
do
filename=$(basename "$pdb")
echo "Processing of $filename file"
cd ..
mkdir ./docking_$filename
done
Many thanks for help,
Gleb
If all your files are contained within the .Repectors folder, you can loop each of them like so:
#!/bin/bash
for pdb in ./Receptors/*.pdb ; do
filename=$(basename "$pdb")
filenamenoextention=${filename/.pdb/}
mkdir "../docking_${filenamenoextention}"
done
Btw:
filenamenoextention=${filename/.pdb/}
Does a search replace in the variable $pdb. The syntax is ${myvariable/FOO/BAR}, and replaces all "FOO" substrings in $myvariable with "BAR". In your case it replaces ".pdb" with nothing, effectively removing it.
Alternatively, and safer (in case $filename contains multiple ".pdb"-substrings) is to remove the last four characters, like so: filenamenoextention=${filename:0:-4}
The syntax here is ${myvariable:s:e} where s and e correspond to numbers for the start and end index (not inclusive). It also let's you use negative numbers, which are offsets from the end. In other words: ${filename:0:-4} says: extract the substring from $filename starting from index 0, until you reach fourth-to-the-last character.
A few problems you have had with your script:
for pdb in ./Receptors loops only "./Receptors", and not each of the files within the folder.
When you change to parent directory (cd ..), you do so for the current shell session. This means that you keep going to the parent directory each time. Instead, you can specify the parent directory in the mkdir call. E.g mkdir ../thedir
You're looping over a one-item list, I think what you wanted to get is the list of the content of ./Receptors:
...
for pdb in $receptors/*
...
to list only file with .pdb extension use $receptors/*.pdb
So instead of just giving the path in for loop, give this:
for pdb in $receptors/*.pdb
To remove the extension :
set the variable ext to the extension you want to remove and using shell expansion operator "%" remove the extension from your filename eg:
ext=.pdb
filename=${filename%${ext}}
You can create the new directory without changing your current directory:
So to create a directory outside your current directory use the following command
mkdir ../docking_$filename
And to copy the file in the new directory use cp command
After correction
Your script should look like:
receptors=./Receptors
ext=.pdb
for pdb in $receptors/*.pdb
do
filename=$(basename "$pdb")
filename=${filename%${ext}}
echo "Processing of $filename file"
mkdir ../docking_$filename
cp $pdb ../docking_$filename
done
The following code includes a command and a string:
files = `ls /tmp`
I would like /tmp to be a variable instead of a static string, and would ideally like it to be like:
dir = '/tmp'
command = 'ls ' + dir
files = `command`
What is the correct Ruby syntax to achieve this?
Use string interpolation:
dir = '/tmp'
files = `ls #{dir}`
files = `#{command}`
Is that what you are looking for ?
Use the standard shellwords library. It will take care of proper escaping, which will help to protect you from shell injection attacks.
require 'shellwords'
command = [
'ls',
dir
].shelljoin
files = `#{command}`
If dir comes from untrusted input, the above code still allows someone to see any directory on your system. However, using shelljoin protects you from someone injecting, for example, a "delete all files on my hard drive" command.
In the particular case of listing a directory, The built-in class Dir will do that rather well:
files = Dir[File.join(dir, '*')]
Here we add a glob onto the end of the directory using File::join. Dir::[] then returns the paths of the files in that directory.