I am looping through urls in a file to download their data and each url represents data for each hour of the day. I want to name the file the date and hour that it comes from. The following doesn't work, but I'm not quite sure why
myDate=$(date -v -1d '+%Y/%m/%d')
for hour in {0..23}
do
...
...
#set file name
name=$myDate.$hour.txt
curl -L -o $name "https://..."
done
I think it's just a problem with the syntax for $name in the curl statement, but I don't know how to correct it.
I get the following error
Warning: Failed to create the file 2019/09/18-0: No such file or directory
curl: (23) Failed writing body (0 != 16360)
date -v -1d '+%Y/%m/%d' returns a string containing slashes, which are used as path separators, so, for example, in:
iMac-ForceBru:~ forcebru$ date -v -1d '+%Y/%m/%d'
2019/09/18
The 2019/09/18 would be treated as a file called 18 in directory 09, which is in turn inside directory 2019. It looks like the path 2019/09 doesn't exist on your system, so the file 2019/09/something.txt can't be created.
Related
I want to use the find command to find these directories:
Access: 2013-12-13 10:59:46.190886900 -0500
Modify: 2013-12-03 07:04:02.995890600 -0500
Change: 2013-12-03 07:04:02.995890600 -0500
Birth: 2013-12-02 07:04:02.000000000 -0500 (I want a time after '12-03')
This is the command I ran but it still lists older directories:
find . -type d -newerBt '2013-12-03 00:00:00' -exec du -h {} \;
How can I modify this line to find the directories created after that date? What is the difference between -newerct and -newerBt. I think I want the birth date.
Note: I am running this with the latest cygwin.
You could use stat instead:
find . -type d -exec bash -c '(( $(stat -c %W "{}") > $(date +%s -d '2013-12-03') )) && du -h "{}"' \;
You are finding directories, but showing files contained therein.
Those files may have birth dates that lie before that of the containing directory. For example, create a file, then a directory, and move the file into that directory.
This is the difference between birth date and change date. If a file is moved into the dir, the dir is changed, so I think -newerct is what you want.
I wonder if it's a TimeZone issue? What is echo $TZ? What happens if you do unset TZ (and unsetenv TZ, too, in csh) and re-try the same commands?
Here's the man page excerpt for -newerXY. Maybe reading it will trigger some thoughts?
-newerXY reference
Compares the timestamp of the current file with reference. The reference
argument is normally the name of a file (and one of its timestamps is used
for the comparison) but it may also be a string describing an absolute time.
X and Y are placeholders for other letters, and these letters select which
time belonging to how reference is used for the comparison.
a The access time of the file reference
B The birth time of the file reference
c The inode status change time of reference
m The modification time of the file reference
t reference is interpreted directly as a time
Some combinations are invalid; for example, it is invalid for X to be t.
Some combinations are not implemented on all systems; for example B is not
supported on all systems. If an invalid or unsupported combination of XY is
specified, a fatal error results. Time specifications are interpreted as for
the argument to the -d option of GNU date. If you try to use the birth time
of a reference file, and the birth time cannot be determined, a fatal error
message results. If you specify a test which refers to the birth time of
files being examined, this test will fail for any files where the birth time
is unknown.
I'm writing a script that takes a list of ~300 URLs as input, which have the following format:
http://long.domain.prefix/folder/subfolder/filename.html
Of that URL, I'd like to save filename.htmlin ./folder/subfolder/ - if that folder structure doesn't exist, it must be created. This works, the folders are being written to disk, however no files are downloaded.
My script looks like this:
#!/bin/bash
for line in `cat list.txt`; do
# strips the URL prefix and trailing slash
name=${line#http://long.domain.prefix\/}
/usr/bin/curl -m 10 -f -o $name --create-dirs $fullname
done;
For some reason, the $name variable is cut off after exactly 74 characters, which obviously results in HTTP error codes. I can't give out the exact URLs, but rest assured they are correct, as long as the full URL is being used.
How can I prevent this odd cutting-off behavior?
Thanks to Etan Reisner, the solution was to convert the file to Unix-Style line endings.
I have a parameter file(parameter.txt) which contain like below:
SASH=/home/ec2-user/installers
installer=/home/hadoop/path1
And My shell script(temp_pull.sh) is like below:
EPATH=`cat $1|grep 'SASH' -w| cut -d'=' -f2`
echo $EPATH
${EPATH}/data-integration/kitchen.sh -file="$KJBPATH/hadoop/temp/maxtem/temp_pull.kjb"
When I run my temp_pull.sh like below:
./temp_pull.sh parameter.txt
$EPATH gives me correct path, but 3rd line of code takes only partial path.
Error code pasted below:
/home/ec2-user/installers-->out put of 2nd line
/data-integration/kitchen.sh: No such file or directory**2-user/installer** -->out put of 3rd line
There is no need to manually parse the values of the file, because it already contains data in the format variables are defined: var=value.
Hence, if the file is safe enough, you can source the file so that SASH value will be available just by saying $SASH.
Then, you can use the following:
source "$1" # source the file given as first parameter
"$SASH"/data-integration/kitchen.sh -file="$KJBPATH/hadoop/temp/maxtem/temp_pull.kjb"
The problem is file which we were using was copied from windows to UNIX.So delimiter issue are the root cause.
By using dos2unix paramfile.txt we are able to fix the isue.
command:
dos2unix paramfile.txt
This will convert all the delemeter of windows to unix format.
I need to access two files in my shell script. The only issue is , I am not sure what the file name is going to be as it is system generated.A part of the file name is always constant , but the rest of it is system generated , hence may vary. I am not sure how to access these files?
Sample File Names
Type 1
MyFile1.yyyy-mm-dd_xx:yy:zz.log
In this case , I know MyFile1 portion is a constant for all the files, the other portion varies based on date and time. I can use date +%Y-%m-%d to get till MyFile1.yyyy-mm-dd_ but I am not sure how to select the correct file. Please note each day will have just one file of the kind. In unix the below command gives me the correct file .
unix> ls MyFile1.yyyy-mm-dd*
Type 2
MyFile2.yyyymmddxxyyxx.RandomText.SomeNumber.txt
In this file , as you can see Myfile2 portion is common,I can user Date +%Y%m%d to get till (current date) MyFile2.yyyymmdd, again not very clear how to go on from there .In unix the below command gives me the correct file .Also I need to have previous date in the dd column for File 2.
unix> ls MyFile2.yyyymmdd*
basically looking for the following line in my shell script
#!/bin/ksh
timeA=$(date +%Y-%m-%d)
timeB=$(date +%Y%m)
sysD=$(date +%d)
sysD=$((sysD-1))
filename1=($Home/folder/MyFile1.$timeA*)
filename2=($Home/folder/MyFile2.$timeB$sysD*)
Just not sure how to get the RHS for these two files.
The result when running the above scripts is as below
Script.ksh[8]: syntax error at line 8 : `(' unexpected
Perhaps this
$ file=(MyFile1.yyyy-mm-dd*)
$ echo $file
MyFile1.yyyy-mm-dd_xx:yy:zz.log
It should be noted that you must declare variables in this manner
foo=123
NOT
foo = 123
Notice carefully, bad
filename1=$($HOME/folder/MyFile1.$timeA*)
good
filename1=($HOME/folder/MyFile1.$timeA*)
I have a bash script which backups my source code on a 10 minute basis thru crontab. Script was working until the end of August. It's not working since September 1st. This is the script:
#!/bin/sh
date=`date +%e-%m-%y`
cd /home/neky/python
tar -zcf lex.tar.gz lex/
echo $date
mv lex.tar.gz lex-$date.tar.gz
mv lex-$date.tar.gz /home/neky/Dropbox/lex/lex-$date.tar.gz
If I execute it manually, it print the current date 4-09-12, and this error mv: target ‘4-09-12.tar.gz’ is not a directory
What could be the problem?
Your date contains a space when the day of month is a single digit (which also explains why it only stopped working in the new month). That results in your command being split up, i.e.
# this is what it you end up with
mv lex.tar.gz lex- 4-09-12.tar.gz
Use date +%d-%m-%y instead which will give you 04-09-12 (note %d instead of %e).
If you really want a space in the name, you'll need to quote your variables, i.e.:
mv lex.tar.gz "lex-$date.tar.gz"
mv "lex-$date.tar.gz" /home/neky/Dropbox/lex/
The character % (part of your date format) is a special one in cron scripts, so you need to escape it.