I have Machine-A and Machine-B and both are Ubuntu servers. Now I want to list all the files on Machine-B using ssh. I want to return the result in a text file so I can analyse the result and the use scp to copy the required files.
ssh my_user_name#192.168.150.4 'bash -s tree /f'
ssh my_user_name#192.168.150.4 'bash -s ls -LR'
Now this command is not giving the result I wanted. Can anyone help with this so I can list all files on the remote computer using ssh and return the output in the form of a text file.
I am using ls -LR to list files and SSH to remote script execution.
From the Answer i worked on my problem and iam updating the question to match one little requirement.
I got the list of files throught this command ssh my_user_name#192.168.150.4 ls /something/sub > output.txt
But i want the md5sum of all files instead of names because 2 file names might get match.So is there any way to list all files and return all md5sum of all files and return to output.txt file.
Copy the file list to a valid path in Machine B and copy it back to Machine A using scp
ssh username#machineB 'ls -LR /path/to/dir > ~/fileList'
To return the md5sum of all the files in the directory, use find as
ssh username#machineB 'find /path/to/dir -type f -exec md5sum {} \; > ~/md5sum_fileList'
Now copy the file back to machine A, using a glob pattern to copy the files having the pattern fileList
scp username#machineB:~/*fileList* username#machineA:~/
All you need to do is specify the command, without using "bash". Your default shell will be used on the remote device to execute the command.
ssh remote-host command
To save the output of your ls command to a file, you can simply use the usual shell redirection:
ssh remote-host command > output.txt
Just in case you end up with multiple file names on a single line, you may need to use -1 on the ls command line. Also, remember that if a filename includes a space, you need quotes in a shell script to support those...
To run multiple commands in a row, although the output won't be as easy to manage, you use quotes and separate commands with semicolons (for example) as in:
ssh remote-host "command1; command2; command3" > output.txt
In regard to md5sum, you can run that against all the files in a directory using the find command along with md5sum:
ssh remote-host "find . -type f -exec md5sum {} \;" > output.txt
Change the path (. in the example) to whatever works for you.
Related
I'm new to linux (using bash) and I wanted to ask about something that I do often while I work, I'll give two examples.
Deleting multiple specific folders inside a certain directory.
Copying multiple specific folders into a ceratin directory.
I succesfully done this with files, using find with some regex and then using -exec and -delete. But for folders I found it more problematic, because I had problem pipelining the list of folders I got to the cp/rm command succescfully, each time getting the "No such file or directory error".
Looking online I found the following command (in my case for copying all folders starting with a Z):
cp -r $(ls -A | grep "Z*") destination
But when I execute it it says nothing and the prompt won't show up again until I hit Ctrl+C and nothing is copied.
How can I achieve what I'm looking for? For both cp and rm.
Thanks in advance!
First of all, you are trying to grep "Z*" but it means you are looking for Z, ZZ, ZZZZ, ZZZZZ ?
also try to execute ls -A - you will get multiple columns. I think need at least ls -1A to print result one per line.
So for your command try something like:
cp -r $(ls -1A|grep "^p") destination
or
cp -r $(ls -1A|grep "^p") -t destination
But all the above is just to correct syntax of your example.
It is much better to use find. Just in case try to put target directory in quotas like:
find <PATH_FROM> -type d -exec cp -r \"{}\" -t target \;
First I made a question here: Unzip a file and then display it in the console in one step
It works and helped me a lot. (please read)
Now I have a second issue. I do not have a single zipped log file but I have a lot of them in defferent folders, which I need to find first. The files have the same names. For example:
/somedir/server1/log.gz
/somedir/server2/log.gz
/somedir/server3/log.gz
and so on...
What I need is a way to:
find all the files like: find /somedir/server* -type f -name log.gz
unzip the files like: gunzip -c log.gz
use grep on the content of the files
Important! The whole should be done in one step.
I cannot first store the extracted files in the filesystem because it is a readonly filesystem. I need somehow to connect, with pipes, the output from one command to the input of the next.
Before, the log files were in text format (.txt), therefore I had not to unzip them first. In this case it was easy:
ex.
find /somedir/server* -type f -name log.txt | xargs grep "term"
Now I have to deal with zipped files. That means, after I find the files, I need first somehow do unzip them and then send the contents to grep.
With one file I do:
gunzip -p /somedir/server1/log.gz | grep term
But for multiple files I don't know how to do it. For example how to pass the output of find to gunzip and the to grep?!
Also if there is another way / "best practise" how to do that, it is welcome :)
find lets you invoke a command on the files it finds:
find /somedir/server* -type f -name log.gz -exec gunzip -c '{}' + | grep ...
From the man page:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of
invocations of the command will be much less than the number
of matched files. The command line is built in much the same
way that xargs builds its command lines. Only one instance of
{} is allowed within the command, and (when find is being
invoked from a shell) it should be quoted (for example, '{}')
to protect it from interpretation by shells. The command is
executed in the starting directory. If any invocation with
the + form returns a non-zero value as exit status, then
find returns a non-zero exit status. If find encounters an
error, this can sometimes cause an immediate exit, so some
pending commands may not be run at all. This variant of -exec
always returns true.
I have an issue on a server whereby on occassion automated backups from the server to a remote host fails.
Currently this leaves me with no recent backups and with a pile of .tar.gz files taking up a large amount of space on the server.
My current process for correcting this when it happens is to manually Putty in and command line FTP these files across individually. This is time consuming and tedious.
I want to write a .sh script I can upload to the folder and tell the server to put across each .tar.gz file in the folder. I can't transfer the folder as a whole but simply each file in it, as some files are already transported correctly, etc.
I found this question which
shows a script that worked for this question asker but I need to adjust parts of this script and I do not know (am not confident enough) with .sh instructions to do this, and also am wary of screwing up anything server side.
#!/bin/sh
USERNAME="user"
PASSWORD="pass"
SERVER="123.456.78.90"
DATE="`date +%Y-%m-%d`"
BACKUPDIR="/${DATE}/accounts/"
find . -type f -name "*.tar.gz" -exec basename {} .tar.gz \; |
while read filename ; do
/bin/ftp -inv $SERVER >> /tmp/ftp_backup.log <<EOF
user $USERNAME $PASSWORD
cd $BACKUPDIR
binary
put $filename
EOF
echo "$date copied $filename" >> /tmp/ftp_backup.log
done
My intention is to make this script that I can upload it in to the server folder in question and then run the script (after chmoding it) in the folder to move the .tar.gz files - one at a time - FTP'd across to the backup directory (/<date>/accounts/) and finishing once they're all moved.
(Then I would delete the server-side .tar.gz files and the .sh script above.)
There are ~60 files up to 15Gb in size. Filenames do not contain spaces.
Filepath structures:
Serve side:
/backupsfolder/2018-07-11/filename1.tar.gz
/backupsfolder/2018-07-11/filename2.tar.gz
/backupsfolder/2018-07-11/backupscript.sh //my script above
/backupsfolder/2018-07-11/master.meta //other files
FTP side:
/2018-07-11/accounts/filename1.tar.gz
What do I need to adjust on the above script to do this?
After some work I found a few issues to be careful of and fix:
1) In order to run, .sh files need to be "enabled" with chmod on the server.
chmod +x ./<filename>
2) Unix line endings; while using Notepad++ it claimed to have correct line endings saved, but the error was coming up on the server of:
/bin/sh^M: bad interpreter: No such file or directory
this was solved with:
sed -i 's/\r//' <filepath>/<filename>
from this answer.
3) The names of the files being pushed to FTP was wrong - it was not including the .tar.gz - I hadn't realised the -exec feature was cutting off the .tar.gz
This was fixed with
-exec basename {} .tar.gz
becomes
-exec basename {}
4) Log file output was not being set on new lines; instead being all on the same line.
This was fixed with reading this anwser and using -e on the echo statements and using the \n syntax.
echo -e "$date copied $filename\n"
Final fully working bash script for my needs:
1) Save the script to the server
2) Run sed -i 's/\r//' /<filepath>/<filename>
3) Run chmod +x ./<filename>
4) Run the file in bash.
5) View results in the tmp directory specified.
The script
This script takes .tar.gz files from the current directory and saves them to the remote FTP, cycling through each file in turn.
#!/bin/sh
USERNAME="user"
PASSWORD="pass"
SERVER="123.456.78.90"
DATE="`date +%Y-%m-%d`"
BACKUPDIR="/${DATE}/accounts/"
find . -type f -name "*.tar.gz" -exec basename {} \; |
while read filename ; do
ftp -inv $SERVER >> /tmp/My_ftp_backup.log <<EOF
user $USERNAME $PASSWORD
cd $BACKUPDIR
binary
put $filename
EOF
echo -e "$date copied $filename\n" >> /tmp/My_ftp_backup.log
done
I am flattening a directory of nested folders/picture files down to a single folder. I want to move all of the nested files up to the root level.
There are 3,381 files (no directories included in the count). I calculate this number using these two commands and subtracting the directory count (the second command):
find ./ | wc -l
find ./ -type d | wc -l
To flatten, I use this command:
find ./ -mindepth 2 -exec mv -i -v '{}' . \;
Problem is that when I get a count after running the flatten command, my count is off by 46. After going through the list of files before and after (I have a backup), I found that the mv command is overwriting files sometimes even though I'm using -i.
Here's details from the log for one of these files being overwritten...
.//Vacation/CIMG1075.JPG -> ./CIMG1075.JPG
..more log
..more log
..more log
.//dog pics/CIMG1075.JPG -> ./CIMG1075.JPG
So I can see that it is overwriting. I thought -i was supposed to stop this. I also tried a -n and got the same number. Note, I do have about 150 duplicate filenames. Was going to manually rename after I flattened everything I could.
Is it a timing issue?
Is there a way to resolve?
NOTE: it is prompting me that some of the files are overwrites. On those prompts I just press Enter so as not to overwrite. In the case above, there is no prompt. It just overwrites.
Apparently the manual entry clearly states:
The -n and -v options are non-standard and their use in scripts is not recommended.
In other words, you should mimic the -n option yourself. To do that, just check if the file exists and act accordingly. In a shell script where the file is supplied as the first argument, this could be done as follows:
[ -f "${1##*/}" ]
The file, as first argument, contains directories which can be stripped using ##*/. Now simply execute the mv using ||, since we want to execute when the file doesn't exist.
[ -f "${1##*/}" ] || mv "$1" .
Using this, you can edit your find command as follows:
find ./ -mindepth 2 -exec bash -c '[ -f "${0##*/}" ] || mv "$0" .' '{}' \;
Note that we now use $0 because of the bash -c usage. It's first argument, $0, can't be the script name because we have no script. This means the argument order is shifted with respect to a usual shell script.
Why not check if file exists, prior move? Then you can leave the file where it is or you can rename it or do something else...
Test -f or, [] should do the trick?
I am on tablet and can not easyly include the source.
Apreciate any help and excuse me if my terminology is incorrect.
this is a script(*.sh file) that:
1-goes to a specific dir A
2-copies files from another dir B to dir A
3-#comented out# it also unzips the files in dir A and its subdirectories
4-#comented out# it also removes rows 1-6 and the last row of all *.csv files in dir A
#!/bin/bash
# Configure bash so the script will exit if a command fails.
set -e
#cd to the dir you want to copy to:
cd /cygdrive/c/path/I/want/to/copy/to
#echo Hello
#cp the files i want
#include the subdirectories
cp -r /cygdrive/c/path/I/want/to/copy/from/* .
# This will unzip all .zip files in all subdirectories under this one.
# -o is required to overwrite everything that is in there
#find -iname '*.zip' -execdir unzip -o {} \;
#find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
Now I can get this script to work in cygwin by going to the dir where the file is stored and giving the following commands:
./filename.sh
or
/cygdrive/c/path/where/the/file/is/filename.sh
or
bash filename.sh
I can also do this in CMD/Windows DOS by doing the following:
C:\cygwin\bin\bash.exe -l
to get into a bash terminal and then give the following command:
/cygdrive/c/path/where/the/file/is/filename.sh
In task scheduler(in Windows) I have tried to schedule the following:
C:\cygwin\bin\bash.exe -l /cygdrive/c/path/where/the/file/is/filename.sh
but this does not work, even though the seperate commands work in CMD/Windows DOS as I have said above
Now what I want to do is be able to schedule this script(filename.sh) like I would a .vbs or .bat file in windows using task scheduler? Can anyone advise on this?
Note I have tried to write a Windows batch file(.bat) to do this(see below), but I could not get my unzip and sed commands to work,see here. So I have tried to write the Bash shell script above.
chdir C:\pointA
C:\cygwin\bin\cp.exe /cygdrive/v/pointB/* .
::find -iname *.zip -execdir unzip {} \;
::find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
A solution is to associate .sh files with a batch file that runs bash. That way whenever you tell windows to execute an sh file it'll use the correct launcher - whether that's via a double click or a scheduled task. Here's mine:
#echo off
d:
chdir d:\cygwin\bin
bash --login %*
Associating a file type means that when you try to execute a file of that type, windows will use the path to that file as an argument passed to the program you've specified. For example, I have LibreOffice4 associated with .ods files. So if I doubleclick a .ods file, or just enter the path to a .ods file at the command prompt, windows will run open office calc, with the first parameter being the ods file. So if I have Untitled.ods on my desktop. I doubleclick it. That's effectively the same as opening up command prompt, typing
D:\Program Files (x86)\LibreOffice 4\program\scalc.exe" "C:\Users\Adam\Desktop\Untitled.ods".
and hitting enter. Indeed, if I do it, the expected happens: open office calc starts up and loads the file.
You can see how this works if you change the association to echo.exe (which I found in D:\cygwin\bin).
If I change the association to echo, open up the command prompt and type
"C:\Users\Adam\Desktop\Untitled.ods"
I'll just see echo.exe echo the filename back to me.
So what I'm suggesting you do is this:
create a batch file to run bash scripts using cygwin's bash (or use mine).
change the association for .sh files to that batch file
execute those .sh files directly, as though they were .exe or .bat files.
why not creating a batchfile (.bat) that loads your cygwin bashscript and schedule this batchfile? like this you dont have to deal with the way M$ handles paramters