Im attempting the simplest use of robocopy with powershell:
robocopy C:\Users\tkeen\Documents\test\ C:\Users\tkeen\Documents\test2\
It doesn't seem to do anything. This is the response I get back:
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Friday, April 22, 2022 5:51:41 PM
Source : C:\Users\tkeen\Documents\test\
Dest : C:\Users\tkeen\Documents\test2\
Files : *.*
Options : *.* /DCOPY:DA /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
0 C:\Users\tkeen\Documents\test\
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 0 1 0 0 0
Files : 0 0 0 0 0 0
Bytes : 0 0 0 0 0 0
Times : 0:00:00 0:00:00 0:00:00 0:00:00
Ended : Friday, April 22, 2022 5:51:41 PM
I tried with other options like \MIR but then I just get a "ERROR : Invalid Parameter #3 : \mir" message. Not sure what Im doing wrong.
Try Below
robocopy "Source" "destination" /MIR /E /XO /xx /tee /R:2 /W:1 /SEC /LOG:"o/p path"
MIR - mirrors directory - deletes files which is present in destination but not in source
E - copies subdirectories even folder is empty
X0 - excludes older files - we can say for incremental
xx - excludes extra files - will not delete extra files in destination (XX overtakes MIR)
tee - gives output in command prompt also
R - Number of Retries on failure
W - Wait time between Retries
SEC - copy with security
LOG - output path
I am trying to rename a single file from an zip archive using zip gem in Ruby.
zip_path = "C:/Downloads/"
zipfile_name = "C:/Downloads/02004 - 1850 W PINHOOK RD Lafayette LA.zip"
old_name = 'Reports.xls'
new_filename = 'Reports.csv'
Zip::ZipFile.open('C:/Downloads/02004 - 1850 W PINHOOK RD Lafayette LA.zip').each do |zipfile|
files = zipfile.select(&:file?)
file = files.find{|f| f.name == old_name}
zipfile.rename(file.name, new_filename) if file
end
end`
This is the code that I use and I get the the following error:
**NoMethodError: private method `select' called for #<Zip::Entry:0x00000000062ff098>
./features/step_definitions/rename_csv_definition.rb:21:in `block (2 levels) in <top (required)>'
./features/step_definitions/rename_csv_definition.rb:20:in `/^I rename the excel file to csv$/'**
Any suggestions?
Working example:
$ touch 1.csv 2.csv 3.csv
$ zip -r files.zip 1.csv 2.csv 3.csv
$ touch zip_rename.rb
# zip_rename.rb
require 'zip/zip'
zip_path = './'
zipfile_name = 'files.zip'
old_name = '1.csv'
new_filename = '777.csv'
Zip::ZipFile.open(zip_path + zipfile_name) do |zipfile|
if zipfile.find_entry(old_name)
zipfile.rename(old_name, new_filename)
end
end
Check archive content:
$ unzip -l files.zip
Archive: files.zip
Length Date Time Name
--------- ---------- ----- ----
0 03-18-2021 18:37 3.csv
0 03-18-2021 18:37 2.csv
0 03-18-2021 19:43 777.csv
--------- -------
0 3 files
Note, that rename method is avaliable for Zip::ZipFile object(zipfile) and take Zip::ZipEntry instance file name as argument
Here is the source
rubyzip Zip::File#rename
I have multiple text files residing in a single folder. These files are created on daily basis with time stamp on it like ABC_set_05082020.txt, DEF_set_050820202.txt. I want to create a single folder like destination_05082020.zip in same directory which contains all these files.
The Script i created is like
FILE_DIR1=home/users/documents/Files
FILE_NAME=$1
FILE_DATE_TIME1="${FILE_NAME}_set_`date +%Y%m%d`"
FULL_PATH1="${FILE_DIR1}${FILE_DATE_TIME1}.dat"
if [ $# -lt 1 ]
then
echo "*************************************************************************"
echo "* ERROR script requires <workbookname> "
echo "*************************************************************************"
exit $EXIT_SYS_ERROR
fi
if [ "$FILE_NAME" = "ABC" ]
then
zip -r $FILE_DIR1/destination.zip $FULL_PATH1
fi
# done
printf "$SCRIPT_NAME `date`: Info. Complete Successfully, exitings |$RC|.\n"
exit $EXIT_SUCCESS
here i am able to create destination.zip in the same directory and destination.zip also contains ABC_set_05082020.txt but when i am unzipping destination.zip. It contains complete folder path like home/users/documents/files/ABC_set_05082020.txt. I don't want complete path in it when i unzip it. It should only contain my Text file. please help.
Try:
zip -rj $FILE_DIR1/destination.zip $FULL_PATH1
From man page
-j
--junk-paths
Store just the name of a saved file (junk the path), and do not store directory names. By default, zip will store the full
path (relative to the current directory).
Demo:
$zip -jr test.zip /home/renegade/test/file*
adding: file1.txt (stored 0%)
adding: file2.txt (stored 0%)
adding: file3.txt (stored 0%)
adding: file4.txt (stored 0%)
$zip -r test_2.zip /home/renegade/test/file*
adding: home/renegade/test/file1.txt (stored 0%)
adding: home/renegade/test/file2.txt (stored 0%)
adding: home/renegade/test/file3.txt (stored 0%)
adding: home/renegade/test/file4.txt (stored 0%)
$unzip -l test.zip
Archive: test.zip
Length Date Time Name
--------- ---------- ----- ----
0 2020-05-09 05:50 file1.txt
0 2020-05-09 05:50 file2.txt
0 2020-05-09 05:50 file3.txt
0 2020-05-09 05:50 file4.txt
--------- -------
0 4 files
$unzip -l test_2.zip
Archive: test_2.zip
Length Date Time Name
--------- ---------- ----- ----
0 2020-05-09 05:50 home/renegade/test/file1.txt
0 2020-05-09 05:50 home/renegade/test/file2.txt
0 2020-05-09 05:50 home/renegade/test/file3.txt
0 2020-05-09 05:50 home/renegade/test/file4.txt
--------- -------
0 4 files
$
Edit : -D does not work with -r
I have been a lurker at stackoverflow.com for many years (great site and users here), but never had the need to ask a question. Now the time has come :-) Let me begin:
OS: x64 Windows 8.0 to Windows 10 (15063.14) (the issue exists since years, but I have never pursued it fully yet, so we can exclude that it is specific to a specific Windows version)
FS: NTFS
Issue: 2 files with the same (long) name in the same directory and I cannot figure out how this is even possible. This happens to me since years whenever I manually upgrade my Email client. The main .EXE file of it (MailClient.exe) is never asking for replacement if copying the new one over to the same directory. Instead they are both placed there, with the exact same long name.
The issue has nothing to do with a specific directory, I can copy around both .EXE files to freshly created directories on the NTFS drive without issues (also getting no "overwrite" question there).
Let me show you:
C:\temp\2>dir
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
13.04.2017 02:29 <DIR> .
13.04.2017 02:29 <DIR> ..
21.10.2016 17:10 24.742.760 MailClient.exe
27.12.2016 03:26 24.911.872 MailCliеnt.exe
2 File(s) 49.654.632 bytes
2 Dir(s) 78.503.038.976 bytes free
However, if doing a dir /x, this comes up:
C:\temp\2>dir /x
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
13.04.2017 02:29 <DIR> .
13.04.2017 02:29 <DIR> ..
21.10.2016 17:10 24.742.760 MAILCL~2.EXE MailClient.exe
27.12.2016 03:26 24.911.872 MAILCL~1.EXE MailCliеnt.exe
2 File(s) 49.654.632 bytes
2 Dir(s) 78.503.038.976 bytes free
So they obviously have a different 8.3 name, OK, but the exact same long name. Here is another screenshot of the situation. Both files show the same location within the Windows "properties" dialog (right click) too. Unfortunately I am not allowed to post images just yet (it seems) - just tried. So you will have to take my word.
I cannot figure out how this is possible and this is bugging me ;) As soon as I rename both files for example to 1.exe, Windows starts telling me that there is already a file with that name in the same directory. So it obviously has something to do with the filename, but they are both exactly identical, no extra spaces, nothing, as you can see from the DIR command.
I´ve also tried to rename them and re-wrote the exact wording "MailCient.exe" manually for both, to make sure the characters are EXCACTLY the same, Windows still won´t complain, they both go there once again under the same name. However, renaming them to "Mail.exe" and "Mail.exe" will NOT work, then Windows is saying that another file with that name already exists. However, naming them both back to "MailClient.exe" is just absolutely fine, no complains by Windows with that.
Another fun fact about this, if I dir for mailclient.exe directly, this happens:
C:\temp\2>dir mailclient.exe
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
21.10.2016 17:10 24.742.760 MailClient.exe
1 File(s) 24.742.760 bytes
0 Dir(s) 78.501.998.592 bytes free
However, if looking for *.exe, this happens:
C:\temp\2>dir *.exe
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
21.10.2016 17:10 24.742.760 MailClient.exe
27.12.2016 03:26 24.911.872 MailCliеnt.exe
2 File(s) 49.654.632 bytes
0 Dir(s) 78.501.990.400 bytes free
This yields also interesting results:
C:\temp\2>ren mailclient.exe *.bak
C:\temp\2>dir
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
13.04.2017 02:50 <DIR> .
13.04.2017 02:50 <DIR> ..
21.10.2016 17:10 24.742.760 MailClient.bak
27.12.2016 03:26 24.911.872 MailCliеnt.exe
2 File(s) 49.654.632 bytes
2 Dir(s) 78.501.990.400 bytes free
And back:
C:\temp\2>ren mailclient.bak MailClient.exe
C:\temp\2>dir
Volume in drive C is SSD 840 Pro
Volume Serial Number is 0C6D-D489
Directory of C:\temp\2
13.04.2017 02:51 <DIR> .
13.04.2017 02:51 <DIR> ..
21.10.2016 17:10 24.742.760 MailClient.exe
27.12.2016 03:26 24.911.872 MailCliеnt.exe
2 File(s) 49.654.632 bytes
2 Dir(s) 78.501.982.208 bytes free
I´ve also checked permissions on the files and took ownership, it changes nothing. Additionally I´ve cleared the NTFS Journal and even the transaction log + run chkdsk, which reveals no errors either.
Any ideas on this mysterious situation? What am I missing?
Thanks so much:)
UPDATE #1:
I´ve just tried this: going to Windows explorer and renaming both files after each other by truncating their names. So I first renamed the first "MailClient.exe" to "MailClien.exe", then the seconds "MailClient.exe" to "MailClien.exe". Again, no message by Windows that they have the same name, it just renamed both fine. I then continued to "MailClie.exe". Worked.
However, as soon as I tried to renamed both to "MailCli.exe", Windows complained and told me that there is already another file with that name. Trying to rename both back from there to "MailClient.exe" also does not work, just for one of them, because then Windows says (and right so too) that a file with that name already exists. So it seems to come down to the "e" possibly having another ANSI-character in both filenames? I, however, wouldn´t know of another one for "e", or am I missing something?
Harry Johnston is right: one of the filenames contains a Unicode character that just looks the same as an ANSI character.
Read Naming Files, Paths, and Namespaces:
On newer file systems, such as NTFS, exFAT, UDFS, and FAT32, Windows
stores the long file names on disk in Unicode, which means that the
original long file name is always preserved. This is true even if a
long file name contains extended characters, regardless of the code
page that is active during a disk read or write operation.
Use the following PowerShell script 43381802b.ps1 to detect and show non-ANSI file names (see different calls below):
param( [string[]]$Path = '.',
[switch]$Cpp, ### list any non-ANSI character in file names like a C++ literal
### i.e. a prefix \u followed by a four digit Unicode code point
[switch]$All ### list all files including pure ANSI-encoded file names
)
Set-StrictMode -Version latest
$strArr = Get-ChildItem -path $Path
$arrDiff = #()
for ($i=0; $i -lt $strArr.Count; $i++) {
$strDiff = 'ANSI'
$strName = ''
$auxName = $strArr[$i].Name
for ( $k=0; $k -lt $auxName.Length; $k++ ) {
if ( [int][char]$auxName[$k] -gt 255 ) {
$strDiff = 'UCS2'
$strName += '\u{0:X4}' -f [int][char]$auxName[$k]
} else {
$strName += $auxName[$k]
}
}
if ( $All.IsPresent -or $strDiff -eq 'UCS2' ) {
$strArr[$i] | Add-Member NoteProperty Code $strDiff
$strArr[$i] | Add-Member NoteProperty CppName $strName
$arrDiff += $strArr[$i]
}
}
if ( $Cpp.IsPresent ) {
$arrDiff | Select-Object -Property Code, Mode, LastWriteTime, Length, CppName | ft
} else {
$arrDiff | Select-Object -Property Code, Mode, LastWriteTime, Length, Name | ft
}
Output:
PS D:\PShell> .\SO\43381802b.ps1 'C:\testC\43381802'
Code Mode LastWriteTime Length Name
---- ---- ------------- ------ ----
UCS2 -a---- 02/05/2017 11:47:53 317 MailCliеnt.txt
UCS2 -a---- 02/05/2017 11:49:04 317 МailClient.txt
UCS2 -a---- 02/05/2017 11:50:16 399 МailCliеnt.txt
PS D:\PShell> .\SO\43381802b.ps1 'C:\testC\43381802' -Cpp
Code Mode LastWriteTime Length CppName
---- ---- ------------- ------ -------
UCS2 -a---- 02/05/2017 11:47:53 317 MailCli\u0435nt.txt
UCS2 -a---- 02/05/2017 11:49:04 317 \u041CailClient.txt
UCS2 -a---- 02/05/2017 11:50:16 399 \u041CailCli\u0435nt.txt
PS D:\PShell> .\SO\43381802b.ps1 'C:\testC\43381802' -Cpp -All
Code Mode LastWriteTime Length CppName
---- ---- ------------- ------ -------
ANSI -a---- 02/05/2017 11:44:05 235 MailClient.txt
UCS2 -a---- 02/05/2017 11:47:53 317 MailCli\u0435nt.txt
UCS2 -a---- 02/05/2017 11:49:04 317 \u041CailClient.txt
UCS2 -a---- 02/05/2017 11:50:16 399 \u041CailCli\u0435nt.txt
Use the following 43381802a.ps1 script to get more info about non-ANSI characters (see the first call bellow) and their position in file names (see the latter call bellow with -Detail switch):
param( [string[]] $strArr = #('ΗGreek', 'НCyril', 'HLatin'),
[switch]$Detail )
Set-StrictMode -Version latest
$auxArr = #()
if ( ( Get-Command -Name Get-CharInfo -ErrorAction SilentlyContinue ) -and
( -not $Detail.IsPresent ) ) {
$auxArr = $strArr | Get-CharInfo |
Where-Object { [int]$_.Codepoint.Replace('U+', '0x') -ge 128 }
} else {
foreach ($strStr in $strArr) {
for ($i = 0; $i -lt $strStr.Length; $i++ ) {
if ( [int][char]$strStr[$i] -ge 128 ) {
$auxArr += [PSCustomObject] #{
Char = $strStr[$i]
CodePoint = 'U+{0:x4}' -f [int][char]$strStr[$i]
Category = $i + 1 ### 1-based index
Description = $strStr ### string itself
}
}
}
}
}
$auxArr
Output:
PS D:\PShell> .\SO\43381802a.ps1 ( Get-childitem -path 'C:\testC\43381802' ).Name
Char CodePoint Category Description
---- --------- -------- -----------
е U+0435 LowercaseLetter Cyrillic Small Letter Ie
М U+041C UppercaseLetter Cyrillic Capital Letter Em
М U+041C UppercaseLetter Cyrillic Capital Letter Em
е U+0435 LowercaseLetter Cyrillic Small Letter Ie
PS D:\PShell> .\SO\43381802a.ps1 ( Get-childitem -path 'C:\testC\43381802' ).Name -detail
Char CodePoint Category Description
---- --------- -------- -----------
е U+0435 8 MailCliеnt.txt
М U+041c 1 МailClient.txt
М U+041c 1 МailCliеnt.txt
е U+0435 8 МailCliеnt.txt
Tested on files:
==> dir /-C /X /A-D C:\testC\43381802\
Volume in drive C has no label.
Volume Serial Number is …
Directory of C:\testC\43381802
02/05/2017 11:44 235 MAILCL~1.TXT MailClient.txt
02/05/2017 11:47 317 MAILCL~2.TXT MailCliеnt.txt
02/05/2017 11:49 317 AILCLI~1.TXT МailClient.txt
02/05/2017 11:50 399 AILCLI~2.TXT МailCliеnt.txt
4 File(s) 1268 bytes
0 Dir(s) 69914857472 bytes free
==>
The problem can be reproduced using a simple test.
The "pig" script is as follows:
SET pig.noSplitCombination true;
dataIn = LOAD 'input/Test';
DEFINE macro `TestScript` input('DummyInput.txt') output('A.csv', 'B.csv', 'C.csv', 'D.csv', 'E.csv') ship('TestScript');
dataOut = STREAM dataIn through macro;
STORE dataOut INTO 'output/Test';
The actual script is a complex R program but here is a simple "TestScript" that reproduces the problem and doesn't require R:
# Ignore the input coming from the 'DummyInput.txt' file
# For now just create some output data files
echo "File A" > A.csv
echo "File B" > B.csv
echo "File C" > C.csv
echo "File D" > D.csv
echo "File E" > E.csv
The input 'DummyInput.txt' is some dummy data for now.
Record1
Record2
Record3
For the test, I've load the the dummy data in HDFS using the following script. This will result in 200 input files.
for i in {0..199}
do
hadoop fs -put DummyInput.txt input/Test/Input$i.txt
done
When I run the pig job, it runs without errors. 200 mappers run as expected. However, I expect to see 200 files in the various HDFS directories. Instead I find that a number of the output files are missing:
1 200 1400 output/Test/B.csv
1 200 1400 output/Test/C.csv
1 189 1295 output/Test/D.csv
1 159 1078 output/Test/E.csv
The root "output/Test" has 200 files, which is correct. Folders "B.csv" and "C.csv" have 200 files as well. However, folders "D.csv" and "E.csv" have missing files.
We have looked at the logs but can't anything which points to why the local output files are not being copied from the data nodes to HDFS.