I am trying to create something in Perl that is basically like the Unix tee command. I'm trying to read each line of STDIN, run a substitution on it, and print it. (And eventually, also print it to a file.) This works if I'm using console input, but if I try to pipe input to the command it doesn't do anything. Here's a simple example:
print "about to loop\n";
while(<STDIN>)
{
s/2010/2009/;
print;
}
print "done!\n";
I try to pipe the dir command to it like this:
C:\perltest>dir | mytee.pl
about to loop
done!
Why is it not seeing the piped input? (I'm using Perl 5.10.0 on WinXP, if that is relevant.)
This is actually a bug in how Windows handles IO redirection. I am looking for the reference right now, but it is that bug that requires you to specify
dir | perl filter.pl
rather than being able to use
dir | filter
See Microsoft KB article STDIN/STDOUT Redirection May Not Work If Started from a File Association:
Start Registry Editor.
Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer
On the Edit menu, click Add Value, and then add the following registry value:
Value name: InheritConsoleHandles
Data type: REG_DWORD
Radix: Decimal
Value data: 1
Quit Registry Editor.
C:\Temp> cat filter.pl
#!/usr/bin/perl
while ( <> ) {
print "piped: $_";
}
C:\Temp> dir | filter
piped: Volume in drive C is MAIN
piped: Volume Serial Number is XXXX-XXXX
piped:
piped: Directory of C:\Temp>
piped:
piped: 2010/03/19 03:48 PM .
piped: 2010/03/19 03:48 PM ..
piped: 2010/03/19 03:33 PM 32 m.pm
piped: 2010/03/19 03:48 PM 62 filter.pl
Try:
C:\perltest>dir | perl mytee.pl
Could it be Microsoft KB #321788?
Scripts that contain standard input
(STDIN) and standard output (STDOUT)
may not work correctly if you start
the program from a command prompt and
you use a file association to start
the script.
There's nothing wrong with trying to learn by doing, but a quick search of CPAN shows a number of possible solutions for the tee in Perl problem.
For example: PerlIO::Tee.
Well IMHO, perl is poor substitute for sed ;)
dir | sed s/2009/2010/
Related
I am trying to redirect outputs of systat -ifstat and systat -vmstat to a file and When I open that file, lot of special chars are added to a file like below
(B)0[?1049h[1;39r[m[4l[H[2J[1;21H/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
[68DLoad Average
[11DInterface[4;27HTraffic[4;49HPeak[4;69HTotal[2;21H[5;13H1/1 in[6C0.000 KB/s[5;46H0.000 KB/s[5;66H260.087 MB[6;18Hout 0.000 KB/s[6;46H0.000 KB/s[6;66H205.319
The command I am using to redirect to a file is below:
systat -ifstat 1 > text.txt
Can someone guide me to get rid of these special chars.Help is appreciated.
Keep in mind that systat(from the man page) display system statistics in a screen-oriented fashion using the curser screen display library, therefore in the try to get something like a screenshot, this partially works:
systat -ifstat | tee /tmp/output
to see the output like when using the command you will need to do cat /tmp/output, otherwise you will see all the shell escape characters.
You could also try script:
$ script
Script started, output file is typescript
$ systat -ifstat
next, you exit systat ctrl+c
$ exit
exit
Script done, output file is typescript
This will create a file named typescript but the output is not as clear as when using tee. (still, haven't found a way of how to render properly typescript within csh)
I monitor a file for changes in a separate thread using kqueues/ kevent(2).
(I monitor a Python file for reparsing)
I subscribe as following:
EV_SET(&file_change, pyFileP, EVFILT_VNODE,
EV_ADD | EV_CLEAR,
NOTE_DELETE | NOTE_WRITE | NOTE_EXTEND |
NOTE_ATTRIB | NOTE_LINK | NOTE_RENAME | NOTE_REVOKE,
0, 0);
When I write to the file "/tmp/somefile.py" using Vim, I get two separate kevents:
The flags of these events (event.fflags) are:
NOTE_RENAME
and
NOTE_DELETE | NOTE_LINK
I never get a "NOTE_WRITE" event!
This seems to have something to do with the way Vim writes these files, since if I do
echo "sometext" >> /tmp/somefile.py
I do get the:
NOTE_WRITE|NOTE_EXTEND
event.
Odd, eh? I haven't checked the Vim source code but it must do something strange, or does it simply use user level functions that are implemented that way?
I wasn't really expecting this. Is this a known problem, I just have to check for all events possible, or is there a known interface that really checks if a file has been written?
What is actually happening is that Vim won't write over the same file, first
it probably renames it to something else and then creates another file (link).
You can confirm that by doing something like:
$ vim file -c wq
This will open a file and write it. Now check the inode:
$ ls -i
30621217 file
Write the file with Vim again and re-check the inode:
$ vim file -c wq
$ ls -i
30621226 file
It's just different. That means the second file is actually another file
(linked to another inode) with the same name, and the old one was unlinked.
Many editors do that. I can't confirm why exactly Vim takes this approach.
Maybe for safety: if you first rename the file and something goes wrong
while writing the new file, you still have the old one. If you start writing
over a file and a problem occurs (even with memory) you'll probably loose part
of it. Maybe.
Say I have too programs a and b that I can run with ./a and ./b.
Is it possible to diff their outputs without first writing to temporary files?
Use <(command) to pass one command's output to another program as if it were a file name. Bash pipes the program's output to a pipe and passes a file name like /dev/fd/63 to the outer command.
diff <(./a) <(./b)
Similarly you can use >(command) if you want to pipe something into a command.
This is called "Process Substitution" in Bash's man page.
Adding to both the answers, if you want to see a side by side comparison, use vimdiff:
vimdiff <(./a) <(./b)
Something like this:
One option would be to use named pipes (FIFOs):
mkfifo a_fifo b_fifo
./a > a_fifo &
./b > b_fifo &
diff a_fifo b_fifo
... but John Kugelman's solution is much cleaner.
For anyone curious, this is how you perform process substitution in using the Fish shell:
Bash:
diff <(./a) <(./b)
Fish:
diff (./a | psub) (./b | psub)
Unfortunately the implementation in fish is currently deficient; fish will either hang or use a temporary file on disk. You also cannot use psub for output from your command.
Adding a little more to the already good answers (helped me!):
The command docker outputs its help to STD_ERR (i.e. file descriptor 2)
I wanted to see if docker attach and docker attach --help gave the same output
$ docker attach
$ docker attach --help
Having just typed those two commands, I did the following:
$ diff <(!-2 2>&1) <(!! 2>&1)
!! is the same as !-1 which means run the command 1 before this one - the last command
!-2 means run the command two before this one
2>&1 means send file_descriptor 2 output (STD_ERR) to the same place as file_descriptor 1 output (STD_OUT)
Hope this has been of some use.
For zsh, using =(command) automatically creates a temporary file and replaces =(command) with the path of the file itself. With normal Process Substitution, $(command) is replaced with the output of the command.
This zsh feature is very useful and can be used like so to compare the output of two commands using a diff tool, for example Beyond Compare:
bcomp =(ulimit -Sa | sort) =(ulimit -Ha | sort)
For Beyond Compare, note that you must use bcomp for the above (instead of bcompare) since bcomp launches the comparison and waits for it to complete. If you use bcompare, that launches comparison and immediately exits due to which the temporary files created to store the output of the commands disappear.
Read more here: http://zsh.sourceforge.net/Intro/intro_7.html
Also notice this:
Note that the shell creates a temporary file, and deletes it when the command is finished.
and the following which is the difference between $(...) and =(...) :
If you read zsh's man page, you may notice that <(...) is another form of process substitution which is similar to =(...). There is an important difference between the two. In the <(...) case, the shell creates a named pipe (FIFO) instead of a file. This is better, since it does not fill up the file system; but it does not work in all cases. In fact, if we had replaced =(...) with <(...) in the examples above, all of them would have stopped working except for fgrep -f <(...). You can not edit a pipe, or open it as a mail folder; fgrep, however, has no problem with reading a list of words from a pipe. You may wonder why diff <(foo) bar doesn't work, since foo | diff - bar works; this is because diff creates a temporary file if it notices that one of its arguments is -, and then copies its standard input to the temporary file.
How can I print the output to terminal and file at the same time?
$ perl foo.pl > foout.txt
does not allow me to see live process.
Is there any way I can see the output process real time and getting at the end the output of the screen on a file?
perl foo.pl | tee foout.txt
The utility tee will do that.
See IO::Tee. This module will allow you to do this selectively with fine grained control within your program (there is also a less mature module called File::Tee which worked for me once but I would not recommend that for any serious project).
See also Log4perl for really fine grained control over what gets logged where and how.
For one off usage from the command line, as others have recommended, you can, of course, utilize the command line utility tee if you have access to it.
Or you could pipe it into a perl programs to print to the screen and a log file (that is if you are not on Unix or have a tee program)
perl foo.pl | perl myPipe.pl myFile.txt
where the data is captureed to myFile.txt and
myPipe.pl is
#
open OUTFILE,">$ARGV[0]" or die "Unable to open file: $ARGV[0]\n";
while(<>)
{
print;
print OUTFILE;
}
close OUTFILE;
This reads a line of input from STDIN and prints it to the screen and then to the file. When the end is reached, it will close the file
I need to join two binary files with a *.bat script on Windows.
How can I achieve that?
Windows type command works similarly to UNIX cat.
Example 1:
type file1 file2 > file3
is equivalent of:
cat file1 file2 > file3
Example 2:
type *.vcf > all_in_one.vcf
This command will merge all the vcards into one.
You can use copy /b like this:
copy /b file1+file2 destfile
If you have control over the machine where you're doing your work, I highly recommend installing GnuWin32. Just "Download All" and let the wget program retrieve all the packages. You will then have access to cat, grep, find, gzip, tar, less, and hundreds of others.
GnuWin32 is one of the first things I install on a new Windows box.
Shameless PowerShell plug (because I think the learning curve is a pain, so teaching something at any opportunity can help)
Get-Content file1,file2
Note that type is an alias for Get-Content, so if you like it better, you can write:
type file1,file2
Just use the dos copy command with multiple source files and one destination file.
copy file1+file2 appendedfile
You might need the /B option for binary files
In Windows 10's Redstone 1 release, the Windows added a real Linux subsystem for the NTOS kernel. I think originally it was intended to support Android apps, and maybe docker type scenarios. Microsoft partnered with Canonical and added an actual native bash shell. Also, you can use the apt package manager to get many Ubuntu packages. For example, you can do apt-get gcc to install the GCC tool chain as you would on a Linux box.
If such a thing existed while I was in university, I think I could have done most of my Unix programming assignments in the native Windows bash shell.
If you simply want to append text to the end of existing file, you can use the >> pipe. ex:
echo new text >>existingFile.txt
So i was looking for a similar solution with the abillity to preserve EOL chars and found out there was no way, so i do what i do best and made my own utillity
This is a native cat executable for windows - https://mega.nz/#!6AVgwQhL!qJ1sxx-tLtpBkPIUx__iQDGKAIfmb21GHLFerhNoaWk
Usage: cat file1 file2 file3 file4 -o output.txt
-o | Specifies the next arg is the output, we must use this rather than ">>" to preserve the line endings
I call it sharp-cat as its built with C#, feel free to scan with an antivirus and source code will be made available at request
I try to rejoin tar archive which has been splitted in a Linux server.
And I found if I use type in Windows's cmd.exe, it will causes the file being joined in wrong order.(i.e. type sometimes will puts XXXX.ad at first and then XXXX.ac , XXXX.aa etc ...)
So, I found a tool named bat in GitHub https://github.com/sharkdp/bat which has a Windows build, and has better code highlight and the important thing is, it works fine on Windows to rejoin tar archive!
Windows type command has problems, for example with Unicode characters on 512 bytes boundary. Try Cygwin's cat.
If you have to use a batch script and have python installed here is a polyglot answer in batch and python:
1>2# : ^
'''
#echo off
python "%~nx0" " %~nx1" "%~nx2" "%~nx3"
exit /b
rem ^
'''
import sys
import os
sys.argv = [argv.strip() for argv in sys.argv]
if len(sys.argv) != 4:
sys.exit(1)
_, file_one, file_two, out_file = sys.argv
for file_name in [file_one, file_two]:
if not os.path.isfile(file_name):
print "Can't find: {0}".format(file_name)
sys.exit(1)
if os.path.isfile(out_file):
print "Output file exists and will be overwritten"
with open(out_file, "wb") as out:
with open(file_one, "rb") as f1:
out.write(f1.read())
with open(file_two, "rb") as f2:
out.write(f2.read())
If saved as join.bat usage would be:
join.bat file_one.bin file_two.bin out_file.bin
Thanks too this answer for the inspiration.