I have been working on some device that allowed login via telnet and I extracted some data from devices and made some reports, without any problems. recently, I had to switch to SSH while rest of the script is all the same, only login procedure has been changed from telnet to SSH. after switching to SSH, I am facing some problem with the data extracted that there are some invalid characters in some of the lines, below is an example: as can be seen, there is an invalid character after PON7 in the line:
OLT:LT6.PON7.ONT1,ALARM,Date time,
problem is that this invalid character is not even visible in the bash/csv file, but it was discovered when I copied the line in notepad++ or while posting it here.
now I have two problems:
1st: if someone knows what is causing these invalid characters while switching between telnet/ssh.
2nd: how to deal with this invalid character in BASH as it is not even visible in BASH, but this report is being used somewhere and these invalid characters are causing problems.
Edit:
Pasting the text into a text-to-hex converter produces this:
4f 4c 54 3a 4c 54 36 2e 50 4f 4e 37 11 2e 4f 4e 54 31 2c 41 4c 41 52 4d 2c 44 61 74 65 20 74 69 6d 65 2c
It looks like there's a DC1 character (hex 11) between the "7" and the ".".
Unfortunately, this edit also has the side effect of removing the character from the sample text.
Passing your text through a text to hexadecimal converter shows that the invisible character is an ASCII DC1 character (hex 11, octal 021). This character is also known as Ctrl-Q or XON. It's sometimes used in flow control.
In a bash script, you could filter it out using the tr program:
echo $badtext | tr -d '\021'
SSH doesn't inherently insert DC1 characters into text streams. If you're getting a DC1 character in the output from a device, presumably the device sent that character.
Related
I'm logging into to a remote ssh session using plink.exe to perform certain tasks using a batch script. Getting the output of these commands in a log file as well on the screen is very important for me.
I tried using usual batch way i.e. plink servername -m cmd.txt>logfile.log way but the problem with this is that it won't display it on the Windows terminal that the batch script is running on.
Then I found the -sshlog option of Plink. This does the work, i.e. I can get the output but on screen and in a log file, but this results in output as follows:
00000f90 56 4c 41 4e 2a 2a 0d 0a 20 65 6e 63 61 70 73 75 VLAN**.. encapsu
00000fa0 6c 61 74 69 6f 6e 20 64 6f 74 31 51 20 34 30 34 lation dot1Q 404
00000fb0 0d 0a 20 69 70 20 61 64 64 72 65 73 73 20 31 30 .. ip address 10
00000fc0 2e 37 31 2e 31 39 31 2e 31 34 35 20 32 35 35 2e .71.191.145 255.
My actual output starts at "VLAN**.. encapsu" in the text above The output has these "00000010 74 65 72 ... "bla bla characters which I do not want. Plus the main output (that would be displayed if i was using Plink interactively is "word-wrapped" and looks horrible which makes it very difficult to understand for a general user
Is there any way to prevent Plink output unwanted 'sshlog' characters in the log file? or Is there any other way to get the output on screen and log fail simultaneously in a Plink/PuTTY session inside a batch script?
I tried both -sshlog and -sshrawlog but same output. Also tried -sessionlog as per the documentation but it does not work!
I tried also > file.txt but it gave an empty file!
Expected results:
encapsulation dot1Q 404
ip address 10.71.191.145
There's no command-line switch in Plink to log only the readable output.
You would have to configure "Printable output" logging in Windows Registry prior to running Plink.
But there are other options.
See Displaying Windows command prompt output and redirecting it to a file.
So this should do what you need:
powershell "plink -batch servername -m cmd.txt | tee logfile.log"
(the tee is an alias to Tee-Object cmdlet)
I'm logging into to a remote ssh session using plink.exe to perform certain tasks using a batch script. Getting the output of these commands in a log file as well on the screen is very important for me.
I tried using usual batch way i.e. plink servername -m cmd.txt>logfile.log way but the problem with this is that it won't display it on the Windows terminal that the batch script is running on.
Then I found the -sshlog option of Plink. This does the work, i.e. I can get the output but on screen and in a log file, but this results in output as follows:
00000f90 56 4c 41 4e 2a 2a 0d 0a 20 65 6e 63 61 70 73 75 VLAN**.. encapsu
00000fa0 6c 61 74 69 6f 6e 20 64 6f 74 31 51 20 34 30 34 lation dot1Q 404
00000fb0 0d 0a 20 69 70 20 61 64 64 72 65 73 73 20 31 30 .. ip address 10
00000fc0 2e 37 31 2e 31 39 31 2e 31 34 35 20 32 35 35 2e .71.191.145 255.
My actual output starts at "VLAN**.. encapsu" in the text above The output has these "00000010 74 65 72 ... "bla bla characters which I do not want. Plus the main output (that would be displayed if i was using Plink interactively is "word-wrapped" and looks horrible which makes it very difficult to understand for a general user
Is there any way to prevent Plink output unwanted 'sshlog' characters in the log file? or Is there any other way to get the output on screen and log fail simultaneously in a Plink/PuTTY session inside a batch script?
I tried both -sshlog and -sshrawlog but same output. Also tried -sessionlog as per the documentation but it does not work!
I tried also > file.txt but it gave an empty file!
Expected results:
encapsulation dot1Q 404
ip address 10.71.191.145
There's no command-line switch in Plink to log only the readable output.
You would have to configure "Printable output" logging in Windows Registry prior to running Plink.
But there are other options.
See Displaying Windows command prompt output and redirecting it to a file.
So this should do what you need:
powershell "plink -batch servername -m cmd.txt | tee logfile.log"
(the tee is an alias to Tee-Object cmdlet)
Good afternoon.
I am running pcl6.exe version 9.15 on Windows 8.1.
I am running into a problem where pcl6.exe in silently converting any APOSTROPHE characters into RIGHT SINGLE QUOTATION MARK characters using the 16602 typeface in a PCL5 file.
Here is the command line I am using:
pcl6.exe -dNOPAUSE -sDEVICE=txtwrite -sOutputFile=test.txt test.prn
test.prn input (hex)
1B 28 30 55 1B 28 73 31 70 31 30 76 31 36 36 30 32 54 1B 26 61 30 76 30 48 3E 27 3C
test.prn input (text ['.' is the escape character])
.(0U.(s1p10v16602T.&a0v0H>'<
test.txt output (hex)
20 20 3E E2 80 99 3C 0D 0A
test.txt output (text)
>’<..
expected test.txt output (hex)
20 20 3E 27 3C 0D 0A
expected test.txt output (text)
>'<..
Is there a flag or an option somewhere that can disable this conversion?
Thank you for your time.
txtwrite does the best it can with the input, PCL tends not to have much information in the PCL file to allow us to determine what the glyph should be (PostScript is better, and PDF often still better).
If you think there is a real problem I would suggest your best bet is to open a bug report. Apart from anything else, I would need to see the PCL file to determine what's going on. Most lkely the character code you are using corresponds to an apostrophe, which in the specific font is a right quote. There is no way for the text extraction device to know what shape the font will draw in response to a character code. At least, not in PCL
The problem was caused by the symbolset.
The sample was using the PCL ISO 6: ASCII symbolset (code 0U )
http://www.pcl.to/symbolset/pcl_0u.pdf
According to the 0U symbolset reference, APOSTROPHE (0x27) is replaced with RIGHT SINGLE QUOTATION MARK (0x2019). Pcl6.exe then converts these UTF-16 bytes into their UTF-8 equivalent: 0xE28099
I am writing a script that takes a UTF-16 encoded text file as input and outputs a UTF-16 encoded text file.
use open "encoding(UTF-16)";
open INPUT, "< input.txt"
or die "cannot open > input.txt: $!\n";
open(OUTPUT,"> output.txt");
while(<INPUT>) {
print OUTPUT "$_\n"
}
Let's just say that my program writes everything from input.txt into output.txt.
This WORKS perfectly fine in my cygwin environment, which is using "This is perl 5, version 14, subversion 2 (v5.14.2) built for cygwin-thread-multi-64int"
But in my Windows environment, which is using "This is perl 5, version 12, subversion 3 (v5.12.3) built for MSWin32-x64-multi-thread",
Every line in output.txt is pre-pended with crazy symbols except the first line.
For example:
<FIRST LINE OF TEXT>
㈀ Ⰰ ㈀Ⰰ 嘀愀 ㌀ 䌀栀椀愀 䐀⸀⸀⸀ 儀甀愀渀最 䠀ഊ<SECOND LINE OF TEXT>
...
Can anyone give some insight on why it works on cygwin but not windows?
EDIT: After printing the encoded layers as suggested.
In Windows environment:
unix
crlf
encoding(UTF-16)
utf8
unix
crlf
encoding(UTF-16)
utf8
In Cygwin environment:
unix
perlio
encoding(UTF-16)
utf8
unix
perlio
encoding(UTF-16)
utf8
The only difference is between the perlio and crlf layer.
[ I was going to wait and give a thorough answer, but it's probably better if I give you a quick answer than nothing. ]
The problem is that crlf and the encoding layers are in the wrong order. Not your fault.
For example, say you do print "a\nb\nc\n"; using UTF-16le (since it's simpler and it's probably what you actually want). You'd end up with
61 00 0D 0A 00 62 00 0D 0A 00 63 00 0D 0A 00
instead of
61 00 0D 00 0A 00 62 00 0D 00 0A 00 63 00 0D 00 0A 00
I don't think you can get the right results with the open pragma or with binmode, but it can be done using open.
open(my $fh, '<:raw:encoding(UTF-16):crlf', $qfn)
You'll need to append a :utf8 with some older version, IIRC.
It works on cygwin because the crlf layer is only added on Windows. There you'd get
61 00 0A 00 62 00 0A 00 63 00 0A 00
You have a typo in your encoding. It should be use open ":encoding(UTF-16)" Note the colon. I don't know why it would work on Cygwin but not Windows, but could also be a 5.12 vs 5.14 thing. Perl seems to make up for it, but it could be what's causing your problem.
If that doesn't do it, check if the encoding is being applied to your filehandles.
print map { "$_\n" } PerlIO::get_layers(*INPUT);
print map { "$_\n" } PerlIO::get_layers(*OUTPUT);
Use lexical filehandles (ie. open my $fh, "<", $file). Glob filehandles are global and thus something else in your program might be interfering with them.
If all that checks out, if lexical filehandles are getting the encoding(UTF-16) applied, let us know and we can try something else.
UPDATE: This may provide your answer: "BOMed UTF files are not suitable for streaming models, and they must be slurped as binary files instead." Looks like you have to read the file in as binary and do the encoding as a string. This may have been a bug fixed in 5.14.
UPDATE 2: Yep, I can confirm this is a bug that was fixed in 5.14.
The Problem
I'm having a problem with a Windows batch file and labels.
I keep getting this error:
The system cannot find the batch label
specified
What I've tried
Two computers; a WindowsXP and a 2003 Server.
Made sure it was encoded as ASCII
Editted the hex code for the line continuation characters. Tried replacing all with CR , LF, and CRLF in turn. All combinations give me the same error.
Tried inserting extra characters before the label to make the label past 512 characters.
Here is the code:
cls
#echo off
SET zip=7za a dependencies.7z
call:dozip "c:\temp\dir.txt"
pause
goto exit
:dozip
echo Testing 1.2.3...
%zip% %1
goto:eof
:exit
Here's the hex with CRLF (0d 0a).
63 6c 73 0d 0a 53 45 54 20 7a 69 70 3d 37 7a 61 20 61 20 64 65 70 65 6e 64 65 6e 63 69 65 73 2e 37 7a 0d 0a 63 61 6c 6c 3a 64 6f 7a 69 70 20 22 63 3a 5c 74 65 6d 70 5c 64 69 72 2e 74 78 74 22 0d 0a 0d 0a 70 61 75 73 65 0d 0a 67 6f 74 6f 20 65 78 69 74 0d 0a 0d 0a 3a 64 6f 7a 69 70 0d 0a 20 20 65 63 68 6f 20 54 65 73 74 69 6e 67 20 31 2e 32 2e 33 2e 2e 2e 0d 0a 20 20 25 7a 69 70 25 20 25 31 0d 0a 67 6f 74 6f 3a 65 6f 66 0d 0a 3a 65 78 69 74
Here's the console's output (when I remove #echo off):
C:\>SET zip=7za a dependencies.7z
C:\>call:dozip "c:\temp\dir.txt"
C:\>echo Testing 1.2.3...
Testing 1.2.3...
C:\>7za a dependencies.7z "c:\temp\dir.txt"
The system cannot find the batch label specified - dozip
C:\>pause
Press any key to continue . . .
It never actually creates the 7zip file, so I think I can assume that its crashing on this line;
7za a dependencies.7z "c:\temp\dir.txt"
If I run that line by itself from a command prompt, it works fine and creates the dependencies.7z, so I don't think its necessarily a problem with 7za.exe.
I've already read this stackoverflow question:
stackoverflow.com/questions/232651/why-the-system-cannot-find-the-batch-label-specified-is-thrown-even-if-label-ex
and the link from that post;
help.wugnet.com/windows/system-find-batch-label-ftopict615555.html
The Answer
So, I found the problem guys.
I was using a technique I commonly use that I could only really describe as "proxy" batch files. I have a folder called c:\scripts, and I put several bat files in there to target commonly used exes. This saves my PATH variable from becoming absolutely massive with all of my command line tools. This way I only need to add c:\scripts to my PATH, and create a proxy batch file when I need something.
I had 7za.bat in c:\scripts, containing only this;
#echo off
"C:\Program Files\7-zip\7za.exe" %*
I changed my script to this;
SET zip="c:\program files\7-zip\7za.exe" a dependencies.7z
instead of this;
SET zip=7za a dependencies.7z
and it worked flawlessly.
The moral of the story...
Avoid calling other batch files from within a batch file. If you do, you will need to prefix them with "call".
I would point out that the "Testing 1.2.3..." and "Press any key to continue . . ." lines indicate that execution has successfully gone to the :dozip label and then successfully returned to the caller.
Is the "7za" executable actually a batch file? If I modify my test script to have the helper be a batch file, I get the same error. The fix is to do 'call %zip% %1'
the moral of the story: when calling external programs/batch files in a batch file, use call
call foo.bat
and/or
call %foo%
(Calling one batch from another has been done since the days of DOS, just remember to call)
The "The system cannot find the batch label specified" error message brought me here (10 years later!) and the problem turned out to be the CRLF was incorrect.
The cause, in our case, was a Git Repository that didn't recognize BAT files as being text files which needed to be CRLF on a Windows 7 machine.
Our solution was to create a .gitattributes file containing the line:
*.bat text eol=crlf
Then deleting the BAT files and checking out from the repository re-wrote our BAT files with the correct line endings. The BAT labels are now working again.
One possibility, although it seems unlikely, is that command extensions aren't enabled, or up-to-date, and this is interfering with call/goto/label behaviour.
Try:
echo [%cmdextversion%]
and if it's less than [2] (or empty -- []) then check to see if cmd.exe is being invoked with /e:off, or just run
cmd /e:on
in the console window where you will run this batch file.
In my case the reasons of "The system cannot find the batch label specified" were:
No CALL before jar- and unzip-commands
No ENDLOCAL for every SETLOCAL DisableDelayedExpansion
Some other batch-magic, which I didn't completely understood near SETLOCAL DisableDelayedExpansion, it was important to place SET param=1 after SETLOCAL DisableDelayedExpansion or something like that
Are you using Windows NT 4/Windows 2000? Only there can you use CALL to call subroutines within the same batch file.
Looking closely at your hex, it does not actually have all CRLF (0d 0a). Several lines end in LF-only (0a without a preceding 0d).
Check in your hex editor to make sure every 0a is preceded by 0d (exactly one).
Or just cut and paste your file into a blank Notepad document and re-save it.