I am trying to print the first hexagram from here in Perl.
The code below doesn't generate any errors, but doesn't print any hexagrams either.
use warnings;
use open ':encoding(utf8)';
binmode(STDOUT, ":utf8");
print "\x{4DC0}\n";
I was hoping to see this "䷀" not "Σ╖Ç".
You tell Perl your terminal is expecting UTF-8, but your terminal appears to expect one of the following:[1]
cp437
cp860
cp861
cp863
cp865
Seeing as these are all Windows code pages, I presume the terminal in question is a Windows console. If so, you can find out which encoding is expected using either of these commands:
chcp
perl -le"use Win32; print Win32::GetACP()"
Prepend cp to the number to get a name you can use with the Encode module (which is used by the :encoding layer).
Knowing the expected encoding won't help you, however. None of the character sets of these encodings contains "䷀", so your terminal can't display "䷀" without change.
You can switch the encoding expected by a Windows console to UTF-8 by issuing the following command:
chcp 65001
You may have to adjust the font in the console's properties.
I obtained the list of possible encodings using the following program:
use strict;
use warnings;
use feature qw( say );
use utf8;
use Encode qw( decode encode_utf8 );
my $output = encode_utf8("\x{4DC0}");
my $displayed = "䷀";
for my $encoding (Encode->encodings(":all")) {
defined( my $got = eval { decode($encoding, $output, Encode::FB_CROAK|Encode::LEAVE_SRC) } )
or next;
say $encoding if $output eq $displayed;
}
(Make sure the file is encoded using UTF-8.)
Related
Basically, I wrote a Perl script that creates an encoded command for Powershell and tries to run it. I had to explicitly convert the command string to utf-16 before base64-encoding it. I'm wondering why that's all I had to do get the script to work. What conversions is Perl on Windows* performing by default in the run of an "ordinary" program that interacts with the console and perhaps the file system? For instance, is argv converted? Is stdin/stdout converted? Does file IO go through a conversion?
✱ in particular, the Strawberry Perl distribution in case ActivePerl does something different
I'm trying to write a Perl script that calls many PowerShell fragments and depends on Strawberry Perl distribution.
PowerShell, rather conveniently, has an -encodedCommand flag that accepts a base64-encoded string and then processes it. This is helpful for avoiding quoting-related problems.
I tried the Simplest Thing That Could Possibly Work.
// powersheller.pl
#! /usr/bin/env perl
use strict;
use warnings;
use MIME::Base64;
use Encode qw/encode decode/;
use vars ('$powershell_command');
sub run_powershell_fragment {
my ($contents) = #_;
my $encoded = encode_base64($contents);
printf "encoded: %s\n", $encoded;
return `powershell.exe -noprofile -encodedCommand $encoded`;
}
printf "%s\n---\n", run_powershell_fragment($powershell_command);
BEGIN {
$powershell_command = <<EOF
echo "hi"
EOF
}
And ran it. Here's the output of the ... standard output channels (?) from running the perl script in the powershell window.
PS C\...> perl .\powersheller.pl
encoded: ZWNobyAiaGkiCQo=
Redundant argument in printf at .\powersheller.pl line 18.
?????? : The term '??????' is not recognized as the name of a cmdlet, function, script file, or operable program.
---
This looked like an encoding issue. I guessed that Perl was using something resembling utf-8 by default and powershell was expecting utf16-le or similar.
sub run_powershell_fragment {
my ($contents) = #_;
my $utf16_le_contents = encode("utf-16le", $contents);
my $encoded = encode_base64($utf16_le_contents);
printf "encoded: %s\n", $encoded;
return `powershell.exe -noprofile -encodedCommand $encoded`;
}
Technically, using "ucs-2le" also works. I don't know which is appropriate.
Anyway, all together, the program works as expected with the extra conversion inserted.
PS C:\...> perl .\powersheller.pl
encoded: ZQBjAGgAbwAgACIAaABpACIACQAKAA==
hi
---
Why was this all that I needed to do? Is Perl handling conversions related to argv and stdout &c?
qx`` performs no conversion. The command is expected to be encoded using the system's ANSI code page as it will be passed unmodified to CreateProcessA or similar.[1]
use Encode qw( encode );
use Win32 qw( );
my $cmd_ansi = encode("cp".Win32::GetACP(), $cmd);
`$cmd_ansi`
Of course, if the command contains only ASCII characters, encoding is moot.
Similarly, the values in #ARGV have not been decoded. They are received from the system encoded using the system's ANSI code page.
use Encode qw( decode );
use Win32 qw( );
my #decode_argv = map { decode("cp".Win32::GetACP(), $_) } #ARGV;
Of course, if the arguments contain only ASCII characters, decoding is moot.
By default, file handles do not perform any encoding or decoding except for CRLF ⇔ LF conversion (CRLF ⇒ LF on read, LF ⇒ CRLF on write). You are expected to provide a string of bytes (a string of characters with values in 0..255) to print/printf/say[1], and you will receive a string of bytes from the readline/read/readpipe.
You may provide an encoding/decoding layer when opening the file.
open(my $fh, '>:encoding(UTF-8)', $qfn)
You may provide an default encoding/decoding layer via the open pragma.
use open ':encoding(UTF-8)';
open(my $fh, '>', $qfn)
In both cases, you will now need to provide a string of Unicode Code Points to print/printf/say, and you will similarly receive a string of bytes from the readline/read/readpipe.
I'm not sure what's best for STDIN/STDOUT/STDERR, but you could start with the following:
use Win32 qw( );
my ($in_enc, $out_enc);
BEGIN {
$in_enc = "cp".Win32::GetConsoleCP();
$out_enc = "cp".Win32::GetConsoleOutputCP();
binmode STDIN, ":encoding($in_enc)";
binmode STDOUT, ":encoding($out_enc)";
binmode STDERR, ":encoding($out_enc)";
}
You should use UTF-16le rather than UCS-2le.
If you provide a string that contains non-bytes (characters outside of 0..255), Perl will assumes you meant to encode the string using UTF-8. It will warn ("Wide character") and encode the string using utf8.
I have a file in Unicode format on a windows machine. Is there any way to convert it to ASCII format on a windows machine using perl script
It's UTF-16 BOM.
If you want to convert unicode to ascii, you must be aware that some characters can't be converted, because they just don't exist in ascii.
If you can live with that, you can try this:
#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
use open IN => ':encoding(UTF-16)';
use open OUT => ':encoding(ascii)';
my $buffer;
open(my $ifh, '<', 'utf16bom.txt');
read($ifh, $buffer, -s $ifh);
close($ifh);
open(my $ofh, '>', 'ascii.txt');
print($ofh $buffer);
close($ofh);
If you do not have autodie, just remove that line - you should then change your open/close statements with a
open(...) or die "error: $!\n";
If you have characters that can't be converted, you will get warnings on the console and your output file will have e.g. text like
\x{00e4}\x{00f6}\x{00fc}\x{00df}
in it.
BTW: If you don't have a mom but know it is Big Endian (Little Endian), you can change the encoding line to
use open IN => ':encoding(UTF-16BE)';
or
use open IN => ':encoding(UTF-16LE)';
Hope it works under Windows as well. I can't give it a try right now.
Take a look at the encoding option on the Perl open command. You can specify the encoding when opening a file for reading or writing:
It'd be something like this would work:
#! /usr/bin/env perl
use strict;
use warnings;
use feature qw(say switch);
use Data::Dumper;
use autodie;
open (my $utf16_fh, "<:encoding(UTF-16BE)", "test.utf16.txt");
open (my $ascii_fh, ">:encoding(ASCII)", ".gvimrc");
while (my $line = <$utf16_fh>) {
print $ascii_fh $line;
}
close $utf16_fh;
close $ascii_fh;
I am struggling creating a file that contains non-ascii characters.
The following script works fine, if it is called with 0 as parameter but dies when called with 1.
The error message is open: Invalid argument at C:\temp\filename.pl line 15.
The script is started within cmd.exe.
I expect it to write a file whose name is either (depending on the paramter) äöü.txt or äöü☺.txt. But I fail to create the filename containing a smiley.
use warnings;
use strict;
use Encode 'encode';
# Text is stored in utf8 within *this* file.
use utf8;
my $with_smiley = $ARGV[0];
my $filename = 'äöü' .
($with_smiley ? '☺' : '' ).
'.txt';
open (my $fh, '>', encode('cp1252', $filename)) or die "open: $!";
print $fh "Filename: $filename\n";
close $fh;
I am probably missing something that is obvious to others, but I can't find, so I'd appreciate any pointer towards solving this.
First of all, saying "UTF-8 character" is weird. UTF-8 can encode any Unicode character, so the UTF-8 character set is the Unicode character set. That means you want to create file whose name contain Unicode characters, and more specifically, Unicode characters that aren't in cp1252.
I've answered this on PerlMonks in the past. Answer copied below.
Perl treats file names as opaque strings of bytes. That means that file names need to be encoded as per your "locale"'s encoding (ANSI code page).
In Windows, code page 1252 is commonly used, and thus the encoding is usually cp1252.* However, cp1252 doesn't support Tamil and Hindi characters [or "☺"].
Windows also provides a "Unicode" aka "Wide" interface, but Perl doesn't provide access to it using builtins**. You can use Win32API::File's CreateFileW, though. IIRC, you need to still need to encode the file name yourself. If so, you'd use UTF-16le as the encoding.
Aforementioned Win32::Unicode appears to handle some of the dirty work of using Win32API::File for you. I'd also recommend starting with that.
* — The code page is returned (as a number) by the GetACP system call. Prepend "cp" to get the encoding.
** — Perl's support for Windows sucks in some respects.
The following runs on Windows 7, ActiveState Perl. It writes "hello there" to a file with hebrew characters in its name:
#-----------------------------------------------------------------------
# Unicode file names on Windows using Perl
# Philip R Brenan at gmail dot com, Appa Apps Ltd, 2013
#-----------------------------------------------------------------------
use feature ":5.16";
use Data::Dump qw(dump);
use Encode qw/encode decode/;
use Win32API::File qw(:ALL);
# Create a file with a unicode name
my $e = "\x{05E7}\x{05EA}\x{05E7}\x{05D5}\x{05D5}\x{05D4}".
"\x{002E}\x{0064}\x{0061}\x{0074}\x{0061}"; # File name in UTF-8
my $f = encode("UTF-16LE", $e); # Format supported by NTFS
my $g = eval dump($f); # Remove UTF ness
$g .= chr(0).chr(0); # 0 terminate string
my $F = Win32API::File::CreateFileW
($g, GENERIC_WRITE, 0, [], OPEN_ALWAYS, 0, 0); # Create file via Win32API
say $^E if $^E; # Write any error message
# Write to the file
OsFHandleOpen(FILE, $F, "w") or die "Cannot open file";
binmode FILE;
print FILE "hello there\n";
close(FILE);
no need to encode the filename (at least not on linux). This code works on my linux system:
use warnings;
use strict;
# Text is stored in utf8 within *this* file.
use utf8;
my $with_smiley = $ARGV[0] || 0;
my $filename = 'äöü' .
($with_smiley ? '?' : '' ).
'.txt';
open my $fh, '>', $filename or die "open: $!";
binmode $fh, ':utf8';
print $fh "Filename: $filename\n";
close $fh;
HTH, Paul
I have the following code
use utf8;
open($file, '>:encoding(UTF-8)', "さっちゃん.txt") or die $!;
print $file "さっちゃん";
But I get the file name as ã•ã£ã¡ã‚ƒã‚“.txt
I was wondering if there was a way of making this work as I would expect (meaning I have a unicode file name) this without resorting to Win32::API, Win32API::* or moving to another platform and using a Samba share to modify the files.
The intent is to ensure we do not have any Win32 specific modules that need to be loaded (even conditionally).
Perl treats file names as opaque strings of bytes. They need to be encoded as per your "locale"'s encoding (ANSI code page).
In Windows, this is is usually cp1252. It is returned by the GetACP system call. (Prepend "cp"). However, cp1252 doesn't support Japanese characters.
Windows also provides a "Unicode" aka "Wide" interface, but Perl doesn't provide access to it using builtins*. You can use Win32API::File's CreateFileW, though. IIRC, you need to still need to encode the file name yourself. If so, you'd use UTF-16le as the encoding.
* — Perl's support for Windows sucks in some respects.
Use Encode::Locale:
use utf8;
use Encode::Locale;
use Encode;
open($file, '>:encoding(UTF-8)', encode(locale_fs => "さっちゃん.txt") ) or die $!;
print $file "さっちゃん";
The following produces a unicoded file name on Windows 7 using Activestate Perl.
#-----------------------------------------------------------------------
# Unicode file names on Windows using Perl
# Philip R Brenan at gmail dot com, Appa Apps Ltd, 2013
#-----------------------------------------------------------------------
use feature ":5.16";
use Data::Dump qw(dump);
use Encode qw/encode decode/;
use Win32API::File qw(:ALL);
# Create a file with a unicode name
my $e = "\x{05E7}\x{05EA}\x{05E7}\x{05D5}\x{05D5}\x{05D4}".
"\x{002E}\x{0064}\x{0061}\x{0074}\x{0061}"; # File name in UTF-8
my $f = encode("UTF-16LE", $e); # Format supported by NTFS
my $g = eval dump($f); # Remove UTF ness
$g .= chr(0).chr(0); # 0 terminate string
my $F = Win32API::File::CreateFileW
($g, GENERIC_WRITE, 0, [], OPEN_ALWAYS, 0, 0); # Create file via Win32API
say $^E if $^E; # Write any error message
# Write to the file
OsFHandleOpen(FILE, $F, "w") or die "Cannot open file";
binmode FILE;
print FILE "hello there\n";
close(FILE);
I have a Perl Script which was built on a Linux platform using Perl 5.8 . However now I am trying to run the Perl Script on a Windows platform command prompt with the same Perl version.
I am using this command perl rgex.pl however it gives me one whole chunk of errors which looks to me like it has already been resolved in the script itself. The weird thing is I am able to run another Perl script without problem consisting of simple functions such as print, input etc.
The Code:
#!/usr/bin/perl
use warnings;
use strict;
use Term::ANSIColor;
my $file = "C:\Documents and Settings\Desktop\logfiles.log";
open LOG, $file or die "The file $file has the error of:\n => $!";
my #lines = <LOG>;
close (LOG);
my $varchar = 0;
foreach my $line ( #lines ) {
if ( $line =~ m/PLLog/ )
{
print("\n\n\n");
my $coloredText = colored($varchar, 'bold underline red');
print colored ("POS :: $coloredText\n\n", 'bold underline red');
$varchar ++;
}
print( $line );
}
When I run on the windows command prompt it gives me errors such as:
Unrecognized escape \D passed through at rgex.pl line 7.
=> No such file or directory at rgex.pl line 8.
Please give some advice on the codes please. Thanks.
A \ in a Perl string enclosed in double quotes marks the beginning of an escape sequence like \n for newline, \t for tab. Since you want \ to be treated literally you need to escape \ like \\ as:
my $file = "C:\\Documents and Settings\\Desktop\\logfiles.log";
Since you are not interpolating any variables in the string it's better to use single quotes:
my $file = 'C:\Documents and Settings\Desktop\logfiles.log';
(Inside single quotes, \ is not special unless the next character is a backslash or single quote.)
These error messages are pretty clear. They tell you exactly which lines the problems are on (unlike some error messages which tell you the line where Perl first though "Hey, wait a minute!").
When you run into these sorts of problems, reduce the program to just the problematic lines and start working on them. Start with the first errors first, since they often cascade to the errors that you see later.
When you want to check the value that you get, print it to ensure it is what you think it is:
my $file = "C:\\D....";
print "file is [$file]\n";
This would have shown you very quickly that there was a problem with $file, and once you know where the problem is, you're most of the way to solving it.
This is just basic debugging technique.
Also, you're missing quite a bit of the basics, so going through a good Perl tutorial will help you immensely. There are several listed in perlfaq2 or perlbook. Many of the problems that you're having are things that Learning Perl deals with in the first couple of chapters.