I am using this command to import csv file data to postgresql in omnidb windows :
COPY owner."order"(id,type,name)
FROM 'C:\Users\Desktop\omnidb_exported.csv' DELIMITER ';' CSV HEADER;
Getting this error, although it exists:
could not open file "C:\Users\Desktop\omnidb_exported.csv" for
reading: No such file or directory
I have also provided everyone security permissions of read and execute on csv file and its folder. Still the problem exists.
The csv file has delimiter ";" with header information.
This owner schema has 3 tables, which are connected by "id" column.
How to import the csv file data correctly? What is the problem with these commands?
OK, as below:
\copy owner."order"(id,type,name) FROM 'C:\Users\Desktop\omnidb_exported.csv' DELIMITER ';' CSV HEADER;
Just replace the copy to \copy, then can load data sucessfully.
I have a file ffmpeg_list_of_files.txt with the content
file '.\Output_0\forces_vs_radii.pdf'
file '.\Output_1\forces_vs_radii.pdf'
file '.\Output_2\forces_vs_radii.pdf'
file '.\Output_3\forces_vs_radii.pdf'
file '.\Output_4\forces_vs_radii.pdf'
and so on...
and then run ffmpeg -f concat -i ffmpeg_list_of_files.txt -c copy output.mkv as is stated at
http://trac.ffmpeg.org/wiki/Concatenate
I, unfortunately, get the error
Line 1: unknown keyword ' ■f'
.\ffmpeg_list_of_files.txt: Invalid data found when processing input
in Windows PowerShell in Windows 10.
What am I doing wrong?
It's an encoding problem I also got that I solved changing the file encoding.
Steps to solve it:
Open the ffmpeg_list_of_files.txt with the notepad, notepad++ or similar
Change the encoding to UTF-8 without BOM. To do it follow one of the next steps:
with windows notepad, this is done using the "save as..." option and at the bottom changing the encoding to "UTF-8" and press "save"
with notepad++ select "encoding" in the main menu and select "encoding with UTF-8 without BOM" and save the file after that
Names could change a little bit depending on the version, but following the steps it's pretty straight forward.
Note: In my case, when redirecting ls or dir to a file in powershell, the default encoding of the file is USC-2
This is my code:
# -*- coding: utf-8 -*-
import subprocess as sp
import locale
LOCAL_ENCODING = locale.getpreferredencoding()
cmds = ['dir', '/b', '*.txt']
out = sp.check_output(cmds, shell=True)
print(out)
print(out.decode(LOCAL_ENCODING))
s = 'レミリア・スカレート.txt'
print(s.encode(LOCAL_ENCODING, 'replace'))
print(LOCAL_ENCODING)
# print(s.encode('utf-8'))
This is the output:
b'\xa5\xec\xa5\xdf\xa5\xea\xa5\xa2?\xa5\xb9\xa5\xab\xa5\xec\xa9`\xa5\xc8.txt\r\n'
レミリア?スカレート.txt
b'\xa5\xec\xa5\xdf\xa5\xea\xa5\xa2?\xa5\xb9\xa5\xab\xa5\xec\xa9`\xa5\xc8.txt'
cp936
(A text file named 'レミリア・スカレート.txt' is in the script directory.)
As the result shows, the bytes of the file name returned has been automatically encoded by local encoding, which can't totally encode the filename(Note the ? in the bytes), thus some information lost.
Environment:
- win10 Chinese Simplified
- python-3.5.1
My question is:
Is it possible to avoid the automatical local-encoding and get an utf-8(or some other specified encoding) bytes?
I read this issue, but got no solution :-(
1.For built-in command, solved by eryksun's answer:
out = sp.check_output('cmd.exe /u /c "dir /b *.txt"').decode('utf-16le'),
/u: Output UNICODE characters (UCS-2 le),
/c: Run Command and then terminate)
2.For external programs:[no general solution]
configure the output using proper encoding(by setting exteral programs' options or configurations, of course, such options may be nonexistent),
for example, in the latest winrar, one can set the encoding of console rar messages:rar lb -scur data > list.txt, will produce Unicode list.txt with archived file names
I am trying to compile Z3 version 4.1.2. AFter a successful configuration, when you do "make", I get the following error:
Makefile:151: lib.srcs: No such file or directory
Makefile:152: shell.srcs: No such file or directory
Makefile:153: test.srcs: No such file or directory
Making test.srcs...
/usr/local/bin/dos2unix takes only stdin and stdout
make: *** [test.srcs] Error 1
I think the problem is that all textual files in z3-src-4.1.2.zip use "carriage return" (cr) and "line feed" (lf) for encoding line termination. The zip was created on a Windows machine. Another problem is the "dos2unix" application. It is an application that converts windows/dos textual files into unix/linux/osx textual files. It is a very simple application. It just replaces cr/lf with a lf.
On Linux, this application takes a single argument: the file name to be modified.
I'm currently working on a new build system that avoids this issues. In the meantime, here a some workarounds.
1) Use git to retrieve the source. git will take care of the cr/lf vs lf issue.
Here is the command for retrieving Z3:
git clone https://git01.codeplex.com/z3
If you do that, you don't need to use dos2unix.
So, you can remove the lines #$(DOS2UNIX) in Makefile.in. Another option is to replace
DOS2UNIX=#D2U#
with
DOS2UNIX=touch
in the beginning of Makefile.in
After these changes, you should be able to compile it on OSX. I successfully compiled it on OSX 10.7.
2) Get the "unstable" branch.
http://z3.codeplex.com/SourceControl/changeset/view/946a06cddbe4
This is the current "working branch". It contains the new build system. It is not ready, but it is good enough to generate the Z3 executable. Here are the instructions to build Z3 using this branch
Download the code from the page above. Or use git to retrieve the "unstable" branch. Then, execute
autoconf
./configure
python scripts/mk_make.py
cd build
make
I managed to compile it on OSX 10.7 last Friday.
3) Keep the .zip, but convert all textual files. I'm using the following python script to convert all files in the new build system. If you execute this python script in the Z3 root directory, it will convert all files.
import os
import glob
import re
import getopt
import sys
import shutil
def is_cr_lf(fname):
# Check whether text files use cr/lf
f = open(fname, 'r')
line = f.readline()
sz = len(line)
return sz >= 2 and line[sz-2] == '\r' and line[sz-1] == '\n'
# dos2unix in python
# cr/lf --> lf
def dos2unix(fname):
if is_cr_lf(fname):
fin = open(fname, 'r')
fname_new = '%s.new' % fname
fout = open(fname_new, 'w')
for line in fin:
line = line.rstrip('\r\n')
fout.write(line)
fout.write('\n')
fin.close()
fout.close()
shutil.move(fname_new, fname)
if is_verbose():
print "dos2unix '%s'" % fname
def dos2unix_tree_core(pattern, dir, files):
for filename in files:
if fnmatch(filename, pattern):
fname = os.path.join(dir, filename)
if not os.path.isdir(fname):
dos2unix(fname)
def dos2unix_tree():
os.path.walk('.', dos2unix_tree_core, '*')
dos2unix_tree()
This isn't really a programming question, is there a command line or Windows tool (Windows 7) to get the current encoding of a text file? Sure I can write a little C# app but I wanted to know if there is something already built in?
Open up your file using regular old vanilla Notepad that comes with Windows.
It will show you the encoding of the file when you click "Save As...".
It'll look like this:
Whatever the default-selected encoding is, that is what your current encoding is for the file.
If it is UTF-8, you can change it to ANSI and click save to change the encoding (or visa-versa).
I realize there are many different types of encoding, but this was all I needed when I was informed our export files were in UTF-8 and they required ANSI. It was a onetime export, so Notepad fit the bill for me.
FYI: From my understanding I think "Unicode" (as listed in Notepad) is a misnomer for UTF-16.
More here on Notepad's "Unicode" option: Windows 7 - UTF-8 and Unicdoe
If you have "git" or "Cygwin" on your Windows Machine, then go to the folder where your file is present and execute the command:
file *
This will give you the encoding details of all the files in that folder.
The (Linux) command-line tool 'file' is available on Windows via GnuWin32:
http://gnuwin32.sourceforge.net/packages/file.htm
If you have git installed, it's located in C:\Program Files\git\usr\bin.
Example:
C:\Users\SH\Downloads\SquareRoot>file *
_UpgradeReport_Files; directory
Debug; directory
duration.h; ASCII C++ program text, with CRLF line terminators
ipch; directory
main.cpp; ASCII C program text, with CRLF line terminators
Precision.txt; ASCII text, with CRLF line terminators
Release; directory
Speed.txt; ASCII text, with CRLF line terminators
SquareRoot.sdf; data
SquareRoot.sln; UTF-8 Unicode (with BOM) text, with CRLF line terminators
SquareRoot.sln.docstates.suo; PCX ver. 2.5 image data
SquareRoot.suo; CDF V2 Document, corrupt: Cannot read summary info
SquareRoot.vcproj; XML document text
SquareRoot.vcxproj; XML document text
SquareRoot.vcxproj.filters; XML document text
SquareRoot.vcxproj.user; XML document text
squarerootmethods.h; ASCII C program text, with CRLF line terminators
UpgradeLog.XML; XML document text
C:\Users\SH\Downloads\SquareRoot>file --mime-encoding *
_UpgradeReport_Files; binary
Debug; binary
duration.h; us-ascii
ipch; binary
main.cpp; us-ascii
Precision.txt; us-ascii
Release; binary
Speed.txt; us-ascii
SquareRoot.sdf; binary
SquareRoot.sln; utf-8
SquareRoot.sln.docstates.suo; binary
SquareRoot.suo; CDF V2 Document, corrupt: Cannot read summary infobinary
SquareRoot.vcproj; us-ascii
SquareRoot.vcxproj; utf-8
SquareRoot.vcxproj.filters; utf-8
SquareRoot.vcxproj.user; utf-8
squarerootmethods.h; us-ascii
UpgradeLog.XML; us-ascii
Another tool that I found useful: https://archive.codeplex.com/?p=encodingchecker
EXE can be found here
Install git ( on Windows you have to use git bash console). Type:
file --mime-encoding *
for all files in the current directory , or
file --mime-encoding */*
for the files in all subdirectories
Here's my take how to detect the Unicode family of text encodings via BOM. The accuracy of this method is low, as this method only works on text files (specifically Unicode files), and defaults to ascii when no BOM is present (like most text editors, the default would be UTF8 if you want to match the HTTP/web ecosystem).
Update 2018: I no longer recommend this method. I recommend using file.exe from GIT or *nix tools as recommended by #Sybren, and I show how to do that via PowerShell in a later answer.
# from https://gist.github.com/zommarin/1480974
function Get-FileEncoding($Path) {
$bytes = [byte[]](Get-Content $Path -Encoding byte -ReadCount 4 -TotalCount 4)
if(!$bytes) { return 'utf8' }
switch -regex ('{0:x2}{1:x2}{2:x2}{3:x2}' -f $bytes[0],$bytes[1],$bytes[2],$bytes[3]) {
'^efbbbf' { return 'utf8' }
'^2b2f76' { return 'utf7' }
'^fffe' { return 'unicode' }
'^feff' { return 'bigendianunicode' }
'^0000feff' { return 'utf32' }
default { return 'ascii' }
}
}
dir ~\Documents\WindowsPowershell -File |
select Name,#{Name='Encoding';Expression={Get-FileEncoding $_.FullName}} |
ft -AutoSize
Recommendation: This can work reasonably well if the dir, ls, or Get-ChildItem only checks known text files, and when you're only looking for "bad encodings" from a known list of tools. (i.e. SQL Management Studio defaults to UTF16, which broke GIT auto-cr-lf for Windows, which was the default for many years.)
A simple solution might be opening the file in Firefox.
Drag and drop the file into firefox
Press Ctrl+I to open the page info
and the text encoding will appear on the "Page Info" window.
Note: If the file is not in txt format, just rename it to txt and try again.
P.S. For more info see this article.
I wrote the #4 answer (at time of writing). But lately I have git installed on all my computers, so now I use #Sybren's solution. Here is a new answer that makes that solution handy from powershell (without putting all of git/usr/bin in the PATH, which is too much clutter for me).
Add this to your profile.ps1:
$global:gitbin = 'C:\Program Files\Git\usr\bin'
Set-Alias file.exe $gitbin\file.exe
And used like: file.exe --mime-encoding *. You must include .exe in the command for PS alias to work.
But if you don't customize your PowerShell profile.ps1 I suggest you start with mine: https://gist.github.com/yzorg/8215221/8e38fd722a3dfc526bbe4668d1f3b08eb7c08be0
and save it to ~\Documents\WindowsPowerShell. It's safe to use on a computer without git, but will write warnings when git is not found.
The .exe in the command is also how I use C:\WINDOWS\system32\where.exe from powershell; and many other OS CLI commands that are "hidden by default" by powershell, *shrug*.
you can simply check that by opening your git bash on the file location then running the command file -i file_name
example
user filesData
$ file -i data.csv
data.csv: text/csv; charset=utf-8
Some C code here for reliable ascii, bom's, and utf8 detection: https://unicodebook.readthedocs.io/guess_encoding.html
Only ASCII, UTF-8 and encodings using a BOM (UTF-7 with BOM, UTF-8 with BOM,
UTF-16, and UTF-32) have reliable algorithms to get the encoding of a document.
For all other encodings, you have to trust heuristics based on statistics.
EDIT:
A powershell version of a C# answer from: Effective way to find any file's Encoding. Only works with signatures (boms).
# get-encoding.ps1
param([Parameter(ValueFromPipeline=$True)] $filename)
begin {
# set .net current directoy
[Environment]::CurrentDirectory = (pwd).path
}
process {
$reader = [System.IO.StreamReader]::new($filename,
[System.Text.Encoding]::default,$true)
$peek = $reader.Peek()
$encoding = $reader.currentencoding
$reader.close()
[pscustomobject]#{Name=split-path $filename -leaf
BodyName=$encoding.BodyName
EncodingName=$encoding.EncodingName}
}
.\get-encoding chinese8.txt
Name BodyName EncodingName
---- -------- ------------
chinese8.txt utf-8 Unicode (UTF-8)
get-childitem -file | .\get-encoding
Looking for a Node.js/npm solution? Try encoding-checker:
npm install -g encoding-checker
Usage
Usage: encoding-checker [-p pattern] [-i encoding] [-v]
Options:
--help Show help [boolean]
--version Show version number [boolean]
--pattern, -p, -d [default: "*"]
--ignore-encoding, -i [default: ""]
--verbose, -v [default: false]
Examples
Get encoding of all files in current directory:
encoding-checker
Return encoding of all md files in current directory:
encoding-checker -p "*.md"
Get encoding of all files in current directory and its subfolders (will take quite some time for huge folders; seemingly unresponsive):
encoding-checker -p "**"
For more examples refer to the npm docu or the official repository.
Similar to the solution listed above with Notepad, you can also open the file in Visual Studio, if you're using that. In Visual Studio, you can select "File > Advanced Save Options..."
The "Encoding:" combo box will tell you specifically which encoding is currently being used for the file. It has a lot more text encodings listed in there than Notepad does, so it's useful when dealing with various files from around the world and whatever else.
Just like Notepad, you can also change the encoding from the list of options there, and then saving the file after hitting "OK". You can also select the encoding you want through the "Save with Encoding..." option in the Save As dialog (by clicking the arrow next to the Save button).
The only way that I have found to do this is VIM or Notepad++.
EncodingChecker
File Encoding Checker is a GUI tool that allows you to validate the text encoding of one or more files. The tool can display the encoding for all selected files, or only the files that do not have the encodings you specify.
File Encoding Checker requires .NET 4 or above to run.