I must convert regularly a file which contains uft-8-mac strings to uft-8. I started to do the job with iconv.
How ever iconv throws an error, if there was too many lines which must be converted.
Here is a script to reproduce the bug
#!/bin/zsh
set -eu
for i in {1..1000}; do
echo "$i:äöüß#€" >> /tmp/xx
iconv -f utf-8-mac -t utf-8 /tmp/xx > /dev/null
done
Obviously I can split the file, but I get then really a lot of files.
Has anyone another workaround or tool? Or a code example in golang?
I tried
func main() {
dat, err := os.ReadFile(".backup_files.unconv")
if err != nil {
log.Fatal(err)
}
output := ".backup_files.goconv"
w, err := os.Create(output)
if err != nil {
log.Fatalf("Can't create %s, %v", output, err)
}
defer closeFile(w)
wc := norm.NFC.Writer(w)
defer wc.Close()
wc.Write(dat)
}
But it differs from iconv result.
Thanks in advance.
Found an appropriate solution:
You can use the uconv utility from ICU. Normalization is achieved through transliteration (-x).
On Debian, Ubuntu and other derivatives, uconv is in the libicu-dev package. On Fedora, Red Hat and other derivatives, and in BSD ports, it's in the icu package.
Thanks to Gilles 'SO- stop being evil'
Related
I try to get file from private gitlab repository using os/exec with curl and get 404 response status:
func Test_curl(t *testing.T) {
cmd := exec.Command(
`curl`,
`-H`, `PRIVATE-TOKEN:token`,
`https://gitlab.some.com/api/v4/projects/23/repository/files/.gitignore/raw\?ref\=master`,
)
t.Log(cmd.String())
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
t.Fatal(err)
}
t.Log(out.String())
}
=== RUN Test_curl
t_test.go:166: /usr/bin/curl -H PRIVATE-TOKEN:token https://gitlab.some.com/api/v4/projects/23/repository/files/.gitignore/raw\?ref\=master
t_test.go:175: {"error":"404 Not Found"}
--- PASS: Test_curl (0.25s)
but when I try to use the same command from zsh, I get right response:
% /usr/bin/curl -H PRIVATE-TOKEN:token https://gitlab.some.com/api/v4/projects/23/repository/files/.gitignore/raw\?ref\=master
.DS_Store
.vs/
.vscode/
.idea/
I think the problem in the url, but don't understand how to fix one.
? and = must not be quoted:
cmd := exec.Command(
`curl`,
`-H`, `PRIVATE-TOKEN:token`,
`https://gitlab.some.com/api/v4/projects/23/repository/files/.gitignore/raw?ref=master`,
)
exec.Command does not spawn a shell, therefore shell globs do not need escaping.
I'm unable to echo a string into a new file , it works if I use a filename with some extension(.txt, .go, etc) but it doesn't work when I just use a filename without file extension
I want the below bash commands to be executed in golang
echo "testDir/*"> .git/info/sparse-checkout
git checkout <Branch Name>
code snippet:
// Remove the redirect from command
cmd := exec.Command("echo", "testDir/*")
// Make test file
testFile, err := os.Create(".git/info/sparse-checkout")
if err != nil {
panic(err)
}
defer testFile.Close()
// Redirect the output here (this is the key part)
cmd.Stdout = testFile
err = cmd.Start(); if err != nil {
panic(err)
}
cmd.Wait()
branchCmd := exec.Command("git checkout <Branch Name>")
The echo command just print the list of arguments. The command interpreter (shell) expand the * based on globing existing filenames.
Without the shell, you can’t echo anything. You may need to call the shell (bash for instance) to expand and echo it, or use the io package to list files based on this pattern
If I understand correctly : in your bash command, echo "testDir/*" doesn't expand anything, it just outputs the string "testDir/*". echo ... > file is just one of the gazillion ways to set the file content to a given value from your shell.
If you want to write a fixed string to a file in go, just write it :
_, err := testFile.Write([]byte("testDir/*\n"))
no need to start some external process to echo a value on stdout.
I have a bash script in which I am receiving response of an api in string format. The response is in the following format:
foo bar test stack over flow
Now I am having following bash script to convert it to array and process further:
#!/bin/bash
result=$(curl "API URL")
resp=($result)
for i in "${resp[#]}"
do
echo "$i"
done
Problem:
If I run this script manually in the terminal (by making it executable) it works fine. But when I try to run it by using Golang sh command
ExecuteCommand("sh /path/to/directory/test.sh")
func ExecuteCommand(command string) error{
cmd := exec.Command("sh", "-c",command)
var out bytes.Buffer
var stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
fmt.Println(fmt.Sprint(err) + ": " + stderr.String())
return err
}
fmt.Println("Result: " + out.String())
return nil
}
It gives me error:
test.sh: Syntax error: "(" unexpected
can someone help me what am I doing wrong ?
change these lines in your example
ExecuteCommand("/path/to/directory/test.sh")
func ExecuteCommand(command string) error{
cmd := exec.Command(command)
The kernel shebang will intercept the #! line and correctly run the script
U can also create a shortcut from "test.sh" file :
#!/bin/bash
$ sudo ln -s /path/to/directory/test.sh /usr/bin/testSH
If u arent root user give permission to the shortcut :
sudo chmod 777 /usr/bin/testSH
command := "testSH"
cmd := exec.Command("sh", "-c",command).Run()
I am attempting to run the following code in Go. I have tried both of the following ways:
out, err := exec.Command("sh", "-c", "tcpdump -i ens0 host 192.168.1.100 -F ./testfile").Output()
fmt.Println(string(out)) // Prints nothing
fmt.Println(err) // exit status 1
I have also tried replacing sh with /bin/bash.
I have also tried the following, with and without sh as the first argument:
out, err := exec.Command("tcpdump", "-i", "ens0", "host", "192.168.1.100", "-F", "./testfile").Output()
fmt.Println(string(out)) // Prints nothing
fmt.Println(err) // exit status 1
None of this is working. Can someone see what I am doing wrong? I have also tried this go package "github.com/kami-zh/go-capturer" to read stderr and again it prints nothing.
Normally I have to use sudo to execute tcpdump from shell, so I build the go binary and execute it as root user.
Something like this should work, i am not sure if there is any specific command like -F available in tcp dump,
If you want to capture plain output of the tcp dump , you can direct the output to file using > file . The -w option is for wireshark/tcpdump format , to read and display
cmd := exec.Command("sh", "-c", "sudo tcpdump -i <eth> host <ip> -w ./testfile")
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
Thanks, #torek, the -c option can be used with tcpdump to exit after capturing n packets
cmd := exec.Command("sh", "-c", "sudo tcpdump -i ens33 -c 100 host localhost -w ./testfile")
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
The other way is to use cmd.Start
cmd := exec.Command("sh", "-c", "sudo tcpdump -i ens33 -c 100 host localhost -w ./testfile")
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
log.Printf("Waiting for command to finish...")
err = cmd.Wait()
log.Printf("Command finished with error: %v", err)
The tcpdump command continue to run infinitely if you use cmd.run without -c option with tcpdump cmd.So you can't see if you put print statement after cmd.Run() call, the reason being the exec.Command failed is, it just work the same way on how it works from the cli, so if you need sodo in front for it , you should put it in command as well or run it from the root user.
I've run into an issue with the os/exec library. I want to run a shell and pass it multiple commands to run, but it's failing when I do. Here's my test code:
package main
import (
"fmt"
"os/exec"
)
func main() {
fmt.Printf("-- Test 1 --\n`")
command1 := fmt.Sprintf("\"%s\"", "pwd") // this one succeeds
fmt.Printf("Running: %s\n", command1)
cmd1 := exec.Command("/bin/sh", "-c", command1)
output1,err1 := cmd1.CombinedOutput()
if err1 != nil {
fmt.Printf("error: %v\n", err1)
return
}
fmt.Printf(string(output1))
fmt.Printf("-- Test 2 --\n")
command2 := fmt.Sprintf("\"%s\"", "pwd && pwd") // this one fails
fmt.Printf("Running: %s\n", command2)
cmd2 := exec.Command("/bin/sh", "-c", command2)
output2,err2 := cmd2.CombinedOutput()
if err2 != nil {
fmt.Printf("error: %v\n", err2)
return
}
fmt.Printf(string(output2))
}
When running this I get an error 127 on the second example. It seems like it's looking for a literal "pwd && pwd" command instead of evaluating it as a script.
If I do the same thing from the command line it works just fine.
$ /bin/sh -c "pwd && pwd"
I'm using Go 1.4 on OS X 10.10.2.
the quotes are for your shell where you typed the command line, they should not be included when programatically launching an app
just make this change and it will work:
command2 := "pwd && pwd" // you don't want the extra quotes