Go build & exec: fork/exec: permission denied - macos

I need to build a program using the Go toolchain and then execute it. For some reasons I get a permission error due the forking. Is there a way to circumvent this error or any best practice? I think my program does something similar with Go test tool, though go test doesn't get this kind of error.
package main
import(
"os"
"os/exec"
"flag"
log "github.com/golang/glog"
)
func main(){
flag.Parse()
tdir := "abc"
if err := os.MkdirAll(tdir, 0777); err !=nil{
log.Error(err)
return
}
f, err := os.Create(tdir + "/main.go")
if err !=nil{
log.Error(err)
return
}
if err = f.Chmod(0777); err !=nil{
log.Error(err)
return
}
defer f.Close()
defer os.Remove(f.Name())
if _, err = f.Write([]byte(tpl)); err !=nil{
log.Error(err)
return
}
cmd := exec.Command("go", "build", "-o", "edoc")
cmd.Path = tdir
b, err := cmd.CombinedOutput()
if err !=nil{
log.Errorf("%s, err %v", b, err)
return
}
}
var tpl = `package main
import(
"fmt"
"flag"
)
func main(){
flag.Parse()
fmt.Printf("Hello World")
}`
Error:
E0202 18:24:42.359008 13600 main.go:36] , err fork/exec abc: permission denied
OS: OSX 10.11

You're changing the command path from the location of your go binary, to abc.
type Cmd struct {
// Path is the path of the command to run.
//
// This is the only field that must be set to a non-zero
// value. If Path is relative, it is evaluated relative
// to Dir.
Path string
If you want to change the working directory, use Cmd.Dir

Related

Writing to file from cmd output

I am trying to write a small code in Go that will collect and save stats from IPFS.
So my Go code will execute IPFS command and save its output in .txt file and keep updating that .txt file.
I am having trouble doing that.
This is my code:
package main
import (
"fmt"
"io"
"log"
"os"
"os/exec"
"time"
)
func ipfsCommand() (ipfsOutput string) {
// output and error
out, err := exec.Command("ipfs","stats","bitswap","--human").Output()
// if there are errors, print/log them
if err != nil {
log.Printf("error!")
log.Fatal(err)
} else {
log.Printf("no error, printing output")
fmt.Printf("%s", out)
}
return
}
func writeToFile(message string) error {
f, err := os.Create("outputTest2_2.txt")
if err != nil {
return err
}
defer f.Close()
l, err := io.WriteString(f, message)
if err != nil {
fmt.Println(err)
f.Close()
return err
}
fmt.Println(l, "bytes written successfully")
return f.Sync()
}
func main() {
// get current time
currentTime := time.Now()
fmt.Println("YYYY.MM.DD : ", currentTime.Format("2006.01.02 15:04:05"))
writeToFile(currentTime)
// get output from ipfs command
msg := ipfsCommand()
// write the output to file
writeToFile(msg)
fmt.Println("file written!!!")
/* // write to file many times
for i:=0;i<3;i++{
// get output from ipfs command
msg := ipfsCommand()
// write the output to file
writeToFile(msg)
}*/
}
When the above code is run, this is the error:
# command-line-arguments
.\test2.go:49:13: cannot use currentTime (type time.Time) as type string in argument to writeToFile
Again, I want to get output from IPFS and save it to .txt file along with current time. I want to do this in loop because I want to save output from IPFS over a long period of time.
I tried to fix your script as is, but it just has too many issues. Here is a
rewrite, maybe you can use it as a new starting point:
package main
import (
"os"
"os/exec"
"time"
)
func main() {
f, err := os.Create("outputTest2_2.txt")
if err != nil {
panic(err)
}
defer f.Close()
currentTime, err := time.Now().MarshalText()
if err != nil {
panic(err)
}
f.Write(append(currentTime, '\n'))
msg, err := exec.Command("go", "env").Output()
if err != nil {
panic(err)
}
f.Write(msg)
}

Can't find a public file from url in go

I am trying to get the content of a publicly available file using ioutil.ReadFile() but it doesn't find the file: panic: open http://www.pdf995.com/samples/pdf.pdf: No such file or directory
Here's my code:
// Reading and writing files are basic tasks needed for
// many Go programs. First we'll look at some examples of
// reading files.
package main
import (
"fmt"
"io/ioutil"
)
// Reading files requires checking most calls for errors.
// This helper will streamline our error checks below.
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
fileInUrl, err := ioutil.ReadFile("http://www.pdf995.com/samples/pdf.pdf")
if err != nil {
panic(err)
}
fmt.Printf("HERE --- fileInUrl: %+v", fileInUrl)
}
Here's a go playground example
ioutil.ReadFile() does not support http.
If you look at the source code(https://golang.org/src/io/ioutil/ioutil.go?s=1503:1549#L42), open the file using os.Open.
I think I can do this coding.
package main
import (
"io"
"net/http"
"os"
)
func main() {
fileUrl := "http://www.pdf995.com/samples/pdf.pdf"
if err := DownloadFile("example.pdf", fileUrl); err != nil {
panic(err)
}
}
func DownloadFile(filepath string, url string) error {
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
return err
}
but, go playgound not protocol(go error dial tcp: Protocol not available).
so, You have to do it PC.

Docker client build error

This defeats me. I am not sure what I am doing wrong. There is so less documentation that searching did not produce good result. I will be happy to see what is the reason behind this strange behaviour.
I am on a MAC (10.11.6) and I am running docker for MAC (beta)
Here is the code I am trying to run
package main
import (
"fmt"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"golang.org/x/net/context"
)
func main() {
defaultHeaders := map[string]string{"User-Agent": "ego-v-0.0.1"}
cli, _ := client.NewClient("unix:///var/run/docker.sock", "v1.24", nil, defaultHeaders)
options := types.ImageBuildOptions{
Dockerfile: "/path/to/my/Dockerfile",
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true}
buildResponse, err := cli.ImageBuild(context.Background(), nil, options)
if err != nil {
fmt.Printf("%s", err.Error())
}
fmt.Printf("%s", buildResponse.OSType)
}
This gives me this error -
Error response from daemon: {"message":"Cannot locate specified Dockerfile: /path/to/my/Dockerfile"}
Whereas when I run this command (from the same directory where my Go code is)
docker build /path/to/my
It works absolutely fine.
What am I doing wrong? I feel like banging my head against a wall now. Please help.
------------ EDIT / ADD ------------
I ended up doing this -
package main
import (
"archive/tar"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"golang.org/x/net/context"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func tarit(source, target string) error {
filename := filepath.Base(source)
target = filepath.Join(target, fmt.Sprintf("%s.tar", filename))
fmt.Println(target)
tarfile, err := os.Create(target)
if err != nil {
return err
}
defer tarfile.Close()
tarball := tar.NewWriter(tarfile)
defer tarball.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
return filepath.Walk(source,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if err := tarball.WriteHeader(header); err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(tarball, file)
return err
})
}
func main() {
tarit("/dir/with/files/and/dockerfile", "repo")
dockerBuildContext, err := os.Open("./repo.tar")
defer dockerBuildContext.Close()
defaultHeaders := map[string]string{"User-Agent": "ego-v-0.0.1"}
cli, _ := client.NewClient("unix:///var/run/docker.sock", "v1.24", nil, defaultHeaders)
options := types.ImageBuildOptions{
Dockerfile: "repo/Dockerfile",
SuppressOutput: false,
Remove: true,
ForceRemove: true,
PullParent: true}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
if err != nil {
fmt.Printf("%s", err.Error())
}
fmt.Printf("********* %s **********", buildResponse.OSType)
}
Now it is not complaining about anything and I can see that the tar is getting made properly and the last println is printing
********* linux **********
Which is a reply from the server. But it does not build anything. I understand that reply is almost immediate as under the hood it is just a POST request. But not sure why it is not building anything although.
The Dockerfile has to be part of the build context that gets TARd and sent to the engine. You shouldn't use an absolute path to the Dockerfile, it must be relative to the context, so you just pass the name.
I don't think the CLI command you demonstrate is right - unless Dockerfile is actually a directory name in your case:
> ls /tmp/path/to/context
Dockerfile
> docker build /tmp/path/to/context/Dockerfile
unable to prepare context: context must be a directory: /tmp/path/to/context/Dockerfile
When you send just the path (which contains the Dockerfile file), that works:
> docker build /tmp/path/to/context
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM alpine
---> ee4603260daa
Successfully built ee4603260daa
The working code to this problem. For anyone who will probably be stuck like me -
package main
import (
"fmt"
"io/ioutil"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"github.com/jhoonb/archivex"
"golang.org/x/net/context"
)
func main() {
tar := new(archivex.TarFile)
tar.Create("/path/to/tar/archieve")
tar.AddAll("/path/to/base/folder", false)
tar.Close()
dockerBuildContext, err := os.Open("/path/to/tar/archieve.tar")
defer dockerBuildContext.Close()
defaultHeaders := map[string]string{"Content-Type": "application/tar"}
cli, _ := client.NewClient("unix:///var/run/docker.sock", "v1.24", nil, defaultHeaders)
options := types.ImageBuildOptions{
SuppressOutput: true,
Remove: true,
ForceRemove: true,
PullParent: true,
Tags: []string{"xxx"}}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, options)
defer buildResponse.Body.Close()
if err != nil {
fmt.Printf("%s", err.Error())
}
//time.Sleep(5000 * time.Millisecond)
fmt.Printf("********* %s **********", buildResponse.OSType)
response, err := ioutil.ReadAll(buildResponse.Body)
if err != nil {
fmt.Printf("%s", err.Error())
}
fmt.Println(string(response))
}
Be careful that a lot of error checks are not done in this code. This is just a working code where I am using docker client lib successfully.
Let me know what you think.
Thanks to -
"github.com/jhoonb/archivex"

Build Docker Image From Go Code

I'm trying to build a Docker image using the Docker API and Docker Go libraries (https://github.com/docker/engine-api/). Code example:
package main
import (
"fmt"
"github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"golang.org/x/net/context"
)
func main() {
defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"}
cli, err := client.NewClient("unix:///var/run/docker.sock", "v1.22", nil, defaultHeaders)
if err != nil {
panic(err)
}
fmt.Print(cli.ClientVersion())
opt := types.ImageBuildOptions{
CPUSetCPUs: "2",
CPUSetMems: "12",
CPUShares: 20,
CPUQuota: 10,
CPUPeriod: 30,
Memory: 256,
MemorySwap: 512,
ShmSize: 10,
CgroupParent: "cgroup_parent",
Dockerfile: "dockerSrc/docker-debug-container/Dockerfile",
}
_, err = cli.ImageBuild(context.Background(), nil, opt)
if err == nil || err.Error() != "Error response from daemon: Server error" {
fmt.Printf("expected a Server Error, got %v", err)
}
}
The error is always same:
Error response from daemon: Cannot locate specified Dockerfile: dockerSrc/docker-debug-container/Dockerfile
or
Error response from daemon: Cannot locate specified Dockerfile: Dockerfile
Things I've checked:
The folder exists in build path
I tried both relative and absolute path
There are no softlinks in the path
I tried the same folder for binary and Dockerfile
docker build <path> works
and bunch of other stuff
My other option was to use RemoteContext which looks like it works, but only for fully self contained dockerfiles, and not the ones with "local file presence".
Update:
Tried passing tar as buffer, but got the same result with the following:
dockerBuildContext, err := os.Open("<path to>/docker-debug- container/docker-debug-container.tar")
defer dockerBuildContext.Close()
opt := types.ImageBuildOptions{
Context: dockerBuildContext,
CPUSetCPUs: "2",
CPUSetMems: "12",
CPUShares: 20,
CPUQuota: 10,
CPUPeriod: 30,
Memory: 256,
MemorySwap: 512,
ShmSize: 10,
CgroupParent: "cgroup_parent",
// Dockerfile: "Dockerfile",
}
_, err = cli.ImageBuild(context.Background(), nil, opt)
The following works for me;
package main
import (
"archive/tar"
"bytes"
"context"
"io"
"io/ioutil"
"log"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
)
func main() {
ctx := context.Background()
cli, err := client.NewEnvClient()
if err != nil {
log.Fatal(err, " :unable to init client")
}
buf := new(bytes.Buffer)
tw := tar.NewWriter(buf)
defer tw.Close()
dockerFile := "myDockerfile"
dockerFileReader, err := os.Open("/path/to/dockerfile")
if err != nil {
log.Fatal(err, " :unable to open Dockerfile")
}
readDockerFile, err := ioutil.ReadAll(dockerFileReader)
if err != nil {
log.Fatal(err, " :unable to read dockerfile")
}
tarHeader := &tar.Header{
Name: dockerFile,
Size: int64(len(readDockerFile)),
}
err = tw.WriteHeader(tarHeader)
if err != nil {
log.Fatal(err, " :unable to write tar header")
}
_, err = tw.Write(readDockerFile)
if err != nil {
log.Fatal(err, " :unable to write tar body")
}
dockerFileTarReader := bytes.NewReader(buf.Bytes())
imageBuildResponse, err := cli.ImageBuild(
ctx,
dockerFileTarReader,
types.ImageBuildOptions{
Context: dockerFileTarReader,
Dockerfile: dockerFile,
Remove: true})
if err != nil {
log.Fatal(err, " :unable to build docker image")
}
defer imageBuildResponse.Body.Close()
_, err = io.Copy(os.Stdout, imageBuildResponse.Body)
if err != nil {
log.Fatal(err, " :unable to read image build response")
}
}
#Mangirdas: staring at a screen long enough DOES help - at least in my case. I have been stuck with the same issue for some time now.
You were right to use the tar file (your second example). If you look at the API doc here https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/build-image-from-a-dockerfile you can see that it expects a tar.
What really helped me was looking at other implementations of the client, perl and ruby in my case. Both create a tar on the fly when being asked to build an image from a directory.
Anyway, you only need to put your dockerBuildContext somewhere else (see the cli.ImageBuild())
dockerBuildContext, err := os.Open("/Path/to/your/docker/tarfile.tar")
defer dockerBuildContext.Close()
buildOptions := types.ImageBuildOptions{
Dockerfile: "Dockerfile", // optional, is the default
}
buildResponse, err := cli.ImageBuild(context.Background(), dockerBuildContext, buildOptions)
if err != nil {
log.Fatal(err)
}
defer buildResponse.Body.Close()
I am not there with naming the images properly yet, but at least I can create them...
Hope this helps.
Cheers
The Docker package has a function for creating a TAR from a file path. It's whats used by the CLI. It's not in the client package so it need to be installed separately:
import (
"github.com/mitchellh/go-homedir"
"github.com/docker/docker/pkg/archive"
)
func GetContext(filePath string) io.Reader {
// Use homedir.Expand to resolve paths like '~/repos/myrepo'
filePath, _ := homedir.Expand(filePath)
ctx, _ := archive.TarWithOptions(filePath, &archive.TarOptions{})
return ctx
}
cli.ImageBuild(context.Background(), GetContext("~/repos/myrepo"), types.ImageBuildOptions{...})
I agree with Marcus Havranek's answer, that method has worked for me. Just want to add how to add a name to an image, since that seemed like an open question:
buildOptions := types.ImageBuildOptions{
Tags: []string{"imagename"},
}
Hope this helps!
Combining a few of the answers, and adding how to correctly parse the returned JSON using DisplayJSONMessagesToStream.
package main
import (
"os"
"log"
"github.com/docker/docker/api/types"
"github.com/docker/docker/pkg/archive"
"github.com/docker/docker/pkg/jsonmessage"
"github.com/docker/docker/pkg/term"
"golang.org/x/net/context"
)
// Build a dockerfile if it exists
func Build(dockerFilePath, buildContextPath string, tags []string) {
ctx := context.Background()
cli := getCLI()
buildOpts := types.ImageBuildOptions{
Dockerfile: dockerFilePath,
Tags: tags,
}
buildCtx, _ := archive.TarWithOptions(buildContextPath, &archive.TarOptions{})
resp, err := cli.ImageBuild(ctx, buildCtx, buildOpts)
if err != nil {
log.Fatalf("build error - %s", err)
}
defer resp.Body.Close()
termFd, isTerm := term.GetFdInfo(os.Stderr)
jsonmessage.DisplayJSONMessagesStream(resp.Body, os.Stderr, termFd, isTerm, nil)
}
I've left our a few convenience functions like getCLI but I'm sure you have your own equivalents.
I encounter same problem. Finally find out the tar file should be docker build context even with Dockerfile.
Here is my code,
package main
import (
"log"
"os"
"github.com/docker/docker/api/types"
"github.com/docker/docker/client"
"golang.org/x/net/context"
)
func main() {
dockerBuildContext, err := os.Open("/Users/elsvent/workspace/Go/src/test/test.tar")
defer dockerBuildContext.Close()
buildOptions := types.ImageBuildOptions{
SuppressOutput: true,
PullParent: true,
Tags: []string{"xxx"},
Dockerfile: "test/Dockerfile",
}
defaultHeaders := map[string]string{"Content-Type": "application/tar"}
cli, _ := client.NewClient("unix:///var/run/docker.sock", "v1.24", nil, defaultHeaders)
buildResp, err := cli.ImageBuild(context.Background(), dockerBuildContext, buildOptions)
if err != nil {
log.Fatal(err)
}
defer buildResp.Body.Close()
}
opt := types.ImageBuildOptions{
Dockerfile: "Dockerfile",
}
filePath, _ = homedir.Expand(".")
buildCtx, _ := archive.TarWithOptions(filePath, &archive.TarOptions{})
x, err := cli.ImageBuild(context.Background(), buildCtx, opt)
io.Copy(os.Stdout, x.Body)

mkdir if not exists using golang

I am learning golang(beginner) and I have been searching on both google and stackoverflow but I could not find an answer so excuse me if already asked, but how can I mkdir if not exists in golang.
For example in node I would use fs-extra with the function ensureDirSync (if blocking is of no concern of course)
fs.ensureDir("./public");
Okay I figured it out thanks to this question/answer
import(
"os"
"path/filepath"
)
newpath := filepath.Join(".", "public")
err := os.MkdirAll(newpath, os.ModePerm)
// TODO: handle error
Relevant Go doc for MkdirAll:
MkdirAll creates a directory named path,
along with any necessary parents, and returns nil,
or else returns an error.
...
If path is already a directory, MkdirAll does nothing
and returns nil.
I've ran across two ways:
Check for the directory's existence and create it if it doesn't exist:
if _, err := os.Stat(path); os.IsNotExist(err) {
err := os.Mkdir(path, mode)
// TODO: handle error
}
However, this is susceptible to a race condition: the path may be created by someone else between the os.Stat call and the os.Mkdir call.
Attempt to create the directory and ignore any issues (ignoring the error is not recommended):
_ = os.Mkdir(path, mode)
This is one alternative for achieving the same but it avoids race condition caused by having two distinct "check ..and.. create" operations.
package main
import (
"fmt"
"os"
)
func main() {
if err := ensureDir("/test-dir"); err != nil {
fmt.Println("Directory creation failed with error: " + err.Error())
os.Exit(1)
}
// Proceed forward
}
func ensureDir(dirName string) error {
err := os.Mkdir(dirName, os.ModeDir)
if err == nil {
return nil
}
if os.IsExist(err) {
// check that the existing path is a directory
info, err := os.Stat(dirName)
if err != nil {
return err
}
if !info.IsDir() {
return errors.New("path exists but is not a directory")
}
return nil
}
return err
}
So what I have found to work for me is:
import (
"os"
"path/filepath"
"strconv"
)
//Get the cur file dir
path, err := filepath.Abs("./") // current opened dir (NOT runner dir)
// If you want runner/executable/binary file dir use `_, callerFile, _, _ := runtime.Caller(0)
// path := filepath.Dir(callerFile)`
if err != nil {
log.Println("error msg", err)
}
//Create output path
outPath := filepath.Join(path, "output")
//Create dir output using above code
if _, err = os.Stat(outPath); os.IsNotExist(err) {
var dirMod uint64
if dirMod, err = strconv.ParseUint("0775", 8, 32); err == nil {
err = os.Mkdir(outPath, os.FileMode(dirMod))
}
}
if err != nil && !os.IsExist(err) {
log.Println("error msg", err)
}
I like the portability of this.
Or you could attempt creating the file and check that the error returned isn't a "file exists" error
if err := os.Mkdir(path, mode); err != nil && !os.IsExist(err) {
log.Fatal(err)
}

Resources