I have Windows 10 OS, docker-compose and want to work with Apache Flink tutorial Playground, docker-compose starting correctly starting docker-compose but after several minutes of work,
Apache Flink has to create checkpoints, but there is some problem with access to the file system.
Exception:
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 104. Failure reason: Failure to finalize checkpoint.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1216) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
…..
Caused by: org.apache.flink.util.SerializedThrowable: Mkdirs failed to create file:/tmp/flink-checkpoints-directory/d73c2f87b0d7ea6748a1913ee4b50afe/chk-104
at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:262) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
Could you please help me with the correct path and docker access?
state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints-directory
state.savepoints.dir: file:///tmp/flink-savepoints-directory
Also I tried use full Windows path but got the same error.
Are you using Windows Docker containers or Linux Docker containers ?
Right-click on the Docker Desktop icon to see your current configuration.
Switch to Windows containers
OR
Switch to Linux containers
You have to configure Flink paths according to you target Docker container type.
Note: You cannot use Windows and Linux containers at the same time.
Related
How Docker Windows and Linux Containers run Simultaneously (how does it works) and why building doesn't work.
I try to run mixed windows and linux containers using docker compose.
I followed this
article and its worked (using compose version 2.4 and latest docker desktop).
Can someone explain me how does it work? Does it work only with wsl or I can use it using linux hyperv?
When I try to build new linux image (pulling works fine) I got an error, some commands gave me Invalid signal: WINCH and some gave me
returned a non-zero code: 4294967295:
failed to shutdown container: container ad12191abf0849d5e49bb5dc0570d6ba8eaf2cc5b4e7d77127ed381901fcb672
encountered an error during hcsshim::System::waitBackground: failure in a Windows system call:
The virtual machine or container with the specified identifier is not running. (0xc0370110):
subsequent terminate failed container ad12191abf0849d5e49bb5dc0570d6ba8eaf2cc5b4e7d77127ed381901fcb672
encountered an error during hcsshim::System::waitBackground: failure in a Windows system call:
The virtual machine or container with the specified identifier
Someone solved it?
We are constantly getting an error while starting our Beam Golang SDK pipeline (driver program) from a docker image which works when started from local / VM instance. We are using Dataflow runner for our pipeline and Kubernetes to deploy.
LOCAL SETUP:
We have GOOGLE_APPLICATION_CREDENTIALS variable set with service account for our GCP cluster. When running the job from local, job gets submitted to dataflow and completes successfully.
DOCKER SETUP:
Build image used is FROM golang:1.14-alpine. When we pack the same program with Dockerfile and try to run, it fails with error
User program exited: fork/exec /bin/worker: no such file or directory
On checking Stackdriver logs for more details, we see this:
Error syncing pod 00014c7112b5049966a4242e323b7850 ("dataflow-go-job-1-1611314272307727-
01220317-27at-harness-jv3l_default(00014c7112b5049966a4242e323b7850)"),
skipping: failed to "StartContainer" for "sdk" with CrashLoopBackOff:
"back-off 2m40s restarting failed container=sdk pod=dataflow-go-job-1-
1611314272307727-01220317-27at-harness-jv3l_default(00014c7112b5049966a4242e323b7850)"
Found reference to this error in Dataflow common errors doc, but it is too generic to figure out whats failing. After multiple retries, we were able to eliminate any permission / access related issues from pods. Not sure what else could be the problem here.
After multiple attempts, we decided to start the job manually from a new Debian 10 based VM instance and it worked. This brought to our notice that we are using alpine based golang image in Docker which may not have all the required dependencies installed to start the job.
On golang docker hub, we found a golang:1.14-buster where buster is codename for Debian 10. Using that for docker build helped us solve the issue. Self answering here to help anyone else facing the same issues.
I am trying the Docker Get Started tutorial, Part 3 (Services). So the part where I need to init a swarm and deploy a stack, all my service status is rejected:
The full error (using --no-trunc) is:
hnsCall failed in Win32: The parameter is incorrect. (0x57)
Here are the steps I am doing:
Ensure my image is correct (the docker run works well, I accessed localhost:4000 successfully). Then I stopped the container to make sure it does not interfere.
When I init the swarm, it says I have multiple addresses, so I chose a random one (I tried with either of them, same result) using --advertise-addr.
docker stack deploy works, but when I check the status with docker service ps, none of them are up. localhost:4000 has no listener.
Note: I switched Docker to a Windows container.
I am new to Docker and this is beyond me. Can anyone please suggest a solution/debug way?
I tried everything but cannot get it to run on a Windows container so I switched to Linux container. The Get Started part 3 runs well.
I have been experimenting with docker the aim is
Install docker on windows 16
for which I am following instructions from https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/quick_start_windows_server
everything looks fine till installation of the docker , after that when I try to test the docker it give
below error.
failed to register layer: re-exec error: exit status 1: output: ProcessBaseLayer C:\ProgramData\docker\windowsfilter\0c9
effd422805350acb1f051eb171399678aff003192c41be274acc4762b786c: The system cannot find the path specified.
here I am just trying to pull "hello-world" from docker hub
My ultimate aim is to run IIS on docker container and deploy the application running on IIS.
Had anyone faced such issue or could come up with any suggestions
its VM machine with
OS :- Win2016
HDD :- 50GB
RAM :- 4GB
You can't pull the hello-world image from docker hub because it's linux image. I used to see it before while playing with Docker Container on windows when trying to do the same thing.
If you want to get different image like IIS, just run command follow the instructions:
First:
Install-PackageProvider ContainerImage -Force
Then
Install-ContainerImage -Name WindowsServerCore
After that, restart your docker container with
Restart-Service docker
And you'll have the images of IIS on your machine.
I've tried to get the images of windows as the way we do in Linux but it always throw error as you have, I guess that all images we have on hub now are for Linux only, so if you want to have particular things, you must build it by yourselves or using existed images for Windows on Docker hub (about 9 or 10 images as I remember).
This is probably because there's no hello-world image for Windows. You can try running docker run windowsservercore cmd /C hello world.
I'm very new to kubernetes and trying to conceptualize it as well as set it up locally in order to try developing something on it.
There's a confound though that I am running on a windows machine.
Their "getting started" documentation in github says you have to run Linux to use kubernetes.
As docker runs on windows, I was wondering if it was possible to create a kubernetes instance as a container in windows docker and use it to manage the rest of the cluster in the same windows docker instance.
From reading the setup instructions, it seems like docker, kubernetes, and something called etcd all have to run "in parallel" on a single host operating system... But part of me thinks it might be possible to
Start docker, boot 'default' machine.
Create kubernetes container - configure to communicate with the existing docker 'default' machine
Use kubernetes to manage existing docker.
Pipe dream? Wrongheaded foolishness? I see there are some options around running it in a vagrant instance. Does that mean docker, etcd, & kubernetes together in a single VM (which in turn creates a cluster of virtual machines inside it?)
I feel like I need to draw a picture of what this all looks like in terms of physical hardware and "memory boxes" to really wrap my head around this.
With Windows, you need docker-machine and boot2docker VMs to run anything docker related.
There is no (not yet) "docker for Windows".
Note that issue 7428 mentioned "Can't run kubernetes within boot2docker".
So even when you follow instructions (from a default VM created with docker-machine), you might still get errors:
➜ workspace docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.14.2 /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests
ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
➜ workspace docker logs -f ee0b490f74f6bc9b70c1336115487b38d124bdcebf09b248cec91832e0e9af1d
W0428 09:09:41.479862 1 server.go:249] Could not load kubernetes auth path: stat : no such file or directory. Continuing with defaults.
I0428 09:09:41.479989 1 server.go:168] Using root directory: /var/lib/kubelet
The alternative would be to try on a full-fledge Linux VM (like the latest Ubuntu), instead of a boot2docker-like VM (based on a TinyCore distro).
All k8s components can be raised up with hyperkube, which helps you bring up a containerized one.
If you're able to run docker on windows, it would probably work. I haven't tried it on windows personally.