My company currently is using S3fs and Ec2 from AWS. We have been mounted our s3 buckets on our Ec2 instances, but after some time (a week, for example) some of the buckets unmount by themselves and our server instances become nearly useless. The error is "Transport endpoint not connected."
S3fs version: 1.61 build from source
FUSE version: 2.84.1 build from source
OS: Linux, Ubuntu 11.04
Is there some kind of safe mechanism for preventing (or at least detecting) these problems?
Great insight. Hadn't thought about this. But here are 3 precautionary steps we can take:
1) Create an auto-mount so that in the very unlikely event that EC2 is down, S3 gets mounted back on once EC2 comes back via /etc/fstab
2) or/and if you prefer, create a secondary auto-mount using cron:
echo "/usr/bin/s3fs [s3 bucket name] [mountpoint path] -o allow_other" >> automount-s3
sudo mv automount-s3 /usr/sbin
sudo chown root:ubuntu /usr/sbin/automount-s3
sudo chmod +x /usr/sbin/automount-s3
crontab -e
add this line
#reboot /usr/sbin/automount-s3
3) I would also create another hourly cron to check whether S3 is still mounted - this can be done by checking if a dummy file exists in your EC2 path. If the file doesn't exist, cron will do a manual mount by calling "/usr/bin/s3fs -o allow_other [s3 bucket name] [mountpoint path]". It would be good to trigger an email to the admin and log it in the system as well.
s3fs is a nice idea but keep in mind that even though the call to s3 might be somewhat internal (or let's say "on their network"), you're still mounting a filesystem over HTTP. That is not going to be stable in the long-run.
Maybe you can re-phrase your question to ask for alternatives and share what you're trying to accomplish by using any kind of (I'm guessing) shared network filesystem. I can see the appeal, but with Amazon EC2 people usually use a shared nothing approach and anything extra network-related should be avoided to be able to recycle instances easier, etc..
Happy to extend my answer.
Related
I'm trying to set up a linux container with isolated mount namespace using unshare tool from util-linux package :
% sudo unshare -m -f /bin/bash
So I'm expecting that bash will be launched in a namespace, where the mount namespace, i.e. filesystems, will be completely isolated form the host one, however I still can modify the host FS (create/delete files on the host FS). What am I doing wrong here?
A mount namespace only creates a separate mount tree by copying the parent tree.
You still have to remount the file systems as read-only, unmount them, mount a tmpfs over them or pivot_root into a clean tree to prevent access. Switching to an umapped user via user namespaces can help to some extent but it won't prevent access to world-readable/writable files.
If you need to setup more complex namespace environments - containers basically - you can use firejail or runc to automate those tasks based on configuration files. systemd-nspawn provides some intermediate featureset between accessing the primitives directly, as unshare does, and container runtimes.
I assume that mount namespace is isolated because mount/unmount in the namespace does not have impact on the host FS. So I think modifying FS is another issue, probably related to userns, but not fully sure about this.
I have a container that I start like
docker run -it --mount type=bind,source=/path/to/my/data,target=/staging -v myvol:/myvol buildandoid bash -l
It has two mounts, one bind mount that I use to get data into the container, and one named volume that I use to persist data. The container is used as a reproducable android (AOSP) build environment, so not your typical web service.
I would like to access the files on myvol from the Windows host. If I use an absolute path for the mount, e.g. -v /c/some/path:/myvol, I can do that, but I believe docker creates copies of all the files and keeps them in sync. I really want to avoid creating these files on the windows side (for space reasons, as it is several GB, and performance reasons, since NTFS doesn't seem to handle many little files well).
Can I somehow "mount" a container directory or a named volume on the host? So the exact reverse of a bind mount. I think alternatively I could install samba or sshd in the container, and use that, but maybe there is something built into docker / VirtualBox to achive this.
Use bind mounts.
https://docs.docker.com/engine/admin/volumes/bind-mounts/
By contrast, when you use a volume, a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.
I would like to automate these steps:
Unzip a zip package (is it possible loading this zip on S3 bucket and downloading it during script? If yes, how?)
Edit apache configuration files (port.conf, /etc/apache2/sites-available/example.com.conf)
Run apt-get commands.
I really do not know how to create a script file to be run on EC2 instance startup.
Could everybody help me, please?
Thank you really much
What you're looking at is User Data, that will give you the possibility to run your script when ec2 instance is launched
When you create your ec2 instance, in step 3 (configure instance details) go to the bottom of the script and click on "Advanced Details". From there you can enter your script.
If you're using a Amazon AMI, the CLI is built in and you can use it, make sure to have ec2 IAM role defined with necessary rights on your AWS resources.
Now in terms of your script, this is vague but roughly:
you would run aws s3 cp s3_file local_file to download a zip file from s3 on the instance, use unzip linux command to unzip the content
Edit your files using sed, cat or >, see this Q&A
run commands with apt-get
Note: you're running the user-data script as root, so you dont need sudo when running commands.
Several articles have been extremely helpful in understanding Docker's volume and data management. These two in particular are excellent:
http://container-solutions.com/understanding-volumes-docker/
http://www.alexecollins.com/docker-persistence/
However, I am not sure if what I am looking for is discussed. Here is my understanding:
When running docker run -v /host/something:/container/something the host files will overlay (but not overwrite) the container files at the specified location. The container will no longer have access to the location's previous files, but instead only have access to the host files at that location.
When defining a VOLUME in a Dockerfile, other containers may share the contents created by the image/container.
The host may also view/modify a Dockerfile volume, but only after discovering the true mountpoint using docker inspect. (usually somewhere like /var/lib/docker/vfs/dir/cde167197ccc3e138a14f1a4f7c....). However, this is hairy when Docker has to run inside a Virtualbox VM.
How can I reverse the overlay so that when mounting a volume, the container files take precedence over my host files?
I want to specify a mountpoint where I can easily access the container filesystem. I understand I can use a data container for this, or I can use docker inspect to find the mountpoint, but neither solution is a good solution in this case.
The docker 1.10+ way of sharing files would be through a volume, as in docker volume create.
That means that you can use a data volume directly (you don't need a container dedicated to a data volume).
That way, you can share and mount that volume in a container which will then keep its content in said volume.
That is more in line with how a container is working: isolating memory, cpu and filesystem from the host: that is why you cannot "mount a volume and have the container's files take precedence over the host file": that would break that container isolation and expose to the host its content.
Begin your container's script with copying files from a read-only mount bind reflecting the host files to a work location in the container. End the script with copying necessary results from the container's work location back to the host using either the same or different mount point.
Alternatively to the end-of-the script command, run the container without automatically removing it at the end, then run docker cp CONTAINER_NAME:CONTAINER_DIR HOST_DIR, then docker rm CONTAINER_NAME.
Alternatively to copying results back to the host, keep them in a separate "named" volume, provided that the container had it mounted (type=volume,src=datavol,dst=CONTAINER_DIR/work). Use the named volume with other docker run commands to retrieve or use the results.
The input files may be modified in the host during development between the repeated runs of the container. Avoid shadowing them with the frozen files in the named volume. Beginning the container script with copying the input files from the host may help.
Using a named volume helps running the container read-only. (One may still need --tmpfs /tmp for temporary files or --tmpfs /tmp:exec if some container commands create and run executable code in the temporary location).
In an automated system, i copy files to a mounted network volume with a sh
In basic i do "cp file.pdf /Volumes/NetworkShare/".
This works well until the remote system is down.
So before copying i can do a ping to detect if it's online.
But... when i get online OSX often remounts on a different path "/Volumes/NetworkShare-1/".
The old path "/Volumes/NetworkShare/" stil exists altough it's useless.
So, how can i find the actual mount point of this share in OSX cli?
I found out that diskutil does something like this for local disks, not for network volumes. Is there an equivalent for diskutil for network volumes?
The mount command (just on its own) will list all mounted filesystems. As for why OS X is creating that extra directory, that is pretty odd. Did you manually mount the filesystem, by any chance? If you created the “NetworkShare” directory yourself, OS X’s auto mounter might do what you’re suggesting.