Magento and Vagrant using rsync: performance and bidirectional sync - magento

Problem: on Windows 7, due to the fact that we can't use nfs (naturally, without hacks), the performance of the couple Vagrant/Magento are really poor.
After so much research, i found that the best way (maybe the only) to solve the problem is to use rsync. Ok, i succeed to use it and the performancies now are really good!
I found a problem: it seems to be that rsync is mono-directional. What i mean? Suppose I do install magento succesfully and then i call "vagrant rsync" command. It will perform a new sync of the folders and, cause it sync the guest file structure with the host file structure, it will "delete" the etc/app/local.xml file that Magento has built for me after the installation, just beacause it doesn't exist in the host file structure.
Now, i read some solution like exclude folders or file from sync, but i think it's really not a great way to solve the problem.
Someone has a better solution? There is a way to sync bidirectionally the two file structures?
UPDATE
I tried to find a solution.
1) I tried to use unison, but i found some kind of error i can't understand.
2) I tried to use the vagrant plugin rsync-back, but it seems it can't find the right folders to sync
3) I finally choose to execute the rsync within the virtual machine. Access through vagrant ssh, execute the command "rsync -av /var/www/ /vagrant.
It seems to work.
So, for the moment, the solution to improve the performance of Vagrant and Magento is to activate the rsync system. To solve the uni-directional sync problem, I need to execute the rsync command from the VM if i need to sync from guest to host (viceversa, it's enough to use vagrant rsync).
If you have a better way, please, give it to me!

Best option i know is unison (realtime bidirectional folder sync)
vagrant-unison plugin from https://github.com/mrdavidlaing/vagrant-unison is outdated and not functioning.
Get updated version of vagrant-unison plugin here https://github.com/dmatora/vagrant-unison

This is indeed the case right now; Vagrant's rsync support is not bidirectional—it only syncs from your host machine to the VM, and not back. There is an open issue to add two-way sync (rsync-push and rsync-pull), but I'm not sure when this issue will see the light of day.
Some other options, in the meanwhile:
vagrant rsync-back plugin - currently allows for only manual one-time rsync pulls, no support for rsync-auto
vagrant unison plugin - uses Unison to synchronize only one folder at a time
The main reason I like rsync is because it's one of the simplest/most robust file sync tools available for Mac/Windows/Linux, and since it's already installed on 2/3 of those platforms, only the Windows devs need to do any extra work to get it going. Most other options (NFS, Unison, etc.) require extra software for everyone, and don't offer much in the way of a performance gain over rsynced files.

The problem I had using rsync was it was a mono directional so if my app creates a file on the remote server, it will be deleted when rsynced next time.
I've tried the above suggestions but I've ended up with using ftp.
I'm using phpstorm and use the automatic deployment. So that if I change anything in the file or create a new one then it will automatically upload your files to the remote server (VM).
It's still one way but it works for me.

Related

Using Git in mounted encfs drive via Dokan

I would like to use Git for a software project which resides inside an encfs enctrypted drive mounted via Dokan (Windows environment). The Encryption of the files works just fine, unfortunately Git does not seem to like working in this environment. When initializing the Git repository I encounter the error message:
error: could not commit config file w:/djangodance/.git/config
When committing I encounter this - disk is writeable and quota is not exceeded:
fatal: Repository has been updated, but unable to write new_index file. Check that disk is not full or quota is not exceeded...
So far I have learned that Git does not seem to like certain drive-mounting-setups. This article (mounting remote filesystem via sshfs) proposes a workaround option (-oworkaround=rename).
My questions:
Did I locate the source of the problem correctly?
Is there some similar setup for Dokan which works with Git?
Is there a different approach which could satisfy on-the-fly encryption using Windows? I'd rather not use TrueCrypt since it is not as useful in combination with Dropbox.
Thank you for any hint!
Dokany is a fork of Dokan. It is very active by the maintainers and the community. It is now know as the main solution for writing driver and porting FUSE to Windows with the same code.
It also have the fix that you would be interested:
https://github.com/dokan-dev/dokany/pull/39
There was a problem with MoveFileEx. Without the fix this API fails with "permission denied" because the target file is opened and cannot be deleted.
Now the git command works via dokan!
The latest release of Dokan appears to be nearly three years ago. I don't imagine you'll get much support with it, even from the community.
You may want to consider using Bitlocker, a full-drive encryption system. Or possibly Encrypting Filesystem to encrypt your development directory. Both of these options are included in recent versions of Windows, and should be fully supported by Microsoft.

Easy FTP publishing for Vim (like Coda)

I've been using Vim (MacVim) exclusively for months now, and I love it. Before using Vim though, I used Coda (I'm on OSX), and the one thing I miss about Coda is the way it marks my files for publishing via FTP whenever I edit them. I can then choose to upload the modified files single files individually, or to publish all of them in one go.
Is there anything that will do this or similar in Vim?
I'm aware netrw can edit directly over FTP, but I have all my sites running locally as mirrors of the online sites, so I need to edit locally and publish files remotely as and when I need. I've Googled for solutions but can't find anything.
I'm far more productive in Vim while in the editor, but having to open up an FTP program and hunt and peck to copy across files as I edit them seperately is a real pain, and makes me less productive overall when working on websites when compared to working in Coda.
Any suggestions welcome :)
Transmit, Cyberduck and YummyFTP (the ones I know) all have their own version of "automated folder syncing" where you work locally and any modified files are uploaded on change: you setup an "observer" and never have to hit a special button or shortcut again.
If you want to only use MacVim you could write a little command that uploads the current file on save: a script that would use the current file's path to construct an scp command. But this doesn't seem very portable/practical to me. It appears some people have already written something like that: here and there.
The right way is (not only in my opinion) to:
have everything under some kind of VCS like Git, Subversion or Mercurial or whatever floats your boat
write your code in a local clone/checkout and test the hell out of it on a local server
commit only working code
push milestones to a staging server used by you and your clients to test everything, this part can be automated via a post-commit hook or something like that
deploy only validated changes on your production server.
All the aforementioned Version Control Systems can be used directly from Vim's command line with :!git commit or :!svn update… If you need more abstraction, Fugitive (Git-only) or VCSCommand (multi-VCS, my choice) are here to help.
On your loss of productivity due to uploading files: I think it's very normal because you essentially perform very different tasks with very different neurological needs. This speed bump can also be experienced when previewing a layout change in your browser, looking up a color in Photoshop or any other similar task. I don't know of a way to pilot Photoshop, Chrome's dev tools or Outlook from within Vim and I don't think such a gizmo could reallistically exist so you will probably have to bear with it.
You can try git-ftp - a git based command line ftp client. Then you can manage your project as a Git repository, and git-ftp will only upload the files marked with Git - and only if they have changed.
The downside is that you are going to have to learn Git - and that's a bit overkill for what you need.
The upside is that you are going to learn and use Git.
I have since stumbled upon a Vim plugin which works with Transmit (which I happen to use as my main FTP client) and allows me to upload the current file to the server with a simple keymap (Ctrl+U) as I edit.
This strikes a nice balance between being a very simple solution, and one that does enough of what I need to improve my productivity significantly.
Any more suggestions are still welcome!

Check in - Check out process/version control for PSDs and Image files

The title may not be so clear but the issue I am facing is this:
Are designers are working on large photoshop files across the network, this has a number of network traffic and file corruption issues which I am trying to overcome.
The way I want to do this is to have the designers copy the the files to their machine (Mac OSX) and work on them locally. But the problem then stands that they may forget to copy them back up or that another designer may start work on the version stored on the network.
What I need is a system where the designer checks out the files or folders from the server which locks those files so no other user can copy them until they are checked back in. We do not need to store revisions for the files.
My initial idea was to use SVN or preferably GIT and force lock on checkout somehow, does this sound feasible or is there a better system?
How big are the files on average? Not sure about GIT haven't used it but SVN should be ok - If you did go with SVN I would trial checking out over Http/Https vs Network Path to the repo as you may get a speed advantage out of one or the other. When we vpn to our repo at work it is literally 100 times faster over http than checking out using a network \\path to the repo.
SVN is a good option, but you will have revisions (this is the whole point of SVN). SVN doesn't lock files by default, but you may configure it so that it does. See http://svnbook.red-bean.com/nightly/en/svn-book.html?bcsi_scan_554E00F99A9AD604=0&bcsi_scan_filename=svn-book.html#svn.advanced.locking
I don't know git very well, but since it's not a centralized VCS, I'm pretty sure it isn't the right tool for your situation.

Editing remote files over SSH, using TextMate?

I LOVE using TextMate on my MacBook. It's great.
Unfortunately, I want to edit some files directly on my dev server, since it's difficult to recreate the environment locally. I'm using Git, so one alternative is to just edit locally, git commit, git push, and then git merge, but that's kind of complicated every time I want to make a simple change.
I'd rather just ... use another solution. One thing I tried is mounting a hard drive via MacFusion, and then loading that in an editor. But that's so freaking laggy/slow!
Has anyone cooked up a better solution?
OK - here is the one that works on Mountain Lion.
Go to http://osxfuse.github.com/
Install FUSE for OS X
Install SSHFS for OS X
Then the following commands on your terminal:
mkdir /Volumes/SSHFS
/usr/local/bin/sshfs username#host:/path/to/dir /Volumes/SSHFS
Done.
I would also recommend using the ReMate plugin as pointed out by another user to prevent TextMate from beach-balling every time you refocus it. Link:
ReMate http://ciaranwal.sh/remate
I use Fetch and TextMate for just such tasks. Fetch can be set to use TextMate as an external editor and can even automatically open files in TextMate by double clicking.
Saving the window in TextMate automatically pushes the file back to the server. Of course you would have to commit the changes on the server at a later time.
I'm sure most Mac FTP clients could do the same.
The best thing would be using TextMate's rmate script, follow the link and you'll find the instructions bellow, I recommend it since it will make your life easier and handle all the Nitty-Gritty.
I use the free version of TextWrangler for just this and it works great. I can load and save files over sftp.
The correct answer is to use sshfs and make sure "Perform atomic saves" is checked in the Textmate preferences window. The easiest way to setup sshfs is to use Macfusion. http://macfusionapp.org/.
Try one of these methods.
see: http://wiki.macromates.com/Main/FAQ#projects
also have a look at:
http://www.gnu.org/software/tramp/
You don't need to push every time you make simple changes; git is a distributed version control system, you commit to your local repository for the small changes. You should only push to the remote repository once you finished working on a feature/bug (or for really huge feature, a complete subfeature). Well, that's assuming you can recreate the environment; which apparently you can't.
A decent text editor can have integration with your favorite control version system; if you cannot configure your editor to commit and push from inside your editor, get a decent editor.
An even better editor can be set to save, commit, push, compile, and run your program all in one click or keypress.
If you are not able — for whatever reason — to replicate your environment locally and still want to use TextMate, the FTP client+TextMate combo is the best solution I can think of. MacFusion and all the other similar solutions are neat on the paper but awfully slow.
If you feel adventurous and confident enough to drop the TextMate requirement, SSH+Vim in the terminal works amazingly well.
Are you positive you can't replicate at all your remote environment?
I know this question already has several answers, and it's been a while, but I wanted to also point out DokanSSHFS - This will use SSH to make a local drive of the directory location on the server that you choose. Then you can use your editor of choice to edit the files as if they were on a local disk.
Most of the proposed solutions are centered around sshfs in one form or another. I have tried these solutions, but I found that reliability of filesystem is not always as good as desired.
There is tool called rmate, which allows editing of remote files in text mate.
Use command from ssh session to edit file on the server:
rmate file_name
The readme on github provides easy to follow instructions on how to set it up.
MacFusion is pretty sweet for free - basically ssh-mounting of directories.
http://macfusionapp.org/
Transmit 4 has a similar feature, tho it costs $$.
Try http://ciaranwal.sh/remate/ if it seems slow, as that will disable textmate from refreshing the file list so often.
Use Fuse for OS X http://osxfuse.github.com and the companion package, SSHFS (same URL) and install them. I installed the MacFUSE compatibility libraries from there too, just for good measure.
Then, install http://macfusionapp.org and follow the instructions located here ( https://github.com/osxfuse/osxfuse/wiki/SSHFS ) to configure macfusion to use the newer libraries.
I had trouble getting authenticated with password, so I set up ssh key authentication and used macfusion without password. Works like a charm.
If you do use an IDE, you could just set up an SSH tunnel to your dev server and edit your files from the comfort of your favourite IDE. Saving the files locally would automatically then push the files on the dev server as well
P.S: I am NOT endorsing the use of IDE
You need rmate it works fine to edit files on your server via ssh using TextMate on you local machine.
Github link here
rmate might be another choice. On server side, you type rmate /path/to/file. The file will be transferred to local machine, where you use some editor like Sublime Text or VS Code (TexMate may also work). To use it, one needs to install both server and client.
For server side, there are several ones in various languages. Choose one you like. Here is the Github repo.
On local machine, as far as I know, VS Code and Sublime Text have their extensions to receive files. For VS Code, refer to here. For Sublime Text, refer to here.
On the other hand, Microsoft just announced an official remote editing extension for VS Code (not released yet).
For those remote machines not having ruby or if bash not compiled with /dev/tcp, but has python, this works: https://github.com/scriptmaster/rmate-python
If you have pip:
pip install rmate
or simply:
wget https://raw.githubusercontent.com/scriptmaster/rmate-python/master/bin/rmate
chmod +x ./rmate
mv ./rmate /usr/local/bin/rmate
then rmate /path/to/file
especially if you are in a containerd-os with restrictions (with only python and docker) such as kubernetes-vm or gce-vm
you don't need vs-code-server, atom-editor,

Keeping FTP Folder Constantly Up To Date Based On Local Folder

Does anybody know a decent freeware FTP utility, preferably that can run in the background, that can mirror a local folder, constantly checking the files there for changes, and then automatically refreshing the FTP folder when the local files change?
WinSCP has a "keep remote directory up to date" feature under the commands tab which I use to do this. It first asks whether you want to sync the whole directory, then automatically uploads any file as soon as you save it. I'm currently looking for a similar solution for Mac.
I am interested in this as well. I know about the Rsync option of course, but would like to have a Windows option that does the same thing..
Thanks Everyone. I ended up using the FastNet NMFTP component in Borland C++ Buidler 5 to write my own Windows Version of Sync.
If you are running on a Linux or UNIX box you can use a shell script that periodically runs under cron to sync a folder to a "master" folder. You could use rsync in that shell script to do the actual sync.
Is that helpful?
David Kerins

Resources