How to replace folder with symlink on a server - bash

We have a git repository for a scientific software where we need to maintain a certain folder structure for our data files.
These folders should remain empty, everything that will be put there should not be tracked by git. However, it is necessary that those folders exist.
The solution to accomplish this was to add a .gitignore file into every directory which looks like this:
*
!.gitignore
which means everything inside this folder is ignored except for the .gitignore file.
This works very well.
We maintain all our data on one particular server.
Our scientists use this server often for their calculations.
It would be very convenient to be able to replace the data folders from the git repository which currently contain only the .gitignore file with a symbolic link to the full data files on this server. The data files on the server also have a .gitignore file which looks exactly the same as in every repository.
I wrote a bash script to do this which looks like this:
rm -r path/to/empty/data/in/repository/name
ln -sfn /absolute/path/to/data/on/server/ path/to/empty/data/in/repository
Now the software runs perfectly and you have access to all the data without copying it into your git repository.
However, git now gets confused.
If I run git status only my changes are listed as expected. It does not complain about the new symbolic links which replaced the existing directories.
As soon as I run git add . to stage my changes the symbolic links appear as new file: and the .gitignore files in the replaced folder are listed as deleted:.
This seems like a problem to me because as soon as somebody pushes his code changes that he made on the server the symbolic links would get uploaded (I guess) and the .gitignore files would get removed and thus the folder structure would not remain.
Is it possible to tell git that it should compare the content of the symbolic linked folders rather than the symbolic link itself?
PS: I know this seems like a software design issue with the static folder structure which is inside git but I do not want to discuss this here. We are all scientists and no programmers and the software is now developed for over 10 years by many different people. It is not possible to change the code to make it more flexible.
EDIT: This bash code reproduces the problem:
cd ~ #setup
mkdir tmp
cd tmp
mkdir server #server data folder (this one is full of data)
mkdir server/data
printf '*\n!.gitignore' > server/data/.gitignore
printf 'data file 1' > server/data/data1.txt
printf 'data file 2' > server/data/data2.txt
mkdir repo #repo data folder (this one only contains .gitignore file)
mkdir repo/data
printf '*\n!.gitignore' > repo/data/.gitignore
cd repo # create a dummy repo
git init
git add .
git commit -am"commit 1"
git status
cd .. # replace data folder with server/data folder which hase exactly the same content
rm -r repo/data/
ln -sfn ~/tmp/server/data/ ./repo/
cd repo
git status
At the end git status should ideally not list any changes in the repository.
EDIT:
I found a workaround: instead of linking the whole directory I'm now linking the content of the directory:
ln -sfn /absolute/path/to/data/on/server/* path/to/empty/data/in/repository/
this works because the symbolic links are irgnored due to the .gitignore file.
Drawback is that it only works with existing data. As soon as there is a new file in the server directory I have to run the bash script again.

Git tracks symbolic links. What you're trying to achieve can be done with bind mounts.
Replace the final ln -sfn ~/tmp/server/data/ ./repo/ with sudo mount --bind $PWD/repo
$HOME/tmp/server/data/

Related

Hidden .git folder created on local

Using git init in my terminal on mac creates a hidden a folder. I used command + shift + . to show folder, but it appears faded and can't be detected for further functions.
enter image description here
Code:
cd "desired folder"
git init .
when i cd .git, it works; however, i cannot ls into .git. I can also add and remove in code but the changes appear as changes to be committed. When I open the .git folder my files aren't saved there.
Could someone please assist?
Thanks
Git stores its data within the .git directory, and that directory is created by running git init (or as a part of running git clone). On macOS, like other Unix systems, files that start with a dot are hidden, which is normal, and it's intended that this folder should be that way.
Once you've committed data, that data will be stored in the .git directory in an internal format, so it will not be readily visible. If you want to access it, you'll need to do so using the git command or a library, such as one based on libgit2.
You should, unless you know what you are doing, not modify the contents of the .git folder, possibly with the exception of the files in .git/info and .git/hooks (after reading the documentation thoroughly). Also, the history of your project is stored there, so if you delete the directory or its contents and haven't pushed the changes elsewhere, you'll lose data.
So it appears everything is working as expected in this case and you can just ignore the .git directory and its contents, using the git command as normal.

Folder capitalization not changing on branch switch

I'm working on a python project and want to rename a (package) folder to small letters, let's say from Myackage to mypackage. As git is case-sensitive and Windows is not, I followed the solutions taken from here and espacially here.
My procedure was as follows:
git mv Mypackage tmp
git mv tmp mypackage
git commit -m "Change capitalization of package name"
This changes the folder Myackage to mypackage with success (for both, git and Windows). But if I switch to another branch, I expect the folder to change back to Mypackage (with capital letter!), as it was before. Background is, that all the imports of the package are also case-sensitve in python and i need this renamng acompanied with adaptions of the imports.
I've tried both, core.ignorecase set to true and false, but no matter what I try, if I checkout an older branch, the folder remains in form of small letters (mypackage) and I run into issues within python.
UPDATE:
I've set up a small example with only one Folder and one file and could succesfully change the capitalization of the folder. It also shows the desired behaviour, that upon branch switch the capitalization of the folder in Windows changes, yet still this won't work for my python project.
Could it be, that, e.g., submodules, play a role here?
UPDATE 2:
I've checked the case sensitivity attribute for both cases via:
fsutil.exe file queryCaseSensitiveInfo .
Both folders claim, that case-sensitivity is deactivated. Still for one project folder name capitalization changes, but for the other folder not.
The attribute case sensitivity is available on Windows 10 but after April 2018 Update and only affect the specific folder to which you apply it. It isn’t automatically inherited by that folder’s subfolders. However, if you use WSL to create folders it's enabled by default and available in that way to Windows. [1]
Although you can use the Git Unite [2] tool to match the case of the current folders with the git index.
If you use the rename approach, try using it with git commands like in "Rename files and folders with git"[3]
git mv foldername tempname && git mv tempname folderName
I found a way to reproduce your behavior :
if my CaSeD folder contains some extra files (untracked files for example), git will not change the case of my folder name when I jump between commits.
Is this the case in your setup ?
If this is your issue : you could go with a post-checkout hook, which forcibly renames the folders according to what is stored in HEAD after a checkout.
One way to get the full list of paths to directories from commit HEAD is :
git ls-tree --name-only -d -r HEAD
If you match this list with a similar list extracted from your local file system (ls -r ? find . -type d ? some python function from os.* ?), you can spot what folders need to be recapitalized.

How to automatically download files from github without copying the repository

I have a number of scripts that I use almost everyday in my work. I develop and maintain these on my personal laptop. I have a local git repository where I track the changes, and I have a repository on github to which I push my changes.
I do a lot of my work on a remote supercomputer, and I use my scripts there a lot. I would like to keep my remote /home/bin updated with my maintained scripts, but without cluttering the system with my repository.
My current solution does not feel ideal. I have added the following code belowto my .bashrc. Whenever I log in, my repository will be deleted, and I then clone my project from github. Then I copy the script files I want to my bin, and make them executable.
This sort of works, but it does not feel like an elegant solution. I would like to simply download the script files directly, without bothering with the git repository. I never edit my script files from the remote computer anyway, so I just want to get the files from github.
I was thinking that perhaps wget could work, but it did not feel very robust to include the urls to the raw file page at github; if I rename the file I suppose I have to update the code as well. At least my current solution is robust (as long as the github link does not change).
Code in my .bashrc:
REPDIR=mydir
if [ -d $REPDIR ]; then
rm -rf $REPDIR
echo "Old repository removed."
fi
cd $HOME
git clone https://github.com/user/myproject
cp $REPDIR/*.py $REPDIR/*.sh /home/user/bin/
chmod +x /home/user/bin/*
Based on Kent's solution, I have defined a function that updates my scripts. To avoid any issues with symlinks, I just unlink everything and relink. that might just be my paranoia, though....
function updatescripts() {
DIR=/home/user/scripts
CURR_DIR=$PWD
cd $DIR
git pull origin master
cd $CURR_DIR
for file in $DIR/*.py $DIR/*.sh; do
if [ -L $HOME/bin/$(basename $file) ]; then
unlink $HOME/bin/$(basename $file)
fi
ln -s $file $HOME/bin/$(basename $file)
done
}
on that remote machine, don't do rm then clone, keep the repository somewhere, just do pull. Since you said you will not change the files on that machine, there won't be conflicts.
For the scripts files. Don't do cp, instead, create symbolic links (ln -s) to your target directory.

Can't get Xcode / JUCE / .gitignore to work

I'm placing my .gitignore file in the same location as the *.jucer file and the .gitignore file contains:
./Builds/*
./JuceLibraryCode/*
As I only want to back up the source code folder only.
But whenever I go to commit, tons of of folders and items inside the Builds and JuceLibraryCode folder are pre checked. How Can I get Xcode to see this ignore file? Is it in the wrong location since the Projucer builds the folder hierarchy in a way that Xcode doesn't understand? I have it in the same location as my .git file.
Also, its worth noting that I recently allowed .git to back up everything, all of the files I'm currently trying to now ignore in those folders, but I did do a git rm --cached on all the files I'm trying to skip, as i read I needed to dump those before the .gitignore would be successful, but still, not luck. Any help would be greatly appreciated.
Remove git cache again.
$ git rm --cached -r .
Add this in .gitignore
Builds/
JuceLibraryCode/
See if git ignore the changes
$ git status

Ignoring directories in Git repositories on Windows

How can I ignore directories or folders in Git using msysgit on Windows?
Create a file named .gitignore in your project's directory. Ignore directories by entering the directory name into the file (with a slash appended):
dir_to_ignore/
More information is here.
By default, Windows Explorer will display .gitignore when in fact the file name is .gitignore.txt.
Git will not use .gitignore.txt
And you can't rename the file to .gitignore, because Windows Explorer thinks it's a file of type gitignore without a name.
Non command line solution:
You can rename a file to ".gitignore.", and it will create ".gitignore"
It seems that for ignoring files and directories there are two main ways:
.gitignore
Placing .gitignore file into the root of your repository besides the .git folder (in Windows, make sure you see the true file extension and then make .gitignore. (with the point at the end to make an empty file extension))
Making the global configuration ~/.gitignore_global and running git config --global core.excludesfile ~/.gitignore_global to add this to your Git configuration
Note: files tracked before can be untracked by running git rm --cached filename
Repository exclude - For local files that do not need to be shared, you just add the file pattern or directory to the file .git/info/exclude. Theses rules are not committed, so they are not seen by other users. More information is here.
To make exceptions in the list of ignored files, see this question.
To ignore an entire directory place a .gitignore of “*” there.
For example,
Example System
/root/
.gitignore
/dirA/
someFile1.txt
someFile2.txt
/dirB/
.gitignore
someFile3.txt
someFile4.txt
Goal
ignore the contents of dirB/
Top Level (/root/.gitignore)
You could just “dirB/“ here
Ignored Directory (/root/dirB/.gitignore)
Or you could “*” here
Git watches for gitignore at every step of the file system. So here I choose dirB/.gitignore as “*” to ignore dirB/, including all files and subdirs within.
Done ☺️
To instruct Git to ignore certain files or folders, you have to create .gitignore file.
But in Windows Explorer you have to provide a name for the file. You just cannot create file with just an extension. The trick is that create a empty text file and go to command prompt and change the name of the file to .gitignore:
ren "New Text Document.txt" .gitignore
Now open the file with your favorite text editor and add the file/folder names you wish you ignore. You can also use wildcards like this: *.txt.
I had some issues creating a file in Windows Explorer with a . at the beginning.
A workaround was to go into the commandshell and create a new file using "edit".
If you want to maintain a folder and not the files inside it, just put a ".gitignore" file in the folder with "*" as the content. This file will make Git ignore all content from the repository. But .gitignore will be included in your repository.
$ git add path/to/folder/.gitignore
If you add an empty folder, you receive this message (.gitignore is a hidden file)
The following paths are ignored by one of your .gitignore files:
path/to/folder/.gitignore
Use -f if you really want to add them.
fatal: no files added
So, use "-f" to force add:
$ git add path/to/folder/.gitignore -f
You can create the ".gitignore" file with the contents:
*
!.gitignore
It works for me.
In Windows there's an extra catch with slashes. Excluding a single directory in .gitignore with
dir_to_exclude/
will possibly work, but excluding all directories with
/
causes problems when you have file names with spaces (like my file.txt) in your directory: Git Bash escapes these spaces with a backslash (like my\ file.txt) and Git for Windows doesn't distinguish between / and \.
To exclude all directories, better use:
**/
Two consecutive asterisks signify directory contents.
Just in case you need to exclude sub folders you can use the ** wildcard to exclude any level of sub directory.
**/build/output/Debug/
Also in your \.git\info projects directory there is an exclude file that is effectively the same thing as .gitignore (I think). You can add files and directories to ignore in that.
When everything else fails try editing the file
/.git/info/exclude
and adding the directories you want to the end of the file, like this:
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
assets/
compiled/
I added the folders "assets" and "compiled" to the list of files and directories to ignore.
I've had some problems getting Git to pick up the .gitignore file on Windows. The $GIT_DIR/info/exclude file always seems to work though.
The downside of this approach, however, is that the files in the $GIT_DIR directory are not included in the check-in, and therefore not shared.
p.s. $GIT_DIR is usually the hidden folder named .git
On Unix:
touch .gitignore
On Windows:
echo > .gitignore
These commands executed in a terminal will create a .gitignore file in the current location.
Then just add information to this .gitignore file (using Notepad++ for example) which files or folders should be ignored. Save your changes. That's it :)
More information: .gitignore
I assume the problem is that your working tree is like:
a-cache/foo
a-cache/index.html
b-cache/bar
b-cache/foo
b-cache/index.html
.gitignore
... with the .gitignore you describe. This will give you git status output like:
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# .gitignore
# a-cache/
# b-cache/
... if the index.html files have not yet been added to the repository. (Git sees that there are unignored files in the cache directories, but it only reports the directories.) To fix this, make sure that you have added and committed the index.html files:
git add *cache/index.html
git commit -m "Adding index.html files to the cache directories"
... and your git status will then look like:
$ git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# .gitignore
nothing added to commit but untracked files present (use "git add" to track)
(Obviously you do want to commit .gitignore as well. I was just being lazy with this test case.)
On Windows and Mac, if you want to ignore a folder named Flower_Data_Folder in the current directory, you can do:
echo Flower_Data_Folder >> .gitignore
If it's a file named data.txt:
echo data.txt >> .gitignore
If it's a path like "Data/passwords.txt"
echo "Data/passwords.txt" >> .gitignore.
I had similar issues. I work on a Windows tool chain with a shared repository with Linux guys, and they happily create files with the same (except for case) names in a given folder.
The effect is that I can clone the repository and immediately have dozens of 'modified' files that, if I checked in, would create havoc.
I have Windows set to case sensitive and Git to not ignore case, but it still fails (in the Win32 API calls apparently).
If I gitignore the files then I have to remember to not track the .gitignore file.
But I found a good answer here:
http://archive.robwilkerson.org/2010/03/02/git-tip-ignore-changes-to-tracked-files/index.html
Just create .gitignore file in your project folder Then add the name of the folder in it for ex:
frontend/node_modules
This might be extremely obvious for some, but I did understand this from the other answers.
Making a .gitignore file in a directory does nothing by itself. You have to open the .gitignore as a text file and write the files/directories you want it to ignore, each on its own line.
so cd to the Git repository directory
touch .gitignore
nano .gitignore
and then write the names of the files and or directories that you want to be ignored and their extensions if relevant.
Also, .gitignore is a hidden file on some OS (Macs for example) so you need ls -a to see it, not just ls.
Temporarily ignore a directory/file that was already in git:
I have a lot of projects in a multi-project gradle project and they can take a long time to delete them, and they're all pretty much the same but different. From time to time I want to remove those from the gradle build by deleting them altogether. git can get them back after all. However I don't want them showing up in git status either. So I use the following simple procedure;
delete files and folders I don't want.
verify build still works
tell git to ignore the deleted files for a bit (we can get them back)
git ls-files --deleted -z | git update-index --assume-unchanged -z
--stdin
go about life without the dirs until you want them back. Then run the same command as before but switch out assume-unchanged for no-assume-unchanged
git ls-files --deleted -z | git update-index --no-assume-unchanged -z
--stdin

Resources