Using sed to scrub a file for environment configs - bash

Currently, my .zshrc file maintains not only a laundry list of aliases i like to keep track, but also a few secrets/tokens that get sourced whenever i start up a new terminal, e.g
# Artifactory Config
export CREDENTIALS_USR=***
export CREDENTIALS_PSW=***
I am working on setting up a remote repository where I can store my .zshrc, mainly so that I can keep my aliases checked into source control, so if my laptop were to break, that's one less thing I'd have to start over with when i get a new computer.
I originally had an alias, cpzsh, which did the following:
alias cpzsh='cp ~/.zshrc .'
and when i execute it from the root of my git repo, copies my zshconfig to the current directory (in this case my local git repo). After this I need to manually go in and scrub my secrets prior to pushing to my remote repo.
This is really easy to screw up if I were to forget, so I tried putting together a better cpzsh command using sed that could edit my file inline, and then pipe the scrubbed .zshrc to my local repo:
alias cpzsh='sed "/CREDENTIALS_/d" ~/.zshrc > .zshrc'
However, this led to some adverse effects, because the above line also got scrubbed from my .zshrc! This is because it's traversing the file for the substring CREDENTIALS_ in a line, which it deletes. So it not only deletes my artifactory credentials (which it should do), but it also deletes the line that contains the substring to delete (which it should not do!)
I am aware that sed also allows you to hard-code the lines to delete, but I'd rather not run the risk of assigning my secrets in certain lines, only for those lines to suddenly change for whatever reason and i unsuspectingly push my secrets to github.
Is there a better way to accomplish this?

How about you scrub the values but leave the export CREDENTIALS_... lines in the file?
alias cpzsh="sed 's/\(CREDENTIALS_[^=]*=\).*/\1REDACTED/;' ~/.zschrc ."

Related

Create a Git Alias specific to one project

I have a Python project that is compiled using pyinstaller. It usually uses a bunch of concatenated operations && to not only compile, but perform several other things. Ok, just a big command.
I can use an alias with git bash to reduce it to a single word, fine. But I want to know, if exists some way to distribute some bash file with that alias with my project.
I mean, in my repo, having something like alias.sh that contains the alias I want, but they just exists inside that repo, so whenever I open a terminal inside that repo, I already have present those alias. And if someone forks or clones my project, they already have that alias too thanks to that specific file.
No, it's not possible.
But you can create a script file alias.sh to set up the alias and instruct users of your repo via README file to source this file once in their terminal session when they need this alias: . ./alias.sh
The file alias.sh might contain:
#!/bin/sh
alias youralias='your | long | command'
After the file was sourced with . ./alias.sh, you can simply run youralias as a standalone command.

Git - rename files with illegal characters

Our team works on repository which includes one directory where there are files with pipe character "|". I'm the only one on Windows so pipe character is illegal for files names.
Is there a way to rename files in the directory "extra" from ex. "2021|08|05" to "2021\08\05" when I make "git pull"?
Is there a way to rename files in the directory "extra" from ex. "2021\08\05" to "2021|08|05" when I make "git push"?
No.
No.
These are the right answers to the question you've asked, but the trick is, you've asked the wrong question. Alas, the answer to the right question is: "yes, but it's a horrible solution". The question is: Is there a solution to the problem of bad / invalid characters in file names stored in Git commits?
In the early days of Git, when it was a collection of shell scripts that only a few people could use successfully, 😀 one would obtain new commits from elsewhere with git fetch, then read these commits into Git's index with git read-tree.
Git's index, which Git also calls the staging area or sometimes the cache, can hold these file names. In fact, even on Windows, Git's index can hold files named aux.h, which Windows won't let you create. The index has no folders either: it just has files with names with embedded (forward) slashes, such as path/to/file. Git can hold two different files, one named README and one named readme, in its index. WIndows can't have two different files whose name only differs in case.
So, Git's index / staging-area can hold these files just fine. The problem comes when you go to work with the files. Files that are in Git's index, are stored there in a special Git-only format, as what Git calls a blob object. You cannot read or write a blob object directly. You have to use yet more Git commands to do this. It's terribly inconvenient.
To use Git conveniently, then, we normally don't use all the individual one-step-at-a-time internal Git operations: we use some sort of higher level, user oriented command, like git checkout. We check out an entire commit: Git will find all the files that are stored in that commit, read them into Git's index, and copy out and expand all the internal Git-only blob objects into ordinary files, with ordinary file names.
This step—copying files out of Git's index, to make them usable—is where things go wrong, on Windows. The file's name in Git's index is, say, path/to/2021|08|05. Git recognizes that path/to/ has to be turned into two folders, path\ and path\to\, on Windows, so that Git can create a a file in the second folder. Unfortunately, Git has no way to remap the 2021|08|05 part. That part is going to stay 2021|08|05, and as you have seen, Git can't create a file with that name: the OS just says "no".
What you can do, at this point, is drop down to those lower-level commands. You can run:
git rev-parse :path/to/2021|08|05
perhaps with quotes if needed, depending on your shell:
git rev-parse ":path/to/2021|08|05"
This git rev-parse command will show the blob hash ID for the file. You can then access the file's contents with:
git cat-file -p <hash>
which prints those contents to the standard output. If your shell supports redirection, you can then redirect the output to a file whose name is your choice. This lets you see and use the file's contents.
The git cat-file -p command can take the index path name directly, so:
git cat-file -p ":path/to/2021|08|05" > path/to/2021-08-05
is a way to extract the file to a usable name.
Unfortunately, git add—which is how you would normally update the file—will insist on using the name you gave the file in the file system. Once again, you must fall back on internal Git plumbing commands to work around this. If you need to update that particular file, for instance, you would:
run git hash-object -w -t blob path/to/2021-08-05 to turn the updated file's data into an internal Git object;
run git update-index with arguments to make Git update the entry for path/to/2021|08|05 using the hash ID obtained in step 1.
Once all of this is done, you can go back to normal Git commands, because git commit makes a new commit from what's in Git's index / staging-area.
The (rather large) drawback here is that you cannot use a lot of normal everyday Git commands:
git pull is often a no-go because it runs git rebase or git merge, both of which need to use your working tree (OS-level files). Run git fetch first, then do as much manual work as needed.
git checkout will fail: you can use it, but then you must manually do something about each of the bad file names that are now in Git's index.
git diff will show differences that include deleting the files with the bad names, and git status will show the adjusted-name files as untracked files (because they are).
git add of any changes you need to make to these files is also a no-go; use git hash-object -w and git update-index instead.
git rebase and git merge become difficult. You probably can deal with them as in steps 2 and 4, but that's painful at best.

Git smudge and clean using local configuration branch

The local configuration of the project I'm working on involves changing several files in complicated ways that cannot be committed to any submitted branches. To work around this I've committed these local configuration changes to a dedicated local branch config, and have been running this bash script config.sh after starting a new work branch:
#!/bin/bash
# put relevant config files in array
mapfile -t files < <(git diff config develop --name-only)
# overwrite only those files to my working directory
git checkout config -- ${files[#]}
# unstage them so they aren't accidentally committed
git reset HEAD ${files[#]}
echo The following files were successfully overwritten for local configuration:
printf '\t%s\n' "${files[#]}"
Along with another .deconfig script that does the same in reverse. Run directly from the terminal, these scripts have been working fine, but I'd like to streamline the process further using git's clean and smudge filters. So I created a .gitattributes file:
*.* filter=config
and then added this to my .git/config file:
[filter "config"]
smudge = ./config.sh
clean = ./deconfig.sh
However, it just isn't working. If I had to guess it's because git isn't expecting me to run an additional checkout as part of a filter, which itself runs after the checkout command against all files. Most use cases for smudge and clean seem to involve simple find and replace operations, but that approach would be complicated to implement and difficult to maintain given the complexity of changes needed. I could store the configuration files in a static, external directory somewhere, but I'd like to smudge and clean based off the same configuration branch because the local configuration itself frequently evolves and benefits from versioning alongside the rest of the project, and ideally the branch could be used as a baseline for other devs for their local configuration. Git's filter-branch might be a better fit but git's own documentation recommends against using it at all. Is there a way to do this? Is there something wrong with my git configuration? Could the script itself be causing a problem? Any other possible approaches?
Although it is not documented anywhere, you cannot change the state of the working tree with a smudge or clean filter. Git expects to invoke the filter once for each file by piping data into it and reading the data from the standard output. In other words, these filters are intended to be invoked on a per-file basis and process only that file, not by modifying the working tree state.
The best solution to your problem is to avoid keeping a separate branch. Simply keep all of the files, both development and production, in some directory, and use a script to copy the correct one into place. The location of the running config file should be ignored, so the script won't cause Git to show anything as modified. Alternatively, keep a template somewhere, and have the script generate the appropriate one based on the environment. This is good if you have secrets for production that should not be checked in; you can pass them to the script through the environment and have the right values generated.
What you're doing is related to ignoring tracked files, which, as outlined in the Git FAQ, generally can't be done successfully.

Git on Windows capitalized file names on origin, lower case locally

We are forced to work on Windows at work, and I have lets say problem, strange situation. We have github repository, inside which we have one directory with name Something (with capitalized first letter 'S'), but in my local I see this directory with name something (note lower case 's'), git status shows that working directory is clean, even if I change this directory locally to, for example SoMeThInG git says that nothing changed. I suspect that Windows is here a problem, as it is case insensitive. Is there possibility to change this directory name from Windows level? Or maybe how to force git bash to be case sensitive?
Update
I've changed that files from mine virtual fedora, but this is just a workaround, the question remains unanswered, how to do it properly on Windows?
On case-insensitive file systems, Git will not detect changes just in casing. However, when committing files, the actual casing is still being reflected in the way it was added to the index.
So a git add file and git add FILE will both work for a file that is named file in any kind of casing (e.g. FiLe or fIlE), but each command will actually stage that exact name into the repository. So git add file will make the name be case-sensitive file and git add FILE will make the name case-sensitive FILE.
That’s why you should try to always use your command line auto completion for file names, so you don’t accidentally add files with a different casing than they actually are. Or use commands that stage the files automatically, e.g. git add ., since that will also use the actual casing.
However, since Git will not detect casing changes, once a file has been added with a particular casing, that casing will be used until you explicitly change it. That’s why it’s possible to have files in a folder src/readme.md and SRC/license.txt that are both physically in the same location on your file system, but are represented using incompatible paths inside of Git. So you should be careful here.
That all being said, you can fix the casing later. But to do that, you need to make the change using Git instead of the file system, as Git is case sensitive while the file system isn’t. So commands like git mv will work. Same as a combination of git rm --cached and git add.
For example, to fix the above situation of the src/SRC directory, one could do (assuming the correct name of the folder should be Src):
git mv src/readme.md Src/readme.md
# or
git rm --cached SRC/license.txt
git add Src/license.txt
You can also fix the casing for every file by removing everything from the index, and then adding it back:
git rm --cached -r .
git add .
That should stage all renames to the correct file casing.

Programmatically overwrite a specific local file with remote file on every git pull

I have an XML file that we consider binary in git. This file is externally modified and committed.
I don't care about who edited it and what's new in the file. I just want to have the latest file version at every pull. At this time, at every git pull I have a merge conflict.
I just want that this file is overwritten on every git pull, without manually doing stuff like git fetch/checkout/reset every time I have to sync my repo.
Careful: I want to overwrite just that file, not every file.
Thanks
I thought you could use Git Hooks, but I don't see one running before a pull...
A possible workaround would be to make a script to delete this file and chain with the needed git pull...
This answer shows how to always select the local version for conflicted merges on a specific file. However, midway through the answer, the author describes also how to always use the remote version.
Essentially, you have to use git attributes to specify a specific merge driver for that specific file, with:
echo binaryfile.xml merge=keepTheirs > dir/with/binary/file/.gitattributes
git config merge.keepTheirs.name "always keep their file during merge"
git config merge.keepTheirs.driver "keepTheirs.sh %O %A %B"
git add -A
git commit -m "commit file for git attributes"
and then create keepTheirs.sh in your $PATH:
cp -f "$3" "$2"
exit 0
Please refer to that answer for a detailed explanation.
If the changes to your files are not actual changes, you should not submit them. This will clutter your version history and cause numerous problems.
From your statement I’m not quite sure which is the case, but there are 2 possibilities:
The file in question is a local storage file, the contents of which are not relevant for your actual sourcecode. In this case the file should be part of your .gitignore.
This file is actually part of your source and will thus have relevant changes in the future. By setting up the merge settings like you are planning to do, you will cause trouble once this file actually changes. Because merges will then be destructive.
In this case the solution is a little bit more complicated (apart from getting a fix for the crappy tool that changes stuff it doesn’t actually change …). What you are probably looking for is the assume unchanged functionality of git. You can access it with this command:
git update-index --assume-unchanged <file>
git docu (git help update-index):
You can set "assume unchanged" bit to
paths you have not changed to cause git not to do this check. Note that setting this bit on a path does not mean git will check the
contents of the file to see if it has changed — it makes git to omit any checking and assume it has not changed. When you make changes
to working tree files, you have to explicitly tell git about it by dropping "assume unchanged" bit, either before or after you modify
them.

Resources