Images corrupt after git push - image

I added some .png images to my git repo and then pushed them to my remote repository. I noticed that the images are in the remote repository, but they are broken.
In order to verify this, I compared the binary data for the images on my local and my remote repos. I also ran the file command on images in both repos.
Image on local repository
First 10 Lines of Binary data:
8950 4e47 0d0a 1a0a 0000 000d 4948 4452
0000 0019 0000 0014 0806 0000 0078 7796
bd00 0000 1974 4558 7453 6f66 7477 6172
6500 4164 6f62 6520 496d 6167 6552 6561
6479 71c9 653c 0000 0373 6954 5874 584d
4c3a 636f 6d2e 6164 6f62 652e 786d 7000
0000 0000 3c3f 7870 6163 6b65 7420 6265
6769 6e3d 22ef bbbf 2220 6964 3d22 5735
4d30 4d70 4365 6869 487a 7265 537a 4e54
637a 6b63 3964 223f 3e20 3c78 3a78 6d70
Result of file command:
bundles/admin/public/images/messages-icon.png: PNG image data, 24 x 16, 8-bit/color RGBA, non-interlaced
Image pushed to remote repository
First 10 Lines of Binary data:
8950 4e47 0a1a 0a00 0000 0d49 4844 5200
0000 1900 0000 1408 0600 0000 7877 96bd
0000 0019 7445 5874 536f 6674 7761 7265
0041 646f 6265 2049 6d61 6765 5265 6164
7971 c965 3c00 0003 7369 5458 7458 4d4c
3a63 6f6d 2e61 646f 6265 2e78 6d70 0000
0000 003c 3f78 7061 636b 6574 2062 6567
696e 3d22 efbb bf22 2069 643d 2257 354d
304d 7043 6568 6948 7a72 6553 7a4e 5463
7a6b 6339 6422 3f3e 203c 783a 786d 706d
Result of file command:
bundles/admin/public/images/notifications-icon.png: data
Why would this be happening? I searched all over google and this site for answers but have found none.

I had this issue too - here's a generic .gitattributes that looks generally safe for php web repos that worked for me: https://github.com/Danimoth/gitattributes/blob/master/Web.gitattributes
running git check-attr --all -- path/to/png before and after adding the new gitattributes showed that the image file was originally treated as a text file, and now it's binary (note I had to commit/push the .gitattributes file for it to take effect on remote)
In case the link goes dead:
# These settings are for any web project
# Handle line endings automatically for files detected as text
# and leave all files detected as binary untouched.
# * text=auto
# NOTE - originally I had the above line un-commented. it caused me a lot of grief related to line endings because I was dealing with WordPress plugins and the website changing line endings out if a user modified a plugin through the web interface. commenting this line out seems to have alleviated the git chaos where simply switching to a branch caused it to believe 500 files were modified.
#
# The above will handle all files NOT found below
#
#
## These files are text and should be normalized (Convert crlf => lf)
#
# source code
*.php text
*.css text
*.sass text
*.scss text
*.less text
*.styl text
*.js text
*.coffee text
*.json text
*.htm text
*.html text
*.xml text
*.svg text
*.txt text
*.ini text
*.inc text
*.pl text
*.rb text
*.py text
*.scm text
*.sql text
*.sh text
*.bat text
# templates
*.ejs text
*.hbt text
*.jade text
*.haml text
*.hbs text
*.dot text
*.tmpl text
*.phtml text
# server config
.htaccess text
# git config
.gitattributes text
.gitignore text
.gitconfig text
# code analysis config
.jshintrc text
.jscsrc text
.jshintignore text
.csslintrc text
# misc config
*.yaml text
*.yml text
.editorconfig text
# build config
*.npmignore text
*.bowerrc text
# Heroku
Procfile text
.slugignore text
# Documentation
*.md text
LICENSE text
AUTHORS text
#
## These files are binary and should be left untouched
#
# (binary is a macro for -text -diff)
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.ico binary
*.mov binary
*.mp4 binary
*.mp3 binary
*.flv binary
*.fla binary
*.swf binary
*.gz binary
*.zip binary
*.7z binary
*.ttf binary
*.eot binary
*.woff binary
*.pyc binary
*.pdf binary

*.png binary
in .gitattribute worked after images were reuploaded.
You can add all images by executing git add *.png --force

Related

Jenkins job adding DOS newlines to shell scripts, causing a "bad interpreter" error on UNIX

I am trying to run the following shell script but I am getting the "/bin/bash^M: bad interpreter" error.
Script:
#!/bin/bash
rm -rf tomcat8/webapps/cpproject*
I know that this is happening because I am editing the file on windows and trying to run it on Unix. I tried all the ways of fixing it as suggested on stackoverflow. I tried the following.
Installed Vim editor and saved it
In Notepad++ changed the End Of Line option to Unix. Tried to save the new documents as Unix format
Installed gedit and saved it.
Installed doc2unix and tried it.
I tried a lot other ways but nothing worked. Can some one suggest me how to fix this issue?
Updated:
"xxd before_install1.sh | head" command is showing the followng.
00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170 rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a ps/cpproject*
Update:
I made sure that I am not running the old file. Because I changed the file name each time and tried it. I made sure that the new file is being executed each time.
Basically I am using codedeploy jenkins plugin which generates the war file and zips it with scripts and appspec.yml file. Since I change the file name each time and update the appspec.yml file with new name, I am making sure that the new file is being uploaded. But it is giving the same error.
Even after it is uploaded to the S3 bucket I downloaded and the zip file and made sure that it contains the latest one and even executed the xxd command and got the following.
$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20 #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261 -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a pps/cpproject*..
Update:
#Charles Duffy, Thank you for your response. I am developing these scripts locally and pushing them to GitHub. I have a Jenkins job, which takes this code from GitHub repository and packages this into a war file. I am using "CodeDeploy" plugin, which will take this war file and zips it with these shell scripts under scripts folder and appspec.yml file and push it to S3 bucket in AWS. The following is what is happening.
xxd command in my local system file:
$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170 rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a0a ps/cpproject*.
xxd command on the file after it is uploaded to GitHub:
$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170 rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a0a ps/cpproject*.
xxd command on the file after it is pulled by Jenkins from GitHub to Local Workspace(in .jenkins/worspace folder)
$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20 #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261 -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a pps/cpproject*..
xxd command on the file after it is pushed to s3 bucket in AWS:
$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20 #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261 -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a pps/cpproject*..
I am seeing that the extra characters '0d0a' are added after the code is taken by Jenkins into Local Workspace. It's not a problem with CodeDeploy plugin at this momement, it seems. Is there any way we can avoid this?
Thank you.
If the script is called 'myscript.dos', you can delete the carriage returns with tr:
tr -d '\015' < myscript.dos > myscript.unix
#Charles Duffy, Thank you for your time and patience in resolving this issue. It is finally working. I am just posting the actual problem and solution here so that it will be useful for other.
Work:
I am working on developing a deployment strategy into AWS using Jenkins and AWS CodeDeploy. I am using CodeDeploy plugin in Jenkins. Jenkins will create a war file and CodeDeploy plugin will create a zip file with the war file, appspec.yml and some shell scripts to stop, start servers etc and this zip file will be pushed to S3 and from there CodeDeploy will be triggered and deploy the war file.
Problem:
The shell scripts used to start the server or stop the server are developed in windows system. So it is adding DOS lines to the scripts so they are not running on Unix in EC2 machines. I have removed the DOS lines on the shell scripts by using different techniques. But since Jenkins is running locally (On Windows Machine), when the code is pulled from GutHub it is again adding DOS lines.
Solution: Please check the above questions and the solutions given by all to find out how to identify if DOS line are being added to shell scripts. Since, in my case, the DOS lines are added by Jenkins I have to clear them in Jenkins before it is going to produce a zip file.
Go to Jenkins and configure the shell executable first. Since I have git installed on my system, I have a Shell executable in my system.
Once this is done, go to the Jenkins job created and add "Execute Shell" build step and add the commands shown in the below picture.
This will remove all the DOS lines added to the scripts and your shell scripts will be executed correctly by codedeploy in AWS on Linux system.
Thank you #Charles Duffy and others who helped me in resolving these issues.
Thank you very much,
Subbu.

SoftQuad DESC or font file binary

I read this question but it doesn't helped me. I am solving a challenge where I have two files, first one was .png which gave me upper half part of an image, second file is SoftQuad DESC or font file binary I am sure that this file should somehow convert into .png file to complete the image. I googled and got hint about magic bytes but I am unable to match the bytes.
These are the first two rows of output of xxd command
00000000: aaaa a6bb 67bb bf18 dd94 15e6 252c 0a2f ....g.......%,./
00000010: fe14 d943 e8b5 6ad5 2264 1632 646e debc ...C..j."d.2dn..
These are the last two rows of output of xxd command
00001c10: 7a05 7f4c 3600 0000 0049 454e 44ae 4260 z..L6....IEND.B`
00001c20: 82
.

How to extract contents from 'Payload' file in a apple macOS update package?

I am extracting macOS sierra update package - macOSUpd10.12.1.pkg using following command to /tmp/macosupd directory.
pkgutil --expand /Volumes/macOS\ Sierra\ Update/macOSUpd10.12.1.pkg /tmp/macosupd
I can see following packages are bundled inside the metapackage.
-rwxr-xr-x 1 Distribution
drwxr-xr-x 6 FirmwareUpdate.pkg
drwxr-xr-x 5 FullBundleUpdate.pkg
drwxr-xr-x 36 Resources
drwxr-xr-x 6 macOSUpd10.12.1.pkg
I am not able to extract contents of Payload file. For example when i try to extract the Payload of macOSUpd10.12.1.pkg, i get following error message:
with tar:
tar -xvf macOSUpd10.12.1.pkg/Payload
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
with xar:
xar -xvf macOSUpd10.12.1.pkg/Payload
Error opening xar archive: macOSUpd10.12.1.pkg/Payload
When I run file command on the payload file, i get a message as:
file macOSUpd10.12.1.pkg/Payload
macOSUpd10.12.1.pkg/Payload: data
By seeing initial bits of Payload file using xxd command , i can see file type looks like pbzx
xxd macOSUpd10.12.1.pkg/Payload
00000000: 7062 7a78 0000 0000 0100 0000 0000 0000 pbzx............
00000010: 0100 0000 0000 0000 0031 0330 fd37 7a58 .........1.0.7zX
00000020: 5a00 0000 ff12 d941 04c0 f385 c401 8080 Z......A........
00000030: 8008 2101 1600 0000 506a 84e2 e3b8 13ef ..!.....Pj......
00000040: fe5d 0018 0ddf 07a4 347c 7c50 9853 8031 .]......4||P.S.1
00000050: 2d14 f703 6903 cf69 f214 76b0 93c0 a4c9 -...i..i..v.....
00000060: 774d 6fb0 8b3a 2257 4a55 04ad 289b cc4d wMo..:"WJU..(..M
00000070: b835 5db7 7e72 f7a8 dc15 7a9c 7755 800c .5].~r....z.wU..
00000080: 6060 d45f e078 f84f e537 4319 2d89 f72e ``._.x.O.7C.-...
00000090: 60c4 cdb0 6b54 9326 9321 3339 4a4f 1e75 `...kT.&.!39JO.u
000000a0: 8eb7 991d 8968 5e6f 45d4 24c9 e364 712b .....h^oE.$..dq+
000000b0: ef4e 9abc af70 e97a e5a3 1810 7f05 54df .N...p.z......T.
000000c0: 08d2 3060 9f8a a1e0 edb8 2b10 df23 789d ..0`......+..#x.
000000d0: 3e52 ee3e d6f0 468a bfee 3366 d39e 28db >R.>..F...3f..(.
Please suggest a way on how to extract the contents of Payload file. I have tried all he methods suggested in link1 and link2, including the method suggested in this external-link still no success. Seems like apple updates prior to Mavericks were pure xar and tar packages. With Mavericks and above apple changed the file format from a tar to pbzx(lzma).
please suggest an easy command line way to extract contents of the Payload file as i have to make use in my automation setup.
There is an undocumented option to pkgutil, --expand-full, which uses the same syntax as pkgutil --expand.
pkgutil --expand-full [pkg] [dir]
This will completely "unarchive" a pkg or mpkg file.
Since OSX 10.10, the Payload in the .pkg files is encoded as pbzx (which is in turn lzma compressed). It can no longer be extracted using gunzip. Check out the pbzx tool (a fork of the original which would not allow you to extract the Payload directly but only by passing the .pkg file directly).
Instead of
tar -xvf macOSUpd10.12.1.pkg/Payload
you can write
pbzx -n macOSUpd10.12.1.pkg/Payload | cpio -i

How to set binary-flag in Zip-file with rubyzip?

I received a complain about a zip file I created with rubyzip 1.2.0:
The text flag for text.pdf corrupted the pdf after unpacking.
I create the zip under windows, the recipient works with Unix/Linux.
You can see the flag with zipinfo (or unzip -Z with windows).
unzip -Z -l test.zip results with:
Archive: test.zip
Zip file size: 1666 bytes, number of entries: 2
-rw---- 5.2 fat 1521 t- 730 defN 16-Sep-30 23:37 test.dtx
-rw---- 5.2 fat 1521 t- 730 defN 16-Sep-30 23:37 test.pdf
2 files, 3042 bytes uncompressed, 1460 bytes compressed: 52.0%
+
+--- This is the t-flag
This is the related testcode:
require "zip"
###add here my patch####
Zip::File.open('test.zip', Zip::File::CREATE) do |zipfile|
%w{
test.dtx
test.pdf
}.each{|filename|
file = begin
#I use __FILE__, so you can run this example out of the box.
#My original code uses the dtx and pdf file.
zipfile.add(filename, __FILE__) #Zip::ZipEntryExistsError if already in zip
rescue Zip::EntryExistsError
zipfile.replace(filename, __FILE__)
end
#This is part of my work around
file.internal_file_attributes = 0 if filename =~ /pdf$/ and file.respond_to?(:internal_file_attributes)
}
end #Zip::ZipFile.open
I checked the source code of Zip::Entry and found the definition of
#internal_file_attributes = 1, but no possibility to influence this value.
Based on this information, I made a patch Zip::Entry like this:
module Zip
class Entry
#my work around to set #internal_file_attributes
attr_accessor :internal_file_attributes
end
end
When I add this patch in my test code above, then I get:
Archive: test.zip
Zip file size: 1726 bytes, number of entries: 2
-rw---- 5.2 fat 1666 t- 760 defN 16-Sep-30 23:40 test.dtx
-rw---- 5.2 fat 1666 b- 760 defN 16-Sep-30 23:40 test.pdf
2 files, 3332 bytes uncompressed, 1520 bytes compressed: 54.4%
+
+--- There is a b-flag for the pdf
So I found a work around for my problem - but is there another and better solution?
Is there a possibility to set the binary flag during adding the file?

Strange character for empty line in TextWrangler and cat -v

I have a text file, which on my Mac I open with TextWrangler. I enable the invisible characters to see the line endings. I see that every empty line has a red, upside down question mark in it. Which character is this?
When in the terminal I type cat -v file.txt, it shows these characters as ^# (and the line endings themselves as ^M). What I need to know is the regex of that specific character, like /n for the end of line.
In the hex dump, I see the following:
0000000: 312e 300d 0a00 0d0a 2231 3130 3030 3030 1.0....."1100000
0000010: 3030 3222 3b22 3922 3b22 5354 4422 3b3b 002";"9";"STD";;
0000020: 3b0d 0a22 3131 3030 3030 3030 3639 223b ;.."1100000069";
If I manually remove the strange characters, and make a new hex dump, I see:
0000000: 312e 300d 0a0d 0a22 3131 3030 3030 3030 1.0...."11000000
0000010: 3032 223b 2239 223b 2253 5444 223b 3b3b 02";"9";"STD";;;
0000020: 0d0a 2231 3130 3030 3030 3036 3922 3b22 .."1100000069";"
The difference is a byte sequence 00. Is there an encoding in which this 00 is required for empty lines?
The red inverted question mark, you are looking at, is apparently a NULL / NUL character. Whether or not it makes any difference does depend on the application writing/reading the files in question. (So, it's most likely not a general encoding issue of sorts. Compare: Wikipedia.)
Once you made the hidden characters visible in TextWrangler, you can mark that/any character (or character sequence for that matter), and copy it to the Find input field using CMD + E. The NULL character shows up as \x{00} on my machine.
Alternatively, you might use -> Text -> Zap Gremlins... with (at least) Null (ASCII 0) characters checked, Replace with code selected, and were told \x00. Either one of these should work when searching for these characters - no matter whether grep is enabled or not. Not sure, though, whether \s should actually find it as well in grep mode - it does not on my machine. But \W does grep it.
Please comment, if and as this requires adjustment / further detail.

Resources