I have a bunch of files in s3 that show in the console as ending in "%0A". I'd like to programmatically delete these files.
When I use this method and set the key with suffix "%0A", I get no error but the file is not deleted.
I don't know how it looked when the file was initially written.
S3 Console should be able to delete it no matter what characters it contains or how broken it is (if we're talking about same thing). Do you have enough permissions?
When doing it other ways, first you need to figure out if "%0A" is actually part of the name or is a url encoded LF character ("\n"). Keep in mind that you might not see other non-printable characters as well. The best way to determine this will be to open it in Amazon Console, choose S3 in Service menu, find your Bucket and click on your file, note the object's Link.
If "%0A" is part of the name, you will see object name as "foo%0A", but in the link it will look like "foo%250A". In this case, s3 rm s3://BUCKET/foo%0A should do the trick.
If "%0A" denotes a url encoded "\n" character, then you should see object name (just above tabs) as just "foo", but will see "foo%0A" as object name in the url. In ths case you need to embed new line into the key when issuing s3 rm command. This depends on you operating system. E.g. on linux/unix you will enter it like this:
aws-cli s3 rm 's3://BUCKET/foo
'
Note the quotes and that the closing quote is at the begining of a new line. This will make new line part of the value passed as the command line parameter.
Also note that aws-cli might be called just aws on some system, but given you provided s3 rm command as an example I assume you already know what it is on your system. And if using PowerShell on Windows, use backtick in place of quotes.
Now, check the object's Link and see what other characters are there. Looking at your examples and issues you report, I suspect there is more then one in your file names. Important point here is that your object name might include more non-printable characters and you won't see then in S3 console, but Link will include them all in URL encoded form. So, you can extract actual exact name from the link. Note that both aws-cli and programmatic interface you linked take actual name, not url encoded one. So, if you use aws-cli, you will need to embed all non-printable characters into command line parameter. And if you're using programmatic interface you'll need to embed them in the key string you supply to the call (e.g. putting "\n" instead of new line). Though I consider this part pretty obvious. And believe you'll know what to do once you see your actual file name with all the non-printables in it.
Try this
aws s3 rm "bucket_name"$'\r' --recursive
Worked for me.
Related
The following script works, but I need help with one change. Right now using lday serach, and a little utility called GETPASS, this script will pull a listy users, from a specified context out of an LDAP directory, compare their LDAP password, to an unused attribute, If different, add it to the unused attribute called carLicense, and then send it in SHA 1 format, up our google APPS domain, with Google Apps directory sync. The one problem, is that it when it returns the ldapsearch, it works fine with a user in this format:
cn=joebloe,ou=googletest,o=someorg
However, if the usename has a space in it like this:
cn=joe bloe,ou=googletest,o=someorg
the script will fail as it does not know how to deal with that space. In this situations, a single quote, around the DN will solve the issue IE:
'cn=joe blow,ou=googletest,o=someorg'
However, I have tried to alter the script to address this need of mine, and I am failing miserably, please see a copy of the script in Pasetebin below. Any help with be deeply and GREATLY appreciated, as this is currently my only stumbling block to success.
http://pastebin.com/htWxsNXj
Replace the first line of generating result file (in your pasting is line 66) with:
echo "dn: '$RESULT'" >> $SCRIPTPATH/gadspwsync.ldif
If it won't help, don't remove the file in line 75. Check the content, show us the first line, then show the expected content.
We're creating an app that is going to generate some text files on *nix systems with hashed filenames to avoid too-long filenames.
However, it would be nice to tag the files with some metadata that gives a better clue as to what their content is.
Hence my question. Does anyone have any experience with creating files with custom metadata in Ruby?
I've done some searching and there seem to be some (very old) gems that read metadata:
https://github.com/kig/metadata
http://oai.rubyforge.org/
I also found: system file, read write o create custom metadata or attributes extended which seems to suggest that what I need may be at the system level, but dropping down there feels dirty and scary.
Anyone know of libraries that could achieve this? How would one create custom metadata for files generated by Ruby?
A very old but interesting question with no answers!
In order for a file to contain metadata, it has to have a format that has some way (implicitly or explicitly) to describe where and how the metadata is stored.
This can be done by the format, such as having a header that says where the "main" data is stored and where the "metadata" is stored, or perhaps implicitly, such as having a length to the "main" data, and storing metadata as anything beyond the "main" data.
This can also be done by the OS/filesystem by storing information along with the files, such as permission info, modtime, user, and more comprehensive file information like "icon" as you would find with iOS/Windows.
(Note that I am using "quotes" around "main" and "metadata" because the reality is that it's all data, and needs to be stored in some way that tools can retrieve it)
A true text file does not contain any headers or any such file format, and is essentially just a continuous block of characters (disregarding how the OS may store it). This also means that it can be generally opened by any text editor, which will merely read and display all the characters it finds.
So the answer in some sense is that you can't, at least not on a true text file that is truly portable to multiple OS.
A few thoughts on how to get around this:
Use binary at the end of the text file with hope/requirements that their text editor will ignore non-ascii.
Store it in the OS metadata for the file and make it OS specific (such as storing it in the "comments" section that an OS may have for a file.
Store it in a separate file that goes "along with" the file (i.e., file.txt and file.meta) and hope that they keep the files together.
Store it in a separate file and zip the text and the meta file together and have your tool be zip aware.
Come up with a new file format that is not just text but has a text section (though then it can no longer be edited with a text editor).
Store the metadata at the end of the text file in a text format with perhaps comments or some indicator to leave the metadata alone. This is similar to the technique that the vi/vim text editor uses to embed vim commands into a file, it just puts them as comments at the beginning or end of the file.
I'm not sure there are many other ways to accomplish what you want, but perhaps one of those will work.
I am currently developing some functionality that implements some complex calculations. The calculations themselves are explained and defined in Word documents.
What I would like to do is create a hyperlink in each code file that references the assocciated Word document - just as you can in Word itself. Ideally this link would be placed in or near the XML comments for each class.
The files reside on a network share and there are no permissions to worry about.
So far I have the following but it always comes up with a file not found error.
file:///\\165.195.209.3\engdisk1\My Tool\Calculations\111-07 MyToolCalcOne.docx
I've worked out the problem is due to the spaces in the folder and filenames.
My Tool
111-07 MyToolCalcOne.docx
I tried replacing the spaces with %20, thus:
file:///\\165.195.209.3\engdisk1\My%20Tool\Calculations\111-07%20MyToolCalcOne.docx
but with no success.
So the question is; what can I use in place of the spaces?
Or, is there a better way?
One way that works beautifully is to write your own URL handler. It's absolutely trivial to do, but so very powerful and useful.
A registry key can be set to make the OS execute a program of your choice when the registered URL is launched, with the URL text being passed in as a command-line argument. It just takes a few trivial lines of code to will parse the URL in any way you see fit in order to locate and launch the documentation.
The advantages of this:
You can use a much more compact and readable form, e.g. mydocs://MyToolCalcOne.docx
A simplified format means no trouble trying to encode tricky file paths
Your program can search anywhere you like for the file, making the document storage totally portable and relocatable (e.g. you could move your docs into source control or onto a website and just tweak your URL handler to locate the files)
Your URL is unique, so you can differentiate files, web URLs, and documentation URLs
You can register many URLs, so can use different ones for specs, designs, API documentation, etc.
You have complete control over how the document is presented (does it launch Word, an Internet Explorer, or a custom viewer to display the docs, for example?)
I would advise against using spaces in filenames and URLs - spaces have never worked properly under Windows, and always cause problems (or require ugliness like %20) sooner or later. The easiest and cleanest solution is simply to remove the spaces or replace them with something like underscores, dashes or periods.
I have an IzPack installer that takes in a lot of User Inputs and substitutes them in an XML file. This XML file is actually the configuration file for my application.
There is a major problem that I have hit and I cant move on from it.
In the Input fields (in the installer) user can enter any text and also special characters like & # % ' etc. These special characters messes up my XML file as they are no allowed in the XML syntax and needs to be escaped. for example for & one would need &
So far I have been asking the user to do this, as in escape the special characters themselves, but thats now working either.
Is there a way to have this done automatically? I really need a solution fast.
I am using IzPack V 4.1
You should use a proper XML Api (SAX, DOM) to generate the XML file, this will apply the correct encoding automatically. This may look more complicated first but guarantees that a well formed, syntactically correct file is written.
Searching for JAXP should give you a proper starting point.
If I wanted to create a string which is guaranteed not to represent a filename, I could put one of the following characters in it on Windows:
\ / : * ? | < >
e.g.
this-is-a-filename.png
?this-is-not.png
Is there any way to identify a string as 'not possibly a file' on Linux?
There are almost no restrictions - apart from '/' and '\0', you're allowed to use anything. However, some people think it's not a good idea to allow this much flexibility.
An empty string is the only truly invalid path name on Linux, which may work for you if you need only one invalid name. You could also use a string like "///foo", which would not be a canonical path name, although it could refer to a file ("/foo"). Another possibility would be something like "/dev/null/foo", since /dev/null has a POSIX-defined non-directory meaning. If you only need strings that could not refer to a regular file you could use "/" or ".", since those are always directories.
Technically it's not invalid but files with dash(-) at the beginning of their name will put you in a lot of troubles. It's because it has conflicts with command arguments.
I personally find that a lot of the time the problem is not Linux but the applications one is using on Linux.
Take for example Amarok. Recently I noticed that certain artists I had copied from my Windows machine where not appearing in the library. I check and confirmed that the files were there and then I noticed that certain characters in the folder names (Named for the artist) were represented with a weird-looking square rather than an actual character.
In a shell terminal the filenames look even stranger: /Music/Albums/Einst$'\374'rzende\ Neubauten is an example of how strange.
While these files were definitely there, Amarok could not see them for some reason. I was able to use some shell trickery to rename them to sane versions which I could then re-name with ASCII-only characters using Musicbrainz Picard. Unfortunately, Picard was also unable to open the files until I renamed them, hence the need for a shell script.
Overall this a a tricky area and it seems to get very thorny if you are trying to synchronise a music collection between Windows and Linux wherein certain folder or file names contain funky characters.
The safest thing to do is stick to ASCII-only filenames.