etcd "Not a directory" result when put some value - etcd

I'm using etcd v3.3 in my application and communicate with it over its WEB API.
According to documentation I don't need to explicitly create directories when putting key-value pairs on some path.
Here is an example what I doing (note that the path /base-test-path/level1/level2/level3/ does not exist yet):
curl -X PUT -d value=foo http://localhost:2379/v2/keys/base-test-path/level1/level2/level3/
The result was:
{"action":"set","node":{"key":"/base-test-path/test/test/test","value":"foo","modifiedIndex":347017,"createdIndex":347017}}
But when I try to add a new value a bit deeper into existing path, I get an error (note that the path /base-test-path/level1/level2/level3/ already exists because I run previous command before):
curl -X PUT -d value=foo http://localhost:2379/v2/keys/base-test-path/level1/level2/level3/level4
Response:
{"errorCode":104,"message":"Not a directory","cause":"/base-test-path/level1/level2/level3","index":347018}
It seems like etcd does not create directories when any part of the path already exists.
The question is: can I keep my code simple so I don't need to care about etcd directories and still be able to put values on every etcd's path I want?

It seems like etcd, when speak it with API v2, can not create directories when some key is already on the place.
E.g. when you have a key
/base-test-path/level1/somekey
You can not create such key ("Not a directory"):
/base-test-path/level1/somekey/subkey
When I switched to API v3 this problem is eliminated as it does not have directories at all. See a comment here or official documentation here.

Related

Directory argument for Bowtie2

I'm using the software Bowtie 2 which aligns genome sequences. I have all my indexes in a directory miniReference1.
When I call Bowtie2 with the -x <dir> option I get the error that my index is not a Bowtie 2 index. What am I doing wrong? Below is a screenshot:
The -x argument takes the path to the basename of the index. That means, if the index called miniReference1.{suffix} is inside a folder also called miniReference1 then it must be -x path/to/miniReference1/miniReference1. There is no need to use this variable, just use the plain path.
I had a similar issue running bowtie 2.5.0 against index files built by bowtie 2.4.5. The error message shows readU: No such file or directory even though the files are there.
I rebuilt the indexes using bowtie 2.5.0 and the issue was resolved. So I'd suggest rebuilding the indexes using the same version as the aligner, and then running it again.
I'm quoting the relevant part of the manual:
-x           The basename of the index for the reference genome. The basename is the name of any of the index files up to but not including the final .1.bt2 / .rev.1.bt2 / etc. bowtie2 looks for the specified index first in the current directory, then in the directory specified in the BOWTIE2_INDEXES environment variable. [Emphasis by me.]
After reading this, I assume that what was missing is to tell BowTie2 the directory it is supposed to look in by setting the environment variable BOWTIE2_INDEXES. In bash you would do that by issuing the command
export BOWTIE2_INDEXES=miniReference1
If that doesn't help try to provide the absolute path. It is unclear to me whether Bowtie2 expects a Posix path name like /c/Users/raghdad/Documents/project-x/sequences/miniReference1 or a Windows path name like C:\Users\.... (pwd in your terminal gives you the Posix name for the current directory).
The -x file base name argument, miniReference1, is correct but does not name a directory; instead, it tells BowTie2 the set of indexes you want to work with, which is identified by the common "base name" of the files (the part up to the first dot). The reason for this scheme is probably that you could have different sets of indexes in the same directory, and each set would have a distinct and unique base name. Of course your way of putting each set in its own directory appears to be much cleaner, unless there is a reason to assess several index sets in one go.

Store an API_KEY in an env var and use in a playlist URL

I use a streaming service (di.fm) that has many channels. Each channel has a playlist I stream from the CLI (using mpv). Each URL in each playlist stores the API KEY.
I want to store the API KEY outside of the individual playlists, so for example, if I change the API KEY, I don't have to change every playlist.
I'm on a Mac.
1) What is the best (safest) place to declare export DI_KEY=""? In .bashrc was my first thought, except I back it up to github. Any other better place to declare the env var that will be created each time I enter bash?
2) In the playlist file, how do I use the $DI_KEY in the URL?
[playlist]
NumberOfEntries=1
File1=http://prem4.di.fm:80/00sclubhits?$DI_KEY
Title1=DI.FM - 00s Club Hits
Length1=0
Version=2
Just referencing it directly doesn't work.
I'm sure this may be answered elsewhere, but in all my searching I couldn't find any helpful answers, particularly to questions 2.
Regarding setting env variables outside of .bashrc, you could create a separate file to define sensitive variables and source this from within your .bashrc.
For example, create a file ~.my-private-variables, add the filename to your .gitignore and add the line export DI_KEY="12345" to this file. Then add the following block in .bashrc:
if [ -f ~/.my-private-variables ]; then
. ~/.my-private-variables
fi
Regarding the playlist file, bash is not running the file, so the environment variable is not expanded.
You could dynamically generate the playlist when bash starts, something like this:
#!/bin/bash
filename=playlist-1.pls
baseurl=http://prem4.di.fm:80
cat << EOF > $filename
[playlist]
NumberOfEntries=1
File1=${baseurl}/00sclubhits?${DI_KEY}
Title1=DI.FM - 00s Club Hits
Length1=0
Version=2
EOF
This will expand the variable and write it to the file, in this case playlist-1.pls in the current working directory. You might add an absolute path to the filename variable that references your playlists directory.
To run this, you could create a script called playlist-generator and source this in .bashrc as described above. You could add as many playlists as you like here.

Move files under GCS with renaming

I want to write the following bash script which copies files from one GCS bucket to another with renaming options.
My input folder is gs://test-rtt-integration/result/frd/*.orc
and my destination folder is gs://test-rtt-integration/recent_files/frd
The renaming of the copied file should be done based on the name provided from gs://test-rtt-integration/complex-files/TAN/recent_files/today/frd
once the copy with renaming is done I need to clean gs://test-rtt-integration/result/frd
I tested the following commands, but they are not working properly
NAME = "$(gsutil ls gs://test-rtt-integration/complex-files/TAN/recent_files/today/frd)"
gsutil mv gs://test-rtt-integration/result/frd/*.orc gs://test-rtt-integration/recent_files/frd/$NAME
gsutil rm -rf gs://test-rtt-integration/result/frd
( all .orc files and other files should be deleted)
But this is not working properly as I have to split the NAME based on / and get the last split , so if the result of split is called SPLIT , I have to do gsutil mv gs://test-rtt-integration/result/frd/*.orc gs://test-rtt-integration/recent_files/frd/$SPLIT
Any idea on how to do this?
The question is a little bit confusing. You say that you want to move files from one Google Cloud Storage bucket to another, but all the operations are made in one single bucket called test-rtt-integration.
However, as soon as you get the file location with the command gsutil ls gs://[BUCKET_NAME]/folder e.g. gs://[BUCKET_NAME]/folder/[FILENAME].orc, since the gs://[BUCKET_NAME]/folder/ part is always the same for all the objects in the folder, just replace it with null and you will get only the object name at the end as [FILENAME].orc etc.
I am not sure if this is exactly what you are looking for, but I did a little bit of coding myself and I have created a bash script that:
Gets the name of each object from gs://[BUCKET_NAME]/from bucket folder
Copy all objects from gs://[BUCKET_NAME]/from bucket folder to the gs://[BUCKET_NAME]/to/ bucket folder
Delete all objects from gs://[BUCKET_NAME]/from bucket folder
Inside there are comments that explain how every operation works in details. If that is not exactly what you are looking for, you can get the basic idea of how that works and implement it in different way that will suit you better. I have tested the scrip myself in Google Cloud Shell and it is working. The example code can be found in GitHub.

autoconf: how do I substitute the library prefix?

CLISP's interface to PARI is configured with the configure.in containing AC_LIB_LINKFLAGS([pari]) from lib-link.m4.
The build process also requires the Makefile to know where the datadir of PARI is located. To this end, Makefile.in has
prefix = #LIBPARI_PREFIX#
DATADIR = #datadir#
and expects to find $(DATADIR)/pari/pari.desc (normally
/usr/share/pari/pari.desc or /usr/local/share/pari/pari.desc).
This seems to work on Mac OS X where PARI is installed by homebrew in /usr/local (and LIBPARI_PREFIX=/usr/local), but not on Ubuntu, where PARI is in /usr, and LIBPARI_PREFIX is empty.
How do I insert the location of the PARI's datadir into the Makefile?
PS. I also asked this on the autoconf mailing list.
PPS. In response to #BrunoHaible's suggestion, here is the meager attempt at debugging on Linux (where LIBPARI_PREFIX is empty).
$ bash -x configure 2>&1 | grep found_dir
+ found_dir=
+ eval ac_val=$found_dir
+ eval ac_val=$found_dir
You are trying to use $(prefix) in an unintended way. In an Autotools-based build system, the $(prefix) represents a prefix to the target installation location of the software you're building. By setting it in your Makefile.in, you are overriding the prefix that configure will try to assign. However, since you appear not to have any installation targets anyway, at least at that level, that's probably more an issue of poor form than a cause for malfunction.
How do I insert the location of the PARI's datadir into the Makefile?
I'd recommend computing or discovering the needed directory in your configure script, and exporting it to the generated Makefile via its own output variable. Let's take the second part first, since it's simple. In configure.in, having in some manner located the wanted data directory and assigned it to a variable
DATADIR=...
, you would make an output variable of that via the AC_SUBST macro:
AC_SUBST([DATADIR])
Since you are using only Autoconf, not Automake, you would then manually receive that into your Makefile by changing the assignment in your Makefile.in:
DATDIR = #DATADIR#
Now, as for locating the data directory in the first place, you have to know what you're trying to implement before you can implement it. From your question and followup comments, it seems to me that you want this:
Use a data directory explicitly specified by the user if there is one. Otherwise,
look for a data directory relative to the location of the shared library. If it's not found there then
(optional) look under the prefix specified to configure, or specifically in the specified datadir (both of which may come from the top-level configure). Finally, if it still has not been found then
look in some standard locations.
To create a configure option by which the user can specify a custom data directory, you would probably use the AC_ARG_WITH macro, maybe like this:
AC_ARG_WITH([pari-datadir], [AS_HELP_STRING([--with-pari-datadir],
[explicitly specifies the PARI data directory])],
[], [with_pari_datadir=''])
Thanks to #BrunoHaible, we see that although the Gnulib manual does not document it, the macro's internal documentation specifies that if AC_LIB_LINKFLAGS locates libpari then it will set LIBPARI_PREFIX to the library directory prefix. You find that that does work when the --with-libpari option is used to give it an alternative location to search, so I suggest working with that. You certainly can try to debug AC_LIB_LINKFLAGS to make it set LIBPARI_PREFIX in all cases in which the lib is found, but if you don't want to go to that effort then you can work around it (see below).
Although the default or specified installation prefix is accessible in configure as $prefix, I would suggest instead going to the specified $datadir. That is slightly tricky, however, because by default it refers to the prefix indirectly. Thus, you might do this:
eval "datadir_expanded=${datadir}"
Finally, you might hardcode a set of prefixes such as /usr and /usr/local.
Following on from all the foregoing, then, your configure.in might do something like this:
DATADIR=
for d in \
${with_pari_datadir} \
${LIBPARI_PREFIX:+${LIBPARI_PREFIX}/share/pari} \
${datadir_expanded}/pari \
/usr/local/share/pari \
/usr/share/pari
do
AS_IF([test -r "$[]d/pari.desc"], [DATADIR="$[]d"; break])
done
AS_IF([test x = "x$DATADIR"], [AC_MSG_ERROR(["Could not identify PARI data directory"])])
AC_SUBST([DATADIR])
Instead of guessing the location of datadir, why don't you ask PARI/GP where its datadir is located? Namely,
$ echo "default(datadir)" | gp -qf
"/usr/share/pari"
does the trick.

How to parse html with wget to download an artifact using pattern matching against Jenkins

I am trying to download an artifact from Jenkins where I need the latest build. If I curl jenkins.mycompany.com/view/iOS/job/build_ios/lastSuccessfulBuild/artifact/build it brings me to the page that contains the artifact I need to download, which in my case is myCompany-1234.ipa
so by changing curl to wget with --auth-no-challenge https://userIsMe:123myapikey321#jenkins.mycompany.com/view/iOS/job/build_ios/lastSuccessfulBuild/artifact/build/ it downloads the index.html file.
if I put --reject index.html it stops the index.html downloading.
if I add the name of the artifact with a wildcard like so MyCompany-*.ipait downloads a 14k file named MyCompany-*.ipa, not the MyCompany-1234.ipa I was hoping for. Keep in mind the page i am requesting only has 1 MyCompany-1234.ipa so there will never be multiple matches found
if I use a flag to pattern match -A "*.ipa" like so: wget --auth-no-challenge -A "*.ipa" https://userIsMe:123myapikey321#jenkins.mycompany.com/view/iOS/job/build_ios/lastSuccessfulBuild/artifact/build/ It still doesn't download the artifact.
It works if I perfectly input the exact url like so: wget --auth-no-challenge https://userIsMe:123myapikey321#jenkins.mycompany.com/view/iOS/job/build_ios/lastSuccessfulBuild/artifact/build/MyCompany-1234.ipa
The problem here is the .ipa is not always going to be 1234, tomorrow will be 1235, and so on. How can I either parse the html or use the wildcard correctly in wget to ensure I am always getting the latest?
NM, working with another engineer here at my work came up with a super elegant solution parsing json.
Install Chrome and get the plugin JSONView
Call the Jenkins API in your Chrome browser using https://$domain/$job/lastSuccessfulBuild/api/json
This will print out the key pair values in json. Denote your Key, for me it was number
brew install jq
In a bash script create a variable that will store the dynamic value as follows
This will store the build number to latest:
latest=$(curl --silent --show-error https://userIsMe:123myapikey321#jenkins.mycompany.com/job/build_ios/lastSuccessfulBuild/api/json | jq '.number')
Print it to screen if you would like:
echo $latest
Now with some string interpolation pass the variable for latest to your wget call:
wget --auth-no-challenge https://userIsMe:123myapikey321#jenkins.mycompany.com/view/iOS/job/build_ios/lastSuccessfulBuild/artifact/build/myCompany-$latest.ipa
Hope this helps someone out as there is limited info out there that is clear and concise, especially given that wget has been around for an eternity.

Resources