I'm trying to do a report of all the objects in all the projects we have in Cloud Storage of our Org. I'm using this repo from the Google Professionnal Services as it's doing exactly what we want: https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/gcs2bq
We want to use containers instead of just the go code on a Cloud Function for portability mainly.
Locally everything is good and the program behave as expected but when I try in Cloud Run things get tricky. From what I understand, the go part needs to listen to a port, which I added at the beginning of the main so the container can be deployed, which it is:
// Determine port for HTTP service
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("defaulting to port %s", port)
}
Start HTTP server.
log.Printf("listening on port %s", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
log.Fatal(err)
}
But as you can see in the repo, the first file called is the run.sh one. Which set environment variables and then call the main.go. It sucessfully complete it's task, which is get all the size of the different files. But after that the run.sh doesnt "resume" and go to the part where it uploads the data in a BigQuery table, which locally work.
Here is the part in the run.sh file where I have a problem. Note : I don't have errors from executing the ./gcs2bq Note 2 : Every environment variable has a correct value
./gcs2bq $GCS2BQ_FLAGS || error "Export failed!" 2 <- doesnt get past this line
gsutil mb -p "${GCS2BQ_PROJECT}" -c standard -l "${GCS2BQ_LOCATION}" -b on "gs://${GCS2BQ_BUCKET}" || echo "Info: Storage bucket already exists: ${GCS2BQ_BUCKET}"
gsutil cp "${GCS2BQ_FILE}" "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}" || error "Failed copying ${GCS2BQ_FILE} to gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}!" 3
bq mk --project_id="${GCS2BQ_PROJECT}" --location="${GCS2BQ_LOCATION}" "${GCS2BQ_DATASET}" || echo "Info: BigQuery dataset already exists: ${GCS2BQ_DATASET}"
bq load --project_id="${GCS2BQ_PROJECT}" --location="${GCS2BQ_LOCATION}" --schema bigquery.schema --source_format=AVRO --use_avro_logical_types --replace=true "${GCS2BQ_DATASET}.${GCS2BQ_TABLE}" "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FIL$
error "Failed to load gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME} to BigQuery table ${GCS2BQ_DATASET}.${GCS2BQ_TABLE}!" 4
gsutil rm "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}" || error "Failed deleting gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}!" 5
rm -f "${GCS2BQ_FILE}"
I'm kinda new to containers and Cloud Run and even after reading projects and documentation, I'm not sure what I'm doing wrong, Is it normal that the .sh is "stuck" when calling the main.go? I can provide more details/explaination if needed.
Okay so for anyone who encounter similar situation this is how I made it work for me.
The container isn't supposed to stop so no exit, it will just go back to the main function.
That means that when I called executable it just looped and never exited and completed the task. So the solution here is to "recode" everything past the call in golang directly into the main.go
Here the run.sh is then useless so I used another .go file that listen for http request and then call the code that gather data and send it to Bigquery.
I want to use Cloudera's MapReduceIndexerTool to understand how morphlines work. I created a basic morphline that just reads lines from the input file and I tried to run that tool using that command:
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar org.apache.solr.hadoop.MapReduceIndexerTool \
--morphline-file morphline.conf \
--output-dir hdfs:///hostname/dir/ \
--dry-run true
Hadoop is installed on the same machine where I run this command.
The error I'm getting is the following:
net.sourceforge.argparse4j.inf.ArgumentParserException: Cannot write parent of file: hdfs:/hostname/dir
at org.apache.solr.hadoop.PathArgumentType.verifyCanWriteParent(PathArgumentType.java:200)
The /dir directory has 777 permissions on it, so it is definitely allowed to write into it. I don't know what I should do to allow it to write into that output directory.
I'm new to HDFS and I don't know how I should approach this problem. Logs don't offer me any info about that.
What I tried until now (with no result):
created a hierarchy of 2 directories (/dir/dir2) and put 777 permissions on both of them
changed the output-dir schema from hdfs:///... to hdfs://... because all the examples in the --help menu are built that way, but this leads to an invalid schema error
Thank you.
It states 'cannot write parent of file'. And the parent in your case is /. Take a look into the source:
private void verifyCanWriteParent(ArgumentParser parser, Path file) throws ArgumentParserException, IOException {
Path parent = file.getParent();
if (parent == null || !fs.exists(parent) || !fs.getFileStatus(parent).getPermission().getUserAction().implies(FsAction.WRITE)) {
throw new ArgumentParserException("Cannot write parent of file: " + file, parser);
}
}
In the message printed is file, in your case hdfs:/hostname/dir, so file.getParent() will be /.
Additionally you can try the permissions with hadoop fs command, for example you can try to create a zero length file in the path:
hadoop fs -touchz /test-file
I solved that problem after days of working on it.
The problem is with that line --output-dir hdfs:///hostname/dir/.
First of all, there are not 3 slashes at the beginning as I put in my continuous trying to make this work, there are only 2 (as in any valid HDFS URI). Actually I put 3 slashes because otherwise, the tool throws an invalid schema exception! You can easily see in this code that the schema check is done before the verifyCanWriteParent check.
I tried to get the hostname by simply running the hostname command on the Cent OS machine that I was running the tool on. This was the main issue. I analyzed the /etc/hosts file and I saw that there are 2 hostnames for the same local IP. I took the second one and it worked. (I also attached the port to the hostname, so the final format is the following: --output-dir hdfs://correct_hostname:8020/path/to/file/from/hdfs
This error is very confusing because everywhere you look for the namenode hostname, you will see the same thing that the hostname command returns. Moreover, the errors are not structured in a way that you can diagnose the problem and take a logical path to solve it.
Additional information regarding this tool and debugging it
If you want to see the actual code that runs behind it, check the cloudera version that you are running and select the same branch on the official repository. The master is not up to date.
If you want to just run this tool to play with the morphline (by using the --dry-run option) without connecting to Solr and playing with it, you can't. You have to specify a Zookeeper endpoint and a Solr collection or a solr config directory, which involves additional work to research on. This is something that can be improved to this tool.
You don't need to run the tool with -u hdfs, it works with a regular user.
I cannot write to a file for the life of me using Sinatra in production.
In my development environment, I can use Logger without a problem and log STDOUT to a file.
It seems like in production, the Logger class is overwritten by the RACK middleware's Logger and it makes things more complicated.
I simply want to write to a file like this:
post '/' do
begin
$log_file = File.open("/home/ec2-user/www/logs/app.log", "w")
...do..stuff...
$log_file.write "INFO -- #{Time.now} --\n #{notification['Message']}"
...do..stuff...
rescue
$log_file.write "ERROR -- #{Time.now} --" + "\njob failed"
ensure
$log_file.close
end
end
The file doesn't get created when I receive a POST request to '/'.
However the file DOES get created when I load the app running pry:
pry -r ./app.rb
I am certain the code inside the POST block is effectively running because new jobs are getting added to the database upon receiving requests..
Any help would be greatly appreciated.
I was finally able to get to the bottom of this.
I changed the nginx user in /etc/nginx/nginx.conf from nginx to ec2-user. (Ideally I would just fix the write permissions for the nginx user but this solution suits me for now.)
Then I ps aux | grep unicorn and saw the timestamp next to the process name: unicorn master -c unicorn.rb -D was 3 days old!!
All this time I was pushing my code the the production server, restarting nginx and never killed and restart the unicorn process.
I removed all the code in my POST block and left only the file creation part
post '/' do
$log_file = File.open("/home/ec2-user/www/logs/app.log", "a")
$log_file.write("test log string")
$log_file.close
end
And the the file was successfully written to upon receiving a POST request.
Using a Raspi/Debian - I have a script that parses the results from an iwlist scan and sends them via UDP to a Pure Data patch. This runs fine in gui mode, but now I'm trying to automate the whole process in another script with the following:
pd-extended -nogui /home/pi/patch.pd & /home/pi/libOSC/scan.sh && fg
But when I run this new script, the UDP appears to only send the info to Pure Data once, and then the scanning continues but Pd does not receive the packet. Any help with this would be appreciated.
What happens when you run /home/pi/libOSC/scan.sh? It sends the results only once? Then maybe you need to do it differently, like calling that script from within pd using the 'shell' or 'popen' objects for instance. Or you implement a polling command via UDP that will return the values.
how does your scan.sh script look like?
you probably want to make it something like:
pdhost=localhost
pdport=9999
do_scan() {
## some code here that does the scan and print's the result to stdout
}
doscan | while read line
do
echo "${line};" | pdsend ${pdhost} ${pdport}
done
rather than the following:
doscan | pdsend ${pdhost} ${pdport}
on webos, I have openssh running and would like to take a picture using the command line script.
I suspect this is going to include some luna-send command, or alternatively a gst-launch
But I am not having any luck with the docs.
webos doesn't have any of the expected capture tools, but I can access the /dev/video0 device.
Edit: i noticed that the touchpad has the ffmpeg utility installed, but it doesn't recognise the video4linux2 format
So far, I am trying Gopherkhan's suggestions with the following code;
luna-send -n 1 palm://com.palm.mediad.MediaCapture/startImageCapture \
'{"path":"/media/internal/foo1.png","options":[{"quality" \
:100,"flash":2,'reviewDuration':0,'exifData':{}}]}'
but its just hanging there doing nothing, after a while is says this;
{"serviceName":"com.palm.mediad.MediaCapture","returnValue":false,"errorCode":-1 \
,"errorText":"com.palm.mediad.MediaCapture is not running."} \
(process:8534): LunaService-CRITICAL **: AppId msg type: 17
So to do this with luna-sends is a bit tricky, and technically not supported.
You're probably going to want to hit the MediaCapture library, which can be found on the device here:
/usr/palm/frameworks/enyo/0.10/framework/lib/mediacapture
To include it in your enyo app drop the following in your depends.js:
"$enyo-lib/mediacapture/"
There are three main steps involved.
Initializing the component
Capturing the image
Unloading the device.
Here's a sample:
Declare the component in your scene
{
kind: "enyo.MediaCapture", name:"mediaCaptureObj",
onLoaded:"_setUpLoadedState", onInitialized:"_setUpInitializedState",
onImageCaptureStart:"_onImageCaptureStart", onImageCaptureComplete:"_onImageCaptureComplete",
onAutoFocusComplete:"_onAutoFocusComplete", onError:"_handleError",
onElapsedTime:"_onElapsedTime", onVuData:"_onVuDataChange", onDuration:"_onDuration"
}
Call the initialize method:
this.$.mediaCaptureObj.initialize(this.$.ViewPort);
In your onInitialized callback
Use the property bag to locate the number of devices that are available. Typically, the descriptions are "Camera/Camcorder", "Front Microphone", and "User facing camera"
var keyString;
for(var i = 0; i < this.pb.deviceKeys.length; i++)
{
if(this.pb.deviceKeys[i].description.indexOf("Camera/Camcorder") >= 0)
{
keyString = this.pb.deviceKeys[i].deviceUri;
break;
}
}
if(keyString)
{
var formatObj = {
imageCaptureFormat: this.pb[keyString].supportedImageFormats[0]
};
this.$.mediaCaptureObj.load(keyString, formatObj);
}
Take a photo.
var obj = {"exifData":"{\"make\": \"Palm\", \"model\": \"Pre3\", \"datetime\": \"2011:05:19 10:39:18\", \"orientation\": 1, \"geotag\": {}}","quality":90,"flash":"FLASH_ON"};
this.$.mediaCaptureObj.startImageCapture("", obj);
Unload the device:
this.$.mediaCaptureObj.unload();
To do this with the old JS frameworks, see:
https://developer.palm.com/content/api/reference/javascript-libraries/media-capture.html
Now, you can do something similar with luna-send, but again, I don't think it's technically supported. You might have trouble with starting-up/keeping-alive the media capture service, etc. BUT, if you want to try, you could do something along the lines of:
1. get the media server instance --- this returns a port instance number
luna-send -a your.app.id -i palm://com.palm.mediad/service/captureV3 '{"args":["subscribe":true]}'
This will return a location of the capture service with a port number, a la:
{"returnValue":true, "location":"palm://com.palm.mediad.MediaCaptureV3_7839/"}
Since this is a subscription, don't kill the request. Just open a new terminal.
2. Open a new terminal. Use the "location" returned in step 1 as your new service uri:
luna-send -a your.app.id -i palm://com.palm.mediad.MediaCaptureV3_7839/load '{"args":["video:1", {"videoCaptureFormat":{"bitrate":2000000,"samplerate":44100,"width":640,"height":480,"mimetype":"video/mp4","codecs":"h264,mp4a.40"},"imageCaptureFormat":{"bitrate":0,"samplerate":1700888,"width":640,"height":480,"mimetype":"image/jpeg","codecs":"jpeg"},"deviceUri":"video:1"}]}'
You should see:
{"returnValue":true}
if the call completed correctly. You can safely ctrl+c out of this call.
3. Take your picture. (you can ctrl+c out of the last call, and just supply the args here)
luna-send -a your.app.id -i palm://com.palm.mediad.MediaCaptureV3_7839/startImageCapture '{"args":["", {"exifData":"{\"orientation\": 1, \"make\": \"HP\", \"model\": \"TouchPad\", \"datetime\": \"2011:09:22 15:34:36\", \"geotag\": {}}","quality":90,"flash":"FLASH_DISABLED","orientation":"faceup"}]}'
Again, you should see:
{"returnValue":true}
if the call completed correctly.
You should hear a shutter click, and the image will show up in the Photos app, in your Photo Roll.
An alternative, which might some benefit of using cross platform tools, is to the use the gst-launch pipeline. So far I have managed to start the web cam using command line;
gst-launch camsrc .src ! video/x-raw-yuv,width=320,height=240,framerate=30/1
! palmvideoencoder ! avimux name=mux ! filesink location=test1.avi alsasrc !
palmaudioencoder
but not take a single image;
gst-launch -v camsrc .src_still take-picture=1 flash-ctrl=2 ! fakesink dump=true
but I can't get it to recognise the .src_still tab. I will update this answer with this alternative method as I proceed.