I am writing an automation script for an old project and I need some help with pvpython from Paraview 3.98.1. The function SaveData() in this version does not exist. I found its implementation here and moved it to my code. How can I save a file as ASCII? Calling it like SaveData(filename, proxy=px, FileType='Ascii') saves my files as binaries (awkward behavior).
I need this version because some of my codes in the scripting pipeline handle very specific vtk files. Using the SaveData() function of the latest versions ended up creating different metadata in my final files, and when I process them it ends up destroying my results. It is easier at the moment to use an older version of Paraview than to modify all my codes.
Edit:
The website is not working now, but it was yesterday. Maybe it is an internal problem? Anyway, the code is attached below.
# -----------------------------------------------------------------------------
def SetProperties(proxy=None, **params):
"""Sets one or more properties of the given pipeline object. If an argument
is not provided, the active source is used. Pass a list of property_name=value
pairs to this function to set property values. For example::
SetProperties(Center=[1, 2, 3], Radius=3.5)
"""
if not proxy:
proxy = active_objects.source
properties = proxy.ListProperties()
for param in params.keys():
pyproxy = servermanager._getPyProxy(proxy)
pyproxy.__setattr__(param, params[param])
# -----------------------------------------------------------------------------
def CreateWriter(filename, proxy=None, **extraArgs):
"""Creates a writer that can write the data produced by the source proxy in
the given file format (identified by the extension). If no source is
provided, then the active source is used. This doesn't actually write the
data, it simply creates the writer and returns it."""
if not filename:
raise RuntimeError ("filename must be specified")
session = servermanager.ActiveConnection.Session
writer_factory = servermanager.vtkSMProxyManager.GetProxyManager().GetWriterFactory()
if writer_factory.GetNumberOfRegisteredPrototypes() == 0:
writer_factory.UpdateAvailableWriters()
if not proxy:
proxy = GetActiveSource()
if not proxy:
raise RuntimeError ("Could not locate source to write")
writer_proxy = writer_factory.CreateWriter(filename, proxy.SMProxy, proxy.Port)
writer_proxy.UnRegister(None)
pyproxy = servermanager._getPyProxy(writer_proxy)
if pyproxy and extraArgs:
SetProperties(pyproxy, **extraArgs)
return pyproxy
# -----------------------------------------------------------------------------
def SaveData(filename, proxy=None, **extraArgs):
"""Save data produced by 'proxy' in a file. If no proxy is specified the
active source is used. Properties to configure the writer can be passed in
as keyword arguments. Example usage::
SaveData("sample.pvtp", source0)
SaveData("sample.csv", FieldAssociation="Points")
"""
writer = CreateWriter(filename, proxy, **extraArgs)
if not writer:
raise RuntimeError ("Could not create writer for specified file or data type")
writer.UpdateVTKObjects()
writer.UpdatePipeline()
del writer
# -----------------------------------------------------------------------------
The question is answered here (also my post). I used SaveData() to save a binary file with the proxy I need and then used DataSetWriter() to change my FileType to ASCII. It is not a beautiful solution since SaveData() is supposed to do that, but it does the job.
Related
I'm trying to learn how to use the google speech to text API but I'm getting an undefined method error when running the code. The code is taken from google cloud client library to use. I'm not sure how to fix the error.
Error: : undefined method alternatives for nil:NilClass (NoMethodError)
#use gem install google-cloud-speech
require "google/cloud/speech"
speech = Google::Cloud::Speech.speech
file_name = "file_path"
audio_file = File.binread file_name
config = { encoding: :LINEAR16,
sample_rate_hertz: 16_000,
language_code: "en-US" }
audio = { content: audio_file }
response = speech.recognize config: config, audio: audio
results = response.results
results.first.alternatives.each do |alternatives|
puts "Transcription: #{alternatives.transcript}"
end
I have tried to replicate your issue and I was able to successfully execute the code and other examples from the documentation. In addition, since you stated that you are learning how to use the Speech to Text API from Google, I will describe the steps I took.
It seems that you have taken the code from the documentation. However, you do not have the first lines and the last lines of the code, which respectively define transcription method and the call to execute it. For this reason, when you execute your code there is no call to any method and neither to the Speech-to-Text API. Your code should be as follows:
def speech_sync_recognize audio_file_path: nil
# [START speech_transcribe_sync]
# audio_file_path = "Path to file on which to perform speech recognition"
require "google/cloud/speech"
speech = Google::Cloud::Speech.speech
# [START speech_ruby_migration_sync_response]
audio_file = File.binread audio_file_path
config = { encoding: :LINEAR16,
sample_rate_hertz: 16_000,
language_code: "en-US" }
audio = { content: audio_file }
response = speech.recognize config: config, audio: audio
results = response.results
alternatives = results.first.alternatives
alternatives.each do |alternative|
puts "Transcription: #{alternative.transcript}"
end
# [END speech_ruby_migration_sync_response]
# [END speech_transcribe_sync]
end
if $PROGRAM_NAME == __FILE__
command = ARGV.shift
#I have added this part in order to use a command after to define which method to call within the code.
case command
when "recognize"
speech_sync_recognize audio_file_path: ARGV.first
end
end
And in order to run the sample,
bundle exec ruby speech_samples.rb
Notice the argument recognize which describes which method to execute from the code. In the above case, there is just one. However,using arguments in the call is very useful when there other methods available to call within the code.
Furthermore, I will describe the steps I took in order to execute the code properly. I have followed the steps described here,
I have have ran the code from Cloud Shell. However the speech.googleapis.com does not support end user authentication from Cloud Shell. For this reason, I have given the Service Account Token Creator from IAM Console to my user, so I could impersonate a Service Account and call the API.
Export your project_id to a environment variable such as export GOOGLE_CLOUD_PROJECT="YOUR-PROJECT-ID".
Download the Gemfile file from the documentation and run the command: bundle install
Copy the code speech_samples.rb to your environment.
Copy your file path to the variable audio_file_path and un-comment it. Notice that there will be many audio_file_path local variables, each one for a specific method. In my case, I copied the path only to the variable in the first function, audio_file_path = "home/alex/audio.wav".
Pay attention that each function has a specific configured argument (from line 437 onward). In my case, I wanted to use the recognize one.
Run the sample code with bundle exec ruby speech_samples.rb recognize.
Check your environment for a new directory called results. Then inspect the outputs.
Please, do not forget to copy your file path to all the functions you will be calling with the proper arguments.
UPDATE:
As I mentioned in the comment section, the code I shared above is part of the file speech_samples.rb which contains various examples of functions that calls the Speech-To-Text API. In my case, I just used part of the sample, which I pasted above.
Regarding the link you shared, there is a button to the GitHub source repo for the code, which is the same as the one with the example I used. Notice that your code in GitHub,here, it is wrapped in a function. Also, this function should be called in order to be executed, you can call it by simply writing its name after the function definition. Thus the code should look as follows,
def quickstart
# [START speech_quickstart]
# Imports the Google Cloud client library
# [START speech_ruby_migration_import]
require "google/cloud/speech"
# [END speech_ruby_migration_import]
# Instantiates a client
# [START speech_ruby_migration_client]
speech = Google::Cloud::Speech.speech
# [END speech_ruby_migration_client]
# The name of the audio file to transcribe
file_name = "./resources/brooklyn_bridge.raw"
# [START speech_ruby_migration_sync_request]
# [START speech_ruby_migration_config]
# The raw audio
audio_file = File.binread file_name
# The audio file's encoding and sample rate
config = { encoding: :LINEAR16,
sample_rate_hertz: 16_000,
language_code: "en-US" }
audio = { content: audio_file }
# Detects speech in the audio file
response = speech.recognize config: config, audio: audio
# [END speech_ruby_migration_config]
results = response.results
# [END speech_ruby_migration_sync_request]
# Get first result because we only processed a single audio file
# Each result represents a consecutive portion of the audio
results.first.alternatives.each do |alternatives|
puts "Transcription: #{alternatives.transcript}"
end
# [END speech_quickstart]
end
#call the function defined above
quickstart
NOTE: I must point out that I was also able to execute the code from your question as well, without wrapping in a function. I followed the steps described in the documentation and it successfully retrieved the output.
I have an AWS Lambda deployed successfully with Terraform:
resource "aws_lambda_function" "lambda" {
filename = "dist/subscriber-lambda.zip"
function_name = "test_get-code"
role = <my_role>
handler = "main.handler"
timeout = 14
reserved_concurrent_executions = 50
memory_size = 128
runtime = "python3.6"
tags = <my map of tags>
source_code_hash = "${base64sha256(file("../modules/lambda/lambda-code/main.py"))}"
kms_key_arn = <my_kms_arn>
vpc_config {
subnet_ids = <my_list_of_private_subnets>
security_group_ids = <my_list_of_security_groups>
}
environment {
variables = {
environment = "dev"
}
}
}
Now, when I run terraform plan command it says my lambda resource needs to be updated because the source_code_hash has changed, but I didn't update lambda Python codebase (which is versioned in a folder of the same repo):
~ module.app.module.lambda.aws_lambda_function.lambda
last_modified: "2018-10-05T07:10:35.323+0000" => <computed>
source_code_hash: "jd6U44lfe4124vR0VtyGiz45HFzDHCH7+yTBjvr400s=" => "JJIv/AQoPvpGIg01Ze/YRsteErqR0S6JsqKDNShz1w78"
I suppose it is because it compresses my Python sources each time and the source changes. How can I avoid that if there are no changes in the Python code? Is my hypothesis coherent if I didn't change the Python codebase (I mean, why then the hash changes)?
This is because you are hashing just main.py but uploading dist/subscriber-lambda.zip. Terraform compares the hash to the hash it calculates when the file is uploaded to lambda. Since the hashing is done on two different files, you end up with different hashes. Try running the hash on the exact same file that is being uploaded.
This works for me and also doesn't trigger an update on the Lambda function when the code hasn't changed
data "archive_file" "lambda_zip" {
type = "zip"
source_dir = "../dist/go"
output_path = "../dist/lambda_package.zip"
}
resource "aws_lambda_function" "aggregator_func" {
description = "MyFunction"
function_name = "my-func-${local.env}"
filename = data.archive_file.lambda_zip.output_path
runtime = "go1.x"
handler = "main"
source_code_hash = data.archive_file.lambda_zip.output_base64sha256
role = aws_iam_role.function_role.arn
timeout = 120
publish = true
tags = {
environment = local.env
}
}
I'm going to add my answer to contrast to the one #ODYN-Kon provided.
The source code hash field in resource "aws_lambda_function" is not compared to some hash of the zip you upload. Instead, the hash is merely checked against the Terraform saved state from the last time it ran. So, the next time you run Terraform, it computes the hash of the actual python file to see if it has changed. If it has, it assumes that the zip has been changed and the Lambda function resource needs to be run again. The source_code_hash can have any value you want to give it or it can be omitted entirely. You could set it to a constant of some arbitrary string, and then it would never change unless you edit your Terraform configuration.
Now, the problem there is that Terraform assumes you updated the zip file. Assuming you only have one directory or one file in the zip archive, you can use the Terraform data source archive_file to create the zip file. I have a case where I cannot use that because I need a directory and a file (JS world: source + node_modules/). But here is how you can use that:
data "archive_file" "lambdaCode" {
type = "zip"
source_file = "lambda_process_firewall_updates.js"
output_path = "${var.lambda_zip}"
}
Alternativly, you can archive an entire directory, if you replace the "source_file" statement with source_dir = "node_modules"
Once you do this, you can reference the hash code of the zip archive file for insertion into resource "aws_lambda_function" "lambda" { block as "${data.archive_file.lambdaCode.output_base64sha256}" for the field source_hash. Then, anytime the zip changes, the lambda function gets updated. And, the data source archive file knows that anytime the source_file changes it must regenerate the zip.
Now, I haven't drilled down to a root cause in your case, but hopefully given some help to get to a better place. You can check the saved state of Terraform via: tf state list - which lists the items of saved state. You can find the one that matches your lambda function block and then execute tf state show <state-name>. For example, for one I am working on:
tf state show aws_lambda_function.test-lambda-networking gives about 30 lines of output, including:
source_code_hash = 2fKX9v/duluQF0H6O9+iRnID2gokhfpXIXpxyeVBUM0=
You can compare the hash via command line commands. Example on MacOS: sha256sum my-lambda.zip, where sha256sum was installed by brew install coreutils.
As mentioned, the use of archive_file doesn't work when you have multiple elements of the zip which are not isolated to a single directory. I think that probably happens a lot, so I wish the Hashicorp guys would extend archive_file to support multiple. I even went looking at the Go code, but that is a rainy day project. One variation I use is to take the source_code_hash to be "${base64sha256(file("my-lambda.zip"))}". But that still requires me to run tf twice.
As others have said, your zip should be used in your filename and your hash.
I want to mention that you can also get similar recreation issues if you use the wrong hash function in your lambda definitions. For example filesha256(.zip) will also recreate your lambdas every time. You have to use filebase64sha256("file.zip") (terraform 0.11.12+) or base64sha256(file("file.zip")) as mentioned under source_code_hash here
I wanna figure out how to download images from internet then store them locally.
Here's what I did:
require 'open-uri' # => true
file = open "https://s3-ap-southeast-1.amazonaws.com/xxx/Snip20180323_40.png"
# => #<Tempfile:/var/folders/k0/.../T/open-uri20180524-60756-1r44uix>
Then I was confused about this Tempfile object. I found I can get the original url by:
file.base_uri
# => #<URI::HTTPS https://s3-ap-southeast-1.amazonaws.com/xxx/Snip20180323_40.png>
But I failed in finding a method that can directly get the original file name Snip20180323_40.png.
Is there a method that can directly get the original file name from a Tempfile object?
What purpose are Tempfile objects mainly used for? Are they different from normal file objects such as: file_object = File.open('how_old.rb') # => #<File:how_old.rb>?
Can I convert a Tempfile object to a File object?
How can I write this Tempfile as the same name file in a local directory, for example /users/user_name/images/Snip20180323_40.png?
The original filename is only really available in the URL. Just take uri.path.split("/").last.
Tempfiles are effective Files, with the distinction that when it is garbage collected, the underlying file is deleted.
You can copy the underlying file with FileUtils.copy, or you can open the Tempfile, read it, and write it into a new File handle of your choosing.
Something like this should work:
def download_url_to(url, base_path)
uri = URI(url)
filename = uri.path.split("/").last
new_file = File.join(base_path, filename)
response = uri.open
open(new_file, "wb") {|fp| fp.puts response.read }
return new_file
end
It's worth noting that if the file is less than 10kb, you'll get a StringIO object rather than a Tempfile object. The above solution handles both cases. This also just accepts whatever the last part of the path parameter is - it's going to be up to you to sanitize it, as well as the contents of the file itself; you don't want to permit clients to download arbitrary files to your system, in most cases. For example, you may want to be extra sure that the filename doesn't include paths like ..\\..\\.."which may be used to write files to non-intended locations.
In ruby...
I have an IO object created by an external process, which I need to get the file name from.
However I only seem to be able to get the File descriptor (3), which is not very useful to me.
Is there a way to get the filename from this object or even to get a File Object?
I am getting the IO object from notifier. So this may be a way of getting the file path as well?
There is a similar question on how to get a the filename in C, I will present here the answer to this question in a ruby way.
Getting the filename in Linux
Suppose io is your IO Object. The following code gives you the filename.
File.readlink("/proc/self/fd/#{io.fileno}")
This does not work for example if the file was removed after the io object was created for it. With this solution you have the filename, but not an File object.
Getting a File object which does not know the filename
The method IO#for_fd can create an IO and it's subclasses for any given integer filedescriptor. Your get your File object for your fd by doing:
File.for_fd(io.fileno)
Unfortunely this File object does not know the filename.
File.for_fd(io.fileno).path # => nil
I scanned through the ruby-1.9.2 sources. There seems to be no way in pure ruby to manipulate the path after the file object was created.
Getting a File object which does know the filename
An extension to ruby can be created in C which first calls File#for_fd and afterwards manipulates the Files internal data structures. This sourcecode does work for ruby-1.9.2, for other versions of ruby it may has to be adjustet.
#include "ruby.h"
#include "ruby/io.h"
VALUE file_fd_filename(VALUE self, VALUE fd, VALUE filename) {
VALUE file= rb_funcall3(self, rb_intern("for_fd"), 1, &fd);
rb_io_t *fptr= RFILE(rb_io_taint_check(file))->fptr;
fptr->pathv= rb_str_dup(filename);
return file;
}
void Init_filename() {
rb_define_singleton_method(rb_cFile, "for_fd_with_filename", file_fd_filename, 2);
}
Now you can do after compiling:
require "./filename"
f= File.for_fd_with_filename(io.fileno, File.readlink("/proc/self/fd/#{io.fileno}"))
f.path # => the filename
The readlink could also be put into the File#for_fd_with_filename definiton. This examples is just to show how it works.
If you are sure that the IO object represents a File you could try something like this
path = io.path if io.respond_to?(:path)
See documentation for File#path
If I am running a long R script from the command line (R --slave script.R), then how can I get it to give line numbers at errors?
I don't want to add debug commands to the script if at all possible; I just want R to behave like most other scripting languages.
This won't give you the line number, but it will tell you where the failure happens in the call stack which is very helpful:
traceback()
[Edit:] When running a script from the command line you will have to skip one or two calls, see traceback() for interactive and non-interactive R sessions
I'm not aware of another way to do this without the usual debugging suspects:
debug()
browser()
options(error=recover) [followed by options(error = NULL) to revert it]
You might want to look at this related post.
[Edit:] Sorry...just saw that you're running this from the command line. In that case I would suggest working with the options(error) functionality. Here's a simple example:
options(error = quote({dump.frames(to.file=TRUE); q()}))
You can create as elaborate a script as you want on an error condition, so you should just decide what information you need for debugging.
Otherwise, if there are specific areas you're concerned about (e.g. connecting to a database), then wrap them in a tryCatch() function.
Doing options(error=traceback) provides a little more information about the content of the lines leading up to the error. It causes a traceback to appear if there is an error, and for some errors it has the line number, prefixed by #. But it's hit or miss, many errors won't get line numbers.
Support for this will be forthcoming in R 2.10 and later. Duncan Murdoch just posted to r-devel on Sep 10 2009 about findLineNum and setBreapoint:
I've just added a couple of functions to R-devel to help with
debugging. findLineNum() finds which line of which function
corresponds to a particular line of source code; setBreakpoint() takes
the output of findLineNum, and calls trace() to set a breakpoint
there.
These rely on having source reference debug information in the code.
This is the default for code read by source(), but not for packages.
To get the source references in package code, set the environment
variable R_KEEP_PKG_SOURCE=yes, or within R, set
options(keep.source.pkgs=TRUE), then install the package from source
code. Read ?findLineNum for details on how to tell it to search
within packages, rather than limiting the search to the global
environment.
For example,
x <- " f <- function(a, b) {
if (a > b) {
a
} else {
b
}
}"
eval(parse(text=x)) # Normally you'd use source() to read a file...
findLineNum("<text>#3") # <text> is a dummy filename used by
parse(text=)
This will print
f step 2,3,2 in <environment: R_GlobalEnv>
and you can use
setBreakpoint("<text>#3")
to set a breakpoint there.
There are still some limitations (and probably bugs) in the code; I'll
be fixing thos
You do it by setting
options(show.error.locations = TRUE)
I just wonder why this setting is not a default in R? It should be, as it is in every other language.
Specifying the global R option for handling non-catastrophic errors worked for me, along with a customized workflow for retaining info about the error and examining this info after the failure. I am currently running R version 3.4.1.
Below, I've included a description of the workflow that worked for me, as well as some code I used to set the global error handling option in R.
As I have it configured, the error handling also creates an RData file containing all objects in working memory at the time of the error. This dump can be read back into R using load() and then the various environments as they existed at the time of the error can be inspected interactively using debugger(errorDump).
I will note that I was able to get line numbers in the traceback() output from any custom functions within the stack, but only if I used the keep.source=TRUE option when calling source() for any custom functions used in my script. Without this option, setting the global error handling option as below sent the full output of the traceback() to an error log named error.log, but line numbers were not available.
Here's the general steps I took in my workflow and how I was able to access the memory dump and error log after a non-interactive R failure.
I put the following at the top of the main script I was calling from the command line. This sets the global error handling option for the R session. My main script was called myMainScript.R. The various lines in the code have comments after them describing what they do. Basically, with this option, when R encounters an error that triggers stop(), it will create an RData (*.rda) dump file of working memory across all active environments in the directory ~/myUsername/directoryForDump and will also write an error log named error.log with some useful information to the same directory. You can modify this snippet to add other handling on error (e.g., add a timestamp to the dump file and error log filenames, etc.).
options(error = quote({
setwd('~/myUsername/directoryForDump'); # Set working directory where you want the dump to go, since dump.frames() doesn't seem to accept absolute file paths.
dump.frames("errorDump", to.file=TRUE, include.GlobalEnv=TRUE); # First dump to file; this dump is not accessible by the R session.
sink(file="error.log"); # Specify sink file to redirect all output.
dump.frames(); # Dump again to be able to retrieve error message and write to error log; this dump is accessible by the R session since not dumped to file.
cat(attr(last.dump,"error.message")); # Print error message to file, along with simplified stack trace.
cat('\nTraceback:');
cat('\n');
traceback(2); # Print full traceback of function calls with all parameters. The 2 passed to traceback omits the outermost two function calls.
sink();
q()}))
Make sure that from the main script and any subsequent function calls, anytime a function is sourced, the option keep.source=TRUE is used. That is, to source a function, you would use source('~/path/to/myFunction.R', keep.source=TRUE). This is required for the traceback() output to contain line numbers. It looks like you may also be able to set this option globally using options( keep.source=TRUE ), but I have not tested this to see if it works. If you don't need line numbers, you can omit this option.
From the terminal (outside R), call the main script in batch mode using Rscript myMainScript.R. This starts a new non-interactive R session and runs the script myMainScript.R. The code snippet given in step 1 that has been placed at the top of myMainScript.R sets the error handling option for the non-interactive R session.
Encounter an error somewhere within the execution of myMainScript.R. This may be in the main script itself, or nested several functions deep. When the error is encountered, handling will be performed as specified in step 1, and the R session will terminate.
An RData dump file named errorDump.rda and and error log named error.log are created in the directory specified by '~/myUsername/directoryForDump' in the global error handling option setting.
At your leisure, inspect error.log to review information about the error, including the error message itself and the full stack trace leading to the error. Here's an example of the log that's generated on error; note the numbers after the # character are the line numbers of the error at various points in the call stack:
Error in callNonExistFunc() : could not find function "callNonExistFunc"
Calls: test_multi_commodity_flow_cmd -> getExtendedConfigDF -> extendConfigDF
Traceback:
3: extendConfigDF(info_df, data_dir = user_dir, dlevel = dlevel) at test_multi_commodity_flow.R#304
2: getExtendedConfigDF(config_file_path, out_dir, dlevel) at test_multi_commodity_flow.R#352
1: test_multi_commodity_flow_cmd(config_file_path = config_file_path,
spot_file_path = spot_file_path, forward_file_path = forward_file_path,
data_dir = "../", user_dir = "Output", sim_type = "spot",
sim_scheme = "shape", sim_gran = "hourly", sim_adjust = "raw",
nsim = 5, start_date = "2017-07-01", end_date = "2017-12-31",
compute_averages = opt$compute_averages, compute_shapes = opt$compute_shapes,
overwrite = opt$overwrite, nmonths = opt$nmonths, forward_regime = opt$fregime,
ltfv_ratio = opt$ltfv_ratio, method = opt$method, dlevel = 0)
At your leisure, you may load errorDump.rda into an interactive R session using load('~/path/to/errorDump.rda'). Once loaded, call debugger(errorDump) to browse all R objects in memory in any of the active environments. See the R help on debugger() for more info.
This workflow is enormously helpful when running R in some type of production environment where you have non-interactive R sessions being initiated at the command line and you want information retained about unexpected errors. The ability to dump memory to a file you can use to inspect working memory at the time of the error, along with having the line numbers of the error in the call stack, facilitate speedy post-mortem debugging of what caused the error.
First, options(show.error.locations = TRUE) and then traceback(). The error line number will be displayed after #