I am using the Watson document conversion service. I have a set of .html files that I want to pass into the Watson API. I created two lists the first one (File_Name_List) contains the names of the .html files and the second (File_Name_List_TXT) contains the names of the .txt files I want the output of the Watson service to write to. Everytime I run the code below it gives me the following error: ValueError: I/O operation on closed file. I cannot figure out what the issue is - any help would be appreciated.
for file in File_Name_List:
for file_txt in File_Name_List_TXT:
f = open(file_txt,'w')
with open(join(dirname(__file__), file), 'r') as document: config = {'conversion_target': DocumentConversionV1.NORMALIZED_TEXT}
f.write(str(document_conversion.convert_document(document=document, config=config, media_type='text/plain').content).lower())
f.close()
Related
I am experimenting with the flow editor in PowerAutomate to merge a bunch of CSVs from OneDrive that I am syncing via rclone.
The structure is:
OneDrive(Root)/Folder/Subfolder/*.csv
I would like to merge them into a dataset (master CSV) that I can use with PowerBI.
Because this dataset is updated daily, new csvs will get added to the folder, thus my triggering event is "When a file is created"
The automation looks like this:
When a file is created >
Initialize a String Variable >
Find files in folder >
Search Query: *
Folder: Same as #1
FileSearch Mode: OneDriveSearch
Apply to each >
Get File content
Append to string variable
Compose (string variable) >
Create file >
File Path: whatever/path/
File Name: whatever.csv
File Contents: Outputs
The automation runs fine, and creates my master csv.
Except it's blank!
What's going on?
When a File is Created:
Initialize Variable:
Find Files in Folder:
Apply to Each:
Append String to Variable:
Compose:
Create a File:
At the end of the day, I get a CSV with some data written to it. *Previously, I thought it was blank, but it does, indeed have data.. but it appears truncated.
Something to note here... it looks like it retains the headers..
It looks like it grabbed the first 31 files, but there are 229 files in that folder.
Thanks in advance
I am creating a CSV file in an ADLS folder.
For example: sample.txt is the file name
instead of a single file, I see sample.txt/..,part-000 files.
My question is is there a method to create sample.txt file instead of a directory in pyspark.
df.write() or df.save() both create folders and multiple files inside that directory.
Using Coalesce(1) I can combine multiple part-000 files into one file. but how to create a single csv file?
Unfortunately, Spark doesn’t support creating a data file without a folder
To workaround,
Firstly using coalesce or repartition, create a single part (partition) file.
df\
.coalesce(1)\
.write\
.format("csv")\
.mode("overwrite")\
.save("mydata")
The above example produces an mydata directory, a part-000* file, and hidden files
However, our data is contained in only one CSV file. The name of this file is not user-friendly. We can rename this file and extract it.
data_location = "/mydata.csv/"
files = dbutils.fs.ls(data_location)
csv_file = [x.path for x in files if x.path.endswith(".csv")][0]
dbutils.fs.mv(csv_file, data_location.rstrip('/') + ".csv")
dbutils.fs.rm(data_location, recurse = True)
set up account key and configure the storage account to access. then move file from databricks location to adls. To move file, we use dbutils.fs.mv
```python
storage_account_name = "Storage account name"
storage_account_access_key = "storage account acesss key"
spark.conf.set("fs.azure.account.key."+storage_account_name+".blob.core.windows.net",storage_account_access_key)
dbutils.fs.cp('/mydata.csv.csv','abfss://demo12#pratikstorage1.dfs.core.windows.net//mydata1.csv')
My Execution:
Output:
I am using following command to generate HTML dashboard:
jmeter -g C:/Users/E01659/Desktop/feb28/feb28_10_3600.csv -o C:/Users/E01659/Downloads/apache-jmeter-5.1/bin/sonali/
Following error is coming: File
'C:\Users\E01659\Desktop\feb28\feb28_10_3600.csv' does not contain the
field names header, ensure the jmeter.save.saveservice.* properties
are the same as when the CSV file was created or the file may be read
incorrectly when generating report An error occurred: Mismatch between
expected number of columns:17 and columns in CSV file:11, check your
jmeter.save.saveservice.* configuration or check line is complete
errorlevel=1
From the error message, it is clear that - CSV file created earlier don't have the header as the first line. When you want to create the APDEX Dashboard from CSV file, JMeter expects that files are created with correct Save Service Configuration. You can change Result file config which are present in jmeter.propertes file.
You may need to change the line as below, so that file created has the headers name as required
jmeter.save.saveservice.label = true
The command you're using is correct, the reason of failure is mismatch of the .jtl file contents and what JMeter expects it to be.
If you got this feb28_10_3600.csv file from execution on another JMeter installation - make sure that JMeter's result save configuration on the machine where you're trying to generate the report is identical to the JMeter installation where JMeter was running.
Inspect the associated JMeter properties and amend them to for 100% match, once done you should be able to normally generate the dashboard.
If you still experience the differences update your question with first 2-3 lines of your .jtl results file.
You are receiving this error (most likely) due to a corrupt line at the end of your file. Just execute a 'tail ' to see the last few lines. Most likely, your last line is missing columns due to an abrupt stop in the test. Once you delete the problematic line, you should be able to generate the HTML report.
I am using twitter ruby gem to fetch twitter search result. The example code from Github extracts the information from search result.I am wondering how to save the search result, which is JSON i think, to a separate JSON file.
Here is part of the example code:
results = #search.perform("$aaa", 1000)
aFile = File.new("data.txt", "w")
results.map do |status|
myStr="#{status.from_user}: #{status.text} #{status.created_at}"
aFile.write(myStr)
aFile.write("\n")
end
Is there any way to save all the search result to a separate JSON file instead of writing strings to a file?
Thanks in advance.
If you want to save to a file all you need to do is open the file, write it it, then close it:
File.open("myFileName.txt", "a") do |mFile|
mFile.syswrite("Your content here")
mFile.close
end
When you use open you will create the file if it doesn't exist.
One thing to be aware of is that there are different ways to open file, of which will determine where the program writes to. The "a" indicates that it will append everything you write to the file, to the end of the current content.
Here is some of the options:
r Read-only mode. The file pointer is placed at the beginning of the file. This is the default mode.
r+ Read-write mode. The file pointer will be at the beginning of the file.
w Write-only mode. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
w+ Read-write mode. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
a Write-only mode. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
a+ Read and write mode. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.
So in your case, you would want to pull out the data you want to save, then write it to a file as I have shown. You can also specify file paths by doing:
File.open("/the/path/to/yourfile/myFileName.txt", "a") do |mFile|
mFile.syswrite("Your content here")
mFile.close
end
Another thing to be aware of is that open does not create directories, so you will either need to create directories yourself, or you can do it with your program. Here is a link that is helpful for file input/output:
http://www.tutorialspoint.com/ruby/ruby_input_output.htm
I am working on writing a ruby script to iterate through a file containing a list of file paths to open in Microsoft Excel. I read the file like this:
file_names = IO.readlines('D:\TEST_1\file_names.txt')
Next, I create an array of file names from each line of the parsed file (thus containing an array of file paths). Finally, I loop through that array with the following code, to open the documents:
require 'win32ole'
xl = WIN32OLE.new('Excel.Application')
xl.Visible = 1
file_names.each do |file_name|
wb1=xl.Workbooks.Open(file_name)
ws1=wb1.worksheets(1)
end
That first call to parse file_names.txt produces this exception, which I am having difficulty understanding:
Test4.rb:6:in 'method_missing'
OLE error code:800A03EC in Microsoft Office
Excel 'D:\Test_1\1.xlsx' couldnot be found. Check the spelling of
the file name, and verify that the file location is correct.
if you are trying to open the file from your list most recently used
files, make sure that the file has not been renamed, moved or deleted
HR Error code : 0x80020009 Exception occurred. from Test4.rb:6:in
'block in ' from Test4.rb:5:in 'each' from Test4.rb:5:in
''
This error does not appear when I pass a single file name (instead of a file path) as my parameter - so why do I get it here? Any help would be much appreciated.
At first look you are not using the variable "file_name" but a symbol :file_name.
file_array.each do |file_name|
wb1=xl.Workbooks.Open(file_name)
ws1=wb1.worksheets(1)
end