I'm trying to get some CSV data into a DataTable in order to create a line graph. The CSV has gone through a function to clean up the data prior to creating a new DataTable, the variable data2Array.
I only want to access 5 of the columns from data2Array. When I try to add a new row with the 5 data values, I get the following exception:
Uncaught Error: Row 29 has 1 columns, but must have 32
This has me confused, because I can see I've only tried to add the one row to data2Table, supplying only the 5 aforementioned values. Here is the log of what the row looks like:
0, Jul 1 2015, 3.379, 6.57, 0
data2Array has 32 columns and 29 rows, but in my DataTable, I've only specified for 5 columns before attempting to add a row. Since I am not adding anything to the original data source, but rather to data2Table, why is the exception mentioning row 29 or needing 32 columns? My new DataTable has been specified with only 5 columns.
Below is the code I am working with currently:
var data2Array = fullDataArray(csvData2);
console.log("data2 clean: " + data2Array);
// request line graph - data2
var data2Table = new google.visualization.DataTable(data2Array);
data2Table.addColumn('number', 'colIndex');
data2Table.addColumn('string', 'colLabel1');
data2Table.addColumn('string', 'colLabel2');
data2Table.addColumn('string', 'colLabel3');
data2Table.addColumn('string', 'colLabel4');
var testArr = [0, data2Array[0][0],data2Array[0][4], data2Array[0][5], data2Array[0][7]];
console.log('test: ' + testArr);
data2Table.addRow([0, data2Array[0][0],data2Array[0][4], data2Array[0][5], data2Array[0][7]]);
Related
Perhaps more a WORD issue than VBSCRIPT but my task is simple I just don't know what I'm doing.
The input file is and RTF with 23 tables separated by section breaks. All content is table except header and footer. I want to either convert each table to text one by or select all and convert all. I will ultimately save document as text file.
The following converts one table and then gives collection error.
1 Const RTF="t_14.4.3.1_lbshift.rtf"
2 Const WDFORMATCODE=2
3 Set objWord = CreateObject("Word.Application")
4 With objWord
5 .Visible = True
6 .DisplayAlerts = True
7 Set objDoc = .Documents.Open(rtf)
8 t0 = objDoc.Tables.Count
9 wscript.echo "T0=",t0
10 for i = 1 to objDoc.Tables.Count
11 wscript.echo i
12 .Selection.Tables(i).Select
13 .Selection.Rows.ConvertToText()
14 next
15 End With
T0= 23
1
2
Stderr output:
RTF2PDF.vbs(12, 7) Microsoft Word: The requested member of the collection does not exist.
I removed the header in my data but I am having troubles removing the first row. Basically I want to turn
1
2
3
4
5
into
2
3
4
5
I keep getting this error. I am using a number because it is the title of my gage.
"11481000" = "11481000"[-1,]
Error in "11481000"[-1, ] : incorrect number of dimensions
"11481000"[-1,]
Error in "11481000"[-1, ] : incorrect number of dimensions
View(11481000)
"11481000" <- "11481000" [-c(1), ]
Error in "11481000"[-c(1), ] : incorrect number of dimensions
TIA
My table test_orc contains (for one partition):
col1 col2 part1
abc def 1
ghi jkl 1
mno pqr 1
koi hai 1
jo pgl 1
hai tre 1
By running
hive --orcfiledump /hive/user.db/test_orc/part1=1/000000_0
I get the following:
Structure for /hive/a0m01lf.db/test_orc/part1=1/000000_0 .
2018-02-18 22:10:24 INFO: org.apache.hadoop.hive.ql.io.orc.ReaderImpl - Reading ORC rows from /hive/a0m01lf.db/test_orc/part1=1/000000_0 with {include: null, offset: 0, length: 9223372036854775807} .
Rows: 6 .
Compression: ZLIB .
Compression size: 262144 .
Type: struct<_col0:string,_col1:string> .
Stripe Statistics:
Stripe 1:
Column 0: count: 6 .
Column 1: count: 6 min: abc max: mno sum: 17 .
Column 2: count: 6 min: def max: tre sum: 18 .
File Statistics:
Column 0: count: 6 .
Column 1: count: 6 min: abc max: mno sum: 17 .
Column 2: count: 6 min: def max: tre sum: 18 .
Stripes:
Stripe: offset: 3 data: 58 rows: 6 tail: 49 index: 67 .
Stream: column 0 section ROW_INDEX start: 3 length 9 .
Stream: column 1 section ROW_INDEX start: 12 length 29 .
Stream: column 2 section ROW_INDEX start: 41 length 29 .
Stream: column 1 section DATA start: 70 length 20 .
Stream: column 1 section LENGTH start: 90 length 12 .
Stream: column 2 section DATA start: 102 length 21 .
Stream: column 2 section LENGTH start: 123 length 5 .
Encoding column 0: DIRECT .
Encoding column 1: DIRECT_V2 .
Encoding column 2: DIRECT_V2 .
What does the part about stripes mean?
First, let's see how an ORC file looks like.
Now some keywords used in above image and also in your question!
Stripe - A chunk of data stored in ORC file. Any ORC file is divided into those chunks, called stripes, each sized 250 MB with index data, actual data and some metadata for actual data stored in that stripe.
Compression - The compression codec used to compress the data stored. ZLIB is the default for ORC.
Index Data - includes min and max values for each column and the row positions within each column. (A bit field or bloom filter could also be included.) Row index entries provide offsets that enable seeking to the right compression block and byte within a decompressed block. Note that ORC indexes are used only for the selection of stripes and row groups and not for answering queries.
Row data - Actual data. Is used in table scans.
Stripe Footer - The stripe footer contains the encoding of each column and the directory of the streams including their location. To describe each stream, ORC stores the kind of stream, the column id, and the stream’s size in bytes. The details of what is stored in each stream depends on the type and encoding of the column.
Postscript - holds compression parameters and the size of the compressed footer.
File Footer - The file footer contains a list of stripes in the file, the number of rows per stripe, and each column's data type. It also contains column-level aggregates count, min, max, and sum.
Now! Talking about your output from orcfiledump.
First is general information about your file. The name, location, compression codec, compression size etc.
Stripe statistics will list all the stripes in your ORC file and its corresponding information. You can see counts and some statistics about Integer columns like min, max, sum etc.
File statistics is similar to #2. Just for the complete file as opposed to each stripe in #2.
Last part, the Stripe section, talks about each column in your file and corresponding index info for each of it.
Also, you can use various options with orcfiledump to get "desired" results. Follows a handy guide.
// Hive version 0.11 through 0.14:
hive --orcfiledump <location-of-orc-file>
// Hive version 1.1.0 and later:
hive --orcfiledump [-d] [--rowindex <col_ids>] <location-of-orc-file>
// Hive version 1.2.0 and later:
hive --orcfiledump [-d] [-t] [--rowindex <col_ids>] <location-of-orc-file>
// Hive version 1.3.0 and later:
hive --orcfiledump [-j] [-p] [-d] [-t] [--rowindex <col_ids>] [--recover] [--skip-dump]
[--backup-path <new-path>] <location-of-orc-file-or-directory>
Follows a quick guide to the options used in the commands above.
Specifying -d in the command will cause it to dump the ORC file data
rather than the metadata (Hive 1.1.0 and later).
Specifying --rowindex with a comma separated list of column ids will
cause it to print row indexes for the specified columns, where 0 is
the top level struct containing all of the columns and 1 is the first
column id (Hive 1.1.0 and later).
Specifying -t in the command will print the timezone id of the
writer.
Specifying -j in the command will print the ORC file metadata in JSON
format. To pretty print the JSON metadata, add -p to the command.
Specifying --recover in the command will recover a corrupted ORC file
generated by Hive streaming.
Specifying --skip-dump along with --recover will perform recovery
without dumping metadata.
Specifying --backup-path with a new-path will let the recovery tool
move corrupted files to the specified backup path (default: /tmp).
is the URI of the ORC file.
is the URI of the ORC file or
directory. From Hive 1.3.0 onward, this URI can be a directory
containing ORC files.
Hope that helps!
I have a Core Data model that includes a Car entity and let's say an EngineData entity. This is a one-to-one relationship.
In a new version of my app I want to add trucks. So I've created a new version of my Core Data model. I now have a Vehicle entity that is the parent entity of Car. I've added a new Truck entity that also has Vehicle as its parent entity. For EngineData, the inverse relationship was generically named object, so the destination entity just changes from Car to Vehicle.
I wasn't entirely sure this would work with lightweight migration, but I first changed it a couple of weeks ago, and until now it seemed fine. I have code that fetches and updates existing data from EngineData using a Car's engineData property, and I haven't seen any issues there. However, there's a single Core Data fetch in my app that causes a crash every time. All I have to do to cause the crash is a simple fetch of all my EngineData objects:
do {
let request: NSFetchRequest<EngineData> = EngineData.fetchRequest()
let objects = try context.fetch(request)
} catch {
NSLog("Error fetching data: \(error)")
}
On the context.fetch line, I get an exception:
[error] error: Background Core Data task threw an exception. Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: error: Background Core Data task threw an exception. Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
And if I try to actually do anything with those objects, I get some more exceptions until the app crashes:
[General] An uncaught exception was raised
[General] *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10]
0 CoreFoundation 0x00007fff8861937b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff9d40d48d objc_exception_throw + 48
2 CoreFoundation 0x00007fff88532b5c -[__NSArrayM objectAtIndex:] + 204
3 CoreData 0x00007fff881978ed -[NSSQLRow newObjectIDForToOne:] + 253
4 CoreData 0x00007fff8819770f -[NSSQLRow _validateToOnes] + 399
5 CoreData 0x00007fff88197571 -[NSSQLRow knownKeyValuesPointer] + 33
6 CoreData 0x00007fff88191868 _prepareResultsFromResultSet + 4312
7 CoreData 0x00007fff8818e47b newFetchedRowsForFetchPlan_MT + 3387
8 CoreData 0x00007fff8835f6d7 _executeFetchRequest + 55
9 CoreData 0x00007fff8828bb35 -[NSSQLFetchRequestContext executeRequestUsingConnection:] + 53
10 CoreData 0x00007fff8832c9c8 __52-[NSSQLDefaultConnectionManager handleStoreRequest:]_block_invoke + 216
11 libdispatch.dylib 0x000000010105478c _dispatch_client_callout + 8
12 libdispatch.dylib 0x00000001010555ad _dispatch_barrier_sync_f_invoke + 307
13 CoreData 0x00007fff8832c89d -[NSSQLDefaultConnectionManager handleStoreRequest:] + 237
14 CoreData 0x00007fff88286c86 -[NSSQLCoreDispatchManager routeStoreRequest:] + 310
15 CoreData 0x00007fff88261189 -[NSSQLCore dispatchRequest:withRetries:] + 233
16 CoreData 0x00007fff8825e21d -[NSSQLCore processFetchRequest:inContext:] + 93
17 CoreData 0x00007fff8817d218 -[NSSQLCore executeRequest:withContext:error:] + 568
18 CoreData 0x00007fff882436de __65-[NSPersistentStoreCoordinator executeRequest:withContext:error:]_block_invoke + 5486
19 CoreData 0x00007fff8823a347 -[NSPersistentStoreCoordinator _routeHeavyweightBlock:] + 407
20 CoreData 0x00007fff8817cd9e -[NSPersistentStoreCoordinator executeRequest:withContext:error:] + 654
21 CoreData 0x00007fff8817af51 -[NSManagedObjectContext executeFetchRequest:error:] + 593
22 libswiftCoreData.dylib 0x0000000100c91648 _TFE8CoreDataCSo22NSManagedObjectContext5fetchuRxSo20NSFetchRequestResultrfzGCSo14NSFetchRequestx_GSax_ + 56
I thought the flag -com.apple.CoreData.SQLDebug 1 might give me some useful information, but I don't see anything very helpful here:
CoreData: annotation: Connecting to sqlite database file at "/Users/name/Library/Group Containers/Folder/Database.sqlite"
CoreData: sql: pragma recursive_triggers=1
CoreData: sql: pragma journal_mode=wal
CoreData: sql: SELECT Z_VERSION, Z_UUID, Z_PLIST FROM Z_METADATA
CoreData: sql: SELECT TBL_NAME FROM SQLITE_MASTER WHERE TBL_NAME = 'Z_MODELCACHE'
CoreData: sql: SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZDATA, t0.ZOBJECT, t0.Z9_OBJECT FROM ZENGINEDATA t0
2017-04-28 16:18:41.693548-0400 AppName[95979:11442888] [error] error: Background Core Data task threw an exception. Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: error: Background Core Data task threw an exception. Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: annotation: sql connection fetch time: 0.0065s
CoreData: annotation: fetch using NSSQLiteStatement <0x6080004805a0> on entity 'ENGINEDATA' with sql text 'SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZDATA, t0.ZOBJECT, t0.Z9_OBJECT FROM ZENGINEDATA t0 ' returned 1185 rows with values: (
"<JUNENGINEDATA: 0x608000480eb0> (entity: ENGINEDATA; id: 0x40000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1> ; data: <fault>)",
"<JUNENGINEDATA: 0x608000480f00> (entity: ENGINEDATA; id: 0x80000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p2> ; data: <fault>)",
"<JUNENGINEDATA: 0x608000480f50> (entity: ENGINEDATA; id: 0xc0000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p3> ; data: <fault>)",
...
"<JUNENGINEDATA: 0x600000288110> (entity: ENGINEDATA; id: 0x128c0000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1187> ; data: <fault>)",
"<JUNENGINEDATA: 0x600000288160> (entity: ENGINEDATA; id: 0x12900000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1188> ; data: <fault>)",
"<JUNENGINEDATA: 0x6000002881b0> (entity: ENGINEDATA; id: 0x12940000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1189> ; data: <fault>)"
)
CoreData: annotation: total fetch execution time: 0.1053s for 1185 rows.
2017-04-28 16:18:41.793201-0400 AppName[95979:11442649] Got results: 1185
The good news is everything in EngineData can be recreated, and I've found that if I do a batch delete of all EngineData objects, it no longer crashes (even after creating some new objects). If possible I'd greatly prefer to understand the cause of the problem though, and find a less drastic solution.
Here are some other things I've discovered:
I tried copying a database from another computer, and I was no able to duplicate the problem using that data. That got me thinking maybe the data was corrupt to begin with…
But if I go back to an old version of my app and its data, the same fetch works fine. If I do nothing other than upgrade the data model, I get the same exceptions.
I've tried setting a version hash modifier for the relationship, hoping that might force Core Data to clean things up when it migrates, but no luck.
If I set a batch size of 10, it's able to loop through 900 results (out of 1185) before the exception.
Using that process, I can delete good records one at a time to narrow down which ones are problematic. (I'm saving after each delete, and I kept a backup of the original database if I need to go back for further testing.)
After a lot of experimenting I was able to narrow it down: I have a handful of EngineData objects that reference a Car object that no longer exists. It looks like since that relationship is broken, the records aren't migrated correctly to the new data model version. If I open the .sqlite file in the Mac app Base, I can see a new Z9_OBJECT field that's set to 10 for all the good records, and NULL for the damaged ones.
Fixing the problem was fairly easy once I discovered the cause. Using NSBatchDeleteRequest I was able to find and delete the damaged objects:
do {
let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "EngineData")
fetchRequest.predicate = NSPredicate(format: "object.uuid == nil")
let batchRequest = NSBatchDeleteRequest(fetchRequest: fetchRequest)
try context.execute(batchRequest)
} catch {
NSLog("Error deleting objects: \(error)")
}
In my predicate, object is the parent object (previously a Car, now a Vehicle), and uuid is a non-optional attribute on that object. As long as the attribution isn't optional—meaning it will have a value on any undamaged object—this should successfully delete only the objects that are damaged.
You might be expecting this to cause an exception, since it's still fetching the damaged objects! Fortunately NSBatchDeleteRequest works by running SQL statements directly on the store, without loading anything into memory—so it avoids the problem.
I am trying to plot Num/Den type percentages using OVER. But my thoughts does not appear to translate into spotfire custom expression syntax.
Sample Input:
RecordID CustomerID DOS Age Gender Marker
9621854 854693 09/22/15 37 M D
9732721 676557 09/18/15 65 M D
9732700 676557 11/18/15 65 M N
9777003 5514882 11/25/15 53 M D
9853242 1753256 09/30/15 62 F D
9826842 1260021 09/30/15 61 M D
9897642 3375185 09/10/15 74 M N
9949185 9076035 10/02/15 52 M D
10088610 3512390 09/16/15 33 M D
10120650 41598 10/11/15 67 F N
9949185 9076035 10/02/15 52 M D
10088610 3512390 09/16/15 33 M D
10120650 41598 09/11/15 67 F N
Expected Out:
Row Labels D Cumulative_D N Cumulative_N Percentage
Sep 6 6 2 2 33.33%
Oct 2 8 1 3 37.50%
Nov 1 9 1 4 44.44%
My counts are working.
I want to take the same Cumulative_N & Cumulative_D count and plot Percentage over [Axis.X] as a line chart.
Here's what I am using:
UniqueCount(If([Marker]="N",[CustomerID])) / UniqueCount(If([Marker]="D",[CustomerID])) THEN SUM([Value]) OVER (AllPrevious([Axis.X])) as [CumulativePercent]
I understand SUM([Value]) is not the way to go. But I don't know what to use instead.
Also tried the one below as well, but did not:
UniqueCount(If([Marker]="N",[CustomerID])) OVER (AllPrevious([Axis.X])) / UniqueCount(If([Marker]="D",[CustomerID])) OVER (AllPrevious([Axis.X])) as [CumulativePercent]
Can you have a look ?
I found a way to make it work, but it may not fit your overall solution. I should mention i used Count() versus UniqueCount() so that the results would mirror your desired output.
Add a transformation to your current data table
Insert a calculated column Month([DOS]) as [TheMonth]
Set Row Identifers = [TheMonth]
Set value columns and aggregation methods to Count([CustomerID])
Set column titles to [Marker]
Leave the column name pattern as %M(%V) for %C
That will give you a new data table. Then, you can do your cumulative functions. I did them in a cross table to replicate your expected results. Insert a new cross table and set the values to:
Sum([D]), Sum([N]), Sum([D]) OVER (AllPrevious([Axis.Rows])) as [Cumulative_D],
Sum([N]) OVER (AllPrevious([Axis.Rows])) as [Cumulative_N],
Sum([N]) OVER (AllPrevious([Axis.Rows])) / Sum([D]) OVER (AllPrevious([Axis.Rows])) as [Percentage]
That should do it.
I don't know if Spotfire released a fix or based on everyone's inputs I could get the syntax right. But here is the solution that worked for me.
For Columns D & N,
COUNT([CustomerID])
For columns Cumulative_D & Cumulative_N,
Count([CustomerID]) OVER (AllPrevious([Axis.X])) where [Axis.X] is DOS(Month), Marker
For column Percentage,
Count(If([Marker]="N",[CustomerID])) OVER (AllPrevious([Axis.X])) / Count(If([Marker]="D",[CustomerID])) OVER (AllPrevious([Axis.X]))
where [Axis.X] is DOS(Month)