I am trying to create a sequence where a folder where multiple subdirectories are present, Each subdirectory has 2 Excel files with multiple sheets (Sheet1, Sheet2).
I am looking a way to merge both the Excel files from each repository wrt.
to their respective sheets. For Eg: Sheet1 data needs to be merged with Sheet1, Sheet2 with Sheet2, and finally give an output Excel file containing both the sheets with merged data.
I have tried using UIPATH_ExcelMerge, BUt its not working in my case.
I assumed that the sheets have the same schema, so you could iterate all Excel files and for each worksheet use "Read range" to store the data into a temp data table. Then you could merge the data tables of the same worksheets together with the "Merge data table" activity. Finally, write the merged data table into the output Excel file.
Something like this:
Regards, Gio
Related
I have a list of VCF files from specific ethnicity such as American Indian, Chinese, European, etc
Under each ethnicity, I have around 100+ files.
Currently, I computed the VARIANT QC metrics such as call_rate, n_het etc for one file as shown in the hail tutorial (refer image below)
image is here
However, now I would like to have one file for each ethnicity and then compute VARIANT_QC metrics.
I already referred to this post and this post but don't think this addresses my query
How can I do this across all files under a specific ethnicity?
Can help me with this?
Is there any hail/python/R/other tools way to do this?
You could use Variant Transforms to achieve this goal. Variant Transforms is a tool for parsing and importing VCF files into BigQuery. It also can perform the reverse transform: export variants stored in BigQuery tables to VCF file. So basically you need to: multiple VCF files -> BigQuery -> Single VCF file
Variant Transforms can easily handle multiple input files. It also can perform more complex logic to merge same variants across multiple files into the same record. After your variants are all loaded into BigQuery you could export them to VCF file.
Note that Variant Transforms creates a separate table for each chromosome to optimize query costs. You can easily create a VCF file for each chromosome and then merge them together to create a single one.
You can reach out to Variant Transforms team if you need help with this task.
I have multiple excel files with two types of metadata, Now i have to push the data into two different tables based on metadata of excel files using SSIS.
There are many, many different ways to do this. You'd need to share a lot more information on how your data is structured to really give a great answer, but here's the general strategy I'd suggest.
In the control flow tab, have a separate data flow for each Excel file. The data flows will all work the same, with the exception of having a different Excel source in each data flow, so it will be enough to get the first version working and then copy and paste for the other files.
In the data flow, use a conditional split transformation to read the metadata coming from Excel and send the row to the correct table.
If you really want to be fancy, however, you could create a child package that includes all your data flow logic. Using the Execute Package Task you can pass the Excel file name to the child package for each Excel file you need to import. This way you consolidate your logic in one package and can still import from multiple Excel files in parallel.
how to work on specific part of cvs file uploaded into HDFS ?
I'm new in Hadoop and i have an a question that is if i export an a relational database into cvs file then uploaded it into HDFS . so how to work on specific part (table) in file using MapReduce .
thanks in advance .
I assume that the RDBMS tables are exported to individual csv files for each table and stored in HDFS. I presume that, you are referring to column(s) data within the table(s) when you mentioned 'specific part (table)'. If so, place the individual csv files into the separate file paths say /user/userName/dbName/tables/table1.csv
Now, you can configure the job for the input path and field occurrences. You may consider to use the default Input Format so that your mapper would get one line at time as input. Based on the configuration/properties, you can read the specific fields and process the data.
Cascading allows you to get started very quickly with MapReduce. It has framework that allows you to set up Taps to access sources (your CSV file) and process it inside a pipeline say to (for example) add column A to column B and place the sum into column C by selecting them as Fields
use BigTable means convert your database to one big table
Can any one help me out how can i merge excel files having same columns which are saved in one folder to one destination excel file using SQL Server Integration services(SSIS) .
add an Excel source for each of your files an a "union all" task to join them, so if you have 20 rows on your first excel and 30 on the second, you will end up with 50 rows:
to set the source and destination paths, just double click each of the tasks and set the connection manager
EDIT:
example of how to read an excel file here
I am new to Informatica so need your help.
I have one staging table where data comes everyday and I need to extract data from this staging table and convert it into Dat file format and place in into a folder. so that these dat files could be a feed for another process.
I dont know how informatica does this (Conversion of data from Staging table to Dat). So please help me to know how Informatica fetch the data from staging table, transform it into Dat file and place it into a folder.
Thanks & Regards,
Vikram
To create a pipe-delimited flat file...
Go to the Target Designer - Select Target->Create then choose Flat File. Then double click on the file, and in the 'Table' tab, at the bottom right select 'Advanced' and choose your delimiter. Then you can add your columns, specify the file location and all is well!
You will need to define a source definition based on your staging table, a target definition based on your final file format and then create the mapping,session and workflow that link the two.
.Dat file is not a complete description for the file, since any file can be renamed to a .dat file. You'll need to decide how the data would be separated in this file (commas? tabs? pipes?). Remember all downstream processes will then use this file as input, so you need to publish this format too.