My folder structure with 2000 folder and every folder having 2000 csv file(every csv file size:upto 3KB).
But my problem is when i copy paste whole folder to other place it takes about 6 hours.
So i need to minimize that time using batch file.I want to know why it takes like this much and any idea to solve this problem.
data
|-----LrK90001
| |----LrK90001.csv
| |----LrK90001_e.csv
| |----LrK90002.csv
| |----LrK90002_e.csv
| .
| .
| .
| |----LrK91000_e.csv
|
|-----LrK90002
| |----LrK90001.csv
| |----LrK90001_e.csv
| |----LrK90002.csv
| |----LrK90002_e.csv
| .
| .
| .
| |----LrK91000_e.csv
.
.
.
|-----LrK92000
| |----LrK90001.csv
| |----LrK90001_e.csv
| |----LrK90002.csv
| |----LrK90002_e.csv
| .
| .
| .
| |----LrK91000_e.csv
|
My pc spec:
Processor - Intel i5
RAM - 16GB
Related
currently, i'm working with project that has some modules which need to run schedule job.
I want to take schedule job into Kernel of each module (not Kernel in app/Console directory).
I did like this Laravel 5 Package Scheduled Tasks but it only run on 1 module. The others did not run.
Can anybody please help me! Thanks
My source code has structure like this:
app
| Console
| | Commands
| | | Command.php
| | Kernel.php
bootstrap
...
Module
| Module 1
| | Console
| | | Commands
| | | | Command11.php
| | | | Command12.php
| | | Kernel.php
| Module 2
| | Console
| | | Commands
| | | | Command21.php
| | | | Command22.php
| | | Kernel.php
I'm currently implementing the execv syscall into my own kernel. However, I managed to calculate the argument count(argc) in user space and passed argc and argv to my kernel exec function via a system call.
Actually, I don't know how to pass these two variables now to the new user threads main function. I allocated a new stack page and stored argc and argv in sequence onto this page space. My I idea was to point either EDI or ESI user register to the first argument (argc) followed by the argv array.
I'm not sure if it is correct to use EDI or ESI for this purpose. Does anybody know how the arguments get passed inside other kernels?
You need to setup the user stack properly and then the entry point of the ELF executable would be responsible for locating argc and argv on the stack. The function in the Linux kernel responsible for seting up the stack is create_elf_tables. For a visual representation, you can check out this nice figure from this great LWN.net article:
------------------------------------------------------------- 0x7fff6c845000
0x7fff6c844ff8: 0x0000000000000000
_ 4fec: './stackdump\0' <------+
env / 4fe2: 'ENVVAR2=2\0' | <----+
\_ 4fd8: 'ENVVAR1=1\0' | <---+ |
/ 4fd4: 'two\0' | | | <----+
args | 4fd0: 'one\0' | | | <---+ |
\_ 4fcb: 'zero\0' | | | <--+ | |
3020: random gap padded to 16B boundary | | | | | |
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -| | | | | |
3019: 'x86_64\0' <-+ | | | | | |
auxv 3009: random data: ed99b6...2adcc7 | <-+ | | | | | |
data 3000: zero padding to align stack | | | | | | | |
. . . . . . . . . . . . . . . . . . . . . . . . . . .|. .|. .| | | | | |
2ff0: AT_NULL(0)=0 | | | | | | | |
2fe0: AT_PLATFORM(15)=0x7fff6c843019 --+ | | | | | | |
2fd0: AT_EXECFN(31)=0x7fff6c844fec ------|---+ | | | | |
2fc0: AT_RANDOM(25)=0x7fff6c843009 ------+ | | | | |
ELF 2fb0: AT_SECURE(23)=0 | | | | |
auxiliary 2fa0: AT_EGID(14)=1000 | | | | |
vector: 2f90: AT_GID(13)=1000 | | | | |
(id,val) 2f80: AT_EUID(12)=1000 | | | | |
pairs 2f70: AT_UID(11)=1000 | | | | |
2f60: AT_ENTRY(9)=0x4010c0 | | | | |
2f50: AT_FLAGS(8)=0 | | | | |
2f40: AT_BASE(7)=0x7ff6c1122000 | | | | |
2f30: AT_PHNUM(5)=9 | | | | |
2f20: AT_PHENT(4)=56 | | | | |
2f10: AT_PHDR(3)=0x400040 | | | | |
2f00: AT_CLKTCK(17)=100 | | | | |
2ef0: AT_PAGESZ(6)=4096 | | | | |
2ee0: AT_HWCAP(16)=0xbfebfbff | | | | |
2ed0: AT_SYSINFO_EHDR(33)=0x7fff6c86b000 | | | | |
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | | | | |
2ec8: environ[2]=(nil) | | | | |
2ec0: environ[1]=0x7fff6c844fe2 ------------------|-+ | | |
2eb8: environ[0]=0x7fff6c844fd8 ------------------+ | | |
2eb0: argv[3]=(nil) | | |
2ea8: argv[2]=0x7fff6c844fd4 ---------------------------|-|-+
2ea0: argv[1]=0x7fff6c844fd0 ---------------------------|-+
2e98: argv[0]=0x7fff6c844fcb ---------------------------+
0x7fff6c842e90: argc=3
At the first instruction of the entry point of the ELF binary (glibc calls it _start), the top of the stack (0(%rsp)) would point to argc, as shown in the figure. Then follows the argv array, which is just an array of pointers. The environment variables array is also part of the ABI, as well as the ELF auxiliary vector (which may be used by the glibc initialization code). Although if you know for sure (or assume) that the ELF binary is not going to use the environment variables array or the auxiliary vector, then you just setup argv and argc as shown in the figure.
I have a massive chunk of files in an extremely fast SAN disk that I like to do Hive query on them.
An obvious option is to copy all files into HDFS by using a command like this:
hadoop dfs -copyFromLocal /path/to/file/on/filesystem /path/to/input/on/hdfs
However, I don't want to create a second copy of my files, just to be to Hive query in them.
Is there any way to point an HDFS folder into a local folder, such that Hadoop sees it as an actual HDFS folder? The files keep adding to the SAN disk, so Hadoop needs to see the new files as they are being added.
This is similar to Azure's HDInsight approach that you copy your files into a blob storage and HDInsight's Hadoop sees them through HDFS.
For playing around with small files using the local file system might be fine, but I wouldn't do it for any other purpose.
Putting a file in an HDFS means that it is being split to blocks which are replicated and distributed.
This gives you later on both performance and availability.
Locations of [external] tables can be directed to the local file system using file:///.
Whether it works smoothly or you'll start getting all kinds of error, that's to be seen.
Please note that for the demo I'm doing here a little trick to direct the location to a specific file, but your basic use will probably be for directories.
Demo
create external table etc_passwd
(
Username string
,Password string
,User_ID int
,Group_ID int
,User_ID_Info string
,Home_directory string
,shell_command string
)
row format delimited
fields terminated by ':'
stored as textfile
location 'file:///etc'
;
alter table etc_passwd set location 'file:///etc/passwd'
;
select * from etc_passwd limit 10
;
+----------+----------+---------+----------+--------------+-----------------+----------------+
| username | password | user_id | group_id | user_id_info | home_directory | shell_command |
+----------+----------+---------+----------+--------------+-----------------+----------------+
| root | x | 0 | 0 | root | /root | /bin/bash |
| bin | x | 1 | 1 | bin | /bin | /sbin/nologin |
| daemon | x | 2 | 2 | daemon | /sbin | /sbin/nologin |
| adm | x | 3 | 4 | adm | /var/adm | /sbin/nologin |
| lp | x | 4 | 7 | lp | /var/spool/lpd | /sbin/nologin |
| sync | x | 5 | 0 | sync | /sbin | /bin/sync |
| shutdown | x | 6 | 0 | shutdown | /sbin | /sbin/shutdown |
| halt | x | 7 | 0 | halt | /sbin | /sbin/halt |
| mail | x | 8 | 12 | mail | /var/spool/mail | /sbin/nologin |
| uucp | x | 10 | 14 | uucp | /var/spool/uucp | /sbin/nologin |
+----------+----------+---------+----------+--------------+-----------------+----------------+
You can mount your hdfs path into local folder, for example with hdfs mount
Please follow this for more info
But if you want speed, it isn't an option
Database: Oracle 12c
I want to take single partition, or a set of partitions, disconnect it from a Table, or set of tables on DB1 and move it to another table on another database. I would like to avoid doing DML to do this for performance reasons (It needs to be fast).
Each Partition will contain between three and four hundred million records.
Each Partition will be broken up into approximately 300 Sub-Partitions.
The task will need to be automated.
Some thoughts I had:
Somehow put each partition in it's own datafile upon creation, then detaching from the source and attaching it to the destination?
Extract the whole partition (not record-by-record)
Any other non-DML Solutions are also welcom
Example (Move Part#33 from both to DB#2, preferably with a single, operation):
__________________ __________________
| DB#1 | | DB#2 |
|------------------| |------------------|
|Table1 | |Table1 |
| Part#1 | | Part#1 |
| ... | | ... |
| Part#33 | ----> | Part#32 |
| Subpart#1 | | |
| ... | | |
| Subpart#300 | | |
|------------------| |------------------|
|Table2 | |Table2 |
| Part#1 | | Part#1 |
| ... | | ... |
| Part#33 | ----> | Part#32 |
| Subpart#1 | | |
| ... | | |
| Subpart#300 | | |
|__________________| |__________________|
Please read the document below with all the examples of exchanging partitions of table.
https://oracle-base.com/articles/misc/partitioning-an-existing-table-using-exchange-partition
I have a .cql query file that I want to run from the Neo4jShell (Windows) with the command:
Neo4j> Neo4jShell -file query.cql
The query returns some rows of data. How can I write that query output into a .csv or .txt file from the shell?
Also, I am using the Windows command prompt so take that into consideration with any solutions. Thanks!
UPDATE 1:
The command suggested by Luane essentially works:
Neo4j> Neo4jShell -file query.cql > out.csv
The only issue is that the output isn't comma separated:
+--------------------------+
| column 1 | column 2 |
+--------------------------+
| "C1611640" | "C1265875" |
| "C1579268" | "C1265875" |
| "C1570906" | "C1265875" |
| "C1522577" | "C1265875" |
| "C1519033" | "C1265875" |
| "C1515119" | "C1265875" |
| . . |
| . . |
| . . |
| "C1533658" | "C1265875" |
| "C1532338" | "C1265875" |
| "C1527144" | "C1265875" |
+--------------------------+
2000 rows
219 ms
Assuming your query returns data in the format you require, you can just send the output to any file. This works on MacOS, but I don't see why it should not work on Windows:
> neo4j-shell -file query.cql > out.txt