Author: aswinir
Posted: Mon Dec 12, 2016 3:39 pm (GMT 5.5)
Its only 1 program.
10m file is the driver file in which id is key record. each id record in parent file has 6 sub records in the 90m file. out of which, the program picks only 3 sub records which are not contiguous. Also the driver file reads 70k records
file to pick few data from this.
Joined the keys of 70K file and 10M file based on the keys and removed duplicates from 70K file. (consider file A)
Joined keys of 10M file and 90M file based on keys and include only the records that are in need. (Consider file B)
Changed the program to drive with new file A and read the 10M file in sequential mode and then reading file B sequentially based on the records in 10 M file.
Note: I have not tested with full set of records. for POC purpose, only 8M records were considered for testing out of 90M. i.e. 8 Million before program changes and with new code. this gave more than 40% improvement in CPU.
Posted: Mon Dec 12, 2016 3:39 pm (GMT 5.5)
Its only 1 program.
10m file is the driver file in which id is key record. each id record in parent file has 6 sub records in the 90m file. out of which, the program picks only 3 sub records which are not contiguous. Also the driver file reads 70k records
file to pick few data from this.
Joined the keys of 70K file and 10M file based on the keys and removed duplicates from 70K file. (consider file A)
Joined keys of 10M file and 90M file based on keys and include only the records that are in need. (Consider file B)
Changed the program to drive with new file A and read the 10M file in sequential mode and then reading file B sequentially based on the records in 10 M file.
Note: I have not tested with full set of records. for POC purpose, only 8M records were considered for testing out of 90M. i.e. 8 Million before program changes and with new code. this gave more than 40% improvement in CPU.