Author: Bill Woodger
Subject: Reply to: How to split the records using the amount field
Posted: Tue Nov 01, 2016 5:32 pm (GMT 5.5)
My main concern with this topic is the data. Asked for a representative sample, data was only shown which is exactly divisible by 250000 (despite the example in the first post).
If that second set of sample data is representative, then the task is easy, one step, no ICETOOL.
You note that all the values on the first output file are 250000, and on the first OUTFIL output 1-10 records (using the slash-operator) depending on the amount (if greater than 2.5 million, output 10, 2.25 million, output nine, etc).
On the second OUTFIL, INCLUDE= for GT 2.5 million, and subtract 2.5 million to give the residual value.
If, as I suspect, the numbers are not nice and "round", it doesn't take much to make the final one a calculation on the first OUTFIL (subtract the value "below" the one you are testing for).
With all the references to the same fields and constants, I'd definitely show the solution with symbols.
But then, do I want to prepare and test all that, only to have TS/OP explain further that the description of the data is not quite right?
Lots of IFTHENs, lots of code to create, only two calculations at maximum per record.
For a different requirement, I'd go for RESIZE over two passes of the data, but Arun is correct, for any given solution it is only known to perform better or worse than another with the actual data. Generally I'd expect the RESIZE to work better, but no guarantees, as it does depend on what goes along with it, and can, not so much in this example, depend on the data.
As to "over engineering" there's a certain amount of that in both proposed solutions :-)
All suggested solutions would suffer from an increase to 50 splits, but all the code for all the solutions could be "generated" should such a thing arise.
My specific advice is to wait for/obtain clarity from TS/OP before getting to code...
Subject: Reply to: How to split the records using the amount field
Posted: Tue Nov 01, 2016 5:32 pm (GMT 5.5)
My main concern with this topic is the data. Asked for a representative sample, data was only shown which is exactly divisible by 250000 (despite the example in the first post).
If that second set of sample data is representative, then the task is easy, one step, no ICETOOL.
You note that all the values on the first output file are 250000, and on the first OUTFIL output 1-10 records (using the slash-operator) depending on the amount (if greater than 2.5 million, output 10, 2.25 million, output nine, etc).
On the second OUTFIL, INCLUDE= for GT 2.5 million, and subtract 2.5 million to give the residual value.
If, as I suspect, the numbers are not nice and "round", it doesn't take much to make the final one a calculation on the first OUTFIL (subtract the value "below" the one you are testing for).
With all the references to the same fields and constants, I'd definitely show the solution with symbols.
But then, do I want to prepare and test all that, only to have TS/OP explain further that the description of the data is not quite right?
Lots of IFTHENs, lots of code to create, only two calculations at maximum per record.
For a different requirement, I'd go for RESIZE over two passes of the data, but Arun is correct, for any given solution it is only known to perform better or worse than another with the actual data. Generally I'd expect the RESIZE to work better, but no guarantees, as it does depend on what goes along with it, and can, not so much in this example, depend on the data.
As to "over engineering" there's a certain amount of that in both proposed solutions :-)
All suggested solutions would suffer from an increase to 50 splits, but all the code for all the solutions could be "generated" should such a thing arise.
My specific advice is to wait for/obtain clarity from TS/OP before getting to code...