Thread Count and Batch Sizes
To best configure thread count and batch size, use the following principles:
Identify the ideal thread count for your environment depending on your system configuration. BDT gives optimal performance with 16 threads on four core machines. Setting thread count too low or too high will slow down the application.
Decide on batch size, configured based on available memory, size of individual record, and configured thread count. Setting batch size too high may result in an out-of-memory error.
Use the following formula to calculate the approximate maximum memory utilization by BDT:
thread count * batch size * record size = memory utilization
Therefore, thread count and batch size will be less than or equal to about 60% of available memory. The remaining memory is required to run the utility and to maintain transformed batches in the write queue.
Example
If you have 4GB available memory and individual record size of approximately 50MB, you can set either:
Batch size to 50 and thread count to 1, utilizing
50*1*50MB = 2500MB, approximately 60% of memory.Batch size to 10 and thread count to 3, utilizing
10*3*100MB = 3000MB.
