Max number of steps

[pkernevez]

Hi all,

I'm creating a batch in a banking context, that recreates Accounts' positions.
During the nominal situation the batch recompute only the missing positions since the last execution, meaning 1 to 4 days depending on WE and closed days.
But on some situation like recreating a new environnement, we may have to recompute 2 years (or more) of positions.

Our first approach was to only have a batch working with chunk of x (~50) accounts and recompute for each account all the missing positions since the last execution.
It works well with the nominal situation, but with a workload of 2+ years, the hibernate session exploses when faces to the unusual amount of data.

I identify 2 alternative approaches:

1. change the chunk size depending on the duration. Like (50/(number of month)) to reduce the size of chunks.
   It reduces the mem pressure, but reduce the efficiency of the batch, because each chunk have to recompute & reload all the context (prices & fx rates). When we are recomputing 5 years at once, it's not really a batch, it becomes a X times transactions. Without cache or optimisations.
2. Create a job with a dynamical number of steps, like one per day. With this approach I can keep chunks of 50 accounts and a stable size for hibernate sessions (50 * one position). But what is the impact of having 1500 steps in a job ? (if it's one per day for 5 years). I don't find any recommandation about the limit of steps number.

What is your recommandations / feeling ?

[pkernevez]

Does someone have an opinion on this question ?

[fmbenhassine]

The only limit to the number of steps is the amount of resources you allocate to the JVM. In my experience, the table step_execution becomes a bottle neck due to the frequent updates to the step executions and contexts (specifically if steps are running in parallel, like with partitioning).

I don't think you need the dynamic number of steps approach (and end up with 1500 steps in your job). Have you thought about partitioning the input data set and setup a reasonable fixed number of parallel steps? Partitioning really works well in most cases (and is restartable in case of failures, only failed partitions are reprocessed). This should solve your problem as each worker step would have its own hibernate session with a subset of the data.