The first thing I do when creating an App Engine is disable restart. Because restartable is the default, it might seem like Restartable is a best practice. But is it?
A restartable App Engine is a batch program that iterates over data and commits changes on an interval. This behavior contrasts with the default behavior, which is to roll back changes on failure. A restartable program restarts at the last checkpoint. Without restart, ALL data must be reprocessed on failure. The obvious benefit is that restartable App Engines don't have to reprocess successful data. This sounds fantastic! But is it?
There are two main data processing strategies:
- Set-based processing
- Row-by-row processing
Set-based processing allows the database to optimize and execute commands at the database. Row-by-row processing, however, requires each row of data to be transferred to the process scheduler server and then processed. Row-by-row processing, therefore, is dramatically slower.
App Engines utilizing either strategy may be restartable, but a restartable set-based program would be rare. Restart is only valuable on failure. Before restarting, we would fix whatever data caused an error so we may reprocess that data. With set-based processing, that error would already be stored in an intermediate downstream processing table. Therefore, row-by-row processing is the only processing method that seems relevant for restart.
Another problem with a Restartable App Engine is that it must commit on intervals to benefit from the restart. What makes this a problem is that databases, such as Oracle, close any open cursors on commits. Therefore, a Restartable App Engine must also re-fetch (close and reopen) SQL cursors on every commit.
What does this mean? A Restartable App Engine may be the slowest, least-performing option available! It is an option we may use when it makes sense, but should it be the default? What do you think?
Ready to take your PeopleTools skills to the next level? Become a subscriber to gain access to our vast library of PeopleTools training courses, hands-on activities, downloads, instruction videos, and more!
