Enabling Search Framework Real Time Indexing (RTI)

As we prepare our HCM PUM 46 training servers, we decided to enable Real Time Indexing (RTI). We found several great sample videos and resources but still made a few mistakes. We are sharing our adventure to help you as you implement RTI.

Before implementing RTI, we have a couple prerequisites. First, we must have a working Elasticsearch (OpenSearch) service. Second, we should have some indexes deployed. Since our training focus is PeopleTools, we usually deploy the PTSF_RECENTMENU and the PTPORTALREGISTRY indexes. To summarize the prerequisites, we need a:

• Working Elasticsearch (OpenSearch) installation and
• Deployed indexes

Elasticsearch (OpenSearch) does not store its data in the PeopleSoft database. Instead, we move data into the Elasticsearch (OpenSearch) index storage through scheduled batch processes. The point of RTI is to reduce latency by eliminating scheduling; get data into indexes as fast as possible: Enter data → Search for data.

RTI works by using the following:

• Triggers to insert rows into a staging table and
• A Process Scheduler server process that checks for new rows in the staging table

Enabling RTI is a multi-step process requiring:

• Process Scheduler-specific configurations and
• Index-specific configurations

Process Scheduler Configuration

Let's start with the Process Scheduler Server. Using the psadmin command, make sure Real Time Indexing is enabled, as demonstrated in the following screenshot:

This turns on the "watcher" process. On interval, the PSRTISRV service polls the RTI staging table, PS_PTSF_RTISTG, for new rows to process. We can tune the number of RTI instances through the psprcs.cfg file.

[PSRTISRV]
;=========================================================================
; Settings for Real Time Indexing watchdog
;=========================================================================
;-------------------------------------------------------------------------
Min Instances=1
Max Instances=1

We may configure the polling interval through the Search Instance Search Options Config (PeopleTools > Search Framework > Administration > Define Search Instances). The last several options relate to RTI:

The Heartbeat Interval determines how often the RTI process checks for new rows in the RTI staging table (PS_PTSF_RTISTG). The Real Time Indexing Set Size specifies how many rows to process on each interval. The point is to tune these two numbers for optimal performance. The shorter the interval, the closer to real time, but with a performance cost. The longer the interval, the less the performance impact. Too long of an interval, however, and the number of transactions per interval may exceed the Set Size threshold, meaning RTI may only clear the queue after hours, creating a larger backlog with each run.

As a side note, PeopleSoft queues transactions for RTI through database triggers. A bulk update to a triggered table may generate more transactions than RTI can clear in a single polling interval, creating a latency effect for online transactions. Given the default configuration parameters, PeopleSoft will index 300 transactions every 2 seconds. If we batch update 30,000 rows of data, we may have to wait 4 minutes before that data clears the queue. 300,000 rows of data? Roughly 35 minutes. With this in mind, we may want to remove an index from RTI before a batch update, run the update, re-enable, and then schedule an incremental build to catch up. That way, intermittent, unusual batch processing doesn't interfere with online RTI processing.

Index Configuration

After configuring server settings, we choose which indexes to enable. Navigate to PeopleTools > Search Framework > Configure Realtime Indexing to select an index. To start with, pick indexes that are Disabled. These are indexes that contain RTI metadata, making them trivial to enable. Indexes marked as Not Configured may be enabled but will require additional configuration before enabling.

In our demo environments, we enabled PTSF_RECENTMENU. To enable an index:

• Check the box Enable Real Time Indexing,
• Save, and
• Click the Trigger SQL link to download database-specific triggers.

When you click the Trigger SQL link, PeopleSoft will either open three new tabs or download three new files (it depends on browser configuration and registered handler for SQL files). The file name identifies the database: ora for Oracle, mss for Microsoft SQL Server, and db2 for DB2. If your browser opens them in new tabs instead of downloads, scroll to the end of the address bar to confirm the file name. This is where I made a mistake. I had so many open tabs that I didn't notice PeopleSoft opened three new ones. I only saw one and kept trying to run the DB2 script against my Oracle database.

Alternatively, as described by Ravi and Ramasimha in their Quest Recorded Webcast, we can let App Designer build the proper database triggers for us. I appreciate the App Designer integration because it ensures future record changes won't delete the RTI trigger scripts. It also means I don't have to log into a separate SQL tool or verify that I chose the correct database script.

After enabling RTI for an index, PeopleSoft recommends running a full or incremental sync to ensure all data exists in the target index. After that, you can cancel any scheduled run controls for that index, as RTI will now manage the incremental updates for that specific index.

Here are some great resources with additional information about RTI:

• Siva Koya's RTI YouTube Video
• PeopleSoft Tutorial
• Quest Recorded Webcast

Is RTI Really "Real time"?

Realtime usually means data automatically flows from one system to the next. Create data → Send data. If RTI were truly real time, each update would send immediately. But if you have up to 300 updates within a 2-second period, a real time interface would create 300 connections to incrementally send 300 updates. PeopleSoft's near real time approach, however, is a lot like App Engine's Bulk Update. Another way to think of it is that RTI is a shuttle with room for 300 passengers. A shuttle arrives once every 2 seconds to pick up however many passengers are waiting. This is far more efficient than sending up to 300 vehicles individually. And even better, through parameters, we can tune the interval and passenger count to achieve the best experience for our users.

Which Indexes Should You Enable?

We recommend starting with PTSF_RECENTMENU. This allows the portal search feature to suggest user and enterprise recent items within search results. Next, we suggest enabling indexes used by component keyword search so your users may view transactions immediately after creation.

At JSMpros, we teach PeopleTools concepts every week. Each student who attends a live training class receives a properly sized, personal training server with relevant PeopleTools and application versions for the duration of training. This is one of the ways we provide you with an unparalleled training experience! Check out our website to see what we are offering next!

Access all of our content anytime from anywhere through our all-access subscriptions. Check it out!

Branding Shortcut

Starting with his post PeopleTools 8.55.x - Branding - Part I - What has changed, Sasank Vemana provides a series of articles describing how to brand Fluid. If your organization supports multiple branding themes, then the PeopleTools delivered branding module and branding macros concept described by Sasank are a perfect fit. Although a fair amount of effort to configure, I didn't mind the macro concept provided in PeopleTools 8.55. But when PeopleTools delivered 8.56 with a brand new macro set and guidance suggesting we either start over with the new macro set or update ours with their new macros (which included evaluating all of our other macro changes), I folded. The scale had tipped. I realized that branding macros were not a "once and done" proposition. It was clear that maintaining branding macros would be more time consuming than injecting a little CSS into Oracle delivered stylesheets. I have this rule: If a configuration alternative exists, but that configuration alternative requires significantly more ongoing maintenance effort than customizing, I will choose the customization. Why? the point of configuration is to simplify Lifecycle Management. If the configuration alternative is more effort, complicating Lifecycle Management, then it is not a good alternative. It is counterproductive. PeopleTools includes very good compare tools for managed definition customizations. It is these great compare tools that sometimes make customizations simpler to maintain than configuration alternatives. This is not the case (yet -- I say "yet" because I believe this will change in the future) for configuration options that may become invalid (or broken) during an update/upgrade/selective adoption.

If your organization has just one global branding theme, you may find this approach much simpler. This is the approach I used with PeopleTools prior to the attribute-based branding module:

1. Open a Fluid homepage.
2. Using your browser's developer tools, mock up the changes desired.
3. Be sure to make your selector more qualified than Oracle's. I suggest including the ID of a higher level element, but do NOT use an ID that starts with win0div as these IDs change with every New Window launched from the base PeopleSoft window.
4. Copy these changes into a new PeopleSoft free-form sub stylesheet.
5. Add this new stylesheet to PSSTYLEDEF_FMODE.
6. Test.
7. Visit a Fluid transaction page to identify further changes required to finalize the Fluid branding theme.

Here is some sample CSS to get you started:

How does this work? Unlike the branding module, which replaces and/or changes Oracle-delivered CSS, we allow Oracle's CSS to be sent to web browsers unchanged. Just as before the customization, a user's web browser will parse Oracle's CSS, building a list of rules. But when the browser reads our rules injected at the very end, the browser will ignore Oracle's rules because ours will be both more specific and interpreted last.

What about Classic and Classic Plus? Same principle, just a different stylesheet. Classic uses PSSTYLEDEF_TANGERINE and DEFAULT_THEME_FLUID. I prefer PSSTYLEDEF_TANGERINE because it is a structured stylesheet, allowing us to inject one object, very minor customization.

What about Lifecycle Management? When applying PeopleTools patches and updates, it is very likely Oracle will replace PSSTYLEDEF_FMODE, erasing your one-line customization. Restoring the customization, however, is trivial. Just re-insert the free form sub stylesheet. It is possible that Oracle may change the HTML structure of Fluid and Classic pages resulting in CSS selector modifications, etc. We, therefore, must test after every update and be prepared to modify accordingly. However, I have used this approach with Fluid from 8.55 through 8.57 with no updates necessary.

Did you find this article helpful? Are you interested in learning more about PeopleTools, including productivity shortcuts such as this one? Take your PeopleTools skills to the next level by registering for one of our courses at jsmspros.com

Unlimited Session Timeout

There are a lot of security admins out there that are going to hate me for this post. There are a lot of system administrators, developers, and users, however, that will LOVE me for this post. The code I'm about to share with you will keep the logged in PeopleSoft user's session active as long as the user has a browser window open that points to a PeopleSoft instance. Why would you do this? I can think of two reasons:

• Your users have several PeopleSoft browser windows open. If one of them times out because of inactivity at the browser window level, then it will kill the session for ALL open windows. That just seems wrong.
• Your users have long running tasks, such as completing performance reviews, that may require more time to complete than is available at a single sitting. For example, imagine you are preparing a performance review and you have to leave for a meeting. You don't have enough information in the transaction to save, but you can't be late for the meeting either. You know if you leave, your session will time out while you are gone and you will lose your work. This also seems wrong.

Before I show you how to keep the logged in user's session active, let's talk about security... Session timeouts exist for two reasons (at least two):

• Security: no one is home, so lock the door
• Server side resource cleanup: PeopleSoft components require web server state. Each logged in user session (and browser window) consumes resources on the web server. If the user is dormant for a specific period of time, reclaim those resources by killing the user's session.

We can "lock the door" without timing out the server side session with strong policies on the workstation: password protected screen savers, etc.

So here is how it works. Add the following JavaScript to the end of the HTML definition PT_COMMON (or PT_COPYURL if using an older version of PeopleTools) (or even better, if you are on PeopleTools 8.54+, use component and/or role based branding to activate this script). Next, turn down your web profile's timeout warning and timeout to something like 3 and 5 minutes or 5 and 10 minutes. On the timeout warning interval, the user's browser will place an Ajax request to keep the session active. When the user closes all browser windows, the reset won't happen so the user's server side session state will terminate.

What values should you use for the warning and timeout? As low as possible, but not so low you create too much network chatter. If the browser makes an ajax request on the warning interval and a user has 10 windows open, then that means the user will trigger up to 10 Ajax requests within the warning interval window. Now multiply that by the number of logged in users at any given moment. See how this could add up?

Here is the JavaScript:

(function (root) {
    // xhr adapted from http://toddmotto.com/writing-a-standalone-ajax-xhr-javascript-micro-library/
    var xhr = function (type, url, data) {
        var methods = {
            success: function () {
            },
            error: function () {
            }
        };

        var parse = function (req) {
            var result;
            try {
                result = JSON.parse(req.responseText);
            } catch (e) {
                result = req.responseText;
            }
            return [result, req];
        };

        var XHR = root.XMLHttpRequest || ActiveXObject;
        var request = new XHR('MSXML2.XMLHTTP.3.0');
        request.open(type, url, true);
        request.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
        request.onreadystatechange = function () {
            if (request.readyState === 4) {
                if (request.status === 200) {
                    methods.success.apply(methods, parse(request));
                } else {
                    methods.error.apply(methods, parse(request));
                }
            }
        };
        
        request.send(data);
        return {
            success: function (callback) {
                methods.success = callback;
                return methods;
            },
            error: function (callback) {
                methods.error = callback;
                return methods;
            }
        };
    }; // END xhr


    var timeoutIntervalId;
    var resetUrl;

    /* replace warning message timeout with Ajax call
     * 
     * clear old timeout after 30 seconds
     * macs don't set timeout until 1000 ms
     */
    root.setTimeout(function () {
        /* some pages don't have timeouts defined */
        if (typeof (timeOutURL) !== "undefined") {
            if (timeOutURL.length > 0) {
                resetUrl = timeOutURL.replace(/expire$/, "resettimeout");
                if (totalTimeoutMilliseconds !== null) {
                    root.clearTimeout(timeoutWarningID);
                    root.clearTimeout(timeoutID);
                    
                    timeoutIntervalId =
                            root.setInterval(resetTimeout /* defined below */,
                                    root.warningTimeoutMilliseconds);
                }
            }
        }
    }, 30000);

    var resetTimeout = function () {
        xhr("GET", resetUrl)
                .success(function (msg) {
                    /* do nothing */
                })
                .error(function (xhr, errMsg, exception) {
                    alert("failed to reset timeout");
                    /* error; fallback to delivered method */
                    (root.setupTimeout || root.setTimeout2)();
                });
    };
}(window));

A special "shout out" to Todd Motto for his Standalone Ajax/XHR JavaScript micro-library which is embedded (albeit modified) in the JavaScript above.

An Interview with Jeff Robbins

At OpenWorld 2013, I had the opportunity to catch up with Jeff Robbins and ask him some questions about PeopleSoft's User Experience. You can watch the video here:

Watch more videos from Oracle's User Experience team on the oracleuseableapps YouTube channel.