Consuming Enormous Datasets with PeopleSoft

Integration Broker is built on the DOM concept, which means it parses data into an in-memory structured document before transforming, processing, or transmitting it. This is fantastic for small and incremental integrations designed to keep two systems synchronized on a per-transaction basis. However, it fails when processing datasets that exceed the amount of available system memory.

An alternative is to use a SAX or stream-based parser that emits events. This type of parser only consumes the memory necessary to emit the next event. Stream-based parsers are highly efficient for one-way, one-time reads through large datasets. Unfortunately, Integration Broker does not support stream-based processing.

While reviewing PeopleSoft's Java class path, I noticed the Jakarta JSON stream-processing Java library. You can find a simple PeopleCode example of processing a JSON file using Jakarta JSON streams in our blog post JSON Stream Processing with PeopleCode. Now that we know how to use Jakarta's stream processing, the next challenge is reading an input stream from an external service. Since Integration Broker does not support streams, we need alternatives. Here are a few Java-based alternatives that readily integrate with PeopleCode through delivered APIs:

• Java Sockets
• Java HttpURLConnection
• Apache HttpClient

Although PeopleCode offers incredible support for Java, the real challenge lies with method and constructor overloading. PeopleCode identifies target Java methods and constructors through parameter count, not parameter type. Java overloading doesn't play well with PeopleCode. The solution I use to overcome method overloading is to leverage Java's support for JavaScript as a translation layer or a "glue." PeopleSoft exposes all functions and objects to Java. Therefore, all PeopleCode functions and objects are also available to JavaScript.

The following is a sample JavaScript that can run from PeopleCode to stream load data into a PeopleSoft table. It uses Jakarta for stream processing and Apache HttpClient to connect to the external service. Both of these external libraries are included with PeopleTools. Notice the use of PeopleCode functions, such as CreateSQL, as well as Java objects such as HttpGet.

I use PeopleCode similar to the following to run JavaScript from PeopleCode. You can find several examples of using Java's ScriptEngineManager on our blog.

Local JavaObject &manager = CreateJavaObject("javax.script.ScriptEngineManager");
Local JavaObject &engine = &manager.getEngineByName("JavaScript");
Local String &script = "JavaScript goes here";
Local Any &result;

&engine.eval(&script);

&result = &engine.get("result");

At JSMpros, we teach PeopleTools and PeopleCode tips like this in every class. Check out our online schedule to see what we're offering next. Would you prefer to learn at your own pace? Purchase a subscription to access all of our on-demand content at discounted rates!

JavaScript on the App Server: Scripting PeopleCode

It has been nearly a decade since I started playing with JavaScript on the PeopleSoft application server. Back then I had to deploy a couple of JAR files to the app server. At that time, maintaining and deploying unmanaged files seemed more headache than benefit. Today Java provides full scripting support through the ScriptEngineManager and embedded Mozilla Rhino JavaScript script engine. Why would I want to script PeopleCode? Here are a few of my favorite reasons:

• Low-level socket communication
• Avoid reflection: JavaScript executes all methods regardless of variable type whereas PeopleCode only recognizes the returned type, not the real type
• Process simple JSON structures that can't be modeled with the Documents module

Here is the PeopleCode required to invoke JavaScript

Local JavaObject &manager =  CreateJavaObject("javax.script.ScriptEngineManager");
Local JavaObject &engine =  &manager.getEngineByName("JavaScript");

REM ** Evaluate a simple JavaScript;
&engine.eval("var result = Math.random();");

REM ** Access the value of the JavaScript variable named result;
Local string &result_text =  &engine.get("result").toString();

Here is some JavaScript that converts the variable &json_string into a JSON Array and then iterates over each entry, inserting values into a table. Notice that I'm invoking the PeopleCode SQLExec function from JavaScript.

var result = (function() {
  var SQLExec = Packages.PeopleSoft.PeopleCode.Func.SQLExec;
  var json = JSON.parse(json_string);
  var count = 0;
  json.forEach(function(item, idx) {
    SQLExec("INSERT INTO ... SYSTIMESTAMP", [idx, item]);
    count++;
  });
  return count + " rows inserted";
}());

Where did that &json_string variable come from? Here:

&engine.put("json_string", "[""item1"", ""item2"", ""item3""]");

101 Ways to Process JSON with PeopleCode

... well... maybe not 101 ways, but there are several!

There is a lot of justified buzz around JSON. Many of us want to (or must) generate and parse JSON with PeopleSoft. Does PeopleSoft support JSON? Yes, actually. The Documents module can generate and parse JSON. Unfortunately, many of us find the Documents module's structure too restrictive. The following is a list of several alternatives available to PeopleSoft developers:

• Documents module
• Undocumented JSON objects delivered by PeopleTools
• JSON.org Java implementation (now included with PeopleTools)
• JavaScript via Java's ScriptEngineManager

We will skip the first two options as there are many examples and references available on the internet. In this post, we will focus on the last two options in the list: JSON.org and JavaScript. Our scenario involves generating a JSON object containing a role and a list of the role's permission lists.

PeopleCode can do a lot of things, but it can't do everything. When I find a task unfit for PeopleCode, I reach out to the Java API. PeopleCode has outstanding support for Java. I regularly scan the class and classes directories of PS_HOME, looking for new libraries I can leverage from PeopleCode. One of the files in my App Server's class path is json.jar. As a person interested in JSON, how could I resist inspecting the file's contents? Upon investigation, I realized that json.jar contains the json.org Java JSON implementation. This is good news as I used to have to add this library myself. So how might we use json.jar to generate a JSON file? Here is an example

JSON.org has this really cool fluent design class named JSONStringer. If the PeopleCode editor supported custom formatting, fluent design would be really, really cool. For now, it is just cool. Here is an example of creating the same JSON using the JSONStringer:

What about reading JSON using json.org? The following example starts from the JSON string generated by JSONStringer. It is a little ugly because it requires Java Reflection to invoke the JSONObject constructor. On the positive side, though, this example demonstrates Java Class casting in PeopleCode (hat tip to tslater2006 for helping me with Java Class casting in PeopleCode)

What is that you say? Your PeopleTools installation doesn't have the json.jar (or jsimple.jar) files? If you like this approach, then I suggest working with your system administrator to deploy the JSON.org jar file to your app and/or process scheduler's Java class path

But do we really need a special library to handle JSON? By definition, JSON describes a JavaScript object. Using Java's embedded JavaScript script engine, we have full access to JavaScript. Here is a sample JavaScript file that generates the exact same JSON as the prior two examples:

... and the PeopleCode to invoke this JavaScript:

Did you see something in this post that interests you? Are you ready to take your PeopleTools skills to the next level? We offer a full line of PeopleTools training courses. Learn more at jsmpros.com.