Consuming Enormous Datasets with PeopleSoft

Integration Broker is built on the DOM concept, which means it parses data into an in-memory structured document before transforming, processing, or transmitting it. This is fantastic for small and incremental integrations designed to keep two systems synchronized on a per-transaction basis. However, it fails when processing datasets that exceed the amount of available system memory.

An alternative is to use a SAX or stream-based parser that emits events. This type of parser only consumes the memory necessary to emit the next event. Stream-based parsers are highly efficient for one-way, one-time reads through large datasets. Unfortunately, Integration Broker does not support stream-based processing.

While reviewing PeopleSoft's Java class path, I noticed the Jakarta JSON stream-processing Java library. You can find a simple PeopleCode example of processing a JSON file using Jakarta JSON streams in our blog post JSON Stream Processing with PeopleCode. Now that we know how to use Jakarta's stream processing, the next challenge is reading an input stream from an external service. Since Integration Broker does not support streams, we need alternatives. Here are a few Java-based alternatives that readily integrate with PeopleCode through delivered APIs:

• Java Sockets
• Java HttpURLConnection
• Apache HttpClient

Although PeopleCode offers incredible support for Java, the real challenge lies with method and constructor overloading. PeopleCode identifies target Java methods and constructors through parameter count, not parameter type. Java overloading doesn't play well with PeopleCode. The solution I use to overcome method overloading is to leverage Java's support for JavaScript as a translation layer or a "glue." PeopleSoft exposes all functions and objects to Java. Therefore, all PeopleCode functions and objects are also available to JavaScript.

The following is a sample JavaScript that can run from PeopleCode to stream load data into a PeopleSoft table. It uses Jakarta for stream processing and Apache HttpClient to connect to the external service. Both of these external libraries are included with PeopleTools. Notice the use of PeopleCode functions, such as CreateSQL, as well as Java objects such as HttpGet.

I use PeopleCode similar to the following to run JavaScript from PeopleCode. You can find several examples of using Java's ScriptEngineManager on our blog.

Local JavaObject &manager = CreateJavaObject("javax.script.ScriptEngineManager");
Local JavaObject &engine = &manager.getEngineByName("JavaScript");
Local String &script = "JavaScript goes here";
Local Any &result;

&engine.eval(&script);

&result = &engine.get("result");

At JSMpros, we teach PeopleTools and PeopleCode tips like this in every class. Check out our online schedule to see what we're offering next. Would you prefer to learn at your own pace? Purchase a subscription to access all of our on-demand content at discounted rates!