Which Technology Would You Use for JSON?

We asked our LinkedIn audience:

Which Technology Would You Use to Process JSON?

And they replied:

These answers are fantastic! Based on performance testing and research, PeopleCode native JSON objects, such as JsonObject and JsonArray, perform much faster than the Documents module or external Java libraries. However, there are times when I might choose an alternative.

Java Libraries

PeopleCode leverages the Java Native Interface to provide bidirectional access to Java. Java classes can interact with PeopleCode functions and objects, and vice versa. In addition to the delivered Java API, PeopleSoft provides several key Java libraries, including the Jakarta JSON library. Jakarta stream processing enables us to process enormous datasets. This stands in contrast to native DOM-based JSON parsing, which is limited by the amount of system memory available.

Documents

A document structure may generate either XML or JSON, which might simplify solution development when creating APIs that support multiple output types. Likewise, a document structure is required when creating a REST URI template.

Additional Resources

• PeopleCode JSON API
• JSON Stream Processing Blog Post
• PeopleSoft Documents module
• Jakarta JSON Processing

How about you? What technology would you choose? What are some reasons you might choose an alternative to the native JsonObject and JsonArray?

At JSMpros, we teach JSON processing techniques regularly through our Integration Tools Update course. Check out our online schedule to see what we are offering next! Alternatively, subscribe to gain access to all of our on-demand courses!

Consuming Enormous Datasets with PeopleSoft

Integration Broker is built on the DOM concept, which means it parses data into an in-memory structured document before transforming, processing, or transmitting it. This is fantastic for small and incremental integrations designed to keep two systems synchronized on a per-transaction basis. However, it fails when processing datasets that exceed the amount of available system memory.

An alternative is to use a SAX or stream-based parser that emits events. This type of parser only consumes the memory necessary to emit the next event. Stream-based parsers are highly efficient for one-way, one-time reads through large datasets. Unfortunately, Integration Broker does not support stream-based processing.

While reviewing PeopleSoft's Java class path, I noticed the Jakarta JSON stream-processing Java library. You can find a simple PeopleCode example of processing a JSON file using Jakarta JSON streams in our blog post JSON Stream Processing with PeopleCode. Now that we know how to use Jakarta's stream processing, the next challenge is reading an input stream from an external service. Since Integration Broker does not support streams, we need alternatives. Here are a few Java-based alternatives that readily integrate with PeopleCode through delivered APIs:

• Java Sockets
• Java HttpURLConnection
• Apache HttpClient

Although PeopleCode offers incredible support for Java, the real challenge lies with method and constructor overloading. PeopleCode identifies target Java methods and constructors through parameter count, not parameter type. Java overloading doesn't play well with PeopleCode. The solution I use to overcome method overloading is to leverage Java's support for JavaScript as a translation layer or a "glue." PeopleSoft exposes all functions and objects to Java. Therefore, all PeopleCode functions and objects are also available to JavaScript.

The following is a sample JavaScript that can run from PeopleCode to stream load data into a PeopleSoft table. It uses Jakarta for stream processing and Apache HttpClient to connect to the external service. Both of these external libraries are included with PeopleTools. Notice the use of PeopleCode functions, such as CreateSQL, as well as Java objects such as HttpGet.

I use PeopleCode similar to the following to run JavaScript from PeopleCode. You can find several examples of using Java's ScriptEngineManager on our blog.

Local JavaObject &manager = CreateJavaObject("javax.script.ScriptEngineManager");
Local JavaObject &engine = &manager.getEngineByName("JavaScript");
Local String &script = "JavaScript goes here";
Local Any &result;

&engine.eval(&script);

&result = &engine.get("result");

At JSMpros, we teach PeopleTools and PeopleCode tips like this in every class. Check out our online schedule to see what we're offering next. Would you prefer to learn at your own pace? Purchase a subscription to access all of our on-demand content at discounted rates!

Five Ways to Consume a REST Service with PeopleSoft

PeopleSoft includes facilities for providing and consuming REST. A service provider is a listener. A provider waits for requests and responds accordingly. To provide web services, we must register metadata with Integration Broker. The process is pretty well fixed. Consumption, on the other hand, is much more flexible. Consuming a REST service means invoking (or calling) someone else's listener. The consumer initiates the conversation. In a consumption scenario, we might be sending data, receiving data, or both.

Here are five ways a PeopleSoft developer might invoke a REST service:

1. Metadata

This is the traditional PeopleSoft approach and follows the same pattern as providing services through Integration Broker. Here are a few of the metadata items a developer would create:

• Documents,
• Messages,
• Service, and
• Service Operation.

After creating and registering somewhat reusable metadata, the developer would then write PeopleCode to:

• Create Document structures,
• Invoke the service, and
• Parse the response.

Even though this approach requires the most effort and offers the least reuse, it takes full advantage of Integration Broker's capabilities, including logging, security, etc.

Metadata, in general, offers the following benefits:

• Reuse (ability to use the same definition multiple times),
• Documentation (describes an object),
• Reporting (query and analysis), and
• Transformation (ability to upgrade and transform based on architecture changes).

Pure code solutions are much harder to query and transform. Metadata is what allowed PeopleSoft to switch from a Windows-based Client/Server solution to its Pure Internet Architecture in (mostly) one release. Metadata is what allowed Integration Broker to transform from an XML-focused solution into an SOA solution in one release (the 8.48 transform). If these benefits sound appealing, then metadata may be the right choice for you.

2. Application Services Framework

Created initially as a REST provider to support digital assistants, the Application Services Framework can now consume REST services. The Application Services Framework is a developer-friendly configuration layer tightly integrated with Integration Broker metadata. Rather than directly working with SOA-oriented metadata, the Application Services Framework allows developers to create solutions using REST terminology and REST patterns.

To use the Application Services Framework for consumption, a developer would:

• Register the target service with the Application Services Framework, which includes describing the URL, parameters, headers, and payload (both request and response).
• Write PeopleCode to invoke the endpoint service.

This is one of the most important chapters in our updated Integration Tools curriculum. You may find basic information (including code samples) on consuming a REST service in PeopleBooks.

3. %IntBroker.ConnectorRequestURL

If your request is a simple URL with no special headers (request or response headers) and any parameters may be passed through the URL, then this is the most straightforward approach:

Local string &response = %IntBroker.ConnectorRequestURL("http://rest.example.com/endpoint");

Unfortunately, this solution offers no logging, header control, or any typical Integration Broker or integration features. It is strictly for the simplest GET requests. Nevertheless, by using URL definitions, this alternative may be one of the better alternatives for cross-environment migrations.

4. Code-only

The challenge with the first option, Metadata, is that it requires creating, migrating, and maintaining a significant number of definitions that provide little (if any) value for REST. For example, the URL of a PeopleSoft REST consumer service becomes part of the metadata. If you have different targets between DEV, TEST, and PROD, then you must modify the metadata after every migration. Implementing a true PeopleSoft Metadata-focused solution (such as option 1) may require you to violate development best practices by making changes after each migration.

Since all approaches require PeopleCode, why not skip the Integration Broker metadata in favor of your own reusable best practice-based metadata? You may find an example of this approach in our blog post Simple Code-only REST Request. With this approach, you are responsible for managing URLs, Credentials, etc., but you may store them in a manner that allows for reuse and better migration management. For example, you might create URL definitions that point to different endpoints in DEV, TEST, and PROD.

It is important to note that this approach DOES use metadata, but it reuses generic PeopleTools-delivered definitions, so we don't have to create new definitions.

As mentioned earlier, code doesn't report or transform well, so this alternative may miss out on future automated transforms, such as the 8.48 automated transform. Likewise, integrations launched in this manner won't log details in the Synchronous Service Operation Monitor. Nevertheless, this approach is becoming one of our favorite alternatives to overly complicated metadata.

5. Java

Using Java allows a PeopleSoft developer to bypass Integration Broker altogether. We put this option last because it should be the choice of last resort. However, it is probably the most flexible option available, allowing for everything from simple low-level sockets to convenient libraries, such as Apache's HttpClient (now included with PeopleTools).

There are many reasons to choose this option, but here are the two most common:

• Processing binary data: Integration Broker is pretty much an XML workhorse. Everything passing through Integration Broker is wrapped in XML at some point. Unfortunately, that might not work well with binary data. Therefore, skipping Integration Broker may be appropriate.
• Processing large amounts of data: Integration Broker works with DOM. It requires all data to be loaded in memory at one time. This makes it a poor choice for handling large datasets. An alternative is to use stream processing with an event-based parser, such as a SAX parser. With Java, you can read bytes, process them through an event-based processor, and then discard the processed bytes.

The challenge with using Java directly from PeopleCode is that many Java methods and constructors use overloading. As a workaround, we devised a strategy for invoking Java through JavaScript from PeopleCode. With a small PeopleCode fragment, we can invoke a JavaScript, and JavaScript handles Java really well. Details about this strategy are available in our Blog Post JavaScript on the App Server: Scripting PeopleCode.

The following is a sample JavaScript we used at Reconnect 2024 to stream-processed a large amount of JSON data from a REST service.

At JSMpros, we teach all five options in our Integration Tools and Integration Tools Update courses. Check out our website for on-demand and live course offerings. Alternatively, subscribe to gain access to all of our on-demand courses.

Generating LARGE JSON Files

The PeopleCode native JsonObject and JsonArray classes allow us to create JSON structures as in-memory representations. But what if you need to generate a really LARGE JSON structure? An in-memory JSON Array may consume more memory than you can reasonably allow. Fortunately, PeopleTools includes the Jakarta JSON library, which allows us to write a JSON structure to a stream during construction.

The following code snippet demonstrates creating 10 million JSON objects in an array without any change in memory consumption. The generated file was 2.5 GB in size, but my memory utilization didn't change the entire time the program ran.

Local JavaObject &Json = GetJavaClass("jakarta.json.Json");
Local JavaObject &writer = CreateJavaObject("java.io.FileWriter", "C:\temp\users-big.json");

Local JavaObject &gen = &Json.createGenerator(&writer);

Local number &iteration = 1;

REM ** 10 million iterations;
Local number &maxIterations = 10000000;

&gen.writeStartArray();

For &iteration = 1 To &maxIterations
   
   REM ** start person/user object;
   &gen.writeStartObject();
   &gen.write("id", "" | &iteration);
   &gen.write("firstName", "John");
   &gen.write("lastName", "Smith");
   
   REM ** start child address object;
   &gen.writeStartObject("address");
   &gen.write("streetAddress", "21 2nd Street");
   &gen.write("city", "New York");
   &gen.write("state", "NY");
   &gen.write("postalCode", "10021");
   &gen.writeEnd();
   
   REM ** start phone number array;
   &gen.writeStartArray("phoneNumber");
   
   REM ** start home phone object;
   &gen.writeStartObject();
   &gen.write("type", "home");
   &gen.write("number", "212 555-1234");
   &gen.writeEnd();
   
   REM ** start fax number object;
   &gen.writeStartObject();
   &gen.write("type", "fax");
   &gen.write("number", "646 555-4567");
   &gen.writeEnd();
   
   REM ** end array of phone numbers;
   &gen.writeEnd();
   
   REM ** end person/user object;
   &gen.writeEnd();
End-For;


REM ** end array;
&gen.writeEnd();

REM ** cleanup to flush buffers;
&gen.close();
&writer.close();

The hard-coded values come directly from the Jakarta generator API documentation. In real life, you would replace these values with database data. I converted numbers to strings to simplify the example to avoid Java Reflection.

Are you interested in parsing rather than generating large JSON files? Check out our post on JSON Stream Parsing.

We teach PeopleTools and PeopleCode tips like this every week! Check out our upcoming course schedule to see what we are offering next! We would love to have you join us. Want to learn at your own pace? Check out our subscriptions and on-demand offerings as well. Or do you have a group you would like to train? Contact us for group and quantity discounts.

JSON Stream Processing with PeopleCode

Our Integration Tools courses emphasize the importance of utilizing PeopleSoft's native JSON support to parse and process JSON. This feature provides excellent functionality for most scenarios, as PeopleSoft's native JsonObject and JsonArray offer fast and efficient JSON processing. They are particularly useful for generating small integration responses or handling tasks like JWT (JSON Web Token) generation.

It's important to note that PeopleSoft's native JSON definitions employ a DOM-based parser, which builds an in-memory JSON structure. While this is effective in many cases, it does require your server to have enough memory to accommodate the entire JSON document without impacting its normal workload. Large files, therefore, can be problematic.

Stream-based parsing provides an alternative approach. With a stream-based parser, events are emitted as they occur, enabling immediate processing of data identified by these events. Once processed, the parser discards the data and proceeds to the next event. For instance, let's consider a scenario where you have a massive array of users. A DOM parser would load the entire array into memory, whereas a stream-based parser would only load one user at a time, allowing you to process the user and then discard it. Stream-based parsers are often more efficient in terms of resource utilization, as they process and discard data right away instead of constructing a traversable in-memory document.

Although PeopleCode itself doesn't include a stream-based parser, it does include the Java Jakarta stream-based JSON parser, which we can leverage through PeopleSoft's built-in Java support. Let me share an example with you. For testing purposes, I obtained a sample JSON file from https://jsonplaceholder.typicode.com/users. This file is small and perfect for testing. I then incorporated the following PeopleCode into an App Engine step, which you can run locally through App Designer:

REM ** JavaDoc: https://jakarta.ee/specifications/jsonp/2.0/apidocs/jakarta.json/jakarta/json/stream/jsonparser;
REM ** Data: https://jsonplaceholder.typicode.com/users;
Local JavaObject &Json = GetJavaClass("jakarta.json.Json");
Local JavaObject &reader = CreateJavaObject("java.io.FileReader", "C:\temp\users.json");
Local JavaObject &parser = &Json.createParser(&reader);

Local string &email;

While (&parser.hasNext())
   Local JavaObject &next = &parser.next();
   
   If (&next.equals(&next.START_OBJECT)) Then
      
      Local JavaObject &user = &parser.getObject();
      
      REM ** do something with the object;
      &email = &user.getString("email");
      MessageBox(0, "", 0, 0, "email: %1", &email);
   End-If;
End-While;

Here is some sample output generated by this short App Engine program:

email: Sincere@april.biz (0,0)
 Message Set Number: 0
 Message Number: 0
 Message Reason: email: Sincere@april.biz (0,0) (0,0)

email: Shanna@melissa.tv (0,0)
 Message Set Number: 0
 Message Number: 0
 Message Reason: email: Shanna@melissa.tv (0,0) (0,0)

email: Nathan@yesenia.net (0,0)
 Message Set Number: 0
 Message Number: 0
 Message Reason: email: Nathan@yesenia.net (0,0) (0,0)

email: Julianne.OConner@kory.org (0,0)
 Message Set Number: 0
 Message Number: 0
 Message Reason: email: Julianne.OConner@kory.org (0,0) (0,0)

Are you interested in learning more about PeopleSoft Integration, PeopleTools, Or PeopleCode? If so, check out our subscriptions, on-demand, and upcoming course schedule. We would love to have you join us!

Base64 Encoding with Emoji

PeopleSoft's Pluggable Encryption Technology (PET) is used to apply base64 encoding. The first step of the base64 algorithm chain is to convert from PeopleSoft Unicode to ASCII. The conversion makes sense since PeopleCode is a Unicode language, and the base64 algorithm is not. But what if you have Unicode characters you want to base64 encode? So I tried the following "Hello World" example with emoji:

&crypto.UpdateData("Hello World 🤔");

Unfortunately, the PET algorithm dropped the Unicode Emoji. Understandable because Emoji is Unicode. So, what alternatives do we have? The good news is my older PL/SQL approach still works. However, my older Java example no longer works. So, for those who want a cross-platform solution, here is an updated PeopleCode/Java code listing. The good news is this version is documented and much simpler!

REM ** What would you like to encode?;
Local string &textToEncode = "Hello World 🤔";

REM ** Pointer to Java encoder;
Local JavaObject &encoder = GetJavaClass("java.util.Base64").getEncoder();

REM ** Be sure to change the character set to match your source;
Local JavaObject &bytes = CreateJavaObject("java.lang.String", &textToEncode).getBytes("UTF-8");

Local string &result = &encoder.encodeToString(&bytes);

REM ** print the result;
MessageBox(0, "", 0, 0, &result);

At JSMpros, we teach advanced PeopleTools concepts such as this all the time! Check out our events page to see what we are offering next, and become a subscriber to get 24x7 access to all of our on-demand videos, activity guides, and code samples!

PeopleCode to Move a File... sort of

You ever need to move a file through PeopleCode? Perhaps an App Engine program that needs to move generated content into the process output directory? I found myself in that position today. Surely there is a PeopleCode function for that... right? If you have one, please share it, because it seems the Internet doesn't know about it. A quick Internet search turned up several examples of using Exec with mv. I'm always weary of Exec. If I have to use the command line, I prefer to control it through the Java Runtime (see Exec Processes while Controlling stdin and stdout). Speaking of Java, can we borrow anything from the JRE? I did find my old ITToolbox response showing how to use java.io.File.renameTo to move files. It is still a great solution, but with multiple mounted file systems... well... it actually might fail. So I got to thinking... what about using java.nio? Here is an example:

Local JavaObject &jSource = CreateJavaObject("java.io.File", "c:\temp\JSM_TEST.log").toPath();
Local JavaObject &jTarget = CreateJavaObject("java.io.File", "c:\temp\JSM_TEST_moved.log").toPath();

Local JavaObject &jFiles = GetJavaClass("java.nio.file.Files");
Local JavaObject &jOptions = CreateJavaObject("java.nio.file.CopyOption[]");

&jFiles.move(&jSource, &jTarget, &jOptions);

And, what if you want to overrwrite the destination? Try it with copy options:

Local JavaObject &jSource = CreateJavaObject("java.io.File", "c:\temp\JSM_TEST.log").toPath();
Local JavaObject &jTarget = CreateJavaObject("java.io.File", "c:\temp\JSM_TEST_moved.log").toPath();

Local JavaObject &jFiles = GetJavaClass("java.nio.file.Files");
Local JavaObject &jStandardCopyOptions = GetJavaClass("java.nio.file.StandardCopyOption");
Local JavaObject &jOptions = CreateJavaObject("java.nio.file.CopyOption[]", &jStandardCopyOptions.REPLACE_EXISTING);

&jFiles.move(&jSource, &jTarget, &jamp;Options);

For more details about copy options, take a look at the Java Docs.

Are you ready to take your PeopleSoft development skills to the next level? Check out our latest course offerings at jsmpros.com. Prefer a custom agenda? Contact us for details.

Using PeopleCode to Read (and process) Binary Excel Files

At HIUG Interact last week, a member asked one of my favorite questions:

"Does anyone know how to read binary Microsoft Excel files from PeopleSoft?"

Nearly 15 years ago my AP manager asked me the same question, but phrased it a little differently:

"We receive invoices as Excel spreadsheets. Can you convert them into AP vouchers in PeopleSoft?"

Of course my answer was "YES!" How? Well... that was the challenge. I started down the typical CSV/FileLayout path, but that seems to be a temporary band aid, and challenging for the best users. I wanted to read real binary Excel files directly through the Process Scheduler, or basically, with PeopleCode. But here is the reality: PeopleCode is really good with data and text manipulation, but stops short of binary operations. Using PeopleCode's Java interface, however, anything is possible. After a little research, I stumbled upon Apache POI, a Java library that can read and write binary Excel files. With a little extra Java code to interface between PeopleCode and POI's Java classes, I had a solution. Keep in mind this was nearly 15 years ago. PeopleSoft and Java were both a little different back then and today's solution is slightly simpler. Here is a summary of PeopleSoft and Java changes that simplify this solution:

• As of PeopleTools 8.54, PeopleSoft now includes POI in the App and Process Scheduler server Java class path. This means I no longer have to manage POI as a custom Java library.
• The standard JRE added support for script engines and included the JavaScript script engine with every deployment. This means I no longer have to write custom Java to interface between POI and PeopleCode, but can leverage the dynamic nature of JavaScript.

How does a solution like this work? The ultimate goal is to process spreadsheet rows through a Component Interface. First we need to get data rows into a format we can process. Each language and operating environment has its strengths:

• PeopleCode can handle simple Java method invocations,
• JavaScript can handle complex Java method invocation without compilation,
• Java is really good at working with binary files, and
• PeopleCode and Component Interfaces play nicely together.

My preference is to capitalize on these strengths. With this in mind, I put together the following flow:

1. Use PeopleCode to create an instance of a JavaScript script interpeter,
2. Use JavaScript to invoke POI and iterate over spreadsheet rows, inserting row data into a temporary table, and
3. Use PeopleCode to process those rows through a component interface.

The code for this solution is in two parts: JavaScript and PeopleCode. Here is the JavaScript:

Next hurdle: where do we store JavaScript definitions so we can process them with PeopleCode? Normally we place JavaScript in HTML definitions. This works great for online JavaScript as we can use GetHTMLText to access our script content. App Engines, however, are not allowed to use that function. An alternative is to use Message Catalog entries for scripts. The following PeopleCode listing uses an HTML definition, but accesses the JavaScript content directly from the HTML definition Metadata table:

To summarize this PeopleCode listing, it first creates a JavaScript script engine manager, it then evaluates the above JavaScript, and finishes by processing rows through a CI (the CI part identified as a TODO segment).

This example is fully encapsulated in as few technologies as possible: PeopleCode and JavaScript, with a little SQL to fetch the JavaScript. The code will work online as well as from an App Engine. If this were in an App Engine, however, I would likely replace the JavaScript GUID section with the AE's PROCESS_INSTANCE. Likewise, I would probably use an App Engine Do-Select instead of a PeopleCode SQL cursor.

Did you see something on this blog that interests you? Are you ready to take your PeopleTools skills to the next level? We offer a full line of PeopleTools training courses. Learn more at jsmpros.com.

JavaScript and PeopleCode Array Parameters

I have been experimenting with scripting PeopleCode using JavaScript. This is possible because Java includes Mozilla's Rhino JavaScript engine. I took one of my experiments to OpenWorld 2015 which shows creating a Microsoft Excel Spreadsheet using POI and JavaScript. Here we are, a couple of weeks later, and I see this in the PeopleSoft OTN Discussion Forum: Java Exception: java.lang.reflect.InvocationTargetException: during call of java.lang.reflect.Method .invoke. Perfect! That is my exact use case from OpenWorld. I just happen to have a code sample to share on the forum. The developer's scenario was a bit more complicated. As you will note from the forum post, the developer needed to invoke SQL.Fetch from JavaScript. The JavaScript version of SQL.Fetch, which uses the PeopleCode Java interface, requires an array of selected columns. My first thought was just to use a standard JavaScript array. Since the SQL only has one column, I just needed an array with one item. This didn't work. JavaScript Arrays clearly are not Java Arrays. Here is an example:

var result = (function() {  
    var ReflectiveArray = java.lang.reflect.Array;  
    var CreateSQL = Packages.PeopleSoft.PeopleCode.Func.CreateSQL;  
    var columns = ReflectiveArray.newInstance(java.lang.Object,
        1 /* number of selected columns */);  
    var results = [];  
      
    SQL = CreateSQL("SELECT OPRDEFNDESC FROM PSOPRDEFN WHERE ROWNUM < 10");  
    while (SQL.Fetch(columns)) {  
        results.push(columns[0]);  
    }  
      
    return results.join();  
      
}());  

PeopleSoft Streams from Oracle University

In February of this year, Oracle University launched the PeopleSoft Learning Stream. Oracle's Learning Streams are short, educational vignettes. I was given the privilege of recording 6 streams:

• Using JavaScript with Pagelet Wizard is a 21 minute video showing you how to use Pagelet Wizard to convert a PeopleSoft query into an interactive D3 chart, a navigation collection into a carousel, a navigation collection into an accordion, and RequireJS for JavaScript dependency management.
• REST Query Access Service is a 15 minute session showing you how to craft a Query Access Service REST URL.
• Working with JSON in PeopleSoft Document Technology is a 23 minute video demonstrating how to use the PeopleCode Document, Compound, and Collection objects to read and write JSON.
• Basic Java API with PeopleCode is a 26 minute session showing you how to use the delivered Java API with PeopleCode. This session covers constructors, instance methods, properties, and static method invocation. Java objects demonstrated include String, Hashtable, Regular Expression Pattern and Matcher, arrays, and String.format.
• Intermediate Java API with PeopleCode is a 38 minute video that shows you how to configure JDeveloper to write Java for the PeopleSoft Application and Process Scheduler servers and provides some examples of writing and deploying Java to a PeopleSoft application server. Note: in this session you get to watch me attempt to troubleshoot an App Engine ABEND.
• Advanced Java API with PeopleCode is a 26 minute recording showing you how to use Java Reflection to remove PeopleCode ambiguity as well as how to use JavaScript to avoid reflection.

You can access all of my streams here. From this page you can preview the first 2 minutes of each video or subscribe for unlimited access to all of the videos in the Oracle PeopleSoft Learning Stream.

JavaScript on the App Server: Scripting PeopleCode

It has been nearly a decade since I started playing with JavaScript on the PeopleSoft application server. Back then I had to deploy a couple of JAR files to the app server. At that time, maintaining and deploying unmanaged files seemed more headache than benefit. Today Java provides full scripting support through the ScriptEngineManager and embedded Mozilla Rhino JavaScript script engine. Why would I want to script PeopleCode? Here are a few of my favorite reasons:

• Low-level socket communication
• Avoid reflection: JavaScript executes all methods regardless of variable type whereas PeopleCode only recognizes the returned type, not the real type
• Process simple JSON structures that can't be modeled with the Documents module

Here is the PeopleCode required to invoke JavaScript

Local JavaObject &manager =  CreateJavaObject("javax.script.ScriptEngineManager");
Local JavaObject &engine =  &manager.getEngineByName("JavaScript");

REM ** Evaluate a simple JavaScript;
&engine.eval("var result = Math.random();");

REM ** Access the value of the JavaScript variable named result;
Local string &result_text =  &engine.get("result").toString();

Here is some JavaScript that converts the variable &json_string into a JSON Array and then iterates over each entry, inserting values into a table. Notice that I'm invoking the PeopleCode SQLExec function from JavaScript.

var result = (function() {
  var SQLExec = Packages.PeopleSoft.PeopleCode.Func.SQLExec;
  var json = JSON.parse(json_string);
  var count = 0;
  json.forEach(function(item, idx) {
    SQLExec("INSERT INTO ... SYSTIMESTAMP", [idx, item]);
    count++;
  });
  return count + " rows inserted";
}());

Where did that &json_string variable come from? Here:

&engine.put("json_string", "[""item1"", ""item2"", ""item3""]");

Writing to stdout and stderr Take II

A few years ago I wrote a post that describes how to print text to the App Engine output file, bypassing the verbose MessageBox statement with its text length limitation (AppEngine Output Tricks, Reporting, Logging, Etc). I recently employed that same technique for logging Taleo Connect Client output when run from an App Engine. Looking at the code from those old println methods, I noticed a couple of inefficiencies:

• Each invocation of println initializes the same JavaObject variables resulting in wasted CPU cycles and wasted memory
• The two functions, println_to_stderr and println_to_stdout, are nearly identical which violates the DRY principle

The solution to both of these problems is inherent in Application Classes. We can solve the first issue by maintaining state inside the App Class with private instance variables. The DRY violation could be solved through properly implemented composition (preferred) or inheritance.

Here is an alternate implementation of the println_to_stdout method that uses an App Class to maintain state between invocations:

class StdoutWriter
   method println(&message As string);
   
private
   
   instance JavaObject &printlnMethod_;
   instance JavaObject &outputStream_;
end-class;

method println
   /+ &message as String +/
   /+ Extends/implements NAA_STDIO:IOWriter.println +/
   
   REM ** Lazy initializtion to ensure initialized between invocations;
   REM ** Need a local copy of member for "None" test;
   Local JavaObject &outputStream = &outputStream_;
   If (None(&outputStream)) Then
      REM ** NOTE: this is the only difference between StderrWriter and StdoutWriter;
      &outputStream_ = GetJavaClass("java.lang.System").out;
      
      Local JavaObject &stringClass = GetJavaClass("java.lang.Class").forName("java.lang.String");
      Local JavaObject &printStreamCls = &outputStream_.getClass();
      Local JavaObject &printlnArgTypes = CreateJavaObject("java.lang.Class[]", &stringClass);
      
      &printlnMethod_ = &printStreamCls.getDeclaredMethod("println", &printlnArgTypes);
   End-If;
   
   &printlnMethod_.invoke(&outputStream_, CreateJavaObject("java.lang.Object[]", &message));
   rem ** I didn't find flushing necessary, but here is where you would flush the buffer if desired;
   rem &outputStream_.flush();
end-method;

Notice that I used lazy initialization and only persist two JavaObject variables. The life of a JavaObject variable is potentially shorter than most PeopleCode variables. The PeopleSoft runtime can persist many types of PeopleCode variables across requests (events, think time functions, etc). This is not the case with Java variables. Lazy initialization and re-initialization ensures those Java variables always have a value. I only persist two variables rather than the seven from the original println_to_stdout function because we only need two of those variables for subsequent invocations.

About the DRY principle violation... I chose not to solve it. The number of lines required to create another app class (for composition or inheritance) was about the same as the number of duplicate lines. If I had multiple targets besides stderr and stdout, then creating a class structure to contain this redundant code would make sense. In this case the clarity seemed worth a little redundancy.

So how do you use it? Assuming you put this class in an Application Package named JJM_STDIO, you would call it like this:

import JJM_STDIO:StdoutWriter;

Local JJM_STDIO:StdoutWriter &out = create JJM_STDIO:StdoutWriter();

&out.println("This could be a very long line of text read from some process output that would exceed the maximum length for MessageBox");

Convert Byte Array into String

In the OTN forums, someone recently asked how to convert a byte array into a String. Assuming the byte array contains characters, not binary data, you can convert a binary array into a string by using the Java String byte array constructor. Here is a short example:

REM ** Create an array of bytes for testing purposes;
Local JavaObject &input = CreateJavaObject("java.lang.String", "A test string.");
Local JavaObject &bytes = &input.getBytes();

REM Convert the bytes back into a String;
Local JavaObject &output = CreateJavaObject("java.lang.String", &bytes);

MessageBox(0, "", 0, 0, &output.toString());

Manipulating Zip Files with PeopleCode

I've seen a few forum posts that show how to zip files using both Exec and the XML Publisher PSXP_RPTDEFNMANAGER:Utility app package. Those are great options, but might not fit every scenario. Since the Java API includes support for zip files, let's investigate how we can use it to create or extract zip files.

Java allows developers to create zip files by writing data to a ZipOutputStream. We've used OutputStreams a few times on this blog to write data to files. A ZipOutputStream is just a wrapper around an OutputStream that writes contents in the zip file format. Here is an example of reading a text file and writing it out to a ZipOutputStream

REM ** The file I want to compress;
Local string &fileNameToZip = "c:\temp\blah.txt";

REM ** The internal zip file's structure -- internal location of blah.txt;
Local string &zipInternalPath = "my/internal/zip/folder/structure";

Local JavaObject &zip = CreateJavaObject("java.util.zip.ZipOutputStream", CreateJavaObject("java.io.FileOutputStream", "c:\temp\compressed.zip", True));

Local JavaObject &file = CreateJavaObject("java.io.File", &fileNameToZip);
REM ** We will read &fileNameToZip into a buffer and write it out to &zip;
Local JavaObject &buf = CreateJavaArray("byte[]", 1024);

Local number &byteCount;
Local JavaObject &in = CreateJavaObject("java.io.FileInputStream", &fileNameToZip);

Local JavaObject &zipEntry = CreateJavaObject("java.util.zip.ZipEntry", &zipInternalPath | "/" | &file.getName());

REM ** Make sure zip entry retains original modified date;
&zipEntry.setTime(&file.lastModified());

&zip.putNextEntry(&zipEntry);

&byteCount = &in.read(&buf);

While &byteCount > 0
   &zip.write(&buf, 0, &byteCount);
   &byteCount = &in.read(&buf);
End-While;

&in.close();
&zip.flush();
&zip.close();

To add multiple files to a single zip file, we can convert the above code into a function (preferably a FUNCLIB function) and then call it multiple times, once for each file:

Function AddFileToZip(&zipInternalPath, &fileNameToZip, &zip)
   Local JavaObject &file = CreateJavaObject("java.io.File", &fileNameToZip);
   REM ** We will read &fileNameToZip into a buffer and write it out to &zip;
   Local JavaObject &buf = CreateJavaArray("byte[]", 1024);
   
   Local number &byteCount;
   Local JavaObject &in = CreateJavaObject("java.io.FileInputStream", &fileNameToZip);
   
   Local JavaObject &zipEntry = CreateJavaObject("java.util.zip.ZipEntry", &zipInternalPath | "/" | &file.getName());
   
   REM ** Make sure zip entry retains original modified date;
   &zipEntry.setTime(&file.lastModified());
   
   &zip.putNextEntry(&zipEntry);
   
   &byteCount = &in.read(&buf);
   
   While &byteCount > 0
      &zip.write(&buf, 0, &byteCount);
      &byteCount = &in.read(&buf);
   End-While;
   
   &in.close();
End-Function;


Local JavaObject &zip = CreateJavaObject("java.util.zip.ZipOutputStream", CreateJavaObject("java.io.FileOutputStream", "c:\temp\compressed.zip", True));

AddFileToZip("folder1", "c:\temp\file1.txt", &zip);
AddFileToZip("folder1", "c:\temp\file2.txt", &zip);
AddFileToZip("folder2", "c:\temp\file1.txt", &zip);
AddFileToZip("folder2", "c:\temp\file2.txt", &zip);

&zip.flush();
&zip.close();

The contents to zip doesn't have to come from a static file in your file system. It could come from the database or... well, anywhere. Here is an example of zipping static text. In this example I intentionally left the internal zip file path (folder) blank to show how to create a zip file with no structure.

Local JavaObject &textToCompress = CreateJavaObject("java.lang.String", "This is some text to compress... probably a bloated XML document or something ;)");
Local string &zipInternalFileName = "contents.txt";

Local JavaObject &zip = CreateJavaObject("java.util.zip.ZipOutputStream", CreateJavaObject("java.io.FileOutputStream", "c:\temp\compressed.zip", True));
Local JavaObject &zipEntry = CreateJavaObject("java.util.zip.ZipEntry", &zipInternalFileName);
Local JavaObject &buf = &textToCompress.getBytes();
Local number &byteCount = &buf.length;

&zip.putNextEntry(&zipEntry);

&zip.write(&buf, 0, &byteCount);

&zip.flush();
&zip.close();

And, finally, unzipping files. The following example prints the text inside each file from a zip file named "compressed.zip" that contains four fictitious text files named file1.txt, file2.txt, file3.txt, and file4.txt.

Local JavaObject &zipFileInputStream = CreateJavaObject("java.io.FileInputStream", "c:\temp\compressed.zip");
Local JavaObject &zipInputStream = CreateJavaObject("java.util.zip.ZipInputStream", &zipFileInputStream);
Local JavaObject &zipEntry = &zipInputStream.getNextEntry();
Local JavaObject &buf = CreateJavaArray("byte[]", 1024);
Local number &byteCount;

While &zipEntry <> Null
   
   If (&zipEntry.isDirectory()) Then
      REM ** do nothing;
   Else
      Local JavaObject &out = CreateJavaObject("java.io.ByteArrayOutputStream");
      &byteCount = &zipInputStream.read(&buf);
      
      While &byteCount > 0
         &out.write(&buf, 0, &byteCount);
         &byteCount = &zipInputStream.read(&buf);
      End-While;
      
      &zipInputStream.closeEntry();
      MessageBox(0, "", 0, 0, &out.toString());
      /*Else
         &log.writeline("&zipEntry is a directory named " | &zipEntry.getName);*/
   End-If;
   
   &zipEntry = &zipInputStream.getNextEntry();
End-While;

&zipInputStream.close();
&zipFileInputStream.close();

What about unzipping binary files into the file system? I'll let you write that one.

Password protected zip files? Java doesn't make this easy. There are a few Java libraries, but as Chris Rigsby points out here, using non-standard Java classes (including your own) can be hazardous. At this time, it seems the best way to password protect a zip file is to use Exec to call a command line zip program. On Linux with the zip utility, use the -P parameter to encrypt with a password.

Creating Binary Arrays

Lately I have been using PeopleCode to manipulate binary files: moving files, copying files, and even creating zip files. A prerequisite for reading from and writing to binary files is the basic binary array -- the buffer. My blog post Base64 Encoding for PeopleSoft demonstrated a very complicated method for creating binary files that worked with PeopleTools 8.49 and earlier, but does not work on my PeopleTools 8.51 systems. While studying PeopleBooks I found a much easier, well documented method for creating binary arrays:

Local JavaObject &bytes = CreateJavaArray("byte[]", 1024 /* length of array */);

For arrays with known values at construction time, you can use the CreateJavaObject function:

Local JavaObject &bytes = CreateJavaObject("byte[]", 5, 10, 15, 20);

Note: Since this is documented, I suspect these functions will work with PeopleTools 8.49 and earlier, but I haven't tested them on earlier PeopleTools versions. If this method won't work in PeopleTools 8.49 or earlier, then you are welcome to use the alternative:

REM ** get a reference to a Java class instance for the primitive byte array;
Local JavaObject &arrayClass = GetJavaClass("java.lang.reflect.Array");
Local JavaObject &bytes = &arrayClass.newInstance(GetJavaClass("java.lang.Byte").TYPE, 5);

I would like to call out Kris who posted a comment on Base64 Encoding for PeopleSoft stating that the older method no longer worked. I happened to be working on zipping files with PeopleCode the week before Kris's comment and discovered the same issue and resolution. Very timely.

Accessing PeopleCode Rowsets from Java

A reader recently asked how to create instances of the Rowset class from Java. I believe the question was more about IDE and classpath setup than it was about actual Java code. But, since it can be difficult to figure out how to use PeopleCode objects in Java, I thought I would post an example:

package test.peoplecode;

import PeopleSoft.PeopleCode.Func;
import PeopleSoft.PeopleCode.Name;
import PeopleSoft.PeopleCode.Rowset;

  public static String getOprDescr(String oprid) {
    Name recName = new Name("RECORD", "PSOPRDEFN");
    Name fieldName = new Name("FIELD", "OPRDEFNDESC");
    Rowset r = Func.CreateRowset(recName, new Object[] { });

    r.Fill(new Object[] { "WHERE OPRID = :1", oprid });

    return (String)r.GetRow(1).GetRecord(recName).GetField(fieldName).getValue();
  }
}

Notice that the first parameter to CreateRowset is a Name object and the second is an empty array. If I were creating a hierarchical Rowset (similar to a component buffer), then I would fill the array with additional Rowset objects, as described by the CreateRowset PeopleBooks entry. Another important difference between PeopleCode and Java is that the "RECORD" and "FIELD" parameters to the Name constructor must be upper case.

Here is some PeopleCode to test this example:

MessageBox(0, "", 0, 0, GetJavaClass("test.peoplecode.RowsetTest").getOprDescr(%OperatorId));

What about the IDE's Java project classpath? If your IDE supports library definitions (like JDeveloper), then add the JAR %PS_HOME%\class\peoplecode.jar as a new library and then add the library to your project.

JSON Encoding in PeopleCode

I am a big fan of the JSON.simple Java library. JSON.simple integrates well with PeopleCode. It produces flawless JSON without ugly PeopleCode Java Reflection and is compatible with Java 1.2 (for older tools versions). Yes, the object/array to JSON conversion in JSON.simple is nice, but my real reason for using a JSON library is JSON encoding. I can mock up and string together variable values to produce JSON, but my main problem is escaping strings so that they represent safe JSON data (quotes, etc). I thought the PeopleCode EscapeJavascriptString function would handle this for me, but I discovered that JSON != JavaScript. Certain character sequences, such as \' are valid for JavaScript, but invalid for JSON. After my latest tools and app upgrade, I decided to see what it would take to encode strings for JSON from PeopleCode. Here is what I created:

class JSONEncoder
   method encode(&input As string) Returns string;
   
private
   instance JavaObject &meta_chars_;
   instance JavaObject &unsafe_chars_pattern_;
   instance JavaObject &int_;
   
   method init();
end-class;

method encode
   /+ &input as String +/
   /+ Returns String +/
   
   Local JavaObject &matcher;
   Local string &output = &input;
   Local string &replacement;
   Local string &match;
   Local number &offset = 1;
   
   REM ** Run lazy init if needed;
   REM ** Protects against stateless PeopleCode/Stateful JVM;
   %This.init();
   
   &matcher = &unsafe_chars_pattern_.matcher(CreateJavaObject("java.lang.String", &input));
   
   While &matcher.find()
      &match = &matcher.group();
      
      If (&meta_chars_.containsKey(&match)) Then
         REM ** replace meta characters first;
         &replacement = &meta_chars_.get(&match).toString();
      Else
         REM ** not meta, so convert to a unicode escape sequence;
         &replacement = "\u" | Right("0000" | &int_.toHexString(Code(&match)), 4);
      End-If;
      &output = Replace(&output, &matcher.start() + &offset, (&matcher.end() - &matcher.start()), &replacement);
      
      REM ** move the starting position based on the size of the string after replacement;
      &offset = &offset + Len(&replacement) - (&matcher.end() - &matcher.start());
   End-While;
   
   Return &output;
end-method;

method init
   REM ** None only works on local vars, so get a pointer;
   Local JavaObject &int = &int_;
   
   REM ** if &int has no value, then initialize all JavaObject vars;
   /*
    * JavaObject vars will have no value in two scenarios:
    *
    *   1. First use, never initialized
    *   2. Think time function, global variable, anything that causes state
    *      serialization.
    *
    * The first case is obvious. The second case, however, is not. PeopleSoft
    * allows you to make App Classes Global and Component scoped objects, but
    * not JavaObject variables. By using JavaObject variables in Component and
    * Global scope, you can get into a bit of trouble. Retesting these values
    * on each use ensures they are always initialized. The same will happen if
    * you use a think-time function like Prompt or a Yes/No/Cancel MessageBox.
    */
   If (None(&int)) Then
      REM ** Lazy initialize Integer class;
      &int_ = GetJavaClass("java.lang.Integer");
      
      REM ** Lazy initialize the regular expression;
      REM ** List other unsafe characters;
      &unsafe_chars_pattern_ = GetJavaClass("java.util.regex.Pattern").compile("[\\""\x00-\x1f\x7f-\x9f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]");
      
      REM ** Lazy initialize the hashtable;
      &meta_chars_ = CreateJavaObject("java.util.Hashtable");
      
      REM ** setup meta characters;
      &meta_chars_.put(Char(8), "\b");
      &meta_chars_.put(Char(9), "\t");
      &meta_chars_.put(Char(10), "\n");
      &meta_chars_.put(Char(12), "\f");
      &meta_chars_.put(Char(13), "\r");
      &meta_chars_.put("\", "\\");
      &meta_chars_.put("""", "\""");
   End-If;
   
end-method;

I adapted this code from the JavaScript quote function in the json.org JSON2 JavaScript parser. Yes, this solution does still use Java (regular expressions and hexadecimal encoding), but it doesn't require external libraries. See, my real motivation was to eliminate external dependencies. I wanted code I could compile and leave in the database; code that didn't require OS file system modifications; code that would upgrade without impacting PS_HOME, psappsrv.cfg, psconfig.sh, or any other upgraded configuration file.

Why an App Class instead of a FUNCLIB? I originally wrote this code as a FUNCLIB function. Step one of the function would populate the hashtable. This meant for each function call, I would incur the overhead of creating and populating the hashtable. Since I know I will call this function multiple times while constructing a JSON string, I wanted a mechanism to persist the hashtable across function calls. An App Class's private instance variable provides this mechanism. What about Global variables? First, I have NEVER used them. Second, you CAN'T use them with variables of type JavaObject. What about serialization, scoping, and think-time functions with Java? I protect against the "First operand of . is Null" error by lazily initializing the hashtable and the regular expression. A postback will reset the JavaObject to Null, and my lazy initialization code will reinitialize it.