Wednesday, April 14, 2010

JSON Encoding in PeopleCode

I am a big fan of the JSON.simple Java library. JSON.simple integrates well with PeopleCode. It produces flawless JSON without ugly PeopleCode Java Reflection and is compatible with Java 1.2 (for older tools versions). Yes, the object/array to JSON conversion in JSON.simple is nice, but my real reason for using a JSON library is JSON encoding. I can mock up and string together variable values to produce JSON, but my main problem is escaping strings so that they represent safe JSON data (quotes, etc). I thought the PeopleCode EscapeJavascriptString function would handle this for me, but I discovered that JSON != JavaScript. Certain character sequences, such as \' are valid for JavaScript, but invalid for JSON. After my latest tools and app upgrade, I decided to see what it would take to encode strings for JSON from PeopleCode. Here is what I created:

class JSONEncoder
method encode(&input As string) Returns string;

private
instance JavaObject &meta_chars_;
instance JavaObject &unsafe_chars_pattern_;
instance JavaObject &int_;

method init();
end-class;

method encode
/+ &input as String +/
/+ Returns String +/

Local JavaObject &matcher;
Local string &output = &input;
Local string &replacement;
Local string &match;
Local number &offset = 1;

REM ** Run lazy init if needed;
REM ** Protects against stateless PeopleCode/Stateful JVM;
%This.init();

&matcher = &unsafe_chars_pattern_.matcher(CreateJavaObject("java.lang.String", &input));

While &matcher.find()
&match = &matcher.group();

If (&meta_chars_.containsKey(&match)) Then
REM ** replace meta characters first;
&replacement = &meta_chars_.get(&match).toString();
Else
REM ** not meta, so convert to a unicode escape sequence;
&replacement = "\u" | Right("0000" | &int_.toHexString(Code(&match)), 4);
End-If;
&output = Replace(&output, &matcher.start() + &offset, (&matcher.end() - &matcher.start()), &replacement);

REM ** move the starting position based on the size of the string after replacement;
&offset = &offset + Len(&replacement) - (&matcher.end() - &matcher.start());
End-While;

Return &output;
end-method;

method init
REM ** None only works on local vars, so get a pointer;
Local JavaObject &int = &int_;

REM ** if &int has no value, then initialize all JavaObject vars;
/*
* JavaObject vars will have no value in two scenarios:
*
* 1. First use, never initialized
* 2. Think time function, global variable, anything that causes state
* serialization.
*
* The first case is obvious. The second case, however, is not. PeopleSoft
* allows you to make App Classes Global and Component scoped objects, but
* not JavaObject variables. By using JavaObject variables in Component and
* Global scope, you can get into a bit of trouble. Retesting these values
* on each use ensures they are always initialized. The same will happen if
* you use a think-time function like Prompt or a Yes/No/Cancel MessageBox.
*/
If (None(&int)) Then
REM ** Lazy initialize Integer class;
&int_ = GetJavaClass("java.lang.Integer");

REM ** Lazy initialize the regular expression;
REM ** List other unsafe characters;
&unsafe_chars_pattern_ = GetJavaClass("java.util.regex.Pattern").compile("[\\""\x00-\x1f\x7f-\x9f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]");

REM ** Lazy initialize the hashtable;
&meta_chars_ = CreateJavaObject("java.util.Hashtable");

REM ** setup meta characters;
&meta_chars_.put(Char(8), "\b");
&meta_chars_.put(Char(9), "\t");
&meta_chars_.put(Char(10), "\n");
&meta_chars_.put(Char(12), "\f");
&meta_chars_.put(Char(13), "\r");
&meta_chars_.put("\", "\\");
&meta_chars_.put("""", "\""");
End-If;

end-method;

I adapted this code from the JavaScript quote function in the json.org JSON2 JavaScript parser. Yes, this solution does still use Java (regular expressions and hexadecimal encoding), but it doesn't require external libraries. See, my real motivation was to eliminate external dependencies. I wanted code I could compile and leave in the database; code that didn't require OS file system modifications; code that would upgrade without impacting PS_HOME, psappsrv.cfg, psconfig.sh, or any other upgraded configuration file.

Why an App Class instead of a FUNCLIB? I originally wrote this code as a FUNCLIB function. Step one of the function would populate the hashtable. This meant for each function call, I would incur the overhead of creating and populating the hashtable. Since I know I will call this function multiple times while constructing a JSON string, I wanted a mechanism to persist the hashtable across function calls. An App Class's private instance variable provides this mechanism. What about Global variables? First, I have NEVER used them. Second, you CAN'T use them with variables of type JavaObject. What about serialization, scoping, and think-time functions with Java? I protect against the "First operand of . is Null" error by lazily initializing the hashtable and the regular expression. A postback will reset the JavaObject to Null, and my lazy initialization code will reinitialize it.

34 comments:

Chili Joe said...

Hi Jim,

I haven't tried it with a JavaObject, however, I recall it is possible to declare variables on top of the FUNCLIB code - before the 1st function. These variables will be shared across all functions on the same code, provided those functions are called in the same execution/event. It's one way to encapsulate data without using an application class.

Jim Marion said...

@Chili Joe, I first created this as a FUNCLIB. Then I recognized the need for reuse. Next, for reuse, I did exactly what you recommend. I actually found that every call to the FUNCLIB reinitialized the JavaObject hashtable. The JavaObject variables were null on every call. This was true even though I called the FUNCLIB function several times from the same PeopleCode event. I was actually surprised because I thought it would keep the local data as long as my code was executing. I expected it to reset after the event ended, since I was using a JavaObject, but I didn't expect it to reset on each call within the same event.

Like you say, this could be unique to JavaObject. I didn't try it with other types of variables (string, number, etc). It was my understanding that declaring variables (but not setting them) above a function declaration would keep them alive, but this wasn't the case with JavaObject.

Jim Marion said...

I just did some testing on this to confirm the behavior of Local scoped variables declared in an event above a FUNCLIB function. As Chili Joe pointed out, variables declared above a function are shared with other functions within the same PeopleCode event. For a FUNCLIB, this means that a FUNCLIB function that calls other functions within the same event will share the same variable instance. Each call to a FUNCLIB function outside the event, however, will result in a new initialization of that variable, eliminating reuse from outside an event. I will post my test code shortly.

Jim Marion said...

Here is my test case for the discussion thus far: FUNCLIB's and Event Scoped Variables.

Raj said...

Hi Jim,
I have a requirement to get all the zip codes in range given the base zip and range in miles. This is a standard function whose code is commonly available. I need to transalate this in peoplecode. The code uses lots of math functions like Sin, Cos, Pow() etc..
Is there a way to just call this funtion (in php, or java or vb) from within peoplecode?

Thanks
Raj

Jim Marion said...

@Raj, if you can get a zipcode range calculator in Java form, then the easiest thing to do is to place the class/jar files in your app server class path and use the PeopleCode CreateJavaObject and GetJavaClass functions. I have several examples of using Java from PeopleCode in this blog.

42N-71W said...

Hi Jim!

So I am trying to serve JSON outside of PeopleSoft and I did all the IB setup - but I would think I won't be able to use %Reponse.write as I have to return a message object when I implement PS_PT:Integration:IRequestHandler. Its easy to create an XML response using return &response_MSG.SetXmlDoc(&somexmldoc); but how to return a JSON message..

Jim Marion said...

@42N-71W,

Use an unstructured message and create a single node XML document containing the attribute psnonxml="yes". Here is a PeopleCode snippet:


Local string &nonXmlData = "" |
"";
Local XmlDoc &doc = CreateXmlDoc(&nonXmlData);

&response_msg.SetXmlDoc(&doc);
Return &response_msg;

Raj said...

Hi Jim,
I have a character encoding issue.I ma trying to send a french character
é to a third party application.
However,the character is coming as garbled characters.Kindly suggest.

Thanks in advance,
Raj

Jim Marion said...

@Raj, I suspect the "garbled" issue has to do with the string encoding (UTF, ASCII, etc). The easiest way to handle this type of character for HTML or XML is to use the HTML or XML entity rather than trust encodings. The HTML entity for this particular character is é. In XML you can use é (the ascii character code for é). Therefore, to properly encode these types of characters, you can convert them to their ascii # and then sandwich them between &# and ;

Unknown said...

Hi Jim,
I have a problem regarding encoding data. I have made my module in arabic and want to make web service but when i populate its JSON on page then its string pulated like "????????" plz guide me the solution?

Jim Marion said...

@Ayesha, I wish I had something to offer you. It sounds like an encoding issue (unicode, ascii, codepage, etc), but I think you already knew that. You may need to open a case with Oracle global support for this.

Unknown said...

Hi Jim,
i want to show Arabic on my browser in web service but Arabic words don't show. ?????? will show instead of Arabic.Here my code samle

Local string &nonXmlData = "هذه هي اللغة العربية.";
Local XmlDoc &DCM = CreateXmlDoc(&nonXmlData);
Local string &outStr = &DCM.GenFormattedXmlString();




&response = CreateMessage(Operation.LOGINSERVICE3_GET, %IntBroker_Response);
&response.SetXmlDoc(&DCM);

please suggest me how i encoded the Arabic in XMlDOC..
Thanks in Advance
regards Sarfaraz

Jim Marion said...

@sarfaraz, I believe the missing piece in your code is the XML wrapper for Integration Broker. Here is an example of code that I use to send non XML data (NOTE: I replaced less than/greater than with {} so blogger would accept this comment):

Local string &nonXmlData = "{?xml version=""1.0""?}" |
"{data psnonxml=""yes""}{![CDATA[هذه هي اللغة العربية]]}{/data}";
Local XmlDoc &doc = CreateXmlDoc(&nonXmlData);

If that doesn't work, the only other thing I can think of is encoding. Integration Broker is UTF-8. Just make sure your characters are within the allowed character range.

Unknown said...

Hi Jim, thank you for sharing your JSON encoding peoplecode. I created an app package class using your code, then wrote extra methods to convert various types of data to JSON strings, for example...

method BooleanToPair
/+ &inLabel as String, +/
/+ &inBoolean as Boolean +/
/+ Returns String +/

If None(&inLabel) Then
Return "";
Else
Return %This.encodeLabel(&inLabel) | Lower(String(&inBoolean));
End-If;

end-method;

It makes it quite easy to create even reasonably complex JSON objects rather than coding them manually.

Cheers,

Stephen

Jim Marion said...

@Stephen, thank you for sharing. I also created helper methods to do things like encode rowsets, etc. Of note, PeopleTools 8.53's Documents module supports rendering and parsing JSON. What is interesting, though is that collections, etc, have to follow a specific pattern, so you can't just create any valid JSON structure you want. The reason I bring this up is because even though PT 8.53 supports JSON, I think we will still be using code like this to encode JSON.

Unknown said...

Hi Jim, do you know of a way to determine the data type of an "Any" variable? I'd like to be able to turn an Array of Any into a JSON array but to do that I have to be able to figure out the data type of each element. I've written a method to do this but it has some limitations, e.g. it can't distinguish between a number value in a Number variable and a number value in a String variable.

Jim Marion said...

@Stephen, that is a good question. Have you considered the "Is" functions, such as IsNumeric, IsDigits, etc? Other than those, you can also try using a try/catch block around an operation that requires numbers or strings. You can put that in a function. If it makes it through the operation, then set the return value to true.

Anonymous said...

Hi, I've recently written a native JSON encoder for Peoplecode, if anyone is interested.

Jim Marion said...

@Cameron, yes, we are definitely interested. Can you share a link to it? Did you blog about it or upload it to github or some other sharing location?

Anonymous said...

Here is my getting started post:

http://cameronbarre.blogspot.com/2014/07/getting-up-and-running-with-jsoft.html

There is also a link in the post to all of the method documentation.

It's on Bitbucket here: https://bitbucket.org/cjbarre/jsoft.json

I copied the project to file, I'm no expert, but I believe anyone with Peopletools 8.53.07 or higher will be able to import the project from file.

I'm relatively new to the Peoplesoft scene, If anyone knows of a better way to package projects like this to make them easier to use, I'd appreciate the tips.

Jim Marion said...

@Cameron, how are you escaping Strings to make them JSON safe? I didn't see that in your source code.

Anonymous said...

I believe I've overlooked this, it's my first library, thanks for taking the time to look through it. Looks like it still needs work!

Jim Marion said...

@Cameron, feel free to include the encoder shown in this post. My post showed how to do the part you are missing, it didn't show how to convert an Array or object into a JSON structure like your code does.

Anonymous said...

@Jim, I've included your code in the project, it fires anytime a string value is added to an object or an array.

Alex Shilman said...

@Jim, great stuff. Keep em coming.
One thing, could you elaborate on
"
I protect against the "First operand of . is Null" error by lazily initializing the hashtable and the regular expression. A postback will reset the JavaObject to Null, and my lazy initialization code will reinitialize it.
",

currently I'm getting that same error when using &json.getClass ().. after parsing json on the inbound from Iscript %Request.getparameter..

Jim Marion said...

@Alex, I was referring to stateful/statelessness of the PeopleSoft session. It is possible to call a PeopleCode function from a PeopleCode function that halts the first PeopleCode and serializes the session (think-time functions). This serialization saves the state of ALL PeopleCode variables EXCEPT JavaObject variables. If you are experiencing this from an iScript, then I doubt you are having a think-time issue. iScripts run top to bottom with no stops... usually. I suspect there is something else causing your &json variable to be null. I would put some %Response.WriteLine(...) statements in the code to see if I can when it has a value, and when it doesn't. You can use the All() and None() functions to test if &json is valid. Hope that helps!

Unknown said...

Hi Jim this is my first post to the group and I am sure that I will get the solution to my problem. We recently upgraded our peoplesoft tools version to 8.55.13 where the following piece of code is not working hence need you expertise on this.
method doFileCharConversion
/+ &mFile_Name as String +/

Local File &lfile_ToConvert, &lfile_Converted;
Local number &lnbr_Pos;
Local string &lstr_ECOutName, &lstr_Data;

&lfile_ToConvert = GetFile(&mFile_Name, "r", "A", %FilePath_Absolute);

&lnbr_Pos = Find(".", &mFile_Name);
&lstr_ECOutName = Substring(&mFile_Name, 1, &lnbr_Pos) | "out";

&lfile_Converted = GetFile(&lstr_ECOutName, "W", "A", %FilePath_Absolute);
If &lfile_ToConvert.IsOpen And
&lfile_Converted.IsOpen Then

While &lfile_ToConvert.ReadLine(&lstr_Data);

For &idx = 1 To &parry_ConversionMatrix.Len

Local JavaObject &conv_regex = CreateJavaObject("java.lang.String", "[\" | &parry_ConversionMatrix [&idx][1] | "]");
Local JavaObject &test_string = CreateJavaObject("java.lang.String", &lstr_Data);

&lstr_Data = &test_string.replaceAll(&conv_regex, &parry_ConversionMatrix [&idx][2]);
End-For;

&lfile_Converted.WriteLine(&lstr_Data);
End-While;

&lfile_ToConvert.Delete();
&lfile_Converted.Close();
End-If;

end-method;

Jim Marion said...

@grover, when you say, "Not working" what do you mean specifically? Is there an error? If so, what is the error?

Unknown said...


I am having an issue with escape character at the following piece of code. In 8.53.14 tools the same code is working fine but after upgrade to peopletools 8.55.13 it is not working king as expected and not conveying the .dat file into .Out file. Not getting any error.
Following is the piece of code which is causing an issue


For &idx = 1 To &parry_ConversionMatrix.Len

Local JavaObject &conv_regex = CreateJavaObject("java.lang.String", "[\" | &parry_ConversionMatrix [&idx][1] | "]");

Jim Marion said...

Got it. You are expecting [\something] and you aren't getting a regex containing that. Do you need to create &conv_regex as a JavaObject String? Can't you just use local string &conv_regex? The PeopleCode runtime should convert it to a Java string automatically. If not, then your way is fine. Also, is your regex supposed to have a '\' in it? If so, you might need '\\' in your PeopleCode.

Without running the code, it is hard to know exactly why this is failing, but these are some of the things I would check.

Unknown said...

Thanks Jim for your suggestions. I will work on it and will update you accordingly.

Appreciate your help!!

Sudheer said...

Hi Jim,

I am facing while reading French special characters (â, ê, î, ô, û) through JSON request message. PeopleSoft reading as ? instead of actual special character. can you please guide me how to fix the issue.

Jim Marion said...

@Sudheer,

PeopleCode is a Unicode language, and properly recognizes all characters. The challenge may be with the encoding used to open the file for reading. Make sure you are using the same character encoding when reading as the encoding used when writing.