Generating LARGE JSON Files

The PeopleCode native JsonObject and JsonArray classes allow us to create JSON structures as in-memory representations. But what if you need to generate a really LARGE JSON structure? An in-memory JSON Array may consume more memory than you can reasonably allow. Fortunately, PeopleTools includes the Jakarta JSON library, which allows us to write a JSON structure to a stream during construction.

The following code snippet demonstrates creating 10 million JSON objects in an array without any change in memory consumption. The generated file was 2.5 GB in size, but my memory utilization didn't change the entire time the program ran.

Local JavaObject &Json = GetJavaClass("jakarta.json.Json");
Local JavaObject &writer = CreateJavaObject("java.io.FileWriter", "C:\temp\users-big.json");

Local JavaObject &gen = &Json.createGenerator(&writer);

Local number &iteration = 1;

REM ** 10 million iterations;
Local number &maxIterations = 10000000;

&gen.writeStartArray();

For &iteration = 1 To &maxIterations
   
   REM ** start person/user object;
   &gen.writeStartObject();
   &gen.write("id", "" | &iteration);
   &gen.write("firstName", "John");
   &gen.write("lastName", "Smith");
   
   REM ** start child address object;
   &gen.writeStartObject("address");
   &gen.write("streetAddress", "21 2nd Street");
   &gen.write("city", "New York");
   &gen.write("state", "NY");
   &gen.write("postalCode", "10021");
   &gen.writeEnd();
   
   REM ** start phone number array;
   &gen.writeStartArray("phoneNumber");
   
   REM ** start home phone object;
   &gen.writeStartObject();
   &gen.write("type", "home");
   &gen.write("number", "212 555-1234");
   &gen.writeEnd();
   
   REM ** start fax number object;
   &gen.writeStartObject();
   &gen.write("type", "fax");
   &gen.write("number", "646 555-4567");
   &gen.writeEnd();
   
   REM ** end array of phone numbers;
   &gen.writeEnd();
   
   REM ** end person/user object;
   &gen.writeEnd();
End-For;


REM ** end array;
&gen.writeEnd();

REM ** cleanup to flush buffers;
&gen.close();
&writer.close();

The hard-coded values come directly from the Jakarta generator API documentation. In real life, you would replace these values with database data. I converted numbers to strings to simplify the example to avoid Java Reflection.

Are you interested in parsing rather than generating large JSON files? Check out our post on JSON Stream Parsing.

We teach PeopleTools and PeopleCode tips like this every week! Check out our upcoming course schedule to see what we are offering next! We would love to have you join us. Want to learn at your own pace? Check out our subscriptions and on-demand offerings as well. Or do you have a group you would like to train? Contact us for group and quantity discounts.