Developer Transformation Guide

10.5
- 10.5.2
- 10.4.1
- 10.4.0

Back Next

Complex Data Type Conversion on the Spark Engine

You can add complex ports to the Java transformation in mappings that run on the Spark engine. A complex port is a port that is assigned a complex data type. The Java transformation converts complex data types to comparable Java complex data types. It also converts elements of complex data types to comparable Java data types, which are a boxed version of primitive data types.

Array Data Type

When a Java transformation reads input rows, it converts array data type to Java List data type. The transformation converts the data type of array elements to a boxed version of Java data types.

For example, the transformation converts the Developer tool complex data type

array<integer>

to a Java data type

List<Integer>

When a Java transformation writes output rows, it converts Java List data type to array data type. The transformation converts the data type of List elements to an unboxed version of the Developer tool data types.

Struct Data Type

When a Java transformation reads input rows of struct data type, it generates an equivalent Java bean class. The name of the Java bean class is the same as the name of the struct data type. The transformation converts the data type of struct elements to a boxed version of Java data types.

If the struct data type uses special characters or Java reserved keywords, such as final or private, the Java transformation replaces the special characters with an underscore (_) in the class name. You can open the full code in the Java view of the transformation properties tab to see the generated classes.

The Java transformation generates class member field names prefixed with an underscore (_). It also generates getters and setters for the member fields.

The generated Java bean class has the namespace of the type definition library name as an outer class. The name of the outer class is the same as the name of the type definition library. The Java data type name of the struct port is of the following format:

type_library_name.struct_type_name

For example, the type library name is m_Type_Definition_Library. The complex data type definition for the struct port is:

Customer {
  name string
  age  integer
 }

The Java transformation generates the Java bean classes with getters and setters for the member fields. The following code snippet shows the outer class and inner class:

public static final class m_Type_Definition_Library {
 public static final class Customer implements Serializable {
  private String _name;
  private Integer _age;

 public Customer() {}

The following code snippet shows the getter and setter for the struct element name of type string:

public String get_name() {
 return _name;
	}

public void set_name(String _name) {
 this._name = _name;
 }

When a Java transformation writes output rows of struct data type, it converts Java bean class to struct data type. The transformation converts the member fields of the class to struct elements. The data type of member fields are converted to an unboxed version of the Developer tool data types.

Map Data Type

When a Java transformation reads input rows, it converts map data type to Java Map data type. The transformation converts the data type of map elements to a boxed version of Java data types.

For example, the Developer tool complex data type

map<string, bigint>

is converted to a Java data type

Map<String, long>

When a Java transformation writes output rows, it converts Java Map data type to map data type. The transformation converts the data type of Map elements to an unboxed version of the Developer tool data types.

Rename Saved Search

Table of Contents

Developer Transformation Guide

Developer Transformation Guide

Complex Data Type Conversion on the Spark Engine

Complex Data Type Conversion on the Spark Engine

Array Data Type

Struct Data Type

Map Data Type