com.ibm.streamsx.topology (Java Application API for IBM Streams)
Date: 2020-08-07

# Package com.ibm.streamsx.topology

Java application API for IBM Streams.

See: Description

- Interface Summary
  Interface | Description
  --- | ---
  TopologyElement | Any element in a `Topology`.
  TSink | A handle to the `sink` of a `TStream`.
  TStream<T> | A `TStream` is a declaration of a continuous sequence of tuples.
  TWindow<T,K> | Declares a window of tuples for a `TStream`.

- Class Summary
  Class | Description
  --- | ---
  Topology | A declaration of a topology of streaming data.

- Enum Summary
  Enum | Description
  --- | ---
  TStream.Routing | Enumeration for routing tuples to parallel channels.

## Package com.ibm.streamsx.topology Description

Java application API for IBM Streams. This API is used to generate streaming topologies for execution by IBM Streams 4.0.1 or later. An instance of `Topology` is created that then is used to build a topology (or graph) of streams, represented by `TStream` instances. This topology is the declaration of the application, and it may then be :

- Submitted to an IBM Streams instance.
- Executed as an IBM Streams standalone application.
- Compiled into an IBM Streams application bundle (`.sab` file).
- Executed within the local JVM (not all toppologies are supported).

A `TStream` represents a continuous stream of tuples with a specific type `T`. Streams are transformed into new streams through functional transformations, defined using a single method of an interface like `Function`.

A transformation consists of an instance of a functional interface, known as the functional logic and how that logic is applied against tuples on the stream. The method on `TStream` the functional logic is passed into to create a new stream, declares how the functional logic is applied, for example:

- `TStream.filter(com.ibm.streamsx.topology.function.Predicate)` declares that the functional logic will be called for each tuple on the stream, and if it returns `true` then the input tuple is submitted to the new stream, otherwise the input tuple is discarded.
- `TStream.transform(com.ibm.streamsx.topology.function.Function)` declares that the functional logic will be called for each tuple and its return value will be submitted to the output stream, unless it is `null`, in which case no tuple will be submitted.

### Functional Logic semantics

#### Declaring Functional Logic

When declaring (building) the topology any functional logic is an instance of a Java interface, such as `Predicate`. These objects are always serialized and deserialized before being executed against tuples on streams, regardless of the type of the `StreamsContext`.

- If the instance is declared using an anonymous class then it should be declared in a static context (e.g. static method). If the anonymous class is not declared in a static context then it will also capture the reference of the object that declares it, typically this object (the code declaring the topology) will not be serializable, and is not required by the anonymous class for execution.

- The code declaring the topology passes parameters into an anonymous class by having local variables or method parameters that are `final` and thus can be captured by the anonymous class.
- The code within an anonymous class must not reference any static fields set up by the code declaring the topology, as the code declaring the topology is not executed when the topology itself is executed. For example a method in an anonymous class that references a static file name set when declaring the topology, will not see that value when executing at runtime.

- When the functional logic is deserialized before being executed at runtime, its constructor is not called. Thus any transient fields either need to be initialized on their first use or using the Java serialization hooks `readObject()` or `readResolve()`. The sample `RegexGrep` has an example of this.
- A single deserialized instance of functional logic is called for all tuples on each `parallel channel`, when the stream is not parallelized a single instance is called.

#### Synchronization of Functional Logic

- Call to a functional logic are not executed concurrently.
- Each call to a functional logic happens-before any subsequent call to it.

Thus there is no need for a functional logic to have synchronization to protect state from multiple concurrent calls or to ensure visibility.

#### Stateful Functional Logic

A functional logic instance lives for the lifetime of its container (embedded JVM, standalone process or processing element in distributed mode), thus it may maintain state across the invocations of its method. For a de-duplicating `Predicate` may maintain a collection of previously seen tuples on the stream to filter out duplicates.
In distributed mode, if a processing element (PE) restarts then any state will be lost and a new functional logic instance set to its initial deserialized state is created.
For future compatibility:

- any state that should not be persisted on a checkpoint must be marked as `transient`, such as connections to external systems.
- any non-changing instance fields should be marked as `final`.

### Pass by reference semantics for tuples

Where possible, tuples are passed by reference from one stream to another, thus in a general case, a tuple (as a Java object) is returned from one functional logic instance and passed into one or more downstream functional logic instances, using its reference. For example, with the flow:
`s.transform(trans1).filter(filt).transform(trans2)`
a tuple `t` returned from `trans1.apply(t)` may be passed directly as `filt.test(t)` and to `trans2.apply(t)` if the former returned `true`.
When a tuple is not, or can not, be passed by reference, for example when functional logic instances are being executed on different hosts in a distributed environment, then the tuple object will passed using serialization and de-serialization.

#### Ensuring consistent behavior

To ensure consistent behavior regardless of how the topology is executed, these recommendations should be followed:

- Where possible `TStream` types should be immutable, which means any downstream functional logic cannot modify any tuple through its reference. Thus a reference to a tuple may be safely passed to multiple downstream functional logic instances.
- Any functional logic discards any reference to any mutable returned item, thus any change made to an mutable tuple object are not seen by any upstream functional logic.
- Any functional logic discards any reference to any mutable argument when it returns from its method..

### Included Libraries

This API requires the IBM Streams Java Operator API which results in these libraries in the class path:

- IBM Streams Java Operator API - `$STREAMS_INSTALL/lib/com.ibm.streams.operator.jar`
- IBM Streams Java Operator samples - `$STREAMS_INSTALL/lib/com.ibm.streams.operator.samples.jar`
- Apache Commons Math 2.2 - `$STREAMS_INSTALL/ext/lib/commons-math-2.1.jar`
- Apache Log4j 1.2.17 - `$STREAMS_INSTALL/ext/lib/log4j-1.2.17.jar`
- JSON4J - `$STREAMS_INSTALL/ext/lib/JSON4J.jar`