Overview (Java Application API for IBM Streams)
Date: 2020-08-07

Java API to allow creation of streaming applications for IBM Streams by Java & Scala developers.

See: Description

Packages
Package | Description
--- | ---
com.ibm.streamsx.rest | Java bindings for the REST APIs of IBM Streams and Streaming Analytics service.
com.ibm.streamsx.rest.build | Java binding for the Build REST APIs of IBM Cloud Pak for Data.
com.ibm.streamsx.rest.internal |
com.ibm.streamsx.topology | Java application API for IBM Streams.
com.ibm.streamsx.topology.consistent | At least once processing using consistent regions.
com.ibm.streamsx.topology.context | Context for executing IBM Streams Java topologies.
com.ibm.streamsx.topology.context.local |
com.ibm.streamsx.topology.file | File related streams.
com.ibm.streamsx.topology.function | Classes that provide the mechanisms to implement functional transformations on streams.
com.ibm.streamsx.topology.inet | Network access to streaming data.
com.ibm.streamsx.topology.jobconfig |
com.ibm.streamsx.topology.json | JSON streams.
com.ibm.streamsx.topology.logic | Utilities and implementations of functional logic.
com.ibm.streamsx.topology.messaging.kafka | Support for integrating with the Apache Kafka messaging system http://kafka.apache.org.
com.ibm.streamsx.topology.messaging.mqtt | Support for integrating with the MQTT messaging system http://mqtt.org.
com.ibm.streamsx.topology.spi | SPI for adding extensions to the functional api.
com.ibm.streamsx.topology.spi.builder |
com.ibm.streamsx.topology.spi.runtime |
com.ibm.streamsx.topology.spl | Integration between Streams SPL and Java applications.
com.ibm.streamsx.topology.streams | Utilities to create and transform streams.
com.ibm.streamsx.topology.tester | Support for testing of topologies.
com.ibm.streamsx.topology.tester.junit |
com.ibm.streamsx.topology.tester.spl |
com.ibm.streamsx.topology.tuple | Interfaces for tuple types.

Java API to allow creation of streaming applications for IBM Streams by Java & Scala developers.

## Overview

IBM® Streams is an advanced analytic platform that allows user-developed applications to quickly ingest, analyze and correlate information as it arrives from thousands of real-time sources. Streams can handle very high data throughput rates, millions of events or messages per second.
With this API Java & Scala developers can build streaming applications that can be executed using IBM Streams, including the processing being distributed across multiple computing resources (hosts or machines) for scalability.

## Java Application API

The fundamental building block of a Java Streams application is a `stream`, which is continuous sequence of tuples (messages, events, records). The API provides the ability to perform some form of processing or analytics on each tuple as it appears on a stream, resulting in a new stream containing tuples representing the result of the processing against the input tuples.
`Source streams` are streams containing tuples from external systems, for example a source stream may be created from a reading messages from a message queue system, such as MQTT. The purpose of a source stream is to bring the external data into the Streams environment, so that it can be processed, analyzed, correlated with other streams, etc.
Streams are terminated using `sink functions` that typically deliver tuples to external systems, such as real-time dashboards, SMS alerts, databases, HDFS files, etc.

An application is represented by a `Topology` object containing instances of `TStream`. The Java interface `TStream` is a declaration of a stream of tuples, each tuple being an instance of the Java class or interface T. For example `TStream<String>` represents a stream where each tuple will be `String` object, while, for example, `TStream<CallDetailRecord>` represents a stream of `com.sometelco.switch.CallDetailRecord` tuples. Thus, tuples on streams are Java objects, rather than SPL tuples with an attribute based schema.

Streams are created (sourced), transformed or terminated (sinked) generally though functions, where a function is represented by an instance of a Java class with a single method. Frequently, these functions are implemented by anonymous classes specific to an application, though utility methods may encapsulate one or more functions. Here is an example of filtering out all empty strings from stream `s` of type `String`

    TStream<String> s = ...
    TStream<String> filtered = s.filter(new Predicate<String>() {
                 @Override
                 public boolean test(String tuple) {
                     return !tuple.isEmpty();
                 }} );

`s.filter()` is passed an instance of `Predicate`, and sets up a filter where the output stream `filtered` will only contain tuples from the input stream `s` if the method `test()` returns `true`. This implementation of `Predicate`, provided as an anonymous class, returns true if the input tuple (a `String` object) is not empty. At runtime, every `String` tuple that appears on `s` will result in a call to `test`, if the `String` tuple is not empty, then `filtered` will contain the `String` tuple.

### Java 8

With Java 8 lambda expressions or method references can be used to provide the function. Using a lambda expression the above example simplifies to:

    TStream<String> s = ...
    TStream<String> filtered = s.filter(tuple -> !tuple.isEmpty());

Java 8 is supported by IBM Streams 4.0.1 and later.

## Scala Support

Since Scala interoperates with Java classes, applications are implemented in Scala by having the code simply call the Java Application API. A set of implicit conversions are provided to support use of Scala functions in stream transformations. See the Scala documentation under `com.ibm.streamsx.topology/doc/scaladoc/index.html`.

The API is provided as as an SPL toolkit `com.ibm.streamsx.topology` containing the Java API in `lib/com.ibm.streamsx.topology.jar` as well as the SPL operators used to execute the Java functional transformations.

## Features

These features are supported:

Feature | Reference | Since
--- | --- | ---
Tuple types are Java and/or Scala objects. | `TStream` | 1.0
Functional programming, streams are transformed, filtered etc. by functional transformations implemented as Java and/or Scala functions. A Java function is an implementation of interface with a single method, or when using Java 8 a lambda expression or a method reference. | `TStream` | 1.0
Execution within the Java virtual machine, IBM Streams 4.0.1+ Streams standalone or distributed & IBM Bluemix | `StreamsContext` | 1.0
Pipeline topologies. | `Topology` | 1.0
Fan-out, multiple independent functions may be applied to a single stream to produce multiple streams of different or the same type. | `TStream` | 1.0
Fan-in, multiple independent streams of the same type may transformed by a single function to produce a single stream. | `union` | 1.0
Window based aggregation and joins, including partitioning. | `TWindow` | 1.0
Parallel streams (UDP, User Defined Parallelism), including partitioning. | `parallel` | 1.0
Topic based publish-subscribe stream model for cross application communication (Streams dynamic connections). | `publish`, `subscribe` | 1.0
Ability to specify where portions of the topology will execute in distributed mode, including running on resources (hosts) with specified tags. | `Placeable`, `isolate`, `lowLatency` | 1.0
Integration with Apache Kafka and MQTT messaging systems | `Kafka`, `MQTT` | 1.0
Testing of topologies, including those using SPL operators, while running in distributed, standalone or embedded. | `Tester` | 1.0
Integration with SPL streams using SPL attribute schemas. | `SPLStream` | 1.0
Invocation of existing SPL primitive or composite operators. | `SPL` | 1.0

## Samples

A number of sample Java applications are provided under samples. The samples declare a topology and then execute it, the Apache Ant `build.xml` file includes some targets for executing the samples, demonstrating the correct class path.
The javadoc for the samples includes the sample source code (click on the class name of a sample), and is also copied into the SPL toolkit for reference, and is available here: Java Functional Samples

## Declaring a Topology

Java code is used to create a streaming topology, or graph, starting with the `Topology` object and then creating instances of `TStream` by:

- • Invoking methods such as `source` to produce a source stream.
- • Invoking utility static methods that declare a source stream, such as those in `BeaconStreams`.
- • Invoking methods such as `filter` to produce a stream derived from another stream.

Streams are terminated by `sinks`, typically the tuples are sent to an external system by the sink function.

A topology may be arbitrarily complex, including multiple sources and sinks, fan-out on any stream by having multiple functional transformations or fan-in by creating a `union` of streams with identical tuple types.

Creating the `Topology` and its streams as instances of `TStream` just declares how tuples will flow, it is not a runtime representation of the graph. The `TStream` is submitted to a `StreamsContext` in order to execute the graph.

## Java compilation and execution

The API requires these jar files to be in the classpath for compilation and execution:

- • `com.ibm.streamsx.topology/lib/com.ibm.streamsx.topology.jar` - Jar for this API.
- • `$STREAMS_INSTALL/lib/com.ibm.streams.operator.samples.jar` - IBM Streams Java Operator API and samples.

## Testing

The API includes the ability to test topologies, by allow the test program to capture the output tuples of a stream (`TStream`) and validate them. This is described in the `com.ibm.streamsx.topology.tester` package overview.

## Integration with SPL

While the design goal for the API is to not require knowledge of SPL, developers familiar with SPL may also utilize some `SPL primitive and composite operators` from existing toolkits and use `streams that have SPL schemas`.

See Also:
  New to IBM Streams, `Testing support for topologies`