streamsx.ec — streamsx 1.15.9 documentation

# streamsx.ec

Access to the IBM Streams execution context.

## Overview

This module (streamsx.ec) provides access to the execution context when Python code is running in a Streams application.

A Streams application runs distributed or standalone.

### Distributed

Distributed is used when an application is submitted to the Streaming Analytics service on IBM Cloud or a IBM Streams distributed instance.

With distributed a running application is a job that contains one or more processing elements (PEs). A PE corresponds to a Linux operating system process. The PEs in a job may be distributed across the resources (hosts) in the Streams instance.

### Standalone

Standalone is a mode where the complete application is run as a single PE (process) outside of a Streams instance.

Standalone is typically used for ad-hoc testing of an application.

## Application log and trace

IBM Streams provides application trace and log services.

### Application log

The Streams application log service is for application logging, where logging is defined as the recording of serviceability information pertaining to application or operator events. The purpose of logging is to provide an administrator with enough information to do problem determination for items they can potentially control. In general, very few events are logged in the normal running scenario of an application or operator. Events pertinent to the failure or partial failure of application runtime scenarios should be logged.

When running in distributed or standalone the com.ibm.streams.log logger has a handler that records messages to the Streams application log service. The level of the logger and its handler are set to the configured application log level at PE start up.

This logger and handler discard any message with level below INFO (20).

Python application code can log a message suitable for an administrator by using the com.ibm.streams.log logger or a child logger that has `logger.propagate` evaulating to `True`. Example of logging a file exception:

    try:
        import numpy
    except ImportError as e:
        logging.getLogger('com.ibm.streams.log').error(e)
        raise

Application code must not modify the com.ibm.streams.log logger, if additional handlers or different levels are required a child logger should be used.

### Application trace

The Streams application trace service is for application tracing, where tracing is defined as the recording of application or operator internal events and data. The purpose of tracing is to allow application or operator developers to debug their applications or operators.

When running in distributed or standalone the root logger has a handler that records messages to the Streams application trace service. The level of the logger and its handler are set to the configured application trace level at PE start up.

Python application code can trace a message using the root logger or a child logger that has `logger.propagate` evaulating to `True`. Example of logging a trace message:

    trace = logging.getLogger(__name__)

    ...

        trace.info("Threshold set to %f", val)

Any existing logging performed by modules will automatically become Streams trace messages if the application is using the logging package.

Application code must not modify the root logger, if additional handlers or different levels are required a child logger should be used.

## Execution Context

This module (streamsx.ec) provides access to the execution context when Python code is running in a Streams application.

Access is only supported when running:

  - Streams 4.2 or later

This module may be used by Python functions or classes used in a Topology or decorated SPL operators.

Most functionality is only available when a Python class is being invoked in a Streams application.

Changed in version 1.9: Support for Python 2.7

## Module contents

Functions

`channel` | Return the parallel region global channel number obj is executing in.
`domain_id` | Return the instance identifier.
`get_application_configuration` | Get a named application configuration.
`get_application_directory` | Get the application directory.
`get_submission_time_value` | Gets the value of a submission time parameter.
`instance_id` | Return the instance identifier.
`is_active` | Tests if code is active within a IBM Streams execution context.
`is_standalone` | Is the execution context standalone.
`job_id` | Return the job identifier.
`local_channel` | Return the parallel region local channel number obj is executing in.
`local_max_channels` | Return the local maximum number of channels for the parallel region obj is executing in.
`max_channels` | Return the global maximum number of channels for the parallel region obj is executing in.
`pe_id` | Return the PE identifier.
`shutdown` | Return the processing element (PE) shutdown event.

Classes

`CustomMetric` | Create a custom metric.
`MetricKind` | Enumeration for the kind of a metric.

`streamsx.ec.` `is_active`()

  Tests if code is active within a IBM Streams execution context.

  Returns a true value when called from within a IBM Streams distributed job or standalone execution.

  Can be used to only run code required at runtime, such as importing a module that is only needed at runtime and not topology declaration time.

  Returns

    True if running in a IBM Streams context false otherwise.

  Return type

    bool

  New in version 1.11.

`streamsx.ec.` `shutdown`()

  Return the processing element (PE) shutdown event.

  The event is set when the PE is being shutdown. Can be used in source iterators that need to block by sleeping:

      # Sleep for 60 seconds unless the PE is being shutdown
      if streamsx.ec.shutdown.wait(60.0):
          return None

  Code must not call `set()` on the returned event.

  Returns

    Event object representing PE shutdown.

  Return type

    threading.Event

  New in version 1.11.

`streamsx.ec.` `domain_id`()

  Return the instance identifier.

`streamsx.ec.` `instance_id`()

  Return the instance identifier.

`streamsx.ec.` `job_id`()

  Return the job identifier.

`streamsx.ec.` `pe_id`()

  Return the PE identifier.

`streamsx.ec.` `is_standalone`()

  Is the execution context standalone.

  Returns

    True if the execution context is standalone, False if it is distributed.

  Return type

    boolean

`streamsx.ec.` `get_application_directory`()

  Get the application directory.

  Returns

    The application directory.

  Return type

    str

  New in version 1.7.

`streamsx.ec.` `get_application_configuration`(name)

  Get a named application configuration.

  An application configuration is a named set of securely stored properties where each key and its value in the property set is a string.

  An application configuration object is used to store information that IBM Streams applications require, such as:

  - Database connection data

  - Credentials that your applications need to use to access external systems

  - Other data, such as the port numbers or URLs of external systems

  Parameters

    name (str) – Name of the application configuration.

  Returns

    Dictionary containing the property names and values for the application configuration.

  Return type

    dict

  Raises

    ValueError – Application configuration does not exist.

`streamsx.ec.` `get_submission_time_value`(name, type_=None)

  Gets the value of a submission time parameter.

  Note

  Submission parameters must be created with `streamsx.topology.topology.Topology.create_submission_parameter()` at topology declaration time before they can be accessed with this function.

  Note

  Submission time parameters, which are defined within other toolkits that are used by this topology or created with `streamsx.spl.op.Expression` in this topology are not accessible with this function.

  Parameters

    - name (str) – The name of the submission time parameter

    - type_ – Type of parameter value. Supported values are `str`, `int`, `float` and `bool`. `None` assumes that the type of the parameter is `str` and needs no conversion. The type must be the same as used in `streamsx.topology.topology.Topology.create_submission_parameter()` or as deduced from the parameters default value.

  Returns

    The value of the submission time parameter. The return type is either `str` or the given `type_`.

  Raises

    ValueError – The submission time parameter with the given name does not exist.

  New in version 1.15.

`streamsx.ec.` `channel`(obj)

  Return the parallel region global channel number obj is executing in.

  The channel number is in the range of 0 to `max_channel(obj)`.

  When the parallel region is not nested this is the same value as `local_channel(obj)`.

  If the parallel region is nested the value will be between zero and `(width*N - 1)` where N is the number of times the parallel region has been replicated due to nesting.

  Parameters

    obj – Instance of a class executing within Streams.

  Returns

    Parallel region global channel number or -1 if not located in a parallel region.

  Return type

    int

`streamsx.ec.` `local_channel`(obj)

  Return the parallel region local channel number obj is executing in.

  The channel number is in the range of zero to `local_max_channel(obj)`.

  Parameters

    obj – Instance of a class executing within Streams.

  Returns

    Parallel region local channel number or -1 if not located in a parallel region.

  Return type

    int

`streamsx.ec.` `max_channels`(obj)

  Return the global maximum number of channels for the parallel region obj is executing in.

  When the parallel region is not nested this is the same value as `local_max_channels(obj)`.

  If the parallel region is nested the value will be `(width*N)` where N is the number of times the parallel region has been replicated due to nesting.

  Parameters

    obj – Instance of a class executing within Streams.

  Returns

    Parallel region global maximum number of channels or 0 if not located in a parallel region.

  Return type

    int

`streamsx.ec.` `local_max_channels`(obj)

  Return the local maximum number of channels for the parallel region obj is executing in.

  The maximum number of channels corresponds to the width of the region.

  Parameters

    obj – Instance of a class executing within Streams.

  Returns

    Parallel region local maximum number of channels or 0 if not located in a parallel region.

  Return type

    int

class `streamsx.ec.` `MetricKind`

  Bases: `enum.Enum`

  Enumeration for the kind of a metric.

  The kind of the metric only indicates the behavior of value, it does not impose any semantics on the value. The kind is typically used by tooling applications.

  `Counter` = 1

    A counter metric observes a value that represents a count of an occurrence.

  `Gauge` = 0

    A gauge metric observes a value that is continuously variable with time.

  `Time` = 2

    A time metric represents a point in time or duration. The recommended unit of time is milliseconds, using the standard epoch of 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 to represent a point in time.

class `streamsx.ec.` `CustomMetric`(obj, name, description=None, kind=<MetricKind.Counter: 1>, initialValue=0)

  Bases: `object`

  Create a custom metric.

  A custom metric holds a 64 bit signed integer value that represents a Counter, Gauge or Time metric.

  Custom metrics are exposed through the IBM Streams monitoring APIs.

  Metric `name` is unique within the execution context of the callable `obj`. Attempts to create multiple metrics with the same name but different kinds will raise an exception. Multiple creations of a metric of the same name and kind all refer to the same metric, the first creation is the only one that will set the initial value.

  The metric’s value is assigned through the `value` property and can be modified through `+=` and `-=`. `CustomMetric` can also be converted to an `int`.

  Parameters

    - obj – Instance of a class executing within Streams.

    - name (str) – Name of the custom metric.

    - kind (MetricKind) – Kind of the metric.

    - description (str) – Description of the metric.

    - initialValue – Initial value of the metric.

  Examples:

  Simple example used as an instance to `Stream.filter`:

      class MyF:
          def __init__(self, substring):
              self.substring = substring
              pass

          def __call__(self, tuple):
              if self.substring in str(tuple)
                  self.my_metric += 1
              return True

          # Create the metric when the it is running
          # in the Streams execution context
          def __enter__(self):
              self.my_metric = ec.CustomMetric(self, "count_" + self.substring)

          # must supply __exit__ if using __enter__
          def __exit__(self, exc_type, exc_val, exc_tb):
              pass

          def __getstate__(self):
              # Remove metric from saved state.
              state = self.__dict__.copy()
              if 'my_metric' in state:
                  del state['my_metric']
              return state

          def __setstate__(self, state):
              self.__dict__.update(state)

  property `value`

    Current value of the metric.