###############
streamsx-runner
###############

********
Overview
********

Submits or builds a Streams application to the Streaming Analytics service.

The application to be submitted can be:

 * A Python application defined through :py:class:`~streamsx.topology.topology.Topology` using the ``--topology`` flag.
 * An SPL application (main composite) using the ``--main-composite`` flag.
 * A Streams application bundle (``sab`` file) using the ``--bundle`` flag.

***************************
Streaming Analytics service
***************************

The Streaming Analytics service is defined by:

 * Service name - ``--service-name`` defaulting to environment variable ``STREAMING_ANALYTICS_SERVICE_NAME``. The service name must exist in the vcap services.
 * Vcap services - Environment variable ``VCAP_SERVICES`` containing JSON representation of the service definitions or a file name containing the service definitions.

**************
Job submission
**************

Job submission occurs unless ``--create-bundle`` is set.

***************
Bundle creation
***************

When ``-create-bundle`` is specified with ``-main-composite`` or ``--topology``
then a Streams application bundle (sab file) is created.

If environment variable `STREAMS_INSTALL` is set the the build is local
otherwise the build occurs in the IBM Cloud using the Streaming Analytics
service.

When ``STREAMS_INSTALL`` is not set then `streamsx-runner` can be executed
with no local Streams install.

When compiling an SPL application (``--main-composite``) then the
path to the application toolkit containing the main composite must
be listed with ``--toolkits``. 

Any other required local toolkits must be listed with with ``--toolkits``.

*****
Usage
*****

.. code-block:: none

    streamsx-runner [-h] [--service-name SERVICE_NAME] | [--create-bundle]
                 (--topology TOPOLOGY | --main-composite MAIN_COMPOSITE | --bundle BUNDLE)
                 [--toolkits TOOLKITS [TOOLKITS ...]] [--job-name JOB_NAME]
                 [--preload] [--trace {error,warn,info,debug,trace}]
                 [--submission-parameters SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...]]
                 [--job-config-overlays file]
    
    Execute a Streams application using a Streaming Analytics service.
    
    optional arguments:
      -h, --help            show this help message and exit
      --service-name SERVICE_NAME
                            Submit to Streaming Analytics service
      --create-bundle       Create a bundle (sab file). No job submission occurs.
      --topology TOPOLOGY   Topology to call
      --main-composite MAIN_COMPOSITE
                            SPL main composite (namespace::composite_name)
      --bundle BUNDLE       Streams application bundle (sab file) to submit to
                            service
    
    Build options:
      Application build options
    
      --toolkits TOOLKITS [TOOLKITS ...]
                            SPL toolkit path containing the main composite and any
                            other required SPL toolkit paths.
    
    Job options:
      Job configuration options
    
      --job-name JOB_NAME   Job name
      --preload             Preload job onto all resources in the instance
      --trace {error,warn,info,debug,trace}
                            Application trace level
      --submission-parameters SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...], -p SUBMISSION_PARAMETERS [SUBMISSION_PARAMETERS ...]
                            Submission parameters as name=value pairs
      --job-config-overlays file
                            Path to file containing job configuration overlays
                            JSON. Overrides any job configuration set by the
                            application.

*****************************************
Submitting to Streaming Analytics service
*****************************************

An application is submitted to a Streaming Analytics service using
``--service-name SERVICE_NAME``. The named service must exist in the
VCAP services definition pointed to by the ``VCAP_SERVICES`` environment
variable.

The application is submitted as source (except ``--bundle``)  and compiled into
a Streams application bundle (``sab`` file) using the build service before
being submitted as a running job to the service instance.

.. seealso:: :ref:`sas-access`

.. _runner-python-apps:

Python applications
===================

To submit a Python application a Python function must be defined
that returns the application (and optionally its configuration)
to be submitted. The fully qualified name of this function is
specified using the ``--topology`` flag.

For example, an application can be submitted as::

    streamsx-runner --service-name Streaming-Analytics-xd \
        --topology com.example.apps.sensor_ingester

The function returns one of:

    * a :py:class:`~streamsx.topology.topology.Topology` instance defining the application
    * a ``tuple`` containing two values, in order:
        * a :py:class:`~streamsx.topology.topology.Topology` instance defining the application
        * job configuration, one of:
            * :py:class:`~streamsx.topology.context.JobConfig` instance
            * ``dict`` corresponding to the configuration object passed into :py:func:`~streamsx.topology.context.submit`

For example the above function might be defined as::

    def _create_sensor_ingester_app():
       topo = Topology('SensorIngesterApp')
       
       # Application declaration omitted
       ...

       return topo

    def sensor_ingester():
        return (_create_sensor_ingester_app(), JobConfig(job_name='SensorIngester'))


Thus when this application is submitted using the `sensor_ingester` function
it is always submitted with the same job name `SensorIngester`.

The function must be accessible from the current Python path
(typically through environment variable ``PYTHONPATH``).

.. _runner-spl-apps:

SPL applications
================

The main composite that defines the application is specified using the ``-main-composite`` flag specifing the fully namespace qualified name.

Any required local SPL toolkits, *including the one containing the main composite*, must be indivdually specified by location to the ``--toolkits`` flag. Any SPL toolkit that is present on the IBM Cloud service need not be included.

For example, an application that uses the Slack toolkit might be submitted as::

    streamsx-runner --service-name Streaming-Analytics-xd \
        --main-composite com.example.alert::SlackAlerter \
        --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack

where ``$HOME/app/alerters`` is the location of the SPL application toolkit containing the ``com.example.alert::SlackAlerter`` main composite.

.. warning::
    The main composite name must be namespace qualified.
    Use of the default namespace for a main composite is not
    recommended as it increases the chance of a name clash with
    another SPL toolkit.

.. _runner-sab:

Streams application bundles
===========================

A Streams application bundle is submitted to a service instance using ``--bundle``.  The argument to ``--bundle`` is a locally accessible file that will be uploaded to the service.

The bundle must have been created on using an IBM Streams install whose architecture and OS version matches the service instance. Currently this is ``x86_64`` and RedHat/CentOS 6 or 7 depending on the service instance.

The ``--toolkits`` flag must not be specified when submitting a bundle.

Job options
===========

Job options, such as ``--job-name``, configure the running job.

For ``--topology`` job options set as arguments to ``streamsx-runner`` override any configuration returned from the function defining the application.

************************************
Creating Streams application bundles
************************************

``--create-bundle`` uses a local IBM Streams install to attempt to mimic the build that would occur with ``-topology`` or ``--main-composite``. Differences between the local environment and the IBM Cloud Streaming Analytics build environment may cause build failures in one and not the other.

This can be used as a mechanism to perform a local test build before using the service, or as a valid mechanism to create bundles for later upload with ``--bundle``.

For example simply changing the ``--service-name name`` to ``--create-bundle`` perfoms a local build of the same application::

    # Submit to an Streaming Analytics service
    streamsx-runner --service-name Streaming-Analytics-xd \
        --main-composite com.example.alert::SlackAlerter \
        --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack

    # Build the same application locally
    streamsx-runner --create-bundle \
        --main-composite com.example.alert::SlackAlerter \
        --toolkits $HOME/app/alerters $HOME/toolkits/com.ibm.streamsx.slack

