Skip to content

Manager Interface

StreamMine3G has basically just two interfaces: The operator interface, for users to develop operators in order to perform any kind of data mining, and on the other side, the manager interface for e.g. the cloud provider to develop a sophisticated manager component that decides on which node within a StreamMine3G cluster those operators should be running.

The manager interface allows to deploy operator slices, removing or even migrating them to other StreamMine3G nodes to better balance load or, to contract/expand a StreamMine3G cluster dynamically (elastic scaling). In addition to taking actions such as deploying an operator slice or moving it to a new node, the manager is also constantly provided with node as well as slice statistics which include CPU, Memory as well as network utilization, the event throughput (in and out) as well as state size for each of the slices running within a StreamMine3G cluster. Using this information, sophisticated managers can be developed, constantly monitoring the StreamMine3G cluster and taking appropriate actions to minimize costs when running in cloud environments such as Amazon EC2.

With the manager, the following actions (through the CloudControl object) can be taken:

To build operator topologies: * registering an operator * removing an operator

  • wiring operators
  • un-wiring operators

To deploy & launch operator slices on a StreamMine3G cluster * deploying a slice * removing a slice * launching a slice * tearing down a slice

To scale elastically a StreamMine3G cluster * synchronizing a slice * migrating a slice

  • shutting down a StreamMine3G node

StreamMine3G is a fully dynamic and elastic event processing platform, hence, new operators can be deployed and operators topologies modified while the system is running and processing events without having to stop, or to shutdown the platform!

Asynchronous API

The manager interface exposes an asynchronous API, i.e., each action such as deploying a slice will be acknowledged with a call back of the onActionComplete() method. The method provides information such as if the action completed successfully or not as well as what action had been taken e.g. what slice identified through its sliceUId has been deployed successfully.

StreamMine3G Cluster Monitoring

The manager component is constantly provided with performance probe of the StreamMine3G cluster. Every 500ms, each StreamMine3G node reports through heart beat messages system health information such as CPU, Memory as well as network utilization, the event throughput (in and out) as well as state size for all slices running within a StreamMine3G cluster. This information is provided through upcalls of the onPerformanceProbe() method call backs.

Adapter Interface

In a similar fashion as for operators, the manager component is equipped with an adapter Interface to connect and exchange information with external entities via TCP.

State Persistence

In order to prevent the manager being a single point of failure, the state/internal data structures of the manager are periodically persisted into zookeeper. However, user defined data structures within the manager component must be persisted as well. Therefore it is recommend to implement the serializeState() and deserializeState() methods appropriately to persist any user defined class members of the manager as well.