Package org.marketcetera.modules.async

Class SimpleAsyncProcessor

  • All Implemented Interfaces:
    DynamicMBean, DataEmitter, DataReceiver
    public class SimpleAsyncProcessor
extends Module
implements DataEmitter, DataReceiver, DynamicMBean
    A processor module that can be inserted between any two modules within a data flow to decouple the delivery of data from the emitter module and processing of that data in the receiver module.

    For each data flow this module is participating in, the module adds every received data item into a queue. Hence the emitter module from which this module is receiving data is not blocked when delivering data. The module creates a separate thread for the data flow that removes the data items from this queue and delivers it to the next receiver module in the data flow.

    Do note that the if the module that is receiving data emitted by this module is not able to keep up with the module that is emittin data into this module, the data items will keep on accumulating in the queue. If this condition continues on for a while, this may lead to JVM running out of memory. When using this module, be sure to size the heap of the process to ensure that the process does not run out of memory, when there's a disparity in the data emit rate and data receive rate of the modules before and after this module in the data flow.

    The module exposes an DynamicMBean interface to monitor the current queue sizes for every flow. The MBean offers list of attributes, one per data flow, that it is participating in. Each attribute has name of the form "Flowdata_flow_id" where data_flow_id is the flowID of the data flow that the attribute represents. The value of the attribute is the current size of the queue for that data flow. The attribute value can be monitored to observe if the module receiving data from this module is able to keep up with the module emitting data into this module in the data flow represented by the attribute.

    Note that when the data flow is canceled, the module does not wait for the all the data in the queue to be delivered, it interrupts the delivery thread right away and allows the flow to be canceled.

    Note that this module is not designed such that the same instance can be used more than once in a single data flow. If you need to use this module more than once in the same data flow make sure that you use different instances of this module in the flow such that the same instance doesn't appear more than once in the same data flow.

    This module is meant to be used as an intermediate module between two modules in a data flow. Although this module can be used as the first or the last module in a data flow, the features provided by this module offer little merit in such a scenario.

    Usage:

    The following strategy agent command demonstrates how this module can be inserted between two modules to decouple the data flow between them. 

     # Start the bogus feed module
 startModule;metc:mdata:bogus:single
 # Create data flow
 createDataFlow;metc:mdata:bogus;single:symbols=AAPL:content=latest_tick,top_of_book^metc:async:simple:myinstance
    The command above will create a data flow where the market data events from the bogus feed are fed into the simple async module. The async module adds all the received market data events to the queue and has a separate thread that removes them from the queue and delivers them to the sink module. Since the market data events are always added to the queue, market data delivery is very fast and the market data delivery thread is not blocked while the sink module processes the event (for example, logs it).

    Module Features

    Describes the module attributes
    CapabilitiesData Emitter, Data Reciever
    DataFlow Request ParametersNone.
    Stops data flowsNo.
    Start OperationInitializes the thread pool for emitting data.
    Stop OperationShuts down the thread pool.
    Management Interfacedynamic: see above
    FactorySimpleAsyncProcessorFactory
    Since:
    2.0.0
    Version:
    $Id$
    Author:
    anshul@marketcetera.com