API reference

Commands and parameters for the python eCube streaming API.

The eCube streaming API is currently supported in python, and is supported on Linux and Windows platforms.

If your use case involves streaming high channel counts (> 800 channels) and simultaneous high-rate data saving, we suggest using a dedicated recording system to ingest from the eCube, and the streaming API over the network on a separate computer.

eCubeStream prerequisites

Python-based API calls are faciliated using the pyeCubeStream wrapper module, and the servernode-control executable for your target platform (Windows 64-bit, Linux 64-bit), and can be downloaded from the eCube Products page. pyeCubeStream can automatically launch and stop servernode-control as needed in normal operation.

You can find the pyeCubeStream.py wrapper under the tools/ directory of the ServerNode software package. Make sure pyeCubeStream.py is included within your import path. Additionally, pyeCubeStream requires servernode-control to function, and will automatically launch it when required.

When auto-launching, pyeCubeStream will look for servernode-control in one of the 3 following locations:

  1. next to pyeCubeStream.py on module import
  2. parent directory (../) of pyeCubeStream.py
  3. in your python working directory
servernode-${version}
├── (servernode-control)  # <--- (location b)
└── tools/
    └── pyeCubeStream.py
    └── (servernode-control)  # <--- (location a)

${cwd} # working directory in your running python environment, or os.getcwd()
└── (servernode-control)  # <--- (location c)

Usage of pyeCubeStream also require the pyzmq and numpy libraries. The easiest way to set up a python installation with these libraries for scientific computing is to use the Anaconda python distribution. We suggest the 64-bit Python 3.8 package for your target platform.

# import necessary libraries
import pyeCubeStream

Initializing an eCubeStream object

By default, an eCubeStream object can be initialized with no parameters, and it should automatically detect instances of active eCubes or servernode on the network. Optional parameters can be set as below:

Parameters Value Description
sources tuple, list of tuples (optional) Immediately use .add() method to add listed data sources after initialization. See .add() for data source syntax. Defaults to None.
asfloat bool (optional) Convert Headstages, AnalogPanel data sources to float. Defaults to False. See .add() for more details.
snaddress IP string (optional) Destination IP for eCube hardware or for servernode. Defaults to None to connect to the first auto-detected eCube or servernode. Specify this if you have multiple eCubes or servernode instances.
ctladdress IP string (optional) Destination IP for servernode-control. In default localhost setting, if servernode-control has not yet been launched, attempt to launch the process. For non-default, advanced usage, see advanced usage guidelines.
ctlport uint (optional) Defaults to automatic.
dataport uint (optional) Defaults to automatic.
autoshutdown bool (optional) Automatically stop streaming and log out of managed saving sessions when eCubeStream object is destroyed. If servernode-control was originally launched by the eCubeStream object, automatically quit servernode-control process. Defaults to True.
debug bool (optional) Print debugging information. Defaults to False.

Examples:

# (A) Default usage: auto-start servernode-control if not started.
# You can add desired data sources later using .add() and .remove()
ec = pyeCubeStream.eCubeStream()

# (B) Same as option (A), but turn debugging on
ec = pyeCubeStream.eCubeStream(debug=True)

# (C) Initialize eCubeStream and immediately subscribe to
# Headstage 2 and AnalogPanel. This is identical to (A),
# followed by issuing ec.add([('Headstages', 2), ('AnalogPanel',)])
ec = pyeCubeStream.eCubeStream([('Headstages', 2), ('AnalogPanel',)])

(Advanced) Running separate servernode-control and pyeCubeStream

Note: this is an advanced usage note for specific streaming setups.

If your use case calls for running a persistent copy of servernode-control separately from your python data stream, you may launch servernode-control manually and connect to it over the network using pyeCubeStream.

# Advanced usage: servernode-control running on separate machine
ec = pyeCubeStream.eCubeStream(ctladdress='192.168.100.1')

Subscribing to data sources

.add(): Adding data sources

Adds data sources to the current subscription. All data sources are represented as python tuples following the format below. Multiple sources can be added at a time by combining the data sources into lists.

Parameters Value Description
sources tuple, list of tuple See data source table
asfloat bool Convert data source to 32-bit float analog value with appropriate gain. See Data content subpacket table for more details. Note that asfloat conversion applied to any Headstages applies to all Headstages sources. Defaults to False.

Data source format for .add()

Data source Value Description
Headstages ('Headstages',#) Literal string Headstages and the 1-indexed headstage number (1 - 10) as a tuple: e.g. ('Headstages', 2) for all channels on headstage plugged into H2
Partial headstage (channel range) ('Headstages',hs#,(ch#,ch#)) Tuple containing literal string Headstages, headstage number, and nested tuple containing 1-indexed channel range (from, to) inclusive: e.g. ('Headstages', 8, (257, 320)) to stream channels 257-320 on headstage 8
Analog Panel ('AnalogPanel',) Literal string AnalogPanel followed by , to format as a tuple.
Partial Analog Panel (channel range) ('AnalogPanel',(ch#,ch#)) Literal string AnalogPanel as a tuple, and nested tuple containing 1-indexed channel range (from, to) inclusive: e.g. ('AnalogPanel', (1, 10)) for the first 10 AnalogPanel channels.
Digital Panel (as single 64-bit integer) ('DigitalPanel',) Literal string DigitalPanel followed by , to format as a tuple. Usually used to communicate digital states, or full binary words.
Digital Panel (each bit as channels) ('DigitalPanelAsChans',) Literal string DigitalPanelAsChans followed by , to format as a tuple. Usually used to communicate single pin or switch toggles. Only one type of Digital Panel format is supported in each stream.
Digital Panel (channel/bit range) ('DigitalPanelAsChans',(ch#,ch#)) Literal string DigitalPanelAsChans as a tuple, and nested tuple containing 1-indexed channel range (from, to) inclusive: e.g. ('DigitalPanelAsChans', (1, 10)) for the first 10 DigitalPanel channels. Only one type of Digital Panel format is supported in each stream. 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream()
ec.add( ('Headstages', 2) ) # add all channels in headstage 2
ec.add( [('Headstages', 8, (65,96)),
         ('Headstages', 8, (256,384))] ) # add headstage 8 channels 65-96, 256-384
ec.add( ('AnalogPanel', (6, 6)), asfloat=True ) # add AnalogPanel channel 6, and convert to 32-bit `float` volts
ec.add( ('DigitalPanelAsChans', (12, 44)) ) # add DigitalPanel as per-bit channels

.remove(): Removing data sources

Removes data sources to the current subscription. All data sources are represented as python tuples following the format below. Multiple sources can be added at a time by combining the data sources into lists.

Removing data sources must apply to the entire data source. Partial / channel range removals are not supported.

Data source Value Description
Headstages ('Headstages',#) Literal string Headstages and the 1-indexed headstage number (1 - 10) as a tuple: e.g. ('Headstages', 2). Channel range not supported for removal.
Analog Panel ('AnalogPanel',) Literal string AnalogPanel followed by , to format as a tuple.
Digital Panel (as single 64-bit integer) ('DigitalPanel',) Literal string DigitalPanel followed by , to format as a tuple.
Digital Panel (each bit as channels) ('DigitalPanelAsChans',) Literal string DigitalPanelAsChans followed by , to format as a tuple. 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream()
ec.remove(('Headstages', 2) # remove headstage 2 from stream
ec.remove(('AnalogPanel',)) # remove AnalogPanel from stream
ec.remove(('DigitalPanelAsChans',)) # remove DigitalPanel from stream

.listadded(): List added data sources

Returns all data sources currently in subscription, formatted as a tuple containing:

(Headstages, Analog Panel, Digital Panel)

Data source Return format
Headstages tuple( numpy.ndarray[headstage #], numpy.ndarray[channel # within headstage] )
Analog Panel list( AnalogPanel # )
Digital Panel list( DigitalPanel # ) 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream([('Headstages', 2),
    ('Headstages', 8, (65,96)),
    ('AnalogPanel', (6, 6)),
    ('DigitalPanelAsChans', (17, 24))], debug=True)
ec.listadded()

# Output:
# Headstage channels added:
# (array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
#        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
#        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 8, 8,
#        8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8,
#        8, 8, 8, 8, 8, 8, 8, 8], dtype=uint32), array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
#        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
#        35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
#        52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
#        69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85,
#        86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96], dtype=uint32))
# AnalogPanel channels added:
# [6]
# DigitalPanel channels added:
# [17 18 19 20 21 22 23 24]

.listavailable(): List all data sources available for adding

Returns all data sources currently available for subscribing, from the upstream eCube or ServerNode. It is formatted as a tuple containing:

(Headstages, Analog Panel, Digital Panel)

Data source Return format
Headstages numpy.ndarray([ headstages 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
Analog Panel int( # of channels )
Digital Panel int( # of channels ) 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream(debug=True)
print(ec.listavailable())

# Output:
# Channels available:
#   HS 2: 64
#   HS 8: 640
#   HS 10: 640
#   AnalogPanel
#   DigitalPanel
# (array([  0,  64,   0,   0,   0,   0,   0, 640,   0, 640], dtype=int32), 32, 64)

Starting / stopping data streams

.start(): Start streaming of subscribed data sources

Starts the concurrent streaming of subscribed data sources, and makes available obtaining streaming data via .get(). This action locks in the data sources in the subscription, and .add() or .remove() commands are no longer valid until stream is stopped.

Returns True on success.

.get(): Obtain data packet

Obtains the latest available data packet from the stream. A data packet is composed of a tuple formatted as:

tuple( timestamp, data source, data content ), where:

Data packet listing

Data packet component Format
timestamp uint64 counter for nanoseconds (valid for relative comparison within one datastream)
data source one of Headstages, AnalogPanel, DigitalPanel, or DigitalPanelAsChans
data content See data content subpacket

Data content subpacket

For data source… Format
Headstages numpy.ndarray([ 728 int16 samples × channels ]) with 1.907e-7 volts / bit
Headstages with asfloat=True numpy.ndarray([ 728 single samples × channels ] with units of volts)
AnalogPanel numpy.ndarray([ 22 int16 samples × channels ]) with 3.052e-3 volts / bit
AnalogPanel with asfloat=True numpy.ndarray([ 22 single samples × channels ]) with units of volts
DigitalPanel numpy.ndarray([ 728 uint64 samples ])
DigitalPanelAsChans numpy.ndarray([ 728 uint8 samples (bool content) × channels ]) 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream([('Headstages', 2),
    ('Headstages', 8, (65,96)),
    ('DigitalPanelAsChans', (12, 44))])

ec.start()
while True:
	data = ec.get()
	print("Source \'{}\'' received at {} ns containing [{}]: {}".format(
		data[1], data[0], data[2].shape, data[2]))

# Output
# Source 'Headstages' received at 22656674875 ns containing [(728, 96)]: [[ 24328 -13556 -13092 ... -32768 -32768 -32768]
#  [ 24532 -13500 -13124 ... -32768 -32768 -32768]
#  [ 24756 -13468 -13144 ... -32768 -32768 -32768]
#  ...
#  [ 12020 -10984  -8152 ... -32768 -32768 -32768]
#  [ 12332 -11192  -8380 ... -32768 -32768 -32768]
#  [ 12604 -11400  -8592 ... -32768 -32768 -32768]]
# Source 'Headstages' received at 22685794875 ns containing [(728, 96)]: [[ 12876 -11592  -8820 ... -32768 -32768 -32768]
#  [ 13188 -11764  -9020 ... -32768 -32768 -32768]
#  [ 13452 -11900  -9180 ... -32768 -32768 -32768]
#  ...
#  [-20248  18148  16624 ... -32768 -32768 -32768]
#  [-20240  18196  16660 ... -32768 -32768 -32768]
#  [-20232  18252  16732 ... -32768 -32768 -32768]]
# Source 'DigitalPanelAsChans' received at 22698805087 ns containing [(728, 33)]: [[1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  ...
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]]
# Source 'DigitalPanelAsChans' received at 22727925087 ns containing [(728, 33)]: [[1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  ...
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]
#  [1 1 1 ... 1 1 1]]

.stop(): Stop streaming of subscribed data sources

Stops the concurrent streaming of subscribed data sources. This action re-enables .add() or .remove() subscription alterations. Data can no longer be obtained via the .get() command.

Returns True on success.

Starting / stopping remote data recording

If the upstream data source is a running version of servernode, the API is capable of triggering the start and stop of remote data recordings on the host running servernode software, based on the currently subscribed data sources within the API.

These functions are not available if eCube hardware is used directly as the upstream data source.

Please note that sample-aligned synchronization and ServerNode sample timestamps are only valid and relatable within concurrently created recording sessions.

For example, to record synchronized Headstage + DigitalPanel sources, add both sources before calling .remotesave(): this constitutes one session (e.g., Session A) and all data files created by this process are synchronized.

Additional recording sessions (Session B, C…) can be started by changing added sources, and calling .remotesave() again, but samples are not synchronized between separate sessions (session A and session B samples and timestamps are not related).

If you have one common source you would like to use across multiple recording sessions (e.g., DigitalPanel channels are used to delineate experimental epochs for both session A and session B), be sure to retain the same source in every single recording session where it is needed.

.remotesave(): Start a remote data recording session

Start a remote data recording on the host running servernode software, based on the currently subscribed set of channels. The recordings will be saved into a newly created directory matching the method parameter sessionname.

The newly created data recordings will be nested within the servernode target recording directory. To change the target recording path on the remote host, please see ServerNode Pre-configuring options.

Remote saving operations are independent from local streaming. Commands such as .start() / .stop() / .get() do not affect remote saving.

Parameters Value Description
sessionname string Create a new sessionname directory in the remote host recording path, and record data 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream([('Headstages', 2),
    ('Headstages', 8, (65,96)),
    ('AnalogPanel', (65, 96))])
ec.remotesave('Session-20210101-exprun3')

.listremotesessions(): List all remote data recording sessions

Lists all currently created remote sessions running on the host running servernode software. You can stop recording using .remotestopsave(). Please note that recording sessions created with Open Ephys software will also be visible in this list, and issuing .remotesave() or .remotestopsave() will override Open Ephys-based recordings.

import pyeCubeStream

ec = pyeCubeStream.eCubeStream()
ec.listremotesessions()

# Output
# ['Session-20210101-exprun1', 'Session-20210101-exprun2', 'Session-20210101-exprun3']

.remotestopsave(): Stop a remote data recording session

Stop an active remote data recording on the host running servernode software.

Remote saving operations are independent from local streaming. Commands such as .start() / .stop() / .get() do not affect remote saving.

Parameters Value Description
sessionname string Stop remote host recording of data matching sessionname 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream([('Headstages', 2),
    ('Headstages', 8, (65,96)),
    ('AnalogPanel', (65, 96))])
ec.remotestopsave('Session-20210101-exprun3')

Hardware utilities

.resetheadstage(): Reset headstage

Sends a reset command to an eCube-connected headstage, and trigger a headstage reboot.

Parameters Value Description
Headstage # int 1-indexed headstage number (1 - 10) 
import pyeCubeStream

ec = pyeCubeStream.eCubeStream()
ec.resetheadstage(10) # reset headstage 10
Last modified April 16, 2021