Data types used for analysis

Transmission

Inherits from BaseTransmission

class mesmerize.Transmission(df: pandas.core.frame.DataFrame, history_trace: mesmerize.analysis.data_types.HistoryTrace, proj_path: str = None, last_output: str = None, last_unit: str = None, ROI_DEFS: list = None, STIM_DEFS: list = None, CUSTOM_COLUMNS: list = None, plot_state: dict = None)[source]

The transmission object used throughout the flowchart

__init__(df: pandas.core.frame.DataFrame, history_trace: mesmerize.analysis.data_types.HistoryTrace, proj_path: str = None, last_output: str = None, last_unit: str = None, ROI_DEFS: list = None, STIM_DEFS: list = None, CUSTOM_COLUMNS: list = None, plot_state: dict = None)

Base class for common Transmission functions

Parameters:
  • df – Transmission dataframe
  • history_trace – HistoryTrace object, keeps track of the nodes & node parameters the transmission has been processed through
  • proj_path (str) – Project path, necessary for the datapoint tracer
  • last_output (str) – Last data column that was appended via a node’s operation
  • last_unit (str) – Current units of the data. Refers to the units of column in last_output
  • plot_state – State of a plot, such as data and label columns. Used when saving interactive plots.
Variables:
  • df – Dataframe instance belonging to a Transmission instance
  • history_trace
    class:HistoryTrace instance
  • proj_path – project path
  • last_output – Name of last data column that was the output of a node
  • last_unit – The data units corresponding to last_output
static empty_df(transmission, addCols: list = None) → pandas.core.frame.DataFrame

Just a helper method to return an empty DataFrame with the same columns

Parameters:
  • transmission – Transmission object that forms the basis
  • addCols – list of columns to add
Returns:

The input transmission with an empty dataframe containing the same columns and any additional columns that were passed

classmethod from_hdf5(path: str)

Create Transmission from an hdf5 file. See HdfTools for information on the file structure.

Parameters:path – file path, usually ends in .trn
Returns:
classmethod from_pickle(path)

Load Transmission from a pickle.

Parameters:path – file path, usually ends in .trn
get_proj_path() → str

Get the project root dir associated to this Transmission.

Returns:Root directory of the project
set_proj_path(path: str)

Set the project root dir for this transmission.

Used for appending relative paths (stored in the project DataFrame) to the various project files. For example, this is used by the Datapoint Tracer to find max and std projections of image sequeences.

Parameters:path – Root directory of the project
to_dict() → dict

Package Transmission as a dict, useful for saving to hdf5 or pickle

to_hdf5(path: str)

Save as an hdf5 file. Uses pytables to save the DataFrame an JSON to save the HistoryTrace. See HdfTools for information on the file structure.

Parameters:path – file path, usually ends in .trn
to_hickle(path)

Save as an hdf5 file using hickle (not recommended, use to_hdf5)

Parameters:path – file path
to_pickle(path: str)

Svae Transmission as a pickle

Parameters:path – file path, usually ends in .trn
classmethod from_proj(proj_path: str, dataframe: pandas.core.frame.DataFrame, sub_dataframe_name: str = 'root', dataframe_filter_history: dict = None)[source]
Parameters:
  • proj_path – root directory of the project
  • dataframe – Chosen Child DataFrame from the Mesmerize Project
  • sub_dataframe_name – Name of the sub DataFrame to load
  • dataframe_filter_history – Filter history of the child dataframe
static _load_files(proj_path: str, row: pandas.core.series.Series) → pandas.core.series.Series[source]

Loads npz of curve data and pickle files containing metadata using the paths specified in each row of the chosen sub-dataframe of the project

classmethod merge(transmissions: list)[source]

Merges a list of Transmissions into one transmission. A single DataFrame is created by simple concatenation. HistoryTrace objects are also merged using HistoryTrace.merge.

Parameters:transmissions – A list containing Transmission objects to merge
Returns:Merged transmission

BaseTransmission

class mesmerize.analysis.data_types.BaseTransmission(df: pandas.core.frame.DataFrame, history_trace: mesmerize.analysis.data_types.HistoryTrace, proj_path: str = None, last_output: str = None, last_unit: str = None, ROI_DEFS: list = None, STIM_DEFS: list = None, CUSTOM_COLUMNS: list = None, plot_state: dict = None)[source]
__init__(df: pandas.core.frame.DataFrame, history_trace: mesmerize.analysis.data_types.HistoryTrace, proj_path: str = None, last_output: str = None, last_unit: str = None, ROI_DEFS: list = None, STIM_DEFS: list = None, CUSTOM_COLUMNS: list = None, plot_state: dict = None)[source]

Base class for common Transmission functions

Parameters:
  • df – Transmission dataframe
  • history_trace – HistoryTrace object, keeps track of the nodes & node parameters the transmission has been processed through
  • proj_path (str) – Project path, necessary for the datapoint tracer
  • last_output (str) – Last data column that was appended via a node’s operation
  • last_unit (str) – Current units of the data. Refers to the units of column in last_output
  • plot_state – State of a plot, such as data and label columns. Used when saving interactive plots.
Variables:
  • df – Dataframe instance belonging to a Transmission instance
  • history_trace
    class:HistoryTrace instance
  • proj_path – project path
  • last_output – Name of last data column that was the output of a node
  • last_unit – The data units corresponding to last_output
to_dict() → dict[source]

Package Transmission as a dict, useful for saving to hdf5 or pickle

to_hickle(path)[source]

Save as an hdf5 file using hickle (not recommended, use to_hdf5)

Parameters:path – file path
to_hdf5(path: str)[source]

Save as an hdf5 file. Uses pytables to save the DataFrame an JSON to save the HistoryTrace. See HdfTools for information on the file structure.

Parameters:path – file path, usually ends in .trn
classmethod from_hdf5(path: str)[source]

Create Transmission from an hdf5 file. See HdfTools for information on the file structure.

Parameters:path – file path, usually ends in .trn
Returns:
classmethod from_hickle(path)[source]

Create Transmission from hdf5 file saved using hickle.

Parameters:path – file path, usually ends in .trn
Returns:
classmethod from_pickle(path)[source]

Load Transmission from a pickle.

Parameters:path – file path, usually ends in .trn
to_pickle(path: str)[source]

Svae Transmission as a pickle

Parameters:path – file path, usually ends in .trn
static empty_df(transmission, addCols: list = None) → pandas.core.frame.DataFrame[source]

Just a helper method to return an empty DataFrame with the same columns

Parameters:
  • transmission – Transmission object that forms the basis
  • addCols – list of columns to add
Returns:

The input transmission with an empty dataframe containing the same columns and any additional columns that were passed

get_proj_path() → str[source]

Get the project root dir associated to this Transmission.

Returns:Root directory of the project
set_proj_path(path: str)[source]

Set the project root dir for this transmission.

Used for appending relative paths (stored in the project DataFrame) to the various project files. For example, this is used by the Datapoint Tracer to find max and std projections of image sequeences.

Parameters:path – Root directory of the project
set_proj_config()[source]

Sets some project config related attributes.

Helper functions

mesmerize.analysis.get_sampling_rate(transmission: mesmerize.analysis.data_types.Transmission, tolerance: Optional[float] = 0.1) → float[source]

Returns the mean sampling rate of all data in a Transmission if it is within the specified tolerance. Otherwise throws an exception.

Parameters:
  • transmission (Transmission) – Transmission object of the data from which sampling rate is obtained.
  • tolerance (float) – Maximum tolerance (in Hertz) of sampling rate variation between different samples
Returns:

The mean sampling rate of all data in the Transmission

Return type:

float

mesmerize.analysis.get_array_size(transmission: mesmerize.analysis.data_types.Transmission, data_column: str) → int[source]

Returns the size of the 1D arrays in the specified data column. Throws an exception if they do not match

Parameters:
  • transmission (Transmission) – Desired Transmission
  • data_column (str) – Data column of the Transmission from which to retrieve the size
Returns:

Size of the 1D arrays of the specified data column

Return type:

int

HistoryTrace

class mesmerize.analysis.data_types.HistoryTrace(history: Dict[Union[uuid.UUID, str], List[Dict]] = None, data_blocks: List[Union[uuid.UUID, str]] = None)[source]

Structure of a history trace:

A dict with keys that are the block_ids. Each dict value is a list of operation_dicts. Each operation_dict has a single key which is the name of the operation and the value of that key is the operation parameters.

{block_id_1: [

{operation_1:

{

param_1: a,

param_2: b,

param_n, z

}

},

{operation_2:

{

param_1: a,

param_n, z

}

},

{operation_n:

{

param_n: x

}

}

]

block_id_2: <list of operation dicts>,

block_id_n: <list of operation dicts>

}

The main dict illustrated above should never be worked with directly.

You must use the helper methods of this class to query or add information

Variables:
  • _history – The dict of the actual data, as illustrated above. Should not be accessed directly. Use the history property or call get_all_data_blocks_history().
  • _data_blocks – List of all data blocks. Should not be called directly, use the property data_blocks instead.
__init__(history: Dict[Union[uuid.UUID, str], List[Dict]] = None, data_blocks: List[Union[uuid.UUID, str]] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

__weakref__

list of weak references to the object (if defined)

_add_data_block(data_block_id: uuid.UUID)[source]

Adds new datablock UUID to the list of datablocks in this instance. Throws exception if UUID already exists.

static _to_uuid(u: Union[str, uuid.UUID]) → uuid.UUID[source]

If argument ‘u’ is type <str> that can be formatted as a UUID, return it as UUID type. If argument ‘u’ is a UUID, just return it.

add_operation(data_block_id: Union[uuid.UUID, str], operation: str, parameters: dict)[source]

Add a single operation, that is usually performed by a node, to the history trace. Added to all or specific datablock(s), depending on which datablock(s) the node performed the operation on

Parameters:
  • data_block_id – data_block_id to log the operation on to. either a UUID or ‘all’ to append the operation to all data blocks
  • operation – name of the operation, usually the same as the name of the node in all lowercase
  • parameters – operation parameters.
check_operation_exists(data_block_id: uuid.UUID, operation: str) → bool[source]

Check if a specific operation was performed on a specific datablock

create_data_block(dataframe: pandas.core.frame.DataFrame) → Tuple[pandas.core.frame.DataFrame, uuid.UUID][source]

Creates a new UUID, assigns it to the input dataframe by setting the UUID in the _BLOCK_ column

Parameters:dataframe – Assigns a block ID to this entire DataFrame.
data_blocks

List of UUIDs that allow you to pin down the history of specific rows of the dataframe to their history as stored in the history trace data structure (self.history) and outlined in the doc string

static from_dict(d: dict) → dict[source]

Format a dict stored using HistoryTrace.to_dict so that it can be used to create a HistoryTrace instance. Converts all the <str> representations of UUID back to <uuid.UUID> types.

Parameters:d – dict containing appropriate ‘history’ and ‘datablocks’ keys. Must be packaged by HistoryTrace.to_dict()
Returns:dict formatted so that it can be used to instantiate a HistoryTrace instance recapitulating the HistoryTrace it was packaged from.
classmethod from_json(path: str)[source]

Instantiate HistoryTrace from JSON file (that was saved using HistoryTrace.to_json)

Parameters:path – file path, usually ends with .json
classmethod from_pickle(path: str)[source]

Load HistoryTrace that was pickled

Parameters:path – file path
get_all_data_blocks_history() → dict[source]

Returns history trace of all datablocks

get_data_block_history(data_block_id: uuid.UUID) → list[source]

Get the full history trace of a single data block

get_operation_params(data_block_id: uuid.UUID, operation: str) → dict[source]

Get the parameters dict for a specific operation that was performed on a specific data block

get_operations_list(data_block_id: Union[uuid.UUID, str]) → list[source]

Returns just a simple list of operations in the order that they were performed on the given datablock. To get the operations along with their parameters call get_data_block_history()

history

The analysis log that is stored in the structure outlined in the doc string

classmethod merge(history_traces: list)[source]

Merge a list of HistoryTrace instances into one HistoryTrace instance. Useful when merging Transmission objects.

Parameters:history_traces – list of HistoryTrace instances
to_dict() → dict[source]

Package the HistoryTrace instance as a dict. Converts all UUIDs to <str> representation for JSON compatibility.

to_json(path: str)[source]

Save HistoryTrace to a JSON file.

Parameters:path – file path, usually ends with .json
to_pickle(path: str)[source]

Dump this instance to a pickle

Parameters:path – file path