pickling module ¶

Utilities for pickling.

rec_info_registry dict ¶

Registry with instances of RecInfo keyed by RecInfo.id_.

Populate with the required information if any instance cannot be unpickled.

dumps function ¶

dumps(
    obj,
    compression=None,
    compress_kwargs=None,
    **kwargs
)

Pickle an object to a byte stream.

Uses dill when available.

For compression options, see extensions.

Keyword arguments compress_kwargs are passed to the compress method of the compressing package.

Other keyword arguments are passed to the dumps() method of the pickling package.

get_class_from_id function ¶

get_class_from_id(
    class_id
)

Get the class from a class id.

get_compression_extensions function ¶

get_compression_extensions(
    cls_name=None
)

Get all supported compression extensions from vectorbtpro._settings.

get_id_from_class function ¶

get_id_from_class(
    obj
)

Get the class id from a class.

If the object is an instance or a subclass of Pickleable and Pickleable._rec_id is not None, uses the reconstruction id. Otherwise, returns the path to the class definition with find_class().

get_serialization_extensions function ¶

get_serialization_extensions(
    cls_name=None
)

Get all supported serialization extensions from vectorbtpro._settings.

load function ¶

load(
    path,
    compression=None,
    **kwargs
)

Read a byte stream from a file and unpickle.

Can recognize the compression algorithm based on the extension (see dumps() for options).

Uses loads() for deserialization.

loads function ¶

loads(
    bytes_,
    compression=None,
    **kwargs
)

Unpickle an object from a byte stream.

Accepts the same options for compression as in the dumps() method.

Other keyword arguments are passed to the loads() method of the pickling package.

reconstruct function ¶

reconstruct(
    cls,
    rec_state
)

Reconstruct an instance using a class and a reconstruction state.

save function ¶

save(
    obj,
    path=None,
    mkdir_kwargs=None,
    compression=None,
    **kwargs
)

Pickle an object to a byte stream and write to a file.

Can recognize the compression algorithm based on the extension (see dumps() for options).

Uses dumps() for serialization.

Pickleable class ¶

Pickleable()

Superclass that defines abstract properties and methods for pickle-able classes.

If any subclass cannot be pickled, override the Pickleable.rec_state property to return an instance of RecState to be used in reconstruction. If the class definition cannot be pickled (e.g., created dynamically), override its _rec_id with an arbitrary id string, dump/save the class, and before loading, map this id to the class in rec_id_map. This will use the mapped class to construct a new instance.

Subclasses

decode_config class method ¶

Pickleable.decode_config(
    str_,
    parse_literals=True,
    run_code=True,
    pack_objects=True,
    use_refs=True,
    use_class_ids=True,
    code_context=None,
    parser_kwargs=None,
    check_type=True,
    **kwargs
)

Decode an instance from a config string.

Can parse configs without sections. Sections can also become sub-dicts if their names use the dot notation. For example, section a.b will become a sub-dict of the section a and section a.b.c will become a sub-dict of the section a.b. You don't have to define the section a explicitly, it will automatically become the outermost key.

If a section contains only one pair _ = _, it will become an empty dict.

If parse_literals is True, will detect any Python literals and containers such as True and []. Will also understand np.nan, np.inf, and -np.inf.

If run_code is True, will run any Python code prepended with !. Will use the context code_context together with already defined np (NumPy), pd (Pandas), and vbt (vectorbtpro).

Warning

Unpickling byte streams and running code has important security implications. Don't attempt to parse configs coming from untrusted sources as those can contain malicious code!

If pack_objects is True, will look for class paths prepended with @ in section names, construct an instance of RecState (any other keyword arguments will be included to init_kwargs), and finally use reconstruct() to reconstruct the unpacked object.

If use_refs is True, will substitute references prepended with & for actual objects. Constructs a DAG using graphlib.

If use_class_ids is True, will substitute any class ids prepended with @ with the corresponding class.

Other keyword arguments are forwarded to Pickleable.decode_config_node().

Usage

File types.ini:

string = 'hello world'
boolean = False
int = 123
float = 123.45
exp_float = 1e-10
nan = np.nan
inf = np.inf
numpy = !np.array([1, 2, 3])
pandas = !pd.Series([1, 2, 3])
expression = !dict(sub_dict2=dict(some="value"))
mult_expression = !import math; math.floor(1.5)

>>> from vectorbtpro import *

>>> vbt.pprint(vbt.pdict.load("types.ini"))
pdict(
    string='hello world',
    boolean=False,
    int=123,
    float=123.45,
    exp_float=1e-10,
    nan=np.nan,
    inf=np.inf,
    numpy=<numpy.ndarray object at 0x7fe1bf84f690 of shape (3,)>,
    pandas=<pandas.core.series.Series object at 0x7fe1c9a997f0 of shape (3,)>,
    expression=dict(
        sub_dict2=dict(
            some='value'
        )
    ),
    mult_expression=1
)

File refs.ini:

[top]
sr = &top.sr

[top.sr @pandas.Series]
data = [10756.12, 10876.76, 11764.33]
index = &top.sr.index
name = 'Open time'

[top.sr.index @pandas.DatetimeIndex]
data = ["2023-01-01", "2023-01-02", "2023-01-03"]

>>> vbt.pdict.load("refs.ini")["sr"]
2023-01-01    10756.12
2023-01-02    10876.76
2023-01-03    11764.33
Name: Open time, dtype: float64

decode_config_node class method ¶

Pickleable.decode_config_node(
    key,
    value,
    **kwargs
)

Decode a config node.

dumps method ¶

Pickleable.dumps(
    rec_state_only=False,
    **kwargs
)

Pickle the instance to a byte stream.

Optionally, you can set rec_state_only to True if the instance will be later unpickled directly by the class.

encode_config method ¶

Pickleable.encode_config(
    top_name=None,
    unpack_objects=True,
    compress_unpacked=True,
    use_refs=True,
    use_class_ids=True,
    nested=True,
    to_dict=False,
    parser_kwargs=None,
    **kwargs
)

Encode the instance to a config string.

Based on Pickleable.rec_state. Raises an error if None.

Encodes to a format that is guaranteed to be parsed using Pickleable.decode_config(). Otherwise, an error will be thrown. If any object cannot be represented using a string, uses dumps() to convert it to a byte stream.

When unpack_objects is True and an object is an instance of Pickleable, saves its reconstruction state to a separate section rather than the byte stream. Appends @ and class name to the section name. If compress_unpacked is True, will hide keys in RecState that have empty values. Keys in RecState will be appended with ~ to avoid collision with user-defined keys having the same name.

If use_refs is True, out of unhashable objects sharing the same id, only the first one will be defined while others will store the reference (& + key path) to the first one.

Note

The initial order of keys can be preserved only by using references.

If use_class_ids is True, substitutes any class defined as a value by its id instead of pickling its definition. If get_id_from_class() returns None, will pickle the definition.

If the instance is nested, set nested to True to represent each sub-dict as a section.

Other keyword arguments are forwarded to Pickleable.encode_config_node().

encode_config_node method ¶

Pickleable.encode_config_node(
    key,
    value,
    **kwargs
)

Encode a config node.

file_exists class method ¶

Pickleable.file_exists(
    *args,
    **kwargs
)

Return whether a file already exists.

Resolves the file path using Pickleable.resolve_file_path().

getsize method ¶

Pickleable.getsize(
    readable=True,
    **kwargs
)

Get size of this object.

load class method ¶

Pickleable.load(
    path=None,
    file_format=None,
    compression=None,
    **kwargs
)

Unpickle/decode the instance from a file.

Resolves the file path using Pickleable.resolve_file_path().

loads class method ¶

Pickleable.loads(
    bytes_,
    check_type=True,
    **kwargs
)

Unpickle an instance from a byte stream.

modify_state class method ¶

Pickleable.modify_state(
    rec_state
)

Modify the reconstruction state before reconstruction.

rec_state property ¶

Reconstruction state of the type RecState.

resolve_file_path class method ¶

Pickleable.resolve_file_path(
    path=None,
    file_format=None,
    compression=None,
    for_save=False
)

Resolve a file path.

A file must have either one or two extensions: file format (required) and compression (optional). For file format options, see pickle_extensions and config_extensions. For compression options, see compression_extensions. Each can be provided either via a suffix in path, or via the argument file_format and compression respectively.

When saving, uses file_format and compression from pickling as default options. When loading, searches for matching files in the current directory.

Path can be also None or directory, in such a case the file name will be set to the class name.

save method ¶

Pickleable.save(
    path=None,
    file_format=None,
    compression=None,
    mkdir_kwargs=None,
    **kwargs
)

Pickle/encode the instance and save to a file.

Resolves the file path using Pickleable.resolve_file_path().

RecInfo class ¶

RecInfo(
    *args,
    **kwargs
)

Class that represents information needed to reconstruct an instance.

Superclasses

Inherited members

cls class variable ¶

Class.

id_ class variable ¶

Identifier.

modify_state class variable ¶

Callback to modify the reconstruction state.

register method ¶

RecInfo.register()

Register self in rec_info_registry.

RecState class ¶

RecState(
    *args,
    **kwargs
)

Class that represents a state used to reconstruct an instance.

Superclasses

Inherited members

attr_dct class variable ¶

Dictionary with names and values of writeable attributes.

init_args class variable ¶

Positional arguments used in initialization.

init_kwargs class variable ¶

Keyword arguments used in initialization.

pdict class ¶

pdict(
    *args,
    **kwargs
)

Pickleable dict.

Superclasses

Comparable
Pickleable
Prettified
builtins.dict

Inherited members

Subclasses

equals method ¶

pdict.equals(
    other,
    check_types=True,
    **kwargs
)

Check two objects for equality.

load_update method ¶

pdict.load_update(
    path=None,
    clear=False,
    **kwargs
)

Load dumps from a file and update this instance in-place.

pickling module¶

rec_info_registry dict¶

dumps function¶

get_class_from_id function¶

get_compression_extensions function¶

get_id_from_class function¶

get_serialization_extensions function¶

load function¶

loads function¶

reconstruct function¶

save function¶

Pickleable class¶

decode_config class method¶

decode_config_node class method¶

dumps method¶

encode_config method¶

encode_config_node method¶

file_exists class method¶

getsize method¶

load class method¶

loads class method¶

modify_state class method¶

rec_state property¶

resolve_file_path class method¶

save method¶

RecInfo class¶

cls class variable¶

id_ class variable¶

modify_state class variable¶

register method¶

RecState class¶

attr_dct class variable¶

init_args class variable¶

init_kwargs class variable¶

pdict class¶

equals method¶

load_update method¶

pickling module ¶

rec_info_registry dict ¶

dumps function ¶

get_class_from_id function ¶

get_compression_extensions function ¶

get_id_from_class function ¶

get_serialization_extensions function ¶

load function ¶

loads function ¶

reconstruct function ¶

save function ¶

Pickleable class ¶

decode_config class method ¶

decode_config_node class method ¶

dumps method ¶

encode_config method ¶

encode_config_node method ¶

file_exists class method ¶

getsize method ¶

load class method ¶

loads class method ¶

modify_state class method ¶

rec_state property ¶

resolve_file_path class method ¶

save method ¶

RecInfo class ¶

cls class variable ¶

id_ class variable ¶

modify_state class variable ¶

register method ¶

RecState class ¶

attr_dct class variable ¶

init_args class variable ¶

init_kwargs class variable ¶

pdict class ¶

equals method ¶

load_update method ¶