Persistence: Saving and Loading Data

These functions can be imported from the ppc_robot_lib.reporting.persistence module.

Job Storage

Each job has a private storage. You can save arbitrary objects to this storage and load them in the subsequent runs of the same job.

Each object is identified by a key. Key is a string from any characters that can be used in a valid file path.

The storage is isolated and private for each job. Example: we have a task type called product_dsa_generator. When this task type finishes, it stores its state into an object with key state. This object is loaded in the subsequent runs. User creates two jobs: Job A for account A, and Job B for account B. The user runs Job A, and when it is done, it stores its state. Afterwards, the user runs Job B.

However, this job does not load state previously stored by Job A, because the storage of each job is isolated. Instead, it will create its own state and stores it in its own storage.

save_job_object(key, obj)

Saves a given object to persistent storage of the current job.

The ``pickle` module is used to store the object, therefor the object to store must be picklable.

Example:

>>> obj = {'key': 'value'}
>>> save_job_object('example', obj)

Parameters:

• key (str) – Object key, can be any string that forms a valid file-system path.

• obj (Any) – Object to store. The object must be picklable.

Return type:

None

save_job_dataframe(key, df)

Saves a dataframe to persistent storage of the current job.

Example:

>>> import pandas
>>> df = pandas.DataFrame(data=[('a', 1), ('b', 2)], columns=['key', 'value'])
>>> save_job_dataframe('example_df', df)

Parameters:

• key (str) – Object key, can be any string that forms a valid file-system path.

• df (DataFrame) – DataFrame to store. The DataFrame is stored as pickle object.

Return type:

None

load_job_object(key)

Loads a previously saved object from the persistent storage of the current job.

Example:

>>> from ppc_robot_lib.tasks.job_storage.exceptions import ObjectDoesNotExist
>>> try:
...     obj = load_job_object('example')
... except ObjectDoesNotExist:
...     obj = {}  # Create empty object.

Parameters:

key (str) – Key of the previously stored object.

Return type:

Any

Returns:

Previously stored object.

Raises:

ObjectDoesNotExist – if the object does not exist.

load_job_dataframe(key)

Loads a previously saved DataFrame from the persistent storage of the current job.

Example:

>>> from ppc_robot_lib.tasks.job_storage.exceptions import ObjectDoesNotExist
>>> try:
...     df = load_job_dataframe('example')
... except ObjectDoesNotExist:
...     df = pandas.DataFrame(data=[], columns=['key', 'value'])  # Create a new DataFrame

Parameters:

key (str) – Key of the previously stored DataFrame.

Return type:

DataFrame

Returns:

Previously stored DataFrame.

Raises:

ObjectDoesNotExist – if the object does not exist.