pytorch_pfn_extras.writing.QueueWriter#

class pytorch_pfn_extras.writing.QueueWriter(savefun=<function save>, fs=None, out_dir='', task=None)#

Bases: Writer, Generic[_Worker]

Base class of queue snapshot writers.

This class is a base class of snapshot writers that use a queue. A Queue is created when this class is constructed, and every time when __call__ is invoked, a snapshot task is put into the queue.

Parameters:

  • savefun – Callable object which is passed to the create_task() if the task is None. It takes three arguments: the output file path, the serialized dictionary object, and the optional keyword arguments.

  • fs – FileSystem abstracting interface to implement all the operations. optional, defaults to None

  • out_dir – str. Specifies the directory this writer will use. It takes precedence over the one specified in __call__ optional, defaults to ''

  • task – Callable object. Its __call__ must have a same interface to Writer.__call__. This object is directly put into the queue.

See also

Methods

__init__([savefun, fs, out_dir, task])

consume(q)

create_consumer(q)

create_queue()

create_task(savefun)

finalize()

Finalizes the writer.

initialize(out_dir)

save(filename, out_dir, target, savefun, ...)

__call__(filename, out_dir, target, *, savefun=None, append=False)#

Does the actual writing to the file.

This method is invoked by a Snapshot object every time it takes a snapshot.

Parameters:

  • filename (str) – Name of the file into which the serialized target is saved. It is a concrete file name, i.e. not a pre-formatted template string.

  • out_dir (str) –

    Output directory. Corresponds to :py:attr:`ExtensionsManager.out

    <pytorch_pfn_extras.training.ExtensionsManager.out>`.

  • target (dict) – Serialized object which will be saved.

  • savefun (callable) – A callable that accepts a two positional arguments (an object to be serialized, file path) like torch.save.

  • append (bool) – Mode used to open the file. True to use the append mode, False to use the write mode (truncates the file if it already exists).

Return type:

None

__init__(savefun=<function save>, fs=None, out_dir='', task=None)#
Parameters:

  • savefun (Callable[[...], None]) –

  • fs (Optional[Any]) –

  • out_dir (str) –

  • task (Optional[Callable[[...], None]]) –

Return type:

None

consume(q)#
Parameters:

q (queue.Queue[_QueUnit]) –

Return type:

None

create_consumer(q)#
Parameters:

q (queue.Queue[_QueUnit]) –

Return type:

_Worker

create_queue()#
Return type:

queue.Queue[_QueUnit]

create_task(savefun)#
Parameters:

savefun (Callable[[...], None]) –

Return type:

Callable[[…], None]

finalize()#

Finalizes the writer.

Calling this method on already-finalized Writer does nothing.

Return type:

None