Skip to content

[wishlist] Make Git() serializable (i.e. pickleable) #504

@yarikoptic

Description

@yarikoptic
Contributor

I decided to become fancy and use some multiprocessing in our datalad, but quickly was bombarded with

  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
PicklingError: Can't pickle <class 'git.cmd.AutoInterrupt'>: attribute lookup git.cmd.AutoInterrupt failed
()

so I looked into cmd.py to see that AutoInterupt is actuall a sub-class of the Git class... so something went hairy here and I have decided to reproduce on a more obvious snippet:

$> python -c 'from git import Git; r = Git("/tmp/123"); import pickle; pickle.dumps(r)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

so failed to pickle directly but differently... but overall story is that as long as I have a Git object involved anyhow -- can't pickle my beast, can't use multiprocessing :-/

Activity

added this to the v2.0.9 - Bugfixes milestone on Sep 11, 2016
Byron

Byron commented on Sep 11, 2016

@Byron
Member

A fix could be as easy as to not use slots at all. Considering the difficulties slots add to subclasses, and that it seems to be generally frowned upon in the python community for all their weaknesses, it would be a good move at least in this case.

I tried it and it seems to not break anything, thus I'd just push the respective change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @yarikoptic@Byron

        Issue actions

          [wishlist] Make Git() serializable (i.e. pickleable) · Issue #504 · gitpython-developers/GitPython