Adding `_to_python_object` in every class

[czgdp1807]

I get it, and I have felt the same while I was working on it... so this is the proposal:

Every parallel algorithm has _parallel appended to it, so putting it under different modules isn't really useful. Though our current classes, may not work with parallel algorithms through processes because sharing user defined objects among processes is not possible with python. A _to_python_object method can be introduced in every class to support that. Opening a separate issue for this.

Originally posted by @czgdp1807 in #207 (comment)

Should this be done? The main point here is to support parallel algorithms with processes. Any other ideas can be proposed.

[HarsheetKakar]

Maybe a naive question but cant this be done by threads instead... I dont think threads have the same issue with its queue.

[czgdp1807]

Yes, we are doing it by threads for now. They don't really provide parallelism as only one thread executes one time due to GIL.
In addition, what if we want to use a mixture of processes and threads. Processes can run in different memory spaces but threads won't. Will be helpful while porting the code to C++ if we try it out in python(fast prototyping).

[HarsheetKakar]

How to approach this problem? I have had it in different module which if i remember correctly gave me a pickle error (it cant be serialized). I have googled a bit and I couldnt find anything useful.

[czgdp1807]

We first need to define, how each class can be translated to a builtin python representation. May be a simple recursive algorithm can be designed for this task. By doing this, we can just convert every object to builtin python object which can be operated upon by processes. In addition, a function will also be needed to translate back the outputs to the classes in PyDataStructs. It will be a small overhead in computation.

[HarsheetKakar]

We first need to define, how each class can be translated to a builtin python representation. May be a simple recursive algorithm can be designed for this task. By doing this, we can just convert every object to builtin python object which can be operated upon by processes. In addition, a function will also be needed to translate back the outputs to the classes in PyDataStructs. It will be a small overhead in computation.

you mean by extending them from object class?

[czgdp1807]

you mean by extending them from object class?

I didn't get your point. Can you please explain more? I meant to represent class every in builtin python object. For example every class can be represented as a dict(probably nested) with key as attributes and their values as values of dict.

[HarsheetKakar]

I cant understand it exactly, can you explain it to me with this example?

class ABC(Base):
   def __new__(cls, att):
       obj = Base.__new__(cls)
       obj.att = att

this is the class, how do u want ABC to change

[czgdp1807]

ABC._to_python_object will return a dict,
pyobj = {attr: getattr(abc_object, attr) for attr in dir(abc_object)}. More filters are required like, removal of methods/properties, since in multiprocessing they won't be of any use. _to_python_object has some use cases,

1. deepcopy will work no matter what, just use,

obj = ABC()
pyobj = obj._to_python_object()
copy_pyobj = copy.deepcopy(pyobj)
copy_obj = ABC()
copy_obj.__setattribute__(attr, val) for attr, val in copy_pyobj 

2. Helpful in sharing objects among processes as only builtin python object work in multiprocessing.

[HarsheetKakar]

ok so if ABC has a data attribute of DODA and then _to_python_object will call DODA._to_python_object recursively?

[czgdp1807]

Yes.

[robotjellyzone]

Hi @HarsheetKakar are you woking on this issue ?

[HarsheetKakar]

@czgdp1807 have you seen the multiprocessing.shared_memory? I think it will solve our problem in a much easily.

[czgdp1807]

have you seen the multiprocessing.shared_memory?

May be it is for python 3.8,

Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydatastructs import SinglyLinkedList
>>> from multiprocessing import shared_memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'shared_memory'

How will deep copy work using shared memory?