python - How to populate shelf with existing dictionary -

python - How to populate shelf with existing dictionary -

lets have large, 100s of megabytes, dictionary want create on disk shelve. i'm using pypar utilize mpi generate cleaned bits of master list. what's best way accomplish this? example:

# much before masterdict = shelve.open( 'ashelvefile' ) # .. . . . . # work out parts of masterdict maintain # , set acleandict # utilize mpi pypar send cleaned bits master rank if pypar.rank() == 0: tempdict = {} p in range(1,pypar.size()): tempdict.append(pypar.receive(p)) l1 in tempdict: l2 in l1: realdict.append(l2) p in range(1,pypar.size()): pypar.send(realdict,p) # realdict has cleaned bits # send other hosts else: pypar.send(acleandict, 0 ) acleandict = pypar.receive( 0 ) # replace masterdict acleandict # note: cannot send shelve dictonaries using pypar # insert stackover flow code here.

here shelve simple dictionary , create accessibly via key mydict:

import shelve mydict = {"a" : 1, "b" : 2} myshelveddict = shelve.open("my_shelved_dictionary.db") myshelveddict["mydict"] = mydict

note contents of dictionary must pickleable, shelved.

if want replicate construction of dictionary in shelve, i.e. not have mydict key keys of dictionary straight keys of shelf, can utilize update method of shelve:

myshelveddict.update(mydict)

the shelve interface has big overlap dict one.

python mpi shelve