python-zodb 1:3.10.5-0ubuntu3 / usr / share / pyshared / ZODB / DemoStorage.test

==========================
DemoStorage demo (doctest)
==========================

DemoStorages provide a way to provide incremental updates to an
existing, base, storage without updating the storage.

.. We need to mess with time to prevent spurious test failures on windows

    >>> now = 1231019584.0
    >>> def faux_time_time():
    ...     global now
    ...     now += .1
    ...     return now
    >>> import time
    >>> real_time_time = time.time
    >>> time.time = faux_time_time

To see how this works, we'll start by creating a base storage and
puting an object (in addition to the root object) in it:

    >>> from ZODB.FileStorage import FileStorage
    >>> base = FileStorage('base.fs')
    >>> from ZODB.DB import DB
    >>> db = DB(base)
    >>> from persistent.mapping import PersistentMapping
    >>> conn = db.open()
    >>> conn.root()['1'] = PersistentMapping({'a': 1, 'b':2})
    >>> import transaction
    >>> transaction.commit()
    >>> db.close()
    >>> import os
    >>> original_size = os.path.getsize('base.fs')

Now, lets reopen the base storage in read-only mode:

    >>> base = FileStorage('base.fs', read_only=True)

And open a new storage to store changes:

    >>> changes = FileStorage('changes.fs')

and combine the 2 in a demofilestorage:

    >>> from ZODB.DemoStorage import DemoStorage
    >>> storage = DemoStorage(base=base, changes=changes)

If there are no transactions, the storage reports the lastTransaction
of the base database:

    >>> storage.lastTransaction() == base.lastTransaction()
    True

Let's add some data:

    >>> db = DB(storage)
    >>> conn = db.open()
    >>> items = conn.root()['1'].items()
    >>> items.sort()
    >>> items
    [('a', 1), ('b', 2)]

    >>> conn.root()['2'] = PersistentMapping({'a': 3, 'b':4})
    >>> transaction.commit()

    >>> conn.root()['2']['c'] = 5
    >>> transaction.commit()

Here we can see that we haven't modified the base storage:

    >>> original_size == os.path.getsize('base.fs')
    True

But we have modified the changes database:

    >>> len(changes)
    2

Our lastTransaction reflects the lastTransaction of the changes:

    >>> storage.lastTransaction() > base.lastTransaction()
    True

    >>> storage.lastTransaction() == changes.lastTransaction()
    True

Let's walk over some of the methods so we can see how we delegate to
the new underlying storages:

    >>> from ZODB.utils import p64, u64
    >>> storage.load(p64(0), '') == changes.load(p64(0), '')
    True
    >>> storage.load(p64(0), '') == base.load(p64(0), '')
    False
    >>> storage.load(p64(1), '') == base.load(p64(1), '')
    True

    >>> serial = base.getTid(p64(0))
    >>> storage.loadSerial(p64(0), serial) == base.loadSerial(p64(0), serial)
    True

    >>> serial = changes.getTid(p64(0))
    >>> storage.loadSerial(p64(0), serial) == changes.loadSerial(p64(0),
    ...                                                          serial)
    True

The object id of the new object is quite random, and typically large:

    >>> print u64(conn.root()['2']._p_oid)
    3553260803050964942

Let's look at some other methods:

    >>> storage.getName()
    "DemoStorage('base.fs', 'changes.fs')"

    >>> storage.sortKey() == changes.sortKey()
    True

    >>> storage.getSize() == changes.getSize()
    True

    >>> len(storage) == len(changes)
    True


Undo methods are simply copied from the changes storage:

    >>> [getattr(storage, name) == getattr(changes, name)
    ...  for name in ('supportsUndo', 'undo', 'undoLog', 'undoInfo')
    ...  ]
    [True, True, True, True]

    >>> db.close()

Closing demo storages
=====================

Normally, when a demo storage is closed, it's base and changes
storage are closed:

    >>> from ZODB.MappingStorage import MappingStorage
    >>> demo = DemoStorage(base=MappingStorage(), changes=MappingStorage())
    >>> demo.close()
    >>> demo.base.opened(), demo.changes.opened()
    (False, False)

You can pass constructor arguments to control whether the base and
changes storages should be closed when the demo storage is closed:

    >>> demo = DemoStorage(
    ...     base=MappingStorage(), changes=MappingStorage(),
    ...     close_base_on_close=False, close_changes_on_close=False,
    ...     )
    >>> demo.close()
    >>> demo.base.opened(), demo.changes.opened()
    (True, True)


Storage Stacking
================

A common use case is to stack demo storages.  DemoStorage provides
some helper functions to help with this.  The push method, just
creates a new demo storage who's base is the original demo storage:

    >>> demo = DemoStorage()
    >>> demo2 = demo.push()
    >>> demo2.base is demo
    True

We can also supply an explicit changes storage, if we wish:

    >>> changes = MappingStorage()
    >>> demo3 = demo2.push(changes)
    >>> demo3.changes is changes, demo3.base is demo2
    (True, True)

The pop method closes the changes storage and returns the base
*without* closing it:

    >>> demo3.pop() is demo2
    True

    >>> changes.opened()
    False

If storage returned by push is closed, the original storage isn't:

    >>> demo3.push().close()
    >>> demo2.opened()
    True

Blob Support
============

DemoStorage supports Blobs if the changes database supports blobs.

    >>> import ZODB.blob
    >>> base = ZODB.blob.BlobStorage('base', FileStorage('base.fs'))
    >>> db = DB(base)
    >>> conn = db.open()
    >>> conn.root()['blob'] = ZODB.blob.Blob()
    >>> conn.root()['blob'].open('w').write('state 1')
    >>> transaction.commit()
    >>> db.close()

    >>> base = ZODB.blob.BlobStorage('base',
    ...                               FileStorage('base.fs', read_only=True))
    >>> changes = ZODB.blob.BlobStorage('changes',
    ...                                 FileStorage('changes.fs', create=True))
    >>> storage = DemoStorage(base=base, changes=changes)

    >>> db = DB(storage)
    >>> conn = db.open()
    >>> conn.root()['blob'].open().read()
    'state 1'
    >>> _ = transaction.begin()
    >>> conn.root()['blob'].open('w').write('state 2')
    >>> transaction.commit()
    >>> conn.root()['blob'].open().read()
    'state 2'

    >>> storage.temporaryDirectory() == changes.temporaryDirectory()
    True

    >>> db.close()

It isn't necessary for the base database to support blobs.

    >>> base = FileStorage('base.fs', read_only=True)
    >>> changes = ZODB.blob.BlobStorage('changes', FileStorage('changes.fs'))
    >>> storage = DemoStorage(base=base, changes=changes)
    >>> db = DB(storage)
    >>> conn = db.open()
    >>> conn.root()['blob'].open().read()
    'state 2'

    >>> _ = transaction.begin()
    >>> conn.root()['blob2'] = ZODB.blob.Blob()
    >>> conn.root()['blob2'].open('w').write('state 1')
    >>> conn.root()['blob2'].open().read()
    'state 1'

    >>> db.close()

If the changes database is created implicitly, it will get a blob
storage wrapped around it when necessary:

    >>> base = ZODB.blob.BlobStorage('base',
    ...                               FileStorage('base.fs', read_only=True))
    >>> storage = DemoStorage(base=base)

    >>> type(storage.changes).__name__
    'MappingStorage'

    >>> db = DB(storage)
    >>> conn = db.open()
    >>> conn.root()['blob'].open().read()
    'state 1'

    >>> type(storage.changes).__name__
    'BlobStorage'

    >>> _ = transaction.begin()
    >>> conn.root()['blob'].open('w').write('state 2')
    >>> transaction.commit()
    >>> conn.root()['blob'].open().read()
    'state 2'

    >>> storage.temporaryDirectory() == storage.changes.temporaryDirectory()
    True

    >>> db.close()

.. Check that the temporary directory is gone

   For now, it won't go until the storage does.

   >>> transaction.abort()
   >>> blobdir = storage.temporaryDirectory()
   >>> del storage, _

   >>> import gc
   >>> _ = gc.collect()

   >>> import os
   >>> os.path.exists(blobdir)
   False

ZConfig support
===============

You can configure demo storages using ZConfig, using name, changes,
and base options:

    >>> import ZODB.config
    >>> storage = ZODB.config.storageFromString("""
    ... <demostorage>
    ... </demostorage>
    ... """)
    >>> storage.getName()
    "DemoStorage('MappingStorage', 'MappingStorage')"

    >>> storage = ZODB.config.storageFromString("""
    ... <demostorage>
    ...    <filestorage base>
    ...       path base.fs
    ...    </filestorage>
    ...
    ...    <filestorage changes>
    ...       path changes.fs
    ...    </filestorage>
    ... </demostorage>
    ... """)
    >>> storage.getName()
    "DemoStorage('base.fs', 'changes.fs')"

    >>> storage.close()

    >>> storage = ZODB.config.storageFromString("""
    ... <demostorage>
    ...    name bob
    ...    <filestorage>
    ...       path base.fs
    ...    </filestorage>
    ...
    ...    <filestorage changes>
    ...       path changes.fs
    ...    </filestorage>
    ... </demostorage>
    ... """)
    >>> storage.getName()
    'bob'
    >>> storage.base.getName()
    'base.fs'

    >>> storage.close()

Generating OIDs
===============

When asked for a new OID DemoStorage chooses a value and then
verifies that neither the base or changes storages already contain
that OID.  It chooses values sequentially from random starting
points, picking new starting points whenever a chosen value us already
in the changes or base.

Under rare circumstances an OID can be chosen that has already been
handed out, but which hasn't yet been comitted.  Lets verify that if
the same OID is chosen twice during a transaction that everything will
still work.

To test this, we need to hack random.randint a bit.

    >>> import random
    >>> randint = random.randint

    >>> rv = 42
    >>> def faux_randint(min, max):
    ...     print 'called randint'
    ...     global rv
    ...     rv += 1000
    ...     return rv

    >>> random.randint = faux_randint

Now, we create a demostorage.

    >>> storage = DemoStorage()
    called randint

If we ask for an oid, we'll get 1042.

    >>> u64(storage.new_oid())
    1042

oids are allocated seuentially:

    >>> u64(storage.new_oid())
    1043

Now, we'll save 1044 in changes so that it has to pick a new one randomly.

    >>> t = transaction.get()
    >>> ZODB.tests.util.store(storage.changes, 1044)

    >>> u64(storage.new_oid())
    called randint
    2042

Now, we hack rv to 1042 is given out again and we'll save 2043 in base
to force another attempt:

    >>> rv -= 1000
    >>> ZODB.tests.util.store(storage.changes, 2043)
    >>> oid = storage.new_oid()
    called randint
    called randint
    >>> u64(oid)
    3042

DemoStorage keeps up with the issued OIDs to know when not to reissue them...

    >>> oid in storage._issued_oids
    True

...but once data is stored with a given OID...

    >>> ZODB.tests.util.store(storage, oid)

...there's no need to remember it any longer:

    >>> oid in storage._issued_oids
    False

    >>> storage.close()

.. restore randint

    >>> random.randint = randint

.. restore time

    >>> time.time = real_time_time