dave
/
relstorage
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge pull request #23 from NextThought/pypy 

Add Support for PyPy.
This commit is contained in:
Jason Madden 2016-06-14 11:27:08 -05:00 committed by GitHub
parent bc470d9333 e9a4b538c4
commit 01ae65b713
19 changed files with 387 additions and 127 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.travis.yml
View File

@ -3,6 +3,9 @@ sudo: false
services:
- mysql
- postgresql
python:
- 2.7
- pypy
env:
matrix:
- ENV=mysql
@ -10,7 +13,9 @@ env:
matrix:
fast_finish: true
script:
- coverage run -m relstorage.tests.alltests
# coverage slows PyPy down from 2minutes to 12+.
- if [[ $TRAVIS_PYTHON_VERSION == 'pypy' ]]; then python -m relstorage.tests.alltests; fi
- if [[ $TRAVIS_PYTHON_VERSION != 'pypy' ]]; then coverage run -m relstorage.tests.alltests; fi
after_success:
- coveralls
notifications:

2
.travis/setup-mysql.sh
View File

@ -1,4 +1,4 @@
pip install -U MySQL-python
pip install -U -e ".[mysql]"
mysql -uroot -e "CREATE USER 'relstoragetest'@'localhost' IDENTIFIED BY 'relstoragetest';"
mysql -uroot -e "CREATE DATABASE relstoragetest;"
mysql -uroot -e "GRANT ALL ON relstoragetest.* TO 'relstoragetest'@'localhost';"

2
.travis/setup-postgres.sh
View File

@ -1,4 +1,4 @@
pip install -U psycopg2
pip install -U -e ".[postgresql]"
psql -U postgres -c "CREATE USER relstoragetest WITH PASSWORD 'relstoragetest';"
psql -U postgres -c "CREATE DATABASE relstoragetest OWNER relstoragetest;"
psql -U postgres -c "CREATE DATABASE relstoragetest2 OWNER relstoragetest;"

6
CHANGES.txt
View File

@ -7,6 +7,12 @@
POSKeyError instead of returning an empty state. (Revealed by
updated tests for FileStorage in ZODB 4.3.1.)
- Add support for PyPy on MySQL and PostgreSQL using PyMySQL and
psycopg2cffi respectively. PyPy can be substantially faster than
CPython in some scenarios; see `PR 23`_.
.. _`PR 23`: https://github.com/zodb/relstorage/pull/23/
1.6.0b3 (2014-12-08)
--------------------

16
README.txt
View File

@ -1,9 +1,9 @@
RelStorage is a storage implementation for ZODB that stores pickles in
a relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL
5.0.32+ / 5.1.34+ (via MySQLdb 1.2.2 and above), and Oracle 10g and 11g
(via cx_Oracle) are currently supported. RelStorage replaced the
PGStorage project.
a relational database. PostgreSQL 8.1 and above (via psycopg2 or
psycopg2cffi), MySQL 5.0.32+ / 5.1.34+ (via MySQLdb 1.2.2 or PyMySQL),
and Oracle 10g and 11g (via cx_Oracle) are currently supported.
RelStorage replaced the PGStorage project.
.. contents::
@ -43,8 +43,12 @@ versions of ZODB with the patch already applied here:
The patches are also included in the source distribution of RelStorage.
You need the Python database adapter that corresponds with your database.
Install psycopg2, MySQLdb 1.2.2+, or cx_Oracle 4.3+.
You need the Python database adapter that corresponds with your
database. On CPython, install psycopg2, MySQLdb 1.2.2+, or cx_Oracle
4.3+. On PyPy, install psycopg2cffi or PyMySQL (PyPy will generally
work with psycopg2 and MySQLdb, but it will be *much* slower);
cx_Oracle is untested.
Configuring Your Database
-------------------------

50
relstorage/adapters/mysql.py
View File

@ -49,7 +49,19 @@ load_infile
"""
import logging
import MySQLdb
try:
import MySQLdb
except ImportError:
import sys
t, v, tb = sys.exc_info()
try:
import pymysql
pymysql.install_as_MySQLdb()
import MySQLdb
except ImportError:
raise t, v, tb
else:
del t, v, b
from zope.interface import implements
from relstorage.adapters.connmanager import AbstractConnectionManager
@ -83,6 +95,42 @@ disconnected_exceptions = (
# when the adapter attempts to close a database connection.
close_exceptions = disconnected_exceptions + (MySQLdb.ProgrammingError,)
try:
# Under PyMySql 0.6.6, closing an already closed
# connection raises a plain pymysql.err.Error.
# It can also raise a DatabaseError, and sometimes
# an IOError doesn't get mapped to a type
import pymysql.err
close_exceptions += (
pymysql.err.Error,
IOError,
pymysql.err.DatabaseError
)
disconnected_exceptions += (
IOError, # This one can escape mapping;
# This one has only been seen as its subclass,
# InternalError, as (0, 'Socket receive buffer full'),
# which should probably be taken as disconnect
pymysql.err.DatabaseError,
)
except ImportError:
pass
try:
import pymysql.converters
# PyPy up through at least 5.3.0 has a bug that raises spurious
# MemoryErrors when run under PyMySQL >= 0.7.
# (https://bitbucket.org/pypy/pypy/issues/2324/bytearray-replace-a-bc-raises-memoryerror)
# Patch around it.
if hasattr(pymysql.converters, 'escape_string'):
orig_escape_string = pymysql.converters.escape_string
def escape_string(value, mapping=None):
if isinstance(value, bytearray) and not(value):
return value
return orig_escape_string(value, mapping)
pymysql.converters.escape_string = escape_string
except ImportError:
pass
class MySQLAdapter(object):
"""MySQL adapter for RelStorage."""

14
relstorage/adapters/postgresql.py
View File

@ -30,7 +30,19 @@ from relstorage.adapters.txncontrol import PostgreSQLTransactionControl
from relstorage.options import Options
from zope.interface import implements
import logging
import psycopg2
try:
import psycopg2
except ImportError:
import sys
t, v, tb = sys.exc_info()
try:
import psycopg2cffi.compat
psycopg2cffi.compat.register()
import psycopg2
except ImportError:
raise t, v, tb
else:
del t, v, tb
import psycopg2.extensions
import re
"""PostgreSQL adapter for RelStorage."""

2
relstorage/tests/blob/blob_cache.test
View File

@ -52,7 +52,7 @@ Now, let's write some data:
>>> conn = db.open()
>>> for i in range(1, 101):
... conn.root()[i] = ZODB.blob.Blob()
... conn.root()[i].open('w').write(chr(i)*100)
... with conn.root()[i].open('w') as f: _ = f.write(chr(i)*100)
>>> transaction.commit()
We've committed 10000 bytes of data, but our target size is 3000. We

6
relstorage/tests/blob/blob_importexport.txt
View File

@ -36,10 +36,10 @@ Put a couple blob objects in our database1 and on the filesystem:
>>> nothing = transaction.begin()
>>> data1 = 'x'*100000
>>> blob1 = ZODB.blob.Blob()
>>> blob1.open('w').write(data1)
>>> with blob1.open('w') as f: _ = f.write(data1)
>>> data2 = 'y'*100000
>>> blob2 = ZODB.blob.Blob()
>>> blob2.open('w').write(data2)
>>> with blob2.open('w') as f: _ = f.write(data2)
>>> d = PersistentMapping({'blob1':blob1, 'blob2':blob2})
>>> root1['blobdata'] = d
>>> transaction.commit()
@ -49,7 +49,7 @@ Export our blobs from a database1 connection:
>>> conn = root1['blobdata']._p_jar
>>> oid = root1['blobdata']._p_oid
>>> exportfile = 'export'
>>> nothing = connection1.exportFile(oid, exportfile)
>>> nothing = connection1.exportFile(oid, exportfile).close()
Import our exported data into database2:

12
relstorage/tests/blob/blob_packing.txt
View File

@ -34,32 +34,32 @@ Put some revisions of a blob object in our database and on the filesystem:
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> blob = Blob()
>>> blob.open('w').write('this is blob data 0')
>>> with blob.open('w') as f: _ = f.write('this is blob data 0')
>>> root['blob'] = blob
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 1')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 1')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 2')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 2')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 3')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 3')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 4')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 4')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
@ -74,7 +74,7 @@ Do a pack to the slightly before the first revision was written:
>>> blob_storage.pack(packtime, referencesf)
>>> [ os.path.exists(x) for x in fns ]
[True, True, True, True, True]
Do a pack to the slightly before the second revision was written:
>>> packtime = times[1]

10
relstorage/tests/blob/blob_packing_history_free.txt
View File

@ -34,32 +34,32 @@ Put some revisions of a blob object in our database and on the filesystem:
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> blob = Blob()
>>> blob.open('w').write('this is blob data 0')
>>> with blob.open('w') as f: _ = f.write('this is blob data 0')
>>> root['blob'] = blob
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 1')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 1')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 2')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 2')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 3')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 3')
>>> transaction.commit()
>>> tids.append(blob._p_serial)
>>> nothing = transaction.begin()
>>> times.append(new_time())
>>> root['blob'].open('w').write('this is blob data 4')
>>> with root['blob'].open('w') as f: _ = f.write('this is blob data 4')
>>> transaction.commit()
>>> tids.append(blob._p_serial)

218
relstorage/tests/blob/blob_transaction.txt
View File

@ -27,7 +27,8 @@ We need a database with a blob supporting storage::
Putting a Blob into a Connection works like any other Persistent object::
>>> blob1 = ZODB.blob.Blob()
>>> blob1.open('w').write('this is blob 1')
>>> with blob1.open('w') as file:
... _ = file.write(b'this is blob 1')
>>> root1['blob1'] = blob1
>>> 'blob1' in root1
True
@ -40,10 +41,11 @@ Aborting a blob add leaves the blob unchanged:
>>> blob1._p_oid
>>> blob1._p_jar
>>> blob1.open().read()
>>> with blob1.open() as fp:
... fp.read()
'this is blob 1'
It doesn't clear the file because there is no previously committed version:
It doesn't clear the file because there is no previously committed version:
>>> fname = blob1._p_blob_uncommitted
>>> import os
@ -62,8 +64,10 @@ state:
False
>>> blob1._p_blob_uncommitted
>>> blob1.open('w').write('this is new blob 1')
>>> blob1.open().read()
>>> with blob1.open('w') as file:
... _ = file.write(b'this is new blob 1')
>>> with blob1.open() as fp:
... fp.read()
'this is new blob 1'
>>> fname = blob1._p_blob_uncommitted
>>> os.path.exists(fname)
@ -74,7 +78,8 @@ state:
False
>>> blob1._p_blob_uncommitted
>>> blob1.open().read()
>>> with blob1.open() as fp:
... fp.read()
'this is blob 1'
Opening a blob gives us a filehandle. Getting data out of the
@ -88,7 +93,7 @@ resulting filehandle is accomplished via the filehandle's read method::
>>> blob1afh1.read()
'this is blob 1'
Let's make another filehandle for read only to blob1a. Aach file
Let's make another filehandle for read only to blob1a. Each file
handle has a reference to the (same) underlying blob::
>>> blob1afh2 = blob1a.open("r")
@ -115,19 +120,20 @@ when we start)::
>>> bool(blob1a._p_changed)
False
>>> blob1a.open('r').read()
>>> with blob1a.open('r') as fp:
... fp.read()
'this is blob 1'
>>> blob1afh3 = blob1a.open('a')
>>> bool(blob1a._p_changed)
True
>>> blob1afh3.write('woot!')
>>> with blob1a.open('a') as blob1afh3:
... assert(bool(blob1a._p_changed))
... _ = blob1afh3.write(b'woot!')
>>> blob1afh3.close()
We can open more than one blob object during the course of a single
transaction::
>>> blob2 = ZODB.blob.Blob()
>>> blob2.open('w').write('this is blob 3')
>>> with blob2.open('w') as file:
... _ = file.write(b'this is blob 3')
>>> root2['blob2'] = blob2
>>> transaction.commit()
@ -135,11 +141,14 @@ Since we committed the current transaction above, the aggregate
changes we've made to blob, blob1a (these refer to the same object) and
blob2 (a different object) should be evident::
>>> blob1.open('r').read()
>>> with blob1.open('r') as fp:
... fp.read()
'this is blob 1woot!'
>>> blob1a.open('r').read()
>>> with blob1a.open('r') as fp:
... fp.read()
'this is blob 1woot!'
>>> blob2.open('r').read()
>>> with blob2.open('r') as fp:
... fp.read()
'this is blob 3'
We shouldn't be able to persist a blob filehandle at commit time
@ -147,11 +156,12 @@ We shouldn't be able to persist a blob filehandle at commit time
pickled appears to be particulary unhelpful for casual users at the
moment)::
>>> root1['wontwork'] = blob1.open('r')
>>> transaction.commit()
>>> with blob1.open('r') as f:
... root1['wontwork'] = f
... transaction.commit()
Traceback (most recent call last):
...
TypeError: coercing to Unicode: need string or buffer, BlobFile found
...
TypeError: ...
Abort for good measure::
@ -166,10 +176,13 @@ connections should result in a write conflict error::
>>> root4 = database.open(transaction_manager=tm2).root()
>>> blob1c3 = root3['blob1']
>>> blob1c4 = root4['blob1']
>>> blob1c3fh1 = blob1c3.open('a').write('this is from connection 3')
>>> blob1c4fh1 = blob1c4.open('a').write('this is from connection 4')
>>> with blob1c3.open('a') as blob1c3fh1:
... _ = blob1c3fh1.write(b'this is from connection 3')
>>> with blob1c4.open('a') as blob1c4fh1:
... _ = blob1c4fh1.write(b'this is from connection 4')
>>> tm1.commit()
>>> root3['blob1'].open('r').read()
>>> with root3['blob1'].open('r') as fp:
... fp.read()
'this is blob 1woot!this is from connection 3'
>>> tm2.commit()
Traceback (most recent call last):
@ -179,10 +192,12 @@ connections should result in a write conflict error::
After the conflict, the winning transaction's result is visible on both
connections::
>>> root3['blob1'].open('r').read()
>>> with root3['blob1'].open('r') as fp:
... fp.read()
'this is blob 1woot!this is from connection 3'
>>> tm2.abort()
>>> root4['blob1'].open('r').read()
>>> with root4['blob1'].open('r') as fp:
... fp.read()
'this is blob 1woot!this is from connection 3'
You can't commit a transaction while blob files are open:
@ -214,21 +229,22 @@ We do support optimistic savepoints:
>>> connection5 = database.open()
>>> root5 = connection5.root()
>>> blob = ZODB.blob.Blob()
>>> blob_fh = blob.open("w")
>>> blob_fh.write("I'm a happy blob.")
>>> blob_fh.close()
>>> with blob.open("w") as blob_fh:
... _ = blob_fh.write(b"I'm a happy blob.")
>>> root5['blob'] = blob
>>> transaction.commit()
>>> root5['blob'].open("r").read()
>>> with root5['blob'].open("r") as fp:
... fp.read()
"I'm a happy blob."
>>> blob_fh = root5['blob'].open("a")
>>> blob_fh.write(" And I'm singing.")
>>> blob_fh.close()
>>> root5['blob'].open("r").read()
>>> with root5['blob'].open("a") as blob_fh:
... _ = blob_fh.write(b" And I'm singing.")
>>> with root5['blob'].open("r") as fp:
... fp.read()
"I'm a happy blob. And I'm singing."
>>> savepoint = transaction.savepoint(optimistic=True)
>>> root5['blob'].open("r").read()
>>> with root5['blob'].open("r") as fp:
... fp.read()
"I'm a happy blob. And I'm singing."
Savepoints store the blobs in temporary directories in the temporary
@ -242,14 +258,16 @@ After committing the transaction, the temporary savepoint files are moved to
the committed location again:
>>> transaction.commit()
>>> savepoint_dir = os.path.join(blob_dir, 'tmp', 'savepoint')
>>> os.path.exists(savepoint_dir) and len(os.listdir(savepoint_dir)) > 0
False
>>> len([name for name in os.listdir(os.path.join(blob_dir, 'tmp'))
... if name.startswith('savepoint')])
0
We support non-optimistic savepoints too:
>>> root5['blob'].open("a").write(" And I'm dancing.")
>>> root5['blob'].open("r").read()
>>> with root5['blob'].open("a") as file:
... _ = file.write(b" And I'm dancing.")
>>> with root5['blob'].open("r") as fp:
... fp.read()
"I'm a happy blob. And I'm singing. And I'm dancing."
>>> savepoint = transaction.savepoint()
@ -259,17 +277,107 @@ Again, the savepoint creates a new savepoints directory:
... if name.startswith('savepoint')])
1
>>> root5['blob'].open("w").write(" And the weather is beautiful.")
>>> with root5['blob'].open("w") as file:
... _ = file.write(b" And the weather is beautiful.")
>>> savepoint.rollback()
>>> root5['blob'].open("r").read()
>>> with root5['blob'].open("r") as fp:
... fp.read()
"I'm a happy blob. And I'm singing. And I'm dancing."
>>> transaction.abort()
The savepoint blob directory gets cleaned up on an abort:
>>> os.path.exists(savepoint_dir) and len(os.listdir(savepoint_dir)) > 0
False
>>> len([name for name in os.listdir(os.path.join(blob_dir, 'tmp'))
... if name.startswith('savepoint')])
0
Reading Blobs outside of a transaction
--------------------------------------
If you want to read from a Blob outside of transaction boundaries (e.g. to
stream a file to the browser), committed method to get the name of a
file that can be opened.
>>> connection6 = database.open()
>>> root6 = connection6.root()
>>> blob = ZODB.blob.Blob()
>>> with blob.open("w") as blob_fh:
... _ = blob_fh.write(b"I'm a happy blob.")
>>> root6['blob'] = blob
>>> transaction.commit()
>>> with open(blob.committed()) as fp:
... fp.read()
"I'm a happy blob."
We can also read committed data by calling open with a 'c' flag:
>>> f = blob.open('c')
This doesn't prevent us from opening the blob for writing:
>>> with blob.open('w') as file:
... _ = file.write(b'x')
>>> with blob.open() as fp: fp.read()
'x'
>>> f.read()
"I'm a happy blob."
>>> f.close()
>>> transaction.abort()
An exception is raised if we call committed on a blob that has
uncommitted changes:
>>> blob = ZODB.blob.Blob()
>>> blob.committed()
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> blob.open('c')
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> with blob.open('w') as file:
... _ = file.write(b"I'm a happy blob.")
>>> root6['blob6'] = blob
>>> blob.committed()
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> blob.open('c')
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> s = transaction.savepoint()
>>> blob.committed()
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> blob.open('c')
Traceback (most recent call last):
...
BlobError: Uncommitted changes
>>> transaction.commit()
>>> with open(blob.committed()) as fp:
... fp.read()
"I'm a happy blob."
You can't open a committed blob file for writing:
>>> try:
... open(blob.committed(), 'w') # doctest: +ELLIPSIS
... except:
... # Produces IOError in Py2 and PermissionError in Py3
... print('Error raised.')
Error raised.
tpc_abort
---------
@ -277,25 +385,16 @@ tpc_abort
If a transaction is aborted in the middle of 2-phase commit, any data
stored are discarded.
>>> connection6 = database.open()
>>> root6 = connection6.root()
>>> blob = ZODB.blob.Blob()
>>> blob_fh = blob.open("w")
>>> blob_fh.write("I'm a happy blob.")
>>> blob_fh.close()
>>> root6['blob'] = blob
>>> transaction.commit()
>>> open(blob.committed()).read()
"I'm a happy blob."
>>> olddata, oldserial = blob_storage.load(blob._p_oid, '')
>>> t = transaction.get()
>>> blob_storage.tpc_begin(t)
>>> open('blobfile', 'w').write('This data should go away')
>>> with open('blobfile', 'wb') as file:
... _ = file.write(b'This data should go away')
>>> s1 = blob_storage.storeBlob(blob._p_oid, oldserial, olddata, 'blobfile',
... '', t)
>>> new_oid = blob_storage.new_oid()
>>> open('blobfile2', 'w').write('This data should go away too')
>>> with open('blobfile2', 'wb') as file:
... _ = file.write(b'This data should go away too')
>>> s2 = blob_storage.storeBlob(new_oid, '\0'*8, olddata, 'blobfile2',
... '', t)
@ -317,11 +416,12 @@ And we shouldn't be able to read the data that we saved:
>>> blob_storage.loadBlob(blob._p_oid, s1)
Traceback (most recent call last):
...
POSKeyError: 'No blob file'
POSKeyError: 'No blob file...
Of course the old data should be unaffected:
>>> open(blob_storage.loadBlob(blob._p_oid, oldserial)).read()
>>> with open(blob_storage.loadBlob(blob._p_oid, oldserial)) as fp:
... fp.read()
"I'm a happy blob."
Similarly, the new object wasn't added to the storage:
@ -334,7 +434,7 @@ Similarly, the new object wasn't added to the storage:
>>> blob_storage.loadBlob(blob._p_oid, s2)
Traceback (most recent call last):
...
POSKeyError: 'No blob file'
POSKeyError: 'No blob file...
.. clean up

65
relstorage/tests/blob/testblob.py
View File

@ -56,6 +56,7 @@ def new_time():
time.sleep(1)
return new_time
with open(__file__) as _f:
# Just use the this module as the source of our data
# Capture it at import time because test cases may
@ -64,6 +65,7 @@ with open(__file__) as _f:
_random_file_data = _f.read().replace('\n', '').split()
del _f
def random_file(size, fd):
"""Create a random data of at least the given size, writing to fd.
@ -138,19 +140,24 @@ class BlobUndoTests(BlobTestBase):
root = connection.root()
transaction.begin()
blob = Blob()
blob.open('w').write('this is state 1')
with blob.open('w') as f:
f.write('this is state 1')
root['blob'] = blob
transaction.commit()
transaction.begin()
blob = root['blob']
blob.open('w').write('this is state 2')
with blob.open('w') as f:
f.write('this is state 2')
transaction.commit()
database.undo(database.undoLog(0, 1)[0]['id'])
transaction.commit()
self.assertEqual(blob.open('r').read(), 'this is state 1')
with blob.open('r') as f:
data = f.read()
self.assertEqual(data, 'this is state 1')
database.close()
@ -159,7 +166,7 @@ class BlobUndoTests(BlobTestBase):
connection = database.open()
root = connection.root()
transaction.begin()
open('consume1', 'w').write('this is state 1')
with open('consume1', 'w') as f: f.write('this is state 1')
blob = Blob()
blob.consumeFile('consume1')
root['blob'] = blob
@ -167,14 +174,16 @@ class BlobUndoTests(BlobTestBase):
transaction.begin()
blob = root['blob']
open('consume2', 'w').write('this is state 2')
with open('consume2', 'w') as f: f.write('this is state 2')
blob.consumeFile('consume2')
transaction.commit()
database.undo(database.undoLog(0, 1)[0]['id'])
transaction.commit()
self.assertEqual(blob.open('r').read(), 'this is state 1')
with blob.open('r') as f:
data = f.read()
self.assertEqual(data, 'this is state 1')
database.close()
@ -185,13 +194,15 @@ class BlobUndoTests(BlobTestBase):
blob = Blob()
transaction.begin()
blob.open('w').write('this is state 1')
with blob.open('w') as f:
f.write('this is state 1')
root['blob'] = blob
transaction.commit()
transaction.begin()
blob = root['blob']
blob.open('w').write('this is state 2')
with blob.open('w') as f:
f.write('this is state 2')
transaction.commit()
database.undo(database.undoLog(0, 1)[0]['id'])
@ -213,7 +224,8 @@ class BlobUndoTests(BlobTestBase):
blob = Blob()
transaction.begin()
blob.open('w').write('this is state 1')
with blob.open('w') as f:
f.write('this is state 1')
root['blob'] = blob
transaction.commit()
@ -253,17 +265,20 @@ class RecoveryBlobStorage(BlobTestBase,
conn.root()[1] = ZODB.blob.Blob()
transaction.commit()
conn.root()[2] = ZODB.blob.Blob()
conn.root()[2].open('w').write('some data')
with conn.root()[2].open('w') as f:
f.write('some data')
transaction.commit()
conn.root()[3] = ZODB.blob.Blob()
conn.root()[3].open('w').write(
(''.join(struct.pack(">I", random.randint(0, (1<<32)-1))
for i in range(random.randint(10000,20000)))
)[:-random.randint(1,4)]
with conn.root()[3].open('w') as f:
f.write(
(''.join(struct.pack(">I", random.randint(0, (1<<32)-1))
for i in range(random.randint(10000,20000)))
)[:-random.randint(1,4)]
)
transaction.commit()
conn.root()[2] = ZODB.blob.Blob()
conn.root()[2].open('w').write('some other data')
with conn.root()[2].open('w') as f:
f.write('some other data')
transaction.commit()
self._dst.copyTransactionsFrom(self._storage)
self.compare(self._storage, self._dst)
@ -292,7 +307,9 @@ class LargeBlobTest(BlobTestBase):
blob = conn.root()[1] = ZODB.blob.Blob()
size = sizeof_fmt(self.testsize)
self._log('Creating %s blob file' % size)
signature = random_file(self.testsize, blob.open('w'))
blob_file = blob.open('w')
signature = random_file(self.testsize, blob_file)
blob_file.close()
self._log('Committing %s blob file' % size)
transaction.commit()
@ -305,9 +322,9 @@ class LargeBlobTest(BlobTestBase):
# Re-download blob
self._log('Caching %s blob file' % size)
conn = db.open()
blob = conn.root()[1].open('r')
self._log('Creating signature for %s blob cache' % size)
self.assertEqual(md5sum(blob), signature)
with conn.root()[1].open('r') as blob:
self._log('Creating signature for %s blob cache' % size)
self.assertEqual(md5sum(blob), signature)
def packing_with_uncommitted_data_non_undoing():
@ -428,7 +445,7 @@ def loadblob_tmpstore():
>>> from ZODB.blob import Blob
>>> root['blob'] = Blob()
>>> connection.add(root['blob'])
>>> root['blob'].open('w').write('test')
>>> with root['blob'].open('w') as f: _ = f.write('test')
>>> import transaction
>>> transaction.commit()
>>> blob_oid = root['blob']._p_oid
@ -465,7 +482,7 @@ def do_not_depend_on_cwd():
>>> db = DB(bs)
>>> conn = db.open()
>>> conn.root()['blob'] = ZODB.blob.Blob()
>>> conn.root()['blob'].open('w').write('data')
>>> with conn.root()['blob'].open('w') as f: _ = f.write('data')
>>> transaction.commit()
>>> os.chdir(here)
>>> conn.root()['blob'].open().read()
@ -515,7 +532,7 @@ def savepoint_cleanup():
>>> db = DB(bs)
>>> conn = db.open()
>>> conn.root().b = ZODB.blob.Blob()
>>> conn.root().b.open('w').write('initial')
>>> with conn.root().b.open('w') as f: _ = f.write('initial')
>>> _ = transaction.savepoint()
>>> len(os.listdir(tdir))
1
@ -524,9 +541,9 @@ def savepoint_cleanup():
>>> os.path.exists(savepoint_dir) and len(os.listdir(savepoint_dir)) > 0
False
>>> conn.root().b = ZODB.blob.Blob()
>>> conn.root().b.open('w').write('initial')
>>> with conn.root().b.open('w') as f: _ = f.write('initial')
>>> transaction.commit()
>>> conn.root().b.open('w').write('1')
>>> with conn.root().b.open('w') as f: _ = f.write('1')
>>> _ = transaction.savepoint()
>>> transaction.abort()
>>> os.path.exists(savepoint_dir) and len(os.listdir(savepoint_dir)) > 0

28
relstorage/tests/reltestbase.py
View File

@ -103,6 +103,29 @@ class GenericRelStorageTests(
):
def checkDropAndPrepare(self):
# XXX: Hangs with PyMySql; hangs dropping the object_state table,
# the 8th table to drop.
# XXX: Also hangs with psycopg2cffi
import sys
if sys.modules.get("MySQLdb") == sys.modules.get('pymysql', self) \
and 'MySQL' in str(type(self._storage._adapter.schema)):
try:
from unittest import SkipTest
raise SkipTest("PyMySQL hangs dropping a table.")
except ImportError:
# Py2.6; nothing to do but return
return
if sys.modules.get("psycopg2") == sys.modules.get("psycopg2cffi", self) \
and "PostgreSQL" in str(type(self._storage._adapter.schema)):
try:
from unittest import SkipTest
raise SkipTest("psycopg2cffi hangs dropping a table.")
except ImportError:
# Py2.6; nothing to do but return
return
self._storage._adapter.schema.drop_all()
self._storage._adapter.schema.prepare()
@ -628,9 +651,10 @@ class GenericRelStorageTests(
# Verify the pack stops with the right exception if it encounters
# a broken pickle.
# Under Python 2, with zodbpickle, there may be a difference depending
# on whether the accelerated implementation is in use.
# on whether the accelerated implementation is in use. Also ,the pure-python
# version on PyPy can raise IndexError
from zodbpickle.pickle import UnpicklingError as pUnpickErr
unpick_errs = (pUnpickErr,)
unpick_errs = (pUnpickErr,IndexError)
try:
from zodbpickle.fastpickle import UnpicklingError as fUnpickErr
except ImportError:

12
relstorage/tests/testmysql.py
View File

@ -146,10 +146,14 @@ db_names = {
def test_suite():
try:
import MySQLdb
except ImportError, e:
import warnings
warnings.warn("MySQLdb is not importable, so MySQL tests disabled")
return unittest.TestSuite()
except ImportError:
try:
import pymysql
pymysql.install_as_MySQLdb()
except ImportError:
import warnings
warnings.warn("MySQLdb is not importable, so MySQL tests disabled")
return unittest.TestSuite()
suite = unittest.TestSuite()
for klass in [

14
relstorage/tests/testpostgresql.py
View File

@ -136,11 +136,15 @@ db_names = {
def test_suite():
try:
import psycopg2
except ImportError, e:
import warnings
warnings.warn(
"psycopg2 is not importable, so PostgreSQL tests disabled")
return unittest.TestSuite()
except ImportError:
try:
from psycopg2cffi import compat
compat.register()
except ImportError:
import warnings
warnings.warn(
"psycopg2 is not importable, so PostgreSQL tests disabled")
return unittest.TestSuite()
suite = unittest.TestSuite()
for klass in [

3
relstorage/zodbconvert.py
View File

@ -112,7 +112,8 @@ def main(argv=sys.argv):
TimeStamp(txn.tid), txn.user, txn.description))
count += 1
log.info("Would copy %d transactions.", count)
source.close()
destination.close()
else:
if options.clear:
log.info("Clearing old data...")

24
setup.py
View File

@ -24,6 +24,8 @@ classifiers = """\
Intended Audience :: Developers
License :: OSI Approved :: Zope Public License
Programming Language :: Python
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy
Topic :: Database
Topic :: Software Development :: Libraries :: Python Modules
Operating System :: Microsoft :: Windows
@ -39,7 +41,9 @@ doclines = __doc__.split("\n")
def read_file(*path):
base_dir = os.path.dirname(__file__)
file_path = (base_dir, ) + tuple(path)
return file(os.path.join(*file_path)).read()
with open(os.path.join(*file_path)) as f:
result = f.read()
return result
setup(
name="RelStorage",
@ -81,9 +85,21 @@ setup(
'zope.testing',
],
extras_require={
'mysql': ['MySQL-python>=1.2.2'],
'postgresql': ['psycopg2>=2.0'],
'oracle': ['cx_Oracle>=4.3.1'],
'mysql:platform_python_implementation=="CPython"': [
'MySQL-python>=1.2.2',
],
'mysql:platform_python_implementation=="PyPy"' : [
'PyMySQL>=0.6.6',
],
'postgresql: platform_python_implementation == "CPython"': [
'psycopg2>=2.0',
],
'postgresql: platform_python_implementation == "PyPy"': [
'psycopg2cffi>=2.7.0',
],
'oracle': [
'cx_Oracle>=4.3.1'
],
},
entry_points = {
'console_scripts': [

23
tox.ini
View File

@ -1,8 +1,11 @@
[tox]
envlist = py27-mysql,py27-postgres
envlist = py27-mysql,py27-postgres,pypy-mysql,pypy-postgres
[testenv]
deps = coverage
deps =
coverage
mock
zope.testing
commands =
coverage run -m relstorage.tests.alltests
@ -19,3 +22,19 @@ deps =
{[testenv]deps}
commands =
{[testenv]commands}
# coverage is relatively quite slow on PyPy
[testenv:pypy-mysql]
deps =
PyMySQL
{[testenv]deps}
commands =
python -m relstorage.tests.alltests
[testenv:pypy-postgres]
deps =
psycopg2cffi
{[testenv]deps}
commands =
python -m relstorage.tests.alltests