dave
/
relstorage
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Doc updates. Bring in caching doc.

This commit is contained in:
Jason Madden 2016-12-20 10:32:12 -06:00
parent 174409775b
commit c081f272f4
No known key found for this signature in database
GPG Key ID: 349F84431A08B99E
9 changed files with 258 additions and 238 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
doc/cache-tracing.rst
View File

@ -73,70 +73,27 @@ configuration file. The default cache size is 10 MB, which is small.
additional overhead produces enough extra hits to be
worthwhile.
The cache_stats.py tool shows its command line syntax when invoked without
arguments. The tracefile argument can be a gzipped file if it has a .gz
extension. It will be read from stdin (assuming uncompressed data) if the
tracefile argument is '-'.::
The cache_stats.py tool (``python -m ZEO.scripts.cache_stats``) shows
its command line syntax when invoked without arguments. The tracefile
argument can be a gzipped file if it has a .gz extension. It will be
read from stdin (assuming uncompressed data) if the tracefile argument
is '-'.
$ python -m ZEO.scripts.cache_stats --help
usage: cache_stats.py [--verbose | --quiet] [--sizes] [--no-stats]
[--load-histogram] [--check] [--interval INTERVAL]
tracefile
Trace file statistics analyzer
positional arguments:
tracefile The trace to read; may be gzipped
optional arguments:
--verbose, -v Be verbose; print each record
--quiet, -q Reduce output; don't print summaries
--sizes, -s print histogram of object sizes
--no-stats, -S don't print statistics
--load-histogram, -h print histogram of object load frequencies
--check, -X enable heuristic checking for misaligned records: oids
> 2**32 will be rejected; this requires the tracefile
to be seekable
--interval INTERVAL, -i INTERVAL
summarizing interval in minutes (default 15;
max 60)
.. This is not ready yet .. program-output:: python -m
.. ZEO.scripts.cache_stats --help
.. program-output:: python -m ZEO.scripts.cache_stats --help
Simulating Different Cache Sizes
================================
Based on a cache trace file, you can make a prediction of how well the cache
might do with a different cache size. The cache_simul.py tool runs a simulation of
the ZEO client cache implementation based upon the events read from a trace
file. A new simulation is started each time the trace file records a client
restart event; if a trace file contains more than one restart event, a
separate line is printed for each simulation, and a line with overall
statistics is added at the end.::
Based on a cache trace file, you can make a prediction of how well the
cache might do with a different cache size. The cache_simul.py
(``python -m ZEO.scripts.cache_simul``) tool runs a simulation of the
ZEO client cache implementation based upon the events read from a
trace file. A new simulation is started each time the trace file
records a client restart event; if a trace file contains more than one
restart event, a separate line is printed for each simulation, and a
line with overall statistics is added at the end.
$ python -m ZEO.scripts.cache_simul --help
usage: cache_simul.py [-h] [--size CACHELIMIT] [--interval INTERVAL]
[--rearrange REARRANGE]
tracefile
Cache simulation. Note: - The simulation isn't perfect. - The simulation will
be far off if the trace file was created starting with a non-empty cache
positional arguments:
tracefile The trace to read; may be gzipped
optional arguments:
-h, --help show this help message and exit
--size CACHELIMIT, -s CACHELIMIT
cache size in MB (default 20MB)
--interval INTERVAL, -i INTERVAL
summarizing interval in minutes (default 15; max 60)
--rearrange REARRANGE, -r REARRANGE
rearrange factor
.. not ready yet
.. .. program-output:: python -m ZEO.scripts.cache_simul --help
.. program-output:: python -m ZEO.scripts.cache_simul --help
Example, assuming the trace file is in /tmp/cachetrace.log::

1
doc/contents.rst
View File

@ -5,6 +5,7 @@
Contents:
.. toctree::
:maxdepth: 2
install
configure-database

149
doc/developing.rst
View File

@ -5,149 +5,8 @@
Hacking on RelStorage should generally be done in a virtual
environment. Buildout may also be used.
Running Tests
=============
.. toctree::
:maxdepth: 1
RelStorage ships with an extensive test suite. It can be run from a
local checkout using tox::
tox -e py27-mysql
There are environments for each database and each Python version.
You can also run tests manually. Most test file contain a ``__main__``
block that can be used with Python's ``-m``::
python -m relstorage.tests.test_zodbconvert
Databases
---------
Many of RelStorage's test require access to a database. You will have
to have the appropriate database adapter (driver) installed to run
these tests; tests for which you don't have an adapter installed will
be skipped.
For example, to get started testing RelStorage against MySQL in a
virtual environment you could write::
pip install -e ".[mysql]"
from the root of the checkout. This sets up an editable installation
of relstorage complete with the correct MySQL driver.
Before actually running the tests, you need to create a test user
account and several databases. Use or adapt the SQL statements below
to create the databases.
.. highlight:: sql
PostgreSQL
~~~~~~~~~~
Execute the following using the ``psql`` command::
CREATE USER relstoragetest WITH PASSWORD 'relstoragetest';
CREATE DATABASE relstoragetest OWNER relstoragetest;
CREATE DATABASE relstoragetest2 OWNER relstoragetest;
CREATE DATABASE relstoragetest_hf OWNER relstoragetest;
CREATE DATABASE relstoragetest2_hf OWNER relstoragetest;
Also, add the following lines to the top of pg_hba.conf (if you put
them at the bottom, they may be overridden by other parameters)::
local relstoragetest relstoragetest md5
local relstoragetest2 relstoragetest md5
local relstoragetest_hf relstoragetest md5
local relstoragetest2_hf relstoragetest md5
host relstoragetest relstoragetest 127.0.0.1/32 md5
host relstoragetest_hf relstoragetest 127.0.0.1/32 md5
PostgreSQL specific tests can be run by the testposgresql module::
python -m relstorage.tests.testpostgresql
If the environment variable ``RS_PG_SMALL_BLOB`` is set when running
the tests, certain blob tests will use a much smaller size, making the
test run much faster.
MySQL
~~~~~
Execute the following using the ``mysql`` command::
CREATE USER 'relstoragetest'@'localhost' IDENTIFIED BY 'relstoragetest';
CREATE DATABASE relstoragetest;
GRANT ALL ON relstoragetest.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest2;
GRANT ALL ON relstoragetest2.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest_hf;
GRANT ALL ON relstoragetest_hf.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest2_hf;
GRANT ALL ON relstoragetest2_hf.* TO 'relstoragetest'@'localhost';
FLUSH PRIVILEGES;
MySQL specific tests can be run by the testmysql module::
python -m relstorage.tests.testmysql
.. note:: For some MySQL tests to pass (check16MObject), it may be
necessary to increase the server's ``max_allowed_packet``
setting. See the `MySQL Documentation
<http://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html>`_
for more information.
Oracle
~~~~~~
.. highlight:: shell
Initial setup will require ``SYS`` privileges. Using Oracle 10g XE, you
can start a ``SYS`` session with the following shell commands::
$ su - oracle
$ sqlplus / as sysdba
.. highlight:: sql
Using ``sqlplus`` with ``SYS`` privileges, execute the
following::
CREATE USER relstoragetest IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest;
CREATE USER relstoragetest2 IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest2;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest2;
CREATE USER relstoragetest_hf IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest_hf;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest_hf;
CREATE USER relstoragetest2_hf IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest2_hf;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest2_hf;
You may need to grant tablespace privileges if you get "no privileges
on tablespace" errors::
grant unlimited tablespace to relstoragetest;
grant unlimited tablespace to relstoragetest2;
grant unlimited tablespace to relstoragetest_hf;
grant unlimited tablespace to relstoragetest2_hf;
Oracle specific tests can be run by the testoracle module::
python -m relstorage.tests.testoracle
When running the tests, you can use the environment variable
ORACLE_TEST_DSN to override the data source name, which defaults to
"XE" (for Oracle 10g XE). For example, using Oracle's Developer Days
Virtual Box VM with an IP of 192.168.1.131, you might set
ORACLE_TEST_DSN to ``192.168.1.131/orcl``. (And you would connect as
sysdba with ``sqlplus 'sys/oracle@192.168.1.131/orcl' as sysdba``.)
If the environment variable ``RS_ORCL_SMALL_BLOB`` is set when running
the tests, certain blob tests will use a much smaller size, making the
test run much faster.
developing/running.rst
developing/caching.rst

33
notes/caching.txt → doc/developing/caching.rst
View File

@ -1,10 +1,10 @@
==========================
Caching with checkpoints
==========================
Caching with checkpoints
------------------------
The memcache strategy includes checkpoints. Checkpoint management is
a bit complex, but important for achieving a decent cache hit rate.
The caching strategy (both local in :class:`~relstorage.cache.storage_cache.StorageCache` and in
memcache) includes checkpoints. Checkpoint management is a bit
complex, but important for achieving a decent cache hit rate.
Checkpoints are 64 bit integer transaction IDs. Invariant: checkpoint0
is greater than or equal to checkpoint1, meaning that if they are
@ -31,15 +31,15 @@ immutable, so it would be nice to share it between threads.
When looking up an object in the cache, try to get:
- The state at delta_after0.
#. The state at delta_after0.
- The state at checkpoint0.
#. The state at checkpoint0.
- The state at delta_after1.
#. The state at delta_after1.
- The state at checkpoint1.
#. The state at checkpoint1.
- The state from the database.
#. The state from the database.
If the retrieved state is older than checkpoint0, but it
was not retrieved from checkpoint0, cache it at checkpoint0.
@ -70,3 +70,14 @@ adding data very quickly. If that happens, the instance should
ignore the checkpoint update, with the expectation that the new checkpoint
will be visible after the next update.
.. toctree::
../relstorage.cache
../relstorage.cache.interfaces
../relstorage.cache.cache_ring
../relstorage.cache.local_client
../relstorage.cache.mapping
../relstorage.cache.persistence
../relstorage.cache.storage_cache
../relstorage.cache.trace

175
doc/developing/running.rst Normal file
View File

@ -0,0 +1,175 @@
===============
Running Tests
===============
RelStorage ships with an extensive test suite. It can be run from a
local checkout using tox::
tox -e py27-mysql
There are environments for each database and each Python version.
You can also run tests manually. Most test file contain a ``__main__``
block that can be used with Python's ``-m``::
python -m relstorage.tests.test_zodbconvert
There is one file that will run all tests::
python -m relstorage.tests.alltests
If not database drivers are installed, it will only run the unit
tests. Otherwise, tests for all installed drivers will be attempted;
this can be bypassed with ``--no-db`` or limited to particular
databases with ``--only-[mysql|pgsql|oracle]``.
Databases
=========
Many of RelStorage's test require access to a database. You will have
to have the appropriate database adapter (driver) installed to run
these tests; tests for which you don't have an adapter installed will
be skipped.
For example, to get started testing RelStorage against MySQL in a
virtual environment you could write::
pip install -e ".[mysql]"
from the root of the checkout. This sets up an editable installation
of relstorage complete with the correct MySQL driver.
Before actually running the tests, you need to create a test user
account and several databases. Use or adapt the SQL statements below
to create the databases. (You can also see example scripts that are
used to set up the continuous integration test environment in the
`.travis <https://github.com/zodb/relstorage/tree/master/.travis>`_ directory.)
.. highlight:: sql
PostgreSQL
----------
Execute the following using the ``psql`` command::
CREATE USER relstoragetest WITH PASSWORD 'relstoragetest';
CREATE DATABASE relstoragetest OWNER relstoragetest;
CREATE DATABASE relstoragetest2 OWNER relstoragetest;
CREATE DATABASE relstoragetest_hf OWNER relstoragetest;
CREATE DATABASE relstoragetest2_hf OWNER relstoragetest;
Also, add the following lines to the top of pg_hba.conf (if you put
them at the bottom, they may be overridden by other parameters)::
local relstoragetest relstoragetest md5
local relstoragetest2 relstoragetest md5
local relstoragetest_hf relstoragetest md5
local relstoragetest2_hf relstoragetest md5
host relstoragetest relstoragetest 127.0.0.1/32 md5
host relstoragetest_hf relstoragetest 127.0.0.1/32 md5
PostgreSQL specific tests can be run by the testposgresql module::
python -m relstorage.tests.testpostgresql
If the environment variable ``RS_PG_SMALL_BLOB`` is set when running
the tests, certain blob tests will use a much smaller size, making the
test run much faster.
MySQL
-----
Execute the following using the ``mysql`` command::
CREATE USER 'relstoragetest'@'localhost' IDENTIFIED BY 'relstoragetest';
CREATE DATABASE relstoragetest;
GRANT ALL ON relstoragetest.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest2;
GRANT ALL ON relstoragetest2.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest_hf;
GRANT ALL ON relstoragetest_hf.* TO 'relstoragetest'@'localhost';
CREATE DATABASE relstoragetest2_hf;
GRANT ALL ON relstoragetest2_hf.* TO 'relstoragetest'@'localhost';
FLUSH PRIVILEGES;
MySQL specific tests can be run by the testmysql module::
python -m relstorage.tests.testmysql
.. note:: For some MySQL tests to pass (check16MObject), it may be
necessary to increase the server's ``max_allowed_packet``
setting. See the `MySQL Documentation
<http://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html>`_
for more information.
Oracle
------
.. highlight:: shell
Initial setup will require ``SYS`` privileges. Using Oracle 10g XE, you
can start a ``SYS`` session with the following shell commands::
$ su - oracle
$ sqlplus / as sysdba
.. highlight:: sql
Using ``sqlplus`` with ``SYS`` privileges, execute the
following::
CREATE USER relstoragetest IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest;
CREATE USER relstoragetest2 IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest2;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest2;
CREATE USER relstoragetest_hf IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest_hf;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest_hf;
CREATE USER relstoragetest2_hf IDENTIFIED BY relstoragetest;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO relstoragetest2_hf;
GRANT EXECUTE ON DBMS_LOCK TO relstoragetest2_hf;
You may need to grant tablespace privileges if you get "no privileges
on tablespace" errors::
grant unlimited tablespace to relstoragetest;
grant unlimited tablespace to relstoragetest2;
grant unlimited tablespace to relstoragetest_hf;
grant unlimited tablespace to relstoragetest2_hf;
Oracle specific tests can be run by the testoracle module::
python -m relstorage.tests.testoracle
When running the tests, you can use the environment variable
ORACLE_TEST_DSN to override the data source name, which defaults to
"XE" (for Oracle 10g XE). For example, using Oracle's Developer Days
Virtual Box VM with an IP of 192.168.1.131, you might set
ORACLE_TEST_DSN to ``192.168.1.131/orcl``. (And you would connect as
sysdba with ``sqlplus 'sys/oracle@192.168.1.131/orcl' as sysdba``.)
If the environment variable ``RS_ORCL_SMALL_BLOB`` is set when running
the tests, certain blob tests will use a much smaller size, making the
test run much faster.
Docs:
http://www.oracle.com/pls/db102/homepage
Excellent setup instructions:
http://www.davidpashley.com/articles/oracle-install.html
Work around session limit (fixes ORA-12520)::
ALTER SYSTEM SET PROCESSES=150 SCOPE=SPFILE;
ALTER SYSTEM SET SESSIONS=150 SCOPE=SPFILE;
(then restart Oracle)
Manually rollback an in-dispute transaction::
select local_tran_id, state from DBA_2PC_PENDING;
rollback force '$local_tran_id';

2
doc/faq.rst
View File

@ -35,4 +35,4 @@ Q: Can RelStorage replace ZRS (Zope Replication Services)?
Q: How do I set up an environment to run the RelStorage tests?
A: See README.txt in the relstorage/tests directory.
A: See :doc:`developing`.

18
doc/generate_rst.py
View File

@ -22,18 +22,26 @@ template = '''.. AUTOGENERATED -- will be overwritten (remove this comment to sa
directory = dirname(abspath(relstorage.__file__))
print('Imported relstorage from %s' % (directory, ))
modules = glob.glob(join(directory, '*.py')) + glob.glob(join(directory, '*.pyc'))
modules = glob.glob(join(directory, '*.py'))
modules = set(basename(filename).split('.')[0] for filename in modules)
adap_mods = {'adapters.' + basename(x).split('.')[0] for x in glob.glob(join(directory, 'adapters', '*.py'))
if not x.startswith("_")}
adap_mods = {'adapters.' + basename(x).split('.')[0]
for x in glob.glob(join(directory, 'adapters', '*.py'))
if not basename(x).startswith("_")}
modules = modules.union(adap_mods)
for driver in 'mysql', 'oracle', 'postgresql':
adap_mods = {'adapters.' + driver + '.' + basename(x).split('.')[0]
for x in glob.glob(join(directory, 'adapters', driver, '*.py'))
if not x.startswith("_")}
if not basename(x).startswith("_")}
modules = modules.union(adap_mods)
cache_mods = {'cache.' + basename(x).split('.')[0]
for x in glob.glob(join(directory, 'cache', '*.py'))
if not basename(x).startswith("_")}
modules = modules.union(cache_mods)
modules = set(name for name in modules
if not name.startswith('_'))
@ -98,7 +106,7 @@ def generate_rst(do=True):
def iter_autogenerated():
for module in modules:
for module in sorted(modules):
rst_filename = 'relstorage.%s.rst' % module
exists = os.path.exists(rst_filename)
if exists:

30
doc/index.rst
View File

@ -35,6 +35,12 @@ Features
See :doc:`the rest of the documentation <contents>` for more information.
.. toctree::
:maxdepth: 2
contents
Development
===========
@ -56,6 +62,27 @@ Windows.
.. image:: https://ci.appveyor.com/api/projects/status/pccddlgujdoqvl83?svg=true
:target: https://ci.appveyor.com/project/jamadden/relstorage/branch/master
.. image:: https://readthedocs.org/projects/relstorage/badge/?version=latest
:target: http://relstorage.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
Builds on Travis CI automatically submit updates to `coveralls.io`_ to
monitor test coverage.
.. image:: https://coveralls.io/repos/zodb/relstorage/badge.svg?branch=master&service=github
:target: https://coveralls.io/github/zodb/relstorage?branch=master
Likewise, builds on Travis CI will automatically submit updates to
`landscape.io`_ to monitor code health (adherence to PEP8, absence of
common code smells, etc).
.. image:: https://landscape.io/github/zodb/relstorage/master/landscape.svg?style=flat
:target: https://landscape.io/github/zodb/relstorage/master
:alt: Code Health
.. _coveralls.io: https://coveralls.io/github/zodb/relstorage
.. _landscape.io: https://landscape.io/github/zodb/relstorage
Project URLs
============
@ -63,9 +90,6 @@ Project URLs
* http://pypi.python.org/pypi/RelStorage (PyPI entry and downloads)
* http://shane.willowrise.com/ (blog)
.. toctree::
contents
Indices and tables
==================

15
notes/oracle_notes.txt
View File

@ -1,15 +0,0 @@
Docs:
http://www.oracle.com/pls/db102/homepage
Excellent setup instructions:
http://www.davidpashley.com/articles/oracle-install.html
Work around session limit (fixes ORA-12520):
ALTER SYSTEM SET PROCESSES=150 SCOPE=SPFILE;
ALTER SYSTEM SET SESSIONS=150 SCOPE=SPFILE;
(then restart Oracle)
Manually rollback an in-dispute transaction:
select local_tran_id, state from DBA_2PC_PENDING;
rollback force '$local_tran_id';