Performance docs, moved from notes. [skip ci]

doc/contents.rst

@@ -17,6 +17,7 @@ Contents:
    zodburi
    cache-tracing
    faq
+   performance
    migration
    developing
    changelog

doc/faq.rst

@@ -20,7 +20,8 @@ Q: Can I perform SQL queries on the data in the database?
 Q: How does RelStorage performance compare with FileStorage?
 
     A: According to benchmarks, RelStorage with PostgreSQL is often faster than
-    FileStorage, especially under high concurrency.
+    FileStorage, especially under high concurrency. See
+    :doc:`performance` for more.
 
 Q: Why should I choose RelStorage?
 

doc/performance.rst

@@ -0,0 +1,124 @@
+=============
+ Performance
+=============
+
+This document captures various performance metrics, comparing
+RelStorage to other ZODB storages at given points in time. The numbers
+given here cannot be compared to each other outside their own test.
+
+
+RelStorage 2.0
+==============
+
+All RelStorage schemas were history free and used a memcache instance.
+ZEO was version 5.0.4 and used a FileStorage backend from ZODB 5.1.1
+MySQL was version 5.5.53 and PostgreSQL was version 9.6.1. The
+database drivers were mysqlclient-1.3.9 and psycopg2 2.6.2. All cache
+settings were at their default values.
+
+The test suite was zodbshootout 0.6 with a concurrency level of 2 (and
+all other options the default), running on CPython 2.7.12 on a 2.5Ghz
+Intel Core i7 (MacBookPro11,5) under macOS 10.12.2.
+
+.. objects_per_txn=1000, object_size=128, mappingtype=<class 'persistent.mapping.PersistentMapping'> and concurrency=2 (threads? False)
+
+======================  ==========  =====  =====
+Transaction             PostgreSQL  MySQL	ZEO
+======================  ==========  =====  =====
+Add 1000 Objects        19534       23184   8152
+Update 1000 Objects     17140       23790   7122
+Read 1000 Warm Objects  14232       16649   4796
+Read 1000 Cold Objects  15595       18070   4860
+Read 1000 Hot Objects   86326       88422  82600
+======================  ==========  =====  =====
+
+.. image:: perf-rs20-no-mem.png
+
+Memcache
+--------
+
+This is exactly the same configuration as above, but shows the effect
+of introducing a local memcache instance. The "warm" test performs
+better, but the other tests perform worse.
+
+======================  ==========  =====  =====
+Transaction             PostgreSQL  MySQL	ZEO
+======================  ==========  =====  =====
+Add 1000 Objects          14789     17378   8220
+Update 1000 Objects       13834     17711   7021
+Read 1000 Warm Objects    27394     27978   4662
+Read 1000 Cold Objects     7621      8415   4344
+Read 1000 Hot Objects     86165     83438  76540
+======================  ==========  =====  =====
+
+.. image:: perf-20.png
+
+PyPy
+----
+
+This is the same as the second configuration, except we're using PyPy
+5.6.0 for everything (including running the ZEO server). The database
+drivers were PyMySQL-0.7.9  and psycopg2cffi-2.7.5.
+
+======================  ========== ======  =====
+Transaction             PostgreSQL  MySQL	ZEO
+======================  ========== ======  =====
+Add 1000 Objects            15235  13849   4681
+Update 1000 Objects         18522  25939   4853
+Read 1000 Warm Object       15138  964041  23843
+Read 1000 Cold Object       11974   9876   3360
+Read 1000 Hot Objects       99241  81685   25552
+======================  ========== ======  =====
+
+.. image:: perf-rs20pypy.png
+
+Under PyPy, the benchmark was run in shared threads mode (``--threads
+shared``) instead of separate processes, to allow PyPy's JIT to warm
+up. Under CPython, shared threads for concurrency two generally
+*reduces* performance (with the exception of one test), but under PyPy
+it substantially improves performance.
+
+For comparison, here's the shared thread results for CPython:
+
+======================  ========== ======  =====
+Transaction             PostgreSQL  MySQL	ZEO
+======================  ========== ======  =====
+Add 1000 Objects             13409  14081  4852
+Update 1000 Objects          11541  14310  4496
+Read 1000 Warm Objects       42646  43699  123079
+Read 1000 Cold Objects       10253   7712  3497
+Read 1000 Hot Objects        24973  26129  25049
+======================  ========== ======  =====
+
+
+RelStorage 1.6.1 vs RelStorage 2.0
+----------------------------------
+
+These tests are run using the same configuration (databases, database
+drivers, CPython version and same machine) as above, to show any
+performance differences between RelStorage 2.0 and RelStorage 1.6.1
+(running with ZODB 4.4.4 and ZEO 4.3.1).
+
+.. image:: perf-rs20-rs16.png
+
+RelStorage 1.4.0b1
+==================
+
+All RelStorage schemas were history free and used a memcache instance.
+ZEO was version 3.9.3 and used a FileStorage backend.
+
+The test suite was zodbshootout 0.2 with a concurrency level of 2,
+running on Python 2.6 on a 2.1Ghz Intel Core 2 Duo (T8100)
+
+
+======================  ==========  =====  =====
+Transaction             PostgreSQL  MySQL	ZEO
+======================  ==========  =====  =====
+Add 1000 Objects        9177        12711   5044
+Update 1000 Objects     10040        9431   4017
+Read 1000 Warm Objects  20897       22310   1924
+Read 1000 Cold Objects  6591         5853   1932
+Read 1000 Hot Objects   36847       37950  36545
+======================  ==========  =====  =====
+
+.. image:: perf-rs14.png

notes/quicktest.py

@@ -1,19 +0,0 @@
-import logging
-import sys
-format = '%(asctime)s [%(name)s] %(levelname)s %(message)s'
-logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format=format)
-import transaction
-from relstorage.storage import RelStorage
-from relstorage.options import Options
-from relstorage.adapters.mysql import MySQLAdapter
-from ZODB.DB import DB
-options = Options()
-adapter = MySQLAdapter(db='shane', options=options)
-storage = RelStorage(adapter, options=options)
-db = DB(storage)
-conn = db.open()
-root = conn.root()
-root['x'] = root.get('x', 0) + 1
-transaction.commit()
-conn.close()
-db.pack()