When we updated PyPy on Travis to 5.8.0, it started producing errors
"Use multi-True when executing multiple statements", even when we are
demonstrably only executing one statement. E.g.,
https://travis-ci.org/zodb/relstorage/jobs/335720512#L923
If this doesn't fix it, then I plan to take PyPy+MySQL Connector out
of the test matrix.
RelStorage storages to be notified of transaction endings for
transactions that don't call the two-phase commit API. This allows
resources to be used more efficiently because it prevents RDBMS
transactions from being held open.
Fixes: https://github.com/zodb/relstorage/issues/147
(At least for ZODB 5.2.)
And truncate them.
For most things this seems to speed up writes by around 10% or more due
to the way the temp tables are allocated.
Tested on both 5.5 and 5.7.
I think the dropping was an attempt to make the tables statement-level
replication compatible, but since they weren't transactianal I don't
think that worked. They were also variable sized so I don't think there
were any speed benefits. It also didn't implicitly end the transaction,
but that's not a concern where this is called.
Here's a complete set of benchmarks against 5.7, comparing zodbshootout
with -c2 and -c6, -n 1000 and '-n 100 -s 256 --test-reps 200'. This
compares 2.0.0 with current master and this change, in that order.
** c=2,s=256 ** 2.0.0 master InnoDB
"Transaction", mysql_hf
"Add 100 Objects", 8883 9199 10157
"Update 100 Objects", 9380 9169 10318
** c=6,s=256 ** 2.0.0 master InnoDB
"Transaction", mysql_hf
"Add 100 Objects", ---- 14913 15524
"Update 100 Objects", ---- 14914 15077
** c=6,s=128 ** 2.0.0 master InnoDB
"Transaction", mysql_hf
"Add 1000 Objects", 29815 31704 38506
"Update 1000 Objects", 28978 29030 29137
In all cases, InnoDB temp tables outperform, in some cases
substantially.
An earlier set of benchmarks taken against 5.5 (but prior to a rebase on
master, so not easily replicated). The absolute value difference in
numbers versus above is that these were taken with Py 3.4 while above
was py 2.7.
zodbshootout -c 2 shootout.conf -n 100 --test-reps 200 -r 1 -s 256
Before
** concurrency=2 **
"Transaction", mysql_hf
"Add 100 Objects", 6493
"Update 100 Objects", 9636
"Read 100 Warm Objects", 9797
"Read 100 Cold Objects", 9336
"Read 100 Hot Objects", 37407
"Read 100 Steamin' Objects", 921761
** concurrency=6 **
"Transaction", mysql_hf
"Add 100 Objects", 10574
"Update 100 Objects", 15492
"Read 100 Warm Objects", 19290
"Read 100 Cold Objects", 17051
"Read 100 Hot Objects", 72304
"Read 100 Steamin' Objects", 1590488
After
** concurrency=2 **
"Transaction", mysql_hf
"Add 100 Objects", 7079
"Update 100 Objects", 9782
"Read 100 Warm Objects", 8726
"Read 100 Cold Objects", 9834
"Read 100 Hot Objects", 40004
"Read 100 Steamin' Objects", 937232
c=6
** concurrency=6 **
"Transaction", mysql_hf
"Add 100 Objects", 10612
"Update 100 Objects", 15789
"Read 100 Warm Objects", 17501
"Read 100 Cold Objects", 16556
"Read 100 Hot Objects", 71868
"Read 100 Steamin' Objects", 1549534
zodbshootout -c 2 shootout.conf -n 100 --test-reps 200 -r 1 -s 96
** concurrency=2 **
"Transaction", mysql_hf
"Add 100 Objects", 15547
"Update 100 Objects", 14337
"Read 100 Warm Objects", 12357
"Read 100 Cold Objects", 13686
"Read 100 Hot Objects", 54559
"Read 100 Steamin' Objects", 945789
"Transaction", mysql_hf
"Add 100 Objects", 16968
"Update 100 Objects", 16184
"Read 100 Warm Objects", 13430
"Read 100 Cold Objects", 13557
"Read 100 Hot Objects", 54625
"Read 100 Steamin' Objects", 966802
* format tweaks [skip ci]
* MySQL: Drop the commit lock after fill_object_refs
In pre-pack gc in a history-free storage.
Oracle and PostgreSQL were already doing this, so this should be a safe
change to make.
Fixes#9.
* Tweak error message.
For MySQL and PostgreSQL only. MySQL uses the MAX query, and PostgreSQL
uses its prepared statement.
Also avoid checking self.keep_history at runtime, overriding the correct
methods dynamically.
cf #89
* First basic version of appveyor.yml for testing. Still need to configure databases. Lets just see if we build.
* stdint.h is not available on MSVC before 2010, but we're not using it anyway. myssqlclient won't compile on windows
* Try a define for broken c99 support in MSVC.
* Broader macro.
* Use old form of 0 initializing struct.
* More fixes for broken c99 support of MSVC.
* Attempt to setup the database.
* need /c
* give up on running the DB tests.
* produce wheels
* produce wheels
* one more try for mysql tests.
* Run the db script as part of the test, it seems services aren't started during install phase.
* MySQL tests are running, try adding postgres.
* badges
* right bin directory for postgres
* Skip one blob test on windows/py3
Take more care with exception handling and being extra sure to clean up
temporary files.
Refactor to make this more obvious.
In addition, don't open files unless we're going to read them, not just
to stat them. This may open some small race conditions, but we should be
robust to them. It makes it more likely that this process will work on
Windows. It does use more system calls, but this shouldn't be a
bottleneck for reasonably sized cache directories.
Let the system generate cache file names for us, and replace old files
instead of renaming on top of them. This should be less racy overall.
This only happened when the cache entries were sized in such a way that
when a new entry could fit we would still like to reject an old entry
from a different ring.
Fixes#140.
Jason Madden
Jim Fulton
Jason Madden