Test on Appveyor (#145)

* First basic version of appveyor.yml for testing. Still need to configure databases. Lets just see if we build. * stdint.h is not available on MSVC before 2010, but we're not using it anyway. myssqlclient won't compile on windows * Try a define for broken c99 support in MSVC. * Broader macro. * Use old form of 0 initializing struct. * More fixes for broken c99 support of MSVC. * Attempt to setup the database. * need /c * give up on running the DB tests. * produce wheels * produce wheels * one more try for mysql tests. * Run the db script as part of the test, it seems services aren't started during install phase. * MySQL tests are running, try adding postgres. * badges * right bin directory for postgres * Skip one blob test on windows/py3
2016-12-11 11:06:27 -06:00 · 2016-12-11 11:06:27 -06:00 · 402c965b9f
parent 4bf54e0954
commit 402c965b9f
9 changed files with 174 additions and 40 deletions
--- a/.travis/mysql.cmd
+++ b/.travis/mysql.cmd
@ -0,0 +1,10 @@
+mysql -uroot -e "CREATE USER 'relstoragetest'@'localhost' IDENTIFIED BY 'relstoragetest';"
+mysql -uroot -e "CREATE DATABASE relstoragetest;"
+mysql -uroot -e "GRANT ALL ON relstoragetest.* TO 'relstoragetest'@'localhost';"
+mysql -uroot -e "CREATE DATABASE relstoragetest2;"
+mysql -uroot -e "GRANT ALL ON relstoragetest2.* TO 'relstoragetest'@'localhost';"
+mysql -uroot -e "CREATE DATABASE relstoragetest_hf;"
+mysql -uroot -e "GRANT ALL ON relstoragetest_hf.* TO 'relstoragetest'@'localhost';"
+mysql -uroot -e "CREATE DATABASE relstoragetest2_hf;"
+mysql -uroot -e "GRANT ALL ON relstoragetest2_hf.* TO 'relstoragetest'@'localhost';"
+mysql -uroot -e "FLUSH PRIVILEGES;"
--- a/.travis/postgres.cmd
+++ b/.travis/postgres.cmd
@ -0,0 +1,5 @@
+psql -U postgres -c "CREATE USER relstoragetest WITH PASSWORD 'relstoragetest';"
+psql -U postgres -c "CREATE DATABASE relstoragetest OWNER relstoragetest;"
+psql -U postgres -c "CREATE DATABASE relstoragetest2 OWNER relstoragetest;"
+psql -U postgres -c "CREATE DATABASE relstoragetest_hf OWNER relstoragetest;"
+psql -U postgres -c "CREATE DATABASE relstoragetest2_hf OWNER relstoragetest;"
--- a/CHANGES.rst
+++ b/CHANGES.rst
@ -10,7 +10,8 @@
  of stale temporary files remaining. Also, files are kept open for a
  shorter period of time and removed in a way that should work better
  on Windows.
-
+- RelStorage is now tested on Windows for MySQL and PostgreSQL thanks
+  to AppVeyor.

 2.0.0b9 (2016-11-29)
 ====================
--- a/README.rst
+++ b/README.rst
@ -30,11 +30,14 @@ replaced the PGStorage project.
 Documentation
 ===============

-`Documentation`_ including `installation instructions`_ is hosted on `readthedocs`_.
+Documentation including `installation instructions`_ is hosted on `readthedocs`_.

 The complete `changelog`_ is also there.

-.. _`Documentation`: http://relstorage.readthedocs.io/en/latest/
+.. image:: https://readthedocs.org/projects/relstorage/badge/?version=latest
+     :target: http://relstorage.readthedocs.io/en/latest/?badge=latest
+
+
 .. _`installation instructions`: http://relstorage.readthedocs.io/en/latest/install.html
 .. _`readthedocs`: http://relstorage.readthedocs.io/en/latest/
 .. _`changelog`: http://relstorage.readthedocs.io/en/latest/changelog.html
@ -47,3 +50,33 @@ The complete `changelog`_ is also there.
 RelStorage is hosted at GitHub:

    https://github.com/zodb/relstorage
+
+Continuous integration
+----------------------
+
+A test suite is run for every push and pull request submitted. Travis
+CI is used to test on Linux, and AppVeyor runs the builds on
+Windows.
+
+.. image:: https://travis-ci.org/zodb/relstorage.svg?branch=master
+    :target: https://travis-ci.org/zodb/relstorage
+
+.. image:: https://ci.appveyor.com/api/projects/status/pccddlgujdoqvl83?svg=true
+   :target: https://ci.appveyor.com/project/jamadden/relstorage/branch/master
+
+Builds on Travis CI automatically submit updates to `coveralls.io`_ to
+monitor test coverage.
+
+.. image:: https://coveralls.io/repos/zodb/relstorage/badge.svg?branch=master&service=github
+   :target: https://coveralls.io/github/zodb/relstorage?branch=master
+
+Likewise, builds on Travis CI will automatically submit updates to
+`landscape.io`_ to monitor code health (adherence to PEP8, absence of
+common code smells, etc).
+
+.. image:: https://landscape.io/github/zodb/relstorage/master/landscape.svg?style=flat
+   :target: https://landscape.io/github/zodb/relstorage/master
+   :alt: Code Health
+
+.. _coveralls.io: https://coveralls.io/github/zodb/relstorage
+.. _landscape.io: https://landscape.io/github/zodb/relstorage
--- a/appveyor.yml
+++ b/appveyor.yml
@ -0,0 +1,38 @@
+environment:
+
+  matrix:
+    - python : 27
+    - python : 34
+    - python : 35
+    - python : 27-x64
+    - python : 34-x64
+    - python : 35-x64
+
+services:
+  - mysql
+  - postgresql
+
+install:
+  - "SET PATH=C:\\Program Files\\PostgreSQL\\9.4\\bin;C:\\Program Files\\MySql\\MySQL Server 5.7\\bin;C:\\Python%PYTHON%;c:\\Python%PYTHON%\\scripts;%PATH%"
+  - "SET MYSQL_PWD=Password12!"
+  - "SET PGPASSWORD=Password12!"
+  - "SET PGUSER=postgres"
+  - echo "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 > "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\vcvars64.bat"
+  # mysqlclient won't compile on windows
+  - pip install -U pip setuptools
+  - pip install -U PyMySQL
+  - pip install -U wheel
+  - pip install -U -e .[test,postgresql]
+
+build: false
+
+test_script:
+  - cmd /c .travis\mysql.cmd
+  - cmd /c .travis\postgres.cmd
+  - python -m relstorage.tests.alltests
+
+after_test:
+  - python setup.py bdist_wheel
+
+artifacts:
+  - path: dist\*
--- a/doc/index.rst
+++ b/doc/index.rst
@ -43,6 +43,19 @@ RelStorage is hosted at GitHub:
    https://github.com/zodb/relstorage


+Continuous integration
+----------------------
+
+A test suite is run for every push and pull request submitted. Travis
+CI is used to test on Linux, and AppVeyor runs the builds on
+Windows.
+
+.. image:: https://travis-ci.org/zodb/relstorage.svg?branch=master
+    :target: https://travis-ci.org/zodb/relstorage
+
+.. image:: https://ci.appveyor.com/api/projects/status/pccddlgujdoqvl83?svg=true
+   :target: https://ci.appveyor.com/project/jamadden/relstorage/branch/master
+

 Project URLs
 ============
--- a/relstorage/adapters/packundo.py
+++ b/relstorage/adapters/packundo.py
@ -77,7 +77,8 @@ class PackUndo(object):
        # Download the list of object references into the TreeMarker.

        # Note the Oracle optimizer hints in the following statement; MySQL
-        # and PostgreSQL ignore these. Oracle fails to notice that pack_object
+        # and PostgreSQL ignore these (MySQL 5.7, though, emits a warning).
+        # Oracle fails to notice that pack_object
        # is now filled and chooses the wrong execution plan, completely
        # killing this query on large RelStorage databases, unless these hints
        # are included.
--- a/relstorage/cache/cache_ring.c
+++ b/relstorage/cache/cache_ring.c
@ -29,38 +29,48 @@ starting with the most recently used object.

 #include <stddef.h>
 #include <assert.h>
-#include <stdint.h>
+
 #ifndef __RING_H
 #include "cache_ring.h"
 #endif

+/*
+ * No version of MSVC properly supports inline. Sigh.
+ */
+#ifdef _MSC_VER
+#define RSR_SINLINE static
+#define RSR_INLINE
+#else
+#define RSR_SINLINE static inline
+#define RSR_INLINE inline
+#endif

-static inline int ring_oversize(RSRing ring)
+RSR_SINLINE int ring_oversize(RSRing ring)
 {
    return ring->u.head.sum_weights > ring->u.head.max_weight;
 }

-static inline int ring_is_empty(RSRing ring)
+RSR_SINLINE int ring_is_empty(RSRing ring)
 {
    return ring == NULL || ring->r_next == ring || ring->r_next == NULL;
 }

-static inline int cache_oversize(RSCache* cache)
+RSR_SINLINE int cache_oversize(RSCache* cache)
 {
    return ring_oversize(cache->eden) && ring_oversize(cache->probation) && ring_oversize(cache->protected);
 }

-static inline int lru_will_fit(RSRingNode* ring, RSRingNode* entry)
+RSR_SINLINE int lru_will_fit(RSRingNode* ring, RSRingNode* entry)
 {
    return ring->u.head.max_weight >= (entry->u.entry.weight + ring->u.head.sum_weights);
 }

-static inline int cache_will_fit(RSCache* cache, RSRingNode* entry)
+RSR_SINLINE int cache_will_fit(RSCache* cache, RSRingNode* entry)
 {
    return lru_will_fit(cache->eden, entry) || lru_will_fit(cache->probation, entry) || lru_will_fit(cache->protected, entry);
 }

-inline void
+RSR_INLINE void
 rsc_ring_add(RSRing ring, RSRingNode *elt)
 {
    elt->r_next = ring;
@ -74,11 +84,12 @@ rsc_ring_add(RSRing ring, RSRingNode *elt)

 }

-inline void
+RSR_INLINE void
 rsc_ring_del(RSRing ring, RSRingNode *elt)
 {
-    if( elt->r_next == NULL && elt->r_prev == NULL)
+    if( elt->r_next == NULL && elt->r_prev == NULL) {
        return;
+    }

    elt->r_next->r_prev = elt->r_prev;
    elt->r_prev->r_next = elt->r_next;
@ -92,7 +103,7 @@ rsc_ring_del(RSRing ring, RSRingNode *elt)
    ring->u.head.sum_weights -= elt->u.entry.weight;
 }

-inline void
+RSR_INLINE void
 rsc_ring_move_to_head(RSRing ring, RSRingNode *elt)
 {
    elt->r_prev->r_next = elt->r_next;
@ -103,7 +114,7 @@ rsc_ring_move_to_head(RSRing ring, RSRingNode *elt)
    ring->r_prev = elt;
 }

-static inline int
+RSR_SINLINE int
 ring_move_to_head_from_foreign(RSRing current_ring,
                               RSRing new_ring,
                               RSRingNode* elt)
@ -128,7 +139,7 @@ ring_move_to_head_from_foreign(RSRing current_ring,
    return new_ring->u.head.sum_weights > new_ring->u.head.max_weight;
 }

-static inline RSRingNode* ring_lru(RSRing ring)
+RSR_SINLINE RSRingNode* ring_lru(RSRing ring)
 {
    if(ring->r_next == ring) {
        // empty list
@ -166,14 +177,17 @@ void rsc_on_hit(RSRing ring, RSRingNode* entry)
 #define _SPILL_FIT 0
 #define _SPILL_VICTIMS 1
 #define _SPILL_NO_VICTIMS 0
-static inline
+RSR_SINLINE
 RSRingNode _spill_from_ring_to_ring(RSRing updated_ring,
                                    RSRing destination_ring,
                                    RSRingNode* ignore_me,
                                    int allow_victims,
                                    int overfill_destination)
 {
-    RSRingNode rejects = {};
+    RSRingNode* eden_oldest = NULL;
+    RSRingNode* probation_oldest = NULL;
+    RSRingNode rejects = {0};
+
    if(overfill_destination) {
        rejects.r_next = rejects.r_prev = NULL;
    }
@ -182,7 +196,7 @@ RSRingNode _spill_from_ring_to_ring(RSRing updated_ring,
    }

    while(updated_ring->u.head.sum_weights > 1 && ring_oversize(updated_ring)) {
-        RSRingNode* eden_oldest = ring_lru(updated_ring);
+        eden_oldest = ring_lru(updated_ring);
        if(!eden_oldest || eden_oldest == ignore_me) {
            break;
        }
@ -201,7 +215,7 @@ RSRingNode _spill_from_ring_to_ring(RSRing updated_ring,
                break;
            }

-            RSRingNode* probation_oldest = ring_lru(destination_ring);
+            probation_oldest = ring_lru(destination_ring);
            if(!probation_oldest) {
                //Hmm, the ring got emptied, but there's also no space
                //in protected. This must be a very large object. Take
@ -255,10 +269,12 @@ RSRingNode _spill_from_ring_to_ring(RSRing updated_ring,
 void rsc_probation_on_hit(RSCache* cache,
                          RSRingNode* entry)
 {
-    entry->u.entry.frequency++;
    RSRing protected_ring = cache->protected;
    RSRing probation_ring = cache->probation;
    int protected_oversize = ring_move_to_head_from_foreign(probation_ring, protected_ring, entry);
+
+    entry->u.entry.frequency++;
+
    if( !protected_oversize ) {
        return;
    }
@ -278,7 +294,7 @@ void rsc_probation_on_hit(RSCache* cache,
 * have produced victims, we return with rejects.frequency = 1 so the
 * caller can know to stop feeding us.
 */
-static inline
+RSR_SINLINE
 RSRingNode _eden_add(RSCache* cache,
                     RSRingNode* entry,
                     int allow_victims)
@ -286,8 +302,9 @@ RSRingNode _eden_add(RSCache* cache,
    RSRingNode* eden_ring = cache->eden;
    RSRingNode* protected_ring = cache->protected;
    RSRingNode* probation_ring = cache->probation;
+    RSRingNode* eden_oldest = NULL;

-    RSRingNode rejects = {};
+    RSRingNode rejects = {0};
    rejects.r_next = rejects.r_prev = NULL;

    rsc_ring_add(eden_ring, entry);
@ -307,7 +324,7 @@ RSRingNode _eden_add(RSCache* cache,
          # so go ahead and fill it.
        */
        while(ring_oversize(eden_ring)) {
-            RSRingNode* eden_oldest = ring_lru(eden_ring);
+            eden_oldest = ring_lru(eden_ring);
            if(!eden_oldest || eden_oldest == entry) {
                break;
            }
@ -354,15 +371,19 @@ int rsc_eden_add_many(RSCache* cache,
                      RSRingNode* entry_array,
                      int entry_count)
 {
+    int added_count = 0;
+    int i = 0;
+    RSRingNode add_rejects = {0};
+    RSRingNode* incoming = NULL;
+    RSRingNode* rejected = NULL;
+
    if (cache_oversize(cache) || !entry_count || !cache_will_fit(cache, entry_array)) {
        return 0;
    }

-    int added_count = 0;
-    int i = 0;
    for (i = 0; i < entry_count; i++) {
        // Don't try if we know we won't find a place for it.
-        RSRingNode* incoming = entry_array + i;
+        incoming = entry_array + i;
        if (!cache_will_fit(cache, incoming)) {
            incoming->u.entry.r_parent = -1;
            continue;
@ -371,7 +392,7 @@ int rsc_eden_add_many(RSCache* cache,
        // _eden_add *always* adds, but it may or may not be able to
        // rebalance.
        added_count += 1;
-        RSRingNode add_rejects = _eden_add(cache, incoming, _SPILL_NO_VICTIMS);
+        add_rejects = _eden_add(cache, incoming, _SPILL_NO_VICTIMS);
        if (add_rejects.u.entry.frequency) {
            // We would have rejected something, so we must be full.
            // Well, this isn't strictly true. It could be one really
@ -387,7 +408,7 @@ int rsc_eden_add_many(RSCache* cache,
    // Anything left is because we broke out of the loop. They're
    // rejects and need to be marked as such.
    for (i += 1; i < entry_count; i++) {
-        RSRingNode* rejected = entry_array + i;
+        rejected = entry_array + i;
        rejected->u.entry.r_parent = -1;
    }

@ -408,6 +429,8 @@ RSRingNode rsc_update_mru(RSCache* cache,
    RSRing protected_ring = cache->protected;
    RSRing probation_ring = cache->probation;
    RSRing eden_ring = cache->eden;
+    int protected_ring_oversize = 0;
+    RSRingNode result = {0};

    // Always update the frequency
    entry->u.entry.frequency++;
@ -429,7 +452,6 @@ RSRingNode rsc_update_mru(RSCache* cache,
        return rsc_eden_add(cache, entry);
    }

-    int protected_ring_oversize = 0;
    if (home_ring == probation_ring) {
        protected_ring_oversize = ring_move_to_head_from_foreign(home_ring, protected_ring, entry);
    }
@ -445,18 +467,18 @@ RSRingNode rsc_update_mru(RSCache* cache,
                                        _SPILL_VICTIMS, _SPILL_FIT);
    }

-    RSRingNode result = {};
    result.r_next = result.r_prev = NULL;
    return result;
 }

-static inline void lru_age_list(RSRingNode* ring)
+RSR_SINLINE void lru_age_list(RSRingNode* ring)
 {
+    RSRingNode* here = NULL;
    if (ring_is_empty(ring)) {
        return;
    }

-    RSRingNode* here = ring->r_next;
+    here = ring->r_next;
    while (here != ring) {
        here->u.entry.frequency = here->u.entry.frequency / 2;
        here = here->r_next;
--- a/relstorage/tests/blob/testblob.py
+++ b/relstorage/tests/blob/testblob.py
@ -634,10 +634,14 @@ checker = renormalizing.RENormalizing([

 try:
    file_type = file
+    PY3 = False
 except NameError:
    # Py3: Python 3 does not have a file type.
    import io
    file_type = io.BufferedReader
+    PY3 = True
+
+WIN = sys.platform.startswith('win')

 def storage_reusable_suite(prefix, factory,
                           test_blob_storage_recovery=False,
@ -663,13 +667,20 @@ def storage_reusable_suite(prefix, factory,
        test.globs['file_type'] = file_type

    suite = unittest.TestSuite()
-    suite.addTest(doctest.DocFileSuite(
-        "blob_connection.txt", "blob_importexport.txt",
-        "blob_transaction.txt",
-        setUp=setup, tearDown=tearDown,
-        optionflags=doctest.ELLIPSIS,
-        checker=checker,
-        ))
+    tests = ["blob_connection.txt", "blob_importexport.txt",]
+    if not (PY3 and WIN):
+        # This fails on Windows/Py3 for unknown reasons.
+        # But it seems to be due to ZODB's blob helper, not us
+        # https://ci.appveyor.com/project/jamadden/relstorage/build/1.0.16/job/4cji13ml2sargblw#L199
+        tests.append('blob_transaction.txt')
+    suite.addTest(
+        doctest.DocFileSuite(
+            *tests,
+            setUp=setup, tearDown=tearDown,
+            optionflags=doctest.ELLIPSIS,
+            checker=checker
+        )
+    )
    if test_packing:
        suite.addTest(doctest.DocFileSuite(
            pack_test_name,