Install pgvector on CI #225

xuebinsu · 2023-11-08T06:50:17Z

Previsously we did not have test for embedding indexing. This patch
installs pgvector from the
PostgreSQL's apt repo so that we can test the embedding-related
features on CI.

To keep it simple, instead of modifying a container at runtime, this
patch adds a new tox env, called test-container, which builds the
docker image of database server with all dependencies before running
the tests. The test env is still available for running tests with
an existing server.

To make tests more effective, this patch also adds an option called
server_use_pickler to pytest, indicating that whether to use the
pickler dill to deserialize UDFs on server. As a result, we can
have test cases for the pickler, which will fail when the pickler is
not applicable. Such test cases are currently missing and we have
encountered several issues because of that.

beeender · 2023-11-13T08:26:57Z

README.dev.md

 ```
-tox -e test_py39
+python3 -m pip install tox-docker
+tox -e test-container


Does this start a container automatically? It doesn't with my testing. How is this supposed to work? like: building image->creating container->running test?

Yes, it will start the container automatically before running tests.

From the CI build, we should see the following lines when running tox -e test-container:

test-container: docker> build server/postgres12-python39.Dockerfile test-container: docker> built: sha256:ae44ebcefd17 test-container: docker> run 'sha256:ae44ebcefd17' (from 'server') test-container: docker> health check None (from 'server')

The workflow is as follows:

build image

run container

perform health check

run tests

stop and remove the container

It fails on my machine:

GreenplumPython on  ci_pgvector [$!?] via 🐍 v3.11.5 (venv311) ❯ tox -e test-container .pkg: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: get_requires_for_build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: get_requires_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ test-container: install_package> python -I -m pip install --force-reinstall --no-deps /home/cc/repo/GreenplumPython/.tox/.tmp/package/3/greenplum-python-1.0.1.tar.gz test-container: commands[0]> pytest --exitfirst ============================================================= test session starts ============================================================= platform linux -- Python 3.11.5, pytest-7.0.1, pluggy-1.3.0 cachedir: .tox/test-container/.pytest_cache rootdir: /home/cc/repo/GreenplumPython, configfile: tox.ini, testpaths: tests, greenplumpython collected 241 items tests/test_binaryExpr.py E =================================================================== ERRORS ==================================================================== __________________________________________________ ERROR at setup of test_expr_bin_equal_int __________________________________________________ server_use_pickler = True @pytest.fixture(scope="session") def db(server_use_pickler: bool): # for the connection both work for GitHub Actions and concourse > db = gp.database( params={ "host": environ.get("PGHOST", "localhost"), "dbname": environ.get("TESTDB", environ.get("USER")), "user": environ.get("PGUSER", environ.get("USER")), "password": environ.get("PGPASSWORD"), } ) tests/__init__.py:33: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ greenplumpython/db.py:281: in database return Database(uri=uri, params=params) greenplumpython/db.py:41: in __init__ self._conn = psycopg2.connect( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dsn = 'host=localhost dbname=cc user=cc', connection_factory = None, cursor_factory = <class 'psycopg2.extras.RealDictCursor'>, kwargs = {} kwasync = {} def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs): """ Create a new database connection. The connection parameters can be specified as a string: conn = psycopg2.connect("dbname=test user=postgres password=secret") or using a set of keyword arguments: conn = psycopg2.connect(database="test", user="postgres", password="secret") Or as a mix of both. The basic connection parameters are: - *dbname*: the database name - *database*: the database name (only as keyword argument) - *user*: user name used to authenticate - *password*: password used to authenticate - *host*: database host address (defaults to UNIX socket if not provided) - *port*: connection port number (defaults to 5432 if not provided) Using the *connection_factory* parameter a different class or connections factory can be specified. It should be a callable object taking a dsn argument. Using the *cursor_factory* parameter, a new default cursor factory will be used by cursor(). Using *async*=True an asynchronous connection will be created. *async_* is a valid alias (for Python versions where ``async`` is a keyword). Any other keyword parameter will be passed to the underlying client library: the list of supported parameters depends on the library version. """ kwasync = {} if 'async' in kwargs: kwasync['async'] = kwargs.pop('async') if 'async_' in kwargs: kwasync['async_'] = kwargs.pop('async_') dsn = _ext.make_dsn(dsn, **kwargs) > conn = _connect(dsn, connection_factory=connection_factory, **kwasync) E psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 65432 failed: Connection refused E Is the server running on that host and accepting TCP/IP connections? .tox/test-container/lib/python3.11/site-packages/psycopg2/__init__.py:122: OperationalError =========================================================== short test summary info =========================================================== ERROR tests/test_binaryExpr.py::test_expr_bin_equal_int - psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 6... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ============================================================== 1 error in 0.26s =============================================================== test-container: exit 1 (0.43 seconds) /home/cc/repo/GreenplumPython> pytest --exitfirst pid=499107 .pkg: _exit> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ test-container: FAIL code 1 (2.90=setup[2.46]+cmd[0.43] seconds) evaluation failed :( (2.94 seconds)

Have you installed tox-docker? Could you please share the result of pip show, like

[gpadmin@localhost GreenplumPython]$ python3 -m pip show tox-docker Name: tox-docker Version: 4.1.0 Summary: Launch a docker instance around test runs Home-page: https://github.com/tox-dev/tox-docker Author: None Author-email: None License: UNKNOWN Location: /home/gpadmin/.local/lib/python3.9/site-packages Requires: tox, docker, packaging Required-by:

GreenplumPython on  ci_pgvector [$!?] via 🐍 v3.11.5 (venv311) ❯ tox -e test-container .pkg: _optional_hooks> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: get_requires_for_build_sdist> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: get_requires_for_build_wheel> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: prepare_metadata_for_build_wheel> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ .pkg: build_sdist> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ test-container: install_package> python -I -m pip install --force-reinstall --no-deps /home/cc/repo/GreenplumPython/.tox/.tmp/package/4/greenplum-python-1.0.1.tar.gz test-container: docker> build server/postgres12-python39.Dockerfile test-container: docker> built: sha256:dd26e95cbf86 test-container: docker> run 'sha256:dd26e95cbf86' (from 'server') test-container: docker> health check None (from 'server') test-container: commands[0]> pytest --exitfirst ============================================================= test session starts ============================================================= platform linux -- Python 3.11.5, pytest-7.0.1, pluggy-1.3.0 cachedir: .tox/test-container/.pytest_cache rootdir: /home/cc/repo/GreenplumPython, configfile: tox.ini, testpaths: tests, greenplumpython collected 241 items tests/test_binaryExpr.py ........................ [ 9%] tests/test_builtin.py .... [ 11%] tests/test_column.py .. [ 12%] tests/test_dataframe.py ................................... [ 26%] tests/test_db.py .... [ 28%] tests/test_file.py ... [ 29%] tests/test_func.py ............................................. [ 48%] tests/test_group.py ...... [ 51%] tests/test_index.py .... [ 52%] tests/test_join.py .................... [ 60%] tests/test_order.py ....... [ 63%] tests/test_pandas_api.py ....... [ 66%] tests/test_schema.py ............. [ 72%] tests/test_type.py ...... [ 74%] tests/test_unaryExpr.py .... [ 76%] tests/test_use_pickler.py .F ================================================================== FAILURES =================================================================== _________________________________________________________ test_pickler_outside_class __________________________________________________________ db = <greenplumpython.db.Database object at 0x7f525184c410> def test_pickler_outside_class(db: gp.Database): @dataclass class Int: val: int @gp.create_function def add_one(i: int) -> Int: return Int(i + 1) > for row in db.apply(lambda: add_one(1), expand=True): tests/test_use_pickler.py:23: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ greenplumpython/dataframe.py:749: in __iter__ self._contents = self._fetch() greenplumpython/dataframe.py:873: in _fetch result = self._db._execute(to_json_dataframe._serialize()) greenplumpython/db.py:77: in _execute cursor.execute(query) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <cursor object at 0x7f5251e1f950; closed: -1> query = 'WITH cte_b643155ad8c34892a3788ec616aa262b AS (SELECT "pg_temp"."func_2d2ef878474e415daa1e20efb0ea4ad4"( 1) AS func_3c...27aa50acd4c44ac696feaeb2c8611)::TEXT FROM cte_bb6e62df32f24138a1af74673fa70e31 AS cte_8cd27aa50acd4c44ac696feaeb2c8611' vars = None def execute(self, query, vars=None): self.column_mapping = [] self._query_executed = True > return super().execute(query, vars) E psycopg2.errors.ExternalRoutineException: NameError: name 'Int' is not defined E CONTEXT: Traceback (most recent call last): E PL/Python function "func_2d2ef878474e415daa1e20efb0ea4ad4", line 16, in <module> E return GD['__func_2d2ef878474e415daa1e20efb0ea4ad4'](i=i) E PL/Python function "func_2d2ef878474e415daa1e20efb0ea4ad4", line 1, in __func_2d2ef878474e415daa1e20efb0ea4ad4 E PL/Python function "func_2d2ef878474e415daa1e20efb0ea4ad4" .tox/test-container/lib/python3.11/site-packages/psycopg2/extras.py:236: ExternalRoutineException ------------------------------------------------------------ Captured stdout setup ------------------------------------------------------------ SELECT version(); CREATE EXTENSION IF NOT EXISTS plpython3u; CREATE EXTENSION IF NOT EXISTS vector; DROP SCHEMA IF EXISTS test CASCADE; CREATE SCHEMA test; CREATE FUNCTION "pg_temp"."func_bc95268ebbaa4b439cc0bf8c8de20c46" ("requirements" "text") RETURNS "text" AS $$ try: return GD['__func_bc95268ebbaa4b439cc0bf8c8de20c46'](requirements=requirements) except KeyError: try: import dill as __lib_442cad7c9adc439c8cd761aa13392dbd import sysconfig as __lib_faeda410372842de814cbafd0378a099 import sys as __lib_b6a4cd7baa0246788b6f226c441ef0a0 if __lib_faeda410372842de814cbafd0378a099.get_python_version() != '3.11': raise ModuleNotFoundError setattr(__lib_b6a4cd7baa0246788b6f226c441ef0a0.modules['plpy'], '_SD', SD) GD['__func_bc95268ebbaa4b439cc0bf8c8de20c46'] = __lib_442cad7c9adc439c8cd761aa13392dbd.loads(b'\x80\x04\x95\xe2\x03\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C \x02\x02\x08\x01\x08\x02"\x02\x0c\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\xfa\x04\x08\x02\x01<\x01\x06\x01\x12\x010\xff\x94K\x01K\x00K\x00K\x06K\x06K\x03C\xdc\x97\x00d\x01d\x00l\x00}\x01d\x01d\x00l\x01}\x02|\x02j\x02\x00\x00\x00\x00\x00\x00\x00\x00s\nJ\x00d\x02\xa6\x00\x00\x00\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00\x82\x01|\x02j\x02\x00\x00\x00\x00\x00\x00\x00\x00d\x03d\x04d\x05d\x06d\x07g\x06}\x03\t\x00|\x01\xa0\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x03d\x08|\x01j\x04\x00\x00\x00\x00\x00\x00\x00\x00|\x00\xac\t\xa6\x04\x00\x00\xab\x04\x00\x00\x00\x00\x00\x00\x00\x00}\x04|\x04S\x00#\x00|\x01j\x05\x00\x00\x00\x00\x00\x00\x00\x00$\x00r\x19}\x05t\r\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x05j\x07\x00\x00\x00\x00\x00\x00\x00\x00\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00\x82\x01d\x00}\x05~\x05w\x01w\x00x\x03Y\x00w\x01\x94(NK\x00\x8c2Python executable is required to install packages.\x94\x8c\x02-m\x94\x8c\x03pip\x94\x8c\x07install\x94\x8c\r--requirement\x94\x8c\n/dev/stdin\x94\x88\x8c\x04text\x94\x8c\x06stderr\x94\x8c\x05input\x94\x87\x94t\x94(\x8c\nsubprocess\x94\x8c\x03sys\x94\x8c\nexecutable\x94\x8c\x0ccheck_output\x94\x8c\x06STDOUT\x94\x8c\x12CalledProcessError\x94\x8c\tException\x94\x8c\x06stdout\x94t\x94(\x8c\x0crequirements\x94\x8c\x02sp\x94h\x13\x8c\x03cmd\x94\x8c\x06output\x94\x8c\x01e\x94t\x94\x8c//home/cc/repo/GreenplumPython/tests/__init__.py\x94\x8c\x0bpip_install\x94h"K\tC\xa0\x80\x00\xe0\x04\x1b\xd0\x04\x1b\xd0\x04\x1b\xd0\x04\x1b\xd8\x04\x0e\x80J\x80J\x80J\xe0\x0b\x0e\x8c>\xd0\x04O\xd0\x04O\xd0\x1bO\xd1\x04O\xd4\x04O\xd0\x04O\xe0\x08\x0b\x8c\x0e\xd8\x08\x0c\xd8\x08\r\xd8\x08\x11\xd8\x08\x17\xd8\x08\x14\xf0\r\x07\x0b\x06\x80C\xf0\x10\x04\x05"\xd8\x11\x13\x97\x1f\x92\x1f\xa0\x13\xa84\xb8\x02\xbc\t\xc8\x1c\x90\x1f\xd1\x11V\xd4\x11V\x88\x06\xd8\x0f\x15\x88\r\xf8\xd8\x0b\r\xd4\x0b \xf0\x00\x01\x05"\xf0\x00\x01\x05"\xf0\x00\x01\x05"\xdd\x0e\x17\x98\x01\x9c\x08\xd1\x0e!\xd4\x0e!\xd0\x08!\xf8\xf8\xf8\xf8\xf0\x03\x01\x05"\xf8\xf8\xf8\x94C\x17\xa8\x1fA\x08\x00\xc1\x08\nA+\x03\xc1\x12\x14A&\x03\xc1&\x05A+\x03\x94))t\x94R\x94}\x94\x8c\x08__name__\x94\x8c\x05tests\x94sh"NNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}\x94(h\x1bh\x00\x8c\n_load_type\x94\x93\x94\x8c\x03str\x94\x85\x94R\x94\x8c\x06return\x94h4us\x86\x94bh\'\x8c\tException\x94h1h\x18\x85\x94R\x94s0.') except ModuleNotFoundError: exec("def __func_bc95268ebbaa4b439cc0bf8c8de20c46(requirements):\n import subprocess as sp\n import sys\n assert sys.executable, 'Python executable is required to install packages.'\n cmd = [sys.executable, '-m', 'pip', 'install', '--requirement', '/dev/stdin']\n try:\n output = sp.check_output(cmd, text=True, stderr=sp.STDOUT, input=requirements)\n return output\n except sp.CalledProcessError as e:\n raise Exception(e.stdout)", globals()) GD['__func_bc95268ebbaa4b439cc0bf8c8de20c46'] = globals()['__func_bc95268ebbaa4b439cc0bf8c8de20c46'] return GD['__func_bc95268ebbaa4b439cc0bf8c8de20c46'](requirements=requirements) $$ LANGUAGE plpython3u; WITH cte_8f8392e1885645618b19385c16508cdb AS (SELECT "pg_temp"."func_bc95268ebbaa4b439cc0bf8c8de20c46"( 'dill==0.3.6') ),cte_7ab3b55ec11d440492e9f9334eb25046 AS (SELECT cte_8f8392e1885645618b19385c16508cdb.* FROM cte_8f8392e1885645618b19385c16508cdb)SELECT to_json(cte_20d7fa2883664f4ca8aa9414de43cfa0)::TEXT FROM cte_7ab3b55ec11d440492e9f9334eb25046 AS cte_20d7fa2883664f4ca8aa9414de43cfa0 ------------------------------------------------------------------------------------------------------------------------------------------- func_bc95268ebbaa4b439cc0bf8c8de20c46 ------------------------------------------------------------------------------------------------------------------------------------------- Requirement already satisfied: dill==0.3.6 in /var/lib/postgresql/venv/lib/python3.9/site-packages (from -r /dev/stdin (line 1)) (0.3.6) ------------------------------------------------------------------------------------------------------------------------------------------- (1 row) ------------------------------------------------------------ Captured stdout call ------------------------------------------------------------- CREATE TYPE "pg_temp"."type_c22184a1bbcb4b1ebc9127dfe09a96d0" AS ( val "int4" ); CREATE FUNCTION "pg_temp"."func_2d2ef878474e415daa1e20efb0ea4ad4" ("i" "int4") RETURNS "type_c22184a1bbcb4b1ebc9127dfe09a96d0" AS $$ try: return GD['__func_2d2ef878474e415daa1e20efb0ea4ad4'](i=i) except KeyError: try: import dill as __lib_b1f5ce1f8d534cad841a7e2116b4dd9a import sysconfig as __lib_9496a77386224c408e453c8a4acf966a import sys as __lib_7ada146a0dc04b058e9af4d9b611a895 if __lib_9496a77386224c408e453c8a4acf966a.get_python_version() != '3.11': raise ModuleNotFoundError setattr(__lib_7ada146a0dc04b058e9af4d9b611a895.modules['plpy'], '_SD', SD) GD['__func_2d2ef878474e415daa1e20efb0ea4ad4'] = __lib_b1f5ce1f8d534cad841a7e2116b4dd9a.loads(b'\x80\x04\x95\xf9\n\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C\x02\x04\x02\x94K\x01K\x00K\x00K\x01K\x04K\x13C \x95\x01\x97\x00\x02\x00\x89\x01|\x00d\x01z\x00\x00\x00\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00S\x00\x94NK\x01\x86\x94)\x8c\x01i\x94\x85\x94\x8c7/home/cc/repo/GreenplumPython/tests/test_use_pickler.py\x94\x8c\x07add_one\x94\x8c+test_pickler_outside_class.<locals>.add_one\x94K\x13C\x15\xf8\x80\x00\xe0\x0f\x12\x88s\x901\x90q\x915\x89z\x8cz\xd0\x08\x19\x94C\x00\x94\x8c\x03Int\x94\x85\x94)t\x94R\x94}\x94\x8c\x08__name__\x94\x8c\x16tests.test_use_pickler\x94sh\x0bNh\x00\x8c\x0c_create_cell\x94\x93\x94N\x85\x94R\x94\x85\x94t\x94R\x94}\x94}\x94(\x8c\x0f__annotations__\x94}\x94(h\x08h\x00\x8c\n_load_type\x94\x93\x94\x8c\x03int\x94\x85\x94R\x94\x8c\x06return\x94h\x00\x8c\x0c_create_type\x94\x93\x94(h"\x8c\x04type\x94\x85\x94R\x94h\x0fh"\x8c\x06object\x94\x85\x94R\x94\x85\x94}\x94(\x8c\n__module__\x94h\x15\x8c\x0f__annotations__\x94}\x94\x8c\x03val\x94h%s\x8c\x07__doc__\x94\x8c\rInt(val: int)\x94\x8c\x14__dataclass_params__\x94\x8c\x0bdataclasses\x94\x8c\x10_DataclassParams\x94\x93\x94)\x81\x94N}\x94(\x8c\x04init\x94\x88\x8c\x04repr\x94\x88\x8c\x02eq\x94\x88\x8c\x05order\x94\x89\x8c\x0bunsafe_hash\x94\x89\x8c\x06frozen\x94\x89u\x86\x94b\x8c\x14__dataclass_fields__\x94}\x94h4h8\x8c\x05Field\x94\x93\x94)\x81\x94N}\x94(\x8c\x04name\x94h4h)h%\x8c\x07default\x94cdataclasses\nMISSING\n\x8c\x0fdefault_factory\x94cdataclasses\nMISSING\nh>\x88\x8c\x04hash\x94Nh=\x88\x8c\x07compare\x94\x88\x8c\x08metadata\x94h"\x8c\x10MappingProxyType\x94\x85\x94R\x94}\x94\x85\x94R\x94\x8c\x07kw_only\x94\x89\x8c\x0b_field_type\x94cdataclasses\n_FIELD\nu\x86\x94bs\x8c\x08__init__\x94h\x02(h\x04(C\x02\x02\x01\x94K\x02K\x00K\x00K\x02K\x02K\x13C\x14\x97\x00|\x01|\x00_\x00\x00\x00\x00\x00\x00\x00\x00\x00d\x00S\x00\x94N\x85\x94h4\x85\x94\x8c\x04self\x94h4\x86\x94\x8c\x08<string>\x94hY\x8c\x1f__create_fn__.<locals>.__init__\x94K\x02C\r\x80\x00\xd8\x0b\x0e\x80$\x84(\x80(\x80(\x94h\x0e))t\x94R\x94}\x94h\x14h\x15shYNNt\x94R\x94}\x94}\x94(h\x1f}\x94(h4h%h&Nu\x8c\x0c__qualname__\x94\x8c0test_pickler_outside_class.<locals>.Int.__init__\x94u\x86\x94b\x8c\x08__repr__\x94h\x02(h\x04(C\x10\x04\x02D\x01\x08\x01\x04\x01*\x01\x02\x01\x16\x02`\x01\x94K\x01K\x00K\x00K\x03K\x05K\x13C\xfa\x95\x02\x97\x00t\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x00\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00t\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00j\x02\x00\x00\x00\x00\x00\x00\x00\x00\xa6\x00\x00\x00\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00f\x02}\x01|\x01\x89\x03v\x00r\x02d\x01S\x00\x89\x03\xa0\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x01\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\t\x00\x02\x00\x89\x04|\x00\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00}\x02\x89\x03\xa0\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x01\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00n\x1a#\x00\x89\x03\xa0\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x01\xa6\x01\x00\x00\xab\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00w\x00x\x03Y\x00w\x01|\x02S\x00\x94N\x8c\x03...\x94\x86\x94(\x8c\x02id\x94\x8c\x07_thread\x94\x8c\tget_ident\x94\x8c\x03add\x94\x8c\x07discard\x94t\x94h^\x8c\x03key\x94\x8c\x06result\x94\x87\x94\x8c"/usr/lib/python3.11/dataclasses.py\x94\x8c\x07wrapper\x94\x8c _recursive_repr.<locals>.wrapper\x94K\xe9C\x8a\xf8\x80\x00\xe5\x0e\x10\x90\x14\x89h\x8ch\x9d\x07\xd4\x18)\xd1\x18+\xd4\x18+\xd0\x0e+\x88\x03\xd8\x0b\x0e\x90,\xd0\x0b\x1e\xd0\x0b\x1e\xd8\x13\x18\x905\xd8\x08\x14\xd7\x08\x18\xd2\x08\x18\x98\x13\xd1\x08\x1d\xd4\x08\x1d\xd0\x08\x1d\xf0\x02\x03\t&\xd8\x15"\x90]\xa04\xd1\x15(\xd4\x15(\x88F\xe0\x0c\x18\xd7\x0c \xd2\x0c \xa0\x13\xd1\x0c%\xd4\x0c%\xd0\x0c%\xd0\x0c%\xf8\x88L\xd7\x0c \xd2\x0c \xa0\x13\xd1\x0c%\xd4\x0c%\xd0\x0c%\xd0\x0c%\xf8\xf8\xf8\xd8\x0f\x15\x88\r\x94C\x0c\xc1\x00\x0bA!\x00\xc1!\x17A8\x03\x94\x8c\x0crepr_running\x94\x8c\ruser_function\x94\x86\x94)t\x94R\x94}\x94h\x14h\x15shnNh\x17N\x85\x94R\x94h\x17N\x85\x94R\x94\x86\x94t\x94R\x94}\x94\x8c\x0b__wrapped__\x94h\x02(h\x04(C\x02\x02\x01\x94K\x01K\x00K\x00K\x01K\x04K\x13C2\x97\x00|\x00j\x00\x00\x00\x00\x00\x00\x00\x00\x00j\x01\x00\x00\x00\x00\x00\x00\x00\x00d\x01|\x00j\x02\x00\x00\x00\x00\x00\x00\x00\x00\x9b\x02d\x02\x9d\x03z\x00\x00\x00S\x00\x94N\x8c\x05(val=\x94\x8c\x01)\x94\x87\x94\x8c\t__class__\x94hkh4\x87\x94h^\x85\x94h`hn\x8c\x1f__create_fn__.<locals>.__repr__\x94K\x02C \x80\x00\xd8\t\r\x8c\x1e\xd4\t$\xd0\'<\xa8t\xacx\xd0\'<\xd0\'<\xd0\'<\xd1\t<\xd0\x02<\x94h\x0e))t\x94R\x94}\x94h\x14h\x15shnNNt\x94R\x94}\x94}\x94(h\x1f}\x94hkh\x98u\x86\x94bs}\x94(h\x1fh\xa1hk\x8c0test_pickler_outside_class.<locals>.Int.__repr__\x94u\x86\x94bh\x86(\x8c\x02id\x94\x8c\x08builtins\x94\x8c\x02id\x94\x93\x94\x8c\x07_thread\x94h\x00\x8c\x0e_import_module\x94\x93\x94ht\x85\x94R\x94u0\x8c\x06__eq__\x94h\x02(h\x04(C\x06\x02\x01\x1c\x01$\x01\x94K\x02K\x00K\x00K\x02K\x02K\x13CP\x97\x00|\x01j\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x00j\x00\x00\x00\x00\x00\x00\x00\x00\x00u\x00r\x12|\x00j\x01\x00\x00\x00\x00\x00\x00\x00\x00f\x01|\x01j\x01\x00\x00\x00\x00\x00\x00\x00\x00f\x01k\x02\x00\x00\x00\x00S\x00t\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00S\x00\x94N\x85\x94h\x95h4\x8c\x0eNotImplemented\x94\x87\x94h^\x8c\x05other\x94\x86\x94h`h\xaf\x8c\x1d__create_fn__.<locals>.__eq__\x94K\x02C*\x80\x00\xd8\x05\n\x84_\x98\x04\x9c\x0e\xd0\x05&\xd0\x05&\xd8\x0b\x0f\x8c8\x88+\x98\x05\x9c\t\x90|\xd2\n#\xd0\x03#\xdd\t\x17\xd0\x02\x17\x94h\x0e))t\x94R\x94}\x94h\x14h\x15sh\xafNNt\x94R\x94}\x94}\x94(h\x1f}\x94hk\x8c.test_pickler_outside_class.<locals>.Int.__eq__\x94u\x86\x94bh\xbb\x8c\x0eNotImplemented\x94h\x00\x8c\n_eval_repr\x94\x93\x94\x8c\x0eNotImplemented\x94\x85\x94R\x94s0\x8c\x08__hash__\x94N\x8c\x0e__match_args__\x94h4\x85\x94ut\x94R\x94h\xa7\x8c\x07setattr\x94\x93\x94h\xcdhk\x8c\'test_pickler_outside_class.<locals>.Int\x94\x87\x94R0uhkh\x0cu\x86\x94bh\xa7\x8c\x07getattr\x94\x93\x94\x8c\x04dill\x94\x8c\x05_dill\x94\x93\x94\x8c\x08_setattr\x94h\xcf\x87\x94R\x94h\x8a\x8c\rcell_contents\x94h\x9e\x87\x94R0h\xdah\x88h\xdb\x8f\x94\x87\x94R0h\xdah\x19h\xdbh\xcd\x87\x94R0.') except ModuleNotFoundError: exec("def __func_2d2ef878474e415daa1e20efb0ea4ad4(i):\n return Int(i + 1)", globals()) GD['__func_2d2ef878474e415daa1e20efb0ea4ad4'] = globals()['__func_2d2ef878474e415daa1e20efb0ea4ad4'] return GD['__func_2d2ef878474e415daa1e20efb0ea4ad4'](i=i) $$ LANGUAGE plpython3u; WITH cte_b643155ad8c34892a3788ec616aa262b AS (SELECT "pg_temp"."func_2d2ef878474e415daa1e20efb0ea4ad4"( 1) AS func_3c00a917e5f74b59bb48887ca4279826 ),cte_bb6e62df32f24138a1af74673fa70e31 AS (SELECT (cte_b643155ad8c34892a3788ec616aa262b."func_3c00a917e5f74b59bb48887ca4279826").* FROM cte_b643155ad8c34892a3788ec616aa262b)SELECT to_json(cte_8cd27aa50acd4c44ac696feaeb2c8611)::TEXT FROM cte_bb6e62df32f24138a1af74673fa70e31 AS cte_8cd27aa50acd4c44ac696feaeb2c8611 =========================================================== short test summary info =========================================================== FAILED tests/test_use_pickler.py::test_pickler_outside_class - psycopg2.errors.ExternalRoutineException: NameError: name 'Int' is not defined !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================= 1 failed, 185 passed in 20.15s ======================================================== test-container: exit 1 (20.33 seconds) /home/cc/repo/GreenplumPython> pytest --exitfirst pid=593035 test-container: docker> remove '7e4af2ef8be8' (from 'server') .pkg: _exit> python /home/cc/gp/venv311/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ test-container: FAIL code 1 (126.25=setup[105.92]+cmd[20.33] seconds) evaluation failed :( (126.34 seconds)

Then I met antoher error. Python3.11.

test-container: docker> build server/postgres12-python39.Dockerfile test-container: docker> built: sha256:dd26e95cbf86 test-container: docker> run 'sha256:dd26e95cbf86' (from 'server') test-container: docker> health check None (from 'server') test-container: commands[0]> pytest --exitfirst ============================================================= test session starts ============================================================= platform linux -- Python 3.11.5, pytest-7.0.1, pluggy-1.3.0 cachedir: .tox/test-container/.pytest_cache rootdir: /home/cc/repo/GreenplumPython, configfile: tox.ini, testpaths: tests, greenplumpython collected 241 items

Why it builds a python39 image, but runs a python311 tests?

This means the server uses python3.9, while the client uses python3.11.

In this case the python versions don't match so dill will not work. In that case, the CI skips test_use_pickler.py since the file contains test cases that can only be passed when dill works.

OK, so when running docker test, the client is not running in the container, only the pg server, correct?

Yes, that's correct.

yihong0618 · 2023-11-13T08:23:19Z

.github/workflows/build_docs.yml

@@ -37,7 +37,7 @@ jobs:
      - name: Build the docs
        run: |
          export TAG_REF=${{ github.ref }}
-          tox -e docs
+          tox -e doc


nice change

yihong0618 · 2023-11-13T08:24:22Z

.github/workflows/test.yml

-        python-version: ["3.9", "3.10"]
-        postgres-version: ['12', '13']
+        python-version: ["3.9", "3.11"]
+        server: ["postgres12-python39", "postgres12-python311"]


did you host the container in dockerhub?

No, the image is built locally when running tox -e test-container.

yihong0618 · 2023-11-13T08:27:51Z

.github/workflows/test.yml

+        python-version: ["3.9", "3.11"]
+        server: ["postgres12-python39", "postgres12-python311"]


python3.9 and python3.11 pg-3.9 and pg-3.11 these are four different jobs am I right?

Yes, 4 jobs in total.

ruxuez · 2023-11-13T08:51:04Z

tests/test_dataframe.py

@@ -304,12 +304,12 @@ def test_table_refresh_add_rows(db: gp.Database):
 def test_table_refresh_add_columns(db: gp.Database):
    # Initial DataFrame
    nums = db.create_dataframe(rows=[(i,) for i in range(10)], column_names=["num"])
-    t = nums.save_as("const_table", column_names=["num"], temp=True)
+    t = nums.save_as(column_names=["num"], temp=True)


Do we need table_name here ?

Yes, we need the name, but we don't need to name it explicitly.

We can just use the auto-generated name to refer to it, like the line below:

db._execute(f"INSERT INTO {t._qualified_table_name}(num) VALUES (10);", has_results=False)

yihong0618 · 2023-11-14T06:00:42Z

most LGTM
one questions can we build the docker image and push them to docker hub then use it as service like we the precious one

xuebinsu · 2023-11-14T07:23:56Z

Quote from #225 (comment)

push them to docker hub

This is well beyond the scope of this PR. Let's discuss later.

yihong0618

LGTM

Install pgvector on CI

dafaf19

vmwclabot added the cla-not-required label Nov 8, 2023

Xuebin Su added 28 commits November 9, 2023 05:24

Build custom images for server

ffeec29

Clean code

e87bf84

Fix dependencies

7f36dd9

Clean again

bc7a67e

Print debug message

1333f38

Fix dependencies again

51280b2

Specify postgres user

8f935a5

Fix bracket

44f92e0

Fix permission issue

52e5f02

Remove services

d32d229

Fix errors in lint and doc

a548b1f

Skip install for doc

db1461f

Remove skip install for doc again

ee155c1

Add VERSION to MANIFEST.in

239491f

Cusomize env with tox --override

5d3f11e

Fix tox errors

bf0af2e

Fix variable errors

dcb9595

Fix error on docker env

c6866b9

Fix tox variable issue again

1bd2eed

Remove basepython

d0c315c

Remove all basepython

b1a5833

Fix typo

54ddd1b

Chnage back to 3.11

f496178

Fix bug on get_type_hints()

78eb7d3

Add option to use dill

a059f21

Use tox-docker to run server

db28bb9

Remove password

5a9cfbf

Move healthcheck to dockerfile

a56034c

Xuebin Su added 12 commits November 12, 2023 21:41

Make dill test optional

2939c7b

Try to fix if in jobs

7f8a0ab

Test if

6ed522c

Retry if

e992dc3

Try fix if again

893995d

Use format() in if

98566f5

Use single quote

58a0cfe

FIx format string

1daa435

Clean github actions config

ecfde8a

Fix error in variables

c0b021e

Fix wrong server config

32e7172

Add doc

226e623

xuebinsu requested review from beeender, ruxuez and yihong0618 and removed request for beeender and ruxuez November 13, 2023 06:30

xuebinsu marked this pull request as ready for review November 13, 2023 06:31

xuebinsu requested review from beeender and ruxuez November 13, 2023 06:31

xuebinsu mentioned this pull request Nov 13, 2023

Support more index types besides ivfflat #224

Merged

beeender reviewed Nov 13, 2023

View reviewed changes

yihong0618 reviewed Nov 13, 2023

View reviewed changes

ruxuez reviewed Nov 13, 2023

View reviewed changes

beeender approved these changes Nov 14, 2023

View reviewed changes

yihong0618 approved these changes Nov 15, 2023

View reviewed changes

xuebinsu merged commit 417dcc8 into main Nov 15, 2023
6 checks passed

xuebinsu deleted the ci_pgvector branch November 15, 2023 02:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install pgvector on CI #225

Install pgvector on CI #225

xuebinsu commented Nov 8, 2023 •

edited

Loading

beeender Nov 13, 2023

xuebinsu Nov 13, 2023 •

edited

Loading

xuebinsu Nov 13, 2023

beeender Nov 13, 2023

xuebinsu Nov 13, 2023

beeender Nov 13, 2023

beeender Nov 13, 2023 •

edited

Loading

xuebinsu Nov 13, 2023 •

edited

Loading

beeender Nov 14, 2023

xuebinsu Nov 14, 2023 •

edited

Loading

yihong0618 Nov 13, 2023

yihong0618 Nov 13, 2023

xuebinsu Nov 13, 2023

yihong0618 Nov 13, 2023

xuebinsu Nov 13, 2023

ruxuez Nov 13, 2023

xuebinsu Nov 13, 2023

yihong0618 commented Nov 14, 2023

xuebinsu commented Nov 14, 2023

yihong0618 left a comment

		python-version: ["3.9", "3.11"]
		server: ["postgres12-python39", "postgres12-python311"]

Install pgvector on CI #225

Install pgvector on CI #225

Conversation

xuebinsu commented Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

xuebinsu Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beeender Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

xuebinsu Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xuebinsu Nov 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihong0618 commented Nov 14, 2023

xuebinsu commented Nov 14, 2023

yihong0618 left a comment

Choose a reason for hiding this comment

xuebinsu commented Nov 8, 2023 •

edited

Loading

xuebinsu Nov 13, 2023 •

edited

Loading

beeender Nov 13, 2023 •

edited

Loading

xuebinsu Nov 13, 2023 •

edited

Loading

xuebinsu Nov 14, 2023 •

edited

Loading