Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack" #286

Open
geverl opened this issue Sep 26, 2024 · 3 comments
Open

connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack" #286

geverl opened this issue Sep 26, 2024 · 3 comments

Comments

@geverl
Copy link

geverl commented Sep 26, 2024

On Ubuntu 24.04 using llmstack 0.2.5:
llmstack
Container llmstack-weaviate-1 Started
Container llmstack-redis-1 Started
Container llmstack-postgres-1 Started
Container llmstack-api-1 Starting
Container llmstack-rqworker-1 Starting
Container llmstack-runner-1 Started
Container llmstack-api-1 Started
Container llmstack-app-1 Starting
Container llmstack-rqworker-1 Started
Container llmstack-app-1 Startedartedg
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Waiting for LLMStack server to be up...
Failed to connect to LLMStack server. Exiting...
redis-1 | 1:C 26 Sep 2024 07:37:01.830 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis-1 | 1:C 26 Sep 2024 07:37:01.830 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1 | 1:C 26 Sep 2024 07:37:01.830 * Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1 | 1:C 26 Sep 2024 07:37:01.830 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 26 Sep 2024 07:37:01.831 * monotonic clock: POSIX clock_gettime
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Running mode=standalone, port=6379.
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Server initialized
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Loading RDB produced by version 7.4.0
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * RDB age 307 seconds
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * RDB memory usage when created 1.30 Mb
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Done loading RDB, keys loaded: 3, keys expired: 3.
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * DB loaded from disk: 0.000 seconds
redis-1 | 1:M 26 Sep 2024 07:37:01.833 * Ready to accept connections tcp
rqworker-1 | postgres:5432 - accepting connections
rqworker-1 | Postgres is up - continuing
rqworker-1 | Starting RQ worker
rqworker-1 | /usr/local/lib/python3.11/site-packages/langrocks/common/models/tools_pb2_grpc.py:23: RuntimeWarning: The grpc package installed is at version 1.64.1, but the generated code in tools_pb2_grpc.py depends on grpcio>=1.65.1. Please upgrade your grpc module to grpcio>=1.65.1 or downgrade your generated code using grpcio-tools<=1.64.1. This warning will become an error in 1.66.0, scheduled for release on August 6, 2024.
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:201: UserWarning: Field name "schema" in "PandasStore" shadows an attribute in parent "BaseDestination"
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_config.py:291: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:273: PydanticDeprecatedSince20: json_encoders is deprecated. See https://docs.pydantic.dev/2.8/concepts/serialization/#custom-serializers for alternatives. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:814: PydanticDeprecatedSince20: Using extra keyword arguments on Field is deprecated and will be removed. Use json_schema_extra instead. (Extra keys: 'maximum', 'minimum'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
rqworker-1 | warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:783: PydanticDeprecatedSince20: min_items is deprecated and will be removed, use min_length instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | postgres:5432 - accepting connections
rqworker-1 | warn('min_items is deprecated and will be removed, use min_length instead', DeprecationWarning)
app-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
api-1 | Starting API server
api-1 | Postgres is up - continuing
app-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
runner-1 | Starting redis server
api-1 | /usr/local/lib/python3.11/site-packages/langrocks/common/models/tools_pb2_grpc.py:23: RuntimeWarning: The grpc package installed is at version 1.64.1, but the generated code in tools_pb2_grpc.py depends on grpcio>=1.65.1. Please upgrade your grpc module to grpcio>=1.65.1 or downgrade your generated code using grpcio-tools<=1.64.1. This warning will become an error in 1.66.0, scheduled for release on August 6, 2024.
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
api-1 | warnings.warn(
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:201: UserWarning: Field name "schema" in "PandasStore" shadows an attribute in parent "BaseDestination"
api-1 | warnings.warn(
rqworker-1 | /usr/local/lib/python3.11/site-packages/pydantic/fields.py:814: PydanticDeprecatedSince20: Using extra keyword arguments on Field is deprecated and will be removed. Use json_schema_extra instead. (Extra keys: 'example'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_config.py:291: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
app-1 | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
runner-1 | Starting playwright server on port 50053
app-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
runner-1 | Listening on ws://localhost:50053/
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
runner-1 | INFO:langrocks.common.display:Created display: 99
app-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
runner-1 | INFO:langrocks.common.display:Created display: 100
app-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
weaviate-1 | {"action":"startup","default_vectorizer_module":"text2vec-openai","level":"info","msg":"the default vectorizer modules is set to "text2vec-openai", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to "true"","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"open cluster service","servers":{"weaviate-node":8300},"time":"2024-09-26T07:37:01Z"}
postgres-1 |
runner-1 | INFO:langrocks.common.display:Created display: 101
postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres-1 |
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: starting PostgreSQL 16.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on IPv6 address "::", port 5432
runner-1 | INFO:langrocks.common.display:Created display: 102
weaviate-1 | {"address":"172.18.0.2:8301","level":"info","msg":"starting cloud rpc server ...","time":"2024-09-26T07:37:01Z"}
runner-1 | INFO:langrocks.common.display:Created display: 103
rqworker-1 | warn(
rqworker-1 | /code/llmstack/common/blocks/llm/openai.py:781: PydanticDeprecatedSince20: pydantic.config.Extra is deprecated, use literal values instead (e.g. extra='allow'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
runner-1 | WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
weaviate-1 | {"level":"info","msg":"starting raft sub-system ...","time":"2024-09-26T07:37:01Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: using the "epoll" event method
api-1 | warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)
app-1 | 2024/09/26 07:37:02 [notice] 1#1: nginx/1.26.1
weaviate-1 | {"address":"172.18.0.2:8300","level":"info","msg":"tcp transport","tcpMaxPool":3,"tcpTimeout":10000000000,"time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"loading local db","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","msg":"database has been successfully loaded","n":0,"time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"level":"info","metadata_only_voters":false,"msg":"construct a new raft node","name":"weaviate-node","time":"2024-09-26T07:37:01Z"}
weaviate-1 | {"action":"raft","index":1,"level":"info","msg":"raft initial configuration","servers":"[[{Suffrage:Voter ID:weaviate-node Address:172.18.0.2:8300}]]","time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"last_snapshot_index":0,"last_store_applied_index":0,"last_store_log_applied_index":0,"level":"info","msg":"raft node constructed","raft_applied_index":0,"raft_last_index":18,"time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"action":"raft","follower":{},"leader-address":"","leader-id":"","level":"info","msg":"raft entering follower state","time":"2024-09-26T07:37:02Z"}
weaviate-1 | {"action":"bootstrap","error":"could not join a cluster from [172.18.0.2:8300]","level":"warning","msg":"failed to join cluster, will notify next if voter","servers":["172.18.0.2:8300"],"time":"2024-09-26T07:37:03Z","voter":true}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: built by gcc 13.2.1 20231014 (Alpine 13.2.1_git20231014)
weaviate-1 | {"action":"bootstrap","candidates":[{"Suffrage":0,"ID":"weaviate-node","Address":"172.18.0.2:8300"}],"level":"info","msg":"starting cluster bootstrapping","time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"bootstrap","error":"bootstrap only works on new clusters","level":"error","msg":"could not bootstrapping cluster","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: OS: Linux 6.8.0-45-generic
rqworker-1 | extra = Extra.forbid
app-1 | 2024/09/26 07:37:02 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker processes
rqworker-1 | System check identified some issues:
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 20
rqworker-1 |
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 21
postgres-1 | 2024-09-26 07:37:01.920 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
weaviate-1 | {"action":"bootstrap","level":"info","msg":"notified peers this node is ready to join as voter","servers":["172.18.0.2:8300"],"time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"raft","last-leader-addr":"","last-leader-id":"","level":"warning","msg":"raft heartbeat timeout reached, starting election","time":"2024-09-26T07:37:03Z"}
weaviate-1 | {"action":"raft","level":"info","msg":"raft entering candidate state","node":{},"term":19,"time":"2024-09-26T07:37:03Z"}
postgres-1 | 2024-09-26 07:37:01.924 UTC [29] LOG: database system was shut down at 2024-09-26 07:31:54 UTC
rqworker-1 | WARNINGS:
runner-1 | I0000 00:00:1727336222.178037 1 config.cc:230] gRPC experiments enabled: call_status_override_on_cancellation, event_engine_dns, event_engine_listener, http2_stats_fix, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
api-1 | /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:273: PydanticDeprecatedSince20: json_encoders is deprecated. See https://docs.pydantic.dev/2.8/concepts/serialization/#custom-serializers for alternatives. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
api-1 | warnings.warn(
api-1 | Traceback (most recent call last):
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 22
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
postgres-1 | 2024-09-26 07:37:01.932 UTC [1] LOG: database system is ready to accept connections
runner-1 | INFO:langrocks.app.server:Server running at http://[::]:50051
weaviate-1 | {"action":"raft","level":"info","msg":"raft election won","tally":1,"term":19,"time":"2024-09-26T07:37:03Z"}
postgres-1 | 2024-09-26 07:37:07.158 UTC [35] FATAL: password authentication failed for user "llmstack"
rqworker-1 | apiabstractor.Feedback.response_quality: (fields.W122) 'max_length' is ignored when used with SmallIntegerField.
weaviate-1 | {"action":"raft","leader":{},"level":"info","msg":"raft entering leader state","time":"2024-09-26T07:37:03Z"}
rqworker-1 | HINT: Remove 'max_length' from field
rqworker-1 | sheets.PromptlySheetRunEntry.data: (fields.E010) JSONField default should be a callable instead of an instance so that it's not shared between all field instances.
rqworker-1 | HINT: Use a callable instead, e.g., use dict instead of {}.
weaviate-1 | {"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-09-26T07:37:03Z"}
api-1 | self.connect()
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 23
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 24
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 25
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 26
rqworker-1 | /usr/local/lib/python3.11/site-packages/rq/connections.py:35: DeprecationWarning: The Connection context manager is deprecated. Use the connection parameter instead.
runner-1 | WebSocket server settings:
rqworker-1 | warnings.warn(
runner-1 | - Listen on :50052
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 27
runner-1 | - Web server. Web root: /usr/share/www/html
runner-1 | - No SSL/TLS support (no cert file)
runner-1 | - proxying from :50052 to targets generated by TokenRedis
rqworker-1 | /usr/local/lib/python3.11/site-packages/rq/connections.py:58: DeprecationWarning: The push_connection function is deprecated. Pass the connection explicitly instead.
rqworker-1 | warnings.warn(
postgres-1 | 2024-09-26 07:37:07.158 UTC [35] DETAIL: Password does not match for user "llmstack".
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 28
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 29
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 30
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 31
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
weaviate-1 | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 32
weaviate-1 | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-09-26T07:37:03Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 33
weaviate-1 | {"address":"172.18.0.2:8300","level":"info","msg":"current Leader","time":"2024-09-26T07:37:04Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 34
weaviate-1 | {"level":"info","msg":"starting migration from old schema","time":"2024-09-26T07:37:04Z"}
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 35
weaviate-1 | {"level":"info","msg":"legacy schema is empty, nothing to migrate","time":"2024-09-26T07:37:04Z"}
weaviate-1 | {"level":"info","msg":"migration from the old schema has been successfully completed","time":"2024-09-26T07:37:04Z"}
postgres-1 | 2024-09-26 07:37:07.285 UTC [36] FATAL: password authentication failed for user "llmstack"
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 36
weaviate-1 | {"action":"bootstrap","level":"info","msg":"node reporting ready, node has probably recovered cluster from raft config. Exiting bootstrap process","time":"2024-09-26T07:37:04Z"}
api-1 | ^^^^^^^^^^^^^^^^^^^^^
postgres-1 | 2024-09-26 07:37:07.285 UTC [36] DETAIL: Password does not match for user "llmstack".
weaviate-1 | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:ca9b4150-2194-4e11-b54a-82c2c41908f2 Type:INIT Version:1.25.7 NumObjects:0 OS:linux Arch:amd64 UsedModules:[]}","time":"2024-09-26T07:37:04Z"}
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 37
api-1 | self.connection = self.get_new_connection(conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
postgres-1 | 2024-09-26 07:37:07.694 UTC [37] FATAL: password authentication failed for user "llmstack"
api-1 | ^^^^^^^^^^^^^^^^^^^^^
postgres-1 | 2024-09-26 07:37:07.694 UTC [37] DETAIL: Password does not match for user "llmstack".
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 277, in get_new_connection
postgres-1 | Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all password"
api-1 | connection = self.Database.connect(**conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/psycopg2/init.py", line 122, in connect
api-1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | psycopg2.OperationalError: connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack"
api-1 |
api-1 |
api-1 | The above exception was the direct cause of the following exception:
api-1 |
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 38
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 39
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 40
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 41
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 42
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 43
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 44
api-1 | Traceback (most recent call last):
api-1 | File "/code/manage.py", line 22, in
api-1 | main()
api-1 | File "/code/manage.py", line 18, in main
api-1 | execute_from_command_line(sys.argv)
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
api-1 | utility.execute()
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/init.py", line 436, in execute
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 45
api-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 46
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
api-1 | self.execute(*args, **cmd_options)
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
api-1 | output = self.handle(*args, **options)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 47
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 48
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 49
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 50
app-1 | 2024/09/26 07:37:02 [notice] 1#1: start worker process 51
app-1 | 2024/09/26 07:37:02 [error] 20#20: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:37:02 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
app-1 | 2024/09/26 07:37:04 [error] 21#21: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | res = handle_func(*args, **kwargs)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1 | 172.18.0.1 - - [26/Sep/2024:07:37:04 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:37:07 [error] 22#22: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:37:07 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:10 [error] 23#23: *7 upstream timed out (110: Operation timed out) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:10 +0000] "GET / HTTP/1.1" 504 167 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:15 [error] 24#24: *9 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:15 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 100, in handle
app-1 | 2024/09/26 07:38:20 [error] 25#25: *11 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:20 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:26 [error] 26#26: *13 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:26 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:31 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:31 [error] 27#27: *15 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 2024/09/26 07:38:37 [error] 28#28: *17 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | self.check(databases=[database])
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:37 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 486, in check
api-1 | all_issues = checks.run_checks(
api-1 | ^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/checks/registry.py", line 88, in run_checks
api-1 | new_errors = check(app_configs=app_configs, databases=databases)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/core/checks/model_checks.py", line 36, in check_all_models
api-1 | errors.extend(model.check(**kwargs))
api-1 | ^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1617, in check
api-1 | *cls._check_constraints(databases),
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 2467, in _check_constraints
api-1 | connection.features.supports_nulls_distinct_unique_constraints
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 47, in get
api-1 | res = instance.dict[self.name] = self.func(instance)
app-1 | 2024/09/26 07:38:43 [error] 29#29: *19 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:43 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:48 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:48 [error] 30#30: *21 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:38:54 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:38:54 [error] 31#31: *23 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | ^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/features.py", line 141, in is_postgresql_15
app-1 | 2024/09/26 07:39:00 [error] 32#32: *25 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:00 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:39:05 [error] 33#33: *27 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:05 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:11 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:39:11 [error] 34#34: *29 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:16 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | return self.connection.pg_version >= 150000
app-1 | 2024/09/26 07:39:16 [error] 35#35: *31 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 47, in get
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:22 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:39:22 [error] 37#37: *33 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 2024/09/26 07:39:27 [error] 36#36: *35 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | res = instance.dict[self.name] = self.func(instance)
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:27 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | ^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 438, in pg_version
api-1 | with self.temporary_connection():
api-1 | File "/usr/local/lib/python3.11/contextlib.py", line 137, in enter
api-1 | return next(self.gen)
api-1 | ^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 691, in temporary_connection
api-1 | with self.cursor() as cursor:
app-1 | 2024/09/26 07:39:33 [error] 39#39: *37 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | ^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
api-1 | ^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
api-1 | return self._cursor()
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:33 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:38 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | ^^^^^^^^^^^^^^
app-1 | 2024/09/26 07:39:38 [error] 38#38: *39 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 2024/09/26 07:39:44 [error] 40#40: *41 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
api-1 | self.ensure_connection()
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
api-1 | ^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection
api-1 | with self.wrap_database_errors:
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:44 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in exit
app-1 | 2024/09/26 07:39:50 [error] 41#41: *43 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:50 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
api-1 | raise dj_exc_value.with_traceback(traceback) from exc_value
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
api-1 | self.connect()
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
api-1 | ^^^^^^^^^^^^^^^^^^^^^
app-1 | 2024/09/26 07:39:56 [error] 42#42: *45 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:39:56 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 172.18.0.1 - - [26/Sep/2024:07:40:02 +0000] "GET / HTTP/1.1" 502 157 "-" "python-requests/2.32.3" "-"
app-1 | 2024/09/26 07:40:02 [error] 43#43: *47 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:9000/", host: "localhost:3000"
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
api-1 | self.connection = self.get_new_connection(conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
api-1 | return func(*args, **kwargs)
api-1 | ^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 277, in get_new_connection
api-1 | connection = self.Database.connect(**conn_params)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | File "/usr/local/lib/python3.11/site-packages/psycopg2/init.py", line 122, in connect
api-1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1 | django.db.utils.OperationalError: connection to server at "postgres" (172.18.0.3), port 5432 failed: FATAL: password authentication failed for user "llmstack"
api-1 |
Stopping LLMStack server...
[+] Running 8/8
✔ Container llmstack-app-1 Removed 0.4s
✔ Container llmstack-runner-1 Removed 10.5s
✔ Container llmstack-weaviate-1 Removed 1.2s
✔ Container llmstack-rqworker-1 Removed 10.4s
✔ Container llmstack-api-1 Removed 0.0s
✔ Container llmstack-postgres-1 Removed 0.3s
✔ Container llmstack-redis-1 Removed 0.3s
✔ Network llmstack_default Removed
@Lpradel1
Copy link

HI all , I have exactly the same issue on ubuntu 22.04.03 and 22.04.05, is there a problem with ubuntu ? If yes, which distribution is recommanded ?

@BWD-Oscar
Copy link

Hi all, I also had similar issue (the error logs are lengthy).
But I saw the last bit of the logs, which indicated authentication failed for postgres db.
Then I edited a line from docker-compose.yml to "POSTGRES_HOST_AUTH_METHOD: "TRUST"".
This docker-compose.yml was under this path "~/.local/lib/python3.11/site-packages/llmstack"

After that it didn't show any postgres auth issue.
But there were further issues from mine.
Hope this works for your case.

@Space11-Robotics
Copy link

anyone has solve? i've the same issue Ubuntu 22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@geverl @Lpradel1 @Space11-Robotics @BWD-Oscar