Split out JSON and HSTORE documentation into separate sections

doc/src/SUMMARY.md

@@ -4,6 +4,8 @@
 - [Installation](./installation.md)
 - [Quickstart](./quickstart.md)
 - [Using pg-el](./usage.md)
+- [JSON and JSONB support](./json.md)
+- [HSTORE support](./hstore.md)
 - [Collation support](./collation.md)
 - [The COPY protocol](./copy-protocol.md)
 - [Using schema-qualified names](./schemas.md)

doc/src/about.md

@@ -37,10 +37,9 @@ PostgreSQL or a compatible database.
 - Connections over TCP or (on Unix machines) a local Unix socket.
 ~~~
 
-The code has been tested with **PostgreSQL versions** 16.3, 15.4, 13.8, 11.17, and 10.22 on Linux.
-It is also tested via GitHub actions on MacOS and Microsoft Windows, using the PostgreSQL version
-which is pre-installed in the virtual images (currently 14.8). This library also works, to a
-variable extent, against other databases that implement the PostgreSQL wire protocol:
+The code has been tested with **PostgreSQL versions** 17beta1, 16.3, 15.4, 13.8, 11.17, and 10.22 on
+Linux. It is also tested via GitHub actions on MacOS and Microsoft Windows. This library also works,
+to a variable extent, against other databases that implement the PostgreSQL wire protocol:
 
 - [YugabyteDB](https://yugabyte.com/): tested against version 2.21. This database uses a lot of
   code from PostgreSQL 11 and is quite compatible, including with the HSTORE and pgvector
@@ -82,7 +81,7 @@ Tested with **Emacs versions** 29.3, 28.2, 27.2 and 26.3. Emacs versions older t
 against a recent PostgreSQL version (whose default configuration requires SCRAM-SHA-256
 authentication), because they don’t include the GnuTLS support which we use to calculate HMACs. They
 may however work against a database set up to allow unauthenticated local connections. Emacs
-versions before 28.1 will not support the extended query protocol, because the bindat package is
+versions before 28.1 will not support the extended query protocol, because the `bindat` package is
 required. We mostly test with Emacs on Linux, but the library also works fine on Microsoft Windows
 and MacOS.
 

doc/src/copy-protocol.md

@@ -1,7 +1,9 @@
 # The COPY protocol
 
-The COPY protocol can be used to send and receive large amounts of data to/from PostgreSQL. It can
-be used with CSV or TSV data.
+The [COPY protocol](https://www.postgresql.org/docs/current/sql-copy.html) can be used to send and
+receive large amounts of data to/from PostgreSQL. It can be used with CSV or TSV data.
+
+## From Emacs to PostgreSQL
 
 The pg-el library allows you to COPY from an Emacs buffer into PostgreSQL using function
 `pg-copy-from-buffer`, as illustrated below.
@@ -56,6 +58,7 @@ ELISP> (pg-result (pg-exec *pg* "SELECT * FROM copy_csv LIMIT 3") :tuples)
 
 ~~~
 
+## From PostgreSQL to Emacs
 
 You can copy from PostgreSQL into an Emacs buffer using the function `pg-copy-to-buffer`, as
 illustrated below.
@@ -68,3 +71,44 @@ ELISP> (let ((res (pg-copy-to-buffer *pg* "COPY copy_csv TO STDOUT WITH (FORMAT
 "COPY 1000"
 ```
 ~~~
+
+
+
+The following more verbose example illustrates fetching CSV data from an online source, importing it
+into PostgreSQL, removing some unneeded columns and querying the data.
+
+~~~admonish example title="Fetching and querying online CSV datasets"
+```lisp
+ELISP> (with-temp-buffer
+         (url-insert-file-contents "https://www.data.gouv.fr/fr/datasets/r/51606633-fb13-4820-b795-9a2a575a72f1")
+         (pg-exec *pg* "CREATE TABLE cities(
+              insee_code TEXT NOT NULL,
+              city_code TEXT,
+              zip_code NUMERIC,
+              label TEXT NOT NULL,
+              latitude FLOAT,
+              longitude FLOAT,
+              department_name TEXT,
+              department_number VARCHAR(3),
+              region_name TEXT,
+              region_geojson_name TEXT)")
+         (pg-result (pg-copy-from-buffer *pg* "COPY cities FROM STDIN WITH (FORMAT CSV, DELIMITER ',', HEADER TRUE)"
+                                 (current-buffer)) :status))
+"COPY 39145"
+ELISP> (pg-result (pg-exec *pg* "ALTER TABLE cities DROP COLUMN region_name") :status)
+"ALTER TABLE"
+ELISP> (pg-result (pg-exec *pg* "ALTER TABLE cities DROP COLUMN region_geojson_name") :status)
+"ALTER TABLE"
+ELISP> (pg-result (pg-exec *pg* "ALTER TABLE cities DROP COLUMN label") :status)
+"ALTER TABLE"
+ELISP> (pg-result (pg-exec *pg* "SELECT * FROM cities WHERE city_code LIKE 'toulouse%'") :tuples)
+(("39533" "toulouse le chateau" 39230 46.821901729 5.583200112 "jura" "39")
+ ("31555" "toulouse" 31100 43.596037953 1.432094901 "haute-garonne" "31")
+ ("31555" "toulouse" 31300 43.596037953 1.432094901 "haute-garonne" "31")
+ ("31555" "toulouse" 31400 43.596037953 1.432094901 "haute-garonne" "31")
+ ("31555" "toulouse" 31500 43.596037953 1.432094901 "haute-garonne" "31")
+ ("31555" "toulouse" 31000 43.596037953 1.432094901 "haute-garonne" "31")
+ ("31555" "toulouse" 31200 43.596037953 1.432094901 "haute-garonne" "31"))
+```
+
+~~~

doc/src/hstore.md

@@ -0,0 +1,36 @@
+# The HSTORE key-value type
+
+There is support for the PostgreSQL [HSTORE
+  extension](https://www.postgresql.org/docs/current/hstore.html), which can store key/value pairs
+  in a single PostgreSQL column. It's necessary to call `pg-hstore-setup` before using this
+  functionality, to load the extension if necessary and to set up our parser support for the HSTORE
+  type.
+
+~~~admonish example title="Using HSTORE values"
+```lisp
+ELISP> (pg-hstore-setup *pg*)
+ELISP> (defvar *hs* (car (pg-result (pg-exec *pg* "SELECT 'foo=>bar'::hstore") :tuple 0)))
+*hs*
+ELISP> (gethash "foo" *hs*)
+"bar"
+ELISP> (hash-table-count *hs*)
+1 (#o1, #x1, ?\C-a)
+;; There is no guarantee as to the value stored for the 'a' key (duplicate)
+ELISP> (setq *hs* (car (pg-result (pg-exec *pg* "SELECT 'a=>1,foobles=>2,a=>66'::hstore") :tuple 0)))
+#<hash-table equal 2/65 0x1574257479d9>
+ELISP> (hash-table-count *hs*)
+2 (#o2, #x2, ?\C-b)
+ELISP> (pg-result (pg-exec *pg* "SELECT akeys('biz=>NULL,baz=>42,boz=>66'::hstore)") :tuple 0)
+(["baz" "biz" "boz"])
+```
+
+~~~
+
+
+
+~~~admonish example title="Serialization support for HSTORE values"
+```lisp
+ELISP> (pg-hstore-setup *pg*)
+```
+
+~~~

doc/src/installation.md

@@ -1,6 +1,7 @@
 # Installation
 
-You can install by the MELPA package archive, or manually.
+You can install via the MELPA package archive, or with package-vc-install, or manually.
+
 
 ## Installing via MELPA
 
@@ -17,6 +18,16 @@ then saying
      M-x package-install RET pg
 
 
+## Installing with package-vc-install
+
+You can install the library from the latest Github revision using:
+
+     (unless (package-installed-p 'pg)
+        (package-vc-install "https://github.com/emarsden/pg-el" nil nil 'pg))
+
+You can later update to the latest version with `M-x package-vc-upgrade RET pg RET`.
+
+
 ## Installing manually
 
 To install manually, place the file `pg.el` in a directory on your `load-path`, byte-compile it

doc/src/json.md

@@ -0,0 +1,59 @@
+# JSON and JSONB values
+
+PostgreSQL has quite a lot of [support for storing, saving and processing JSON and JSONB
+data](https://www.postgresql.org/docs/current/functions-json.html). pg-el is able to deserialize
+JSON and JSONB values into Emacs Lisp structures such as hashtables (for dicts), arrays, numbers,
+strings and so on.
+
+This library will parse and represent JSON/JSONB data either using the JSON support built into Emacs
+with libjansson (see function `json-available-p`, from version 28.1), or using the `json.el`
+library. There are some differences in the ways these methods handle dictionaries and specific
+values such as NULL, false, [] and {}. Our examples below use the builtin JSON support in Emacs.
+
+
+~~~admonish example title="Retrieving and manipulating JSON data"
+```lisp
+ELISP> (defun scalar (sql) (car (pg-result (pg-exec *pg* sql) :tuple 0)))
+scalar
+ELISP> (let ((json (scalar "SELECT '[5,7]'::json")))
+         (aref json 0))
+5 (#o5, #x5, ?\C-e)
+ELISP> (let ((json (scalar "SELECT '[42.0,77.7]'::jsonb")))
+         (aref json 1))
+77.7
+ELISP> (scalar "SELECT '[]'::json")
+[]
+ELISP> (scalar "SELECT '{}'::json")
+#<hash-table equal 0/1 0x1586e6cc2813>
+ELISP> (let ((json (scalar "SELECT '{\"a\": 42, \"b\": \"foo\"}'::json")))
+         (gethash "b" json))
+"foo"
+ELISP> (let ((json (scalar "SELECT '{\"a\": [0,1,2,null]}'::json")))
+         (gethash "a" json))
+[0 1 2 :null]
+```
+~~~
+
+
+pg-el can also serialize Emacs Lisp structures into the PostgreSQL JSON format, for use in prepared
+statements.
+
+~~~admonish example title="Serializing objects to JSON / JSONB"
+```lisp
+ELISP> (let ((ht (make-hash-table)))
+         (puthash "biz" 45 ht)
+         (puthash "boz" -5.5 ht)
+         (puthash "comment" "good stuff" ht)
+         (pg-result (pg-exec-prepared *pg* "SELECT $1->'boz'" `((,ht . "json"))) :tuple 0))
+(-5.5)
+ELISP> (let ((ht (make-hash-table)))
+         (puthash "biz" 45 ht)
+         (puthash "boz" -5.5 ht)
+         (puthash "comment" "good stuff" ht)
+         ;; the '-' jsonb operator deletes a matching key/value mapping
+         (let* ((res (pg-exec-prepared *pg* "SELECT $1 - 'boz'" `((,ht . "jsonb"))))
+                (row (pg-result res :tuple 0)))
+           (gethash "comment" (cl-first row) )))
+"good stuff"
+```
+~~~

doc/src/quickstart.md

@@ -15,7 +15,7 @@ postgres=# alter user pgeltestuser with encrypted password 'pgeltest';
 If you want to enable and test the support for the HSTORE and pgvector extensions, you will need to
 load them into the test database as PostgreSQL superuser (the normal user `pgeltestuser` we created
 above is not allowed to load extensions). The pgvector extension generally needs to be installed
-separately from PostgreSQL (for example by installing the ̀ postgresql-16-pgvector` package on Debian).
+separately from PostgreSQL (for example by installing the `postgresql-16-pgvector` package on Debian).
 
 ```shell
 sudo -u postgres psql

doc/src/special-features.md

@@ -8,7 +8,7 @@ them a special `ParameterStatus` message. These notifications are sent for `GUC_
 which include the `client_encoding`, the `DateStyle`, `TimeZone`, `server_encoding`,
 `in_hot_standby` and `is_superuser`. You can register your interest in these messages by adding a
 handler function to `pg-parameter-change-functions`. Each of these handler functions will be called
-when such as message is received, with three arguments: the connection to PostgreSQL, the parameter
+when such a message is received, with three arguments: the connection to PostgreSQL, the parameter
 name and the parameter value.
 
 These messages are sent asynchronously. 

doc/src/usage.md

@@ -281,67 +281,5 @@ t
 
 ## PostgreSQL arrays
 
+(To be documented)
 
-
-## JSON and JSONB values
-
-PostgreSQL has quite a lot of [support for storing, saving and processing JSON and JSONB
-data](https://www.postgresql.org/docs/current/functions-json.html). This library will parse and
-represent JSON/JSONB data either using the JSON support built into Emacs with libjansson (see
-function `json-available-p`, from version 28.1), or using the `json.el` library. There are some
-differences in the ways these methods handle dictionaries and specific values such as NULL, false,
-[] and {}. We will focus here on the builtin JSON support in Emacs.
-
-
-~~~admonish example title="Retrieving and manipulating JSON data"
-```lisp
-ELISP> (defun scalar (sql) (car (pg-result (pg-exec *pg* sql) :tuple 0)))
-scalar
-ELISP> (let ((json (scalar "SELECT '[5,7]'::json")))
-         (aref json 0))
-5 (#o5, #x5, ?\C-e)
-ELISP> (let ((json (scalar "SELECT '[42.0,77.7]'::jsonb")))
-         (aref json 1))
-77.7
-ELISP> (scalar "SELECT '[]'::json")
-[]
-ELISP> (scalar "SELECT '{}'::json")
-#<hash-table equal 0/1 0x1586e6cc2813>
-ELISP> (let ((json (scalar "SELECT '{\"a\": 42, \"b\": \"foo\"}'::json")))
-         (gethash "b" json))
-"foo"
-ELISP> (let ((json (scalar "SELECT '{\"a\": [0,1,2,null]}'::json")))
-         (gethash "a" json))
-[0 1 2 :null]
-```
-~~~
-
-
-
-## The HSTORE key-value type
-
-There is support for the PostgreSQL [HSTORE
-  extension](https://www.postgresql.org/docs/current/hstore.html), which can store key/value pairs
-  in a single PostgreSQL cell. It's necessary to call `pg-hstore-setup` before using this
-  functionality, to load the extension if necessary and to set up our parser support for the HSTORE
-  type.
-
-~~~admonish example title="Using HSTORE values"
-```lisp
-ELISP> (pg-hstore-setup *pg*)
-ELISP> (defvar *hs* (car (pg-result (pg-exec *pg* "SELECT 'foo=>bar'::hstore") :tuple 0)))
-*hs*
-ELISP> (gethash "foo" *hs*)
-"bar"
-ELISP> (hash-table-count *hs*)
-1 (#o1, #x1, ?\C-a)
-;; There is no guarantee as to the value stored for the 'a' key (duplicate)
-ELISP> (setq *hs* (car (pg-result (pg-exec *pg* "SELECT 'a=>1,foobles=>2,a=>66'::hstore") :tuple 0)))
-#<hash-table equal 2/65 0x1574257479d9>
-ELISP> (hash-table-count *hs*)
-2 (#o2, #x2, ?\C-b)
-ELISP> (pg-result (pg-exec *pg* "SELECT akeys('biz=>NULL,baz=>42,boz=>66'::hstore)") :tuple 0)
-(["baz" "biz" "boz"])
-```
-
-~~~