[BUG]: Customize PySR

[jun199704]

What happened?

I am trying to customize PySR, I have revised the juliapkg.json file following the instructions as :

 {
    "julia": "=1.10.0, 1.10.3",
    "packages": {
        "SymbolicRegression": {
            "uuid": "8254be44-1295-4e6a-a16d-46603ac705cb",
            "dev":true,
            "path":"/home/drivesim/Documents/SR/SymbolicRegression.jl"
        },

and it works.
Then I try to adding additional functions and package ("JSON" and "PythonCall") in the Population.jl file. I have written them in the project.toml and Manifest.toml in SymbolicRegression.jl. and adding the packages in the Population.jl as follows:

using JSON
using PythonCall

However, when I try to run the code, I got the error like:

Failed to precompile SymbolicRegression [8254be44-1295-4e6a-a16d-46603ac705cb] to "/home/drivesim/.julia/compiled/v1.11/SymbolicRegression/jl_5hy8dP".
ERROR: LoadError: ArgumentError: Package SymbolicRegression does not have JSON in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have SymbolicRegression checked out for development and have
  added JSON as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with SymbolicRegression

Version

1.3.0

Operating System

Linux

Package Manager

pip

Interface

Script (i.e., python my_script.py)

Relevant log output

Failed to precompile SymbolicRegression [8254be44-1295-4e6a-a16d-46603ac705cb] to "/home/drivesim/.julia/compiled/v1.11/SymbolicRegression/jl_5hy8dP".
ERROR: LoadError: ArgumentError: Package SymbolicRegression does not have JSON in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have SymbolicRegression checked out for development and have
  added JSON as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with SymbolicRegression

Extra Info

No response

[MilesCranmer]

Oh it means you are missing the packages JSON and PythonCall in your custom version. To add them do this:

 julia --project="/home/drivesim/Documents/SR/SymbolicRegression.jl" -e 'using Pkg; pkg"add JSON PythonCall"'

[jun199704]

I run the above code
(base) drivesim@drivesim-4090:~/Documents/SR/SymbolicRegression.jl$ julia --project="/home/drivesim/Documents/SR/SymbolicRegression.jl" -e 'using Pkg; pkg"add JSON PythonCall"'
┌ Warning: The Pkg REPL mode is intended for interactive use only, and should not be used from scripts. It is recommended to use the functional API instead.
└ @ Pkg.REPLMode ~/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/REPLMode/REPLMode.jl:388
Resolving package versions...
Compat entries added for
No Changes to ~/Documents/SR/SymbolicRegression.jl/Project.toml
No Changes to ~/Documents/SR/SymbolicRegression.jl/Manifest.toml
Precompiling project...
2 dependencies successfully precompiled in 46 seconds. 128 already precompiled.
1 dependency had output during precompilation:
┌ SymbolicRegression
│ CondaPkg Found dependencies: /home/drivesim/.julia/packages/PythonCall/Nr75f/CondaPkg.toml
│ CondaPkg Resolving changes
│ + libstdcxx-ng
│ + python
│ CondaPkg Creating environment
│ │ /home/drivesim/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
│ │ -r /home/drivesim/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
│ │ create
│ │ -y
│ │ -p /home/drivesim/Documents/SR/SymbolicRegression.jl/.CondaPkg/env
│ │ --override-channels
│ │ --no-channel-priority
│ │ libstdcxx-ng[version='>=3.4,<13.0']
│ │ python[version='>=3.8,<4',channel='conda-forge',build='cpython']
│ └ -c conda-forge
│
│ Transaction
│
│ Prefix: /home/drivesim/Documents/SR/SymbolicRegression.jl/.CondaPkg/env
│
│ Updating specs:
│
│ - libstdcxx-ng[version='>=3.4,<13.0']
│ - conda-forge::python[version='>=3.8,<4',build=cpython]
│
│
│ Package Version Build Channel Size
│ ─────────────────────────────────────────────────────────────────────────────
│ Install:
│ ─────────────────────────────────────────────────────────────────────────────
│
│ + _libgcc_mutex 0.1 conda_forge conda-forge 3kB
│ + ld_impl_linux-64 2.43 h712a8e2_2 conda-forge 669kB
│ + ca-certificates 2024.12.14 hbcca054_0 conda-forge 157kB
│ + libgomp 12.4.0 h77fa898_1 conda-forge 419kB
│ + _openmp_mutex 4.5 2_gnu conda-forge 24kB
│ + libgcc 12.4.0 h77fa898_1 conda-forge 783kB
│ + libgcc-ng 12.4.0 h69a702a_1 conda-forge 52kB
│ + libstdcxx 12.4.0 hc0a3c3a_1 conda-forge 3MB
│ + openssl 3.3.1 h4bc722e_2 conda-forge 3MB
│ + libzlib 1.3.1 h4ab18f5_1 conda-forge 62kB
│ + libxcrypt 4.4.36 hd590300_1 conda-forge 100kB
│ + libffi 3.4.2 h7f98852_5 conda-forge 58kB
│ + bzip2 1.0.8 h4bc722e_7 conda-forge 253kB
│ + ncurses 6.5 he02047a_1 conda-forge 889kB
│ + libuuid 2.38.1 h0b41bf4_0 conda-forge 34kB
│ + libnsl 2.0.1 hd590300_0 conda-forge 33kB
│ + libexpat 2.6.2 h59595ed_0 conda-forge 74kB
│ + xz 5.2.6 h166bdaf_0 conda-forge 418kB
│ + libstdcxx-ng 12.4.0 h4852527_1 conda-forge 52kB
│ + tk 8.6.13 noxft_h4845f30_101 conda-forge 3MB
│ + libsqlite 3.46.0 hde9e2c9_0 conda-forge 865kB
│ + readline 8.2 h8228510_1 conda-forge 281kB
│ + tzdata 2024b hc8b5060_0 conda-forge 122kB
│ + python 3.12.5 h2ad013b_0_cpython conda-forge 32MB
│ + wheel 0.45.1 pyhd8ed1ab_1 conda-forge 63kB
│ + setuptools 75.8.0 pyhff2d567_0 conda-forge 776kB
│ + pip 24.3.1 pyh8b19718_2 conda-forge 1MB
│
│ Summary:
│
│ Install: 27 packages
│
│ Total download: 49MB
│
│ ─────────────────────────────────────────────────────────────────────────────
│
│
│
│ Transaction starting
│ Linking _libgcc_mutex-0.1-conda_forge
│ Linking ld_impl_linux-64-2.43-h712a8e2_2
│ Linking ca-certificates-2024.12.14-hbcca054_0
│ Linking libgomp-12.4.0-h77fa898_1
│ Linking _openmp_mutex-4.5-2_gnu
│ Linking libgcc-12.4.0-h77fa898_1
│ Linking libgcc-ng-12.4.0-h69a702a_1
│ Linking libstdcxx-12.4.0-hc0a3c3a_1
│ Linking openssl-3.3.1-h4bc722e_2
│ Linking libzlib-1.3.1-h4ab18f5_1
│ Linking libxcrypt-4.4.36-hd590300_1
│ Linking libffi-3.4.2-h7f98852_5
│ Linking bzip2-1.0.8-h4bc722e_7
│ Linking ncurses-6.5-he02047a_1
│ Linking libuuid-2.38.1-h0b41bf4_0
│ Linking libnsl-2.0.1-hd590300_0
│ Linking libexpat-2.6.2-h59595ed_0
│ Linking xz-5.2.6-h166bdaf_0
│ Linking libstdcxx-ng-12.4.0-h4852527_1
│ Linking tk-8.6.13-noxft_h4845f30_101
│ Linking libsqlite-3.46.0-hde9e2c9_0
│ Linking readline-8.2-h8228510_1
│ Linking tzdata-2024b-hc8b5060_0
│ Linking python-3.12.5-h2ad013b_0_cpython
│ Linking wheel-0.45.1-pyhd8ed1ab_1
│ Linking setuptools-75.8.0-pyhff2d567_0
│ Linking pip-24.3.1-pyh8b19718_2
│
│ Transaction finished
│
│ To activate this environment, use:
│
│ micromamba activate /home/drivesim/Documents/SR/SymbolicRegression.jl/.CondaPkg/env
│
│ Or to execute a single command in this environment, use:
│
│ micromamba run -p /home/drivesim/Documents/SR/SymbolicRegression.jl/.CondaPkg/env mycommand
└
But still, I have the error like this:
Precompiling SymbolicRegression...
Info Given SymbolicRegression was explicitly requested, output will be shown live
ERROR: LoadError: ArgumentError: Package SymbolicRegression does not have JSON in its dependencies:

• You may have a partially installed environment. Try Pkg.instantiate()
  to ensure all packages in the environment are installed.
• Or, if you have SymbolicRegression checked out for development and have
  added JSON as a dependency but haven't updated your primary
  environment's manifest file, try Pkg.resolve().
• Otherwise you may need to report an issue with SymbolicRegression
  Stacktrace:
  [1] macro expansion
  @ ./loading.jl:2299 [inlined]
  [2] macro expansion
  @ ./lock.jl:273 [inlined]
  [3] __require(into::Module, mod::Symbol)
  @ Base ./loading.jl:2271
  [4] #invoke_in_world#3
  @ ./essentials.jl:1089 [inlined]
  [5] invoke_in_world
  @ ./essentials.jl:1086 [inlined]
  [6] require(into::Module, mod::Symbol)
  @ Base ./loading.jl:2260
  [7] include(mod::Module, _path::String)
  @ Base ./Base.jl:557
  [8] include(x::String)
  @ SymbolicRegression ~/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:1
  [9] top-level scope
  @ ~/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:212
  [10] include
  @ ./Base.jl:557 [inlined]
  [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
  @ Base ./loading.jl:2881
  [12] top-level scope
  @ stdin:6
  in expression starting at /home/drivesim/Documents/SR/SymbolicRegression.jl/src/Population.jl:1
  in expression starting at /home/drivesim/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:1
  in expression starting at stdin:6
  ✗ SymbolicRegression
  0 dependencies successfully precompiled in 3 seconds. 108 already precompiled.

Traceback (most recent call last):
File "/home/drivesim/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.py", line 261, in
from pysr import PySRRegressor, TemplateExpressionSpec
File "/home/drivesim/anaconda3/lib/python3.12/site-packages/pysr/init.py", line 11, in
from .julia_import import jl, SymbolicRegression # isort:skip
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/drivesim/anaconda3/lib/python3.12/site-packages/pysr/julia_import.py", line 64, in
jl.seval("using SymbolicRegression")
File "/home/drivesim/.julia/packages/PythonCall/Nr75f/src/JlWrap/module.jl", line 27, in seval
return self._jl_callmethod($(pyjl_methodnum(pyjlmodule_seval)), expr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juliacall.JuliaError: The following 1 direct dependency failed to precompile:

SymbolicRegression

Failed to precompile SymbolicRegression [8254be44-1295-4e6a-a16d-46603ac705cb] to "/home/drivesim/.julia/compiled/v1.11/SymbolicRegression/jl_fNC00b".
ERROR: LoadError: ArgumentError: Package SymbolicRegression does not have JSON in its dependencies:

• You may have a partially installed environment. Try Pkg.instantiate()
  to ensure all packages in the environment are installed.
• Or, if you have SymbolicRegression checked out for development and have
  added JSON as a dependency but haven't updated your primary
  environment's manifest file, try Pkg.resolve().
• Otherwise you may need to report an issue with SymbolicRegression
  Stacktrace:
  [1] macro expansion
  @ ./loading.jl:2299 [inlined]
  [2] macro expansion
  @ ./lock.jl:273 [inlined]
  [3] __require(into::Module, mod::Symbol)
  @ Base ./loading.jl:2271
  [4] #invoke_in_world#3
  @ ./essentials.jl:1089 [inlined]
  [5] invoke_in_world
  @ ./essentials.jl:1086 [inlined]
  [6] require(into::Module, mod::Symbol)
  @ Base ./loading.jl:2260
  [7] include(mod::Module, _path::String)
  @ Base ./Base.jl:557
  [8] include(x::String)
  @ SymbolicRegression ~/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:1
  [9] top-level scope
  @ ~/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:212
  [10] include
  @ ./Base.jl:557 [inlined]
  [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
  @ Base ./loading.jl:2881
  [12] top-level scope
  @ stdin:6
  in expression starting at /home/drivesim/Documents/SR/SymbolicRegression.jl/src/Population.jl:1
  in expression starting at /home/drivesim/Documents/SR/SymbolicRegression.jl/src/SymbolicRegression.jl:1
  in expression starting at stdin:

[jun199704]

This is my Project.toml, it seems that PythonCall and JSON have been added to the deps.

name = "SymbolicRegression"
uuid = "8254be44-1295-4e6a-a16d-46603ac705cb"
authors = ["MilesCranmer [email protected]"]
version = "1.5.0"

[deps]
ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"
Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"
ConstructionBase = "187b0558-2788-49d3-abe0-74a17ed4e7c9"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
DifferentiationInterface = "a0c0ee7d-e4b9-4e03-894e-1c5f64a51d63"
DispatchDoctor = "8d63f2c5-f18a-4cf2-ba9d-b3f60fc568c8"
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
DynamicDiff = "7317a516-7a03-4707-b902-c6dba1468ba0"
DynamicExpressions = "a40a106e-89c9-4ca8-8020-a735e8728b6b"
DynamicQuantities = "06fc5a27-2a28-4c7c-a15d-362465fb6821"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
LineSearches = "d3d80556-e9d4-5f37-9878-2ab0fcc64255"
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
LossFunctions = "30fc2ffe-d236-52d8-8643-a9d8f7c094a7"
MLJModelInterface = "e80e1ace-859a-464e-9ed9-23947d8ae3ea"
MacroTools = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
Optim = "429524aa-4258-5aef-a3af-852621145aeb"
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Reexport = "189a3867-3050-52da-a836-e630ba90ab69"
SpecialFunctions = "276daf66-3868-5448-9aa4-cd146d93841b"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
StyledStrings = "f489334b-da3d-4c2e-b8f0-e476e12c162b"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[weakdeps]
Enzyme = "7da242da-08ed-463a-9acd-ee780be4f1d9"
JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
SymbolicUtils = "d1185830-fcd6-423d-90d6-eec64667417b"

[extensions]
SymbolicRegressionEnzymeExt = "Enzyme"
SymbolicRegressionJSON3Ext = "JSON3"
SymbolicRegressionSymbolicUtilsExt = "SymbolicUtils"

[compat]
ADTypes = "^1.4.0"
Compat = "^4.16"
ConstructionBase = "1.0.0 - 1.5.6, 1.5.8 - 1"
Dates = "1"
DifferentiationInterface = "0.5, 0.6"
DispatchDoctor = "^0.4.17"
Distributed = "<0.0.1, 1"
DynamicDiff = "0.2"
DynamicExpressions = "~1.9"
DynamicQuantities = "1"
Enzyme = "0.12, 0.13"
JSON = "0.21.4"
JSON3 = "1"
LineSearches = "7"
Logging = "1"
LossFunctions = "0.10, 0.11, 1"
MLJModelInterface = "~1.5, ~1.6, ~1.7, ~1.8, ~1.9, ~1.10, ~1.11"
MacroTools = "0.4, 0.5"
Optim = "~1.8, ~1.9, ~1.10"
Pkg = "<0.0.1, 1"
PrecompileTools = "1"
Printf = "<0.0.1, 1"
ProgressMeter = "1.10"
PythonCall = "0.9.23"
Random = "<0.0.1, 1"
Reexport = "1"
SpecialFunctions = "0.10.1, 1, 2"
StatsBase = "0.33, 0.34"
StyledStrings = "1"
SymbolicUtils = "0.19, ^1.0.5, 2, 3"
TOML = "<0.0.1, 1"
julia = "1.10"

[extras]
Enzyme = "7da242da-08ed-463a-9acd-ee780be4f1d9"
JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
SymbolicUtils = "d1185830-fcd6-423d-90d6-eec64667417b"

[MilesCranmer]

That's good! Maybe try deleting the local python environment and re-installing PySR with the custom juliapkg.json that points to your SymbolicRegression?

Worst case you could just directly use juliapkg: https://github.com/JuliaPy/pyjuliapkg.

Just run the juliapkg.add(pkg, uuid) before importing PySR, and it will modify the environment with those extra dependencies.

[MilesCranmer]

Speaking of which, maybe just try using that directly to edit the juliapkg.json? I wonder if it's just not being stored correctly.

The full signature is

add(pkg, uuid, dev=False, version=None, path=None, subdir=None, url=None, rev=None, target=None)

so you can tag SymbolicRegression and put dev=True, path="your/path".

Again, do this before importing PySR.

[jun199704]

I am still working on this problem and I want to check when adding "JSON" and "PythonCall", should them be added to the project in "SymbolicRegression.jl" or "PySR" or both. Currently I only added them in the project.toml and manifest in "SymbolicRegression.jl" and tried to use them in the "population.jl" file in "SymbolicRegression.jl".

[MilesCranmer]

Make sure to instantiate the environment of SymbolicRegression.jl with Pkg.instantiate. Then you can add it to PySR with the development setting