From 692cc9b19a4e763cd703d4fb167d51a6c76bded2 Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Tue, 27 Aug 2024 23:43:56 -0400 Subject: [PATCH 01/10] Change `u` to `x` and the order of args in `cons_vjp` and `cons_jvp` --- src/scimlfunctions.jl | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/scimlfunctions.jl b/src/scimlfunctions.jl index 076cfc3bc..dcd32831e 100644 --- a/src/scimlfunctions.jl +++ b/src/scimlfunctions.jl @@ -1780,7 +1780,7 @@ $(TYPEDEF) A representation of an objective function `f`, defined by: ```math -\\min_{u} f(u,p) +\\min_{u} f(x,p) ``` and all of its related functions, such as the gradient of `f`, its Hessian, @@ -1807,7 +1807,7 @@ OptimizationFunction{iip}(f, adtype::AbstractADType = NoAD(); ## Positional Arguments -- `f(u,p,args...)`: the function to optimize. `u` are the optimization variables and `p` are parameters used in definition of +- `f(x,p,args...)`: the function to optimize. `x` are the optimization variables and `p` are parameters used in definition of the objective, even if no such parameters are used in the objective it should be an argument in the function. This can also take any additional arguments that are relevant to the objective function, for example minibatches used in machine learning, take a look at the minibatching tutorial [here](https://docs.sciml.ai/Optimization/stable/tutorials/minibatch/). This should return @@ -1817,12 +1817,12 @@ function described in [Callback Functions](https://docs.sciml.ai/Optimization/st ## Keyword Arguments -- `grad(G,u,p)` or `G=grad(u,p)`: the gradient of `f` with respect to `u`. If `f` takes additional arguments - then `grad(G,u,p,args...)` or `G=grad(u,p,args...)` should be used. -- `hess(H,u,p)` or `H=hess(u,p)`: the Hessian of `f` with respect to `u`. If `f` takes additional arguments - then `hess(H,u,p,args...)` or `H=hess(u,p,args...)` should be used. -- `hv(Hv,u,v,p)` or `Hv=hv(u,v,p)`: the Hessian-vector product ``(d^2 f / du^2) v``. If `f` takes additional arguments - then `hv(Hv,u,v,p,args...)` or `Hv=hv(u,v,p, args...)` should be used. +- `grad(G,x,p)` or `G=grad(x,p)`: the gradient of `f` with respect to `x`. If `f` takes additional arguments + then `grad(G,x,p,args...)` or `G=grad(x,p,args...)` should be used. +- `hess(H,x,p)` or `H=hess(x,p)`: the Hessian of `f` with respect to `x`. If `f` takes additional arguments + then `hess(H,x,p,args...)` or `H=hess(x,p,args...)` should be used. +- `hv(Hv,x,v,p)` or `Hv=hv(x,v,p)`: the Hessian-vector product ``(d^2 f / dx^2) v``. If `f` takes additional arguments + then `hv(Hv,x,v,p,args...)` or `Hv=hv(x,v,p, args...)` should be used. - `cons(res,x,p)` or `res=cons(x,p)` : the constraints function, should mutate the passed `res` array with value of the `i`th constraint, evaluated at the current values of variables inside the optimization routine. This takes just the function evaluations @@ -1830,8 +1830,8 @@ function described in [Callback Functions](https://docs.sciml.ai/Optimization/st bounds passed as `lcons` and `ucons` to [`OptimizationProblem`](@ref), in case of equality constraints `lcons` and `ucons` should be passed equal values. - `cons_j(J,x,p)` or `J=cons_j(x,p)`: the Jacobian of the constraints. -- `cons_jvp(Jv,v,x,p)` or `Jv=cons_jvp(v,x,p)`: the Jacobian-vector product of the constraints. -- `cons_vjp(Jv,v,x,p)` or `Jv=cons_vjp(v,x,p)`: the Jacobian-vector product of the constraints. +- `cons_jvp(Jv,x,v,p)` or `Jv=cons_jvp(x,v,p)`: the Jacobian-vector product of the constraints. +- `cons_vjp(Jv,x,v,p)` or `Jv=cons_vjp(x,v,p)`: the Jacobian-vector product of the constraints. - `cons_h(H,x,p)` or `H=cons_h(x,p)`: the Hessian of the constraints, provided as an array of Hessians with `res[i]` being the Hessian with respect to the `i`th output on `cons`. - `hess_prototype`: a prototype matrix matching the type that matches the Hessian. For example, From 7bb16888310a36aff843b1f12cea59d5d170b686 Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Tue, 27 Aug 2024 23:55:09 -0400 Subject: [PATCH 02/10] `getproperty(::OptimziationProblem, :x)` --- src/problems/optimization_problems.jl | 34 ++++++++++++++------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/src/problems/optimization_problems.jl b/src/problems/optimization_problems.jl index c9dd315df..e0dda45d3 100644 --- a/src/problems/optimization_problems.jl +++ b/src/problems/optimization_problems.jl @@ -8,16 +8,16 @@ Documentation Page: https://docs.sciml.ai/Optimization/stable/API/optimization_p ## Mathematical Specification of an Optimization Problem To define an optimization problem, you need the objective function ``f`` -which is minimized over the domain of ``u``, the collection of optimization variables: +which is minimized over the domain of ``x``, the collection of optimization variables: ```math -min_u f(u,p) +min_u f(x,p) ``` -``u₀`` is an initial guess for the minimizer. `f` should be specified as `f(u,p)` +``x₀`` is an initial guess for the minimizer. `f` should be specified as `f(x,p)` and `u₀` should be an `AbstractArray` whose geometry matches the -desired geometry of `u`. Note that we are not limited to vectors -for `u₀`; one is allowed to provide `u₀` as arbitrary matrices / +desired geometry of `x`. Note that we are not limited to vectors +for `x₀`; one is allowed to provide `x₀` as arbitrary matrices / higher-dimension tensors as well. ## Problem Type @@ -25,7 +25,7 @@ higher-dimension tensors as well. ### Constructors ```julia -OptimizationProblem{iip}(f, u0, p = SciMLBase.NullParameters(),; +OptimizationProblem{iip}(f, x0, p = SciMLBase.NullParameters(),; lb = nothing, ub = nothing, lcons = nothing, @@ -44,12 +44,12 @@ will be used, which will throw nice errors if you try to index non-existent parameters. `lb` and `ub` are the upper and lower bounds for box constraints on the -optimization variables. They should be an `AbstractArray` matching the geometry of `u`, -where `(lb[i],ub[i])` is the box constraint (lower and upper bounds) for `u[i]`. +optimization variables. They should be an `AbstractArray` matching the geometry of `x`, +where `(lb[i],ub[i])` is the box constraint (lower and upper bounds) for `x[i]`. `lcons` and `ucons` are the upper and lower bounds in case of inequality constraints on the optimization and if they are set to be equal then it represents an equality constraint. -They should be an `AbstractArray` matching the geometry of `u`, where `(lcons[i],ucons[i])` +They should be an `AbstractArray` matching the geometry of `x`, where `(lcons[i],ucons[i])` are the lower and upper bounds for `cons[i]`. The `f` in the `OptimizationProblem` should typically be an instance of [`OptimizationFunction`](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction) @@ -65,10 +65,10 @@ Any extra keyword arguments are captured to be sent to the optimizers. ### Fields * `f`: the function in the problem. -* `u0`: the initial guess for the optimization variables. +* `x0`: the initial guess for the optimization variables. * `p`: the constant parameters used for defining the problem. Defaults to `NullParameters`. -* `lb`: the lower bounds for the optimization variables `u`. -* `ub`: the upper bounds for the optimization variables `u`. +* `lb`: the lower bounds for the optimization variables `x`. +* `ub`: the upper bounds for the optimization variables `x`. * `int`: integrality indicator for `u`. If `int[i] == true`, then `u[i]` is an integer variable. Defaults to `nothing`, implying no integrality constraints. * `lcons`: the vector of lower bounds for the constraints passed to [OptimizationFunction](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction). @@ -81,12 +81,12 @@ Any extra keyword arguments are captured to be sent to the optimizers. ## Inequality and Equality Constraints Both inequality and equality constraints are defined by the `f.cons` function in the [`OptimizationFunction`](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction) -description of the problem structure. This `f.cons` is given as a function `f.cons(u,p)` which computes -the value of the constraints at `u`. For example, take `f.cons(u,p) = u[1] - u[2]`. +description of the problem structure. This `f.cons` is given as a function `f.cons(x,p)` which computes +the value of the constraints at `x`. For example, take `f.cons(x,p) = x[1] - x[2]`. With these definitions, `lcons` and `ucons` define the bounds on the constraint that the solvers try to satisfy. -If `lcons` and `ucons` are `nothing`, then there are no constraints bounds, meaning that the constraint is satisfied when `-Inf < f.cons < Inf` (which of course is always!). If `lcons[i] = ucons[i] = 0`, then the constraint is satisfied when `f.cons(u,p)[i] = 0`, and so this implies the equality constraint `u[1] = u[2]`. If `lcons[i] = ucons[i] = a`, then ``u[1] - u[2] = a`` is the equality constraint. +If `lcons` and `ucons` are `nothing`, then there are no constraints bounds, meaning that the constraint is satisfied when `-Inf < f.cons < Inf` (which of course is always!). If `lcons[i] = ucons[i] = 0`, then the constraint is satisfied when `f.cons(x,p)[i] = 0`, and so this implies the equality constraint `u[1] = u[2]`. If `lcons[i] = ucons[i] = a`, then ``x[1] - x[2] = a`` is the equality constraint. -Inequality constraints are then given by making `lcons[i] != ucons[i]`. For example, `lcons[i] = -Inf` and `ucons[i] = 0` would imply the inequality constraint ``u[1] <= u[2]`` since any `f.cons[i] <= 0` satisfies the constraint. Similarly, `lcons[i] = -1` and `ucons[i] = 1` would imply that `-1 <= f.cons[i] <= 1` is required or ``-1 <= u[1] - u[2] <= 1``. +Inequality constraints are then given by making `lcons[i] != ucons[i]`. For example, `lcons[i] = -Inf` and `ucons[i] = 0` would imply the inequality constraint ``x[1] <= x[2]`` since any `f.cons[i] <= 0` satisfies the constraint. Similarly, `lcons[i] = -1` and `ucons[i] = 1` would imply that `-1 <= f.cons[i] <= 1` is required or ``-1 <= x[1] - x[2] <= 1``. Note that these vectors must be sized to match the number of constraints, with one set of conditions for each constraint. @@ -150,3 +150,5 @@ end isinplace(f::OptimizationFunction{iip}) where {iip} = iip isinplace(f::OptimizationProblem{iip}) where {iip} = iip + +Base.getproperty(prob::OptimizationProblem, sym::Symbol) = sym == :x0 ? getfield(prob, :u0) : getfield(prob, sym) From c232aecbfa7fa2cbdc9ffc051c9baf2875ac943e Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Tue, 27 Aug 2024 23:57:07 -0400 Subject: [PATCH 03/10] `x` getproperty in optsol --- src/solutions/optimization_solutions.jl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/solutions/optimization_solutions.jl b/src/solutions/optimization_solutions.jl index b7daeca99..cf2877ca9 100644 --- a/src/solutions/optimization_solutions.jl +++ b/src/solutions/optimization_solutions.jl @@ -211,6 +211,8 @@ Base.@propagate_inbounds function Base.getproperty(x::AbstractOptimizationSoluti Base.depwarn("`sol.minimizer` is deprecated. Use `sol.u` instead.", "sol.minimizer") return getfield(x, :u) + elseif s === :x + return getfield(x, :u) elseif s === :minimum Base.depwarn("`sol.minimum` is deprecated. Use `sol.objective` instead.", "sol.minimum") From 74c316984aa03b6103ef0139776aa81c06c3c052 Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Tue, 27 Aug 2024 23:58:01 -0400 Subject: [PATCH 04/10] Update optimization_solutions.jl --- src/solutions/optimization_solutions.jl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/solutions/optimization_solutions.jl b/src/solutions/optimization_solutions.jl index cf2877ca9..0ea8f6753 100644 --- a/src/solutions/optimization_solutions.jl +++ b/src/solutions/optimization_solutions.jl @@ -49,14 +49,14 @@ Representation of the solution to a non-linear optimization defined by an Optimi ## Fields -- `u`: the representation of the optimization's solution. +- `x`: the representation of the optimization's solution. - `cache::AbstractOptimizationCache`: the optimization cache` that was solved. - `alg`: the algorithm type used by the solver. - `objective`: Objective value of the solution - `retcode`: the return code from the solver. Used to determine whether the solver solved successfully or whether it exited due to an error. For more details, see [the return code documentation](https://docs.sciml.ai/SciMLBase/stable/interfaces/Solutions/#retcodes). -- `original`: if the solver is wrapped from an alternative solver ecosystem, such as +- `original`: if the solver is wrapped from a external solver, e.g. Optim.jl, then this is the original return from said solver library. - `stats`: statistics of the solver, such as the number of function evaluations required. """ From b7dd4fdfc41cb980ab1373dc2f7f92bed95fd89f Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Tue, 27 Aug 2024 23:58:48 -0400 Subject: [PATCH 05/10] Update src/problems/optimization_problems.jl --- src/problems/optimization_problems.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/problems/optimization_problems.jl b/src/problems/optimization_problems.jl index e0dda45d3..bd7c8ea06 100644 --- a/src/problems/optimization_problems.jl +++ b/src/problems/optimization_problems.jl @@ -151,4 +151,4 @@ end isinplace(f::OptimizationFunction{iip}) where {iip} = iip isinplace(f::OptimizationProblem{iip}) where {iip} = iip -Base.getproperty(prob::OptimizationProblem, sym::Symbol) = sym == :x0 ? getfield(prob, :u0) : getfield(prob, sym) +Base.getproperty(prob::OptimizationProblem, sym::Symbol) = sym === :x0 ? getfield(prob, :u0) : getfield(prob, sym) From d53332090ab4ea6ad5dcf19d393ee641927c77f4 Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Wed, 28 Aug 2024 01:54:21 -0400 Subject: [PATCH 06/10] Update src/problems/optimization_problems.jl Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- src/problems/optimization_problems.jl | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/problems/optimization_problems.jl b/src/problems/optimization_problems.jl index bd7c8ea06..1e8932ef4 100644 --- a/src/problems/optimization_problems.jl +++ b/src/problems/optimization_problems.jl @@ -151,4 +151,6 @@ end isinplace(f::OptimizationFunction{iip}) where {iip} = iip isinplace(f::OptimizationProblem{iip}) where {iip} = iip -Base.getproperty(prob::OptimizationProblem, sym::Symbol) = sym === :x0 ? getfield(prob, :u0) : getfield(prob, sym) +function Base.getproperty(prob::OptimizationProblem, sym::Symbol) + sym === :x0 ? getfield(prob, :u0) : getfield(prob, sym) +end From 4f80de9d04527e75ae1e1499cb2bf32354af62bc Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Wed, 28 Aug 2024 19:55:54 -0400 Subject: [PATCH 07/10] Update remake_tests.jl --- test/remake_tests.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/remake_tests.jl b/test/remake_tests.jl index ad679bc61..2e23f963d 100644 --- a/test/remake_tests.jl +++ b/test/remake_tests.jl @@ -249,7 +249,7 @@ for prob in deepcopy(probs) function fakeloss!(p) prob2 = @inferred baseType remake(prob; p = [:a => p]) @test eltype(prob2.p) <: ForwardDiff.Dual - return prob2.ps[:a] + return prob2.p[:a] end ForwardDiff.derivative(fakeloss!, 1.0) end From 6682a7100b1896ac307e33b3837355f340ff174f Mon Sep 17 00:00:00 2001 From: Vaibhav Dixit Date: Wed, 28 Aug 2024 20:18:03 -0400 Subject: [PATCH 08/10] revert all u to x changes --- src/problems/optimization_problems.jl | 36 ++++++++++++--------------- src/scimlfunctions.jl | 28 ++++++++++----------- src/solutions/ode_solutions.jl | 7 ++++-- test/remake_tests.jl | 2 +- 4 files changed, 36 insertions(+), 37 deletions(-) diff --git a/src/problems/optimization_problems.jl b/src/problems/optimization_problems.jl index 1e8932ef4..c9dd315df 100644 --- a/src/problems/optimization_problems.jl +++ b/src/problems/optimization_problems.jl @@ -8,16 +8,16 @@ Documentation Page: https://docs.sciml.ai/Optimization/stable/API/optimization_p ## Mathematical Specification of an Optimization Problem To define an optimization problem, you need the objective function ``f`` -which is minimized over the domain of ``x``, the collection of optimization variables: +which is minimized over the domain of ``u``, the collection of optimization variables: ```math -min_u f(x,p) +min_u f(u,p) ``` -``x₀`` is an initial guess for the minimizer. `f` should be specified as `f(x,p)` +``u₀`` is an initial guess for the minimizer. `f` should be specified as `f(u,p)` and `u₀` should be an `AbstractArray` whose geometry matches the -desired geometry of `x`. Note that we are not limited to vectors -for `x₀`; one is allowed to provide `x₀` as arbitrary matrices / +desired geometry of `u`. Note that we are not limited to vectors +for `u₀`; one is allowed to provide `u₀` as arbitrary matrices / higher-dimension tensors as well. ## Problem Type @@ -25,7 +25,7 @@ higher-dimension tensors as well. ### Constructors ```julia -OptimizationProblem{iip}(f, x0, p = SciMLBase.NullParameters(),; +OptimizationProblem{iip}(f, u0, p = SciMLBase.NullParameters(),; lb = nothing, ub = nothing, lcons = nothing, @@ -44,12 +44,12 @@ will be used, which will throw nice errors if you try to index non-existent parameters. `lb` and `ub` are the upper and lower bounds for box constraints on the -optimization variables. They should be an `AbstractArray` matching the geometry of `x`, -where `(lb[i],ub[i])` is the box constraint (lower and upper bounds) for `x[i]`. +optimization variables. They should be an `AbstractArray` matching the geometry of `u`, +where `(lb[i],ub[i])` is the box constraint (lower and upper bounds) for `u[i]`. `lcons` and `ucons` are the upper and lower bounds in case of inequality constraints on the optimization and if they are set to be equal then it represents an equality constraint. -They should be an `AbstractArray` matching the geometry of `x`, where `(lcons[i],ucons[i])` +They should be an `AbstractArray` matching the geometry of `u`, where `(lcons[i],ucons[i])` are the lower and upper bounds for `cons[i]`. The `f` in the `OptimizationProblem` should typically be an instance of [`OptimizationFunction`](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction) @@ -65,10 +65,10 @@ Any extra keyword arguments are captured to be sent to the optimizers. ### Fields * `f`: the function in the problem. -* `x0`: the initial guess for the optimization variables. +* `u0`: the initial guess for the optimization variables. * `p`: the constant parameters used for defining the problem. Defaults to `NullParameters`. -* `lb`: the lower bounds for the optimization variables `x`. -* `ub`: the upper bounds for the optimization variables `x`. +* `lb`: the lower bounds for the optimization variables `u`. +* `ub`: the upper bounds for the optimization variables `u`. * `int`: integrality indicator for `u`. If `int[i] == true`, then `u[i]` is an integer variable. Defaults to `nothing`, implying no integrality constraints. * `lcons`: the vector of lower bounds for the constraints passed to [OptimizationFunction](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction). @@ -81,12 +81,12 @@ Any extra keyword arguments are captured to be sent to the optimizers. ## Inequality and Equality Constraints Both inequality and equality constraints are defined by the `f.cons` function in the [`OptimizationFunction`](https://docs.sciml.ai/Optimization/stable/API/optimization_function/#optfunction) -description of the problem structure. This `f.cons` is given as a function `f.cons(x,p)` which computes -the value of the constraints at `x`. For example, take `f.cons(x,p) = x[1] - x[2]`. +description of the problem structure. This `f.cons` is given as a function `f.cons(u,p)` which computes +the value of the constraints at `u`. For example, take `f.cons(u,p) = u[1] - u[2]`. With these definitions, `lcons` and `ucons` define the bounds on the constraint that the solvers try to satisfy. -If `lcons` and `ucons` are `nothing`, then there are no constraints bounds, meaning that the constraint is satisfied when `-Inf < f.cons < Inf` (which of course is always!). If `lcons[i] = ucons[i] = 0`, then the constraint is satisfied when `f.cons(x,p)[i] = 0`, and so this implies the equality constraint `u[1] = u[2]`. If `lcons[i] = ucons[i] = a`, then ``x[1] - x[2] = a`` is the equality constraint. +If `lcons` and `ucons` are `nothing`, then there are no constraints bounds, meaning that the constraint is satisfied when `-Inf < f.cons < Inf` (which of course is always!). If `lcons[i] = ucons[i] = 0`, then the constraint is satisfied when `f.cons(u,p)[i] = 0`, and so this implies the equality constraint `u[1] = u[2]`. If `lcons[i] = ucons[i] = a`, then ``u[1] - u[2] = a`` is the equality constraint. -Inequality constraints are then given by making `lcons[i] != ucons[i]`. For example, `lcons[i] = -Inf` and `ucons[i] = 0` would imply the inequality constraint ``x[1] <= x[2]`` since any `f.cons[i] <= 0` satisfies the constraint. Similarly, `lcons[i] = -1` and `ucons[i] = 1` would imply that `-1 <= f.cons[i] <= 1` is required or ``-1 <= x[1] - x[2] <= 1``. +Inequality constraints are then given by making `lcons[i] != ucons[i]`. For example, `lcons[i] = -Inf` and `ucons[i] = 0` would imply the inequality constraint ``u[1] <= u[2]`` since any `f.cons[i] <= 0` satisfies the constraint. Similarly, `lcons[i] = -1` and `ucons[i] = 1` would imply that `-1 <= f.cons[i] <= 1` is required or ``-1 <= u[1] - u[2] <= 1``. Note that these vectors must be sized to match the number of constraints, with one set of conditions for each constraint. @@ -150,7 +150,3 @@ end isinplace(f::OptimizationFunction{iip}) where {iip} = iip isinplace(f::OptimizationProblem{iip}) where {iip} = iip - -function Base.getproperty(prob::OptimizationProblem, sym::Symbol) - sym === :x0 ? getfield(prob, :u0) : getfield(prob, sym) -end diff --git a/src/scimlfunctions.jl b/src/scimlfunctions.jl index dcd32831e..d3b0ef651 100644 --- a/src/scimlfunctions.jl +++ b/src/scimlfunctions.jl @@ -1780,7 +1780,7 @@ $(TYPEDEF) A representation of an objective function `f`, defined by: ```math -\\min_{u} f(x,p) +\\min_{u} f(u,p) ``` and all of its related functions, such as the gradient of `f`, its Hessian, @@ -1807,7 +1807,7 @@ OptimizationFunction{iip}(f, adtype::AbstractADType = NoAD(); ## Positional Arguments -- `f(x,p,args...)`: the function to optimize. `x` are the optimization variables and `p` are parameters used in definition of +- `f(u,p,args...)`: the function to optimize. `u` are the optimization variables and `p` are parameters used in definition of the objective, even if no such parameters are used in the objective it should be an argument in the function. This can also take any additional arguments that are relevant to the objective function, for example minibatches used in machine learning, take a look at the minibatching tutorial [here](https://docs.sciml.ai/Optimization/stable/tutorials/minibatch/). This should return @@ -1817,22 +1817,22 @@ function described in [Callback Functions](https://docs.sciml.ai/Optimization/st ## Keyword Arguments -- `grad(G,x,p)` or `G=grad(x,p)`: the gradient of `f` with respect to `x`. If `f` takes additional arguments - then `grad(G,x,p,args...)` or `G=grad(x,p,args...)` should be used. -- `hess(H,x,p)` or `H=hess(x,p)`: the Hessian of `f` with respect to `x`. If `f` takes additional arguments - then `hess(H,x,p,args...)` or `H=hess(x,p,args...)` should be used. -- `hv(Hv,x,v,p)` or `Hv=hv(x,v,p)`: the Hessian-vector product ``(d^2 f / dx^2) v``. If `f` takes additional arguments - then `hv(Hv,x,v,p,args...)` or `Hv=hv(x,v,p, args...)` should be used. -- `cons(res,x,p)` or `res=cons(x,p)` : the constraints function, should mutate the passed `res` array +- `grad(G,u,p)` or `G=grad(u,p)`: the gradient of `f` with respect to `u`. If `f` takes additional arguments + then `grad(G,u,p,args...)` or `G=grad(u,p,args...)` should be used. +- `hess(H,u,p)` or `H=hess(u,p)`: the Hessian of `f` with respect to `u`. If `f` takes additional arguments + then `hess(H,u,p,args...)` or `H=hess(u,p,args...)` should be used. +- `hv(Hv,u,v,p)` or `Hv=hv(u,v,p)`: the Hessian-vector product ``(d^2 f / dx^2) v``. If `f` takes additional arguments + then `hv(Hv,u,v,p,args...)` or `Hv=hv(u,v,p, args...)` should be used. +- `cons(res,u,p)` or `res=cons(u,p)` : the constraints function, should mutate the passed `res` array with value of the `i`th constraint, evaluated at the current values of variables inside the optimization routine. This takes just the function evaluations and the equality or inequality assertion is applied by the solver based on the constraint bounds passed as `lcons` and `ucons` to [`OptimizationProblem`](@ref), in case of equality constraints `lcons` and `ucons` should be passed equal values. -- `cons_j(J,x,p)` or `J=cons_j(x,p)`: the Jacobian of the constraints. -- `cons_jvp(Jv,x,v,p)` or `Jv=cons_jvp(x,v,p)`: the Jacobian-vector product of the constraints. -- `cons_vjp(Jv,x,v,p)` or `Jv=cons_vjp(x,v,p)`: the Jacobian-vector product of the constraints. -- `cons_h(H,x,p)` or `H=cons_h(x,p)`: the Hessian of the constraints, provided as +- `cons_j(J,u,p)` or `J=cons_j(u,p)`: the Jacobian of the constraints. +- `cons_jvp(Jv,u,v,p)` or `Jv=cons_jvp(u,v,p)`: the Jacobian-vector product of the constraints. +- `cons_vjp(Jv,u,v,p)` or `Jv=cons_vjp(u,v,p)`: the Jacobian-vector product of the constraints. +- `cons_h(H,u,p)` or `H=cons_h(u,p)`: the Hessian of the constraints, provided as an array of Hessians with `res[i]` being the Hessian with respect to the `i`th output on `cons`. - `hess_prototype`: a prototype matrix matching the type that matches the Hessian. For example, if the Hessian is tridiagonal, then an appropriately sized `Hessian` matrix can be used @@ -1845,7 +1845,7 @@ function described in [Callback Functions](https://docs.sciml.ai/Optimization/st This is defined as an array of matrices, where `hess[i]` is the Hessian w.r.t. the `i`th output. For example, if the Hessian is sparse, then `hess` is a `Vector{SparseMatrixCSC}`. The default is `nothing`, which means a dense constraint Hessian. -- `lag_h(res,x,sigma,mu,p)` or `res=lag_h(x,sigma,mu,p)`: the Hessian of the Lagrangian, +- `lag_h(res,u,sigma,mu,p)` or `res=lag_h(u,sigma,mu,p)`: the Hessian of the Lagrangian, where `sigma` is a multiplier of the cost function and `mu` are the Lagrange multipliers multiplying the constraints. This can be provided instead of `hess` and `cons_h` to solvers that directly use the Hessian of the Lagrangian. diff --git a/src/solutions/ode_solutions.jl b/src/solutions/ode_solutions.jl index 06df720af..e4d0deda5 100644 --- a/src/solutions/ode_solutions.jl +++ b/src/solutions/ode_solutions.jl @@ -605,11 +605,14 @@ function sensitivity_solution(sol::ODESolution, u, t) return @set sol.interp = interp end -struct LazyInterpolationException <: Exception +struct LazyInterpolationException <: Exception var::Symbol end -Base.showerror(io::IO, e::LazyInterpolationException) = print(io, "The algorithm", e.var, " uses lazy interpolation, which is incompatible with `strip_solution`.") +function Base.showerror(io::IO, e::LazyInterpolationException) + print(io, "The algorithm", e.var, + " uses lazy interpolation, which is incompatible with `strip_solution`.") +end function strip_solution(sol::ODESolution) if has_lazy_interpolation(sol.alg) diff --git a/test/remake_tests.jl b/test/remake_tests.jl index 2e23f963d..ad679bc61 100644 --- a/test/remake_tests.jl +++ b/test/remake_tests.jl @@ -249,7 +249,7 @@ for prob in deepcopy(probs) function fakeloss!(p) prob2 = @inferred baseType remake(prob; p = [:a => p]) @test eltype(prob2.p) <: ForwardDiff.Dual - return prob2.p[:a] + return prob2.ps[:a] end ForwardDiff.derivative(fakeloss!, 1.0) end From d0632f89d59f50fe5960f04e27fdd45b254b31c6 Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Wed, 28 Aug 2024 20:18:58 -0400 Subject: [PATCH 09/10] Update src/scimlfunctions.jl --- src/scimlfunctions.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scimlfunctions.jl b/src/scimlfunctions.jl index d3b0ef651..cb242a2b4 100644 --- a/src/scimlfunctions.jl +++ b/src/scimlfunctions.jl @@ -1821,7 +1821,7 @@ function described in [Callback Functions](https://docs.sciml.ai/Optimization/st then `grad(G,u,p,args...)` or `G=grad(u,p,args...)` should be used. - `hess(H,u,p)` or `H=hess(u,p)`: the Hessian of `f` with respect to `u`. If `f` takes additional arguments then `hess(H,u,p,args...)` or `H=hess(u,p,args...)` should be used. -- `hv(Hv,u,v,p)` or `Hv=hv(u,v,p)`: the Hessian-vector product ``(d^2 f / dx^2) v``. If `f` takes additional arguments +- `hv(Hv,u,v,p)` or `Hv=hv(u,v,p)`: the Hessian-vector product ``(d^2 f / du^2) v``. If `f` takes additional arguments then `hv(Hv,u,v,p,args...)` or `Hv=hv(u,v,p, args...)` should be used. - `cons(res,u,p)` or `res=cons(u,p)` : the constraints function, should mutate the passed `res` array with value of the `i`th constraint, evaluated at the current values of variables From 0c8bb2fec1ab9ddb32bf4c72750c1dfc989c9c5e Mon Sep 17 00:00:00 2001 From: Vaibhav Kumar Dixit Date: Wed, 28 Aug 2024 20:19:33 -0400 Subject: [PATCH 10/10] Update src/solutions/optimization_solutions.jl --- src/solutions/optimization_solutions.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/solutions/optimization_solutions.jl b/src/solutions/optimization_solutions.jl index 0ea8f6753..a250ebf32 100644 --- a/src/solutions/optimization_solutions.jl +++ b/src/solutions/optimization_solutions.jl @@ -49,7 +49,7 @@ Representation of the solution to a non-linear optimization defined by an Optimi ## Fields -- `x`: the representation of the optimization's solution. +- `u`: the representation of the optimization's solution. - `cache::AbstractOptimizationCache`: the optimization cache` that was solved. - `alg`: the algorithm type used by the solver. - `objective`: Objective value of the solution