Skip to content

Commit

Permalink
fix package name issue in documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
qiauil committed Aug 27, 2024
1 parent d5d7e5d commit 1099605
Show file tree
Hide file tree
Showing 7 changed files with 29 additions and 29 deletions.
12 changes: 6 additions & 6 deletions docs/api/grad_operator.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
The `grad_operator` module contains the main operators of ConFIG algorithm. You can use these operators to perform the ConFIG update step for your optimization problem.

## Operation Functions
::: ConFIG.grad_operator.ConFIG_update
::: ConFIG.grad_operator.ConFIG_update_double
::: conflictfree.grad_operator.ConFIG_update
::: conflictfree.grad_operator.ConFIG_update_double

## Operator Classes
::: ConFIG.grad_operator.ConFIGOperator
::: ConFIG.grad_operator.PCGradOperator
::: ConFIG.grad_operator.IMTLGOperator
::: conflictfree.grad_operator.ConFIGOperator
::: conflictfree.grad_operator.PCGradOperator
::: conflictfree.grad_operator.IMTLGOperator

## Base Class of Operators
::: ConFIG.grad_operator.GradientOperator
::: conflictfree.grad_operator.GradientOperator
4 changes: 2 additions & 2 deletions docs/api/length_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ The `length_model` module contains classes for rescaling the magnitude of the fi
The `ProjectionLength` class is the default length model for the ConFIG algorithm. You can create a custom length model by inheriting from the `LengthModel` class.

## Length Model
::: ConFIG.length_model.ProjectionLength
::: conflictfree.length_model.ProjectionLength

## Base Class of Length Model
::: ConFIG.length_model.LengthModel
::: conflictfree.length_model.LengthModel
6 changes: 3 additions & 3 deletions docs/api/loss_recorder.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ The `loss_recorder` module contains classes for recording the loss values during
It is used in the momentum version of the ConFIG algorithm to record the loss values. Not every loss is calculated in a single iteration with the momentum version of the ConFIG algorithm. However, sometimes we need to know the information of all the loss values, e.g., logging and calculating length/weight model. You can create a custom loss recorder by inheriting from the `LossRecorder` class.

## Loss Recorder
::: ConFIG.loss_recorder.LatestLossRecorder
::: ConFIG.loss_recorder.MomentumLossRecorder
::: conflictfree.loss_recorder.LatestLossRecorder
::: conflictfree.loss_recorder.MomentumLossRecorder

## Base Class of Loss Recorder
::: ConFIG.loss_recorder.LossRecorder
::: conflictfree.loss_recorder.LossRecorder
6 changes: 3 additions & 3 deletions docs/api/momentum_operator.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
The `momentum_operator` module contains the main operators for the momentum version ConFIG algorithm.

## Operator Classes
::: ConFIG.momentum_operator.PseudoMomentumOperator
::: ConFIG.momentum_operator.SeparateMomentumOperator
::: conflictfree.momentum_operator.PseudoMomentumOperator
::: conflictfree.momentum_operator.SeparateMomentumOperator

## Base Class of Operators
::: ConFIG.momentum_operator.LatestLossRecorder
::: conflictfree.momentum_operator.LatestLossRecorder
20 changes: 10 additions & 10 deletions docs/api/utils.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
The `utils` module contains utility functions for the ConFIG algorithm.

## Network Utility Functions
::: ConFIG.utils.get_para_vector
::: ConFIG.utils.apply_para_vector
::: ConFIG.utils.get_gradient_vector
::: ConFIG.utils.apply_gradient_vector
::: conflictfree.utils.get_para_vector
::: conflictfree.utils.apply_para_vector
::: conflictfree.utils.get_gradient_vector
::: conflictfree.utils.apply_gradient_vector

## Math Utility Functions
::: ConFIG.utils.get_cos_similarity
::: ConFIG.utils.unit_vector
::: conflictfree.utils.get_cos_similarity
::: conflictfree.utils.unit_vector

## Conflict Utility Functions
::: ConFIG.utils.estimate_conflict
::: conflictfree.utils.estimate_conflict

## Slice Selector Classes
::: ConFIG.utils.OrderedSliceSelector
::: ConFIG.utils.RandomSliceSelector
::: conflictfree.utils.OrderedSliceSelector
::: conflictfree.utils.RandomSliceSelector

## Others
::: ConFIG.utils.has_zero
::: conflictfree.utils.has_zero
4 changes: 2 additions & 2 deletions docs/api/weight_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ The `weight_model` module contains classes for calculating the direction weight
The `EqualWeight` class is the default weight model for the ConFIG algorithm. You can create a custom weight model by inheriting from the `WeightModel` class.

## Weight Model
::: ConFIG.weight_model.EqualWeight
::: conflictfree.weight_model.EqualWeight

## Base Class of Weight Model
::: ConFIG.weight_model.WeightModel
::: conflictfree.weight_model.WeightModel
6 changes: 3 additions & 3 deletions docs/start/theory.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ $$
[\mathcal{U}(\mathbf{g}_1),\mathcal{U}(\mathbf{g}_2),\cdots, \mathcal{U}(\mathbf{g}_m)]^{-\top} \mathbf{1}_m\right].
$$

Here, $\mathbf{1}_m$ is a unit vector with $m$ components. These two equations are implemented as [ConFIG.grad_operator.ConFIG_update()](../../api/grad_operator/#ConFIG.grad_operator.ConFIG_update) and [ConFIG.grad_operator.ConFIGOperator.calculate_gradient()](../../api/grad_operator/#ConFIG.grad_operator.ConFIGOperator.calculate_gradient). Meanwhile, we also provide [weight_model](../../api/weight_model/) to allow you implement different direction weights ($\hat{\mathbf{w}}=\mathbf{1}_m$ as default) and [length_model](../../api/length_model/) to allow you design different length projection (the above adaptive strategy as default). We encourage you to design and try different weight/length models and compare the result with default configurations.
Here, $\mathbf{1}_m$ is a unit vector with $m$ components. These two equations are implemented as [conflictfree.grad_operator.ConFIG_update()](../../api/grad_operator/#conflictfree.grad_operator.ConFIG_update) and [conflictfree.grad_operator.ConFIGOperator.calculate_gradient()](../../api/grad_operator/#conflictfree.grad_operator.ConFIGOperator.calculate_gradient). Meanwhile, we also provide [weight_model](../../api/weight_model/) to allow you implement different direction weights ($\hat{\mathbf{w}}=\mathbf{1}_m$ as default) and [length_model](../../api/length_model/) to allow you design different length projection (the above adaptive strategy as default). We encourage you to design and try different weight/length models and compare the result with default configurations.


## ConFIG in two-loss scenario
Expand All @@ -73,15 +73,15 @@ $$

where $\mathcal{O}(\mathbf{g}_1,\mathbf{g}_2)=\mathbf{g}_2-\frac{\mathbf{g}_1^\top\mathbf{g}_2}{|\mathbf{g}1|^2}\mathbf{g}_1$ is the orthogonality operator. It returns a vector orthogonal to $\mathbf{g}_1$ from the plane spanned by $\mathbf{g}_{1}$ and $\mathbf{g}_{2}$.

This equivlance is implemented as [ConFIG.grad_operator.ConFIG_update_double()](../../api/grad_operator/#ConFIG.grad_operator.ConFIG_update_double). You can also set `allow_simplified_model` to true in [ConFIG.grad_operator.ConFIGOperator](../../api/grad_operator/#ConFIG.grad_operator.ConFIGOperator) to enable using this form in two-loss scenario.
This equivlance is implemented as [conflictfree.grad_operator.ConFIG_update_double()](../../api/grad_operator/#conflictfree.grad_operator.ConFIG_update_double). You can also set `allow_simplified_model` to true in [conflictfree.grad_operator.ConFIGOperator](../../api/grad_operator/#conflictfree.grad_operator.ConFIGOperator) to enable using this form in two-loss scenario.

## M-ConFIG

Gradient-based methods like the proposed ConFIG method require separate backpropagation steps to compute the gradient for each loss term, which could be computationally expensive. To address this issue, we introduce an accelerated momentum-based variant of ConFIG: **M-ConFIG**. Our core idea is to leverage the momentum of the gradient for the ConFIG operation and update momentum variables in an alternating fashion to avoid backpropagating all losses in a single step. In each iteration, only a single momentum is updated with its corresponding gradient, while the others are carried over from previous steps. Algorithm 1 details the entire procedure of M-ConFIG.

![M-ConFIG](../assets/algorithm.png)

The M-ConFIG method is implemented as [ConFIG.momentum_operator.PseudoMomentumOperator](../../api/momentum_operator/#ConFIG.momentum_operator.PseudoMomentumOperator). This momentum method can also be used for other gradient-based methods. In [ConFIG.momentum_operator.PseudoMomentumOperator](../../api/momentum_operator/#ConFIG.momentum_operator.PseudoMomentumOperator), you can modify the `gradient_operator` parameter to enable momentum acceleration for other methods.
The M-ConFIG method is implemented as [conflictfree.momentum_operator.PseudoMomentumOperator](../../api/momentum_operator/#conflictfree.momentum_operator.PseudoMomentumOperator). This momentum method can also be used for other gradient-based methods. In [conflictfree.momentum_operator.PseudoMomentumOperator](../../api/momentum_operator/#conflictfree.momentum_operator.PseudoMomentumOperator), you can modify the `gradient_operator` parameter to enable momentum acceleration for other methods.

---

Expand Down

0 comments on commit 1099605

Please sign in to comment.