How to add config schemas in pytest (or avoid them altogether)

[hogru]

Sorry, another one… in my program I add validation like so:

cs = ConfigStore.instance()
cs.store(name="config_schema", node=MyConfig)

@hydra.main(config_path="../conf", config_name="config", version_base=None)
def main(cfg: DictConfig) -> None:
    cfg_instance = OmegaConf.to_object(cfg)

The corresponding config.yaml:

defaults:
  - config_schema
  - ...

This works. Now I want to read the hydra config in pytest

@pytest.fixture
def hydra_config():
    # Not sure if the context is the right approach here; or the method: don't know how often fixtures are called
    with initialize(version_base=None, config_path="../src/conf", job_name="test_data"):
       cfg = compose(config_name="config")
     return cfg

I get this error: hydra.errors.MissingConfigException: In 'config': Could not load 'config_schema'

So, I guess the question is: What is the “equivalent” of cs.store() in the pytest context? Or tell hydra that I'm not concerned about the schema in this context? Can I override a defaults list (I tried, but that did not change the error and I feel that there's a more elegant way)?

[Jasha10]

So, I guess the question is: What is the “equivalent” of cs.store() in the pytest context?

cs.store() should work the same way in the for initialize+compose as it does for @hydra.main.
The key is to make sure that you call cs.store before you call compose. I suspect that in your case, the call to cs.store is not being run.

---

If your call to cs.store is at the top level of my_configs.py, you could import the my_configs module in your pytest fixture. That should be enough to guarantee that the config_schema node get stored.

# my_configs.py
...
cs = ConfigStore.instance()
cs.store(name="config_schema", node=MyConfig)
...

# test_my_configs.py
...
@pytest.fixture
def hydra_config():
    import my_configs  # to make sure the cs.store calls are executed
    with initialize(version_base=None, config_path="../src/conf", job_name="test_data"):
       cfg = compose(config_name="config")
     return cfg

---

Another approach is to put all of your calls to cs.store inside a function (e.g. store_configs). You'd then call that function before running @hydra.main or compose:

# my_configs.py
...
def store_configs():
    cs = ConfigStore.instance()
    cs.store(name="config_schema", node=MyConfig)
...

# test_my_configs.py
from my_configs import store_configs
...
@pytest.fixture
def hydra_config():
    store_configs()  # execute cs.store calls
    with initialize(version_base=None, config_path="../src/conf", job_name="test_data"):
       cfg = compose(config_name="config")
     return cfg

# my_app.py
from my_configs import store_configs
...
@hydra.main(...)
def run_app(cfg):
    ...

if __name__ == "__main__":
    store_configs()
    run_app()

---

# Not sure if the context is the right approach here; or the method: don't know how often fixtures are called

You can control how often fixtures are called via pytest's fixture scope settings.

[hogru]

A big thank you again, @Jasha10

I think I understand what you're saying and did implement it. I can access hardcoded config values. But the variable interpolation does not seem to run. I have tried a couple of things (see code below) and researched a bit (this issue seems to be related?).

The code:

@pytest.fixture  # (scope='session', autouse=True)
def hydra_config():
    import molgen.config
    with initialize(version_base=None, config_path="../src/conf", job_name="test_data"):
        cfg = compose(config_name="config", return_hydra_config=True)  # Tried out return_hydra_config
    OmegaConf.resolve(cfg)  # tried this in different places
    return cfg

def test_hydra(hydra_config):
    # I can access "hard-coded" config values here
    # But this value needs to be interpolated
    # in config.yaml: data_raw: ${path.data_raw}/${dataset.raw_file}
    print(hydra_config.file.data_raw)  # accessing it raises the exceptions (see below)

These are the last (and relevant?) lines of the error message when running the test in pytest:

    @staticmethod
    def get() -> HydraConf:
        instance = HydraConfig.instance()
        if instance.cfg is None:
>           raise ValueError("HydraConfig was not set")
E           omegaconf.errors.InterpolationResolutionError: ValueError raised while resolving interpolation: HydraConfig was not set

/Users/stephan/Library/Caches/pypoetry/virtualenvs/molgen-KF7BYZEi-py3.9/lib/python3.9/site-packages/hydra/core/hydra_config.py:31: InterpolationResolutionError

As long as you don't lose your patience with me I am trying to understand as much as I can. I am coding for my AI master thesis and the tricky part - integrating all sorts of frameworks (pytorch lightning, wandb, hydra, ...) will be a challenge, esp. doing sweeps, hyper-param searches.

[Jasha10]

(this issue seems to be related?).

Yes, it is indeed related.
Ordinary interpolations should work (i.e. interpolations like ${my_config.foo}), but interpolations using the Hydra resolver are not supported by the compose API.

As long as you don't lose your patience with me I am trying to understand as much as I can.

I'm happy to help :)
FYI I will be on vacation next week though and will not be around to answer questions.

[hogru]

@Jasha10, hope you enjoyed your vacation :-)

I think I can live without the Hydra resolver, used it for accessing the cwd only (I needed it only until Hydra 1.1), and I can do that with HydraConfig.get().runtime.cwd (or so) in the main code anyway. The (py)tests do work now.

[Jasha10]

Fyi something like import os; os.getcwd() should work too!