Problem in candidate-based generation on GENRE using transformers >= 4.36.0

[y0uCeF]

Since version v4.36.0 of huggingface transformers, it is not allowed to have prefix_allowed_tokens_fn return an empty set of tokens 27797.
When doing non-free generation on GENRE, which is based on candidates, the output of the lambda assigned to prefix_allowed_tokens_fn may indeed be an empty list, raising a ValueError with the following message:

f"`prefix_allowed_tokens_fn` returned an empty list for batch ID {batch_id}."
"This means that the constraint is unsatisfiable. Please check your implementation"
f"of `prefix_allowed_tokens_fn` "

I have not reproduced the case for free generation, and it seems the code I use for per-mention Trie creation is no longer presented in examples.ipynb :

trie = Trie([
                  [2] + model.encode(e).tolist()[1:]
                  for e in doc["candidates"]
          ])
          if doc["candidates"] else Trie([[2] + self.model.encode("NIL").tolist()[1:]])

Anyway, it is recommended to stick to versions < v4.36.0 (such as v4.35.2) if one falls into that error.
This condition may be added to requirements.txt

[Klaifer]

I had this same problem with the GENRE huggingface implementation, and I developed a workaround.

I'm going to insert it here because I think it might help someone, or someone might adjust it to work in this implementation.

The implementation I'm referring to is: https://huggingface.co/facebook/mgenre-wiki

There is the following instruction:

outputs = model.generate(
    **tokenizer(sentences, return_tensors="pt"),
    num_beams=5,
    num_return_sequences=5,
    # OPTIONAL: use constrained beam search
    prefix_allowed_tokens_fn=lambda batch_id, sent: trie.get(sent.tolist()),
)

The solution is to replace with:

outputs = model.generate(
    **tokenizer(sentences, return_tensors="pt"),
    num_beams=5,
    num_return_sequences=5,
    # OPTIONAL: use constrained beam search
    prefix_allowed_tokens_fn=lambda batch_id, sent: trie.get(sent.tolist()) or [tokenizer.eos_token_id],
)