Failed to add community in final community report: Faulty Json

[hyeoungho]

I ran the Christmas story and thought it worked well. Then I used different test set and realized the community report wasn't generated at all (well there is one entry). I was curious to check the demo input's report and found below error messages. I looked at the string but don't know what's wrong in the format. But the community report was not generated for the id : 42 (there are many other communities that have similar problems).

14:12:45,197 graphrag.llm.openai.utils INFO Warning: Error decoding faulty json, attempting repair
14:12:45,209 graphrag.index.graph.extractors.community_reports.community_reports_extractor ERROR error generating community report
Traceback (most recent call last):
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/index/graph/extractors/community_reports/community_reports_extractor.py", line 58, in __call__
    await self._llm(
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/openai/json_parsing_llm.py", line 34, in __call__
    result = await self._delegate(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/openai/openai_token_replacing_llm.py", line 37, in __call__
    return await self._delegate(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/openai/openai_history_tracking_llm.py", line 33, in __call__
    output = await self._delegate(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/caching_llm.py", line 96, in __call__
    result = await self._delegate(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 177, in __call__
    result, start = await execute_with_retry()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 159, in execute_with_retry
    async for attempt in retryer:
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 165, in execute_with_retry
    return await do_attempt(), start
           ^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 147, in do_attempt
    return await self._delegate(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/base/base_llm.py", line 48, in __call__
    return await self._invoke_json(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/Work/GraphRag2/.venv/lib/python3.11/site-packages/graphrag/llm/openai/openai_chat_llm.py", line 90, in _invoke_json
    raise RuntimeError(error_msg)

RuntimeError: Failed to generate valid JSON output - Faulty JSON: {'title': "Scrooge's Community", 'summary': "The community revolves around Ebenezer Scrooge, who is visited by the Ghost of Christmas Past. The Ghost of Christmas Past shows Scrooge visions of his past, highlighting his mistakes and regrets. The community includes various entities such as the boys, bridge, church, farmers, fields, gigs, market-town, and shadows, all of which are associated with Scrooge's experiences and the Ghost of Christmas Past's visitations. The relationships between these entities are crucial in understanding the dynamics of this community. The community's structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter. The community's significance lies in its ability to provide insight into Scrooge's past and his potential for change. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge's interactions with the Ghost of Christmas Past and the entities they encounter do not appear to be illegal. The community's noteworthy claims include the ability to provide insight into an individual's past and their potential for change, as well as the ability to facilitate personal growth and transformation. The community's overall structure is centered around Scrooge's interactions with the Ghost of Christmas Past and the various entities they encounter, and the relationships between these entities are crucial in understanding the dynamics of this community. The community's impact severity rating is moderate due to the potential for personal growth and change that Scrooge may experience as a result of his interactions with the Ghost of Christmas Past and the entities they encounter. The community's reputation is that of a place of personal growth and transformation, where individuals can reflect on their past and gain a deeper understanding of themselves and their actions. The community's technical capabilities are not explicitly stated, but it can be inferred that the community relies on the supernatural abilities of the Ghost of Christmas Past to facilitate Scrooge's interactions with the various entities. The community's legal compliance is not explicitly stated, but it can be inferred that the community operates within the bounds of the law, as Scrooge"}
14:12:45,222 graphrag.index.reporting.file_workflow_callbacks INFO Community Report Extraction Error details=None
14:12:45,222 graphrag.index.verbs.graph.report.strategies.graph_intelligence.run_graph_intelligence WARNING No report found for community: 42

It looks like the issues is due to the limit of the maximum token that the LLM can handle. In this example, the returned json from the model does not contain all the field required (due to the length of the 'summary' section).

While final_community_report requires below fields:

final_community_reports.columns
Index(['community', 'full_content', 'level', 'rank', 'title',
       'rank_explanation', 'summary', 'findings', 'full_content_json', 'id'],
      dtype='object')

Is there anything that I can do to walk around the issue (other than use larger model :) )

[RainDayDream]

I have the same problem now. Did you get a solution？

[RainDayDream]

When I use the same model， I get the same error in which the erroneous community ID is the same

[javey9110]

I have the same problem now. Did you get a solution？

[hyeoungho]

Unfortunately, I haven't tried a solution for this. Like I mentioned at the later part, this is due to the limit of context window of the model that I use. I am pretty sure it will work if I use the same model that is used by the graphrag demo (OpenAI model) or any model with the same or longer context length. The alternative solution is modifying the prompt to generate community report to force the model to squeeze the report into the limit, which I haven't tried, yet.

[RainDayDream]

Thanks for your reply!

[BobbyAxelrods]

Hi , i try to test 2 model with graphrag indexing. I found out that when using gpt 4o it work wells but fail for mixtra 8x7b due to faulty json. Same issue due to truncated output by mixtral where the json output not as per expected.

Mixtral 8x7B:
Context Window Size: 32,000 tokens.

GPT-4o:
Window Size: 128,000 tokens.

[MisterAndry]

the reason for this error is that the model does not follow the prompt and its response does not conform to the expected format in which the community should be organized.
Either your model can't accommodate the entire prompt and the prompt is cut off, or your model is too weak to follow such a prompt.
I use Ollama, not ChatGPT and I had the same error and it took me a long time to figure out what the cause was. Because almost all models in ollama have a context window of 2000 tokens, which is very small for prompts in graphrag. You can enlarge this window by hand, there is an instruction on the ollama github page. You can also play with the max_tokens parameter for community_report in the settings.yaml file - this is the length of the community report that will be generated by llm.
I found for myself the ideal model in terms of quality and speed for grafrag - gemma2_27b, with increased context up to 8k

[RainDayDream]

The strangest thing is that the incorrect community ID remains the same every time it runs. Even though the model is weak, why is the error always the same? Do you have the same error ?

[MisterAndry]

What llm are you using? Is it ChatGpt or ollama\LmStudio\llama_cpp hosted model?
Look in the cache folder for the complete prompt that is sent to llm to create a specific community. See how long this prompt is in tokens and whether it fits in the context window of the model being used.
Why it is always the same id - I don't know.