Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Space leak when executing multiple Text.XML.Stream.Parse.parseBytes conduits #136

Open
mbid opened this issue Aug 9, 2018 · 0 comments
Open

Comments

@mbid
Copy link

mbid commented Aug 9, 2018

I've encountered a weird space leak using Text.XML.Stream.Parse. I believe a reasonably minimal example is this:

doTwice :: Applicative f => f () -> f ()
doTwice x = x *> x

leakSpace :: IO ()
leakSpace =
  runResourceT $ runConduit $
  doTwice (sourceFile "large-file.xml" .| Text.XML.Stream.parseBytes def) .|
  sinkNull

If large-file.xml is large enough, this crashes with OOM, even before the second iteration over the file.

If the line

  doTwice (sourceFile "large-file.xml" .| Text.XML.Stream.parseBytes def) .|

is replaced with either

  sourceFile "large-file.xml" .| Text.XML.Stream.parseBytes def .|

(i.e. only parsing the file once) or

  doTwice (sourceFile "large-file.xml.gz" .| ungzip) .|

(i.e. not parsing at all, just connecting sourceFile to something else), everything works as expected. This makes me think that the cause of the issue is somewhere in parseBytes.

I've encountered this issue when trying to combine multiple conduits with for_, i.e. something like

for_ files $ \file -> sourceFile file .|  Text.XML.Stream.parseBytes .| ...

and then I triggered the issue even when files was a singleton list. I've not been able to reproduce this with constant singleton lists though, perhaps because of optimization. If files was a dynamic Maybe instead, the issue did not occure.

I'm using stackage's lts-12.5, i.e. xml-conduit-1.8.0.

@mbid mbid changed the title Space leak when executing multiple Stream.parseBytes conduits Space leak when executing multiple Text.XML.Stream.Parse.parseBytes conduits Aug 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant