Newlines at end of post

[hhoeflin]

Hi,

here an issue I tripped over (or minor problem of faithful round-tripping). I wanted to read a post, manipulate the frontmatter and write it back out. Afterwards, I programmatically append things to the post.

when I do

page_post = frontmatter.load(post_path)                           
# change the metadata somehow  
frontmatter.dump(page_post, post_path)                            

Now, the newlines at the end of the post have disappeared. In my case, this was a problem as I relied on them for syntactically correct appending.

[eyeseast]

Newlines have always been stripped, I think.

What kind of content are you putting in each post? And what's the significance of the number of newlines? Is it possible, knowing that extra whitespace will be stripped on the post, to append one or more newlines later?

[hhoeflin]

Sure, I wrote a short helper function that uses loads instead of load and re-appends any newlines.

I create markdown programmatically and remember the previous number of newlines.

So overall, this completely is not a big deal, but philosophically, the library in a roundtrip doesn't return what it found (i.e. strips the newlines, so results in a changed file). If you don't want to fix, I understand, just wanted to raise the issue.

[eyeseast]

I'm going to close this just because I don't want to change existing behavior here. I'll check the docs to make sure it's clear what's happening. I'm pretty sure I have a test case showing this, but it's worth making this less of a surprise.

[kcarnold]

I was also surprised by newlines getting stripped. I expected this code to modify just the frontmatter, not the rest of the file.

[eyeseast]

Reopening this since @nk9 brings it up in #100. I don't have a strong opinion on this except that I am somewhat hesitant to change default output, at least without a discussion. So please discuss.

[nk9]

Well, I would argue that the point of this utility is to add/manipulate front matter, not any of the rest of the content. If a file ends with a bunch of newlines, what business is it of frontmatter? The present behavior is destructive, for no particular reason.

At the very least, I'd argue that frontmatter should be outputting POSIX-compliant files when asked to dump the Post to a file. It seems very strange to me to force users to opt-into POSIX compliance. I don't really see the point of adding an option for this. The tiny minority of users who absolutely need their file strings stripped can do that easily with dumps().rstrip(). Everyone else would rightly expect it to Just Work in all situations, which means ending files with a newline.

I get that changing the way the utility deals with leading/trailing whitespace has the potential to be disruptive (in tests, and causing whitespace git churn). For that reason, it makes sense to me to put it in a major version and clearly call out the change in the changelog.

[SirUli]

You can use your own handler to fix this.

class yamlFrontMatterHandler(YAMLHandler):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def format(self, post, **kwargs):
        """
        Turn a post into a string, used in ``frontmatter.dumps``.
        Changed from default handler to not remove the last empty line
        """
        start_delimiter = kwargs.pop("start_delimiter", self.START_DELIMITER)
        end_delimiter = kwargs.pop("end_delimiter", self.END_DELIMITER)

        metadata = self.export(post.metadata, **kwargs)

        return DEFAULT_POST_TEMPLATE.format(
            metadata=metadata,
            content=post.content,
            start_delimiter=start_delimiter,
            end_delimiter=end_delimiter,
        ).lstrip()

Notice the lstrip instead of strip in the last line

Then simply add this for the respective methods:

post = frontmatter.load(file_path, handler=yamlFrontMatterHandler())

[justmars]

I appreciate the library and also would like to throw support to enabling the option rather than as a default. For my specific use case, within VS Code, I tend to use the markdownlint extension with the following rule re: terminal newlines.

[eyeseast]

I've run into the same thing with VS Code, and I think the arguments above are persuasive. I'm overdue to handle a few updates, so this will probably all get rolled up into a 2.0 release, since it's technically a breaking change.

Can't say exactly when it'll happen, but it's on the roadmap.

[darkdragon-001]

Just fell into this trap as well. Looking forward to see POSIX-compliant files output from the library. Thanks for your work!

[tusharsadhwani]

Well, small caveat that POSIX no longer mandates a newline at the end of a file, and that requirement wasn't respected by any shell anyway.

Furthermore I think it is valuable to keep the same behaviour as json.dump and json.dumps, which don't emit a newline:

>>> import json
>>> json.dumps({1: 2})
'{"1": 2}'
>>> with open('foo', 'w') as file:
...   json.dump({1: 2}, file)
... 
>>> with open('foo') as file:
...   contents = file.read()
... 
>>> contents
'{"1": 2}'

[zedtools]

I just got tripped up by this issue too.

I am editing Obsidian .md files with YAML frontmatter and content, and noticed that if I edit the YAML in a file that has no content, it is output with no new line after the ---. When I open the file in Obisdian, Obsidian reinserts the newline, changing the file's modification date (which I don't want).

I add my vote to preserving newlines at the end of the file, or at least providing the option to do so.

[nk9]

I now realize that "always emit a newline at the end" and "leave newlines alone" are contradictory in some cases. It sounds like having a terminal newline is less important than leaving the provided text as-is. If users want a newline at the end of the file, they can add one, and frontmatter should respect it.