add support for xz compression #6523

[ChrisLundquist]

Closes #6523

Seems like it works, but I haven't used this before.

This pulls in a new dep, so I may need some help on how to pull that in.

[ltrager]

Thanks for the quick response! The new dep is missing from the Makefile, I applied this patch which allowed me to build and test.

One thing I noticed is that while packer told me it was compressing the image with 8 core it only pegged one core to %100.

[ChrisLundquist]

Cool! I'll see if I can fix those things up!

[ChrisLundquist]

Looks like this particular xz implementation is single core only ulikunitz/xz#18
I didn't see any threading digging through the source. I'm also unclear on how difficult it would be to add.

The pros of the above library is that it is is pure golang so we don't have any runtime deps.

I also saw: https://github.com/danielrh/go-xz/ which seems to call into the C libs which gained multithreading around 2014 https://www.phoronix.com/forums/forum/software/general-linux-open-source/47329-xz-5-2-adds-new-multi-threaded-options

The third option would be to write the code to shell out to the command line utilities, but nothing else does that from what I can tell in this compress post-processor.

So this sounds like a trade off for the packer project to choose.

I can fix up the printed message for now.

[SwampDragons]

Normally we use govendor to manage dependencies, rather than the makefile. So you'd go into your local copy of Packer, then run govendor fetch github.com/ulikunitz/xz, add the files pulled into the vendor folder, and commit the changes to the vendor/vendor.json file.

[SwampDragons]

re: library choice: I think I'd rather single-threading and no runtime dependencies.

[ChrisLundquist]

then run govendor fetch github.com/ulikunitz/xz, add the files pulled into the vendor folder

Yep! I eventually figured this out in the contributing doc. I can remove the line from the Makefile.

re: library choice: I think I'd rather single-threading and no runtime dependencies.

Cool cool, sounds good.

[azr]

Nice one, thanks !
I found just a tiny minor change and I think it's ready to merge.

[azr]

🌵here I'd use _ instead of compressionLevel since it's not used

[azr]

Howdy @ChrisLundquist,

And I forgot to test try it locally !
I tried to run an xz compression locally and I had the following error:

==> googlecompute: Running post-processor: manifest
==> googlecompute: Running post-processor: compress
==> googlecompute (compress): Using xz compression with 1 core for googlecompute.xz (library does not support MT)
Build 'googlecompute' errored: 1 error(s) occurred:

* Post-processor failed: Can only have 1 input file when not using tar/zip. Found 0 files: []

🙂

[rickard-von-essen]

@azr don't think running compress on a cloud builder make much sense since they always have 0 files. I'm not sure it fails more gracefully with another compression algorithm.

[azr]

Hey @rickard-von-essen,

Yes, tried running a windows .iso but it crashed my machine.

But I also tried it using .tar.gz and pgzip & it did work

=> googlecompute: Running post-processor: compress
==> googlecompute (compress): Using pgzip compression with 8 cores for googlecompute.tar.gz
==> googlecompute (compress): Tarring googlecompute.tar.gz with pgzip
==> googlecompute (compress): Archive googlecompute.tar.gz completed
Build 'googlecompute' finished.

==> Builds finished. The artifacts of successful builds are:
--> googlecompute: A disk image was created: packer-1534515078
--> googlecompute:
--> googlecompute: compressed artifacts in: googlecompute.tar.gz

And for this try I was on the wrong branch :

==> googlecompute (compress): Using pgzip compression with 8 cores for googlecompute.xz
==> googlecompute (compress): Tarring googlecompute.xz with pgzip
==> googlecompute (compress): Archive googlecompute.xz completed
Build 'googlecompute' finished.

Do you have an example of a real usage-case of compression ?

[rickard-von-essen]

I guess all of those above googlecompute.XXX archives are empty, a more realistic example is to start with the docker builder (with commit: false).

[azr]

Yup, all empty, good catch, thanks. Trying with docker then 🙂

[azr]

Okay, so I tried with docker and works; It creates an XZ file. The behaviour is the same as with gz but with XZ.

[SwampDragons]

Okay so it sounds like this is ready to merge after the one small change to the makeXZWriter function signature.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.