reproducible tarballs have unexpected checksums on mac

[lexming]

This is related to PR easybuilders/easybuild-easyconfigs#21392

The checksum of the generated lsp-types-0.95.1.tar.gz with our reproducible tarball code is 455f2c12f64c2e293c72efdf2b90b96f5903d6590bd2e496bab5f12750402631 on several Linux systems (RHEL8, Rocky 8, Fedora 40 and Debian Bookworm).

However, on MacOS it has an unexpected 59869db34853933b239f1e2219cf7d431da006aa919635478511fabbfc8849d2.

Steps to generate tarball from git repo:

1. git clone https://github.com/astral-sh/lsp-types.git
2. cd lsp-types && git checkout 3512a9f33eadc5402cfab1b8f7340824c8ca1439 && cd ..

3. find lsp-types -name ".git" -prune -o -print0 -exec touch --date=@0 {} \; -exec chmod "go+u,go-w" {} \; | 
   LC_ALL=C sort --zero-terminated | tar --create --no-recursion --owner=0 --group=0 --numeric-owner 
   --format=gnu --null --files-from - | gzip --no-name > lsp-types-0.95.1.tar.gz
   

We need to troubleshoot this on a mac:

• compare the checksum of the ungzip tarballs on both systems
• compare the raw contents of the tarball header, there is some metadata there that can point to differences. Basically check the outcome of head -n 100 on the .tar files

[lexming]

@boegel this one is for you 😃

[boegel]

@lexming This explains it:

$ find lsp-types -name ".git" -prune -o -print0 -exec touch --date='@0' {} \; -exec chmod 'go+u,go-w' {} \; | LC_ALL=C sort --zero-terminated | tar --create --no-recursion --owner=0 --group=0 --numeric-owner  --format=gnu --null --files-from - | gzip --no-name > lsp-types-0.95.1.tar.gz
tar: Option --owner=0 is not supported
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
touch: illegal option -- -
usage: touch [-A [-][[hh]mm]SS] [-achm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]]
       [-d YYYY-MM-DDThh:mm:SS[.frac][tz]] file ...
...
$ echo $?
0
$ ls -l lsp-types-0.95.1.tar.gz
-rw-r--r--  1 kehoste  staff  20 Sep 27 09:06 lsp-types-0.95.1.tar.gz

So, the resulting tarball is basically an empty file.

Two (well, three) problems here:

• incorrect usage of tar (--owner) and touch (--date);
• we're not correctly checking the exit code of the pipeline of commands we're running, we should also run "set -o pipefail"

For tar, I don't see an equivalent to --owner on macOS, we may need to add chown in the mix?

For touch, there's -d as option, but that only takes a proper timestamp, can't use @0:

     -d      Change the access and modification times to the specified date time instead of the current time of day.  The argument is of the form “YYYY-MM-DDThh:mm:SS[.frac][tz]” where the letters represent the following:
                   YYYY    At least four decimal digits representing the year.
                   MM, DD, hh, mm, SS
                           As with -t time.
                   T       The letter T or a space is the time designator.
                   .frac   An optional fraction, consisting of a period or a comma followed by one or more digits.  The number of significant digits depends on the kernel configuration and the filesystem, and may be zero.
                   tz      An optional letter Z indicating the time is in UTC.  Otherwise, the time is assumed to be in local time.  Local time is affected by the value of the TZ environment variable.

[boegel]

TZ=UTC touch -t 197001010000.00 may works as equivalent to touch --date='@0'?

[lexming]

It doesn't. Using -t 197001010000.00 is timezone dependent, depending on your locale you might get a timestamp zero or not.

[lexming]

@boegel using touch --date=1970-01-01T00:00:00.00Z does seem to be equivalent to touch --date='@0', can you try this one?

[lexming]

@boegel the ownership issue is a big problem. Only root can chown the owner of a file. So there is no workaround possible here.

Good news is that BSD tar does actually support --owner, see https://man.freebsd.org/cgi/man.cgi?tar(1) . So maybe this is limited to some obscure version of macOS.

[lexming]

PR #4660 adds a more portable setting for timestamps and enables pipeline failures.

The issue about --owner stays unchanged.

[boegel]

For tar, one option we could explore could be to use the tarfile module in the Python standard library.

Quick & dirty try:

import tarfile
import os

def reset(tarinfo):
    tarinfo.uid = tarinfo.gid = 0
    tarinfo.uname = tarinfo.gname = "root"
    tarinfo.mtime = 0
    return tarinfo

# Create a tar archive
with tarfile.open("example.tar", "w") as tar:
    tar.add("file1.txt", filter=reset)
    tar.add("file2.txt", filter=reset)

print("Tar archive created.")

(doesn't reproduce exact same tarball on macOS + Linux yet though using empty file1.txt and file2.txt)