update dependabot fixes

[beejones]

Resolve dependabot PR's
Update npm packages
Update pip packages
Update CCF docker images

[beejones]

tests:
https://github.com/microsoft/privacy-sandbox-dev/actions/runs/12274227739

[DomAyre]

Do we know why the e2e tests failed?

[beejones]

Do we know why the e2e tests failed?

passed

[DomAyre]

Okay but we didn't change anything between the runs (ca93a22), so the test is flaky which isn't necessarily a problem, but I'm wondering if we know why?

[beejones]

I reran and now test_listpubkeys running container failed:
==================================== ERRORS ====================================
__________________ ERROR at setup of test_refresh_key_appears __________________
[gw1] linux -- Python 3.10.12 /usr/bin/python3

@pytest.fixture()
def setup_ccf():
    deployment_name = os.getenv("DEPLOYMENT_NAME", f"kms-{unique_string()}")
    try:

      call_script(

            [f"scripts/{TEST_ENVIRONMENT}/up.sh", "--force-recreate"],
            env={
                **os.environ,
                "DEPLOYMENT_NAME": deployment_name,
            },
        )

test/system-test/conftest.py:78:

---

test/system-test/conftest.py:34: in call_script
res = subprocess.run(

---

input = None, capture_output = False, timeout = None, check = True
popenargs = (['scripts/ccf/az-cleanroom-aci/up.sh', '--force-recreate'],)
kwargs = {'cwd': '/home/runner/work/azure-privacy-sandbox-kms/azure-privacy-sandbox-kms', 'env': {'ACCEPT_EULA': 'Y', 'ACTIONS_...ee690d8/idtoken?api-version=2.0', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', ...}, 'stdout': -1}
process = <Popen: returncode: 1 args: ['scripts/ccf/az-cleanroom-aci/up.sh', '--force-...>
stdout = b'', stderr = None, retcode = 1

def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
    or pass capture_output=True to capture both.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:

          raise CalledProcessError(retcode, process.args,

                                     output=stdout, stderr=stderr)

E subprocess.CalledProcessError: Command '['scripts/ccf/az-cleanroom-aci/up.sh', '--force-recreate']' returned non-zero exit status 1.

/usr/lib/python3.10/subprocess.py:526: CalledProcessError
=========================== short test summary info ============================
ERROR test/system-test/test_listpubkeys.py::test_refresh_key_appears - subprocess.CalledProcessError: Command '['scripts/ccf/az-cleanroom-aci/up.sh', '--force-recreate']' returned non-zero exit status 1.

[beejones]

CI passed again. Will rerun them

[beejones]

Failed again but now key.
Similar issue during up. I don't believe it is the change causing this

_____________________ ERROR at setup of test_key_fmt_tink ______________________
[gw2] linux -- Python 3.10.12 /usr/bin/python3

@pytest.fixture()
def setup_ccf():
    deployment_name = os.getenv("DEPLOYMENT_NAME", f"kms-{unique_string()}")
    try:

      call_script(

            [f"scripts/{TEST_ENVIRONMENT}/up.sh", "--force-recreate"],
            env={
                **os.environ,
                "DEPLOYMENT_NAME": deployment_name,
            },
        )

test/system-test/conftest.py:78:

---

test/system-test/conftest.py:34: in call_script
res = subprocess.run(

---

input = None, capture_output = False, timeout = None, check = True
popenargs = (['scripts/ccf/az-cleanroom-aci/up.sh', '--force-recreate'],)
kwargs = {'cwd': '/home/runner/work/azure-privacy-sandbox-kms/azure-privacy-sandbox-kms', 'env': {'ACCEPT_EULA': 'Y', 'ACTIONS_...a923bb9/idtoken?api-version=2.0', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', ...}, 'stdout': -1}
process = <Popen: returncode: 1 args: ['scripts/ccf/az-cleanroom-aci/up.sh', '--force-...>
stdout = b'', stderr = None, retcode = 1

def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
    or pass capture_output=True to capture both.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:

          raise CalledProcessError(retcode, process.args,

                                     output=stdout, stderr=stderr)

E subprocess.CalledProcessError: Command '['scripts/ccf/az-cleanroom-aci/up.sh', '--force-recreate']' returned non-zero exit status 1.

/usr/lib/python3.10/subprocess.py:526: CalledProcessError

[DomAyre]

I'm not worried about flakiness in az-cleanroom-aci/up.sh There's a few improvements for that in my various PRs so once they're merged it should get better