Merge branch 'master' into ruff

.appveyor.yml

@@ -8,7 +8,7 @@ image:
   # linux builds done in Travis CI for now
   - Visual Studio 2017
   - Visual Studio 2019
-  - Visual Studio 2022
+  #- Visual Studio 2022  # Temporary disable while failing on 14.40 build tools
 
 cache:
   - downloads -> appveyor.yml
@@ -37,38 +37,38 @@ environment:
     # Test oldest and newest supported Pythons, and a subset in between.
     # Skipping 3.7 and 3.9 at this time
     - WINPYTHON: "Python312"
-
     - WINPYTHON: "Python310"
-
     - WINPYTHON: "Python38"
-
     - WINPYTHON: "Python36"
 
 # remove sets of build jobs based on criteria below
 # to fine tune the number and platforms tested
 matrix:
   exclude:
-    # test python 3.6 on Visual Studio 2017 image
+    # XXX test python 3.6 on Visual Studio 2017 image
+    # test python 3.8 on Visual Studio 2017 image
     - image: Visual Studio 2017
       WINPYTHON: "Python312"
     - image: Visual Studio 2017
       WINPYTHON: "Python310"
     - image: Visual Studio 2017
-      WINPYTHON: "Python38"
+      WINPYTHON: "Python36"
 
-    # test python 3.8 on Visual Studio 2019 image
+    # test python 3.10 on Visual Studio 2019 image
     - image: Visual Studio 2019
       WINPYTHON: "Python312"
     - image: Visual Studio 2019
-      WINPYTHON: "Python310"
+      WINPYTHON: "Python38"
     - image: Visual Studio 2019
       WINPYTHON: "Python36"
 
-    # test python 3.10 and 3.11 on Visual Studio 2022 image
+    # test python 3.12 on Visual Studio 2022 image
     - image: Visual Studio 2022
-      WINPYTHON: "Python36"
+      WINPYTHON: "Python310"
     - image: Visual Studio 2022
       WINPYTHON: "Python38"
+    - image: Visual Studio 2022
+      WINPYTHON: "Python36"
 
 # Remove some binaries we don't want to be found
 # Note this is no longer needed, git-windows bin/ is quite minimal now.

CHANGES.txt

@@ -4,8 +4,11 @@
 
                             Change Log
 
-NOTE: The 4.0.0 Release of SCons dropped Python 2.7 Support
-NOTE: 4.3.0 now requires Python 3.6.0 and above. Python 3.5.x is no longer supported
+NOTE: The 4.0.0 release of SCons dropped Python 2.7 support. Use 3.1.2 if
+  Python 2.7 support is required (but note old SCons releases are unsupported).
+NOTE: Since SCons 4.3.0, Python 3.6.0 or above is required.
+NOTE: Python 3.6 support is deprecated and will be dropped in a future reease.
+  python.org no longer supports 3.6 or 3.7, and will drop 3.8 in Oct. 2024.
 
 RELEASE  VERSION/DATE TO BE FILLED IN LATER
 
@@ -77,6 +80,40 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
     - Framework for scons-time tests adjusted so a path with a long username
       Windows has squashed doesn't get re-expanded. Fixes a problem seen
       on GitHub Windows runner which uses a name "runneradmin".
+    - SCons.Environment.is_valid_construction_var() now returns a boolean to
+      match the convention that functions beginning with "is" have yes/no
+      answers (previously returned either None or an re.match object).
+      Now matches the annotation and docstring (which were prematurely
+      updated in 4.6). All SCons usage except unit test was already fully
+      consistent with a bool.
+    - When a variable is added to a Variables object, it can now be flagged
+      as "don't perform substitution" by setting the argument subst.
+      This allows variables to contain characters which would otherwise
+      cause expansion. Fixes #4241.
+    - The test runner now recognizes the unittest module's return code of 5,
+      which means no tests were run. SCons/Script/MainTests.py currently
+      has no tests, so this particular error code is expected - should not
+      cause runtest to give up with an "unknown error code".
+    - Updated the notes about reproducible builds with SCons and the example.
+    - The Clone() method now respects the variables argument (fixes #3590)
+    - is_valid_construction_var() (not part of the public API) moved from
+      SCons.Environment to SCons.Util to avoid the chance of import loops. Variables
+      and Environment both use the routine and Environment() uses a Variables()
+      object so better to move to a safer location.
+    - AddOption and the internal add_local_option which AddOption calls now
+      recognize a "settable" keyword argument to indicate a project-added
+      option can also be modified using SetOption. Fixes #3983.
+      NOTE: If you were using ninja and using SetOption() for ninja options
+      in your SConscripts prior to loading the ninja tool, you will now
+      see an error. The fix is to move the SetOption() to after you've loaded
+      the ninja tool.
+    - ListVariable now has a separate validator, with the functionality
+      that was previously part of the converter. The main effect is to
+      allow a developer to supply a custom validator, which previously
+      could be inhibited by the converter failing before the validator
+      is reached.
+    - Regularized header (copyright, licens) at top of documentation files
+      using SPDX.
 
 
 RELEASE 4.7.0 -  Sun, 17 Mar 2024 17:22:20 -0700

README.rst

@@ -249,6 +249,19 @@ notifications and other GitHub events (``#github-update``),
 if those are of interest. See the website for more contact information:
 https://scons.org/contact.html.
 
+Reproducible Builds
+===================
+SCons itself is set up to do "reproducible builds"
+(see (https://reproducible-builds.org/specs/source-date-epoch/)
+if environment variables ``SOURCE_DATE_EPOCH`` is set - that is,
+fields in the package which could change each time the package is
+constructed are forced to constant values.
+
+To support other projects which wish to do the same, a sample script
+is provided which can be placed in a site directory, which imports
+``SOURCE_DATE_EPOCH`` and sets it in the execution environment of
+every created construction envirionment. There's also an installer
+script (POSIX shell only). See packaging/etc/README.txt for more details.
 
 Donations
 =========
@@ -258,15 +271,6 @@ software, or hardware) to support continued work on the project.  Information
 is available at https://www.scons.org/donate.html
 or the GitHub Sponsors button on https://github.com/scons/scons.
 
-Reproducible Builds
-===================
-In order to suppor those users who which to produce reproducible builds
-(https://reproducible-builds.org/specs/source-date-epoch/) we're now including
-logic to force SCons to propagate SOURCE_DATE_EPOCH from your shell environment for
-all SCons builds to support reproducible builds we're now providing an example
-site_init.py and a script to install it in your ~/.scons. See packaging/etc/README.txt
-for more info
-
 For More Information
 ====================
 

RELEASE.txt

@@ -44,6 +44,27 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY
   of CCFLAGS; the latter variable could cause a compiler warning.
 - The implementation of Variables was slightly refactored, there should
   not be user-visible changes.
+- SCons.Environment.is_valid_construction_var() now returns a boolean to
+  match the convention that functions beginning with "is" have yes/no
+  answers (previously returned either None or an re.match object).
+  Now matches the annotation and docstring (which were prematurely
+  updated in 4.6). All SCons usage except unit test was already fully
+  consistent with a bool.
+- The Variables object Add method now accepts a subst keyword argument
+  (defaults to True) which can be set to inhibit substitution prior to
+  calling the variable's converter and validator.
+- AddOption and the internal add_local_option which AddOption calls now
+  recognize a "settable" keyword argument to indicate a project-added
+  option can also be modified using SetOption.
+  NOTE: If you were using ninja and using SetOption() for ninja options
+  in your SConscripts prior to loading the ninja tool, you will now
+  see an error. The fix is to move the SetOption() to after you've loaded
+  the ninja tool.
+- ListVariable now has a separate validator, with the functionality
+  that was previously part of the converter. The main effect is to
+  allow a developer to supply a custom validator, which previously
+  could be inhibited by the converter failing before the validator
+  is reached.
 
 FIXES
 -----
@@ -52,6 +73,7 @@ FIXES
 - Improved the conversion of a "foreign" exception from an action
   into BuildError by making sure our defaults get applied even in
   corner cases. Fixes Issue #4530
+- The Clone() method now respects the variables argument (fixes #3590)
 
 IMPROVEMENTS
 ------------
@@ -76,6 +98,8 @@ DOCUMENTATION
 - Restructured API Docs build so main package contents are listed
   before contents of package submodules.
 - Updated manpage description of Command "builder" and function.
+- Updated the notes about reproducible builds with SCons and the example.
+- Regularized header (copyright, licens) at top of documentation files using SPDX.
 
 
 
@@ -95,7 +119,14 @@ DEVELOPMENT
 - Repository linter/formatter changed from flake8/black to ruff, as the
   latter grants an insane speed boost without compromising functionality.
   Existing settings were migrated 1-to-1 where possible.
-
+- The test runner now recognizes the unittest module's return code of 5,
+  which means no tests were run. SCons/Script/MainTests.py currently
+  has no tests, so this particular error code is expected - should not
+  cause runtest to give up with an "unknown error code".
+- is_valid_construction_var() (not part of the public API) moved from
+  SCons.Environment to SCons.Util to avoid the chance of import loops. Variables
+  and Environment both use the routine and Environment() uses a Variables()
+  object so better to move to a safer location.
 
 Thanks to the following contributors listed below for their contributions to this release.
 ==========================================================================================

SCons/Action.xml

@@ -1,9 +1,10 @@
 <?xml version="1.0"?>
 <!--
-Copyright The SCons Foundation
+SPDX-FileCopyrightText: Copyright The SCons Foundation (https://scons.org)
+SPDX-License-Identifier: MIT
+SPDX-FileType: DOCUMENTATION
 
 This file is processed by the bin/SConsDoc.py module.
-See its __doc__ string for a discussion of the format.
 -->
 
 <!DOCTYPE sconsdoc [

SCons/Defaults.xml

@@ -1,9 +1,10 @@
 <?xml version="1.0"?>
 <!--
-Copyright The SCons Foundation
+SPDX-FileCopyrightText: Copyright The SCons Foundation (https://scons.org)
+SPDX-License-Identifier: MIT
+SPDX-FileType: DOCUMENTATION
 
 This file is processed by the bin/SConsDoc.py module.
-See its __doc__ string for a discussion of the format.
 -->
 
 <!DOCTYPE sconsdoc [

SCons/Environment.py

@@ -76,6 +76,7 @@
     to_String_for_subst,
     uniquer_hashables,
 )
+from SCons.Util.envs import is_valid_construction_var
 from SCons.Util.sctyping import ExecutorType
 
 class _Null:
@@ -510,13 +511,6 @@ def update(self, mapping) -> None:
             self.__setitem__(i, v)
 
 
-_is_valid_var = re.compile(r'[_a-zA-Z]\w*$')
-
-def is_valid_construction_var(varstr) -> bool:
-    """Return True if *varstr* is a legitimate construction variable."""
-    return _is_valid_var.match(varstr)
-
-
 class SubstitutionEnvironment:
     """Base class for different flavors of construction environments.
 
@@ -605,7 +599,7 @@ def __setitem__(self, key, value):
             # key and we don't need to check.  If we do check, using a
             # global, pre-compiled regular expression directly is more
             # efficient than calling another function or a method.
-            if key not in self._dict and not _is_valid_var.match(key):
+            if key not in self._dict and not is_valid_construction_var(key):
                 raise UserError("Illegal construction variable `%s'" % key)
             self._dict[key] = value
 
@@ -1568,16 +1562,28 @@ def AppendUnique(self, delete_existing: bool=False, **kw) -> None:
                     self._dict[key] = dk + val
         self.scanner_map_delete(kw)
 
-    def Clone(self, tools=[], toolpath=None, parse_flags = None, **kw):
+    def Clone(self, tools=[], toolpath=None, variables=None, parse_flags=None, **kw):
         """Return a copy of a construction Environment.
 
-        The copy is like a Python "deep copy"--that is, independent
-        copies are made recursively of each objects--except that
-        a reference is copied when an object is not deep-copyable
-        (like a function).  There are no references to any mutable
-        objects in the original Environment.
-        """
+        The copy is like a Python "deep copy": independent copies are made
+        recursively of each object, except that a reference is copied when
+        an object is not deep-copyable (like a function).  There are no
+        references to any mutable objects in the original environment.
 
+        Unrecognized keyword arguments are taken as construction variable
+        assignments.
+
+        Arguments:
+           tools: list of tools to initialize.
+           toolpath: list of paths to search for tools.
+           variables: a :class:`~SCons.Variables.Variables` object to
+              use to populate construction variables from command-line
+              variables.
+           parse_flags: option strings to parse into construction variables.
+
+        .. versionadded:: 4.8.0
+              The optional *variables* parameter was added.
+        """
         builders = self._dict.get('BUILDERS', {})
 
         clone = copy.copy(self)
@@ -1603,6 +1609,8 @@ def Clone(self, tools=[], toolpath=None, parse_flags = None, **kw):
         for key, value in kw.items():
             new[key] = SCons.Subst.scons_subst_once(value, self, key)
         clone.Replace(**new)
+        if variables:
+            variables.Update(clone)
 
         apply_tools(clone, tools, toolpath)
 

SCons/Environment.xml

@@ -1,9 +1,10 @@
 <?xml version="1.0"?>
 <!--
-Copyright The SCons Foundation
+SPDX-FileCopyrightText: Copyright The SCons Foundation (https://scons.org)
+SPDX-License-Identifier: MIT
+SPDX-FileType: DOCUMENTATION
 
 This file is processed by the bin/SConsDoc.py module.
-See its __doc__ string for a discussion of the format.
 -->
 
 <!DOCTYPE sconsdoc [
@@ -1079,11 +1080,12 @@ Clean(docdir, os.path.join(docdir, projectname))
 </arguments>
 <summary>
 <para>
-Returns a separate copy of a construction environment.
-If there are any keyword arguments specified,
-they are added to the returned copy,
+Returns an independent copy of a &consenv;.
+If there are any unrecognized keyword arguments specified,
+they are added as &consvars; in the copy,
 overwriting any existing values
-for the keywords.
+for those keywords.
+See the manpage section "Construction Environments" for more details.
 </para>
 
 <para>
@@ -1096,8 +1098,9 @@ env3 = env.Clone(CCFLAGS='-g')
 </example_commands>
 
 <para>
-Additionally, a list of tools and a toolpath may be specified, as in
-the &f-link-Environment; constructor:
+A list of <parameter>tools</parameter>
+and a <parameter>toolpath</parameter> may be specified,
+as in the &f-link-Environment; constructor:
 </para>
 
 <example_commands>
@@ -1110,7 +1113,7 @@ env4 = env.Clone(tools=['msvc', MyTool])
 <para>
 The
 <parameter>parse_flags</parameter>
-keyword argument is also recognized to allow merging command-line
+keyword argument is also recognized, to allow merging command-line
 style arguments into the appropriate construction
 variables (see &f-link-env-MergeFlags;).
 </para>
@@ -1119,6 +1122,17 @@ variables (see &f-link-env-MergeFlags;).
 # create an environment for compiling programs that use wxWidgets
 wx_env = env.Clone(parse_flags='!wx-config --cflags --cxxflags')
 </example_commands>
+
+<para>
+The <parameter>variables</parameter>
+keyword argument is also recognized, to allow (re)initializing
+&consvars; from a <literal>Variables</literal> object.
+</para>
+
+<para>
+<emphasis>Changed in version 4.8.0:</emphasis>
+the <parameter>variables</parameter> parameter was added.
+</para>
 </summary>
 </scons_function>
 
@@ -1760,7 +1774,7 @@ will print:
 </arguments>
 <summary>
 <para>
-Return a new construction environment
+Return a new &consenv;
 initialized with the specified
 <parameter>key</parameter>=<replaceable>value</replaceable>
 pairs.
@@ -1770,7 +1784,8 @@ The keyword arguments
 <parameter>toolpath</parameter>,
 <parameter>tools</parameter>
 and <parameter>variables</parameter>
-are also specially recognized.
+are specially recognized and do not lead to
+&consvar; creation.
 See the manpage section "Construction Environments" for more details.
 </para>
 </summary>