bpo-31650: PEP 552 (Deterministic pycs) implementation

Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.

Author: benjaminp

Doc/glossary.rst

@@ -458,6 +458,12 @@ Glossary
       is believed that overcoming this performance issue would make the
       implementation much more complicated and therefore costlier to maintain.
 
+
+   hash-based pyc
+      A bytecode cache file that uses the the hash rather than the last-modified
+      time of the corresponding source file to determine its validity. See
+      :ref:`pyc-invalidation`.
+
    hashable
       An object is *hashable* if it has a hash value which never changes during
       its lifetime (it needs a :meth:`__hash__` method), and can be compared to

Doc/library/compileall.rst

@@ -83,6 +83,15 @@ compile Python sources.
    If ``0`` is used, then the result of :func:`os.cpu_count()`
    will be used.
 
+.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]
+
+   Control how the generated pycs will be invalidated at runtime. The default
+   setting, ``timestamp``, means that pyc files with the source timestamp and
+   size embedded will be generated. The ``checked-hash`` and ``unchecked-hash``
+   values cause hash-based pycs to be generated. Hash-based pycs embed a hash of
+   the source file contents rather than a timestamp. See :ref:`pyc-invalidation`
+   for more information on how Python validates bytecode cache files at runtime.
+
 .. versionchanged:: 3.2
    Added the ``-i``, ``-b`` and ``-h`` options.
 
@@ -91,6 +100,9 @@ compile Python sources.
    was changed to a multilevel value.  ``-b`` will always produce a
    byte-code file ending in ``.pyc``, never ``.pyo``.
 
+.. versionchanged:: 3.7
+   Added the ``--invalidation-mode`` parameter.
+
 
 There is no command-line option to control the optimization level used by the
 :func:`compile` function, because the Python interpreter itself already
@@ -99,7 +111,7 @@ provides the option: :program:`python -O -m compileall`.
 Public functions
 ----------------
 
-.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1)
+.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
    files along the way. Return a true value if all the files compiled successfully,
@@ -140,6 +152,10 @@ Public functions
    then sequential compilation will be used as a fallback.  If *workers* is
    lower than ``0``, a :exc:`ValueError` will be raised.
 
+   *invalidation_mode* should be a member of the
+   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
+   pycs are invalidated at runtime.
+
    .. versionchanged:: 3.2
       Added the *legacy* and *optimize* parameter.
 
@@ -156,7 +172,10 @@ Public functions
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
 
-.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1)
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
+.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Compile the file with path *fullname*. Return a true value if the file
    compiled successfully, and a false value otherwise.
@@ -184,6 +203,10 @@ Public functions
    *optimize* specifies the optimization level for the compiler.  It is passed to
    the built-in :func:`compile` function.
 
+   *invalidation_mode* should be a member of the
+   :class:`py_compile.PycInvalidationMode` enum and controls how the generated
+   pycs are invalidated at runtime.
+
    .. versionadded:: 3.2
 
    .. versionchanged:: 3.5
@@ -193,7 +216,10 @@ Public functions
       The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
       no matter what the value of *optimize* is.
 
-.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1)
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
+.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
 
    Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
    true value if all the files compiled successfully, and a false value otherwise.
@@ -213,6 +239,9 @@ Public functions
       The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
       no matter what the value of *optimize* is.
 
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added.
+
 To force a recompile of all the :file:`.py` files in the :file:`Lib/`
 subdirectory and all its subdirectories::
 

Doc/library/importlib.rst

@@ -67,6 +67,9 @@ generically as an :term:`importer`) to participate in the import process.
     :pep:`489`
         Multi-phase extension module initialization
 
+    :pep:`552`
+        Deterministic pycs
+
     :pep:`3120`
         Using UTF-8 as the Default Source Encoding
 
@@ -1327,6 +1330,14 @@ an :term:`importer`.
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
 
+.. function:: source_hash(source_bytes)
+
+   Return the hash of *source_bytes* as byte string. A hash-based pyc embeds the
+   :func:`source_hash` of the corresponding source file's contents in its
+   header.
+
+   .. versionadded:: 3.7
+
 .. class:: LazyLoader(loader)
 
    A class which postpones the execution of the loader of a module until the

Doc/library/py_compile.rst

@@ -27,7 +27,7 @@ byte-code cache files in the directory containing the source code.
    Exception raised when an error occurs while attempting to compile the file.
 
 
-.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1)
+.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP)
 
    Compile a source file to byte-code and write out the byte-code cache file.
    The source code is loaded from the file named *file*.  The byte-code is
@@ -53,6 +53,9 @@ byte-code cache files in the directory containing the source code.
    :func:`compile` function.  The default of ``-1`` selects the optimization
    level of the current interpreter.
 
+   *invalidation_mode* should be a member of the :class:`PycInvalidationMode`
+   enum and controls how the generated pycs are invalidated at runtime.
+
    .. versionchanged:: 3.2
       Changed default value of *cfile* to be :PEP:`3147`-compliant.  Previous
       default was *file* + ``'c'`` (``'o'`` if optimization was enabled).
@@ -65,6 +68,36 @@ byte-code cache files in the directory containing the source code.
       caveat that :exc:`FileExistsError` is raised if *cfile* is a symlink or
       non-regular file.
 
+   .. versionchanged:: 3.7
+      The *invalidation_mode* parameter was added as specified in :pep:`552`.
+
+
+.. class:: PycInvalidationMode
+
+   A enumeration of possible methods the interpreter can use to determine
+   whether a bytecode file is up to date with a source file. The pyc indicates
+   the desired invalidation mode in its header. See :ref:`pyc-invalidation` for
+   more information on how Python invalidates pycs at runtime.
+
+   .. versionadded:: 3.7
+
+   .. attribute:: TIMESTAMP
+
+      The pyc should include the timestamp and size of the source file, which
+      Python will compare against the metadata of the source file at runtime to
+      determine if the pyc needs to be regenerated.
+
+   .. attribute:: CHECKED_HASH
+
+      The pyc should include a hash of the source file which Python will compare
+      against the source at runtime to determine if the pyc needs to be
+      regenerated.
+
+   .. attribute:: UNCHECKED_HASH
+
+      The pyc should include a hash of the source file but Python should not
+      validate it against the source file at runtime.
+
 
 .. function:: main(args=None)
 

Doc/reference/import.rst

@@ -675,6 +675,32 @@ Here are the exact rules used:
    :meth:`~importlib.abc.Loader.module_repr` method, if defined, before
    trying either approach described above.  However, the method is deprecated.
 
+.. _pyc-invalidation:
+
+Cached bytecode invalidation
+----------------------------
+
+Before Python loads cached bytecode from ``.pyc`` file, it checks whether the
+cache is up-to-date with the source ``.py`` file. By default, Python does this
+by storing the source's last-modified timestamp and size in the cache file when
+writing it. At runtime, the import system then validates the cache file by
+checking the stored metadata in the cache file against at source's
+metadata.
+
+Python also supports "hash-based" cache files, which store a hash of a source
+file contents rather than its metadata. There are two variants of hash-based
+pycs: checked and unchecked. For checked hash-based pycs, Python validates the
+cache file by hashing the source file and comparing the resulting hash with the
+hash in the cache file. If a checked hash-based cache file is found to be
+invalid, Python regenerates it and writes a new checked hash-based cache
+file. For unchecked hash-based pycs, Python simply assumes the cache file is
+valid if it exists. Hash-based pyc validation behavior may be override with the
+:option:`--check-hash-based-pycs` flag.
+
+.. versionchanged:: 3.7
+   Added hash-based pycs. Previously, Python only supported timestamp-based pyc
+   invalidation.
+
 
 The Path Based Finder
 =====================

Doc/using/cmdline.rst

@@ -210,6 +210,19 @@ Miscellaneous options
    import of source modules.  See also :envvar:`PYTHONDONTWRITEBYTECODE`.
 
 
+.. cmdoption:: --check-hash-based-pycs default|always|never
+
+   Control the validation behavior of hash-based pycs. See
+   :ref:`pyc-invalidation`. When set to ``default``, checked and unchecked
+   hash-based bytecode cache files are validated according to their default
+   semantics. When set to ``always``, all hash-based pycs, whether checked or
+   unchecked, are validated against their corresponding source file. When set to
+   ``never``, hash-based pycs are not validated against their corresponding
+   source files.
+
+   The semantics of timestamp-based pycs are unaffected by this option.
+
+
 .. cmdoption:: -d
 
    Turn on parser debugging output (for expert only, depending on compilation

Doc/whatsnew/3.7.rst

@@ -197,6 +197,32 @@ variable is not set in practice.
 
 See :option:`-X` ``dev`` for the details.
 
+Hash-based pycs
+---------------
+
+Python has traditionally checked the up-to-dateness of bytecode cache files
+(i.e., pycs) by comparing the source metadata (last-modified timestamp and size)
+with source metadata saved in the cache file header when it was generated. While
+effective, this invalidation method has its drawbacks. When filesystem
+timestamps are too coarse, Python can miss source updates, leading to user
+confusion. Additionally, having a timestamp in the cache file is problematic for
+`build reproduciblity <https://reproducible-builds.org/>`_ and content-based
+build systems.
+
+:pep:`552` extends the pyc format to allow the hash of the source file to be
+used for invalidation instead of the source timestamp. Such pycs are called
+"hash-based". By default, Python still uses timestamp-based invalidation and
+does not generate hash-based pycs at runtime. Hash-based pycs may be generated
+with :mod:`py_compile` or :mod:`compileall`.
+
+Hash-based pycs come in two variants: checked and unchecked. Python validates
+checked hash-based pycs against the source file at runtime but doesn't do so for
+unchecked hash-based pycs. Unchecked hash-based pycs are a useful performance
+optimization for environments where a system external to Python (e.g., the build
+system) is responsible for keeping pycs up-to-date.
+
+See :ref:`pyc-invalidation` for more information.
+
 
 Other Language Changes
 ======================

Include/internal/hash.h

@@ -0,0 +1,6 @@
+#ifndef Py_INTERNAL_HASH_H
+#define Py_INTERNAL_HASH_H
+
+uint64_t _Py_KeyedHash(uint64_t, const char *, Py_ssize_t);
+
+#endif

Include/internal/import.h

@@ -0,0 +1,6 @@
+#ifndef Py_INTERNAL_IMPORT_H
+#define Py_INTERNAL_IMPORT_H
+
+extern const char *_Py_CheckHashBasedPycsMode;
+
+#endif

Include/pygetopt.h

@@ -12,7 +12,14 @@ PyAPI_DATA(wchar_t *) _PyOS_optarg;
 
 PyAPI_FUNC(void) _PyOS_ResetGetOpt(void);
 
-PyAPI_FUNC(int) _PyOS_GetOpt(int argc, wchar_t **argv, wchar_t *optstring);
+typedef struct {
+    const wchar_t *name;
+    int has_arg;
+    int val;
+} _PyOS_LongOption;
+
+PyAPI_FUNC(int) _PyOS_GetOpt(int argc, wchar_t **argv, wchar_t *optstring,
+                             const _PyOS_LongOption *longopts, int *longindex);
 #endif /* !Py_LIMITED_API */
 
 #ifdef __cplusplus