[libcxx-commits] [libcxx] [libc++][lit] Atomically update the persistent cache (PR #66538)

[Alexander Richardson via libcxx-commits]

https://github.com/arichardson created https://github.com/llvm/llvm-project/pull/66538

When running multiple shards in parallel, one shard might write to the cache while another one is reading this cache. Instead of updating the file in place, write to a temporary file and swap the cache file using os.replace(). This is an atomic operation and means shards will either see the old state or the new one.

>From d2884871218a8803b53a22a18eb1a183fbdcdd91 Mon Sep 17 00:00:00 2001
From: Alex Richardson <alexrichardson at google.com>
Date: Fri, 15 Sep 2023 12:07:39 -0700
Subject: [PATCH] [libc++][lit] Atomically update the persistent cache

When running multiple shards in parallel, one shard might write to the
cache while another one is reading this cache. Instead of updating the
file in place, write to a temporary file and swap the cache file using
os.replace(). This is an atomic operation and means shards will either
see the old state or the new one.
---
 libcxx/utils/libcxx/test/dsl.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/libcxx/utils/libcxx/test/dsl.py b/libcxx/utils/libcxx/test/dsl.py
index 847cebf5962f6aa..7d4df6b01eabdae 100644
--- a/libcxx/utils/libcxx/test/dsl.py
+++ b/libcxx/utils/libcxx/test/dsl.py
@@ -69,8 +69,12 @@ def f(config, *args, **kwargs):
             if cacheKey not in cache:
                 cache[cacheKey] = function(config, *args, **kwargs)
                 # Update the persistent cache so it knows about the new key
-                with open(persistentCache, "wb") as cacheFile:
+                # We write to a temporary file and rename the result to ensure
+                # that the cahe is not corrupted when running the test suite
+                # with multiple shards.
+                with open(persistentCache + ".tmp", "wb") as cacheFile:
                     pickle.dump(cache, cacheFile)
+                os.replace(persistentCache + ".tmp", persistentCache)
             return cache[cacheKey]
 
         return f