Add HW 6

homework/06-membership/readme.md

@@ -0,0 +1,138 @@
+# Group Membership
+
+В этом задании вам предстоит реализовать распределённый сервис для отслеживания состава группы. Он должен позволять процессам вступать в группу, покидать её и узнавать её текущий состав.
+
+В системе _N_ узлов, на каждом из которых запущен процесс с уникальным идентификатором _id_. Процессы вступают в _группу_ по команде `JOIN` и покидают её по команде `LEAVE`. Сам по себе запуск процесса не делает его участником группы.
+
+Помимо добровольного выхода, состав группы может меняться из-за отказов. В нашей системе возможны:
+
+- _Падение процесса или всего узла_. Процесс прекращает выполнение и теряет состояние в памяти. Позднее он может быть перезапущен с тем же _id_ и чистым состоянием. Это модель _crash-recovery_ из лекции 6. Работающий процесс будем называть _живым_.
+- _Потери сообщений_ между некоторыми парами узлов, в обоих или только в одном направлении. Живые процессы могут оказаться полностью или частично изолированы от остальных. Сетевые сбои могут быть временными, после чего связь восстанавливается.
+
+Для обнаружения недоступных участников вам потребуется _детектор отказов_ (см. лекцию 6). Он сообщает о подозрении: отсутствие ответа может быть вызвано как падением процесса, так и проблемами сети. От детектора требуется _полнота_: падение участника должно в конечном счёте обнаруживаться всеми оставшимися участниками. Подберите баланс между скоростью обнаружения, числом ложных подозрений и расходами на обмен сообщениями.
+
+Списки участников на разных процессах могут временно различаться после входа, выхода, отказа или восстановления. Требуется _согласованность в конечном счёте_: когда условия работы перестают меняться, списки должны стабилизироваться. Каждый участник должен видеть в своём списке себя и всех живых участников, с которыми возможен обмен сообщениями в обе стороны, в том числе через других участников.
+
+Упавшие и добровольно вышедшие процессы должны автоматически исключаться из списков остальных участников. То же относится к живому процессу, который не может обмениваться сообщениями в обе стороны ни с одним другим участником.
+
+Если сеть распалась на подгруппы, между которыми нет двусторонней связи, происходит _разделение сети_. В этом случае списки должны совпадать внутри каждой подгруппы, включая меньшую, и содержать всех её участников, без процессов из других подгрупп. В списке участников изолированного процесса должен остаться только он сам. Односторонней связи между подгруппами недостаточно для сохранения общей группы.
+
+После перезапуска процесс снова вступает в группу по команде `JOIN`, и остальные участники должны вернуть его в свои списки. При восстановлении связи с временно недоступным живым участником они должны сделать это автоматически, без новой команды `JOIN`. Это нужно учитывать и при использовании идей SWIM из лекции: исключение из локального списка не должно навсегда препятствовать возвращению участника.
+
+Чтобы получить максимальный балл, нужно также обеспечить масштабируемость решения. При увеличении числа узлов в _X_ раз сетевой трафик и число сообщений в единицу времени должны увеличиваться не более чем в _2X_ раз. Нагрузка на процессы измеряется суммой отправленных и полученных сообщений; отношение максимальной нагрузки к минимальной не должно превышать 5.
+
+Подробные сценарии проверок приведены в тестах — они являются частью условия.
+
+## Реализация
+
+Для реализации и тестирования решения используется фреймворк AnySystem (см. материалы первого семинара). В папке `solution` размещена заготовка для решения [member.py](solution/member.py). Вам надо доработать реализацию процесса в классе `GroupMember` так, чтобы проходили все тесты.
+
+Из библиотек разрешены только AnySystem и стандартная библиотека Python. Можно добавлять собственные вспомогательные модули. Сжимать сетевые сообщения запрещено, в том числе с помощью `zlib`, `gzip`, `bz2`, `lzma` или собственного алгоритма сжатия. Снижайте сетевые расходы за счёт устройства протокола: например, передавайте только изменения состава группы вместо полного списка.
+
+Ваши реализации процессов AnySystem должны соблюдать правило [изоляции процессов](../readme.md#изоляция-процессов-anysystem): изменяемое состояние каждого процесса должно храниться только в `self`, а общая изменяемая память между процессами запрещена. Все взаимодействия между процессами должны происходить через сообщения, как и в реальной распределённой системе.
+
+При инициализации процессу передается его уникальный _id_. Процесс должен поддерживать обработку следующих локальных сообщений:
+
+- _JOIN_ - команда присоединиться к группе. В результате обработки этой команды процесс должен стать участником группы. В поле `seed` передается идентификатор одного из живых участников группы. Если в `seed` находится идентификатор самого процесса, то он должен создать новую пустую группу и добавить в неё себя. Ответ на это сообщение не нужен.
+- _LEAVE_ - команда покинуть группу. В результате обработки этой команды процесс должен перестать быть участником группы. Ответ на это сообщение не нужен.
+- _GET_MEMBERS_ - запрос локального списка участников группы. Процесс должен сразу ответить одним локальным сообщением _MEMBERS_ с хранимым списком идентификаторов (см. заготовку). Обращаться к другим процессам или ждать таймера для ответа нельзя. Эта семантика проверяется отдельным тестом.
+
+Для взаимодействия между процессами вы можете использовать любые собственные типы сообщений.
+
+**Важно!** Для измерения прошедшего времени в коде процесса используйте метод `ctx.time()`, возвращающий локальное время "внутри симуляции". Использовать обычные способы, например `time.time()`, некорректно, так как скорость течения времени в симуляции отличается от реального времени. Также не стоит опираться на синхронизацию часов в системе - у каждого узла свои локальные часы, и показания `ctx.time()` у разных процессов могут расходиться. После перезапуска узла смещение его часов может измениться.
+
+В коде процесса можно использовать генератор случайных чисел из `random`, но не следует его явно инициализировать с помощью `random.seed()`. Это уже делается в тестах на основе переданного random seed, а дополнительная инициализация может нарушить воспроизводимость результатов.
+
+## Тестирование
+
+### Локальное тестирование
+
+Тесты находятся в папке `tests`. Есть два варианта их запуска.
+
+Рекомендуемый вариант запуска тестов - через готовый Docker-образ. В этом случае используемое окружение будет аналогично тестирующей системе. Для запуска тестов выполните команду:
+
+```commandline
+docker run --pull always --rm -t -v ./solution:/solution distsys.ru/course/membership:latest [ЗДЕСЬ МОЖНО УКАЗАТЬ ОПЦИИ]
+```
+
+Для запуска полного набора тестов с теми же параметрами, что и в тестирующей системе, выполните команду:
+
+```commandline
+docker run --pull always --rm -t -v ./solution:/solution distsys.ru/course/membership:latest -m 1000
+```
+
+Вы также можете собрать и запустить тесты непосредственно в Linux с помощью компилятора Rust. На Windows можно использовать WSL или Docker, на macOS — Docker. Сборка из исходников может быть удобна, если вы хотите лучше изучить или доработать тесты. Скомпилируйте тесты с помощью команды `cargo install --locked --path tests`. Для запуска тестов выполните команду:
+
+```commandline
+distsys-membership [ЗДЕСЬ МОЖНО УКАЗАТЬ ОПЦИИ]
+```
+
+Запустить один тест можно с помощью опции `-t` (например, `... -t "PROCESS CRASH"`). По умолчанию тестер выводит результаты проверок, диагностику долгих запусков и итоговую сводку. Флаг `-d` добавляет трассу событий, а `-h` показывает все доступные опции. При отдельном запуске теста масштабируемости через `-t` предварительное прохождение функциональных тестов не требуется.
+
+### Что проверяют тесты
+
+Сначала выполняются обычные функциональные тесты, затем `CHAOS MONKEY`, `MODEL CHECKING` и тесты масштабируемости.
+
+На каждом этапе обычных тестов списки участников должны сойтись к ожидаемому составу за 300 секунд симуляции. В тестах изоляции проверяются и оставшаяся группа, и изолированный участник: в его списке должен остаться только он сам.
+
+Тесты `NETWORK PARTITION UNEQUAL` и `NETWORK PARTITION UNEQUAL RECOVER` разделяют сеть на две непустые подгруппы разного размера: 4 и 6 при десяти процессах. Состав группы должен стабилизироваться в обеих подгруппах, включая меньшую. Второй тест затем восстанавливает связь и проверяет, что все процессы снова видят общую группу.
+
+Тесты `TWO PROCESSES CANNOT COMMUNICATE` и `TWO PROCESSES CANNOT COMMUNICATE IN TRIANGLE` отключают связь между парой узлов, сохраняя связь через остальных участников. Во втором тесте всегда три узла. Все участники должны остаться в группе.
+
+В этих двух тестах, а также после восстановления в `PROCESS CRASH RECOVER`, `PROCESS OFFLINE RECOVER`, `NETWORK PARTITION RECOVER` и `NETWORK PARTITION UNEQUAL RECOVER` проверяется непрерывная стабильность состава группы. За 300 секунд симуляции должен найтись интервал длительностью 30 секунд, в котором списки участников остаются корректными. Внутри этого интервала списки проверяются после каждого события. Переходные ошибки до стабилизации допускаются.
+
+`CHAOS MONKEY` проверяет последовательности отказов. По умолчанию выполняются 100 прогонов, а в тестирующей системе — 1000 (`-m 1000`). Каждый прогон включает до шести шагов с отказами; при стандартных десяти процессах в проверяемой группе может остаться четыре участника.
+
+Для отладки `CHAOS MONKEY` используйте флаг `-v` (`--verbose`): он выводит параметры запуска, порядок входа процессов и отказы с номерами процессов и временем в симуляции. Этот вывод также включается с `-d`. Чтобы повторить запуск, укажите то же значение `-s` (`--seed`) и то же число процессов `-p`, а в `-m` — число не меньше номера нужного прогона. Например, `... -t "CHAOS MONKEY (run 850)" -m 1000 -s 123 -p 10 -v`. В тестах выбор процессов воспроизводим при одинаковых параметрах; случайность внутри вашего решения также должна быть воспроизводимой.
+
+`MODEL CHECKING` (MC) проверяет сборку группы из трёх процессов без отказов и ответы на `GET_MEMBERS` при разных порядках доставки сообщений. Полный состав ожидается в состояниях, где каждый процесс получил не менее трёх сетевых сообщений или больше нет событий. В тестирующей системе MC включён; для быстрой локальной проверки его можно отключить флагом `--disable-mc-tests`.
+
+Тесты `SCALABILITY NORMAL` и `SCALABILITY CRASH` измеряют расходы на поддержание группы. В первом тесте измерение длится 10 секунд симуляции после сборки группы, во втором — от падения одного процесса до стабилизации состава. Расходы на первоначальную сборку группы не учитываются; во втором тесте равномерность нагрузки проверяется среди оставшихся живых процессов.
+
+При стандартных параметрах оба теста масштабируемости запускаются для групп из 10, 20, 50 и 100 узлов. Сетевой трафик и число сообщений в единицу времени сравниваются между последовательными запусками: 10 → 20, 20 → 50 и 50 → 100. По правилу _2X_ допустимый рост расходов составляет соответственно 4, 5 и 4 раза.
+
+### Лимиты и диагностика
+
+Тесты выполняются в отдельных процессах; прогоны `CHAOS MONKEY` объединены в серию с сохранением результата каждого прогона. При зависании, исчерпании памяти или аварийном завершении исполнителя проверка продолжается, пока остаётся общий бюджет времени. Баллы за пройденные функциональные компоненты и сетевые подгруппы сохраняются. Непроверенные тесты отмечаются в сводке и не считаются пройденными.
+
+По умолчанию и в тестирующей системе действуют следующие ограничения реального времени. Все проверки делят общий бюджет 540 секунд:
+
+| Проверка | Лимит | Опция |
+| --- | --- | --- |
+| Обычный тест или один прогон CHAOS MONKEY | 15 секунд | `--test-timeout` |
+| Все прогоны CHAOS MONKEY вместе | 300 секунд | `--chaos-timeout` |
+| Обе фазы MODEL CHECKING вместе | 300 секунд | `--mc-timeout` |
+| Каждый тест SCALABILITY | 60 секунд | `--scalability-timeout` |
+| Полный набор тестов | 540 секунд | `--suite-timeout` |
+
+Адресное пространство исполнителя ограничено 4 ГиБ для обычных тестов, CHAOS MONKEY и SCALABILITY (`--memory-limit 4096`) и 14 ГиБ для MODEL CHECKING (`--mc-memory-limit 14336`). В тестирующей системе эти пределы фиксированы. Окружение должно также иметь запас 256 МиБ для управляющего процесса.
+
+Если памяти окружения недостаточно для заданного предела и запаса для управляющего процесса, тестер завершится с сообщением об ошибке окружения без оценки. Для локального запуска в меньшем объёме памяти можно явно уменьшить соответствующий предел, но результат может отличаться от проверки на сервере.
+
+При остановке тестер сообщает причину и последний этап проверки: например, размер собираемой группы или фазу model checking. Долгие проверки также периодически выводят текущий этап и расход памяти.
+
+Бонусы за пробелы в тестах начисляются по [общим правилам](../readme.md#бонусы-за-пробелы-в-тестах).
+
+### Проверка в тестирующей системе
+
+Отправьте решение в тестирующую систему по [инструкции](../readme.md) и дождитесь результатов.
+
+## Оценивание
+
+Это задание имеет повышенный вес 1.5, то есть максимум можно набрать **15 баллов**.
+
+Базовый функционал, отказы процессов и четыре сетевые подгруппы оцениваются независимо друг от друга. Внутри каждого компонента или подгруппы нужно пройти все перечисленные обычные тесты; исключения для MC описаны ниже. Масштабируемость даёт баллы только при зачёте всей функциональной части.
+
+- **Базовый функционал без отказов — до 4 баллов:**
+  - **3 балла** за прохождение всех пяти тестов: `SIMPLE`, `GET MEMBERS SEMANTICS`, `RANDOM SEED`, `PROCESS JOIN`, `PROCESS LEAVE`.
+  - **Ещё 1 балл**, если пройдены и все пять обычных тестов, и `MODEL CHECKING`. Отключение MC при локальном запуске или остановка уже запущенного MC по лимиту времени (в том числе общего бюджета) или памяти сохраняет первые 3 балла.
+  - **0 баллов за весь компонент**, если MC обнаружил ошибку, не запустился из-за исчерпания общего бюджета или его исполнитель аварийно завершился с неизвестной причиной.
+- **Отказы процессов — 3 балла:** `PROCESS CRASH`, `SEED PROCESS CRASH`, `PROCESS CRASH RECOVER`.
+- **Сетевые и комбинированные отказы — до 4 баллов, по 1 баллу за каждую подгруппу:**
+  - **Отключения и разделения сети с восстановлением:** `PROCESS OFFLINE`, `SEED PROCESS OFFLINE`, `PROCESS OFFLINE RECOVER`, `NETWORK PARTITION`, `NETWORK PARTITION RECOVER`, `NETWORK PARTITION UNEQUAL`, `NETWORK PARTITION UNEQUAL RECOVER`.
+  - **Односторонняя и частичная связность:** `PROCESS CANNOT RECEIVE`, `PROCESS JOIN CANNOT RECEIVE`, `PROCESS CANNOT SEND`, `TWO PROCESSES CANNOT COMMUNICATE`, `TWO PROCESSES CANNOT COMMUNICATE IN TRIANGLE`.
+  - **Задержки и потери сообщений:** `SLOW NETWORK`, `FLAKY NETWORK`, `FLAKY NETWORK ON START`, `FLAKY NETWORK AND CRASH`.
+  - **Последовательности отказов:** все 1000 прогонов `CHAOS MONKEY`.
+- **Масштабируемость — 4 балла:** `SCALABILITY NORMAL`, `SCALABILITY CRASH`. Эти тесты запускаются в полном наборе только после зачёта всех функциональных компонентов: базовой группы (не менее 3 баллов), отказов процессов и всех четырёх сетевых подгрупп. Остановка уже запущенного MC по времени или памяти и его отключение при локальном запуске не блокируют масштабируемость. Если функциональные компоненты не зачтены, тесты масштабируемости пропускаются с указанием причины. Для получения 4 баллов нужно пройти оба теста; при исчерпании общего бюджета незапущенные тесты не считаются пройденными.
+
+К решению приложите отчёт в файле `solution/readme.md` по [общим требованиям](../readme.md#отчёт). Без отчёта автоматические тесты запускаются, но защита не проводится и решение не засчитывается.

homework/06-membership/solution/member.py

@@ -0,0 +1,34 @@
+from anysystem import Context, Message, Process
+
+
+class GroupMember(Process):
+    def __init__(self, proc_id: str):
+        self._id = proc_id
+
+    def on_local_message(self, msg: Message, ctx: Context):
+        if msg.type == 'JOIN':
+            # Add process to the group
+            seed = msg['seed']
+            if seed == self._id:
+                # create new empty group and add process to it
+                pass
+            else:
+                # join existing group
+                pass
+        elif msg.type == 'LEAVE':
+            # Remove process from the group
+            pass
+        elif msg.type == 'GET_MEMBERS':
+            # Get a list of group members
+            # - return the list of all known alive processes in MEMBERS message
+            ctx.send_local(Message('MEMBERS', {'members': [self._id]}))
+
+    def on_start(self, ctx: Context):
+        pass
+
+    def on_message(self, msg: Message, sender: str, ctx: Context):
+        # Implement communication between processes using any message types
+        pass
+
+    def on_timer(self, timer_name: str, ctx: Context):
+        pass

homework/06-membership/tests/.dockerignore

@@ -0,0 +1,5 @@
+target
+**/target
+**/__pycache__
+**/.pytest_cache
+*.pyc

homework/06-membership/tests/Cargo.lock

@@ -0,0 +1,993 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "aho-corasick"
+version = "1.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "anysystem"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e01badd3bc0d042e4399f8f685c394a79a72e7f233b3d3b5eabb3b94734f4dbe"
+dependencies = [
+ "colored",
+ "downcast-rs",
+ "dyn-clone",
+ "indexmap 2.14.0",
+ "lazy_static",
+ "log",
+ "ordered-float",
+ "pyo3",
+ "rand",
+ "rand_pcg",
+ "regex",
+ "rstest",
+ "serde",
+ "serde_json",
+ "simcore",
+ "sugars",
+]
+
+[[package]]
+name = "assertables"
+version = "3.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d84b91d63c67e6c0b1a74728dad0057c4a6124b74e671ff91b7fe9f8bbd671aa"
+
+[[package]]
+name = "atty"
+version = "0.2.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
+dependencies = [
+ "hermit-abi",
+ "libc",
+ "winapi",
+]
+
+[[package]]
+name = "autocfg"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
+
+[[package]]
+name = "bitflags"
+version = "1.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+
+[[package]]
+name = "cfg-if"
+version = "1.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2fd1289c04a9ea8cb22300a459a72a385d7c73d3259e2ed7dcb2af674838cfa9"
+
+[[package]]
+name = "clap"
+version = "3.2.25"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4ea181bf566f71cb9a5d17a59e1871af638180a18fb0035c92ae62b705207123"
+dependencies = [
+ "atty",
+ "bitflags",
+ "clap_derive",
+ "clap_lex",
+ "indexmap 1.9.3",
+ "once_cell",
+ "strsim",
+ "termcolor",
+ "textwrap",
+]
+
+[[package]]
+name = "clap_derive"
+version = "3.2.25"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ae6371b8bdc8b7d3959e9cf7b22d4435ef3e79e138688421ec654acf8c81b008"
+dependencies = [
+ "heck 0.4.1",
+ "proc-macro-error",
+ "proc-macro2",
+ "quote",
+ "syn 1.0.109",
+]
+
+[[package]]
+name = "clap_lex"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2850f2f5a82cbf437dd5af4d49848fbdfc27c157c3d010345776f952765261c5"
+dependencies = [
+ "os_str_bytes",
+]
+
+[[package]]
+name = "colored"
+version = "2.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "117725a109d387c937a1533ce01b450cbde6b88abceea8473c4d7a85853cda3c"
+dependencies = [
+ "lazy_static",
+ "windows-sys 0.59.0",
+]
+
+[[package]]
+name = "distsys-membership"
+version = "0.1.0"
+dependencies = [
+ "anysystem",
+ "assertables",
+ "clap",
+ "env_logger",
+ "indexmap 2.14.0",
+ "libc",
+ "log",
+ "pyo3",
+ "rand",
+ "rand_pcg",
+ "serde",
+ "serde_json",
+ "sugars",
+]
+
+[[package]]
+name = "downcast-rs"
+version = "1.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "75b325c5dbd37f80359721ad39aca5a29fb04c89279657cffdda8736d0c0b9d2"
+
+[[package]]
+name = "dyn-clone"
+version = "1.0.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
+
+[[package]]
+name = "env_logger"
+version = "0.9.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a12e6657c4c97ebab115a42dcee77225f7f482cdd841cf7088c657a42e9e00e7"
+dependencies = [
+ "atty",
+ "humantime",
+ "log",
+ "regex",
+ "termcolor",
+]
+
+[[package]]
+name = "equivalent"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
+
+[[package]]
+name = "erased-serde"
+version = "0.4.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e004d887f51fcb9fef17317a2f3525c887d8aa3f4f50fed920816a688284a5b7"
+dependencies = [
+ "serde",
+ "typeid",
+]
+
+[[package]]
+name = "futures"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "65bc07b1a8bc7c85c5f2e110c476c7389b4554ba72af57d8445ea63a576b0876"
+dependencies = [
+ "futures-channel",
+ "futures-core",
+ "futures-executor",
+ "futures-io",
+ "futures-sink",
+ "futures-task",
+ "futures-util",
+]
+
+[[package]]
+name = "futures-channel"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10"
+dependencies = [
+ "futures-core",
+ "futures-sink",
+]
+
+[[package]]
+name = "futures-core"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e"
+
+[[package]]
+name = "futures-executor"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f"
+dependencies = [
+ "futures-core",
+ "futures-task",
+ "futures-util",
+]
+
+[[package]]
+name = "futures-io"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6"
+
+[[package]]
+name = "futures-macro"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.106",
+]
+
+[[package]]
+name = "futures-sink"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7"
+
+[[package]]
+name = "futures-task"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988"
+
+[[package]]
+name = "futures-timer"
+version = "3.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24"
+
+[[package]]
+name = "futures-util"
+version = "0.3.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81"
+dependencies = [
+ "futures-channel",
+ "futures-core",
+ "futures-io",
+ "futures-macro",
+ "futures-sink",
+ "futures-task",
+ "memchr",
+ "pin-project-lite",
+ "pin-utils",
+ "slab",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.2.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "wasi",
+]
+
+[[package]]
+name = "glob"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
+
+[[package]]
+name = "hashbrown"
+version = "0.12.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
+
+[[package]]
+name = "hashbrown"
+version = "0.17.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
+
+[[package]]
+name = "heck"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
+
+[[package]]
+name = "heck"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
+
+[[package]]
+name = "hermit-abi"
+version = "0.1.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33"
+dependencies = [
+ "libc",
+]
+
+[[package]]
+name = "humantime"
+version = "2.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424"
+
+[[package]]
+name = "indexmap"
+version = "1.9.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99"
+dependencies = [
+ "autocfg",
+ "hashbrown 0.12.3",
+]
+
+[[package]]
+name = "indexmap"
+version = "2.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
+dependencies = [
+ "equivalent",
+ "hashbrown 0.17.1",
+]
+
+[[package]]
+name = "itoa"
+version = "1.0.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c"
+
+[[package]]
+name = "lazy_static"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+
+[[package]]
+name = "libc"
+version = "0.2.175"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6a82ae493e598baaea5209805c49bbf2ea7de956d50d7da0da1164f9c6d28543"
+
+[[package]]
+name = "log"
+version = "0.4.28"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432"
+
+[[package]]
+name = "memchr"
+version = "2.7.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a282da65faaf38286cf3be983213fcf1d2e2a58700e808f83f4ea9a4804bc0"
+
+[[package]]
+name = "num-traits"
+version = "0.2.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
+dependencies = [
+ "autocfg",
+]
+
+[[package]]
+name = "once_cell"
+version = "1.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d"
+
+[[package]]
+name = "ordered-float"
+version = "4.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7bb71e1b3fa6ca1c61f383464aaf2bb0e2f8e772a1f01d486832464de363b951"
+dependencies = [
+ "num-traits",
+ "rand",
+ "serde",
+]
+
+[[package]]
+name = "os_str_bytes"
+version = "6.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e2355d85b9a3786f481747ced0e0ff2ba35213a1f9bd406ed906554d7af805a1"
+
+[[package]]
+name = "pin-project-lite"
+version = "0.2.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b"
+
+[[package]]
+name = "pin-utils"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
+
+[[package]]
+name = "portable-atomic"
+version = "1.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
+
+[[package]]
+name = "ppv-lite86"
+version = "0.2.21"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
+dependencies = [
+ "zerocopy",
+]
+
+[[package]]
+name = "proc-macro-crate"
+version = "3.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "edce586971a4dfaa28950c6f18ed55e0406c1ab88bbce2c6f6293a7aaba73d35"
+dependencies = [
+ "toml_edit",
+]
+
+[[package]]
+name = "proc-macro-error"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "da25490ff9892aab3fcf7c36f08cfb902dd3e71ca0f9f9517bea02a73a5ce38c"
+dependencies = [
+ "proc-macro-error-attr",
+ "proc-macro2",
+ "quote",
+ "syn 1.0.109",
+ "version_check",
+]
+
+[[package]]
+name = "proc-macro-error-attr"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a1be40180e52ecc98ad80b184934baf3d0d29f979574e439af5a55274b35f869"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "version_check",
+]
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.101"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "pyo3"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4688ddedf473e32662b9b067670129a8afb8c18e351482c70d62ba4a88171e8b"
+dependencies = [
+ "libc",
+ "once_cell",
+ "portable-atomic",
+ "pyo3-build-config",
+ "pyo3-ffi",
+ "pyo3-macros",
+]
+
+[[package]]
+name = "pyo3-build-config"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f41027e41b4bd03f6e60f9f417fe24a6341a6bb744edd62b6f709f2a52ea30e9"
+dependencies = [
+ "target-lexicon",
+]
+
+[[package]]
+name = "pyo3-ffi"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e591a95526fead067432c3b3a33fc74770b87b1e04e73671090d9c2055a2b327"
+dependencies = [
+ "libc",
+ "pyo3-build-config",
+]
+
+[[package]]
+name = "pyo3-macros"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "73225868fc1cd84eef2c3c230ddb91273bf1de46aeb8a4248da76d32a0924a1c"
+dependencies = [
+ "proc-macro2",
+ "pyo3-macros-backend",
+ "quote",
+ "syn 2.0.106",
+]
+
+[[package]]
+name = "pyo3-macros-backend"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "571575aa3749fa6216757dd47d2a3e7ef360f329a40f0666a9fbd14889024952"
+dependencies = [
+ "heck 0.5.0",
+ "proc-macro2",
+ "quote",
+ "syn 2.0.106",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.40"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1885c039570dc00dcb4ff087a89e185fd56bae234ddc7f056a945bf36467248d"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "rand"
+version = "0.8.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
+dependencies = [
+ "libc",
+ "rand_chacha",
+ "rand_core",
+ "serde",
+]
+
+[[package]]
+name = "rand_chacha"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
+dependencies = [
+ "ppv-lite86",
+ "rand_core",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+dependencies = [
+ "getrandom",
+ "serde",
+]
+
+[[package]]
+name = "rand_pcg"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "59cad018caf63deb318e5a4586d99a24424a364f40f1e5778c29aca23f4fc73e"
+dependencies = [
+ "rand_core",
+]
+
+[[package]]
+name = "regex"
+version = "1.11.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "23d7fd106d8c02486a8d64e778353d1cffe08ce79ac2e82f540c86d0facf6912"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b9458fa0bfeeac22b5ca447c63aaf45f28439a709ccd244698632f9aa6394d6"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "caf4aa5b0f434c91fe5c7f1ecb6a5ece2130b02ad2a590589dda5146df959001"
+
+[[package]]
+name = "relative-path"
+version = "1.9.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2"
+
+[[package]]
+name = "rstest"
+version = "0.21.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9afd55a67069d6e434a95161415f5beeada95a01c7b815508a82dcb0e1593682"
+dependencies = [
+ "futures",
+ "futures-timer",
+ "rstest_macros",
+ "rustc_version",
+]
+
+[[package]]
+name = "rstest_macros"
+version = "0.21.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4165dfae59a39dd41d8dec720d3cbfbc71f69744efb480a3920f5d4e0cc6798d"
+dependencies = [
+ "cfg-if",
+ "glob",
+ "proc-macro-crate",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "relative-path",
+ "rustc_version",
+ "syn 2.0.106",
+ "unicode-ident",
+]
+
+[[package]]
+name = "rustc-hash"
+version = "2.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d"
+
+[[package]]
+name = "rustc_version"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
+dependencies = [
+ "semver",
+]
+
+[[package]]
+name = "ryu"
+version = "1.0.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "28d3b2b1366ec20994f1fd18c3c594f05c5dd4bc44d8bb0c1c632c8d6829481f"
+
+[[package]]
+name = "semver"
+version = "1.0.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "56e6fa9c48d24d85fb3de5ad847117517440f6beceb7798af16b4a87d616b8d0"
+
+[[package]]
+name = "serde"
+version = "1.0.219"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5f0e2c6ed6606019b4e29e69dbaba95b11854410e5347d525002456dbbb786b6"
+dependencies = [
+ "serde_derive",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.219"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5b0276cf7f2c73365f7157c8123c21cd9a50fbbd844757af28ca1f5925fc2a00"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.106",
+]
+
+[[package]]
+name = "serde_json"
+version = "1.0.143"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d401abef1d108fbd9cbaebc3e46611f4b1021f714a0597a71f41ee463f5f4a5a"
+dependencies = [
+ "indexmap 2.14.0",
+ "itoa",
+ "memchr",
+ "ryu",
+ "serde",
+]
+
+[[package]]
+name = "serde_type_name"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "92acc4cd6ae20767c54d6cf1a075624e7f4d9e99d7ebc685398ff243144d8714"
+dependencies = [
+ "serde",
+]
+
+[[package]]
+name = "simcore"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a20dfec85e94e569fa5b810b2d992c68b361f31bfba471447427536488c9c292"
+dependencies = [
+ "colored",
+ "downcast-rs",
+ "dyn-clone",
+ "erased-serde",
+ "futures",
+ "log",
+ "rand",
+ "rand_pcg",
+ "rustc-hash",
+ "serde",
+ "serde_json",
+ "serde_type_name",
+]
+
+[[package]]
+name = "slab"
+version = "0.4.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7a2ae44ef20feb57a68b23d846850f861394c2e02dc425a50098ae8c90267589"
+
+[[package]]
+name = "strsim"
+version = "0.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "73473c0e59e6d5812c5dfe2a064a6444949f089e20eec9a2e5506596494e4623"
+
+[[package]]
+name = "sugars"
+version = "3.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc0db74f9ee706e039d031a560bd7d110c7022f016051b3d33eeff9583e3e67a"
+
+[[package]]
+name = "syn"
+version = "1.0.109"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "syn"
+version = "2.0.106"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ede7c438028d4436d71104916910f5bb611972c5cfd7f89b8300a8186e6fada6"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "target-lexicon"
+version = "0.13.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
+
+[[package]]
+name = "termcolor"
+version = "1.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755"
+dependencies = [
+ "winapi-util",
+]
+
+[[package]]
+name = "textwrap"
+version = "0.16.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057"
+
+[[package]]
+name = "toml_datetime"
+version = "0.6.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
+
+[[package]]
+name = "toml_edit"
+version = "0.22.27"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
+dependencies = [
+ "indexmap 2.14.0",
+ "toml_datetime",
+ "winnow",
+]
+
+[[package]]
+name = "typeid"
+version = "1.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c"
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d"
+
+[[package]]
+name = "version_check"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
+
+[[package]]
+name = "wasi"
+version = "0.11.1+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+
+[[package]]
+name = "winapi"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
+dependencies = [
+ "winapi-i686-pc-windows-gnu",
+ "winapi-x86_64-pc-windows-gnu",
+]
+
+[[package]]
+name = "winapi-i686-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
+
+[[package]]
+name = "winapi-util"
+version = "0.1.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
+dependencies = [
+ "windows-sys 0.61.0",
+]
+
+[[package]]
+name = "winapi-x86_64-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
+
+[[package]]
+name = "windows-link"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "45e46c0661abb7180e7b9c281db115305d49ca1709ab8242adf09666d2173c65"
+
+[[package]]
+name = "windows-sys"
+version = "0.59.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
+dependencies = [
+ "windows-targets",
+]
+
+[[package]]
+name = "windows-sys"
+version = "0.61.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e201184e40b2ede64bc2ea34968b28e33622acdbbf37104f0e4a33f7abe657aa"
+dependencies = [
+ "windows-link",
+]
+
+[[package]]
+name = "windows-targets"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
+dependencies = [
+ "windows_aarch64_gnullvm",
+ "windows_aarch64_msvc",
+ "windows_i686_gnu",
+ "windows_i686_gnullvm",
+ "windows_i686_msvc",
+ "windows_x86_64_gnu",
+ "windows_x86_64_gnullvm",
+ "windows_x86_64_msvc",
+]
+
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+
+[[package]]
+name = "windows_i686_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
+
+[[package]]
+name = "windows_i686_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+
+[[package]]
+name = "windows_i686_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
+
+[[package]]
+name = "winnow"
+version = "0.7.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "21a0236b59786fed61e2a80582dd500fe61f18b5dca67a4a067d0bc9039339cf"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "zerocopy"
+version = "0.8.27"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0894878a5fa3edfd6da3f88c4805f4c8558e2b996227a3d864f47fe11e38282c"
+dependencies = [
+ "zerocopy-derive",
+]
+
+[[package]]
+name = "zerocopy-derive"
+version = "0.8.27"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.106",
+]

homework/06-membership/tests/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+name = "distsys-membership"
+version = "0.1.0"
+edition = "2021"
+resolver = "3"
+
+[dependencies]
+anysystem = "=0.3.0"
+indexmap = "2.12"
+assertables = "3.2.2"
+clap = { version = "3.2.17", features = ["cargo", "derive"] }
+env_logger = "0.9.0"
+log = "0.4.17"
+pyo3 = { version = "=0.29.2", features = ["auto-initialize"] }
+rand = "0.8.5"
+rand_pcg = "0.3.1"
+serde = { version = "1.0.142", features = ["derive"] }
+serde_json = "1.0.83"
+sugars = "3.0.1"
+
+[target.'cfg(target_os = "linux")'.dependencies]
+libc = "0.2.175"

homework/06-membership/tests/Dockerfile

@@ -0,0 +1,19 @@
+# syntax=docker/dockerfile:1
+FROM rust:1.97.1-bookworm@sha256:0e2bcaef56d041a486784e54104a81aebe0da44bd03019bd70bc0401e42e4a97 AS builder
+
+RUN apt-get update && apt-get install -y build-essential python3-dev
+
+WORKDIR /tests
+COPY . .
+RUN --mount=type=cache,id=distsys-course-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
+    --mount=type=cache,id=distsys-course-cargo-target-rust-1-97-1,target=/tests/target,sharing=locked \
+    cargo install --locked --path . --root /opt/course
+
+FROM debian:bookworm-slim
+
+RUN apt-get update && apt-get install -y python3-dev && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /opt/course/bin/distsys-membership /usr/local/bin/distsys-membership
+
+WORKDIR /solution
+ENTRYPOINT ["timeout", "--verbose", "-k", "10", "600", "distsys-membership", "-i", "member.py"]

homework/06-membership/tests/library_policy.py

@@ -0,0 +1,134 @@
+"""HW6 library rules; inspect student source without importing it.
+
+The general shared-state checker is passed in by the embedded Rust wrapper so
+its import resolution is shared without changing the policy of other homework.
+This checks common compression APIs, not arbitrary compression algorithms.
+"""
+
+import ast
+from pathlib import Path
+import sys
+
+COMPRESSION_APIS = {
+    "zlib": {"compress", "compressobj", "decompress", "decompressobj"},
+    "gzip": {"compress", "decompress", "open", "GzipFile"},
+    "bz2": {"compress", "decompress", "open", "BZ2File", "BZ2Compressor", "BZ2Decompressor"},
+    "lzma": {"compress", "decompress", "open", "LZMAFile", "LZMACompressor", "LZMADecompressor"},
+}
+
+
+def _check_file(tree, current, root, checker):
+    violations = set()
+    pending = checker._local_imports(tree, current, root)
+    modules, functions = checker._dynamic_import_names(tree)
+    strings = checker._static_string_values(tree)
+    symbols = {}
+
+    def report(node, message):
+        violations.add(f"{current}:{node.lineno}:{node.col_offset + 1}: {message}")
+
+    def check_module(name, node, base=root):
+        parts = name.split(".")
+        local = checker._candidate_module(base, parts)
+        # Inspect package initializers as well as the explicitly imported file.
+        for index in range(1, len(parts) + 1):
+            candidate = checker._candidate_module(base, parts[:index])
+            if candidate is not None:
+                pending.add(candidate)
+        namespace = base.joinpath(*parts).resolve()
+        local_namespace = namespace.is_relative_to(root) and namespace.is_dir()
+        if local is None and not local_namespace and parts[0] not in sys.stdlib_module_names and parts[0] != "anysystem":
+            report(node, f"library '{name}' is not allowed: use AnySystem and the Python standard library")
+
+    def dynamic_name(node):
+        argument = node.args[0] if node.args else next(
+            (item.value for item in node.keywords if item.arg == "name"), None
+        )
+        return checker._static_string(argument, strings) if argument is not None else None
+
+    def symbol(node):
+        if isinstance(node, ast.Name):
+            return symbols.get(node.id)
+        if isinstance(node, ast.Call) and checker._is_dynamic_import_call(node, modules, functions):
+            return dynamic_name(node)
+        resolved = checker._resolved_attribute(node)
+        if resolved is not None:
+            parent = symbol(resolved[0])
+            if parent is not None:
+                return f"{parent}.{resolved[1]}"
+        return None
+
+    def check_compression(name, node):
+        parts = name.split(".")
+        if len(parts) >= 2 and parts[1] in COMPRESSION_APIS.get(parts[0], set()):
+            report(node, f"message compression is forbidden in HW6: '{parts[0]}.{parts[1]}'")
+
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Import):
+            for alias in node.names:
+                check_module(alias.name, node)
+                symbols[alias.asname or alias.name.split(".")[0]] = alias.name if alias.asname else alias.name.split(".")[0]
+        elif isinstance(node, ast.ImportFrom):
+            base = root
+            if node.level:
+                base = current.parent
+                for _ in range(node.level - 1):
+                    base = base.parent
+                if not base.is_relative_to(root):
+                    report(node, "local imports must stay inside the solution directory")
+                    continue
+            if node.module:
+                check_module(node.module, node, base)
+            for alias in node.names:
+                name = f"{node.module}.{alias.name}" if node.module else alias.name
+                if checker._candidate_module(base, name.split(".")) is not None:
+                    check_module(name, node, base)
+                if alias.name == "*" and node.module in COMPRESSION_APIS:
+                    report(node, f"message compression is forbidden in HW6: star import from '{node.module}'")
+                else:
+                    check_compression(name, node)
+                symbols[alias.asname or alias.name] = name
+        elif isinstance(node, ast.Call) and checker._is_dynamic_import_call(node, modules, functions):
+            name = dynamic_name(node)
+            if name is None or name.startswith("."):
+                report(node, "dynamic imports must use a statically known absolute module name")
+            else:
+                check_module(name, node)
+
+    # Follow aliases, including compressor functions saved for later use.
+    assignments = [parts for node in ast.walk(tree) if (parts := checker._assignment_parts(node)) is not None]
+    for _ in range(len(assignments) + 1):
+        changed = False
+        for value, targets in assignments:
+            name = symbol(value)
+            if name is None:
+                continue
+            for target in targets:
+                if isinstance(target, ast.Name) and target.id not in symbols:
+                    symbols[target.id] = name
+                    changed = True
+        if not changed:
+            break
+    for node in ast.walk(tree):
+        name = symbol(node)
+        if name is not None:
+            check_compression(name, node)
+    return violations, pending
+
+
+def validate_solution(path, checker):
+    entrypoint = Path(path).resolve()
+    root = entrypoint.parent
+    pending = {entrypoint}
+    visited = set()
+    violations = set()
+    while pending:
+        current = pending.pop()
+        if current in visited:
+            continue
+        visited.add(current)
+        tree = ast.parse(current.read_text(encoding="utf-8-sig"), filename=str(current))
+        errors, imports = _check_file(tree, current, root, checker)
+        violations.update(errors)
+        pending.update(imports - visited)
+    return sorted(violations)

homework/06-membership/tests/src/common.rs

@@ -0,0 +1,44 @@
+use serde::{Deserialize, Serialize};
+use sugars::boxed;
+
+use anysystem::python::PyProcessFactory;
+use anysystem::System;
+
+#[derive(Serialize)]
+pub struct JoinMessage<'a> {
+    pub seed: &'a str,
+}
+
+#[derive(Serialize)]
+pub struct LeaveMessage {}
+
+#[derive(Serialize)]
+pub struct GetMembersMessage {}
+
+#[derive(Deserialize)]
+pub struct MembersMessage {
+    pub members: Vec<String>,
+}
+
+#[derive(Clone, Copy)]
+pub struct TestConfig<'a> {
+    pub process_factory: &'a PyProcessFactory,
+    pub process_count: u32,
+    pub seed: u64,
+}
+
+pub fn build_system(config: &TestConfig) -> System {
+    crate::runner::progress(format!("creating simulation: N={}", config.process_count));
+    let mut sys = System::new(config.seed);
+    sys.network().set_delays(0.01, 0.1);
+    for n in 0..config.process_count {
+        // process and node on which it runs have the same name
+        let name = n.to_string();
+        sys.add_node(&name);
+        let clock_skew = sys.gen_range(0.0..10.0);
+        sys.set_node_clock_skew(&name, clock_skew);
+        let process = config.process_factory.build((&name,), config.seed);
+        sys.add_process(&name, boxed!(process), &name);
+    }
+    sys
+}

homework/06-membership/tests/src/library_policy.rs

@@ -0,0 +1,36 @@
+use std::ffi::CString;
+
+use pyo3::prelude::*;
+use pyo3::types::PyModule;
+
+const POLICY_CODE: &str = include_str!("../library_policy.py");
+
+fn validate_solution(path: &str) -> Result<Vec<String>, String> {
+    Python::attach(|py| {
+        let checker =
+            crate::shared_state::validator_module(py).map_err(|error| error.to_string())?;
+        let code = CString::new(POLICY_CODE).expect("library policy contains a null byte");
+        let policy = PyModule::from_code(py, &code, c"library_policy.py", c"library_policy")
+            .map_err(|error| error.to_string())?;
+        policy
+            .getattr("validate_solution")
+            .and_then(|function| function.call1((path, checker)))
+            .and_then(|result| result.extract::<Vec<String>>())
+            .map_err(|error| error.to_string())
+    })
+}
+
+pub fn validate_or_report(path: &str) -> bool {
+    match validate_solution(path) {
+        Ok(violations) if violations.is_empty() => true,
+        Ok(violations) => {
+            println!("ERROR: HW6 library policy violation");
+            for violation in violations {
+                println!("{violation}");
+            }
+            println!("\nSCORE: 0\n");
+            false
+        }
+        Err(error) => panic!("failed to validate solution libraries: {error}"),
+    }
+}

homework/06-membership/tests/src/main.rs

@@ -0,0 +1,462 @@
+mod common;
+mod library_policy;
+mod python_memory;
+mod resources;
+mod runner;
+mod shared_state;
+mod tests;
+mod tests_mc;
+
+use indexmap::IndexMap;
+use std::collections::HashSet;
+
+use clap::Parser;
+use runner::Outcome;
+
+/// Membership Homework Tests
+#[derive(Parser, Debug)]
+#[clap(about, long_about = None)]
+struct Args {
+    /// Path to Python file with solution
+    #[clap(long = "impl", short = 'i', default_value = "solution/member.py")]
+    solution_path: String,
+
+    /// Test to run (optional)
+    #[clap(long = "test", short)]
+    test: Option<String>,
+
+    /// Print execution trace
+    #[clap(long, short)]
+    debug: bool,
+
+    /// Print test parameters and chaos monkey failures without the execution trace
+    #[clap(long, short = 'v')]
+    verbose: bool,
+
+    /// Random seed used in tests
+    #[clap(long, short, default_value = "123")]
+    seed: u64,
+
+    /// Number of processes
+    #[clap(long, short, default_value = "10")]
+    process_count: u32,
+
+    /// Number of chaos monkey runs
+    #[clap(long, short, default_value = "100")]
+    monkeys: u32,
+
+    /// Do not run model checking tests
+    #[clap(long)]
+    disable_mc_tests: bool,
+
+    /// Wall-clock limit in seconds per ordinary test and per chaos run
+    #[clap(long, default_value = "15", value_parser = clap::value_parser!(u64).range(1..=300))]
+    test_timeout: u64,
+
+    /// Shared wall-clock budget in seconds for all chaos runs
+    #[clap(long, default_value = "300", value_parser = clap::value_parser!(u64).range(1..=540))]
+    chaos_timeout: u64,
+
+    /// Wall-clock limit in seconds per scalability test
+    #[clap(long, default_value = "60", value_parser = clap::value_parser!(u64).range(1..=300))]
+    scalability_timeout: u64,
+
+    /// Wall-clock limit in seconds for both model-checking phases
+    #[clap(long, default_value = "300", value_parser = clap::value_parser!(u64).range(1..=300))]
+    mc_timeout: u64,
+
+    /// Total wall-clock budget in seconds (the Docker watchdog is 600s)
+    #[clap(long, default_value = "540", value_parser = clap::value_parser!(u64).range(1..=540))]
+    suite_timeout: u64,
+
+    /// Fixed address-space limit in MiB for ordinary, chaos and scalability workers
+    #[clap(long, default_value = "4096", value_parser = clap::value_parser!(u64).range(128..=65536))]
+    memory_limit: u64,
+
+    /// Fixed address-space limit in MiB for the model-checking worker
+    #[clap(long, default_value = "14336", value_parser = clap::value_parser!(u64).range(128..=65536))]
+    mc_memory_limit: u64,
+
+    #[clap(long, hide = true, requires = "test-result")]
+    test_worker: Option<String>,
+
+    #[clap(long, hide = true)]
+    test_result: Option<String>,
+
+    #[clap(long, hide = true)]
+    test_progress: Option<String>,
+
+    #[clap(long, hide = true, default_value = "1")]
+    chaos_start: u32,
+}
+
+fn main() {
+    let args = Args::parse();
+    if args.test_worker.is_some() {
+        runner::worker(&args);
+        return;
+    }
+    if !library_policy::validate_or_report(&args.solution_path) {
+        return;
+    }
+    if !shared_state::validate_or_report(&args.solution_path, &["GroupMember"]) {
+        return;
+    }
+    let results = match runner::run(&args) {
+        Ok(results) => results,
+        Err(error) => {
+            eprintln!("ERROR: test environment: {error}");
+            std::process::exit(2);
+        }
+    };
+    if args.test.is_none() {
+        let score = score(results, args.monkeys, args.disable_mc_tests);
+        println!("\nSCORE: {score}\n");
+    }
+}
+
+macro_rules! string_set {
+    ($($x:expr),+ $(,)?) => (
+        HashSet::from([$($x.to_string()),+])
+    );
+}
+
+fn score(results: IndexMap<String, Outcome>, monkeys: u32, disable_mc_tests: bool) -> f32 {
+    let mut basic_group = string_set! {
+        "SIMPLE",
+        "GET MEMBERS SEMANTICS",
+        "RANDOM SEED",
+        "PROCESS JOIN",
+        "PROCESS LEAVE",
+        "MODEL CHECKING",
+    };
+    let process_crash_group = string_set! {
+        "PROCESS CRASH",
+        "SEED PROCESS CRASH",
+        "PROCESS CRASH RECOVER",
+    };
+    let network_groups = [
+        (
+            "Network partitions tests",
+            string_set! {
+                "PROCESS OFFLINE",
+                "SEED PROCESS OFFLINE",
+                "PROCESS OFFLINE RECOVER",
+                "NETWORK PARTITION",
+                "NETWORK PARTITION RECOVER",
+                "NETWORK PARTITION UNEQUAL",
+                "NETWORK PARTITION UNEQUAL RECOVER",
+            },
+        ),
+        (
+            "Network connectivity tests",
+            string_set! {
+                "PROCESS CANNOT RECEIVE",
+                "PROCESS JOIN CANNOT RECEIVE",
+                "PROCESS CANNOT SEND",
+                "TWO PROCESSES CANNOT COMMUNICATE",
+                "TWO PROCESSES CANNOT COMMUNICATE IN TRIANGLE",
+            },
+        ),
+        (
+            "Network delays and losses tests",
+            string_set! {
+                "SLOW NETWORK",
+                "FLAKY NETWORK",
+                "FLAKY NETWORK ON START",
+                "FLAKY NETWORK AND CRASH",
+            },
+        ),
+        (
+            "Chaos monkey tests",
+            (1..=monkeys)
+                .map(|run| format!("CHAOS MONKEY (run {run})"))
+                .collect(),
+        ),
+    ];
+    let scalability_group = string_set! {"SCALABILITY NORMAL", "SCALABILITY CRASH"};
+    // Explicit local opt-out changes the selected basic tests. An MC skipped
+    // because the suite ran out of time remains required during grading.
+    if disable_mc_tests {
+        basic_group.remove("MODEL CHECKING");
+    }
+    let mut all_groups = &(&basic_group | &process_crash_group) | &scalability_group;
+    let mut test_count = basic_group.len() + process_crash_group.len() + scalability_group.len();
+    for (_, group) in &network_groups {
+        all_groups.extend(group.iter().cloned());
+        test_count += group.len();
+    }
+    assert_eq!(all_groups.len(), test_count, "overlapping scoring groups");
+    assert_eq!(all_groups, results.keys().cloned().collect());
+
+    let scalability_skipped = scalability_group
+        .iter()
+        .all(|name| matches!(results.get(name), Some(Outcome::SkippedPrerequisites(_))));
+    let mut passed = HashSet::new();
+    let mut model_checking_ok = false;
+    for (test, result) in results {
+        if test == "MODEL CHECKING" && matches!(result, Outcome::Passed) {
+            model_checking_ok = true;
+        }
+        if result.satisfies_functional_requirement(&test) {
+            passed.insert(test);
+        }
+    }
+
+    let basic_group_ok = basic_group.is_subset(&passed);
+    let mut score = 0.;
+    print!("Basic tests: ");
+    if basic_group_ok {
+        score += 3.;
+        println!("passed");
+    } else {
+        let mut failed: Vec<&String> = basic_group.difference(&passed).collect();
+        failed.sort();
+        println!("not passed {failed:?}");
+    }
+    print!("Process crashes tests: ");
+    if process_crash_group.is_subset(&passed) {
+        score += 3.;
+        println!("passed");
+    } else {
+        let mut failed: Vec<&String> = process_crash_group.difference(&passed).collect();
+        failed.sort();
+        println!("not passed {failed:?}");
+    }
+    for (name, group) in &network_groups {
+        print!("{name}: ");
+        if group.is_subset(&passed) {
+            score += 1.;
+            println!("passed");
+        } else {
+            let mut failed: Vec<&String> = group.difference(&passed).collect();
+            failed.sort();
+            println!("not passed {failed:?}");
+        }
+    }
+    // The basic-group eligibility already includes the resource-limit and
+    // explicit local opt-out exceptions for MC.
+    let functional_tests_ok = basic_group_ok
+        && process_crash_group.is_subset(&passed)
+        && network_groups
+            .iter()
+            .all(|(_, group)| group.is_subset(&passed));
+    print!("Scalability tests: ");
+    if scalability_skipped {
+        println!("skipped (functional tests not passed)");
+    } else if scalability_group.is_subset(&passed) {
+        if functional_tests_ok {
+            score += 4.;
+            println!("passed");
+        } else {
+            println!("passed (no points: functional tests not passed)");
+        }
+    } else {
+        let mut failed: Vec<&String> = scalability_group.difference(&passed).collect();
+        failed.sort();
+        println!("not passed {failed:?}");
+    }
+    print!("Model checking test: ");
+    if model_checking_ok && basic_group_ok {
+        score += 1.;
+        println!("passed");
+    } else if model_checking_ok {
+        println!("passed (no bonus: basic tests not passed)");
+    } else {
+        println!("not passed");
+    }
+    score
+}
+
+#[cfg(test)]
+mod score_tests {
+    use super::*;
+
+    #[test]
+    fn functional_groups_keep_their_points_after_resource_failures() {
+        assert_eq!(score(runner::passed_results(2), 2, false), 15.);
+        for (name, expected) in [
+            ("SCALABILITY NORMAL", 11.),
+            ("PROCESS CRASH", 8.),
+            ("CHAOS MONKEY (run 2)", 10.),
+        ] {
+            for failure in [
+                Outcome::TimeLimit("test budget".into()),
+                Outcome::MemoryLimit("256 MiB".into()),
+                Outcome::WorkerError("SIGKILL".into()),
+                Outcome::Skipped("not started".into()),
+            ] {
+                let mut results = runner::passed_results(2);
+                results.insert(name.into(), failure);
+                assert_eq!(score(results, 2, false), expected, "{name}");
+            }
+        }
+    }
+
+    #[test]
+    fn each_network_test_only_blocks_its_subgroup_and_scalability() {
+        for name in [
+            "PROCESS OFFLINE",
+            "SEED PROCESS OFFLINE",
+            "PROCESS OFFLINE RECOVER",
+            "NETWORK PARTITION",
+            "NETWORK PARTITION RECOVER",
+            "NETWORK PARTITION UNEQUAL",
+            "NETWORK PARTITION UNEQUAL RECOVER",
+            "PROCESS CANNOT RECEIVE",
+            "PROCESS JOIN CANNOT RECEIVE",
+            "PROCESS CANNOT SEND",
+            "TWO PROCESSES CANNOT COMMUNICATE",
+            "TWO PROCESSES CANNOT COMMUNICATE IN TRIANGLE",
+            "SLOW NETWORK",
+            "FLAKY NETWORK",
+            "FLAKY NETWORK ON START",
+            "FLAKY NETWORK AND CRASH",
+            "CHAOS MONKEY (run 1)",
+            "CHAOS MONKEY (run 2)",
+        ] {
+            for failure in [
+                Outcome::Failed("wrong members".into()),
+                Outcome::TimeLimit("test budget".into()),
+                Outcome::MemoryLimit("4096 MiB".into()),
+                Outcome::WorkerError("SIGKILL".into()),
+                Outcome::Skipped("not started".into()),
+            ] {
+                let mut results = runner::passed_results(2);
+                results.insert(name.into(), failure.clone());
+                assert_eq!(score(results, 2, false), 10., "{name}: {failure:?}");
+            }
+        }
+    }
+
+    #[test]
+    fn multiple_failures_in_one_network_subgroup_have_the_same_penalty() {
+        for names in [
+            vec!["PROCESS OFFLINE", "NETWORK PARTITION RECOVER"],
+            vec!["PROCESS CANNOT RECEIVE", "TWO PROCESSES CANNOT COMMUNICATE"],
+            vec!["SLOW NETWORK", "FLAKY NETWORK AND CRASH"],
+            vec!["CHAOS MONKEY (run 1)", "CHAOS MONKEY (run 2)"],
+        ] {
+            let mut results = runner::passed_results(2);
+            for name in &names {
+                results.insert((*name).into(), Outcome::Failed("wrong members".into()));
+            }
+            assert_eq!(score(results, 2, false), 10., "{names:?}");
+        }
+    }
+
+    #[test]
+    fn unsuccessful_ordinary_basic_tests_block_mc_bonus_and_scalability() {
+        for name in [
+            "SIMPLE",
+            "GET MEMBERS SEMANTICS",
+            "RANDOM SEED",
+            "PROCESS JOIN",
+            "PROCESS LEAVE",
+        ] {
+            for failure in [
+                Outcome::Failed("wrong members".into()),
+                Outcome::TimeLimit("test budget".into()),
+                Outcome::MemoryLimit("4096 MiB".into()),
+                Outcome::WorkerError("SIGKILL".into()),
+                Outcome::Skipped("not started".into()),
+            ] {
+                let mut results = runner::passed_results(2);
+                results.insert(name.into(), failure.clone());
+                assert_eq!(score(results, 2, false), 7., "{name}: {failure:?}");
+            }
+        }
+    }
+
+    #[test]
+    fn mc_outcomes_match_the_scoring_table_for_all_other_group_combinations() {
+        // outcome (None = explicit opt-out), basic eligibility, bonus if all basic tests pass.
+        let cases = [
+            (Some(Outcome::Passed), true, 1.),
+            (None, true, 0.),
+            (Some(Outcome::Failed("counterexample".into())), false, 0.),
+            (
+                Some(Outcome::WorkerError("Python exception".into())),
+                false,
+                0.,
+            ),
+            (Some(Outcome::WorkerError("SIGKILL".into())), false, 0.),
+            (Some(Outcome::TimeLimit("MC budget".into())), true, 0.),
+            (
+                Some(Outcome::TimeLimit("suite budget after MC started".into())),
+                true,
+                0.,
+            ),
+            (Some(Outcome::MemoryLimit("14336 MiB".into())), true, 0.),
+            (
+                Some(Outcome::Skipped("suite budget before MC started".into())),
+                false,
+                0.,
+            ),
+        ];
+        let basic = [
+            "SIMPLE",
+            "GET MEMBERS SEMANTICS",
+            "RANDOM SEED",
+            "PROCESS JOIN",
+            "PROCESS LEAVE",
+        ];
+        let other = [
+            ("PROCESS CRASH", 3.),
+            ("PROCESS OFFLINE", 1.),
+            ("PROCESS CANNOT RECEIVE", 1.),
+            ("SLOW NETWORK", 1.),
+            ("CHAOS MONKEY (run 2)", 1.),
+            ("SCALABILITY NORMAL", 4.),
+        ];
+        for (mc, basic_eligible, bonus) in cases {
+            // All 32 combinations of the five ordinary basic tests, and all
+            // 64 combinations of crash, four network subgroups, and scalability.
+            for basic_failures in 0..32 {
+                for other_failures in 0..64 {
+                    let mut results = runner::passed_results(2);
+                    if let Some(outcome) = &mc {
+                        results.insert("MODEL CHECKING".into(), outcome.clone());
+                    } else {
+                        results.shift_remove("MODEL CHECKING");
+                    }
+                    for (index, name) in basic.iter().enumerate() {
+                        if basic_failures & (1 << index) != 0 {
+                            results.insert((*name).into(), Outcome::Failed("wrong members".into()));
+                        }
+                    }
+                    let mut expected = 0.;
+                    if basic_eligible && basic_failures == 0 {
+                        expected += 3. + bonus;
+                    }
+                    for (index, (name, points)) in other.iter().enumerate() {
+                        if other_failures & (1 << index) != 0 {
+                            results.insert((*name).into(), Outcome::Failed("wrong members".into()));
+                        } else if *name != "SCALABILITY NORMAL" {
+                            expected += points;
+                        }
+                    }
+                    // Scalability needs both its own tests and every functional group.
+                    if basic_eligible && basic_failures == 0 && other_failures == 0 {
+                        expected += 4.;
+                    }
+                    let blockers = runner::scalability_blockers(&results, 2, mc.is_none());
+                    let eligible =
+                        basic_eligible && basic_failures == 0 && other_failures & 31 == 0;
+                    assert_eq!(blockers.is_empty(), eligible,
+                        "MC={mc:?}, basic failures={basic_failures:05b}, other failures={other_failures:06b}");
+                    if !eligible {
+                        for name in ["SCALABILITY NORMAL", "SCALABILITY CRASH"] {
+                            results.insert(
+                                name.into(),
+                                Outcome::SkippedPrerequisites(blockers.clone()),
+                            );
+                        }
+                    }
+                    assert_eq!(score(results, 2, mc.is_none()), expected,
+                        "MC={mc:?}, basic failures={basic_failures:05b}, other failures={other_failures:06b}");
+                }
+            }
+        }
+    }
+}

homework/06-membership/tests/src/python_memory.rs

@@ -0,0 +1,83 @@
+//! Observe actual CPython allocation failures before AnySystem replaces a
+//! Python exception with a generic panic. Student stdout is not a result channel.
+use std::ffi::c_void;
+use std::mem::MaybeUninit;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::Once;
+
+use pyo3::ffi::{self, PyMemAllocatorDomain, PyMemAllocatorEx};
+use pyo3::Python;
+
+static INSTALLED: Once = Once::new();
+static FAILED: AtomicBool = AtomicBool::new(false);
+
+fn observe(pointer: *mut c_void, nonzero: bool) -> *mut c_void {
+    if nonzero && pointer.is_null() {
+        FAILED.store(true, Ordering::Relaxed);
+    }
+    pointer
+}
+
+extern "C" fn malloc(ctx: *mut c_void, size: usize) -> *mut c_void {
+    // SAFETY: ctx points to an immutable original allocator, leaked for the
+    // interpreter's entire lifetime. Calls retain CPython's allocator contract.
+    let allocator = unsafe { &*ctx.cast::<PyMemAllocatorEx>() };
+    observe((allocator.malloc.unwrap())(allocator.ctx, size), size != 0)
+}
+extern "C" fn calloc(ctx: *mut c_void, count: usize, size: usize) -> *mut c_void {
+    // SAFETY: same lifetime and callback contract as malloc above.
+    let allocator = unsafe { &*ctx.cast::<PyMemAllocatorEx>() };
+    observe(
+        (allocator.calloc.unwrap())(allocator.ctx, count, size),
+        count != 0 && size != 0,
+    )
+}
+extern "C" fn realloc(ctx: *mut c_void, pointer: *mut c_void, size: usize) -> *mut c_void {
+    // SAFETY: same lifetime and callback contract as malloc above.
+    let allocator = unsafe { &*ctx.cast::<PyMemAllocatorEx>() };
+    observe(
+        (allocator.realloc.unwrap())(allocator.ctx, pointer, size),
+        size != 0,
+    )
+}
+extern "C" fn free(ctx: *mut c_void, pointer: *mut c_void) {
+    // SAFETY: same lifetime and callback contract as malloc above.
+    let allocator = unsafe { &*ctx.cast::<PyMemAllocatorEx>() };
+    (allocator.free.unwrap())(allocator.ctx, pointer);
+}
+
+pub fn install() {
+    Python::attach(|_| {
+        INSTALLED.call_once(|| {
+            for domain in [
+                PyMemAllocatorDomain::PYMEM_DOMAIN_RAW,
+                PyMemAllocatorDomain::PYMEM_DOMAIN_MEM,
+                PyMemAllocatorDomain::PYMEM_DOMAIN_OBJ,
+            ] {
+                // SAFETY: installation holds the GIL before any student code runs.
+                // CPython requires post-initialization hooks to wrap the previous
+                // allocator. Its four callbacks are supplied by the interpreter.
+                unsafe {
+                    let mut original = MaybeUninit::uninit();
+                    ffi::PyMem_GetAllocator(domain, original.as_mut_ptr());
+                    let original = Box::into_raw(Box::new(original.assume_init()));
+                    let mut wrapper = PyMemAllocatorEx {
+                        ctx: original.cast(),
+                        malloc: Some(malloc),
+                        calloc: Some(calloc),
+                        realloc: Some(realloc),
+                        free: Some(free),
+                    };
+                    ffi::PyMem_SetAllocator(domain, &mut wrapper);
+                }
+            }
+        })
+    });
+}
+
+pub fn reset() {
+    FAILED.store(false, Ordering::Relaxed);
+}
+pub fn failed() -> bool {
+    FAILED.load(Ordering::Relaxed)
+}

homework/06-membership/tests/src/resources.rs

@@ -0,0 +1,165 @@
+//! Hard address-space limits are applied before exec, including Python imports.
+//! Unlike an RSS polling loop, RLIMIT_AS also stops a single oversized allocation.
+use std::fs::File;
+use std::process::{Child, Command, ExitStatus};
+
+#[cfg(target_os = "linux")]
+const MIB: u64 = 1024 * 1024;
+
+#[cfg(target_os = "linux")]
+mod linux {
+    use std::fs;
+    use std::os::fd::OwnedFd;
+    use std::os::unix::net::UnixStream;
+    use std::os::unix::process::CommandExt;
+    use std::process::Stdio;
+
+    use super::*;
+
+    fn number(path: &str) -> Option<u64> {
+        fs::read_to_string(path).ok()?.trim().parse().ok()
+    }
+
+    fn memory_capacity() -> Option<u64> {
+        let meminfo = fs::read_to_string("/proc/meminfo").ok()?;
+        let host = meminfo.lines().find_map(|line| {
+            line.strip_prefix("MemTotal:")?
+                .split_whitespace()
+                .next()?
+                .parse::<u64>()
+                .ok()
+        })? * 1024;
+        // Both namespaced cgroup v2 and the conventional v1 Docker mount.
+        let container = [
+            "/sys/fs/cgroup/memory.max",
+            "/sys/fs/cgroup/memory/memory.limit_in_bytes",
+        ]
+        .into_iter()
+        .filter_map(number)
+        .min();
+        Some(container.map_or(host, |container| host.min(container)))
+    }
+
+    pub fn limit(requested_mib: u64) -> Result<u64, String> {
+        // Check static capacity, never current free memory or cgroup usage.
+        // Insufficient capacity is an environment error, not a lower grading limit.
+        let capacity = memory_capacity().ok_or("cannot determine memory capacity")? / MIB;
+        if requested_mib + 256 > capacity {
+            return Err(format!("fixed worker limit is {requested_mib} MiB, but environment capacity is {capacity} MiB; provide at least {} MiB including the supervisor reserve, or explicitly select a smaller local limit", requested_mib + 256));
+        }
+        let mut inherited = libc::rlimit {
+            rlim_cur: 0,
+            rlim_max: 0,
+        };
+        // SAFETY: getrlimit writes into an initialized rlimit structure.
+        if unsafe { libc::getrlimit(libc::RLIMIT_AS, &mut inherited) } != 0 {
+            return Err(std::io::Error::last_os_error().to_string());
+        }
+        if inherited.rlim_max / MIB < requested_mib {
+            return Err(format!("fixed worker limit is {requested_mib} MiB, but inherited address-space hard limit is {} MiB", inherited.rlim_max / MIB));
+        }
+        Ok(requested_mib)
+    }
+
+    pub fn capture(command: &mut Command) -> std::io::Result<File> {
+        let (reader, writer) = UnixStream::pair()?;
+        command.stdout(Stdio::from(OwnedFd::from(writer.try_clone()?)));
+        command.stderr(Stdio::from(OwnedFd::from(writer)));
+        Ok(File::from(OwnedFd::from(reader)))
+    }
+
+    pub fn configure(command: &mut Command, mib: u64) {
+        let supervisor = std::process::id() as libc::pid_t;
+        // SAFETY: only async-signal-safe libc calls run between fork and exec.
+        unsafe {
+            command.pre_exec(move || {
+                // Interrupting the supervisor must not leave a hanging worker.
+                if libc::prctl(libc::PR_SET_PDEATHSIG, libc::SIGKILL) != 0 {
+                    return Err(std::io::Error::last_os_error());
+                }
+                if libc::getppid() != supervisor {
+                    return Err(std::io::Error::from_raw_os_error(libc::ECHILD));
+                }
+                if libc::setsid() < 0 {
+                    return Err(std::io::Error::last_os_error());
+                }
+                let memory = libc::rlimit {
+                    rlim_cur: mib * MIB,
+                    rlim_max: mib * MIB,
+                };
+                let core = libc::rlimit {
+                    rlim_cur: 0,
+                    rlim_max: 0,
+                };
+                if libc::setrlimit(libc::RLIMIT_AS, &memory) != 0
+                    || libc::setrlimit(libc::RLIMIT_CORE, &core) != 0
+                {
+                    return Err(std::io::Error::last_os_error());
+                }
+                Ok(())
+            });
+        }
+    }
+
+    pub fn terminate(child: &mut Child) {
+        // The child starts a new session: clean up its descendants as well.
+        // SAFETY: the negative pid targets this worker's process group.
+        unsafe {
+            libc::kill(-(child.id() as i32), libc::SIGKILL);
+        }
+        let _ = child.kill();
+        let _ = child.wait();
+    }
+
+    pub fn memory_usage(child: &Child) -> Option<u64> {
+        fs::read_to_string(format!("/proc/{}/status", child.id()))
+            .ok()?
+            .lines()
+            .find_map(|line| {
+                line.strip_prefix("VmSize:")?
+                    .split_whitespace()
+                    .next()?
+                    .parse()
+                    .ok()
+            })
+            .map(|kib: u64| kib / 1024)
+    }
+}
+
+#[cfg(target_os = "linux")]
+pub use linux::{capture, configure, limit, memory_usage, terminate};
+
+#[cfg(not(target_os = "linux"))]
+pub fn limit(_: u64) -> Result<u64, String> {
+    Err("resource-limited workers require Linux; use the homework Docker image".into())
+}
+#[cfg(not(target_os = "linux"))]
+pub fn capture(_: &mut Command) -> std::io::Result<File> {
+    Err(std::io::Error::other("use the homework Docker image"))
+}
+#[cfg(not(target_os = "linux"))]
+pub fn configure(_: &mut Command, _: u64) {}
+#[cfg(not(target_os = "linux"))]
+pub fn terminate(child: &mut Child) {
+    let _ = child.kill();
+    let _ = child.wait();
+}
+#[cfg(not(target_os = "linux"))]
+pub fn memory_usage(_: &Child) -> Option<u64> {
+    None
+}
+
+pub fn allocation_crash(status: Option<ExitStatus>, output: &str) -> bool {
+    #[cfg(target_os = "linux")]
+    {
+        use std::os::unix::process::ExitStatusExt;
+        status.is_some_and(|status| status.signal() == Some(libc::SIGABRT))
+            && output.contains("memory allocation of ")
+            && output.contains(" bytes failed")
+    }
+    #[cfg(not(target_os = "linux"))]
+    {
+        let _ = (status, output);
+        false
+    }
+}

homework/06-membership/tests/src/runner.rs

@@ -0,0 +1,878 @@
+//! Based on HW4's killable-process runner. Only workers execute solution code.
+//! Chaos runs share a worker, with an append-only result journal after every run.
+use std::cell::OnceCell;
+use std::fs::{self, File, OpenOptions};
+use std::io::{BufRead, BufReader, Read, Write};
+use std::path::PathBuf;
+use std::process::Command;
+use std::sync::OnceLock;
+use std::thread;
+use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
+
+use anysystem::python::PyProcessFactory;
+use anysystem::test::TestResult;
+use env_logger::Builder;
+use indexmap::IndexMap;
+use log::{info, LevelFilter};
+use pyo3::prelude::*;
+use rand::{RngCore, SeedableRng};
+use rand_pcg::Pcg64;
+use serde::{Deserialize, Serialize};
+
+use crate::common::TestConfig;
+use crate::tests::*;
+use crate::tests_mc::test_mc_group;
+use crate::Args;
+use crate::{python_memory, resources};
+
+type TestFn = fn(&TestConfig) -> TestResult;
+const ORDINARY: &[(&str, TestFn)] = &[
+    ("SIMPLE", test_simple),
+    ("GET MEMBERS SEMANTICS", test_get_members_semantics),
+    ("RANDOM SEED", test_random_seed),
+    ("PROCESS JOIN", test_process_join),
+    ("PROCESS LEAVE", test_process_leave),
+    ("PROCESS CRASH", test_process_crash),
+    ("SEED PROCESS CRASH", test_seed_process_crash),
+    ("PROCESS CRASH RECOVER", test_process_crash_recover),
+    ("PROCESS OFFLINE", test_process_offline),
+    ("SEED PROCESS OFFLINE", test_seed_process_offline),
+    ("PROCESS OFFLINE RECOVER", test_process_offline_recover),
+    ("PROCESS CANNOT RECEIVE", test_process_cannot_receive),
+    (
+        "PROCESS JOIN CANNOT RECEIVE",
+        test_process_join_cannot_receive,
+    ),
+    ("PROCESS CANNOT SEND", test_process_cannot_send),
+    ("NETWORK PARTITION", test_network_partition),
+    ("NETWORK PARTITION RECOVER", test_network_partition_recover),
+    ("NETWORK PARTITION UNEQUAL", test_network_partition_unequal),
+    (
+        "NETWORK PARTITION UNEQUAL RECOVER",
+        test_network_partition_unequal_recover,
+    ),
+    (
+        "TWO PROCESSES CANNOT COMMUNICATE",
+        test_two_processes_cannot_communicate,
+    ),
+    (
+        "TWO PROCESSES CANNOT COMMUNICATE IN TRIANGLE",
+        test_two_processes_cannot_communicate_in_triangle,
+    ),
+    ("SLOW NETWORK", test_slow_network),
+    ("FLAKY NETWORK", test_flaky_network),
+    ("FLAKY NETWORK ON START", test_flaky_network_on_start),
+    ("FLAKY NETWORK AND CRASH", test_flaky_network_and_crash),
+];
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum Outcome {
+    Passed,
+    Failed(String),
+    TimeLimit(String),
+    MemoryLimit(String),
+    WorkerError(String),
+    Skipped(String),
+    SkippedPrerequisites(Vec<String>),
+}
+
+impl Outcome {
+    pub fn resource_limited(&self) -> bool {
+        matches!(self, Self::TimeLimit(_) | Self::MemoryLimit(_))
+    }
+
+    // A resource failure before the Started record is not an attempted test.
+    // In particular it must not grant the MC exemption for the basic group.
+    fn require_started(self, started: bool) -> Self {
+        if !started && self.resource_limited() {
+            Self::Skipped(format!("test did not start; {}", self.description()))
+        } else {
+            self
+        }
+    }
+
+    pub fn satisfies_functional_requirement(&self, name: &str) -> bool {
+        matches!(self, Self::Passed) || (name == "MODEL CHECKING" && self.resource_limited())
+    }
+
+    fn description(&self) -> String {
+        match self {
+            Self::Passed => "passed".into(),
+            Self::Failed(message) => message.clone(),
+            Self::TimeLimit(message) => format!("time limit exceeded: {message}"),
+            Self::MemoryLimit(message) => format!("memory limit exceeded: {message}"),
+            Self::WorkerError(message) => format!("test worker error: {message}"),
+            Self::Skipped(message) => format!("not checked: {message}"),
+            Self::SkippedPrerequisites(tests) => {
+                let mut names = tests.iter().take(8).cloned().collect::<Vec<_>>().join(", ");
+                if tests.len() > 8 {
+                    names.push_str(&format!(" (and {} more)", tests.len() - 8));
+                }
+                format!("required functional tests not passed: {names}")
+            }
+        }
+    }
+}
+
+#[derive(Serialize, Deserialize)]
+enum Event {
+    Started(String),
+    Finished(String, Outcome),
+}
+
+struct Test {
+    name: String,
+    run: TestFn,
+    seed: u64,
+    chaos_run: Option<u32>,
+}
+
+fn tests(args: &Args) -> Vec<Test> {
+    let mut tests: Vec<_> = ORDINARY
+        .iter()
+        .map(|&(name, run)| Test {
+            name: name.into(),
+            run,
+            seed: args.seed,
+            chaos_run: None,
+        })
+        .collect();
+    let mut rand = Pcg64::seed_from_u64(args.seed);
+    for run in 1..=args.monkeys {
+        tests.push(Test {
+            name: format!("CHAOS MONKEY (run {run})"),
+            run: test_chaos_monkey,
+            seed: rand.next_u64(),
+            chaos_run: Some(run),
+        });
+    }
+    for (name, run) in [
+        ("MODEL CHECKING", test_mc_group as TestFn),
+        ("SCALABILITY NORMAL", test_scalability_normal as TestFn),
+        ("SCALABILITY CRASH", test_scalability_crash as TestFn),
+    ] {
+        if name != "MODEL CHECKING" || !args.disable_mc_tests {
+            tests.push(Test {
+                name: name.into(),
+                run,
+                seed: args.seed,
+                chaos_run: None,
+            });
+        }
+    }
+    tests
+}
+
+/// Missing, failed and unstarted functional tests block scalability. Only an
+/// explicitly disabled MC or a resource limit after it started is exempt.
+pub fn scalability_blockers(
+    results: &IndexMap<String, Outcome>,
+    monkeys: u32,
+    disable_mc_tests: bool,
+) -> Vec<String> {
+    ORDINARY
+        .iter()
+        .map(|(name, _)| (*name).to_string())
+        .chain((1..=monkeys).map(|run| format!("CHAOS MONKEY (run {run})")))
+        .chain((!disable_mc_tests).then(|| "MODEL CHECKING".to_string()))
+        .filter(|name| {
+            !results
+                .get(name)
+                .is_some_and(|outcome| outcome.satisfies_functional_requirement(name))
+        })
+        .collect()
+}
+
+static PROGRESS: OnceLock<PathBuf> = OnceLock::new();
+
+/// Bounded, replaceable diagnostics; never mixed with the structured result stream.
+pub fn progress(message: impl AsRef<str>) {
+    if let Some(path) = PROGRESS.get() {
+        let temporary = path.with_extension("next");
+        if fs::write(&temporary, message.as_ref()).is_ok() {
+            let _ = fs::rename(&temporary, path);
+        }
+    }
+}
+
+fn event(file: &mut File, event: Event) {
+    serde_json::to_writer(&mut *file, &event).unwrap();
+    writeln!(file).unwrap();
+    file.flush().unwrap();
+}
+
+fn report(name: &str, outcome: &Outcome) {
+    match outcome {
+        Outcome::Passed => println!("\nPASSED\n"),
+        Outcome::Skipped(_) | Outcome::SkippedPrerequisites(_) => {
+            println!("\nSKIPPED: {}\n", outcome.description())
+        }
+        _ => println!("\nFAILED: {}\n", outcome.description()),
+    }
+    if name == "MODEL CHECKING" && outcome.resource_limited() {
+        println!("INCOMPLETE: model checking reached its resource limit; no MC bonus, basic tests remain eligible\n");
+    }
+    std::io::stdout().flush().unwrap();
+}
+
+pub fn worker(args: &Args) {
+    if let Some(path) = &args.test_progress {
+        let _ = PROGRESS.set(PathBuf::from(path));
+    }
+    if args.debug || args.verbose {
+        let mut logger = Builder::new();
+        logger.filter(Some("distsys_membership"), LevelFilter::Info);
+        if args.debug {
+            logger.filter(Some("anysystem"), LevelFilter::Debug);
+        }
+        logger
+            .format(|buf, record| writeln!(buf, "{}", record.args()))
+            .init();
+    }
+    info!(
+        "Test parameters: seed={}, processes={}",
+        args.seed, args.process_count
+    );
+    let mut journal = OpenOptions::new()
+        .append(true)
+        .open(args.test_result.as_ref().unwrap())
+        .unwrap();
+    let factory = OnceCell::new();
+    for test in tests(args).into_iter().filter(|test| {
+        if args.test_worker.as_deref() == Some("CHAOS MONKEY") {
+            test.chaos_run.is_some_and(|run| run >= args.chaos_start)
+        } else {
+            args.test_worker.as_deref() == Some(test.name.as_str())
+        }
+    }) {
+        progress("loading Python implementation");
+        event(&mut journal, Event::Started(test.name.clone()));
+        println!("\n--- {} ---\n", test.name);
+        std::io::stdout().flush().unwrap();
+        python_memory::reset();
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+            python_memory::install();
+            let factory = factory.get_or_init(|| {
+                // Match the preflight's local-import root when --impl is outside cwd.
+                let solution = fs::canonicalize(&args.solution_path).unwrap();
+                let directory = solution.parent().unwrap().to_string_lossy();
+                Python::attach(|py| {
+                    py.import("sys")
+                        .and_then(|sys| sys.getattr("path"))
+                        .and_then(|path| path.call_method1("insert", (0, directory.as_ref())))
+                        .unwrap();
+                });
+                PyProcessFactory::new(&args.solution_path, "GroupMember")
+            });
+            progress("initializing simulation");
+            let config = TestConfig {
+                process_factory: factory,
+                process_count: if test.name == "MODEL CHECKING" {
+                    3
+                } else {
+                    args.process_count
+                },
+                seed: test.seed,
+            };
+            (test.run)(&config)
+        }));
+        let mut outcome = match result {
+            Ok(Ok(_)) => Outcome::Passed,
+            Ok(Err(message))
+                if message == "time limit of 300s exceeded" && test.name == "MODEL CHECKING" =>
+            {
+                Outcome::TimeLimit(message)
+            }
+            Ok(Err(message)) => Outcome::Failed(message),
+            Err(payload) => {
+                let message = payload
+                    .downcast_ref::<String>()
+                    .map(String::as_str)
+                    .or_else(|| payload.downcast_ref::<&str>().copied())
+                    .unwrap_or("test panicked")
+                    .to_owned();
+                if python_memory::failed() {
+                    Outcome::MemoryLimit(message)
+                } else {
+                    Outcome::WorkerError(message)
+                }
+            }
+        };
+        if let Outcome::MemoryLimit(message) = &outcome {
+            let phase = PROGRESS
+                .get()
+                .and_then(|path| fs::read_to_string(path).ok())
+                .unwrap_or_default();
+            outcome = Outcome::MemoryLimit(format!(
+                "worker address-space limit {} MiB; {phase}; {message}",
+                memory_limit(args, &test)
+            ));
+        }
+        // Bound IPC diagnostics independently of the worker's memory budget.
+        match &mut outcome {
+            Outcome::Failed(message)
+            | Outcome::TimeLimit(message)
+            | Outcome::MemoryLimit(message)
+            | Outcome::WorkerError(message)
+            | Outcome::Skipped(message)
+                if message.len() > 16384 =>
+            {
+                let mut end = 16384;
+                while !message.is_char_boundary(end) {
+                    end -= 1;
+                }
+                message.truncate(end);
+                message.push_str(" [diagnostic truncated]");
+            }
+            _ => {}
+        }
+        event(
+            &mut journal,
+            Event::Finished(test.name.clone(), outcome.clone()),
+        );
+        // Report panics after draining their diagnostics from the output pipe.
+        if !matches!(outcome, Outcome::WorkerError(_)) {
+            report(&test.name, &outcome);
+        }
+        // A panic can leave the embedded interpreter or simulator in an unknown state.
+        if matches!(outcome, Outcome::WorkerError(_) | Outcome::MemoryLimit(_)) {
+            break;
+        }
+    }
+}
+
+struct Files(PathBuf);
+impl Files {
+    fn new() -> std::io::Result<Self> {
+        let nonce = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap()
+            .as_nanos();
+        let path =
+            std::env::temp_dir().join(format!("membership-test-{}-{nonce}", std::process::id()));
+        fs::create_dir(&path)?;
+        let files = Self(path);
+        File::create(files.journal())?;
+        Ok(files)
+    }
+    fn journal(&self) -> PathBuf {
+        self.0.join("results.jsonl")
+    }
+    fn progress(&self) -> PathBuf {
+        self.0.join("progress.txt")
+    }
+}
+impl Drop for Files {
+    fn drop(&mut self) {
+        for name in ["results.jsonl", "progress.txt", "progress.next"] {
+            let _ = fs::remove_file(self.0.join(name));
+        }
+        let _ = fs::remove_dir(&self.0);
+    }
+}
+
+// Retain a bounded diagnostic tail for native allocator failures. Python
+// allocation failures are tracked separately; output alone is not an outcome.
+fn output_reader(mut pipe: impl Read + Send + 'static) -> thread::JoinHandle<String> {
+    thread::spawn(move || {
+        let mut tail = Vec::new();
+        let mut chunk = [0; 4096];
+        while let Ok(count) = pipe.read(&mut chunk) {
+            if count == 0 {
+                break;
+            }
+            let _ = std::io::stdout().lock().write_all(&chunk[..count]);
+            tail.extend_from_slice(&chunk[..count]);
+            if tail.len() > 65536 {
+                tail.drain(..tail.len() - 65536);
+            }
+        }
+        String::from_utf8_lossy(&tail).into_owned()
+    })
+}
+
+struct Journal {
+    reader: BufReader<File>,
+    pending: String,
+    next: usize,
+    started: Instant,
+    announced: bool,
+}
+impl Journal {
+    fn read(
+        &mut self,
+        batch: &[Test],
+        results: &mut IndexMap<String, Outcome>,
+    ) -> Result<(), String> {
+        loop {
+            let remaining = 65537_usize.saturating_sub(self.pending.len());
+            let count = (&mut self.reader)
+                .take(remaining as u64)
+                .read_line(&mut self.pending)
+                .map_err(|error| error.to_string())?;
+            if self.pending.len() > 65536 {
+                return Err("worker result exceeds 64 KiB".into());
+            }
+            if !self.pending.ends_with('\n') {
+                return Ok(());
+            }
+            let event: Event = serde_json::from_str(&self.pending)
+                .map_err(|error| format!("invalid worker result: {error}"))?;
+            self.pending.clear();
+            let name = match &event {
+                Event::Started(name) | Event::Finished(name, _) => name,
+            };
+            if batch.get(self.next).map(|test| &test.name) != Some(name) {
+                return Err(format!("unexpected worker result for {name}"));
+            }
+            match event {
+                Event::Started(_) => {
+                    self.started = Instant::now();
+                    self.announced = true;
+                }
+                Event::Finished(name, outcome) => {
+                    results.insert(name, outcome);
+                    self.next += 1;
+                    self.started = Instant::now();
+                    self.announced = false;
+                }
+            }
+            if count == 0 {
+                return Ok(());
+            }
+        }
+    }
+}
+
+fn memory_limit(args: &Args, test: &Test) -> u64 {
+    if test.name == "MODEL CHECKING" {
+        args.mc_memory_limit
+    } else {
+        args.memory_limit
+    }
+}
+
+fn seconds(args: &Args, test: &Test) -> u64 {
+    if test.name == "MODEL CHECKING" {
+        args.mc_timeout
+    } else if test.name.starts_with("SCALABILITY") {
+        args.scalability_timeout
+    } else {
+        args.test_timeout
+    }
+}
+
+fn run_worker(
+    args: &Args,
+    batch: &[Test],
+    results: &mut IndexMap<String, Outcome>,
+    suite: Instant,
+    phase: Instant,
+) -> usize {
+    let setup = || -> Result<_, String> {
+        let files = Files::new().map_err(|error| error.to_string())?;
+        let limit = resources::limit(memory_limit(args, &batch[0]))?;
+        let mut command = Command::new(std::env::current_exe().map_err(|error| error.to_string())?);
+        command
+            .arg("--test-worker")
+            .arg(if batch.len() > 1 {
+                "CHAOS MONKEY"
+            } else {
+                &batch[0].name
+            })
+            .args([
+                "-i",
+                &args.solution_path,
+                "-s",
+                &args.seed.to_string(),
+                "-p",
+                &args.process_count.to_string(),
+                "-m",
+                &args.monkeys.to_string(),
+                "--chaos-start",
+                &batch[0].chaos_run.unwrap_or(1).to_string(),
+                "--memory-limit",
+                &args.memory_limit.to_string(),
+                "--mc-memory-limit",
+                &args.mc_memory_limit.to_string(),
+                "--test-result",
+            ])
+            .arg(files.journal())
+            .arg("--test-progress")
+            .arg(files.progress())
+            .env("PYTHONHASHSEED", (args.seed as u32).to_string())
+            .env("PYTHONUNBUFFERED", "1");
+        if args.debug {
+            command.arg("-d");
+        }
+        if args.verbose {
+            command.arg("-v");
+        }
+        if args.disable_mc_tests {
+            command.arg("--disable-mc-tests");
+        }
+        resources::configure(&mut command, limit);
+        let reader =
+            BufReader::new(File::open(files.journal()).map_err(|error| error.to_string())?);
+        let output = resources::capture(&mut command).map_err(|error| error.to_string())?;
+        let child = command.spawn().map_err(|error| error.to_string())?;
+        Ok((files, limit, reader, output, child))
+    };
+    let (files, limit, reader, output, mut child) = match setup() {
+        Ok(setup) => setup,
+        Err(error) => {
+            let outcome = Outcome::WorkerError(error);
+            println!("\n--- {} ---\n", batch[0].name);
+            report(&batch[0].name, &outcome);
+            results.insert(batch[0].name.clone(), outcome);
+            return 1;
+        }
+    };
+    let stderr = output_reader(output);
+    let mut journal = Journal {
+        reader,
+        pending: String::new(),
+        next: 0,
+        started: Instant::now(),
+        announced: false,
+    };
+    let mut last_progress = Instant::now();
+    let mut detail = "starting worker".to_owned();
+    let mut stopped = None;
+    let status = loop {
+        if let Err(error) = journal.read(batch, results) {
+            stopped = Some(Outcome::WorkerError(error));
+            break None;
+        }
+        match child.try_wait() {
+            Ok(Some(status)) => break Some(status),
+            Err(error) => {
+                stopped = Some(Outcome::WorkerError(error.to_string()));
+                break None;
+            }
+            Ok(None) => {}
+        }
+        if let Ok(message) = fs::read_to_string(files.progress()) {
+            detail = message;
+        }
+        let now = Instant::now();
+        let current = &batch[journal.next.min(batch.len() - 1)];
+        let limit_seconds = seconds(args, current);
+        if now >= suite
+            || now >= phase
+            || now.duration_since(journal.started).as_secs() >= limit_seconds
+        {
+            let budget = if now >= suite {
+                format!("suite budget of {}s", args.suite_timeout)
+            } else if now >= phase {
+                format!("chaos budget of {}s", args.chaos_timeout)
+            } else {
+                format!("test budget of {limit_seconds}s")
+            };
+            stopped = Some(Outcome::TimeLimit(format!("{budget}; {detail}")));
+            break None;
+        }
+        if last_progress.elapsed() >= Duration::from_secs(5) {
+            let memory = resources::memory_usage(&child).map_or(String::new(), |mib| {
+                format!("; address space {mib}/{limit} MiB")
+            });
+            eprintln!(
+                "RUNNING: {}; {}; elapsed {:.0}s{}",
+                current.name,
+                detail,
+                journal.started.elapsed().as_secs_f64(),
+                memory
+            );
+            last_progress = now;
+        }
+        thread::sleep(Duration::from_millis(10));
+    };
+    resources::terminate(&mut child);
+    let stderr = stderr.join().unwrap_or_default();
+    // The worker may have committed another result between the last poll and exit.
+    if let Err(error) = journal.read(batch, results) {
+        stopped = Some(Outcome::WorkerError(error));
+    }
+    if journal.next > 0 {
+        let finished = &batch[journal.next - 1];
+        if let Some(Outcome::WorkerError(message)) = results.get(&finished.name) {
+            let phase = fs::read_to_string(files.progress()).unwrap_or_default();
+            let outcome = Outcome::WorkerError(format!("{message}; {phase}"));
+            report(&finished.name, &outcome);
+            results.insert(finished.name.clone(), outcome);
+        }
+    }
+    if journal.next == batch.len() {
+        return journal.next;
+    }
+    // A handled panic/allocation failure ends the batch cleanly. Restart the
+    // remaining runs without inventing a failure for a test that never started.
+    if stopped.is_none() && journal.next > 0 && !journal.announced {
+        return journal.next;
+    }
+    if let Ok(message) = fs::read_to_string(files.progress()) {
+        detail = message;
+    }
+    let test = &batch[journal.next];
+    let outcome = stopped.unwrap_or_else(|| {
+        if resources::allocation_crash(status, &stderr) {
+            Outcome::MemoryLimit(format!("worker address-space limit {limit} MiB; {detail}"))
+        } else {
+            Outcome::WorkerError(format!(
+                "{}; {detail}",
+                status.map_or("worker stopped".into(), |status| format!(
+                    "exited with {status} without a result"
+                ))
+            ))
+        }
+    });
+    // Read the final journal first: Started may have arrived after the timeout
+    // poll. Only an actually started test is eligible for a resource exemption.
+    let outcome = outcome.require_started(journal.announced);
+    if !journal.announced {
+        println!("\n--- {} ---\n", test.name);
+    }
+    report(&test.name, &outcome);
+    println!(
+        "REPRODUCE: -t {:?} -s {} -p {} -m {} (effective run seed: {})\n",
+        test.name, args.seed, args.process_count, args.monkeys, test.seed
+    );
+    results.insert(test.name.clone(), outcome);
+    journal.next + 1
+}
+
+pub fn run(args: &Args) -> Result<IndexMap<String, Outcome>, String> {
+    let selected: Vec<_> = tests(args)
+        .into_iter()
+        .filter(|test| args.test.as_ref().is_none_or(|name| name == &test.name))
+        .collect();
+    if selected.is_empty() {
+        println!("FAILED: unknown test {:?}", args.test);
+        return Ok(IndexMap::new());
+    }
+    let maximum_memory = selected
+        .iter()
+        .map(|test| memory_limit(args, test))
+        .max()
+        .unwrap();
+    resources::limit(maximum_memory)?;
+    let suite = Instant::now() + Duration::from_secs(args.suite_timeout);
+    let mut results: IndexMap<_, _> = selected
+        .iter()
+        .map(|test| {
+            (
+                test.name.clone(),
+                Outcome::Skipped(format!(
+                    "suite budget of {}s exhausted before this test started",
+                    args.suite_timeout
+                )),
+            )
+        })
+        .collect();
+    let mut next = 0;
+    while next < selected.len() && Instant::now() < suite {
+        // A single explicitly selected test remains independently runnable.
+        if args.test.is_none() && selected[next].name.starts_with("SCALABILITY") {
+            let blockers = scalability_blockers(&results, args.monkeys, args.disable_mc_tests);
+            if !blockers.is_empty() {
+                for test in &selected[next..] {
+                    let outcome = Outcome::SkippedPrerequisites(blockers.clone());
+                    println!("\n--- {} ---\n", test.name);
+                    report(&test.name, &outcome);
+                    results.insert(test.name.clone(), outcome);
+                }
+                break;
+            }
+        }
+        let mut end = next + 1;
+        let chaos = selected[next].chaos_run.is_some();
+        if chaos {
+            while end < selected.len() && selected[end].chaos_run.is_some() {
+                end += 1;
+            }
+        }
+        let phase = if chaos {
+            Instant::now() + Duration::from_secs(args.chaos_timeout)
+        } else {
+            suite
+        };
+        while next < end && Instant::now() < suite && Instant::now() < phase {
+            next += run_worker(args, &selected[next..end], &mut results, suite, phase);
+        }
+        if next < end {
+            if Instant::now() < suite {
+                for test in &selected[next..end] {
+                    results.insert(
+                        test.name.clone(),
+                        Outcome::Skipped(format!(
+                            "chaos budget of {}s exhausted before this run started",
+                            args.chaos_timeout
+                        )),
+                    );
+                }
+            }
+            next = end;
+        }
+    }
+    if args.test.is_none() {
+        let passed = results
+            .values()
+            .filter(|result| matches!(result, Outcome::Passed))
+            .count();
+        println!("-------------------------------------------------------------------------------\n\nPassed {passed} from {} tests\n", selected.len());
+        for (name, result) in &results {
+            if !matches!(result, Outcome::Passed) {
+                println!("- {name}: {}", result.description());
+            }
+        }
+        println!();
+    }
+    Ok(results)
+}
+
+#[cfg(test)]
+pub fn passed_results(monkeys: u32) -> IndexMap<String, Outcome> {
+    let args = <Args as clap::Parser>::parse_from(["membership", "-m", &monkeys.to_string()]);
+    tests(&args)
+        .into_iter()
+        .map(|test| (test.name, Outcome::Passed))
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resource_failure_requires_a_started_record_even_at_the_deadline() {
+        // Model the final journal drain on either side of the startup boundary.
+        // With no Started record, the result must remain ineligible for MC credit.
+        for resource in [
+            Outcome::TimeLimit("suite budget exhausted during startup".into()),
+            Outcome::MemoryLimit("allocation failed during startup".into()),
+        ] {
+            let files = Files::new().unwrap();
+            let args = <Args as clap::Parser>::parse_from(["membership", "-m", "0"]);
+            let registry = tests(&args);
+            let mc_index = registry
+                .iter()
+                .position(|test| test.name == "MODEL CHECKING")
+                .unwrap();
+            let batch = &registry[mc_index..mc_index + 1];
+            let mut journal = Journal {
+                reader: BufReader::new(File::open(files.journal()).unwrap()),
+                pending: String::new(),
+                next: 0,
+                started: Instant::now(),
+                announced: false,
+            };
+            let mut results = IndexMap::new();
+            journal.read(batch, &mut results).unwrap();
+            let outcome = resource.clone().require_started(journal.announced);
+            assert!(matches!(outcome, Outcome::Skipped(_)));
+            assert!(!outcome.resource_limited());
+
+            // Started committed just before termination is picked up by the
+            // final drain, and makes a genuine interrupted run resource-limited.
+            let mut writer = OpenOptions::new()
+                .append(true)
+                .open(files.journal())
+                .unwrap();
+            event(&mut writer, Event::Started(batch[0].name.clone()));
+            journal.read(batch, &mut results).unwrap();
+            assert!(resource
+                .require_started(journal.announced)
+                .resource_limited());
+        }
+    }
+
+    #[test]
+    fn default_memory_limits_are_separate_for_mc_and_simulation() {
+        let args = <Args as clap::Parser>::parse_from(["membership"]);
+        for test in tests(&args) {
+            assert_eq!(
+                memory_limit(&args, &test),
+                if test.name == "MODEL CHECKING" {
+                    14336
+                } else {
+                    4096
+                }
+            );
+        }
+    }
+
+    #[test]
+    fn registry_preserves_all_tests_and_chaos_seeds() {
+        let args = <Args as clap::Parser>::parse_from(["membership", "-m", "1000"]);
+        let registry = tests(&args);
+        assert_eq!(registry.len(), 1027);
+        let ordinary = ORDINARY.len();
+        assert_eq!(registry[0].name, "SIMPLE");
+        assert_eq!(registry[ordinary].name, "CHAOS MONKEY (run 1)");
+        assert_eq!(registry[ordinary + 999].name, "CHAOS MONKEY (run 1000)");
+        assert_eq!(registry[ordinary + 1000].name, "MODEL CHECKING");
+        assert_eq!(registry[ordinary + 1001].name, "SCALABILITY NORMAL");
+        assert_eq!(registry[ordinary + 1002].name, "SCALABILITY CRASH");
+        let mut rand = Pcg64::seed_from_u64(123);
+        for test in &registry[ordinary..ordinary + 1000] {
+            assert_eq!(test.seed, rand.next_u64());
+        }
+    }
+
+    #[test]
+    fn oversized_ipc_records_do_not_exhaust_supervisor_memory() {
+        let files = Files::new().unwrap();
+        fs::write(files.journal(), vec![b'x'; 100_000]).unwrap();
+        let args = <Args as clap::Parser>::parse_from(["membership"]);
+        let mut journal = Journal {
+            reader: BufReader::new(File::open(files.journal()).unwrap()),
+            pending: String::new(),
+            next: 0,
+            started: Instant::now(),
+            announced: false,
+        };
+        let error = journal
+            .read(&tests(&args), &mut IndexMap::new())
+            .unwrap_err();
+        assert_eq!(error, "worker result exceeds 64 KiB");
+        assert!(journal.pending.len() <= 65537);
+    }
+
+    #[test]
+    fn journal_keeps_completed_results_and_waits_for_partial_records() {
+        let files = Files::new().unwrap();
+        let args = <Args as clap::Parser>::parse_from(["membership", "-m", "2"]);
+        let registry = tests(&args);
+        let batch = &registry[ORDINARY.len()..ORDINARY.len() + 2];
+        let mut writer = OpenOptions::new()
+            .append(true)
+            .open(files.journal())
+            .unwrap();
+        event(&mut writer, Event::Started(batch[0].name.clone()));
+        event(
+            &mut writer,
+            Event::Finished(batch[0].name.clone(), Outcome::Passed),
+        );
+        let mut journal = Journal {
+            reader: BufReader::new(File::open(files.journal()).unwrap()),
+            pending: String::new(),
+            next: 0,
+            started: Instant::now(),
+            announced: false,
+        };
+        let mut results = IndexMap::new();
+        journal.read(batch, &mut results).unwrap();
+        assert_eq!(journal.next, 1);
+        assert!(matches!(results[&batch[0].name], Outcome::Passed));
+        let record = serde_json::to_string(&Event::Started(batch[1].name.clone())).unwrap();
+        writer.write_all(record.as_bytes()).unwrap();
+        writer.flush().unwrap();
+        journal.read(batch, &mut results).unwrap();
+        assert!(!journal.announced);
+        writeln!(writer).unwrap();
+        writer.flush().unwrap();
+        journal.read(batch, &mut results).unwrap();
+        assert!(journal.announced);
+        assert_eq!(results.len(), 1);
+    }
+}

homework/06-membership/tests/src/shared_state.rs

@@ -0,0 +1,78 @@
+use std::ffi::CString;
+
+use pyo3::prelude::*;
+use pyo3::types::{PyList, PyModule};
+
+const VALIDATOR_CODE: &str = include_str!("../shared_state_validator.py");
+
+pub(crate) fn validator_module<'py>(py: Python<'py>) -> PyResult<Bound<'py, PyModule>> {
+    let code = CString::new(VALIDATOR_CODE).expect("validator source contains a null byte");
+    PyModule::from_code(
+        py,
+        &code,
+        c"shared_state_validator.py",
+        c"shared_state_validator",
+    )
+}
+
+#[cfg(test)]
+fn validate_source(
+    source: &str,
+    filename: &str,
+    class_names: &[&str],
+) -> Result<Vec<String>, String> {
+    Python::attach(|py| {
+        let module = validator_module(py).map_err(|error| error.to_string())?;
+        let class_names = PyList::new(py, class_names).map_err(|error| error.to_string())?;
+        module
+            .getattr("validate_source")
+            .and_then(|function| function.call1((source, filename, class_names)))
+            .and_then(|result| result.extract::<Vec<String>>())
+            .map_err(|error| error.to_string())
+    })
+}
+
+fn validate_solution(path: &str, class_names: &[&str]) -> Result<Vec<String>, String> {
+    Python::attach(|py| {
+        let module = validator_module(py).map_err(|error| error.to_string())?;
+        let class_names = PyList::new(py, class_names).map_err(|error| error.to_string())?;
+        module
+            .getattr("validate_solution")
+            .and_then(|function| function.call1((path, class_names)))
+            .and_then(|result| result.extract::<Vec<String>>())
+            .map_err(|error| error.to_string())
+    })
+}
+
+pub fn validate_or_report(path: &str, class_names: &[&str]) -> bool {
+    match validate_solution(path, class_names) {
+        Ok(violations) if violations.is_empty() => true,
+        Ok(violations) => {
+            println!("ERROR: shared state between AnySystem processes is forbidden");
+            for violation in violations {
+                println!("{violation}");
+            }
+            println!("\nSCORE: 0\n");
+            false
+        }
+        Err(error) => panic!("failed to validate solution for shared state: {error}"),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::validate_source;
+
+    #[test]
+    fn embeds_validator_and_rejects_shared_state() {
+        let violations = validate_source(
+            "class ProcessImpl:\n    shared = {}\n",
+            "solution.py",
+            &["ProcessImpl"],
+        )
+        .unwrap();
+
+        assert_eq!(violations.len(), 1);
+        assert!(violations[0].contains("ProcessImpl.shared"));
+    }
+}

homework/06-membership/tests/src/tests.rs

@@ -0,0 +1,813 @@
+use std::collections::{BTreeSet, HashMap};
+
+use assertables::{assume, assume_eq};
+use log::info;
+use rand::prelude::*;
+use rand_pcg::Pcg64;
+use sugars::boxed;
+
+use anysystem::test::TestResult;
+use anysystem::{Message, System};
+
+use crate::common::*;
+
+pub fn test_simple(config: &TestConfig) -> TestResult {
+    let mut sys = build_system(config);
+    let group = sys.process_names();
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)
+}
+
+pub fn test_get_members_semantics(config: &TestConfig) -> TestResult {
+    let mut sys = build_system(config);
+    let group = sys.process_names();
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+    for proc in sys.process_names() {
+        get_members(&mut sys, &proc)?;
+    }
+    Ok(true)
+}
+
+pub fn test_random_seed(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = Vec::new();
+    for proc in sys.process_names() {
+        let seed = match group.len() {
+            0 => &proc,
+            _ => group.choose(&mut rand).unwrap(),
+        };
+        sys.send_local_message(&proc, Message::json("JOIN", &JoinMessage { seed }));
+        group.push(proc);
+    }
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_process_join(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let new_proc = group.remove(rand.gen_range(0..group.len()));
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process joins the system
+    sys.send_local_message(&new_proc, Message::json("JOIN", &JoinMessage { seed }));
+    group.push(new_proc);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_process_leave(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process leaves the system
+    let left_proc = group.remove(rand.gen_range(0..group.len()));
+    sys.send_local_message(&left_proc, Message::json("LEAVE", &LeaveMessage {}));
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_process_crash(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process crashes
+    let crashed = group.remove(rand.gen_range(0..group.len()));
+    crash_process(&crashed, &mut sys);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_seed_process_crash(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0].clone();
+    initialize_group(&mut sys, &group, seed)?;
+
+    // seed process crashes
+    group.remove(0);
+    crash_process(seed, &mut sys);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_process_crash_recover(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0].clone();
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process crashes
+    let crashed = group.remove(rand.gen_range(0..group.len()));
+    crash_process(&crashed, &mut sys);
+    step_until_stabilized(&mut sys, group.clone().into_iter().collect())?;
+
+    // process recovers
+    recover_process(&crashed, &mut sys, config);
+    sys.send_local_message(&crashed, Message::json("JOIN", &JoinMessage { seed }));
+
+    group.push(crashed);
+    step_until_membership_stable(&mut sys, group.into_iter().collect(), "process recovery")
+}
+
+pub fn test_process_offline(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process goes offline
+    let offline_proc = group.remove(rand.gen_range(0..group.len()));
+    sys.network().disconnect_node(&offline_proc);
+    step_until_stabilized(&mut sys, group.into_iter().collect())?;
+    step_until_stabilized_at(&mut sys, BTreeSet::from([offline_proc]), "isolated process")
+}
+
+pub fn test_seed_process_offline(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0].clone();
+    initialize_group(&mut sys, &group, seed)?;
+
+    // seed process goes offline
+    group.remove(0);
+    sys.network().disconnect_node(seed);
+    step_until_stabilized(&mut sys, group.into_iter().collect())?;
+    step_until_stabilized_at(&mut sys, BTreeSet::from([seed.clone()]), "isolated process")
+}
+
+pub fn test_process_offline_recover(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process goes offline
+    let offline_proc = group.remove(rand.gen_range(0..group.len()));
+    sys.network().disconnect_node(&offline_proc);
+    step_until_stabilized(&mut sys, group.clone().into_iter().collect())?;
+    step_until_stabilized_at(
+        &mut sys,
+        BTreeSet::from([offline_proc.clone()]),
+        "isolated process",
+    )?;
+
+    // process goes back online
+    sys.network().connect_node(&offline_proc);
+    group.push(offline_proc);
+    step_until_membership_stable(
+        &mut sys,
+        group.into_iter().collect(),
+        "network reconnection",
+    )
+}
+
+pub fn test_network_partition(config: &TestConfig) -> TestResult {
+    run_network_partition(config, false, false)
+}
+
+pub fn test_network_partition_recover(config: &TestConfig) -> TestResult {
+    run_network_partition(config, false, true)
+}
+
+pub fn test_network_partition_unequal(config: &TestConfig) -> TestResult {
+    run_network_partition(config, true, false)
+}
+
+pub fn test_network_partition_unequal_recover(config: &TestConfig) -> TestResult {
+    run_network_partition(config, true, true)
+}
+
+fn run_network_partition(config: &TestConfig, unequal: bool, recover: bool) -> TestResult {
+    if unequal {
+        assume!(
+            config.process_count >= 3,
+            "Unequal network partition requires at least 3 processes"
+        )?;
+    }
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    let (group1, group2): (Vec<_>, Vec<_>) = if unequal {
+        // Always a nonempty strict minority, including odd process counts.
+        let (minority, majority) = group.split_at((group.len() - 1) / 2);
+        (
+            minority.iter().map(String::as_str).collect(),
+            majority.iter().map(String::as_str).collect(),
+        )
+    } else {
+        // Preserve the existing random partition and its seeds.
+        group
+            .iter()
+            .map(String::as_str)
+            .partition(|_| rand.gen_range(0.0..1.0) > 0.6)
+    };
+    info!(
+        "Network partition: [{}] ({} processes) | [{}] ({} processes)",
+        group1.join(", "),
+        group1.len(),
+        group2.join(", "),
+        group2.len()
+    );
+    sys.network().make_partition(&group1, &group2);
+    step_until_stabilized_at(
+        &mut sys,
+        group1.into_iter().map(String::from).collect(),
+        "network partition: first subgroup",
+    )?;
+    step_until_stabilized_at(
+        &mut sys,
+        group2.into_iter().map(String::from).collect(),
+        "network partition: second subgroup",
+    )?;
+
+    if recover {
+        sys.network().reset();
+        step_until_membership_stable(&mut sys, group.into_iter().collect(), "network recovery")?;
+    }
+    Ok(true)
+}
+
+pub fn test_process_cannot_receive(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process goes partially offline (cannot receive incoming messages)
+    let blocked_proc = group.remove(rand.gen_range(0..group.len()));
+    sys.network().drop_incoming(&blocked_proc);
+    step_until_stabilized(&mut sys, group.into_iter().collect())?;
+    step_until_stabilized_at(&mut sys, BTreeSet::from([blocked_proc]), "isolated process")
+}
+
+pub fn test_process_join_cannot_receive(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let new_proc = group.remove(rand.gen_range(0..group.len()));
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process tries to join the system but cannot receive messages
+    // it must not be kept in the group
+    sys.network().drop_incoming(&new_proc);
+    sys.send_local_message(&new_proc, Message::json("JOIN", &JoinMessage { seed }));
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_process_cannot_send(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // process goes partially offline (cannot send outgoing messages)
+    let blocked_proc = group.remove(rand.gen_range(0..group.len()));
+    sys.network().drop_outgoing(&blocked_proc);
+    step_until_stabilized(&mut sys, group.into_iter().collect())?;
+    step_until_stabilized_at(&mut sys, BTreeSet::from([blocked_proc]), "isolated process")
+}
+
+pub fn test_two_processes_cannot_communicate(config: &TestConfig) -> TestResult {
+    run_pairwise_link_failure(config, false)
+}
+
+pub fn test_two_processes_cannot_communicate_in_triangle(config: &TestConfig) -> TestResult {
+    let config = TestConfig {
+        process_count: 3,
+        ..*config
+    };
+    run_pairwise_link_failure(&config, true)
+}
+
+fn run_pairwise_link_failure(config: &TestConfig, triangle: bool) -> TestResult {
+    assume!(
+        config.process_count >= 3,
+        "Pairwise link failure with a connected group requires at least 3 processes"
+    )?;
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    if triangle {
+        group.sort();
+    } else {
+        group.shuffle(&mut rand);
+    }
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // In a triangle, the seed remains the bridge between the other two nodes.
+    let proc1 = if triangle { &group[1] } else { seed };
+    let proc2 = if triangle {
+        &group[2]
+    } else {
+        group.get(rand.gen_range(1..group.len())).unwrap()
+    };
+    info!(
+        "Network links disabled: {proc1} <-> {proc2}; group remains connected (N={})",
+        group.len()
+    );
+    sys.network().disable_link(proc1, proc2);
+    sys.network().disable_link(proc2, proc1);
+    // Preserve the original test's execution before checking convergence.
+    sys.steps(1000);
+    step_until_membership_stable(
+        &mut sys,
+        group.into_iter().collect(),
+        "pairwise link failure",
+    )
+}
+
+/// Once membership is correct, observe every event without advancing time in GET_MEMBERS.
+/// A transient mismatch restarts the stability window, allowing eventual convergence.
+fn step_until_membership_stable(
+    sys: &mut System,
+    group: BTreeSet<String>,
+    stage: &str,
+) -> TestResult {
+    const STABLE_WINDOW: f64 = 30.0;
+    const CONVERGENCE_LIMIT: f64 = 300.0;
+    let deadline = sys.time() + CONVERGENCE_LIMIT;
+    let mut stable_since = None;
+    let mut checked_events = 0_u64;
+    let mut search_exhausted = false;
+    let mut last_mismatch = String::from("no correct membership snapshot");
+    loop {
+        let now = sys.time();
+        // The next event may lie beyond the observed window. Its effects must
+        // not invalidate the unchanged state inside the completed window.
+        if stable_since
+            .is_some_and(|since| now > since + STABLE_WINDOW && since + STABLE_WINDOW <= deadline)
+        {
+            break;
+        }
+        if now > deadline {
+            return Err(format!(
+                "Group members lists are not stable for {STABLE_WINDOW}s within {CONVERGENCE_LIMIT}s; {last_mismatch}"
+            ));
+        }
+        if checked_events.is_multiple_of(100) {
+            crate::runner::progress(format!(
+                "{stage}: membership stability: N={}, simulation time={now:.1}, stable for {:.1}/{STABLE_WINDOW}s, checked events={checked_events}",
+                group.len(), stable_since.map_or(0.0, |since| now - since)
+            ));
+        }
+        let mut correct = true;
+        for proc in &group {
+            let data = get_members(sys, proc)?;
+            let members: BTreeSet<_> = data.members.into_iter().collect();
+            if members != group {
+                correct = false;
+                last_mismatch = format!(
+                    "process {proc} at simulation time {now:.3}: expected {group:?}, got {members:?}"
+                );
+            }
+        }
+        if correct {
+            let since = *stable_since.get_or_insert(now);
+            if now - since >= STABLE_WINDOW {
+                break;
+            }
+        } else {
+            stable_since = None;
+        }
+        if !correct && !search_exhausted && now < deadline {
+            // Search for the next candidate snapshot at the usual 5s cadence.
+            // Only a correct candidate starts event-by-event observation. This
+            // avoids filling AnySystem's retained trace with failed queries.
+            search_exhausted = !sys.step_for_duration(5.0_f64.min(deadline - now));
+            continue;
+        }
+        if !correct {
+            return Err(format!(
+                "Group members lists are not stable for {STABLE_WINDOW}s within {CONVERGENCE_LIMIT}s; {last_mismatch}"
+            ));
+        }
+        if !sys.step() {
+            // With no pending events, a correct membership cannot change.
+            break;
+        }
+        checked_events += 1;
+    }
+    info!(
+        "Membership remained stable for {STABLE_WINDOW}s ({stage}; checked after {checked_events} simulation events)"
+    );
+    Ok(true)
+}
+
+pub fn test_slow_network(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // slow down network for a while
+    sys.network().set_delays(0.1, 1.0);
+    sys.steps(200);
+    sys.network().set_delays(0.01, 0.1);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_flaky_network(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // make network unreliable for a while
+    sys.network().set_drop_rate(0.5);
+    sys.steps(1000);
+    sys.network().set_drop_rate(0.0);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_flaky_network_on_start(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+
+    // make network unreliable from the start
+    sys.network().set_drop_rate(0.2);
+    for proc in &group {
+        sys.send_local_message(proc, Message::json("JOIN", &JoinMessage { seed }));
+    }
+    sys.steps(1000);
+    sys.network().set_drop_rate(0.0);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_flaky_network_and_crash(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    // make network unreliable for a while + crash process
+    sys.network().set_drop_rate(0.5);
+    let crashed = group.remove(rand.gen_range(0..group.len()));
+    crash_process(&crashed, &mut sys);
+    sys.steps(1000);
+    sys.network().set_drop_rate(0.0);
+    step_until_stabilized(&mut sys, group.into_iter().collect())
+}
+
+pub fn test_chaos_monkey(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let mut group = sys.process_names();
+    group.shuffle(&mut rand);
+    let seed = &group[0];
+    initialize_group(&mut sys, &group, seed)?;
+
+    info!(
+        "CHAOS MONKEY run seed: {}; join order: [{}]; seed process: {seed}",
+        config.seed,
+        group.join(", ")
+    );
+    // Keep at least two participants; pairwise link failures must not isolate
+    // a participant still expected in the control group.
+    let failure_count = 6.min(group.len().saturating_sub(2));
+    for step in 1..=failure_count {
+        let p = rand.gen_range(0.0..1.0);
+        // do some nasty things
+        match p {
+            p if p < 0.25 => {
+                // crash process
+                let victim = group.remove(rand.gen_range(0..group.len()));
+                info!(
+                    "- Step {step} at {:.2}: process {victim} crashes",
+                    sys.time()
+                );
+                crash_process(&victim, &mut sys);
+            }
+            p if p < 0.5 => {
+                // disconnect process
+                let victim = group.remove(rand.gen_range(0..group.len()));
+                info!(
+                    "- Step {step} at {:.2}: process {victim} disconnects",
+                    sys.time()
+                );
+                sys.network().disconnect_node(&victim);
+            }
+            p if p < 0.75 => {
+                // partially disconnect process (cannot receive)
+                let victim = group.remove(rand.gen_range(0..group.len()));
+                info!(
+                    "- Step {step} at {:.2}: process {victim} cannot receive messages",
+                    sys.time()
+                );
+                sys.network().drop_incoming(&victim);
+            }
+            _ => {
+                // two processes cannot communicate with each other
+                let proc1 = group.get(rand.gen_range(0..group.len())).unwrap();
+                let mut proc2 = group.get(rand.gen_range(0..group.len())).unwrap();
+                while proc1 == proc2 {
+                    proc2 = group.get(rand.gen_range(0..group.len())).unwrap();
+                }
+                info!(
+                    "- Step {step} at {:.2}: processes {proc1} and {proc2} cannot communicate",
+                    sys.time()
+                );
+                sys.network().disable_link(proc1, proc2);
+                sys.network().disable_link(proc2, proc1);
+            }
+        }
+        info!(
+            "Control group after step {step}: [{}] ({} processes)",
+            group.join(", "),
+            group.len()
+        );
+        step_until_stabilized(&mut sys, group.clone().into_iter().collect())?;
+    }
+    Ok(true)
+}
+
+pub fn test_scalability_normal(config: &TestConfig) -> TestResult {
+    let sys_sizes = [
+        config.process_count,
+        config.process_count * 2,
+        config.process_count * 5,
+        config.process_count * 10,
+    ];
+    let mut measurements = Vec::new();
+    for size in sys_sizes {
+        let mut run_config = *config;
+        run_config.process_count = size;
+        let mut rand = Pcg64::seed_from_u64(config.seed);
+        let mut sys = build_system(&run_config);
+        let mut group = sys.process_names();
+        group.shuffle(&mut rand);
+        let seed = &group[0];
+        initialize_group(&mut sys, &group, seed)?;
+
+        let init_time = sys.time();
+        let init_net_traffic = sys.network().traffic();
+        let init_msg_count = sys.network().network_message_count();
+        let mut init_loads = HashMap::new();
+        for proc in sys.process_names() {
+            init_loads.insert(
+                proc.clone(),
+                sys.sent_message_count(&proc) + sys.received_message_count(&proc),
+            );
+        }
+
+        crate::runner::progress(format!(
+            "steady-state measurement: N={size}, simulation time={:.1}",
+            sys.time()
+        ));
+        sys.step_for_duration(10.0);
+
+        let mut loads = Vec::new();
+        for proc in sys.process_names() {
+            let load = sys.sent_message_count(&proc) + sys.received_message_count(&proc);
+            loads.push(load - init_loads.get(&proc).unwrap());
+        }
+        let min_load = *loads.iter().min().unwrap();
+        let max_load = *loads.iter().max().unwrap();
+        let duration = sys.time() - init_time;
+        let traffic = sys.network().traffic();
+        let message_count = sys.network().network_message_count();
+        measurements.push((
+            duration,
+            (traffic - init_net_traffic) as f64 / duration,
+            (message_count - init_msg_count) as f64 / duration,
+            max_load as f64 / duration,
+            max_load as f64 / min_load as f64,
+        ));
+    }
+    let mut scaling_ok = true;
+    let mut load_ratio_ok = true;
+    for i in 0..sys_sizes.len() {
+        let (time, traffic, message_count, max_load, load_ratio) = measurements[i];
+        println!(
+            "- N = {}: time - {:.2}, traffic/s - {:.2}, messages/s - {:.2}, max load - {:.2}, max/min load - {:.2}",
+            sys_sizes[i], time, traffic, message_count, max_load, load_ratio
+        );
+        if load_ratio > 5.0 {
+            load_ratio_ok = false;
+        }
+        if i > 0 {
+            let size_ratio = sys_sizes[i] as f64 / sys_sizes[i - 1] as f64;
+            let traffic_ratio = traffic / measurements[i - 1].1;
+            let messages_ratio = message_count / measurements[i - 1].2;
+            if traffic_ratio > 2.0 * size_ratio || messages_ratio > 2.0 * size_ratio {
+                scaling_ok = false;
+            }
+        }
+    }
+    assume!(scaling_ok, "Bad network load scaling")?;
+    assume!(load_ratio_ok, "Bad max/min process load")?;
+    Ok(true)
+}
+
+pub fn test_scalability_crash(config: &TestConfig) -> TestResult {
+    let sys_sizes = [
+        config.process_count,
+        config.process_count * 2,
+        config.process_count * 5,
+        config.process_count * 10,
+    ];
+    let mut measurements = Vec::new();
+    for size in sys_sizes {
+        let mut run_config = *config;
+        run_config.process_count = size;
+        let mut rand = Pcg64::seed_from_u64(config.seed);
+        let mut sys = build_system(&run_config);
+        let mut group = sys.process_names();
+        group.shuffle(&mut rand);
+        let seed = &group[0];
+        initialize_group(&mut sys, &group, seed)?;
+
+        let init_time = sys.time();
+        let init_net_traffic = sys.network().traffic();
+        let init_msg_count = sys.network().network_message_count();
+        let mut init_loads = HashMap::new();
+        for proc in sys.process_names() {
+            init_loads.insert(
+                proc.clone(),
+                sys.sent_message_count(&proc) + sys.received_message_count(&proc),
+            );
+        }
+
+        let crashed = group.remove(rand.gen_range(0..group.len()));
+        crash_process(&crashed, &mut sys);
+        step_until_stabilized_at(
+            &mut sys,
+            group.clone().into_iter().collect(),
+            "stabilization after crash",
+        )?;
+
+        let mut loads = Vec::new();
+        for proc in sys.process_names() {
+            if proc != crashed {
+                let load = sys.sent_message_count(&proc) + sys.received_message_count(&proc);
+                loads.push(load - init_loads.get(&proc).unwrap());
+            }
+        }
+        let min_load = *loads.iter().min().unwrap();
+        let max_load = *loads.iter().max().unwrap();
+        let duration = sys.time() - init_time;
+        let traffic = sys.network().traffic();
+        let message_count = sys.network().network_message_count();
+        measurements.push((
+            duration,
+            (traffic - init_net_traffic) as f64 / duration,
+            (message_count - init_msg_count) as f64 / duration,
+            max_load as f64 / duration,
+            max_load as f64 / min_load as f64,
+        ));
+    }
+    let mut scaling_ok = true;
+    let mut load_ratio_ok = true;
+    for i in 0..sys_sizes.len() {
+        let (time, traffic, message_count, max_load, load_ratio) = measurements[i];
+        println!(
+            "- N = {}: time - {:.2}, traffic/s - {:.2}, messages/s - {:.2}, max load - {:.2}, max/min load - {:.2}",
+            sys_sizes[i], time, traffic, message_count, max_load, load_ratio
+        );
+        if load_ratio > 5.0 {
+            load_ratio_ok = false;
+        }
+        if i > 0 {
+            let size_ratio = sys_sizes[i] as f64 / sys_sizes[i - 1] as f64;
+            let traffic_ratio = traffic / measurements[i - 1].1;
+            let messages_ratio = message_count / measurements[i - 1].2;
+            if traffic_ratio > 2.0 * size_ratio || messages_ratio > 2.0 * size_ratio {
+                scaling_ok = false;
+            }
+        }
+    }
+    assume!(scaling_ok, "Bad network load scaling")?;
+    assume!(load_ratio_ok, "Bad max/min process load")?;
+    Ok(true)
+}
+
+fn initialize_group(sys: &mut System, group: &Vec<String>, seed: &str) -> TestResult {
+    for proc in group {
+        sys.send_local_message(proc, Message::json("JOIN", &JoinMessage { seed }));
+    }
+    step_until_stabilized_at(
+        sys,
+        group.clone().into_iter().collect(),
+        "initial group assembly",
+    )
+}
+
+fn crash_process(name: &str, sys: &mut System) {
+    // we just crash the node on which the process is running
+    sys.crash_node(name);
+}
+
+fn recover_process(name: &str, sys: &mut System, config: &TestConfig) {
+    sys.recover_node(name);
+    let clock_skew = sys.gen_range(0.0..10.0);
+    // A reboot may change the local clock. Apply it before on_start is called.
+    sys.set_node_clock_skew(name, clock_skew);
+    info!(
+        "Process {name} recovers at simulation time {:.6} with clock skew {clock_skew:.6}",
+        sys.time()
+    );
+    let process = config.process_factory.build((name,), config.seed);
+    sys.add_process(name, boxed!(process), name);
+}
+
+fn get_members(sys: &mut System, proc: &str) -> Result<MembersMessage, String> {
+    let sent_before = sys.sent_message_count(proc);
+    sys.send_local_message(proc, Message::json("GET_MEMBERS", &GetMembersMessage {}));
+    assume_eq!(
+        sys.sent_message_count(proc),
+        sent_before,
+        format!("GET_MEMBERS must not send network messages: process {proc}")
+    )?;
+    let messages = sys
+        .step_until_local_message_max_steps(proc, 0)
+        .map_err(|_| {
+            format!("GET_MEMBERS must return immediately without simulation steps: process {proc}")
+        })?;
+    assume_eq!(messages.len(), 1, "expected exactly one MEMBERS message")?;
+    assume_eq!(messages[0].tip, "MEMBERS", "Wrong message type")?;
+    serde_json::from_str(&messages[0].data)
+        .map_err(|error| format!("Invalid MEMBERS message from {proc}: {error}"))
+}
+
+fn step_until_stabilized(sys: &mut System, group: BTreeSet<String>) -> TestResult {
+    step_until_stabilized_at(sys, group, "membership stabilization")
+}
+
+fn step_until_stabilized_at(sys: &mut System, group: BTreeSet<String>, stage: &str) -> TestResult {
+    let max_time = sys.time() + 300.; // timeout is 5 minutes
+    let mut stabilized = BTreeSet::new();
+    let mut memberlists = HashMap::new();
+
+    while stabilized.len() < group.len() && sys.time() < max_time {
+        crate::runner::progress(format!("{stage}: N={}, correct members lists={}/{}, simulation time={:.1}, network messages={}",
+            group.len(), stabilized.len(), group.len(), sys.time(), sys.network().network_message_count()));
+        let cont = sys.step_for_duration(5.);
+        stabilized.clear();
+        // Query processes in a stable order without advancing simulation time.
+        for proc in group.iter() {
+            let data = get_members(sys, proc)?;
+            let members: BTreeSet<String> = data.members.clone().into_iter().collect();
+            if members.eq(&group) {
+                stabilized.insert(proc.clone());
+            }
+            memberlists.insert(proc.clone(), data.members);
+        }
+        if !cont {
+            break;
+        }
+    }
+
+    if stabilized != group && group.len() <= 10 {
+        println!("Members lists:");
+        for proc in sys.process_names() {
+            if group.contains(&proc) {
+                let members = memberlists.get_mut(&proc).unwrap();
+                members.sort();
+                println!("- [{}] {}", proc, members.join(", "));
+            }
+        }
+        let mut expected = group.clone().into_iter().collect::<Vec<_>>();
+        expected.sort();
+        println!("Expected group: {}", expected.join(", "));
+    }
+    assume_eq!(stabilized, group, "Group members lists are not stabilized")?;
+    Ok(true)
+}

homework/06-membership/tests/src/tests_mc.rs

@@ -0,0 +1,159 @@
+use std::collections::HashSet;
+use std::time::Duration;
+
+use rand::prelude::*;
+use rand_pcg::Pcg64;
+use sugars::boxed;
+
+use anysystem::logger::{LogEntry, LogEntry::McMessageReceived};
+use anysystem::mc::{
+    predicates::{collects, goals, invariants, prunes},
+    strategies::Bfs,
+    CollectFn, EventOrderingMode, InvariantFn, McState, ModelChecker, StrategyConfig,
+};
+use anysystem::test::TestResult;
+use anysystem::{Message, System};
+
+use crate::common::*;
+
+pub fn test_mc_group(config: &TestConfig) -> TestResult {
+    let mut rand = Pcg64::seed_from_u64(config.seed);
+    let mut sys = build_system(config);
+    let group = sys.process_names();
+    let seed = group.choose(&mut rand).unwrap();
+
+    crate::runner::progress("MC: exploring group assembly, N=3");
+    let collected_states = mc_explore_after_joins(&mut sys, seed.to_string())?;
+    if collected_states.is_empty() {
+        return Err("no states collected during explore stage".to_string());
+    }
+    crate::runner::progress(format!(
+        "MC: checking membership in {} collected states, N=3",
+        collected_states.len()
+    ));
+    mc_check_members(&mut sys, collected_states)
+}
+
+fn mc_explore_after_joins(sys: &mut System, seed_proc: String) -> Result<HashSet<McState>, String> {
+    let procs = sys.process_names();
+    let mut mc = ModelChecker::new(sys);
+
+    let strategy_config = StrategyConfig::default()
+        // Explore only states with up to 2 timer firings per process
+        // (we expect each process to communicate with others at most 3 times:
+        // 1 time on a local message and 2 times on a timer).
+        .prune(prunes::events_limit_per_proc(
+            |entry: &LogEntry, proc_name: &String| match entry {
+                LogEntry::McTimerFired { proc, .. } => proc_name == proc,
+                _ => false,
+            },
+            procs.clone(),
+            2,
+        ))
+        // Stop when no events left or reached depth 20 (steps of simulation).
+        .goal(goals::any_goal(vec![
+            goals::no_events(),
+            goals::depth_reached(20),
+        ]))
+        // Inner exploration safeguard; the supervisor also bounds both MC phases.
+        .invariant(invariants::time_limit(Duration::from_secs(300)))
+        // Collect states in which the group should be stabilized, namely:
+        // either no events left, or every process received at least 3 messages.
+        //
+        // Considering a system with 3 processes and a stable network, we can show this is enough:
+        // * seed process (A) should get information from both other processes (B & C)
+        // * because C joins later than B, it will know the full group as well
+        // * B is able to get information about the full group either from A or C, depending on implementation
+        //
+        // Technically, there can be execution A <-> B (x3), A <-> C (x3) where B's information becomes outdated,
+        // but we consider solution wrong if it allows such execution.
+        .collect(collects::any_collect(vec![
+            collects::no_events(),
+            collects::all_collects(
+                procs
+                    .clone()
+                    .into_iter()
+                    .map(|proc_name| {
+                        collects::event_happened_n_times_current_run(
+                            move |log_entry| match log_entry {
+                                McMessageReceived { dst: proc, .. } => proc == &proc_name,
+                                _ => false,
+                            },
+                            3,
+                        )
+                    })
+                    .collect::<Vec<CollectFn>>(),
+            ),
+        ]));
+
+    let res = mc.run_with_change::<Bfs>(strategy_config, |sys| {
+        // Use event ordering mode which prioritizes messages over timers
+        // (this emulates perfect network where timeouts do not occur).
+        sys.set_event_ordering_mode(EventOrderingMode::MessagesFirst);
+        for proc in &procs {
+            sys.send_local_message(
+                proc.clone(),
+                proc.clone(),
+                Message::json("JOIN", &JoinMessage { seed: &seed_proc }),
+            );
+        }
+    });
+    match res {
+        Err(e) => {
+            e.print_trace();
+            Err(e.message())
+        }
+        Ok(stats) => {
+            // println!("collected {} states", stats.collected_states.len());
+            Ok(stats.collected_states)
+        }
+    }
+}
+
+fn mc_check_members(sys: &mut System, collected: HashSet<McState>) -> TestResult {
+    let procs = sys.process_names();
+    let mut mc = ModelChecker::new(sys);
+    let strategy_config = StrategyConfig::default()
+        .invariant(mc_invariant_check_stabilized(procs.clone()))
+        .goal(goals::always_ok());
+
+    let res = mc.run_from_states_with_change::<Bfs>(strategy_config, collected, |sys| {
+        for node in procs.iter() {
+            sys.send_local_message(
+                node,
+                node,
+                Message::json("GET_MEMBERS", &GetMembersMessage {}),
+            );
+        }
+    });
+    if let Err(e) = res {
+        e.print_trace();
+        Err(e.message())
+    } else {
+        Ok(true)
+    }
+}
+
+fn mc_invariant_check_stabilized(group: Vec<String>) -> InvariantFn {
+    boxed!(move |state| {
+        let group = group.clone().into_iter().collect::<HashSet<String>>();
+        for node in state.node_states.keys() {
+            if let Some(msg) = state.node_states[node].proc_states[node]
+                .local_outbox
+                .first()
+            {
+                if msg.tip != "MEMBERS" {
+                    return Err("wrong message type".to_owned());
+                }
+                let data: MembersMessage = serde_json::from_str(&msg.data).unwrap();
+                let members = HashSet::from_iter(data.members.into_iter());
+                if !members.eq(&group) {
+                    return Err(format!(
+                        "expected a stabilized group {group:?} but got {members:?}"
+                    ));
+                }
+            }
+        }
+        Ok(())
+    })
+}