Merge branch 'dev'

README.md

@@ -45,6 +45,7 @@ The merge is still actively in development. The exact specification has not been
   * [Merge fork](specs/merge/fork.md)
   * [Fork Choice changes](specs/merge/fork-choice.md)
   * [Validator additions](specs/merge/validator.md)
+  * [Client settings](specs/merge/client_settings.md)
 
 ### Sharding
 
@@ -53,7 +54,7 @@ Sharding follows the merge, and is divided into three parts:
 * Sharding base functionality - In early engineering phase
   * [Beacon Chain changes](specs/sharding/beacon-chain.md)
   * [P2P Network changes](specs/sharding/p2p-interface.md)
-* Custody Game - Ready, dependent on sharding 
+* Custody Game - Ready, dependent on sharding
   * [Beacon Chain changes](specs/custody_game/beacon-chain.md)
   * [Validator custody work](specs/custody_game/validator.md)
 * Data Availability Sampling - In active R&D

SECURITY.md

@@ -0,0 +1,11 @@
+# Security Policy
+
+## Supported Versions
+
+Please see [Releases](https://github.com/ethereum/consensus-specs/releases/). We recommend using the [most recently released version](https://github.com/ethereum/consensus-specs/releases/latest).
+
+## Reporting a Vulnerability
+
+**Please do not file a public ticket** mentioning the vulnerability.
+
+To find out how to disclose a vulnerability in the Ethereum Consensus Layer visit [https://eth2bounty.ethereum.org](https://eth2bounty.ethereum.org) or email [email protected]. Please read the [disclosure page](https://eth2bounty.ethereum.org) for more information about publicly disclosed security vulnerabilities.

configs/minimal.yaml

@@ -58,8 +58,8 @@ INACTIVITY_SCORE_RECOVERY_RATE: 16
 EJECTION_BALANCE: 16000000000
 # 2**2 (= 4)
 MIN_PER_EPOCH_CHURN_LIMIT: 4
-# 2**16 (= 65,536)
-CHURN_LIMIT_QUOTIENT: 65536
+# [customized] scale queue churn at much lower validator counts for testing
+CHURN_LIMIT_QUOTIENT: 32
 
 
 # Deposit contract

setup.py

@@ -223,7 +223,7 @@ def get_spec(file_name: Path, preset: Dict[str, str], config: Dict[str, str]) ->
 
                     if not _is_constant_id(name):
                         # Check for short type declarations
-                        if value.startswith("uint") or value.startswith("Bytes") or value.startswith("ByteList") or value.startswith("Union"):
+                        if value.startswith(("uint", "Bytes", "ByteList", "Union")):
                             custom_types[name] = value
                         continue
 

specs/merge/client_settings.md

@@ -0,0 +1,26 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [The Merge -- Client Settings](#the-merge----client-settings)
+    - [Override terminal total difficulty](#override-terminal-total-difficulty)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# The Merge -- Client Settings
+
+**Notice**: This document is a work-in-progress for researchers and implementers.
+
+This document specifies configurable settings that clients must implement for the Merge.
+
+### Override terminal total difficulty
+
+To coordinate manual overrides to [`terminal_total_difficulty`](fork-choice.md#transitionstore), clients
+must provide `--terminal-total-difficulty-override` as a configurable setting.
+
+If `TransitionStore` has already [been initialized](./fork.md#initializing-transition-store), this alters the previously initialized value of
+`TransitionStore.terminal_total_difficulty`, otherwise this setting initializes `TransitionStore` with the specified, bypassing `compute_terminal_total_difficulty` and the use of an `anchor_pow_block`.
+`terminal_total_difficulty`.
+
+Except under exceptional scenarios, this setting is expected to not be used, and `terminal_total_difficulty` will operate with [default functionality](./fork.md#initializing-transition-store). Sufficient warning to the user about this exceptional configurable setting should be provided.
+[here](fork.md#initializing-transition-store).

specs/merge/fork-choice.md

@@ -73,7 +73,7 @@ def finalize_block(self: ExecutionEngine, block_hash: Hash32) -> bool:
 ```python
 @dataclass
 class TransitionStore(object):
-    transition_total_difficulty: uint256
+    terminal_total_difficulty: uint256
 ```
 
 ### `PowBlock`
@@ -101,8 +101,8 @@ Used by fork-choice handler, `on_block`.
 
 ```python
 def is_valid_terminal_pow_block(transition_store: TransitionStore, block: PowBlock, parent: PowBlock) -> bool:
-    is_total_difficulty_reached = block.total_difficulty >= transition_store.transition_total_difficulty
-    is_parent_total_difficulty_valid = parent.total_difficulty < transition_store.transition_total_difficulty
+    is_total_difficulty_reached = block.total_difficulty >= transition_store.terminal_total_difficulty
+    is_parent_total_difficulty_valid = parent.total_difficulty < transition_store.terminal_total_difficulty
     return block.is_valid and is_total_difficulty_reached and is_parent_total_difficulty_valid
 ```
 

specs/merge/fork.md

@@ -97,18 +97,18 @@ def upgrade_to_merge(pre: altair.BeaconState) -> BeaconState:
         # Execution-layer
         latest_execution_payload_header=ExecutionPayloadHeader(),
     )
-    
+
     return post
 ```
 
 ### Initializing transition store
 
-If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == MERGE_FORK_EPOCH`, a transition store is initialized to be further utilized by the transition process of the Merge.
+If `state.slot % SLOTS_PER_EPOCH == 0`, `compute_epoch_at_slot(state.slot) == MERGE_FORK_EPOCH`, and the transition store has not already been initialized, a transition store is initialized to be further utilized by the transition process of the Merge.
 
 Transition store initialization occurs after the state has been modified by corresponding `upgrade_to_merge` function.
 
 ```python
-def compute_transition_total_difficulty(anchor_pow_block: PowBlock) -> uint256:
+def compute_terminal_total_difficulty(anchor_pow_block: PowBlock) -> uint256:
     seconds_per_voting_period = EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH * SECONDS_PER_SLOT
     pow_blocks_per_voting_period = seconds_per_voting_period // SECONDS_PER_ETH1_BLOCK
     pow_blocks_to_merge = TARGET_SECONDS_TO_MERGE // SECONDS_PER_ETH1_BLOCK
@@ -119,11 +119,14 @@ def compute_transition_total_difficulty(anchor_pow_block: PowBlock) -> uint256:
 
 
 def get_transition_store(anchor_pow_block: PowBlock) -> TransitionStore:
-    transition_total_difficulty = compute_transition_total_difficulty(anchor_pow_block)
-    return TransitionStore(transition_total_difficulty=transition_total_difficulty)
+    terminal_total_difficulty = compute_terminal_total_difficulty(anchor_pow_block)
+    return TransitionStore(terminal_total_difficulty=terminal_total_difficulty)
 
 
 def initialize_transition_store(state: BeaconState) -> TransitionStore:
     pow_block = get_pow_block(state.eth1_data.block_hash)
     return get_transition_store(pow_block)
 ```
+
+*Note*: Transition store can also be initialized at client startup by [overriding terminal total
+difficulty](client_settings.md#override-terminal-total-difficulty).

specs/merge/validator.md

@@ -95,7 +95,7 @@ def get_execution_payload(state: BeaconState,
                           execution_engine: ExecutionEngine,
                           pow_chain: Sequence[PowBlock]) -> ExecutionPayload:
     if not is_merge_complete(state):
-        terminal_pow_block = get_pow_block_at_total_difficulty(transition_store.transition_total_difficulty, pow_chain)
+        terminal_pow_block = get_pow_block_at_total_difficulty(transition_store.terminal_total_difficulty, pow_chain)
         if terminal_pow_block is None:
             # Pre-merge, empty payload
             return ExecutionPayload()

tests/core/pyspec/eth2spec/VERSION.txt

@@ -1 +1 @@
-1.1.0-beta.3
+1.1.0-beta.4

tests/core/pyspec/eth2spec/gen_helpers/gen_base/gen_runner.py

@@ -1,4 +1,5 @@
 import os
+import time
 import shutil
 import argparse
 from pathlib import Path
@@ -22,6 +23,9 @@
 context.is_pytest = False
 
 
+TIME_THRESHOLD_TO_PRINT = 1.0  # seconds
+
+
 def validate_output_dir(path_str):
     path = Path(path_str)
 
@@ -73,6 +77,13 @@ def run_generator(generator_name, test_providers: Iterable[TestProvider]):
         required=False,
         help="specify presets to run with. Allows all if no preset names are specified.",
     )
+    parser.add_argument(
+        "-c",
+        "--collect-only",
+        action="store_true",
+        default=False,
+        help="if set only print tests to generate, do not actually run the test and dump the target data",
+    )
 
     args = parser.parse_args()
     output_dir = args.output_dir
@@ -96,11 +107,15 @@ def run_generator(generator_name, test_providers: Iterable[TestProvider]):
     if len(presets) != 0:
         print(f"Filtering test-generator runs to only include presets: {', '.join(presets)}")
 
+    collect_only = args.collect_only
+    collected_test_count = 0
     generated_test_count = 0
     skipped_test_count = 0
+    provider_start = time.time()
     for tprov in test_providers:
-        # runs anything that we don't want to repeat for every test case.
-        tprov.prepare()
+        if not collect_only:
+            # runs anything that we don't want to repeat for every test case.
+            tprov.prepare()
 
         for test_case in tprov.make_cases():
             case_dir = (
@@ -110,6 +125,11 @@ def run_generator(generator_name, test_providers: Iterable[TestProvider]):
             )
             incomplete_tag_file = case_dir / "INCOMPLETE"
 
+            collected_test_count += 1
+            if collect_only:
+                print(f"Collected test at: {case_dir}")
+                continue
+
             if case_dir.exists():
                 if not args.force and not incomplete_tag_file.exists():
                     skipped_test_count += 1
@@ -122,6 +142,7 @@ def run_generator(generator_name, test_providers: Iterable[TestProvider]):
                     shutil.rmtree(case_dir)
 
             print(f'Generating test: {case_dir}')
+            test_start = time.time()
 
             written_part = False
 
@@ -179,10 +200,22 @@ def output_part(out_kind: str, name: str, fn: Callable[[Path, ], None]):
                     generated_test_count += 1
                     # Only remove `INCOMPLETE` tag file
                     os.remove(incomplete_tag_file)
+            test_end = time.time()
+            span = round(test_end - test_start, 2)
+            if span > TIME_THRESHOLD_TO_PRINT:
+                print(f'    - generated in {span} seconds')
 
-    summary_message = f"completed generation of {generator_name} with {generated_test_count} tests"
-    summary_message += f" ({skipped_test_count} skipped tests)"
-    print(summary_message)
+    provider_end = time.time()
+    span = round(provider_end - provider_start, 2)
+
+    if collect_only:
+        print(f"Collected {collected_test_count} tests in total")
+    else:
+        summary_message = f"completed generation of {generator_name} with {generated_test_count} tests"
+        summary_message += f" ({skipped_test_count} skipped tests)"
+        if span > TIME_THRESHOLD_TO_PRINT:
+            summary_message += f" in {span} seconds"
+        print(summary_message)
 
 
 def dump_yaml_fn(data: Any, name: str, file_mode: str, yaml_encoder: YAML):

tests/core/pyspec/eth2spec/gen_helpers/gen_from_tests/gen.py

@@ -1,6 +1,6 @@
 from importlib import import_module
 from inspect import getmembers, isfunction
-from typing import Any, Callable, Dict, Iterable, Optional
+from typing import Any, Callable, Dict, Iterable, Optional, List, Union
 
 from eth2spec.utils import bls
 from eth2spec.test.helpers.constants import ALL_PRESETS, TESTGEN_FORKS
@@ -59,8 +59,10 @@ def generate_from_tests(runner_name: str, handler_name: str, src: Any,
 def get_provider(create_provider_fn: Callable[[SpecForkName, PresetBaseName, str, str], TestProvider],
                  fork_name: SpecForkName,
                  preset_name: PresetBaseName,
-                 all_mods: Dict[str, Dict[str, str]]) -> Iterable[TestProvider]:
+                 all_mods: Dict[str, Dict[str, Union[List[str], str]]]) -> Iterable[TestProvider]:
     for key, mod_name in all_mods[fork_name].items():
+        if not isinstance(mod_name, List):
+            mod_name = [mod_name]
         yield create_provider_fn(
             fork_name=fork_name,
             preset_name=preset_name,
@@ -75,16 +77,17 @@ def prepare_fn() -> None:
         return
 
     def create_provider(fork_name: SpecForkName, preset_name: PresetBaseName,
-                        handler_name: str, tests_src_mod_name: str) -> TestProvider:
+                        handler_name: str, tests_src_mod_name: List[str]) -> TestProvider:
         def cases_fn() -> Iterable[TestCase]:
-            tests_src = import_module(tests_src_mod_name)
-            return generate_from_tests(
-                runner_name=runner_name,
-                handler_name=handler_name,
-                src=tests_src,
-                fork_name=fork_name,
-                preset_name=preset_name,
-            )
+            for mod_name in tests_src_mod_name:
+                tests_src = import_module(mod_name)
+                yield from generate_from_tests(
+                    runner_name=runner_name,
+                    handler_name=handler_name,
+                    src=tests_src,
+                    fork_name=fork_name,
+                    preset_name=preset_name,
+                )
 
         return TestProvider(prepare=prepare_fn, make_cases=cases_fn)
     return create_provider