guzhujushi
a1cc26e042
## 审查结论 算法移植是对的：大样本正态下百分位区间半宽与解析 `2 × 1.96 × se` 差 < 2%， 按块重采样（batch_size=1000）也照 docstring 的 Notes 做了。以下是要改的部分。 ## Fixed - `random_state=None` 时另开 `np.random.default_rng()`,**绕过了 `core.get_rng`** —— 与决策 24 冲突：`set_seed` 对它无效，同一段脚本重跑区间就变。改成无条件 `get_rng(random_state)`（决策 46）。 - `metrics["statistic_name"]` 用 `hasattr(statistic, "__name__")` 判不出 lambda： lambda 的 `__name__` 恒为 `"<lambda>"`，于是 `else "custom"` 是**死代码**， 实测自定义统计量拿到的是 `"<lambda>"`。改成 `getattr(f, "__name__", None)` + 判 `"<lambda>"`（易错点 17）。 - docstring 补齐 `Parameters` / `Returns` / `Raises`（§4.7），并如实写明 `method="bca"` **尚未实现** （此前 docstring 承诺支持、实现却抛 `NotImplementedError`）。 - ruff 两条：I001（导入块拆成三行且用了绝对导入）、F541（`f"暂不支持bca"` 无占位符）。 ## Tests - **摘掉 3 条 `skip`**：等 `bootstrap_ci` 的那 3 条一直挂着，实现写完照样全绿 —— 易错点 9 复发， 已在 `personal-pitfalls.md` 把该条 `复发` 记到 1 并注明本次经过。 - 新增 6 条：`estimate` 等于原样本统计量、`confidence` 单调性、五种容器一致、 自定义 lambda 走回退分支、**不给种子时跟随 `set_seed`**（钉住上面那个 rng 修复）、非法/未实现 method。 - 写测试时踩到一个真性质：极差的重采样值**永远 ≤ 原样本极差**，`high` 会正好等于 `estimate`， 断言不能写 `estimate < high`。这条已写进卡片 §5。 - 全量 **177 条：143 passed / 34 skipped / 0 xfail**；ruff 0，pyright 0；`examples/01` 跑通。 ## Docs - `docs/algorithms/bootstrap_ci.md`：**第 2 张 S 档卡片**（模板九节 + §10 待拍板）。 `docs/algorithms/README.md` 进度表打勾。 - `docs/dev/personal-pitfalls.md`：新增第 17 条（`hasattr(f, "__name__")` 挡不住 lambda，`else` 分支即死代码）， 索引补到 17；第 9 条记一次复发。 - `STATUS.md`：E 节记 `bootstrap_ci`（L0 10/12 → 剩余 `monte_carlo` `save_figure`），决策 46， 「下一步」第 1 条改写、新增第 8 条列 5 个待拍板口径。 ## 待用户拍板（Codex 一律未动，按决策 39） - `x` 里的 NaN 静默传进统计量 → `nan` 区间（`confidence_interval` / `t_test` 都是先丢 NaN 的，口径不齐） - `n_resamples <= 0` 现在抛 `IndexError: index -1 is out of bounds`（应在入口给可读 ValueError） - `confidence` 越界交给 numpy 抛 `Percentiles must be in the range [0, 100]` - `method="bca"`：补实现（≈25 行，S 档）还是从 docstring 删掉 - `bootstrap_statistic` 仍是 `NotImplementedError`，而 `bootstrap_ci` 把那段重采样循环内联了 —— 建议抽出去让 `bootstrap_ci` 调它（易错点 16 的变体）
603 lines
21 KiB
Python
603 lines
21 KiB
Python
"""stats 包测试。
|
||
|
||
原则（见 docs/02-architecture.md §5）：**测已知解析解，不测实现细节**。
|
||
|
||
未实现的函数**逐条** `skip`——**不要**用模块级 `pytestmark`：那会把同文件里
|
||
已经写好的函数一起跳过，绿灯就是假的（决策 26 / 个人易错点 9）。
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import numpy as np
|
||
import pandas as pd
|
||
import pytest
|
||
|
||
_NOT_IMPLEMENTED = "未实现：见 docs/03-roadmap.md L1"
|
||
|
||
|
||
def _pearson_p_from_r(r: float, n: int) -> float:
|
||
"""Pearson 相关系数的 p 值**解析式**（测解析解，不测实现细节）：
|
||
|
||
``t = r * sqrt((n - 2) / (1 - r^2))``，``p = 2 * P(T > |t|)``，``T ~ t(n - 2)``。
|
||
"""
|
||
from scipy import stats as sps
|
||
|
||
t_stat = r * np.sqrt((n - 2) / (1 - r**2))
|
||
return float(2 * sps.t.sf(abs(t_stat), n - 2))
|
||
|
||
|
||
def test_bootstrap_ci_matches_normal_theory_on_large_n(rng):
|
||
"""大样本下 bootstrap 百分位区间应逼近正态解析区间（±10% 容差）。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(10, 2, size=5000)
|
||
r = bootstrap_ci(x, statistic=np.mean, confidence=0.95, n_resamples=5000, random_state=0)
|
||
se = x.std(ddof=1) / np.sqrt(x.size)
|
||
assert abs((r.params["high"] - r.params["low"]) - 2 * 1.96 * se) < 0.1 * 2 * 1.96 * se
|
||
|
||
|
||
def test_bootstrap_ci_works_for_arbitrary_statistic(rng):
|
||
"""bootstrap 的核心价值：任意统计量都能给区间（这里用中位数）。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.lognormal(0, 1, size=2000)
|
||
r = bootstrap_ci(x, statistic=np.median, random_state=0)
|
||
assert r.params["low"] < r.params["high"]
|
||
assert r.params["low"] < np.median(x) < r.params["high"]
|
||
|
||
|
||
def test_bootstrap_ci_is_reproducible(rng):
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(size=500)
|
||
a = bootstrap_ci(x, statistic=np.mean, random_state=123)
|
||
b = bootstrap_ci(x, statistic=np.mean, random_state=123)
|
||
assert a.params["low"] == b.params["low"]
|
||
|
||
|
||
def test_bootstrap_ci_reports_the_original_estimate(rng):
|
||
"""`params["estimate"]` 必须是**原样本**上的统计量，不是重采样分布的均值。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(10, 2, size=200)
|
||
r = bootstrap_ci(x, statistic=np.mean, n_resamples=500, random_state=0)
|
||
|
||
assert r.params["estimate"] == pytest.approx(float(np.mean(x)))
|
||
assert set(r.params) == {"estimate", "low", "high", "confidence", "n_resamples"}
|
||
assert r.params["n_resamples"] == 500
|
||
assert r.metrics["statistic_name"] == "mean"
|
||
|
||
|
||
def test_bootstrap_ci_interval_widens_with_confidence(rng):
|
||
"""置信水平越高区间必须越宽 —— 结构性检查，不依赖任何具体分布。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(size=300)
|
||
width = [
|
||
bootstrap_ci(x, np.mean, confidence=c, n_resamples=2000, random_state=7).params["high"]
|
||
- bootstrap_ci(x, np.mean, confidence=c, n_resamples=2000, random_state=7).params["low"]
|
||
for c in (0.80, 0.95)
|
||
]
|
||
assert width[0] < width[1]
|
||
|
||
|
||
def test_bootstrap_ci_accepts_the_documented_containers(rng):
|
||
"""DataFrame 只取数值列后摊平；Series / ndarray / list 都收。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(size=200)
|
||
ref = bootstrap_ci(x, np.mean, n_resamples=500, random_state=0).params
|
||
|
||
for wrapped in (pd.Series(x), pd.DataFrame({"a": x}), list(x), tuple(x)):
|
||
got = bootstrap_ci(wrapped, np.mean, n_resamples=500, random_state=0).params
|
||
assert got["low"] == pytest.approx(ref["low"])
|
||
|
||
|
||
def test_bootstrap_ci_works_for_a_custom_lambda(rng):
|
||
"""不认识的统计量走逐样本回退分支（这里是极差）。"""
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(size=200)
|
||
r = bootstrap_ci(x, statistic=lambda s: float(s.max() - s.min()), n_resamples=500, random_state=0)
|
||
|
||
# 极差的重采样值**永远不会超过原样本极差**（原样本本身就是上界），
|
||
# 所以百分位区间的上界会正好落在 estimate 上，不能再断言 estimate < high。
|
||
assert r.params["estimate"] == pytest.approx(float(x.max() - x.min()))
|
||
assert r.params["low"] > 0
|
||
assert r.params["low"] < r.params["high"] <= r.params["estimate"]
|
||
assert r.metrics["statistic_name"] == "custom"
|
||
|
||
|
||
def test_bootstrap_ci_follows_get_rng_when_no_seed_is_given(rng):
|
||
"""不给种子时必须跟随 `get_rng` 的模块级生成器（决策 24），不能另开一个 default_rng。"""
|
||
from mcm_toolkit.core import set_seed
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
x = rng.normal(size=200)
|
||
set_seed(0)
|
||
first = bootstrap_ci(x, np.mean, n_resamples=300).params["low"]
|
||
set_seed(0)
|
||
second = bootstrap_ci(x, np.mean, n_resamples=300).params["low"]
|
||
|
||
assert first == second
|
||
|
||
|
||
def test_bootstrap_ci_rejects_unknown_method(rng):
|
||
from mcm_toolkit.stats import bootstrap_ci
|
||
|
||
with pytest.raises(ValueError, match="bogus"):
|
||
bootstrap_ci(rng.normal(size=50), np.mean, method="bogus", n_resamples=10)
|
||
with pytest.raises(NotImplementedError):
|
||
bootstrap_ci(rng.normal(size=50), np.mean, method="bca", n_resamples=10)
|
||
|
||
|
||
def test_confidence_interval_matches_scipy(rng):
|
||
from scipy import stats
|
||
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
x = rng.normal(5, 1, size=50)
|
||
lo, hi = confidence_interval(x, confidence=0.95, method="t")
|
||
ref = stats.t.interval(0.95, len(x) - 1, loc=x.mean(), scale=stats.sem(x))
|
||
assert np.allclose([lo, hi], ref, rtol=1e-6)
|
||
|
||
|
||
def test_confidence_interval_normal_matches_z_formula(rng):
|
||
"""normal 走 z 解析式：`mean ± z·s/√n`，s 用**样本**标准差（ddof=1）。"""
|
||
from scipy import stats as sps
|
||
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
x = rng.normal(8.5, 1, size=200)
|
||
lo, hi = confidence_interval(x, confidence=0.95, method="normal")
|
||
|
||
half = float(sps.norm.ppf(0.975)) * x.std(ddof=1) / np.sqrt(len(x))
|
||
assert lo == pytest.approx(x.mean() - half)
|
||
assert hi == pytest.approx(x.mean() + half)
|
||
|
||
|
||
def test_confidence_interval_mean_is_not_clamped_to_unit_range(rng):
|
||
"""均值区间不该被夹到 [0, 1]：平均身高 170cm 的下界不可能是 0。"""
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
x = rng.normal(170, 8, size=50)
|
||
lo, hi = confidence_interval(x, method="t")
|
||
|
||
assert lo > 100.0
|
||
assert lo < hi
|
||
|
||
|
||
def test_confidence_interval_proportion_stays_in_unit_interval():
|
||
"""比例的区间必须落在 [0, 1] 里 —— 只有这一种 method 需要夹。"""
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
y = [0.0, 1.0, 1.0, 1.0, 0.0, 1.0]
|
||
lo, hi = confidence_interval(y, method="proportion")
|
||
|
||
assert 0.0 <= lo < hi <= 1.0
|
||
|
||
|
||
def test_confidence_interval_accepts_documented_containers():
|
||
"""docstring 的 Examples 传 list、L0 用例传 ndarray，两边都得收。"""
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
data = [1.0, 2.0, 3.0, 4.0, 5.0]
|
||
expected = confidence_interval(data)
|
||
|
||
assert confidence_interval(pd.Series(data)) == pytest.approx(expected)
|
||
assert confidence_interval(np.asarray(data)) == pytest.approx(expected)
|
||
assert confidence_interval(pd.DataFrame({"x": data})) == pytest.approx(expected)
|
||
|
||
|
||
def test_confidence_interval_rejects_bad_input(rng):
|
||
from mcm_toolkit.stats import confidence_interval
|
||
|
||
x = rng.normal(size=20)
|
||
with pytest.raises(TypeError):
|
||
confidence_interval("abc")
|
||
with pytest.raises(ValueError):
|
||
confidence_interval(x, method="bogus")
|
||
with pytest.raises(ValueError):
|
||
confidence_interval(x, confidence=1.5)
|
||
with pytest.raises(ValueError):
|
||
confidence_interval(x, confidence=0.0)
|
||
with pytest.raises(ValueError):
|
||
confidence_interval(pd.DataFrame({"city": ["X", "Y"]}))
|
||
with pytest.raises(ValueError):
|
||
confidence_interval([1.0])
|
||
|
||
|
||
# ------------------------------------------------------ stats: t_test
|
||
|
||
|
||
def test_t_test_one_sample_matches_analytic_formula(rng):
|
||
"""单样本：`t = (mean - value) / (s/√n)`，`df = n - 1`，`p = 2·P(T > |t|)`。"""
|
||
from scipy import stats as sps
|
||
|
||
from mcm_toolkit.stats import t_test
|
||
|
||
x = rng.normal(5, 1, size=30)
|
||
r = t_test(x, value=5.0)
|
||
|
||
t_stat = (x.mean() - 5.0) / (x.std(ddof=1) / np.sqrt(len(x)))
|
||
assert r["statistic"] == pytest.approx(t_stat)
|
||
assert r["pvalue"] == pytest.approx(float(2 * sps.t.sf(abs(t_stat), len(x) - 1)))
|
||
assert r["df"] == pytest.approx(29.0)
|
||
assert r["significant"] == (r["pvalue"] < 0.05)
|
||
assert isinstance(r["conclusion"], str) and r["conclusion"]
|
||
|
||
|
||
def test_t_test_two_sample_welch_df_matches_formula(rng):
|
||
"""默认 Welch：`df` 用 Welch–Satterthwaite 公式解析对照。"""
|
||
from mcm_toolkit.stats import t_test
|
||
|
||
a = rng.normal(0, 1, size=25)
|
||
b = rng.normal(1, 4, size=35)
|
||
r = t_test(a, b)
|
||
|
||
v1, v2 = a.var(ddof=1), b.var(ddof=1)
|
||
n1, n2 = len(a), len(b)
|
||
expected_df = (v1 / n1 + v2 / n2) ** 2 / ((v1 / n1) ** 2 / (n1 - 1) + (v2 / n2) ** 2 / (n2 - 1))
|
||
assert r["df"] == pytest.approx(expected_df)
|
||
assert r["pvalue"] < 0.05
|
||
|
||
|
||
def test_t_test_equal_var_uses_pooled_df(rng):
|
||
from mcm_toolkit.stats import t_test
|
||
|
||
a = rng.normal(0, 1, size=25)
|
||
b = rng.normal(1, 1, size=35)
|
||
|
||
assert t_test(a, b, equal_var=True)["df"] == pytest.approx(58.0)
|
||
|
||
|
||
def test_t_test_paired_reports_key_set_and_df(rng):
|
||
from mcm_toolkit.stats import t_test
|
||
|
||
a = rng.normal(size=20)
|
||
r = t_test(a, a + rng.normal(0, 0.1, 20), paired=True)
|
||
|
||
assert set(r) == {"statistic", "pvalue", "df", "significant", "conclusion"}
|
||
assert r["df"] == pytest.approx(19.0)
|
||
|
||
|
||
def test_t_test_rejects_bad_input(rng):
|
||
from mcm_toolkit.stats import t_test
|
||
|
||
a = rng.normal(size=20)
|
||
with pytest.raises(ValueError):
|
||
t_test(a, alpha=1.5)
|
||
with pytest.raises(ValueError):
|
||
t_test([1.0])
|
||
with pytest.raises(ValueError):
|
||
t_test(a, rng.normal(size=19), paired=True)
|
||
with pytest.raises(ValueError):
|
||
t_test(pd.DataFrame({"city": ["X", "Y"]}))
|
||
with pytest.raises(TypeError):
|
||
t_test("abc")
|
||
|
||
|
||
def test_correlation_matrix_diagonal_is_one(rng):
|
||
from mcm_toolkit.stats import correlation_matrix
|
||
|
||
x = pd.DataFrame(rng.normal(size=(200, 4)), columns=["a", "b", "c", "d"])
|
||
c = correlation_matrix(x)
|
||
|
||
assert np.allclose(np.diag(np.asarray(c)), 1.0)
|
||
|
||
|
||
def test_correlation_matrix_is_numeric_and_symmetric(rng):
|
||
"""方案 A：返回**数值**矩阵（不是带星标的字符串表），能直接参与后续计算。"""
|
||
from mcm_toolkit.stats import correlation_matrix
|
||
|
||
x = pd.DataFrame(rng.normal(size=(200, 3)), columns=["a", "b", "c"])
|
||
c = correlation_matrix(x)
|
||
|
||
assert all(np.issubdtype(dtype, np.floating) for dtype in c.dtypes)
|
||
assert np.allclose(np.asarray(c), np.asarray(x.corr(method="pearson", min_periods=1)))
|
||
assert np.allclose(np.asarray(c), np.asarray(c.T))
|
||
|
||
|
||
def test_correlation_matrix_runs_on_multicolumn_frame(rng):
|
||
"""回归：`if numeric_cols == 0` 拿 Index 跟 0 比，多列数值表当场 ValueError 崩掉。"""
|
||
from mcm_toolkit.stats import correlation_matrix
|
||
|
||
x = pd.DataFrame(rng.normal(size=(50, 3)), columns=["a", "b", "c"])
|
||
out = correlation_matrix(x)
|
||
|
||
assert out.shape == (3, 3)
|
||
assert list(out.columns) == ["a", "b", "c"]
|
||
assert list(out.index) == ["a", "b", "c"]
|
||
|
||
|
||
def test_correlation_matrix_skips_non_numeric_columns(rng):
|
||
from mcm_toolkit.stats import correlation_matrix
|
||
|
||
x = pd.DataFrame({"a": rng.normal(size=30), "b": rng.normal(size=30), "city": ["X"] * 30})
|
||
assert list(correlation_matrix(x).columns) == ["a", "b"]
|
||
|
||
|
||
def test_correlation_matrix_rejects_bad_input():
|
||
from mcm_toolkit.stats import correlation_matrix
|
||
|
||
with pytest.raises(TypeError):
|
||
correlation_matrix([[1.0, 2.0]])
|
||
with pytest.raises(ValueError):
|
||
correlation_matrix(pd.DataFrame({"city": ["X", "Y"]}))
|
||
with pytest.raises(ValueError):
|
||
correlation_matrix(pd.DataFrame({"a": [1.0, 2.0]}), method="bogus") # type: ignore[arg-type]
|
||
|
||
|
||
def test_correlation_pvalues_matches_analytic_formula(rng):
|
||
from mcm_toolkit.stats import correlation_matrix, correlation_pvalues
|
||
|
||
x = pd.DataFrame(rng.normal(size=(80, 3)), columns=["a", "b", "c"])
|
||
corr = correlation_matrix(x)
|
||
p = correlation_pvalues(x)
|
||
|
||
for i in ["a", "b", "c"]:
|
||
for j in ["a", "b", "c"]:
|
||
if i == j:
|
||
continue
|
||
assert p.loc[i, j] == pytest.approx(_pearson_p_from_r(corr.loc[i, j], 80))
|
||
|
||
|
||
def test_correlation_pvalues_diagonal_is_nan(rng):
|
||
"""对角线写 0 会被下游读成"极显著"；"自己和自己相关"没有 p 值，必须是 NaN。"""
|
||
from mcm_toolkit.stats import correlation_pvalues
|
||
|
||
p = correlation_pvalues(pd.DataFrame(rng.normal(size=(50, 2)), columns=["a", "b"]))
|
||
|
||
assert np.isnan(p.loc["a", "a"])
|
||
assert np.isnan(p.loc["b", "b"])
|
||
|
||
|
||
def test_correlation_pvalues_uses_the_same_observations_as_r(rng):
|
||
"""含缺失时 p 要和 r 用**同一批**观测：成对删缺失，而不是整行删、也不是干脆不删。"""
|
||
from mcm_toolkit.stats import correlation_pvalues
|
||
|
||
a = rng.normal(size=40)
|
||
b = rng.normal(size=40)
|
||
a[[3, 7, 11]] = np.nan
|
||
x = pd.DataFrame({"a": a, "b": b})
|
||
|
||
pair = x.dropna()
|
||
r = float(np.corrcoef(pair["a"], pair["b"])[0, 1])
|
||
|
||
assert correlation_pvalues(x).loc["a", "b"] == pytest.approx(_pearson_p_from_r(r, len(pair)))
|
||
|
||
|
||
def test_correlation_pvalues_respects_min_periods(rng):
|
||
from mcm_toolkit.stats import correlation_pvalues
|
||
|
||
a = rng.normal(size=10)
|
||
a[:5] = np.nan
|
||
x = pd.DataFrame({"a": a, "b": rng.normal(size=10)})
|
||
|
||
assert np.isnan(correlation_pvalues(x, min_periods=6).loc["a", "b"])
|
||
assert np.isfinite(correlation_pvalues(x, min_periods=5).loc["a", "b"])
|
||
|
||
|
||
def test_correlation_table_diagonal_has_no_stars():
|
||
"""对角线是"自己和自己"，不该有显著性星标；完全负相关则必然是 ***。"""
|
||
from mcm_toolkit.stats import correlation_table
|
||
|
||
x = pd.DataFrame({"a": np.arange(30.0), "b": np.arange(30.0) * -2.0})
|
||
table = correlation_table(x)
|
||
|
||
assert table.loc["a", "a"] == "1.000"
|
||
assert table.loc["a", "b"] == "-1.000***"
|
||
|
||
|
||
def test_correlation_table_matches_the_numeric_outputs(rng):
|
||
from mcm_toolkit.stats import correlation_matrix, correlation_pvalues, correlation_table
|
||
|
||
x = pd.DataFrame(rng.normal(size=(60, 3)), columns=["a", "b", "c"])
|
||
corr = correlation_matrix(x)
|
||
p = correlation_pvalues(x)
|
||
table = correlation_table(x)
|
||
|
||
assert list(table.columns) == list(corr.columns)
|
||
assert table.loc["a", "a"] == "1.000"
|
||
assert np.isnan(p.loc["a", "a"])
|
||
|
||
pv = p.loc["a", "b"]
|
||
stars = "***" if pv < 0.001 else "**" if pv < 0.01 else "*" if pv < 0.05 else ""
|
||
value = corr.loc["a", "b"]
|
||
assert table.loc["a", "b"] == f"{value:.3f}" + stars
|
||
|
||
|
||
# ------------------------------------------------------ stats: describe
|
||
|
||
|
||
def test_describe_reports_expected_keys(rng):
|
||
from mcm_toolkit.stats import describe
|
||
|
||
table = describe(rng.normal(size=(100, 3)))
|
||
for key in ["mean", "std", "min", "max", "median"]:
|
||
assert key in table.index
|
||
|
||
|
||
def test_describe_rows_are_metrics_and_columns_are_variables():
|
||
"""方向约定：行 = 指标，列 = 变量。多转置一次就反了。"""
|
||
from mcm_toolkit.stats import describe
|
||
|
||
df = pd.DataFrame({"a": [1.0, 2.0, 3.0], "b": [10.0, 20.0, 30.0]})
|
||
table = describe(df)
|
||
|
||
assert list(table.columns) == ["a", "b"]
|
||
assert table.loc["mean", "a"] == pytest.approx(2.0)
|
||
assert table.loc["max", "b"] == pytest.approx(30.0)
|
||
assert table.loc["median", "b"] == pytest.approx(20.0)
|
||
|
||
|
||
def test_describe_missing_and_shape_rows_are_not_nan():
|
||
"""曾经的静默 bug：转置后 `.loc[指标名] = <按变量索引的 Series>` 对不上列，
|
||
整行会被填成 NaN —— 不报错，但数据全丢。"""
|
||
from mcm_toolkit.stats import describe
|
||
|
||
df = pd.DataFrame({"a": [1.0, np.nan, 3.0, 4.0], "b": [1.0, 2.0, 3.0, 4.0]})
|
||
table = describe(df)
|
||
|
||
assert table.loc["missing", "a"] == pytest.approx(1.0)
|
||
assert table.loc["missing", "b"] == pytest.approx(0.0)
|
||
assert table.loc["missing_ratio", "a"] == pytest.approx(0.25)
|
||
assert table.loc["missing_ratio", "b"] == pytest.approx(0.0)
|
||
assert table.loc["skewness", "b"] == pytest.approx(0.0, abs=1e-12)
|
||
extra = table.loc[["missing", "missing_ratio", "skewness", "kurtosis"]]
|
||
assert not bool(extra.isna().all(axis=None))
|
||
|
||
|
||
def test_describe_skips_non_numeric_columns():
|
||
from mcm_toolkit.stats import describe
|
||
|
||
df = pd.DataFrame({"a": [1.0, 2.0], "city": ["X", "Y"]})
|
||
table = describe(df)
|
||
|
||
assert list(table.columns) == ["a"]
|
||
|
||
|
||
def test_describe_accepts_ndarray_list_and_series(rng):
|
||
"""兄弟函数都收 ndarray / list，describe 没理由只收 DataFrame。"""
|
||
from mcm_toolkit.stats import describe
|
||
|
||
assert "mean" in describe(rng.normal(size=(50, 2))).index
|
||
assert "mean" in describe([[1.0, 2.0], [3.0, 4.0]]).index
|
||
assert "mean" in describe(pd.Series([1.0, 2.0, 3.0])).index
|
||
|
||
|
||
def test_describe_weighted_rows_match_manual_formula():
|
||
from mcm_toolkit.stats import describe
|
||
|
||
df = pd.DataFrame({"a": [1.0, 2.0, 3.0]})
|
||
w = np.array([1.0, 1.0, 2.0])
|
||
table = describe(df, weights=w)
|
||
|
||
expect_mean = np.average(df["a"], weights=w)
|
||
expect_std = np.sqrt(np.average((df["a"] - expect_mean) ** 2, weights=w))
|
||
assert table.loc["weighted_mean", "a"] == pytest.approx(expect_mean)
|
||
assert table.loc["weighted_std", "a"] == pytest.approx(expect_std)
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("weights", "exc"),
|
||
[
|
||
(np.ones(2), ValueError),
|
||
(np.zeros(3), ValueError),
|
||
(np.array([1.0, -1.0, 1.0]), ValueError),
|
||
(np.array([1.0, np.nan, 1.0]), ValueError),
|
||
("abc", TypeError),
|
||
],
|
||
)
|
||
def test_describe_rejects_bad_weights(weights, exc):
|
||
from mcm_toolkit.stats import describe
|
||
|
||
with pytest.raises(exc):
|
||
describe(pd.DataFrame({"a": [1.0, 2.0, 3.0]}), weights=weights)
|
||
|
||
|
||
def test_describe_rejects_unsupported_type_and_all_text_frame():
|
||
from mcm_toolkit.stats import describe
|
||
|
||
with pytest.raises(TypeError):
|
||
describe({"a": [1.0]})
|
||
with pytest.raises(ValueError):
|
||
describe(pd.DataFrame({"city": ["X", "Y"]}))
|
||
|
||
|
||
# ------------------------------------------------------ stats: zscore
|
||
|
||
|
||
def test_zscore_matches_manual_formula():
|
||
from mcm_toolkit.stats import zscore
|
||
|
||
x = np.array([1.0, 2.0, 3.0, 4.0])
|
||
assert np.allclose(np.asarray(zscore(x)), (x - x.mean()) / x.std())
|
||
|
||
|
||
def test_zscore_is_per_column_and_keeps_frame(rng):
|
||
from mcm_toolkit.stats import zscore
|
||
|
||
df = pd.DataFrame({"a": rng.normal(size=200), "b": rng.normal(5, 3, size=200)})
|
||
out = zscore(df)
|
||
|
||
assert isinstance(out, pd.DataFrame)
|
||
assert list(out.columns) == ["a", "b"]
|
||
assert np.allclose(np.asarray(out).mean(axis=0), 0.0, atol=1e-12)
|
||
assert np.allclose(np.asarray(out).std(axis=0), 1.0, atol=1e-12)
|
||
|
||
|
||
def test_zscore_constant_column_stays_zero():
|
||
from mcm_toolkit.stats import zscore
|
||
|
||
assert np.allclose(np.asarray(zscore([2.0, 2.0, 2.0])), 0.0)
|
||
|
||
|
||
def test_zscore_series_keeps_index_and_name(rng):
|
||
"""Series 进 Series 出，索引和 name 都要带回来 —— 否则管道里 concat 会丢列名。"""
|
||
from mcm_toolkit.stats import zscore
|
||
|
||
s = pd.Series(rng.normal(size=50), index=[f"r{i}" for i in range(50)], name="price")
|
||
out = zscore(s)
|
||
|
||
assert isinstance(out, pd.Series)
|
||
assert out.name == "price"
|
||
assert list(out.index) == list(s.index)
|
||
|
||
|
||
def test_zscore_rejects_bad_type_and_all_text_frame():
|
||
from mcm_toolkit.stats import zscore
|
||
|
||
with pytest.raises(TypeError):
|
||
zscore("abc")
|
||
with pytest.raises(ValueError):
|
||
zscore(pd.DataFrame({"city": ["X"]}))
|
||
|
||
|
||
# ------------------------------------------- stats: coefficient_of_variation
|
||
|
||
|
||
def test_coefficient_of_variation_matches_manual_formula():
|
||
from mcm_toolkit.stats import coefficient_of_variation
|
||
|
||
x = np.array([1.0, 2.0, 3.0, 4.0])
|
||
assert coefficient_of_variation(x) == pytest.approx(x.std() / x.mean())
|
||
|
||
|
||
def test_coefficient_of_variation_accepts_single_column_frame():
|
||
from mcm_toolkit.stats import coefficient_of_variation
|
||
|
||
df = pd.DataFrame({"a": [1.0, 2.0, 3.0]})
|
||
single = coefficient_of_variation([1.0, 2.0, 3.0])
|
||
assert coefficient_of_variation(df) == pytest.approx(single)
|
||
|
||
|
||
def test_coefficient_of_variation_uses_absolute_mean():
|
||
"""均值为负时 CV 仍是**正数**：符号不该被读成"波动更小"。"""
|
||
from mcm_toolkit.stats import coefficient_of_variation
|
||
|
||
x = np.array([-1.0, -2.0, -3.0, -4.0])
|
||
assert coefficient_of_variation(x) == pytest.approx(x.std() / abs(x.mean()))
|
||
assert coefficient_of_variation(x) > 0
|
||
|
||
|
||
def test_coefficient_of_variation_zero_mean_is_value_error():
|
||
"""均值为 0 是**取值**问题不是类型问题：必须是 ValueError 而不是 TypeError。"""
|
||
from mcm_toolkit.stats import coefficient_of_variation
|
||
|
||
with pytest.raises(ValueError):
|
||
coefficient_of_variation([-1.0, 1.0])
|
||
|
||
|
||
def test_coefficient_of_variation_rejects_empty_2d_and_bad_type():
|
||
from mcm_toolkit.stats import coefficient_of_variation
|
||
|
||
with pytest.raises(ValueError):
|
||
coefficient_of_variation([])
|
||
with pytest.raises(ValueError):
|
||
coefficient_of_variation([[1.0, 2.0], [3.0, 4.0]])
|
||
with pytest.raises(ValueError):
|
||
coefficient_of_variation(pd.DataFrame({"a": [1.0], "b": [2.0]}))
|
||
with pytest.raises(TypeError):
|
||
coefficient_of_variation(5.0)
|