How to assess if there is a strong signal in your dirty data [Project]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
GitHub 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一款名为 Entropic Scree 的新型表格数据诊断工具已推出，旨在评估脏乱、高维度的真实世界数据集中的强信号。该工具可用于估计此类数据集的各种属性。提供了快速入门指南，包括用于从 GitHub 加载脚本、下载并将其导入 R 环境的 R 函数代码。示例展示了如何运行 Entropic.Scree 函数并提取双极模块，然后查看提取的结构子网络。
I'm sharing this new tabular data diagnostic tool (Entropic Scree). It can be used to estimate these properties of your high-d, real-world, dirty dataset:
- The informational volume of the signal (i.e., helps you assess whether the signal is strong enough to survive the dataset's idiosyncratic volume).
- The overall signal-to-idiosyncratic volume ratio (SNR).
- The intrinsic rank.
- Provides an exploratory map that allows for the identification of decoupled sub-networks of variables.
- The linear sufficiency (i.e., does the dataset align with the linear assumptions of standard PCA?).
Instead of evaluating linear variance, rank order, or Euclidean distance like traditional PCA variants, this new method evaluates a transformed mutual information metric. Relative to these baselines, it is less reliant on strong parametric or distance assumptions, making it appropriate to apply more broadly.
It also serves as a practical diagnostic of the theory explored in the From Garbage to Gold framework, which describes when and why uncurated, error-prone data can be used directly to create accurate prediction models.
There is a preprint that presents the full technical details, and Python and R packages will be released soon. Right now though, the original function is already available in R (see Quick Start R Function Code below).
- Entropic Scree Preprint: https://doi.org/10.5281/zenodo.22028087
- Entropic Scree GitHub: https://github.com/tjleestjohn/entropic-scree
- From Garbage to Gold Preprint: https://arxiv.org/abs/2603.12288
Let me know how it goes if you give it a try... or if you have any questions or comments of course.
############ # Quick Start R Function Code. # To load the function, copy and paste the following into your R console, then hit enter. ############ # 1. Define the direct URL to the raw function script on GitHub url <- "https://raw.githubusercontent.com/tjleestjohn/entropic-scree/main/Entropic.Scree.v1.0.0%20-%20ENLI.R" # 2. Define what you want to name the file on your computer file_name <- "Entropic.Scree.v1.0.0 - ENLI.R" # 3. Download the script to your current working directory download.file(url, destfile = file_name) # 4. Source the core function into your R environment source(file_name) # 5. Ex. To run the function and extract bipolar modules: # results <- Entropic.Scree(dt # , extract_bipolar_modules = TRUE) # # View the extracted structural sub-networks for the primary axes: # results$bipolar_modules