Remove duplicate URLs from any list
Normalize and deduplicate crawl exports, copied links and URL datasets. See duplicate counts instantly while every address remains in your browser.
清洁选项
干净的网址
2 跟踪参数已删除
仅将此实用程序用于您有权评估的数据和系统。设置 URL 列表格式不会授予测试其目标的权限。
一个 URL 列表的六种输出格式
切换工作区输出而不编辑输入。每种格式都会自动标准化和重复数据删除。
干净的网址
小写方案和主机名、排序查询字符串、无片段、凭据或重复行。
域名
将混合列表折叠成一个唯一的主机名清单，为范围审查和资产列表做好准备。
端点
保留原点和路径，删除每个查询字符串，因此相同的端点仅列出一次。
路径
跨主机提取路由结构以比较环境或映射应用程序表面。
参数
无需编写任何正则表达式，即可将 URL 列表转换为去重查询关键词列表。
模糊模板
将每个参数值替换为常见授权测试工作流程使用的 FUZZ 占位符。
永远看不到您的列表的 URL 实用程序
无需帐户、无需上传，也不会请求您粘贴的域名——只在您的浏览器标签页中处理文本。
100%
本地加工
每个 URL 都会由您的浏览器进行解析。没有任何内容上传到 DorkPlus，也没有联系粘贴的域。
01
6
输出格式
从单个输入中清理 URL、域、端点、路径、参数单词列表和 FUZZ 模板。
02
0
限额和账户
无需登录，无上传上限，无需排队。粘贴列表，选择输出并立即复制或导出。
03How to deduplicate a URL list
Standardize superficial URL differences first, then return each remaining normalized address exactly once.
Paste the complete list
Add repeated URLs one per line or paste text containing links. The workspace counts valid, invalid and duplicate outputs.
https://example.com/?b=2&a=1
https://EXAMPLE.com/?a=1&b=2Set the comparison rules
Choose whether tracking keys, fragments, credentials and trailing slashes should be removed before URLs are compared.
Review the unique output
Exact normalized matches collapse to one line. The duplicate counter shows how many repeated output lines were removed.
https://example.com/?a=1&b=2Copy or export the list
Download the deduplicated URLs as text or copy them into the next authorized workflow.
自动处理
Why normalize before removing duplicates?
String-only deduplication misses URLs that differ in hostname capitalization, parameter order or removable campaign tags. Normalization exposes those equivalent representations before comparison.
Two URLs are not guaranteed to serve the same response merely because they look similar. The optional controls let you choose the definition of a duplicate that fits your dataset.
- Count removed duplicate outputs
- Control normalization before comparison
- Preserve one stable URL per exact result
常见用例
Crawler exports
Reduce repeat URLs before a second processing pass.
Content inventories
Remove formatting variants from a page list.
Redirect migrations
Prepare stable old and new URL sets for comparison.
Research datasets
Save one normalized copy of each collected link.
URL deduplicator questions
Answers about duplicate detection, normalization choices and privacy.
需要大规模发现和处理 URL？
DorkPlus 将多引擎搜索解析、爬行、分类和授权漏洞评估结合在一个工作流程中。
DorkPlus 已经存在三年多。它最初只是一个 Google 无代理采集器，后来逐步发展为一体化渗透测试解决方案，并且每周都在持续进化。
Copyright © 2026 DorkPlus
所有系统运行正常