Skip to content

Commit 9e5c2ed

Browse files
titanismtitanism
committed
feat: added daily tangerine benchmarks, support node v24+ type property
1 parent d338a8d commit 9e5c2ed

File tree

5 files changed

+409
-183
lines changed

5 files changed

+409
-183
lines changed

‎.github/workflows/daily-benchmarks.yml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
name: Daily Tangerine Benchmarks
2+
3+
on:
4+
schedule:
5+
# Run daily at 2:00 AM UTC
6+
- cron: '0 2 * * *'
7+
workflow_dispatch: # Allow manual triggering
8+
push:
9+
branches:
10+
- main
11+
paths:
12+
- 'benchmarks/**'
13+
- '.github/workflows/daily-benchmarks.yml'
14+
15+
permissions:
16+
contents: write
17+
18+
jobs:
19+
benchmark:
20+
name: Run Benchmarks on Node ${{ matrix.node-version }}
21+
runs-on: ubuntu-latest
22+
strategy:
23+
matrix:
24+
node-version: [18, 20, 22, 24, 'latest']
25+
fail-fast: false
26+
27+
steps:
28+
- name: Checkout repository
29+
uses: actions/checkout@v4
30+
31+
- name: Setup Node.js ${{ matrix.node-version }}
32+
uses: actions/setup-node@v4
33+
with:
34+
node-version: ${{ matrix.node-version }}
35+
36+
- name: Install dependencies
37+
run: npm install
38+
39+
- name: Run benchmarks
40+
run: npm run benchmarks -- --json > benchmark_results_node_${{ matrix.node-version }}.json 2>&1 || true
41+
continue-on-error: true
42+
43+
- name: Run benchmarks (fallback to text output)
44+
run: |
45+
node -e "
46+
const { execSync } = require('child_process');
47+
const fs = require('fs');
48+
const version = process.version;
49+
50+
const results = {
51+
node_version: version,
52+
platform: process.platform,
53+
arch: process.arch,
54+
timestamp: new Date().toISOString(),
55+
benchmarks: {}
56+
};
57+
58+
// Run each benchmark and capture output
59+
const benchmarks = ['lookup', 'resolve', 'reverse'];
60+
for (const bench of benchmarks) {
61+
try {
62+
const output = execSync(\`node benchmarks/\${bench}\`, {
63+
encoding: 'utf8',
64+
timeout: 300000
65+
});
66+
results.benchmarks[bench] = output;
67+
} catch (err) {
68+
results.benchmarks[bench] = err.message;
69+
}
70+
}
71+
72+
fs.writeFileSync(
73+
\`benchmark_results_node_\${version}.json\`,
74+
JSON.stringify(results, null, 2)
75+
);
76+
console.log('Benchmark results saved for Node.js', version);
77+
"
78+
continue-on-error: true
79+
80+
- name: Upload benchmark results
81+
uses: actions/upload-artifact@v4
82+
with:
83+
name: benchmark-results-node-${{ matrix.node-version }}
84+
path: benchmark_results_node_*.json
85+
if-no-files-found: warn
86+
87+
consolidate:
88+
name: Consolidate Results and Update README
89+
needs: benchmark
90+
runs-on: ubuntu-latest
91+
92+
steps:
93+
- name: Checkout repository
94+
uses: actions/checkout@v4
95+
with:
96+
fetch-depth: 0
97+
98+
- name: Setup Node.js 20
99+
uses: actions/setup-node@v4
100+
with:
101+
node-version: 20
102+
103+
- name: Download all benchmark results
104+
uses: actions/download-artifact@v4
105+
with:
106+
path: benchmark-artifacts
107+
108+
- name: Consolidate benchmark results
109+
run: |
110+
# Move all JSON files to root directory
111+
find benchmark-artifacts -name "*.json" -exec cp {} . \;
112+
113+
# List all result files
114+
ls -la benchmark_results_*.json || echo "No benchmark result files found"
115+
116+
- name: Install dependencies for update script
117+
run: npm install
118+
119+
- name: Generate updated README
120+
run: node scripts/update-readme.js
121+
122+
- name: Configure Git
123+
run: |
124+
git config --local user.email "github-actions[bot]@users.noreply.github.com"
125+
git config --local user.name "github-actions[bot]"
126+
127+
- name: Commit and push changes
128+
run: |
129+
git add benchmark_results_*.json README.md
130+
if git diff --staged --quiet; then
131+
echo "No changes to commit"
132+
else
133+
git commit -m "chore: update benchmark results [skip ci]"
134+
git push
135+
fi
136+
env:
137+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎README.md‎

Lines changed: 18 additions & 155 deletions
Original file line numberDiff line numberDiff line change
@@ -471,6 +471,19 @@ This purge cache feature is useful for DNS records that have recently changed an
471471
472472
## Compatibility
473473
474+
> \[!NOTE]
475+
> **Node.js v24+ DNS Record Type Property**
476+
>
477+
> Starting with Node.js v24, the native DNS resolver adds a `type` property to certain DNS record objects (MX, CAA, SRV, SOA, and NAPTR records). Tangerine automatically includes this property when running on Node.js v24+ to maintain 1:1 compatibility with the native `dns` module. For example:
478+
>
479+
> ```js
480+
> // Node.js v22 and earlier
481+
> { exchange: 'smtp.google.com', priority: 10 }
482+
>
483+
> // Node.js v24+
484+
> { exchange: 'smtp.google.com', priority: 10, type: 'MX' }
485+
> ```
486+
474487
The only known compatibility issue is for locally running DNS servers that have wildcard DNS matching.
475488
476489
If you are using `dnsmasq` with a wildcard match on "localhost" to "127.0.0.1", then the results may vary. For example, if your `dnsmasq` configuration has `address=/localhost/127.0.0.1`, then any match of `localhost` will resolve to `127.0.0.1`. This means that `dns.promises.lookup('foo.localhost')` will return `127.0.0.1` – however with :tangerine: Tangerine it will not return a value.
@@ -514,170 +527,20 @@ BENCHMARK_PROTOCOL="http" BENCHMARK_HOST="127.0.0.1" BENCHMARK_PORT="4000" BENCH
514527
515528
We have written extensive benchmarks to show that :tangerine: Tangerine is as fast as the native Node.js DNS module (with the exception of the `lookup` command). Note that performance is opinionated – since rate limiting plays a factor dependent on the DNS servers you are using and since caching is most likely going to takeover.
516529
517-
The latest benchmark results are viewable on GitHub under this repository's [GitHub CI actions logs](https://github.com/forwardemail/nodejs-dns-over-https-tangerine/actions?query=event%3Apush):
518-
519-
> [Node 16 on ubuntu-latest](https://github.com/forwardemail/nodejs-dns-over-https-tangerine/actions/runs/4297805550/jobs/7491228635#step:6:1)
520-
521-
```diff
522-
node benchmarks/lookup && node benchmarks/resolve && node benchmarks/reverse
523-
524-
Started: lookup
525-
tangerine.lookup POST with caching using Cloudflare x 735 ops/sec ±195.35% (88 runs sampled)
526-
tangerine.lookup POST without caching using Cloudflare x 142 ops/sec ±0.58% (84 runs sampled)
527-
tangerine.lookup GET with caching using Cloudflare x 222,397 ops/sec ±0.52% (88 runs sampled)
528-
+tangerine.lookup GET without caching using Cloudflare x 142 ops/sec ±0.46% (83 runs sampled)
529-
dns.promises.lookup with caching using Cloudflare x 6,169,417 ops/sec ±1.67% (84 runs sampled)
530-
-dns.promises.lookup without caching using Cloudflare x 4,186 ops/sec ±0.58% (89 runs sampled)
531-
Fastest without caching is: dns.promises.lookup without caching using Cloudflare
532-
533-
Started: resolve
534-
tangerine.resolve POST with caching using Cloudflare x 951 ops/sec ±195.84% (87 runs sampled)
535-
tangerine.resolve POST without caching using Cloudflare x 135 ops/sec ±1.27% (79 runs sampled)
536-
tangerine.resolve GET with caching using Cloudflare x 1,134,724 ops/sec ±0.27% (87 runs sampled)
537-
+tangerine.resolve GET without caching using Cloudflare x 135 ops/sec ±1.34% (81 runs sampled)
538-
tangerine.resolve POST with caching using Google x 1,103,189 ops/sec ±0.44% (86 runs sampled)
539-
tangerine.resolve POST without caching using Google x 55.76 ops/sec ±3.57% (80 runs sampled)
540-
tangerine.resolve GET with caching using Google x 1,140,499 ops/sec ±0.32% (87 runs sampled)
541-
tangerine.resolve GET without caching using Google x 70.51 ops/sec ±0.93% (84 runs sampled)
542-
resolver.resolve with caching using Cloudflare x 4,790,171 ops/sec ±0.43% (87 runs sampled)
543-
-resolver.resolve without caching using Cloudflare x 158 ops/sec ±1.26% (83 runs sampled)
544-
Fastest without caching is: resolver.resolve without caching using Cloudflare
545-
546-
Started: reverse
547-
tangerine.reverse GET with caching x 771 ops/sec ±195.37% (85 runs sampled)
548-
+tangerine.reverse GET without caching x 135 ops/sec ±0.74% (81 runs sampled)
549-
resolver.reverse with caching x 5,353,130 ops/sec ±0.36% (89 runs sampled)
550-
-resolver.reverse without caching x 1.90 ops/sec ±210.52% (16 runs sampled)
551-
dns.promises.reverse with caching x 5,123,900 ops/sec ±0.96% (85 runs sampled)
552-
-dns.promises.reverse without caching x 0.29 ops/sec ±171.85% (18 runs sampled)
553-
+Fastest without caching is: tangerine.reverse GET without caching
554-
```
530+
---
555531
556-
> [Node 18 on ubuntu latest](https://github.com/forwardemail/nodejs-dns-over-https-tangerine/actions/runs/4297805550/jobs/7491228742#step:6:1)
532+
<!-- BENCHMARK_RESULTS_START -->
557533
558-
```diff
559-
node benchmarks/lookup && node benchmarks/resolve && node benchmarks/reverse && node benchmarks/http
560-
561-
Started: lookup
562-
tangerine.lookup POST with caching using Cloudflare x 666 ops/sec ±195.48% (87 runs sampled)
563-
tangerine.lookup POST without caching using Cloudflare x 90.81 ops/sec ±8.06% (89 runs sampled)
564-
tangerine.lookup GET with caching using Cloudflare x 256,141 ops/sec ±1.72% (87 runs sampled)
565-
+tangerine.lookup GET without caching using Cloudflare x 96.39 ops/sec ±0.31% (89 runs sampled)
566-
dns.promises.lookup with caching using Cloudflare x 1,473 ops/sec ±195.95% (87 runs sampled)
567-
-dns.promises.lookup without caching using Cloudflare x 4,191 ops/sec ±0.54% (85 runs sampled)
568-
Fastest without caching is: dns.promises.lookup without caching using Cloudflare
569-
570-
Started: resolve
571-
tangerine.resolve POST with caching using Cloudflare x 683 ops/sec ±195.88% (87 runs sampled)
572-
tangerine.resolve POST without caching using Cloudflare x 93.37 ops/sec ±0.48% (87 runs sampled)
573-
tangerine.resolve GET with caching using Cloudflare x 1,146,727 ops/sec ±0.58% (88 runs sampled)
574-
+tangerine.resolve GET without caching using Cloudflare x 93.33 ops/sec ±0.51% (87 runs sampled)
575-
tangerine.resolve POST with caching using Google x 1,133,683 ops/sec ±2.74% (89 runs sampled)
576-
tangerine.resolve POST without caching using Google x 83.91 ops/sec ±6.32% (76 runs sampled)
577-
tangerine.resolve GET with caching using Google x 1,147,212 ops/sec ±0.32% (90 runs sampled)
578-
tangerine.resolve GET without caching using Google x 79.73 ops/sec ±4.02% (77 runs sampled)
579-
resolver.resolve with caching using Cloudflare x 5,318,406 ops/sec ±0.67% (86 runs sampled)
580-
-resolver.resolve without caching using Cloudflare x 100 ops/sec ±1.55% (79 runs sampled)
581-
Fastest without caching is: resolver.resolve without caching using Cloudflare
582-
583-
Started: reverse
584-
tangerine.reverse GET with caching x 722 ops/sec ±195.42% (88 runs sampled)
585-
+tangerine.reverse GET without caching x 93.19 ops/sec ±0.74% (87 runs sampled)
586-
resolver.reverse with caching x 5,520,569 ops/sec ±0.59% (85 runs sampled)
587-
-resolver.reverse without caching x 17.42 ops/sec ±162.63% (70 runs sampled)
588-
dns.promises.reverse with caching x 5,164,258 ops/sec ±0.96% (86 runs sampled)
589-
-dns.promises.reverse without caching x 0.20 ops/sec ±184.87% (25 runs sampled)
590-
+Fastest without caching is: tangerine.reverse GET without caching
591-
```
534+
<!-- BENCHMARK_RESULTS_END -->
592535
593536
---
594537
538+
The benchmarks above are automatically updated daily via `.github/workflows/daily-benchmarks.yml`.
539+
595540
You can also [run the benchmarks yourself](#benchmarks).
596541
597542
---
598543
599-
Provided below are additional benchmark tests we have run:
600-
601-
> Node v18.14.2 on MacBook Air M1 16GB (without VPN):
602-
603-
```diff
604-
node --version
605-
v18.14.2
606-
607-
node benchmarks/lookup && node benchmarks/resolve && node benchmarks/reverse
608-
609-
Started: lookup
610-
tangerine.lookup POST with caching using Cloudflare x 1,035 ops/sec ±195.73% (91 runs sampled)
611-
tangerine.lookup POST without caching using Cloudflare x 52.76 ops/sec ±51.29% (53 runs sampled)
612-
tangerine.lookup GET with caching using Cloudflare x 694,910 ops/sec ±1.54% (87 runs sampled)
613-
+tangerine.lookup GET without caching using Cloudflare x 40.18 ops/sec ±60.19% (49 runs sampled)
614-
dns.promises.lookup with caching using Cloudflare x 12,645,103 ops/sec ±0.26% (90 runs sampled)
615-
-dns.promises.lookup without caching using Cloudflare x 2,664 ops/sec ±0.54% (88 runs sampled)
616-
Fastest without caching is: dns.promises.lookup without caching using Cloudflare
617-
618-
Started: resolve
619-
tangerine.resolve POST with caching using Cloudflare x 1,005 ops/sec ±195.93% (91 runs sampled)
620-
tangerine.resolve POST without caching using Cloudflare x 55.52 ops/sec ±46.26% (57 runs sampled)
621-
tangerine.resolve GET with caching using Cloudflare x 2,879,865 ops/sec ±0.35% (86 runs sampled)
622-
+tangerine.resolve GET without caching using Cloudflare x 71.11 ops/sec ±2.94% (74 runs sampled)
623-
tangerine.resolve POST with caching using Google x 1,292 ops/sec ±195.91% (88 runs sampled)
624-
tangerine.resolve POST without caching using Google x 36.88 ops/sec ±41.76% (53 runs sampled)
625-
tangerine.resolve GET with caching using Google x 2,885,428 ops/sec ±0.22% (88 runs sampled)
626-
tangerine.resolve GET without caching using Google x 70.38 ops/sec ±3.72% (68 runs sampled)
627-
resolver.resolve with caching using Cloudflare x 10,645,813 ops/sec ±0.23% (91 runs sampled)
628-
-resolver.resolve without caching using Cloudflare x 71.80 ops/sec ±2.84% (67 runs sampled)
629-
+Fastest without caching is: resolver.resolve without caching using Cloudflare, tangerine.resolve GET without caching using Cloudflare, tangerine.resolve GET without caching using Google, tangerine.resolve POST without caching using Cloudflare
630-
631-
Started: reverse
632-
tangerine.reverse GET with caching x 917 ops/sec ±195.78% (88 runs sampled)
633-
+tangerine.reverse GET without caching x 51.15 ops/sec ±51.92% (61 runs sampled)
634-
resolver.reverse with caching x 11,058,579 ops/sec ±0.37% (88 runs sampled)
635-
-resolver.reverse without caching x 62.30 ops/sec ±24.83% (64 runs sampled)
636-
dns.promises.reverse with caching x 11,276,123 ops/sec ±0.17% (90 runs sampled)
637-
-dns.promises.reverse without caching x 73.46 ops/sec ±1.99% (69 runs sampled)
638-
Fastest without caching is: dns.promises.reverse without caching, resolver.reverse without caching
639-
```
640-
641-
> Node v18.14.2 on MacBook Air M1 16GB (with DNS blackholed VPN) – **this highlights the DNS blackhole problem**:
642-
643-
```diff
644-
node --version
645-
v18.14.2
646-
647-
node benchmarks/lookup && node benchmarks/resolve && node benchmarks/reverse
648-
649-
Started: lookup
650-
tangerine.lookup POST with caching using Cloudflare x 1,327 ops/sec ±195.65% (89 runs sampled)
651-
tangerine.lookup POST without caching using Cloudflare x 71.11 ops/sec ±8.24% (71 runs sampled)
652-
tangerine.lookup GET with caching using Cloudflare x 759,816 ops/sec ±0.46% (90 runs sampled)
653-
+tangerine.lookup GET without caching using Cloudflare x 73.98 ops/sec ±1.78% (69 runs sampled)
654-
dns.promises.lookup with caching using Cloudflare x 1,744 ops/sec ±195.97% (88 runs sampled)
655-
-dns.promises.lookup without caching using Cloudflare x 2,717 ops/sec ±0.82% (87 runs sampled)
656-
Fastest without caching is: dns.promises.lookup without caching using Cloudflare
657-
658-
Started: resolve
659-
tangerine.resolve POST with caching using Cloudflare x 947 ops/sec ±195.93% (91 runs sampled)
660-
tangerine.resolve POST without caching using Cloudflare x 44.33 ops/sec ±73.30% (75 runs sampled)
661-
tangerine.resolve GET with caching using Cloudflare x 2,814,737 ops/sec ±0.17% (91 runs sampled)
662-
+tangerine.resolve GET without caching using Cloudflare x 57.25 ops/sec ±51.61% (73 runs sampled)
663-
tangerine.resolve POST with caching using Google x 1,087 ops/sec ±195.92% (91 runs sampled)
664-
tangerine.resolve POST without caching using Google x 36.84 ops/sec ±7.04% (62 runs sampled)
665-
tangerine.resolve GET with caching using Google x 2,784,199 ops/sec ±0.15% (92 runs sampled)
666-
tangerine.resolve GET without caching using Google x 47.55 ops/sec ±5.66% (76 runs sampled)
667-
resolver.resolve with caching using Cloudflare x 0.09 ops/sec ±6.41% (5 runs sampled)
668-
-resolver.resolve without caching using Cloudflare x 0.10 ops/sec ±6.52% (5 runs sampled)
669-
+Fastest without caching is: tangerine.resolve GET without caching using Google
670-
671-
Started: reverse
672-
tangerine.reverse GET with caching x 1,345 ops/sec ±195.66% (92 runs sampled)
673-
+tangerine.reverse GET without caching x 71.73 ops/sec ±3.03% (73 runs sampled)
674-
resolver.reverse with caching x 0.10 ops/sec ±6.54% (5 runs sampled)
675-
-resolver.reverse without caching x 0.10 ops/sec ±0.01% (5 runs sampled)
676-
dns.promises.reverse with caching x 0.10 ops/sec ±6.54% (5 runs sampled)
677-
-dns.promises.reverse without caching x 0.10 ops/sec ±0.01% (5 runs sampled)
678-
+Fastest without caching is: tangerine.reverse GET without caching
679-
```
680-
681544
Also see this [write-up](https://samknows.com/blog/dns-over-https-performance) on UDP-based DNS versus DNS over HTTPS ("DoH") benchmarks.
682545
683546
**Speed could be increased** by switching to use [undici streams](https://undici.nodejs.org/#/?id=undicistreamurl-options-factory-promise) and [getStream.buffer](https://github.com/sindresorhus/get-stream) (pull request is welcome).

0 commit comments

Comments
 (0)