I have noticed that when fetching some URLs with nodejs fetch
a TypeError is thrown when trying to parse the text on the response using the .text()
method.
I have tried it on Ubuntu-20.04 WSL2 with node 20.9.0 and on Windows with node 20.11.0 both produce the same result.
In the following example the request to https://www.giant-bicycles.com/de/glory-advanced-rahmenset-2024
will throw an error, its counterpart https://www.giant-bicycles.com/us/glory-advanced-legends-edition-frameset-2024
too. All others work. Somehow different links on the giant site work but others will cause an error in nodejs fetch.
Am I missing something, is this an error on the server side or even a node bug?
Example code:
const urls = [
["gzip", "https://carsten.codimi.de/gzip.yaws/daniels.html"],
["deflate", "https://carsten.codimi.de/gzip.yaws/daniels.html?deflate=on&zlib=on"],
["giant", "https://www.giant-bicycles.com/de/glory-advanced-rahmenset-2024"], // This request fails
["giant", "https://www.giant-bicycles.com/de/anthem-advanced-1-2024"],
];
urls.reduce(async (prev, [name, url]) => {
await prev;
try {
return fetch(url)
.then((res) => res.text())
.then((text) => console.log(name, url, "ok", text.slice(0, 10)))
.catch((e) => {
console.error(`Error fetching page "${url}": ${e.toString()}`, e);
});
} catch (e) {}
}, Promise.resolve()).then(() => process.exit());
$ node test.js
gzip https://carsten.codimi.de/gzip.yaws/daniels.html ok <!DOCTYPE
deflate https://carsten.codimi.de/gzip.yaws/daniels.html?deflate=on&zlib=on ok <!DOCTYPE
Error fetching page "https://www.giant-bicycles.com/de/glory-advanced-rahmenset-2024": TypeError: terminated TypeError: terminated
at Fetch.onAborted (node:internal/deps/undici/undici:9975:53)
at Fetch.emit (node:events:514:28)
at Fetch.terminate (node:internal/deps/undici/undici:9174:14)
at fetchParams.controller.resume (node:internal/deps/undici/undici:9952:36)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[cause]: Error: unknown compression method
at Zlib.zlibOnError [as onerror] (node:zlib:189:17) {
errno: -3,
code: 'Z_DATA_ERROR'
}
}
giant https://www.giant-bicycles.com/de/anthem-advanced-1-2024 ok <!DOCTYPE
EDIT
When disabling compression using Accept-Encoding: identify
it works but id still like to use compression if possible.