Comments (6)
Please provide me with more details:
- Node version?
- Print the Http Headers array & share it in a comment
- Mention any frameworks you are using
- Minor example to reproduce
from es6-crawler-detect.
@vatanuki can you please update the issue, otherwise it will be closed
from es6-crawler-detect.
@JefferyHus sure
node -v
v14.18.2
test code:
'use strict';
const express = require('express')
const { Crawler } = require('es6-crawler-detect')
const app = express()
app.get('/curl', function async (request, response) {
var CrawlerDetector = new Crawler(request)
response.send( CrawlerDetector.isCrawler() ? 'detected' : 'not detected')
})
app.listen(3000)
curl test request:
curl -v http://127.0.0.1:3000/curl
* Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /curl HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/html; charset=utf-8
< Content-Length: 12
< ETag: W/"c-O+YmnRNLEJrWQnm49UdT/kB8u0I"
< Date: Sat, 25 Dec 2021 03:11:24 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host 127.0.0.1 left intact
not detected
posible fix:
diff --git a/src/lib/crawler.js b/src/lib/crawler.js
index 047f2c7..43e2314 100644
--- a/src/lib/crawler.js
+++ b/src/lib/crawler.js
@@ -76,7 +76,7 @@ class Crawler {
) {
for (const header of this.getUaHttpHeaders()) {
if (Object.keys(this.httpHeaders).indexOf(header.toLowerCase()) >= 0) {
- userAgent += this.httpHeaders[header] + ' ';
+ userAgent += this.httpHeaders[header.toLowerCase()] + ' ';
}
}
}
curl test request after fix:
curl -v http://127.0.0.1:3000/curl
* Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /curl HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/html; charset=utf-8
< Content-Length: 8
< ETag: W/"8-vL//2K5LHFy1YRyMCLFAJ5fXQXc"
< Date: Sat, 25 Dec 2021 03:14:34 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host 127.0.0.1 left intact
detected
from es6-crawler-detect.
@vatanuki can you test this branch and see if it works for you? #41
In the meantime I will try to add more test cases which includes your case here and see if the changes you are introducing are not going to break the module
from es6-crawler-detect.
@JefferyHus tested (branch feat/testing-cases), its still not detected
from es6-crawler-detect.
@vatanuki #41 is now fixing your bug, thanks for flagging this bug. You can also check the test cases that performs some simulations on the headers & request object
from es6-crawler-detect.
Related Issues (15)
- TelegramBot not detected HOT 9
- Library cannot be imported in Node.js HOT 1
- Support for Typescript file ? HOT 7
- How to print something on crawler hit with middleware? HOT 2
- update the documentation of how to use in the homepage. HOT 1
- TypeError: Cannot read property 'replace' of undefined at Crawler.isCrawler HOT 2
- Does this support Googlebot? HOT 1
- Potential DDOS vulnerability? HOT 5
- isCrawler() doesnt detect Crawler HOT 10
- Is `projectdiscovery` supposed to be considered as a crawler? HOT 1
- ReferenceError: Crawler is not defined HOT 2
- (Browser runtime) `ReferenceError: Buffer is not defined` HOT 10
- TS Support HOT 2
- Incorrect Crawler.getMatches HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from es6-crawler-detect.