Comments (3)
anjackson
commented on August 24, 2024
Looking at the documentation we are using the 'Standalone Consumer' pattern, and it is implied that we must commit manually rather than being able to rely on the auto.commit behaviour. Not 100% clear though.
from ukwa-heritrix.
anjackson
commented on August 24, 2024
Hmm, locally, couldn't force re-processing the queue on just pause/unpause, but stop and restart did do it because KAFKA_SEEK_TO_BEGINNING was true. Odd. Inspecting logs from the live system.
from ukwa-heritrix.
anjackson
commented on August 24, 2024
I think I was misinterpreting the way they were being reported. The behaviour appears to be as expected now.
from ukwa-heritrix.
Related Issues (20)
- URL launch with launch timestamp not forcing recrawl HOT 2
- Add 'The Internet' test image to the suite
- DC2021 issues HOT 4
- Update User Agent URL
- Broken checkpoints HOT 1
- Add support for configuring the Redis DB via an environment variable
- Quotas reset on restart
- Verify Redis implementation
- Add support for Redis-compatible alternatives like KvRocks
- Add faster/parallel queues for known CDNs HOT 1
- Create a url-frontier Frontier implementation HOT 1
- Update security.txt well-known URI HOT 1
- Firewall issues when crawling some websites
- Domain crawl tripping 'abuse' alerts HOT 1
- Ensure blocked sites are fully blocked. HOT 1
- Timsort comparison error for specific robots.txt URL HOT 1
- Add NLA/Pandas crawler trap rules
- Add unit test for difficult sitemap case
- Crawler ignoring robots.txt on a particular site HOT 1
- WARC fields not populated in Kafka crawl log
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ukwa-heritrix.