Comments (10)
I assume you're talking about adding it to a central Maven repository. Do you
know the process?
-Yasser
Original comment by [email protected]
on 4 May 2011 at 8:12
from crawler4j.
Hello Yasser,
I created a maven pom file which defines the projects dependencies.
I put it on github (synced the svn history with git).
you can view it here https://github.com/mren/crawler4j
One dependency (dsiutils) is from a non-standard repository,
unfortunately the maintainers did not put a recent version to central
maven.
Original comment by [email protected]
on 5 May 2011 at 8:19
from crawler4j.
[deleted comment]
from crawler4j.
awesome
Original comment by [email protected]
on 5 May 2011 at 8:28
from crawler4j.
hi mark , i tried to run the pom file as an aggregated project of parent pom:
<groupId>net.she.swengineering.diplomarbeit</groupId>
<artifactId>Diff2W3C</artifactId>
<packaging>war</packaging>
<version>1.0</version>
<name>Diff2W3C Maven Webapp</name>
<url>http://maven.apache.org</url>
<modules>
<module>crawler4j</module>
</modules>
the module "crawler4j" specifies you described the directory you put to your
github svn account.in this directory there are the pom file and the source of
crawler4j.
it doesn't work because work here is my trace:
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - crawler4j:crawler4j:jar:2.61
[INFO] task-segment: [eclipse:eclipse]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing eclipse:eclipse
[INFO] No goals needed for project - skipping
[INFO] [eclipse:eclipse {execution: default-cli}]
[INFO] Using Eclipse Workspace: C:\Dokumente und Einstellungen\andjock\workspace
[WARNING] Workspace defines a VM that does not contain a valid jre/lib/rt.jar: C
:\Programme\Java\jre6
[INFO] no substring wtp server match.
[INFO] Using as WTP server : Apache Tomcat v6.0
[INFO] Adding default classpath container: org.eclipse.jdt.launching.JRE_CONTAIN
ER
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
central (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
oracle (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository centr
al (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
central (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
oracle (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
Failed to retrieve it.unimi.dsi:dsiutils-1.0.10
Caused by: Unable to download the artifact from any repository
Try downloading the file manually from the project website.
Then, install it using the command:
mvn install:install-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dvers
ion=1.0.10 -Dpackaging=jar -Dfile=/path/to/file
Alternatively, if you host your own repository you can deploy the file there:
mvn deploy:deploy-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dversio
n=1.0.10 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
Path to dependency:
1) crawler4j:crawler4j:jar:2.61
2) it.unimi.dsi:dsiutils:jar:1.0.10
it.unimi.dsi:dsiutils:jar:1.0.10
from the specified remote repositories:
nexus (http://loki.she.de:8080/nexus/content/groups/public)
[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\it\unimi\dsi\dsiutils\1.0.10\dsiutils-1.0.10.jar
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository centr
al (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
Failed to retrieve com.sleepycat:je-4.0.71
Caused by: Unable to download the artifact from any repository
Try downloading the file manually from the project website.
Then, install it using the command:
mvn install:install-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4
.0.71 -Dpackaging=jar -Dfile=/path/to/file
Alternatively, if you host your own repository you can deploy the file there:
mvn deploy:deploy-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4.0
.71 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
Path to dependency:
1) crawler4j:crawler4j:jar:2.61
2) com.sleepycat:je:jar:4.0.71
com.sleepycat:je:jar:4.0.71
from the specified remote repositories:
nexus (http://loki.she.de:8080/nexus/content/groups/public)
[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\com\sleepycat\je\4.0.71\je-4.0.71.jar
[INFO] Not writing settings - defaults suffice
[INFO] Wrote Eclipse project for "crawler4j" to C:\Dokumente und Einstellungen\a
ndjock\workspace\Diff2W3C\crawler4j.
Original comment by [email protected]
on 26 Sep 2011 at 10:06
from crawler4j.
[deleted comment]
from crawler4j.
Hi,
I done working setup for netbeans project which is using crawler4j, jsoup and
JPA,
here's my pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.gmail.krzysztof7kowalski</groupId>
<artifactId>scrap-maven</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>medscrap-maven</name>
<url>http://maven.apache.org</url>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>oracleReleases</id>
<name>Oracle Released Java Packages</name>
<url>http://download.oracle.com/maven</url>
<layout>default</layout>
</repository>
<repository>
<id>piwowar</id>
<url>http://ir.dcs.gla.ac.uk/~bpiwowar/maven</url>
</repository>
</repositories>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.1.2</version>
</dependency>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>dsiutils</artifactId>
<version>1.0.10</version>
</dependency>
<dependency>
<groupId>com.sleepycat</groupId>
<artifactId>je</artifactId>
<version>3.3.75</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<!-- jsoup HTML parser library @ http://jsoup.org/ -->
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.1.3</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.14</version>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>javax.persistence</artifactId>
<version>2.0.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>com.mysql</groupId>
<artifactId>management</artifactId>
<version>5-0-2-beta</version>
<exclusions>
<exclusion>
<artifactId>mysql-connector-java</artifactId>
<groupId>mysql</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil</artifactId>
<version>6.4.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.18</version>
</dependency>
</dependencies>
</project>
Original comment by [email protected]
on 21 Dec 2011 at 6:02
from crawler4j.
As of version 3.0 support for Maven is added. See
http://code.google.com/p/crawler4j/source/browse/pom.xml for the pom.xml file.
crawler4j will also soon be available through a public Maven repository.
-Yasser
Original comment by [email protected]
on 2 Jan 2012 at 7:42
- Changed state: Started
- Added labels: Type-Enhancement
- Removed labels: Type-Defect
from crawler4j.
crawler4j is now available from central Maven repository:
http://code.google.com/p/crawler4j/wiki/MavenConfig
-Yasser
Original comment by [email protected]
on 6 Jan 2012 at 6:51
- Changed state: Done
from crawler4j.
Great job, ganjisaffar !!!
Original comment by [email protected]
on 1 Feb 2012 at 11:41
from crawler4j.
Related Issues (20)
- HttpResponse response = httpClient.execute(get) in PageFetcher has no Timeout HOT 2
- Change logging from System.out.print to SLF4J HOT 1
- WebCrawler should throw exceptions instead of returning at the middle of the method HOT 2
- Upgrade CustomFetchStatus HOT 1
- NullPointerException when crawling links with no HREF HOT 1
- emove Non official http status codes HOT 1
- PageFetcher is unreadable HOT 1
- crawler fail due to http 303 see other HOT 8
- Scraping iframes, base64,vb scripts HOT 6
- FileNotFoundException: .m2\repository\edu\uci\ics\crawler4j\4.0\crawler4j-4.0.jar!\tld-names.zip HOT 8
- Proxy information get lost when using basic authentication
- Slf4j libraries missing on some configurations HOT 1
- Jar with dependencies
- Crawlers storage folder taking up too much space. HOT 4
- Crawling over disallowed paths from robots.txt HOT 1
- robots.txt isn't crawled HOT 4
- Hanging on file process HOT 4
- Memory usage
- Crawler time delay when exiting
- Crawler is slow
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crawler4j.