are there any plans to move to maven? about crawler4j HOT 10 CLOSED

I assume you're talking about adding it to a central Maven repository. Do you 
know the process?

-Yasser

Hello Yasser,

I created a maven pom file which defines the projects dependencies.
I put it on github (synced the svn history with git).
you can view it here https://github.com/mren/crawler4j

One dependency (dsiutils) is from a non-standard repository,
unfortunately the maintainers did not put a recent version to central
maven.

[deleted comment]

awesome

hi mark , i tried to run the pom file as an aggregated project of parent pom:

<groupId>net.she.swengineering.diplomarbeit</groupId>
  <artifactId>Diff2W3C</artifactId>
  <packaging>war</packaging>
  <version>1.0</version>
  <name>Diff2W3C Maven Webapp</name>
  <url>http://maven.apache.org</url>


  <modules>
    <module>crawler4j</module>
  </modules>

the module "crawler4j" specifies you described the directory you put to your  
github svn account.in this directory there are the pom file and the source of 
crawler4j.


it doesn't work because work here is my trace:

[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - crawler4j:crawler4j:jar:2.61
[INFO]    task-segment: [eclipse:eclipse]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing eclipse:eclipse
[INFO] No goals needed for project - skipping
[INFO] [eclipse:eclipse {execution: default-cli}]
[INFO] Using Eclipse Workspace: C:\Dokumente und Einstellungen\andjock\workspace

[WARNING] Workspace defines a VM that does not contain a valid jre/lib/rt.jar: C
:\Programme\Java\jre6
[INFO] no substring wtp server match.
[INFO] Using as WTP server : Apache Tomcat v6.0
[INFO] Adding default classpath container: org.eclipse.jdt.launching.JRE_CONTAIN
ER
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
central (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.pom
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:pom:1.0.10' in repository
oracle (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository centr
al (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.pom
[INFO] Unable to find resource 'com.sleepycat:je:pom:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
central (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
repo-for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/it/unimi/dsi/ds
iutils/1.0.10/dsiutils-1.0.10.jar
[INFO] Unable to find resource 'it.unimi.dsi:dsiutils:jar:1.0.10' in repository
oracle (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
    Failed to retrieve it.unimi.dsi:dsiutils-1.0.10
Caused by: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command:
    mvn install:install-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dvers
ion=1.0.10 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
    mvn deploy:deploy-file -DgroupId=it.unimi.dsi -DartifactId=dsiutils -Dversio
n=1.0.10 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
        1) crawler4j:crawler4j:jar:2.61
        2) it.unimi.dsi:dsiutils:jar:1.0.10


  it.unimi.dsi:dsiutils:jar:1.0.10

from the specified remote repositories:
  nexus (http://loki.she.de:8080/nexus/content/groups/public)


[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\it\unimi\dsi\dsiutils\1.0.10\dsiutils-1.0.10.jar
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository centr
al (http://central)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository repo-
for-dsiutils (http://ir.dcs.gla.ac.uk/~bpiwowar/maven/)
Downloading: http://loki.she.de:8080/nexus/content/groups/public/com/sleepycat/j
e/4.0.71/je-4.0.71.jar
[INFO] Unable to find resource 'com.sleepycat:je:jar:4.0.71' in repository oracl
e (http://download.oracle.com/maven)
[WARNING] An error occurred during dependency resolution.
    Failed to retrieve com.sleepycat:je-4.0.71
Caused by: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command:
    mvn install:install-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4
.0.71 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
    mvn deploy:deploy-file -DgroupId=com.sleepycat -DartifactId=je -Dversion=4.0
.71 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
        1) crawler4j:crawler4j:jar:2.61
        2) com.sleepycat:je:jar:4.0.71


  com.sleepycat:je:jar:4.0.71

from the specified remote repositories:
  nexus (http://loki.she.de:8080/nexus/content/groups/public)


[INFO] Unable to read jar manifest from C:\Dokumente und Einstellungen\andjock\.
m2\repository\com\sleepycat\je\4.0.71\je-4.0.71.jar
[INFO] Not writing settings - defaults suffice
[INFO] Wrote Eclipse project for "crawler4j" to C:\Dokumente und Einstellungen\a
ndjock\workspace\Diff2W3C\crawler4j.

[deleted comment]

Hi,

I done working setup for netbeans project which is using crawler4j, jsoup and 
JPA,

here's my pom.xml



<project xmlns="http://maven.apache.org/POM/4.0.0" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.gmail.krzysztof7kowalski</groupId>
    <artifactId>scrap-maven</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>medscrap-maven</name>
    <url>http://maven.apache.org</url>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <repositories>
        <repository>
            <id>oracleReleases</id>
            <name>Oracle Released Java Packages</name>
            <url>http://download.oracle.com/maven</url>
            <layout>default</layout>
        </repository>
        <repository>
            <id>piwowar</id>
            <url>http://ir.dcs.gla.ac.uk/~bpiwowar/maven</url>
        </repository>
    </repositories>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.1.2</version>
        </dependency>

        <dependency>
            <groupId>it.unimi.dsi</groupId>
            <artifactId>dsiutils</artifactId>
            <version>1.0.10</version>
        </dependency>

        <dependency>
            <groupId>com.sleepycat</groupId>
            <artifactId>je</artifactId>
            <version>3.3.75</version>
        </dependency>

        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.1.1</version>
        </dependency>

        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.6</version>
        </dependency>

        <dependency>
        <!-- jsoup HTML parser library @ http://jsoup.org/ -->
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.1.3</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.14</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>javax.persistence</artifactId>
            <version>2.0.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>com.mysql</groupId>
            <artifactId>management</artifactId>
            <version>5-0-2-beta</version>
            <exclusions>
                <exclusion>
                    <artifactId>mysql-connector-java</artifactId>
                    <groupId>mysql</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>it.unimi.dsi</groupId>
            <artifactId>fastutil</artifactId>
            <version>6.4.1</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.18</version>
        </dependency>
    </dependencies>

</project>

As of version 3.0 support for Maven is added. See 
http://code.google.com/p/crawler4j/source/browse/pom.xml for the pom.xml file. 
crawler4j will also soon be available through a public Maven repository.

-Yasser

crawler4j is now available from central Maven repository: 
http://code.google.com/p/crawler4j/wiki/MavenConfig

-Yasser

Great job, ganjisaffar !!!