vertica-stream-writer's Introduction

Introduction

This library is a helper library to stream data in the Vertica Native Binary Format. The goal is to use a VerticaCopyStream to import data to Vertica in the most efficient way possible.

Dependency

<dependency>
    <groupId>com.github.jcustenborder</groupId>
    <artifactId>vertica-stream-writer</artifactId>
    <version>[0.0.1.1,]</version>
</dependency>

LZO Support

LZO compression support is optional due to the licensing with the upstream library. To enable it you must add the following to your pom to bring in the library.

<dependency>
    <groupId>org.anarres.lzo</groupId>
    <artifactId>lzo-core</artifactId>
    <version>1.0.5</version>    
</dependency>

Example

Below is a direct example of building the example file defined in the Vertica Documentation Creating Native Binary Format Files

    VerticaStreamWriterBuilder streamWriterBuilder = new VerticaStreamWriterBuilder()
        .table("allTypes")
        .column("INTCOL", VerticaType.INTEGER, 8)
        .column("FLOATCOL", VerticaType.FLOAT)
        .column("CHARCOL", VerticaType.CHAR, 10)
        .column("VARCHARCOL", VerticaType.VARCHAR)
        .column("BOOLCOL", VerticaType.BOOLEAN)
        .column("DATECOL", VerticaType.DATE)
        .column("TIMESTAMPCOL", VerticaType.TIMESTAMP)
        .column("TIMESTAMPTZCOL", VerticaType.TIMESTAMPTZ)
        .column("TIMECOL", VerticaType.TIME)
        .column("TIMETZCOL", VerticaType.TIMETZ)
        .column("VARBINCOL", VerticaType.VARBINARY)
        .column("BINCOL", VerticaType.BINARY, 3)
        .column("NUMCOL", VerticaType.NUMERIC, 38, 0)
        .column("INTERVALCOL", VerticaType.INTERVAL);

    Object[] row = new Object[]{
        1,
        -1.11D,
        "one       ",
        "ONE",
        true,
        new Date(915753600000L),
        new Date(919739512350L),
        date("yyyy-MM-dd HH:mm:ssX", "1999-01-08 07:04:37-05"),
        date("HH:mm:ss", "07:09:23"),
        date("HH:mm:ssX", "15:12:34-05"),
        BaseEncoding.base16().decode("ABCD"),
        BaseEncoding.base16().decode("ABCD"),
        BigDecimal.valueOf(1234532),
        (Duration.ofHours(3).plusMinutes(3).plusSeconds(3).toMillis() * 1000L)
    };

    assertEquals(14, streamWriterBuilder.columnInfos.size(), "column count should match.");

    final String actual;

    try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
      try (VerticaStreamWriter streamWriter = streamWriterBuilder.build(outputStream)) {
        streamWriter.write(row);
      }
    }

vertica-stream-writer's People

Contributors

Stargazers

vertica-stream-writer's Issues

Null Value Issue When using binary file

When using a binary file output for loading into Vertica with then native binary load any null value prevent the output .bin file from being loaded.
example:

VerticaStreamWriterBuilder builder = new VerticaStreamWriterBuilder()
                .streamWriterType(VerticaStreamWriterType.NATIVE)
                .table("allTypes")
                .column("INTCOL", INTEGER, 8)
                .column("FLOATCOL", FLOAT)
                .column("CHARCOL", CHAR, 10)
                .column("VARCHARCOL", VARCHAR)
                .column("BOOLCOL", BOOLEAN)
                .column("DATECOL", DATE)
                .column("TIMESTAMPCOL", TIMESTAMP)
                .column("TIMESTAMPTZCOL", TIMESTAMPTZ)
                .column("TIMECOL", TIME)
                .column("TIMETZCOL", TIMETZ)
                .column("VARBINCOL", VARBINARY)
                .column("BINCOL", BINARY, 3)
                .column("NUMCOL", NUMERIC, 38, 0)
                .column("INTERVALCOL", INTERVAL);

        Object[] row = new Object[]{
                1L,
                -1.11D,
                "one       ",
                null,
                true,
                new Date(915753600000L),
                new Date(919739512350L),
                date("yyyy-MM-dd HH:mm:ssX", "1999-01-08 07:04:37-05"),
                date("HH:mm:ss", "07:09:23"),
                OffsetTime.parse("15:12:34-04", DateTimeFormatter.ofPattern("HH:mm:ssX")),
                BaseEncoding.base16().decode("ABCD"),
                BaseEncoding.base16().decode("ABCD"),
                BigDecimal.valueOf(1234532),
                (Duration.ofHours(3).plusMinutes(3).plusSeconds(3).toMillis() * 1000L)
        };

        try (FileOutputStream fileOutputStream = new FileOutputStream("target/allTypes.bin")) {
             try (VerticaStreamWriter streamWriter = builder.build(fileOutputStream)) {
                streamWriter.write(row);
        }

command to load native file 'COPY allTypes FROM '/home/dbadmin/allTypes.bin' NATIVE DIRECT;'

Recommend Projects