Package

com.intenthq.pucket

writer

Permalink

package writer

Visibility
  1. Public
  2. All

Type Members

  1. case class IncrementalPartitionedWriter[T] extends Writer[T, (Long, Throwable)] with PartitionedWriterFunctions[T, (Long, Throwable), IncrementalPartitionedWriter[T]] with Product with Serializable

    Permalink

    Incremental partitioned parquet writer Provides incremental writing for partitioned puckets Any error will return with the minimum checkpoint of the current set of writers

  2. case class IncrementalWriter[T] extends Writer[T, (Long, Throwable)] with Product with Serializable

    Permalink

    Incremental functional writer class for parquet

    Incremental functional writer class for parquet

    Overview

    Allows large files to be written to parquet in smaller incements. Closes the underlying writer and creates a new file at a defined writer count.

    Also keeps a checkpoint of when the last file was sucessfully flushed to the filesystem so in the event of an error processing can be restarted from that point.

    T

    type of data to be written

  3. case class PartitionedWriter[T] extends Writer[T, Throwable] with PartitionedWriterFunctions[T, Throwable, PartitionedWriter[T]] with Product with Serializable

    Permalink

    Partitioned parquet writer Allows a pucket to be written to with a partitioning scheme

  4. trait PartitionedWriterFunctions[T, Ex, ImplementingType] extends AnyRef

    Permalink

    Partitioned writer mixin provides partitioning to a writer type

    Partitioned writer mixin provides partitioning to a writer type

    Overview

    This trait provides common partitioning functions to implementing classes. It maintains a cache of open writer instances with a configurable size.

    T

    type of data to be written

    Ex

    type of error

    ImplementingType

    implementing type to be used when creating new instances

  5. trait Writer[T, Ex] extends AnyRef

    Permalink

    Trait for functional wrappers of parquet writers

    Trait for functional wrappers of parquet writers

    T

    type of data to be written

    Ex

    type of error returned if write fails

Ungrouped