Build #69

Build: #69 was successful Changes by 9 people

Code commits

cdapio/hydrator-plugins

  • vinishavyasa <CuriousVini@users.noreply.github.com>

    vinishavyasa <CuriousVini@users.noreply.github.com> b8c8df76a72f1d22ef3cca1f7c3e79b571df231a

    Merge pull request #561 from caskdata/release/1.5
    Merge release/1.5 to develop

  • Yaser

    Yaser 51a46fb9066490b184fd10d77e66a218d516541c

    Merge pull request #559 from caskdata/revert-555-update-docs-for-ignoreNonExistingFolders
    Revert "Update docs for ignoreNonExistingFolders. It is not nullable."

  • Yaser

    Yaser bf170a696342eb7516aedfaf3540f03b03cca0a3

    Revert "Update docs for ignoreNonExistingFolders. It is not nullable."

    • core-plugins/docs/AzureBlobStore-batchsource.md (version bf170a696342eb7516aedfaf3540f03b03cca0a3)
    • core-plugins/docs/FTP-batchsource.md (version bf170a696342eb7516aedfaf3540f03b03cca0a3)
    • core-plugins/docs/File-batchsource.md (version bf170a696342eb7516aedfaf3540f03b03cca0a3)
    • core-plugins/docs/S3-batchsource.md (version bf170a696342eb7516aedfaf3540f03b03cca0a3)
  • Chris Gianelloni

    Chris Gianelloni a377e428ca58b8a5e71f2e5abe92042765e178ce

    Merge pull request #557 from caskdata/feature_release/update-readme-submodules
    Update information on Git Submodules

  • Chris Gianelloni

    Chris Gianelloni 26496c7404d73bccd7b08528c84a1cacc04ced31

    Update information on Git Submodules

    • README.rst (version 26496c7404d73bccd7b08528c84a1cacc04ced31)
  • Chris Gianelloni

    Chris Gianelloni 68a50818d961e6b66836a86c6e710e56cd9df8a9

    Update refs: git submodule update --init --remote

    • wrangler-transform (version 68a50818d961e6b66836a86c6e710e56cd9df8a9)
  • Yaser

    Yaser 52900f8405727767295fb74fe65729d12938ef38

    Merge pull request #556 from caskdata/fix/FileBatchSource
    Making Ignore Non-Existing Folders optional in all file batch batch sources

  • Rohit Sinha

    Rohit Sinha 989e5312678b22056c0645372cfbe219d0474409

    Making Ignore Non-Existing Folders optional in all file batch batch sources

    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/AzureBatchSource.java (version 989e5312678b22056c0645372cfbe219d0474409)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/FTPBatchSource.java (version 989e5312678b22056c0645372cfbe219d0474409)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/FileBatchSource.java (version 989e5312678b22056c0645372cfbe219d0474409)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/S3BatchSource.java (version 989e5312678b22056c0645372cfbe219d0474409)
  • Ali Anwar

    Ali Anwar 056f495d0a03fe9d005f994199dbc83e7abcf4f2

    Merge pull request #555 from caskdata/update-docs-for-ignoreNonExistingFolders
    Update docs for ignoreNonExistingFolders. It is not nullable.

  • Ali Anwar

    Ali Anwar a830ba5637f853b707496a3a8734f8635cbf6392

    Update docs for ignoreNonExistingFolders. It is not nullable/default-able from the plugin perspective. The default is in the UI widget, actually.

    • core-plugins/docs/AzureBlobStore-batchsource.md (version a830ba5637f853b707496a3a8734f8635cbf6392)
    • core-plugins/docs/FTP-batchsource.md (version a830ba5637f853b707496a3a8734f8635cbf6392)
    • core-plugins/docs/File-batchsource.md (version a830ba5637f853b707496a3a8734f8635cbf6392)
    • core-plugins/docs/S3-batchsource.md (version a830ba5637f853b707496a3a8734f8635cbf6392)
  • albertshau <albert@cask.co>

    albertshau <albert@cask.co> be532806cff020d8246aabba768ff63861268596

    Merge pull request #554 from caskdata/bugfix_release/HYDRATOR-1163-fix-offset-lookup
    HYDRATOR-1163 fix topic partition offset lookup

  • Albert Shau <albert@cask.co>

    Albert Shau <albert@cask.co> 2134406b2ebe4a942bb94ea03749a85b4165ae25

    HYDRATOR-1163 fix topic partition offset lookup
    Fixed a bug where partitions configured to use the latest or
    earliest offset were not querying all brokers to find the
    information.

    • spark-plugins/src/main/java/co/cask/hydrator/plugin/spark/KafkaStreamingSource.java (version 2134406b2ebe4a942bb94ea03749a85b4165ae25)
  • Yaser

    Yaser 7d55f263ecc2d008d883dd5cc262146d6089766e

    Merge pull request #545 from ananya-bhattacharya/feature/HYDRATOR-976-compression-options
    [HYDRATOR-976] - Compression options in hdfs sink plugins

  • ananya-bhattacharya <ananya_bhattacharya@persistent.com>

    ananya-bhattacharya <ananya_bhattacharya@persistent.com> b4cec37d3ca2d2fa902cd7a0c78af4fd7b64d7c1

    Resolving conflicts

  • Yaser

    Yaser 6223bc58fd5d55d8adbf341f10d9f04f9473c174

    Merge pull request #534 from ananya-bhattacharya/feature/HYDRATOR-1004
    [HYDRATOR-1004] - Ignore non-existing files/folders  in File Source

  • Albert Shau <albert@cask.co>

    Albert Shau <albert@cask.co> f329cb530956a5b66bdf6ab54fb519daa7b8c5c0

    add commands to readme about submodules

    • README.rst (version f329cb530956a5b66bdf6ab54fb519daa7b8c5c0)
  • vinishavyasa <CuriousVini@users.noreply.github.com>

    vinishavyasa <CuriousVini@users.noreply.github.com> 854570deb064d3762cd8e06ad2bb683b0f044263

    Merge pull request #551 from caskdata/fix/bump-twill-version
    Bump Twill version

  • ananya-bhattacharya <ananya_bhattacharya@persistent.com>

    ananya-bhattacharya <ananya_bhattacharya@persistent.com> ebeb135fe0dad2767de19d8c933f06e15147e9e7

    Adding option to ignore non-existing files/folders

    • core-plugins/docs/AzureBlobStore-batchsource.md (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/docs/FTP-batchsource.md (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/docs/File-batchsource.md (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/docs/S3-batchsource.md (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/AzureBatchSource.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/FTPBatchSource.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/FileBatchSource.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/S3BatchSource.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/common/Properties.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLFTPTestRun.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLMapReduceTestRun.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/source/FileBatchSourceTest.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/source/S3BatchSourceTest.java (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/widgets/AzureBlobStore-batchsource.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/widgets/FTP-batchsource.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/widgets/File-batchsource.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • core-plugins/widgets/S3-batchsource.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • database-plugins/widgets/Database-action.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • database-plugins/widgets/Database-batchsink.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • database-plugins/widgets/Database-batchsource.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
    • database-plugins/widgets/DatabaseQuery-postaction.json (version ebeb135fe0dad2767de19d8c933f06e15147e9e7)
  • Russ Savage <russ@cask.co>

    Russ Savage <russ@cask.co> f4ef3c80813cbc11508a751c55f3f8bdc1d711a0

    adding compression options to hdfs sinks

    • core-plugins/docs/S3Avro-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/S3Parquet-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/SnapshotAvro-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/SnapshotParquet-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/TPFSAvro-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/TPFSOrc-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/docs/TPFSParquet-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/S3AvroBatchSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/S3ParquetBatchSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/SnapshotFileBatchAvroSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/SnapshotFileBatchParquetSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/TimePartitionedFileSetDataSetORCSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/TimePartitionedFileSetDatasetAvroSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/TimePartitionedFileSetDatasetParquetSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/sink/TimePartitionedFileSetSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/batch/source/FTPBatchSource.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/main/java/co/cask/hydrator/plugin/common/FileSetUtil.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLBatchTestBase.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/ETLTPFSTestRun.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/src/test/java/co/cask/hydrator/plugin/batch/sink/S3BatchSinkTest.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/S3Avro-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/S3Parquet-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/SnapshotAvro-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/SnapshotParquet-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/TPFSAvro-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/TPFSOrc-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • core-plugins/widgets/TPFSParquet-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • hdfs-plugins/docs/HDFS-batchsink.md (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • hdfs-plugins/src/main/java/co/cask/hydrator/plugin/HDFSSink.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • hdfs-plugins/src/test/java/co/cask/hydrator/plugin/HDFSSinkTest.java (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)
    • hdfs-plugins/widgets/HDFS-batchsink.json (version f4ef3c80813cbc11508a751c55f3f8bdc1d711a0)