Build #239

Automatically provisions CDAP cluster and runs integration test suite against it.

Build: #239 was successful Manual run by Vinisha Shah

Code commits

caskdata/cdap-integration-tests

  • greeshmaswaminathan <greeshmaswaminathan@users.noreply.github.com>

    greeshmaswaminathan <greeshmaswaminathan@users.noreply.github.com> 02d898951ea502fef614aab144e7ba36d9669a9d

    Merge pull request #1076 from cdapio/feature/PLUGIN-385-Integration-tests-BQMultiSink
    Integration tests for BQ Multisink

  • Greeshma Swaminathan

    Greeshma Swaminathan f28af0cf9fef383328fb497da72e448e579b4b36

    Integration tests for BQ Multisink

    • integration-test-remote/pom.xml (version f28af0cf9fef383328fb497da72e448e579b4b36)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryMultiSinkTest.java (version f28af0cf9fef383328fb497da72e448e579b4b36)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryTest.java (version f28af0cf9fef383328fb497da72e448e579b4b36)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryUtils.java (version f28af0cf9fef383328fb497da72e448e579b4b36)
    • pom.xml (version f28af0cf9fef383328fb497da72e448e579b4b36)
  • Vinisha Shah <CuriousVini@users.noreply.github.com>

    Vinisha Shah <CuriousVini@users.noreply.github.com> 71173bdc33f80bc1810373c4a20d2ecafeb38c52

    Merge pull request #1073 from cdapio/bugfix/fix-itn
    Fix broken ITN

  • Vinisha Shah

    Vinisha Shah 17b0b63dabdef19476e9cbb890269f5982e8048c

    Fix broken ITN

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DataprocETLTestBase.java (version 17b0b63dabdef19476e9cbb890269f5982e8048c)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> f9f9ced42f3367d303a4e6abbbfb34f90e02f2ad

    Merge pull request #1068 from cdapio/bugfix/dlp-test
    CDAP-16604 fix dlp test

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> b6c75e26b94094adc9fac51d9b4002ae61863078

    CDAP-16604 fix dlp test
    Fixed the dlp test, which was incorrectly counting an empty file
    as an output line of an empty string.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DLPTest.java (version b6c75e26b94094adc9fac51d9b4002ae61863078)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> df0455bf08744f5e8a0a3fd0af773906b1806ddb

    Merge pull request #1063 from cdapio/fix-integration-tests
    fix integration tests

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7113f402e6e6dad7db9604dab1036d28f9976125

    fix integration tests
    Fix the google library dependencies to versions that all use
    compatible dependencies, and remove the direct gax-grpc depdendency
    that was causing Spanner to fail.

    Also use macros in the BQ tests to both test macros and to
    workaround an issue when the CDAP instance is running on a GCE
    VM.

    • integration-test-remote/pom.xml (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DLPTest.java (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryTest.java (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleCloudSpannerTest.java (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/hub/DLPTest.java (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • integration-test-remote/src/test/java/io/cdap/cdap/test/suite/GCPSuite.java (version 7113f402e6e6dad7db9604dab1036d28f9976125)
    • pom.xml (version 7113f402e6e6dad7db9604dab1036d28f9976125)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 4537dc463645dca1411f566f0c244bc2e339f0c2

    Merge pull request #1062 from cdapio/increase-cluster-size
    increase dataproc cluster size

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> f8e81e6da79311b5ac54541fda7a64459ee714e2

    increase dataproc cluster size
    This fixes an issue where some pipelines in some environments
    are stuck forever with MapReduce jobs that stay in pending state.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DataprocETLTestBase.java (version f8e81e6da79311b5ac54541fda7a64459ee714e2)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 12e20df6a676e58aff38d8f07b13d26552402367

    Merge pull request #1061 from cdapio/try-ordering-gcp-tests
    explicitly list tests in the GCP suite

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> c633d4ab031264f272889584a910f81d8d8b6bcd

    don't wait for status before starting

    • integration-test-remote/src/test/java/io/cdap/cdap/apps/fileset/FileSetTest.java (version c633d4ab031264f272889584a910f81d8d8b6bcd)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> e45bb8fd3dac07813a9466550a9cbca7f119a8e4

    Merge pull request #1060 from cdapio/increase-time-between-polls
    CDAP-16454 increase time between status polls

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 36ad42346cdb0f56bfa62de306c193f14c4d45d2

    explicitly list tests in the GCP suite
    Explicitly listing out tests in the GCP suite to control the
    order and see if some failures can be explained by ordering.

    • integration-test-remote/src/test/java/io/cdap/cdap/test/suite/GCPSuite.java (version 36ad42346cdb0f56bfa62de306c193f14c4d45d2)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 161f05fa633bc2d1277169ebd852864b7d4ffcc6

    CDAP-16454 increase time between status polls
    Increase the time between program status polls, since most programs
    take minutes to run and don't need to be polled every 50 milliseconds.
    This also greatly reduces the amount of logging, making it
    easier to investigate failures.

    • integration-test-core/src/main/java/io/cdap/cdap/test/AudiTestBase.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/KVTableWithProjectionTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/TPFSAvroSinkSourceTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/BatchAggregatorTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/BatchCubeSinkTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/BatchJoinerTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/ETLMapReduceTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/ExcelInputReaderTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/HivePluginTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/NormalizeTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/RowDenormalizerTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/TPFSParquetSinkSourceTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/ValueMapperTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/XMLParserTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/XMLReaderTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DataprocETLTestBase.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/PubSubTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/realtime/DataStreamsTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/wrangler/WranglerServiceTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/wrangler/WranglerTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/fileset/PermissionTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/mapreduce/readless/ReadlessIncrementTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/restart/HangingWorkerTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/serviceworker/ServiceWorkerTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/ApplicationTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/dataset/DatasetTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/fileset/FileSetTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/fileset/PartitionCorrectorTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/fileset/PartitionedFileSetUpdateTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/metadata/ProgramMetadataStressTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/metadata/ProgramMetadataTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/report/ReportGenerationAppTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/schedule/ScheduleTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/spark/SparkPageRankAppTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/apps/workflow/WorkflowTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
    • integration-test-remote/src/test/java/io/cdap/cdap/security/AppImpersonationTest.java (version 161f05fa633bc2d1277169ebd852864b7d4ffcc6)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 82f085e14e04ac4db8a4afb3cdc4f73022f2ab30

    Merge pull request #1058 from cdapio/increase-dataproc-resources
    make dataproc worker resources configurable

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7e18cc5d8ac7ddf118cb58b94cefb957bd68517a

    make dataproc worker resources configurable
    Add a way to modify the dataproc worker cpus and memory through
    passing in system properties. Also set the default to a higher
    value, since some pipelines seem to be stuck waiting for
    resources.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DataprocETLTestBase.java (version 7e18cc5d8ac7ddf118cb58b94cefb957bd68517a)
  • Venudhar Ravishankar <venu.ravishankar@gmail.com>

    Venudhar Ravishankar <venu.ravishankar@gmail.com> 878122bb62f29bfb707f45fc724a408545860256

    Merge pull request #1057 from cdapio/feature/CDAP-16439-dlp-integ-tests
    [CDAP-16439] Added cases for DLP plugins

  • Venudhar Ravishankar

    Venudhar Ravishankar 3895f38c4b2230c2ac8befdafff58303d39e5def

    [CDAP-16439] Added cases for DLP plugins

    • integration-test-remote/pom.xml (version 3895f38c4b2230c2ac8befdafff58303d39e5def)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/hub/DLPTest.java (version 3895f38c4b2230c2ac8befdafff58303d39e5def)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7e19abdd205b4a52461c1e10822afff44b877b75

    Merge pull request #1056 from cdapio/merge-release-6-1
    Merge release 6 1

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> fec799373e5d2fb71cd7f6a42f1f777a0f409011

    Merge branch 'release/6.1' into merge-release-6-1

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 06894ffb553d2cf7ab4db0c3c9cc721e104af1c4

    Merge pull request #1055 from cdapio/bugfix/CDAP-16430-fix-hub-plugin-deploy
    CDAP-16430 fix deploy hub plugin

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 349b7ae6c543ae3133b8d01265cece473e834076

    CDAP-16430 fix deploy hub plugin
    Fix the deploy hub plugin method to use the public CDAP REST APIs
    instead of some UI endpoint that is subject to change and which is
    currently broken.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/ETLTestBase.java (version 349b7ae6c543ae3133b8d01265cece473e834076)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/HivePluginTest.java (version 349b7ae6c543ae3133b8d01265cece473e834076)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/hub/PluginAttributes.java (version 349b7ae6c543ae3133b8d01265cece473e834076)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 9607e8dc83a7b1ded1c4bbde09b852b44a08ae15

    Merge pull request #1053 from cdapio/bugfix/CDAP-16429-fix-pubsub-stop-flakiness
    CDAP-16429 log a warning if pipeline cleanup fails

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> adcdd7d6d9c5f7bf43bf2999bd9ebb286da2d5fc

    CDAP-16429 log a warning if pipeline cleanup fails
    The pubsub test was flaky because it would occasionally get a 400
    error that the pipeline is not running when the stop call is made.
    Changed the test to just log a warning in this scenario since the
    test was actually successful. CDAP may have some underlying bugs
    causing this behavior though.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/PubSubTest.java (version adcdd7d6d9c5f7bf43bf2999bd9ebb286da2d5fc)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 410b5d84501df62f1cae139af8f31c8ceffbd1d7 m

    Merge pull request #1040 from cdapio/remove-metric-assertions-datastore
    remove datastore metric assertions

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> a53b770fdebd8ed111c3bd73ee1b9fa9ef737b2e m

    remove datastore metric assertions
    Checking metrics is very unreliable due to race conditions. It can
    fail incorrectly because they are not polled for a long enough time,
    and they can pass incorrectly because the number being checked is
    too low.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleCloudDatastoreTest.java (version a53b770fdebd8ed111c3bd73ee1b9fa9ef737b2e)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> a8b593de1f6d3fa0ca846ec73293e1c632bd2a85 m

    Merge pull request #1037 from cdapio/disable-yarn-pmem-check
    disable yarn pmem check

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> c1bf76daab79bd501d86326f5bf6638418de5afe m

    disable yarn pmem check
    CDAP services are getting killed by yarn due to physical memory
    constraints. Disable the pmem check to avoid this.

    • conf/cdap.json (version c1bf76daab79bd501d86326f5bf6638418de5afe)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 08dcee5bdfd9c1094ccb72ac5f7ed3b87b65df69 m

    Merge pull request #1036 from cdapio/CDAP-15804-gcp-test-macros
    CDAP-15804 use macros in tests for gcp plugins

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 18c0f67a3bbdda9103e03fe200bf09e448749f2d m

    CDAP-15804 use macros in tests for gcp plugins
    Enhanced bigquery, datastore, and spanner tests to use macros.

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/DataprocETLTestBase.java (version 18c0f67a3bbdda9103e03fe200bf09e448749f2d)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryTest.java (version 18c0f67a3bbdda9103e03fe200bf09e448749f2d)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleCloudDatastoreTest.java (version 18c0f67a3bbdda9103e03fe200bf09e448749f2d)
    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleCloudSpannerTest.java (version 18c0f67a3bbdda9103e03fe200bf09e448749f2d)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> a8599f005650c36465311faf53190ff3cafd5044 m

    Merge pull request #1035 from cdapio/remove-validator-test-6-1
    remove validator test

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> ecb9085c50b2fd15f3e1e130f6524049ce99bee0 m

    remove validator test
    the validator plugin was removed, so removing the test as well

    • integration-test-remote/src/test/java/io/cdap/cdap/app/etl/batch/ETLMapReduceTest.java (version ecb9085c50b2fd15f3e1e130f6524049ce99bee0)