Build #540

Build: #540 failed Scheduled with changes by 6 people

Code commits

caskdata/cdap

  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> ecc70b28d6014f8265f84af78f745f8012510d30

    Merge pull request #12247 from cdapio/feature-ui/CDAP-16886-run-ui-unit-tests
    [CDAP-16886] add pom profile to run UI unit tests

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 4313c35cb2475365e03b5a95a478343805bf9bdc

    Merge pull request #12252 from cdapio/feature/CDAP-16852-auto-join-macro-schemas
    CDAP-16852 handle dynamic schemas in auto join

  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 4d3fccccac6f6563b717a41b0d2a6ac3e497f72d

    Merge pull request #12255 from cdapio/feature/CDAP-16669-long-running-dataproc
    (CDAP-16669) New Dataproc provisioner for existing cluster

  • Terence Yim

    Terence Yim f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21

    (CDAP-16669) Added a new Dataproc provisioner for existing cluster

    • cdap-master/pom.xml (version f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocClient.java (version f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/ExistingDataprocProvisioner.java (version f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21)
    • cdap-runtime-ext-dataproc/src/main/resources/META-INF/services/io.cdap.cdap.runtime.spi.provisioner.Provisioner (version f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21)
    • cdap-runtime-ext-dataproc/src/main/resources/gcp-existing-dataproc.json (version f655c17dfd40eeb2d1f5663f0ecdcf05915e7d21)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7a67c119ca4920bb33a84af541adbbe00b368cc3

    CDAP-16852 handle dynamic schemas in auto join
    Properly handle the case when the inputs into an auto-join stage
    have null schemas due to the schema being unknown at deploy time,
    usually because of macros.

    In order to support this type of use case, added a way for a
    plugin to specify the output schema in the JoinDefinition.
    This should be used when the output schema cannot be derived
    at deployment time due to macros. This is what the existing
    Joiner currently requires the user to do for dynamic join
    use cases.

    For Spark, the implementation becomes significantly more difficult
    due to the fact that the schema of each input stage needs to be
    known in the Spark driver, in order to convert RDDs into
    DataFrames.

    Added logic that derives the input schema using the output schema,
    selected fields, and join keys. It is possible to derive
    a usable schema when all the join keys are present in the final
    output schema.

    • cdap-app-templates/cdap-etl/cdap-data-pipeline/src/test/java/io/cdap/cdap/datapipeline/AutoJoinerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-data-streams/src/main/java/io/cdap/cdap/datastreams/SparkStreamingPipelineRunner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-data-streams/src/test/java/io/cdap/cdap/datastreams/DataStreamsTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/JoinDefinition.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/error/JoinError.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/error/OutputSchemaError.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/test/java/io/cdap/cdap/etl/api/join/JoinDefinitionTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-batch/src/main/java/io/cdap/cdap/etl/batch/mapreduce/MapReduceTransformExecutorFactory.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-batch/src/test/java/io/cdap/cdap/etl/batch/BatchPhaseSpecTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/AbstractStageContext.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/plugin/JoinerBridge.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/submit/PipelinePhasePreparer.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/spec/PipelineSpecGenerator.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/lineage/FieldLineageProcessorTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/planner/PipelinePlannerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/spec/PipelineSpecGeneratorTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-proto/src/main/java/io/cdap/cdap/etl/proto/v2/spec/StageSpec.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-proto/src/test/java/io/cdap/cdap/etl/proto/v2/spec/StageSpecTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/SparkPipelineRunner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/BaseRDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/function/JoinMergeFunction.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/function/JoinOnFunction.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core/src/main/java/io/cdap/cdap/etl/spark/batch/RDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core/src/test/java/io/cdap/cdap/etl/spark/SparkPipelineRunnerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core2_2.11/src/main/java/io/cdap/cdap/etl/spark/batch/RDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-test/src/main/java/io/cdap/cdap/etl/mock/batch/MockSource.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-test/src/main/java/io/cdap/cdap/etl/mock/batch/joiner/MockAutoJoiner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
  • Ajai <1452845+ajainarayanan@users.noreply.github.com>

    Ajai <1452845+ajainarayanan@users.noreply.github.com> fd28121a76a602a90e9a3abe12922b61a398fd1b

    Merge pull request #12087 from cdapio/feature-ui/CDAP-16607
    [CDAP-16607] Expands plugin configuration, plugin templates and post …

  • Terence Yim

    Terence Yim dc050313e3c1ae54fea42ea1fd6df603bcfd430b

    (CDAP-16669) Refactor Dataproc Provisioner
    - Cleanup private instance logic. Delay private instance network check to create cluster call instead of from the create() method
    - Create abstract base Dataproc provisioner for different types of Dataproc based provisioner

    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocClient.java (version dc050313e3c1ae54fea42ea1fd6df603bcfd430b)
  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 57580aff9675f66b4d09f227c712b303980af8b9

    Merge pull request #12239 from cdapio/feature/CDAP-16884-retry-pause
    (CDAP-16884) Fix SparkDriverService to pause between failure retry

  • Ajai Narayan

    Ajai Narayan e408e0e2246990a86ab7d27ed716b6b5504a6fec

    [CDAP-16607] Expands plugin configuration, plugin templates and post run action, import & confirmation modals to expand to full screen

    • cdap-ui/app/cdap/components/PostRunActions/PostRunActionsWizard.js (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/directives/my-post-run-action-wizard/my-post-run-action-wizard.js (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/directives/my-post-run-action-wizard/wizard-configure-confirm-step/wizard-configure-confirm-step.less (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/directives/plugin-templates/plugin-templates.less (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/controllers/create/leftpanel-ctrl.js (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/controllers/create/partials/nodeconfig-ctrl.js (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/hydrator-modal.less (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/leftpanel.less (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/templates/detail/bottom-panel.html (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/templates/partial/node-config-modal/popover-template.html (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/hydrator/templates/partial/node-config-modal/popover.html (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
    • cdap-ui/app/styles/themes/cdap/theme.less (version e408e0e2246990a86ab7d27ed716b6b5504a6fec)
  • Terence Yim

    Terence Yim d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f

    (CDAP-16669) Refactor Dataproc Provisioner
    - Cleanup private instance logic. Delay private instance network check to create cluster call instead of from the create() method
    - Create abstract base Dataproc provisioner for different types of Dataproc based provisioner

    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/common/DataprocUtils.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/AbstractDataprocProvisioner.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocClient.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocConf.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocProvisioner.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocRuntimeException.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/main/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocTool.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/test/java/io/cdap/cdap/runtime/spi/common/DataprocUtilsTest.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/test/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/DataprocProvisionerTest.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
    • cdap-runtime-ext-dataproc/src/test/java/io/cdap/cdap/runtime/spi/provisioner/dataproc/MockProvisionerContext.java (version d08a6ddc40c4711e14a239d0e9cc3cca33f82d1f)
  • Ajai <1452845+ajainarayanan@users.noreply.github.com>

    Ajai <1452845+ajainarayanan@users.noreply.github.com> 999bb3d1ab09219e8772f507bf79bfae229120b7

    Merge pull request #12171 from cdapio/bugfix-ui/CDAP-11661
    [CDAP-11661][CDAP-16797] Fix post run actions to remove success message + valid user feedback

  • Ajai Narayan

    Ajai Narayan 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82

    [CDAP-11661][CDAP-16797] Fix post run actions to remove success message + valid user feedback
      - [CDAP-11661] Removes success messages from saving/deleting post run actions in pipeline studio
      - [CDAP-16797] Provide feedback on invalid configuration for post run actions in pipeline studio
      - Fixes post run action modal footer to be consistent
      - Fixes post run action wizard to show proper loading icons when validating

    • cdap-ui/app/cdap/components/PostRunActions/PostRunActions.scss (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/cdap/components/PostRunActions/PostRunActionsWizard.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/my-post-run-action-wizard-ctrl.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/my-post-run-action-wizard-modal.html (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/my-post-run-action-wizard.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/wizard-configure-confirm-step/wizard-configure-confirm-step-ctrl.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/wizard-configure-confirm-step/wizard-configure-confirm-step.html (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/wizard-configure-confirm-step/wizard-configure-confirm-step.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-action-wizard/wizard-configure-confirm-step/wizard-configure-confirm-step.less (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/directives/my-post-run-actions/my-post-run-actions.js (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
    • cdap-ui/app/hydrator/hydrator-modal.less (version 16c7853d74cfe3f6dcf84e4404a09f6dde5f2a82)
  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> f875306754984d4d3ae4536520f197ce711ade29 m

    [CDAP-16886] add pom profile to run UI unit tests

    • cdap-ui/pom.xml (version f875306754984d4d3ae4536520f197ce711ade29)
  • Terence Yim

    Terence Yim 5400b9e01cac3eb0d8ed91e967ba09d300b02681 m

    (CDAP-16884) Fix SparkDriverService to pause between failure retry
    - Also added more information to the error logs

    • cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/distributed/SparkDriverService.java (version 5400b9e01cac3eb0d8ed91e967ba09d300b02681)