Cask Data Application Platform - Develop Build and Test

Build: #3067 was successful Child of CDAP-DRC-5091

Code commits

caskdata/cdap

  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 3e8fd78330e17e3d907dd8222bd04748584b60b5

    Merge pull request #12782 from cdapio/feature/CDAP-17287-collect-spark-history
    (CDAP-17287) Collects Spark execution event logs

  • Terence Yim

    Terence Yim 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427

    (CDAP-17287) Collection of Spark execution event logs
    - Introducing the Spark event logging and upload of logs through the runtime service.
      Based on the cConf settings, the event logging collection will be on/off regardless
      of the Spark job spark.eventLog.* configurations.
    - Refactoring of the SparkRuntimeEnv class to simplify the logic by using CompletableFuture.
    - Make runtime system services shutdown after the SparkExecutionContext stopped.
      Currently they stop concurrently. It is not a problem before this PR since we
      are not doing any important work at the context stop.
      However, this PR introduces uploading the event logs at the context stop,
      which needs to be completed before the system services shutdown and process termination.

    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/runtime/monitor/RuntimeHandler.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-app-fabric/src/test/java/io/cdap/cdap/internal/app/runtime/monitor/RuntimeClientServerTest.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-app-fabric/src/test/java/io/cdap/cdap/internal/app/runtime/monitor/RuntimeClientServiceTest.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-app-fabric/src/test/java/io/cdap/cdap/internal/app/runtime/monitor/RuntimeServiceRoutingTest.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-common/src/main/java/io/cdap/cdap/common/conf/Constants.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-common/src/main/resources/cdap-default.xml (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-master/src/main/java/io/cdap/cdap/master/environment/k8s/RuntimeServiceMain.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/distributed/DistributedSparkProgramRunner.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/distributed/SparkDriverService.java (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-spark-core-base/src/main/scala/io/cdap/cdap/app/runtime/spark/AbstractSparkExecutionContext.scala (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
    • cdap-spark-core-base/src/main/scala/io/cdap/cdap/app/runtime/spark/SparkRuntimeEnv.scala (version 63aa9a75cc72dbe0cc11b61f7a15fc0c12274427)
  • Ajai <1452845+ajainarayanan@users.noreply.github.com>

    Ajai <1452845+ajainarayanan@users.noreply.github.com> 4a8898a7fc4cc7abf128b03a9360e236e725f989

    Merge pull request #12778 from cdapio/bugfix-ui/CDAP-17264-6.3
    [CDAP-17264] [CDAP-17268] [CDAP-17267] Fixes handling undefined schemas in schema editor

  • Ajai Narayan

    Ajai Narayan f5f537adb15eaec5816770eb0ade08e696aaafc0 m

    Fixes SchemaParser to default keys in map type to string when not present

    • cdap-ui/app/cdap/components/AbstractWidget/SchemaEditor/Context/SchemaParser.ts (version f5f537adb15eaec5816770eb0ade08e696aaafc0)
  • Ajai Narayan

    Ajai Narayan 7bb1b6ee4d42b1f3ae55da062c85c4a8145a2f6e m

    Adds additional check while copying nodes in pipeline canvas

    • cdap-ui/app/directives/dag-plus/my-dag-ctrl.js (version 7bb1b6ee4d42b1f3ae55da062c85c4a8145a2f6e)
  • Ajai Narayan

    Ajai Narayan 9b75ac54c4828ebdf191d3e929735fb7f0a7d5b3 m

    [CDAP-17624] Fixes schema editor to handle undefined schemas and default to an empty schema editor