Build #3,022

Cask Data Application Platform - Develop Build and Test

Build: #3022 failed Child of CDAP-DRC-5031

Code commits

caskdata/cdap

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> b1f303d704750c8a8fad879020e8a874ea4d8eff

    Merge pull request #12558 from cdapio/feature/CDAP-17078-spark-stage-consolidation
    CDAP-17078 consolidate stages within a group

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb m

    CDAP-17078 consolidate stages within a group
    Changed the SparkPipelineRunner to use a CombinerDag to group
    sinks and their preceding transforms together. These grouped
    stages are treated similarly to how a single sink is treated,
    with flatMapToPair() called on the input RDD to transform it
    into a PairRDD, then calling save() to write the RDD out.
    This capability is off by default, but can be turned on by
    setting a runtime argument.

    Instead of flatMapToPair() calling just the sink's transform
    method, a new MultiSinkFunction class is used to direct incoming
    records to the correct logical branches of the pipeline.
    This requires that each input be tagged with which stage it
    came from (stage and port), as well as its type (output, or error).
    In order to do this, refactored the SparkPipelineRunner a bit
    to maintain the RDD<RecordInfo> for each stage rather than
    RDD<StructuredRecord>, as the RecordInfo class contains that
    extra information.

    Also added a MultiOutputFormat that will take the output of the
    MultiSinkFunction and delegate writes to the correct underlying
    OutputFormat. Since the OutputFormat lives in the pipeline
    app, this approach means CDAP datasets cannot be combined.
    This caused a problem with dataset lineage, since it is
    implemented by implemented by wrapping OutputFormats into a hidden
    ExternalDataset class in CDAP. Instead of doing this indirect
    wrapping, changed the SparkSinkFactory class to explicitly
    register lineage through direct calls instead of hiding it
    under several layers of abstraction.

    • cdap-app-templates/cdap-etl/cdap-data-pipeline/src/test/java/io/cdap/cdap/datapipeline/DataPipelineTest.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-data-streams/src/main/java/io/cdap/cdap/datastreams/SparkStreamingPipelineDriver.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-batch/src/main/java/io/cdap/cdap/etl/batch/customaction/BasicActionContext.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/Constants.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/ExternalDatasets.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/PhaseSpec.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/PipelinePhase.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/PipelineRuntime.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/output/MultiOutputCommitter.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/output/MultiOutputFormat.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/output/MultiRecordWriter.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/exec/TransformExecutorFactory.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/planner/Dag.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/SparkCollection.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/SparkPipelineRunner.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/SparkTransformExecutorFactory.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/BaseRDDCollection.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/BatchSparkPipelineDriver.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/ETLSpark.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/OutputFormatProviderTypeAdapter.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/SparkBatchSinkContext.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/SparkBatchSinkFactory.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/SparkPreparer.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/function/MultiSinkFunction.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/streaming/DStreamCollection.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/streaming/DefaultStreamingSourceContext.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/streaming/function/RecordInfoWrapper.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/streaming/function/StreamingBatchSinkFunction.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-spark-core/src/main/java/io/cdap/cdap/etl/spark/SparkTransformExecutorFactory.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
    • cdap-app-templates/cdap-etl/hydrator-test/src/main/java/io/cdap/cdap/etl/mock/transform/IntValueFilterTransform.java (version 7e4f1e1fe9cf01b9fa3efeb05b3c76865a0633bb)
  • sagarkapare <sagarkapare@google.com>

    sagarkapare <sagarkapare@google.com> 3b90e4091934ebdcb8353dd87c10a47d7855a51d

    Merge pull request #12524 from cdapio/feature/CDAP-16712-remote-fetcher-impl
    [CDAP-16712] Separate out preview manager and preview runners so that they can be run independently in their own containers.

  • Jenna Choi <jena.choi1106@gmail.com>

    Jenna Choi <jena.choi1106@gmail.com> 5686de64c4e234f11ec9c6bffab375fc5257502a

    Merge pull request #12573 from cdapio/feature-ui/CDAP-17167
    [CDAP-17167] Resolve CDAP UI unit test failure

  • Jenna Choi <jena.choi1106@gmail.com>

    Jenna Choi <jena.choi1106@gmail.com> a89a829ffaef4ac72db9846a23c66dfdf38e0f46

    [CDAP-17167] Resolve CDAP UI unittest failure

    • cdap-ui/.babelrc (version a89a829ffaef4ac72db9846a23c66dfdf38e0f46)
  • Anudeep Katragadda <51677126+itsanudeep@users.noreply.github.com>

    Anudeep Katragadda <51677126+itsanudeep@users.noreply.github.com> 75d97410676db4570ed22aa009354bf16c8eb7ab

    Merge pull request #12562 from cdapio/feature-ui/CDAP-17157
    [CDAP-17157] System delay snackbar changes

  • sagarkapare <sagarkapare@google.com>

    sagarkapare <sagarkapare@google.com> d69fc4b0eb0691156e515343e0fa87e72e409471 m

    [CDAP-16712] Implementation of RemotePreviewRequestFetcher. Separating preview manager and preview runner to be run in different pods.

    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/DefaultPreviewRunnerManager.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewConfigModule.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewHttpServer.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewManager.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewRequestQueue.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewRunnerManager.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewRunnerManagerModule.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/app/preview/PreviewRunnerServiceFactory.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/gateway/handlers/preview/PreviewHttpHandlerInternal.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/DefaultPreviewManager.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/DefaultPreviewRequestQueue.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/DirectPreviewRequestFetcher.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/DirectPreviewRequestFetcherFactory.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/PreviewRequestPollerInfoProvider.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/PreviewRunStopper.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/PreviewRunnerService.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/PreviewRunnerServiceStopper.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/PreviewTMSLogSubscriber.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/preview/RemotePreviewRequestFetcher.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/runtime/k8s/PreviewRequestPollerInfo.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/main/java/io/cdap/cdap/internal/app/runtime/k8s/PreviewRunnerOptions.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/test/java/io/cdap/cdap/internal/app/preview/DefaultPreviewManagerTest.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-app-fabric/src/test/java/io/cdap/cdap/internal/app/preview/PreviewRunnerServiceTest.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-common/src/main/resources/cdap-default.xml (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-master/src/main/java/io/cdap/cdap/master/environment/k8s/PreviewServiceMain.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-master/src/test/java/io/cdap/cdap/master/environment/k8s/MasterServiceMainTestBase.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-master/src/test/java/io/cdap/cdap/master/environment/k8s/PreviewServiceMainTest.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-standalone/src/main/java/io/cdap/cdap/StandaloneMain.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-standalone/src/test/java/io/cdap/cdap/StandaloneMainTest.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-unit-test/src/main/java/io/cdap/cdap/test/TestBase.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-watchdog/src/main/java/io/cdap/cdap/logging/appender/tms/PreviewTMSLogAppender.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
    • cdap-watchdog/src/main/java/io/cdap/cdap/logging/appender/tms/TMSLogAppender.java (version d69fc4b0eb0691156e515343e0fa87e72e409471)
  • itsanudeep <itsanudeep@google.com>

    itsanudeep <itsanudeep@google.com> 902c5e227ceb8667f6c23ffafecd6f8c4cf72db6 m

    snackbar positioning and styling

    • cdap-ui/app/cdap/components/SystemServicesDelay/index.tsx (version 902c5e227ceb8667f6c23ffafecd6f8c4cf72db6)