Build #2,942

Cask Data Application Platform - Develop Build and Test

Build: #2942 failed Child of CDAP-DRC-4918

Code commits

caskdata/cdap

  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> ecc70b28d6014f8265f84af78f745f8012510d30

    Merge pull request #12247 from cdapio/feature-ui/CDAP-16886-run-ui-unit-tests
    [CDAP-16886] add pom profile to run UI unit tests

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 4313c35cb2475365e03b5a95a478343805bf9bdc

    Merge pull request #12252 from cdapio/feature/CDAP-16852-auto-join-macro-schemas
    CDAP-16852 handle dynamic schemas in auto join

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 7a67c119ca4920bb33a84af541adbbe00b368cc3 m

    CDAP-16852 handle dynamic schemas in auto join
    Properly handle the case when the inputs into an auto-join stage
    have null schemas due to the schema being unknown at deploy time,
    usually because of macros.

    In order to support this type of use case, added a way for a
    plugin to specify the output schema in the JoinDefinition.
    This should be used when the output schema cannot be derived
    at deployment time due to macros. This is what the existing
    Joiner currently requires the user to do for dynamic join
    use cases.

    For Spark, the implementation becomes significantly more difficult
    due to the fact that the schema of each input stage needs to be
    known in the Spark driver, in order to convert RDDs into
    DataFrames.

    Added logic that derives the input schema using the output schema,
    selected fields, and join keys. It is possible to derive
    a usable schema when all the join keys are present in the final
    output schema.

    • cdap-app-templates/cdap-etl/cdap-data-pipeline/src/test/java/io/cdap/cdap/datapipeline/AutoJoinerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-data-streams/src/main/java/io/cdap/cdap/datastreams/SparkStreamingPipelineRunner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-data-streams/src/test/java/io/cdap/cdap/datastreams/DataStreamsTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/JoinDefinition.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/error/JoinError.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/main/java/io/cdap/cdap/etl/api/join/error/OutputSchemaError.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-api/src/test/java/io/cdap/cdap/etl/api/join/JoinDefinitionTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-batch/src/main/java/io/cdap/cdap/etl/batch/mapreduce/MapReduceTransformExecutorFactory.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-batch/src/test/java/io/cdap/cdap/etl/batch/BatchPhaseSpecTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/AbstractStageContext.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/plugin/JoinerBridge.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/common/submit/PipelinePhasePreparer.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/main/java/io/cdap/cdap/etl/spec/PipelineSpecGenerator.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/lineage/FieldLineageProcessorTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/planner/PipelinePlannerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-core/src/test/java/io/cdap/cdap/etl/spec/PipelineSpecGeneratorTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-proto/src/main/java/io/cdap/cdap/etl/proto/v2/spec/StageSpec.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/cdap-etl-proto/src/test/java/io/cdap/cdap/etl/proto/v2/spec/StageSpecTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/SparkPipelineRunner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/batch/BaseRDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/function/JoinMergeFunction.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core-base/src/main/java/io/cdap/cdap/etl/spark/function/JoinOnFunction.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core/src/main/java/io/cdap/cdap/etl/spark/batch/RDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core/src/test/java/io/cdap/cdap/etl/spark/SparkPipelineRunnerTest.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-spark-core2_2.11/src/main/java/io/cdap/cdap/etl/spark/batch/RDDCollection.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-test/src/main/java/io/cdap/cdap/etl/mock/batch/MockSource.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
    • cdap-app-templates/cdap-etl/hydrator-test/src/main/java/io/cdap/cdap/etl/mock/batch/joiner/MockAutoJoiner.java (version 7a67c119ca4920bb33a84af541adbbe00b368cc3)
  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> f875306754984d4d3ae4536520f197ce711ade29 m

    [CDAP-16886] add pom profile to run UI unit tests

    • cdap-ui/pom.xml (version f875306754984d4d3ae4536520f197ce711ade29)