Cask Data Application Platform - Develop Build and Test

Build: #3112 was successful Child of CDAP-DRC-5146

Build result summary


142 minutes
57e5469d89a279aebeacb59b4c1caab3a5f6b0bc 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc
Total tests
First to pass since
#3111 (Child of CDAP-DRC-5145)

Code commits

Author Commit Message Commit date
Terence Yim <> Terence Yim <> 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc Merge pull request #12879 from cdapio/feature/CDAP-17439-hadoop-3
(CDAP-17439) Added runtime support for Hadoop + Spark 3
Terence Yim Terence Yim 4570295c99bce9aab75fb2d1929193c7f777140c m 4570295c99bce9aab75fb2d1929193c7f777140c (CDAP-17439) Added runtime support for Hadoop + Spark 3
This change added support for Hadoop 3 and Spark 3, yet maintaining compatibility with
Hadoop 2 and Spark 1 and Spark 2.

Hadoop YARN
* Don’t use YARN localization to expand program jar file due to YARN bug YARN-9591
** In twill containers, expand the jar locally
** In Spark, renames the jar to .zip before localization

Guava depdendency
* Add missing methods to the Guava library
** Various missing Preconditions.checkArgument() methods
** Various missing Preconditions.checkState() methods
** MoreExecutors.directExecutor() method

Hadoop 3
* Upgrade to Avro 1.8.2
** SpecificData constructor becomes public
* Remove usage of ConcurrentHashSet from the jetty library as Hadoop 3 uses an incompatible version, and CDAP shouldn’t be using that library outside of the cdap-security module

Spark 3
* Use Reflection to alter SparkListenerApplicationStart event
** Spark 2 and 3 are binary incompatible for that class

Also, there is some small refactoring to modernize the code to use the standard Java library instead of Guava

JIRA issues

Unknown Issue TypeCDAP-17439Could not obtain issue details from JIRA
Unknown Issue TypeYARN-9591Could not obtain issue details from JIRA