Cask Data Application Platform - Develop Build and Test

Build: #3112 was successful Child of CDAP-DRC-5146

Build result summary

Details

Completed
Duration
142 minutes
Labels
None
Agent
bamboo-agent14
Revision
57e5469d89a279aebeacb59b4c1caab3a5f6b0bc 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc
Total tests
2866
First to pass since
#3111 (Child of CDAP-DRC-5145)

Code commits

Author Commit Message Commit date
Terence Yim <885032+chtyim@users.noreply.github.com> Terence Yim <885032+chtyim@users.noreply.github.com> 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc 57e5469d89a279aebeacb59b4c1caab3a5f6b0bc Merge pull request #12879 from cdapio/feature/CDAP-17439-hadoop-3
(CDAP-17439) Added runtime support for Hadoop + Spark 3
Terence Yim Terence Yim 4570295c99bce9aab75fb2d1929193c7f777140c m 4570295c99bce9aab75fb2d1929193c7f777140c (CDAP-17439) Added runtime support for Hadoop + Spark 3
This change added support for Hadoop 3 and Spark 3, yet maintaining compatibility with
Hadoop 2 and Spark 1 and Spark 2.

Hadoop YARN
* Don’t use YARN localization to expand program jar file due to YARN bug YARN-9591
** In twill containers, expand the jar locally
** In Spark, renames the jar to .zip before localization

Guava depdendency
* Add missing methods to the Guava library
** Various missing Preconditions.checkArgument() methods
** Various missing Preconditions.checkState() methods
** MoreExecutors.directExecutor() method

Hadoop 3
* Upgrade to Avro 1.8.2
** SpecificData constructor becomes public
* Remove usage of ConcurrentHashSet from the jetty library as Hadoop 3 uses an incompatible version, and CDAP shouldn’t be using that library outside of the cdap-security module

Spark 3
* Use Reflection to alter SparkListenerApplicationStart event
** Spark 2 and 3 are binary incompatible for that class

Also, there is some small refactoring to modernize the code to use the standard Java library instead of Guava

JIRA issues

IssueDescriptionStatus
Unknown Issue TypeCDAP-17439Could not obtain issue details from JIRA
Unknown Issue TypeYARN-9591Could not obtain issue details from JIRA