đź”—CDAP Documentation v4.1.1

The Cask™ Data Application Platform (CDAP) is an integrated, open source application development platform for the Apache Hadoop® ecosystem that provides developers with data and application abstractions to simplify and accelerate application development, address a broader range of real-time and batch use cases, and deploy applications into production while satisfying enterprise requirements.

CDAP is a layer of software running on top of Hadoop platforms such as the Cloudera Enterprise Data Hub, the Hortonworks® Data Platform, or the MapR Distribution. CDAP provides these essential capabilities:

  • Abstraction of data in the Hadoop environment through logical representations of underlying data;
  • Portability of applications through decoupling underlying infrastructures;
  • Services and tools that enable faster application creation in development;
  • Integration of the components of the Hadoop ecosystem into a single platform; and
  • Higher degrees of operational control in production through enterprise best practices.

CDAP exposes developer APIs (Application Programming Interfaces) for creating applications and accessing core CDAP services. CDAP defines and implements a diverse collection of services that support applications and data on existing Hadoop infrastructure such as HBase, HDFS, YARN, MapReduce, Hive, and Spark.

These documents are your complete reference to the Cask Data Application Platform: they help you get started and set up your development environment; explain how CDAP works; and teach how to develop and test CDAP applications.

It includes the CDAP programming APIs and client interfaces, with instructions on the installation, monitoring and diagnosing fully distributed CDAP in a Hadoop cluster. CDAP capabilities include CDAP Pipelines and Metadata Management.

  • Developers’ Manual:
    • Getting Started Developing: A quick, hands-on introduction to developing with CDAP
    • Overview: The overall architecture, abstractions, modes, and components behind CDAP
    • Building Blocks: The two core abstractions in CDAP: Data and Applications, and their components
    • Metadata: CDAP can automatically capture metadata and let you see how data is flowing
    • Pipelines: A capability of CDAP that that allows users to build, deploy, and manage data pipelines
    • Security: Perimeter security, configuration and client authentication
    • Testing and Debugging: Test framework plus tools and practices for debugging your applications
    • Ingesting Data: Different techniques for ingesting data into CDAP
    • Advanced Topics: Adding a custom logback, best practices for CDAP development, class loading in CDAP, configuring program resources and retry policies
  • Administration Manual:
    • Installation: Putting CDAP into production, with installation, configuration and upgrading for different distributions
    • Security: CDAP supports securing clusters using a perimeter security model
    • Operations: Logging, monitoring, metrics, runtime arguments, scaling instances, resource guarantees, transaction service maintenance, and introduces the CDAP UI
    • Appendices: Covers the CDAP installation and security configuration files
  • Integrations:
    • Cask Market: A source for re-usable applications, data, and code for all CDAP users
    • Cloudera: Integrating CDAP into Cloudera, using Cloudera Manager, running interactive queries with Impala, and bridging CDAP Metadata with Cloudera's data management tool, Navigator
    • Apache Sentry: Configuring and integrating CDAP with Apache Sentry
    • Apache Hadoop KMS: Configuring and integrating CDAP with Apache Hadoop Key Management Service (KMS)
    • JDBC: The CDAP JDBC driver, included with CDAP
    • ODBC: The CDAP ODBC driver available for CDAP
    • Pentaho: Pentaho Data Integration, a business intelligence tool that can be used with CDAP
    • Squirrel: SquirrelSQL, a simple JDBC client that can be integrated with CDAP
  • Glossary: Definitions of terms and phrases used in CDAP
  • Search: Search this documentation