Build #678

Build: #678 was successful Child of CDAP-DRC-4262

Code commits

caskdata/cdap

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 28456716dc26320ccfaae0c1ba2dcec3114197ef

    Merge pull request #10993 from cdapio/feature/CDAP-14827-get-namespace-info
    CDAP-14827 Added admin call to get namespace information

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 2234e8f27b15c1d5a4a875ae77feaa25cac4c673 m

    CDAP-14827 Added admin call to get namespace information
    Added a method to get information about a namespace through the
    Admin. Also added a generation field to namespaces that can
    be used by consumers to know when a namespace has been deleted
    and then recreated.

    This feature will be used in dataprep to include the namespace
    generation in entity row keys to avoid showing entities from
    a previously existing namespace that was deleted and then
    recreated.

    • cdap-api/src/main/java/co/cask/cdap/api/Admin.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-api/src/main/java/co/cask/cdap/api/NamespaceSummary.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/main/java/co/cask/cdap/app/runtime/scheduler/SchedulerQueueResolver.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/main/java/co/cask/cdap/gateway/handlers/NamespaceHttpHandler.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/runtime/DefaultAdmin.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/bootstrap/executor/DefaultNamespaceCreator.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/test/java/co/cask/cdap/internal/AppFabricClient.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/test/java/co/cask/cdap/internal/app/services/http/handlers/NamespaceHttpHandlerTest.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-app-fabric/src/test/java/co/cask/cdap/internal/app/store/DefaultStoreTest.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-common/src/main/java/co/cask/cdap/common/namespace/NamespaceAdmin.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-common/src/test/java/co/cask/cdap/common/test/NoopAdmin.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-operational-stats-core/src/test/java/co/cask/cdap/operations/cdap/CDAPOperationalStatsTest.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-proto/src/main/java/co/cask/cdap/proto/NamespaceMeta.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-security/src/main/java/co/cask/cdap/security/authorization/DefaultAuthorizationContext.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-unit-test/src/test/java/co/cask/cdap/admin/AdminApp.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
    • cdap-unit-test/src/test/java/co/cask/cdap/admin/AdminAppTestRun.java (version 2234e8f27b15c1d5a4a875ae77feaa25cac4c673)
  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> 5ef98ddf2c2e0c1895199b6c0d7da24510883bb5

    Merge pull request #11025 from cdapio/feature/CDAP-14646-isolate-master-services
    (CDAP-14646) Create isolated main classes for TMS and Metrics services

  • Terence Yim

    Terence Yim 0b3778ea45e569e6bdd2ca530c2350a7f153718c

    (CDAP-14646) Introduct cdap-master-spi module
    - Created MessagingServiceMain to run as isolated process
    - Created MetricsServiceMain to run as isolated process
    - Introduced K8s implementation of cdap-master-spi

    • cdap-common/src/main/java/co/cask/cdap/common/conf/CConfigurationUtil.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-common/src/main/java/co/cask/cdap/common/conf/Constants.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-common/src/main/resources/cdap-default.xml (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-common/src/test/java/co/cask/cdap/common/conf/CConfigurationMapTest.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-kubernetes/pom.xml (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-kubernetes/src/main/java/co/cask/cdap/k8s/discovery/KubeDiscoveryService.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-kubernetes/src/main/java/co/cask/cdap/master/environment/k8s/KubeMasterEnvironment.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-kubernetes/src/main/resources/META-INF/services/META-INF/services/co.cask.cdap.master.spi.environment.MasterEnvironment (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master-spi/pom.xml (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master-spi/src/main/java/co/cask/cdap/master/spi/environment/MasterEnvironment.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master-spi/src/main/java/co/cask/cdap/master/spi/environment/MasterEnvironmentContext.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/pom.xml (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/data/runtime/main/MessagingServiceTwillRunnable.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/DefaultMasterEnvironmentContext.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/MasterEnvironmentExtensionLoader.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/AbstractServiceMain.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MessagingServiceMain.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/main/java/co/cask/cdap/master/environment/k8s/MetricsServiceMain.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/test/java/co/cask/cdap/master/environment/MockMasterEnvironment.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/test/java/co/cask/cdap/master/environment/k8s/MasterMainTestBase.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/test/java/co/cask/cdap/master/environment/k8s/MessagingServiceMainTest.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/test/java/co/cask/cdap/master/environment/k8s/MetricsServiceMainTest.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-master/src/test/resources/META-INF/services/co.cask.cdap.master.spi.environment.MasterEnvironment (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • cdap-tms/src/main/java/co/cask/cdap/messaging/server/MessagingHttpService.java (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
    • pom.xml (version 0b3778ea45e569e6bdd2ca530c2350a7f153718c)
  • Andreas Neumann <andreas@cask.co>

    Andreas Neumann <andreas@cask.co> 1d3ef51cac52561a081be8e1812d28bc8ea68af0

    Merge pull request #11019 from cdapio/feature/CDAP-14584-elastic-meta-storage
    [CDAP-15584] Add an Elasticsearch Metadata Storage Provider

  • Andreas Neumann

    Andreas Neumann a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5 m

    [CDAP-15584] Add an Elasticsearch Metadata Storage Provider

    • cdap-api/src/main/java/co/cask/cdap/api/metadata/MetadataScope.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/Metadata.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/MetadataKind.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/MetadataStorage.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/ScopedName.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/ScopedNameOfKind.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/dataset/DatasetMetadataStorage.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/dataset/SearchHelper.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/main/java/co/cask/cdap/spi/metadata/noop/NoopMetadataStorage.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/test/java/co/cask/cdap/spi/metadata/MetadataStorageTest.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-data-fabric/src/test/java/co/cask/cdap/spi/metadata/dataset/DatasetMetadataStorageTest.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/Cursor.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/ElasticsearchMetadataStorage.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/MetadataDocument.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/main/java/co/cask/cdap/metadata/elastic/ScopedNameTypeAdapter.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/main/resources/index.mapping.json (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
    • cdap-elastic/src/test/java/co/cask/cdap/metadata/elastic/ElasticsearchMetadataStorageTest.java (version a06bee86d09aeb7c8baa81d74aaf6f1c63d9e0d5)
  • Terence Yim <885032+chtyim@users.noreply.github.com>

    Terence Yim <885032+chtyim@users.noreply.github.com> b8b6db70b596a20f3bdaa165b2e961ee28784387

    Merge pull request #11024 from cdapio/feature/CDAP-14869-remove-direct-metric-store-usage
    (CDAP-14869) Remove direct metric store usage

  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> f68219323ef4becba2f6b5df95fc2f6b8fbd75d2

    Merge pull request #11016 from cdapio/feature/CDAP-14855-healthcheck-endpoints-system-apps
    CDAP-14855 add healthcheck endpoints to pipeline and report apps

  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> c771f12259b04e7b2addfd13f222d15b53bfce93

    Merge pull request #11017 from cdapio/bugfix-ui/CDAP-14856-minimap-interaction-detail-view
    [CDAP-14856] fix minimap interaction in pipeline detail view

  • Terence Yim

    Terence Yim 2520decea9972880704baab98bed89d794820d76 m

    (CDAP-14869) Remove direct usage of MetricStore in non-metric system code
    - Separate out the TMS polling logic into two classes
      - The generic polling logic doesn’t manage subscriber state
      - The more specific subscriber service added subscriber state management
    - Remove reliance on DatasetFramework for Metrics table
    - Make MetricDeleteQuery serializable
    - Introduce MetricsSystemClient and implementation
    - Introduce MetricsAdminSubscriberService for admin messages processing
    - Replace all production usages of MetricStore to MetricsSystemClient

    • cdap-app-fabric/src/main/java/co/cask/cdap/app/mapreduce/DistributedMRJobInfoFetcher.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/app/mapreduce/LocalMRJobInfoFetcher.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/gateway/handlers/WorkflowStatsSLAHttpHandler.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/deploy/LocalApplicationManager.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/deploy/pipeline/DeletedProgramHandlerStage.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/namespace/DefaultNamespaceResourceDeleter.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/app/services/ApplicationLifecycleService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-app-fabric/src/main/java/co/cask/cdap/internal/profile/ProfileService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-common/src/main/java/co/cask/cdap/common/conf/Constants.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-common/src/main/java/co/cask/cdap/common/metrics/NoOpMetricsSystemClient.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-common/src/main/resources/cdap-default.xml (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-data-fabric-tests/src/test/java/co/cask/cdap/data2/dataset2/lib/table/hbase/MetricHBaseTableUtilTest.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-data-fabric/src/main/java/co/cask/cdap/data/runtime/SystemDatasetRuntimeModule.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-data-fabric/src/main/java/co/cask/cdap/data2/dataset2/lib/table/hbase/HBaseMetricsTableDefinition.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-master/src/main/java/co/cask/cdap/data/runtime/main/MetricsProcessorTwillRunnable.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-master/src/main/java/co/cask/cdap/data/runtime/main/MetricsTwillRunnable.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-master/src/main/java/co/cask/cdap/data/tools/UpgradeTool.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-master/src/main/java/co/cask/cdap/master/startup/ConfigurationCheck.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-master/src/test/java/co/cask/cdap/master/startup/ConfigurationCheckTest.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-operational-stats-core/src/main/java/co/cask/cdap/operations/cdap/CDAPLoad.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-operational-stats-core/src/main/java/co/cask/cdap/operations/cdap/CDAPTransactions.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-tms/src/main/java/co/cask/cdap/messaging/subscriber/AbstractMessagingPollingService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-tms/src/main/java/co/cask/cdap/messaging/subscriber/AbstractMessagingSubscriberService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog-api/src/main/java/co/cask/cdap/api/metrics/MetricDeleteQuery.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog-api/src/main/java/co/cask/cdap/api/metrics/MetricsSystemClient.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/guice/DistributedMetricsClientModule.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/guice/MetricsClientRuntimeModule.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/DirectMetricsSystemClient.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/MessagingMetricsProcessorService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/MetricsAdminMessage.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/MetricsAdminSubscriberService.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/MetricsConsumerMetaTable.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/process/RemoteMetricsSystemClient.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/store/DefaultMetricDatasetFactory.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/main/java/co/cask/cdap/metrics/store/LocalMetricsDatasetFactory.java (version 2520decea9972880704baab98bed89d794820d76)
    • cdap-watchdog/src/test/java/co/cask/cdap/metrics/process/MetricsAdminSubscriberServiceTest.java (version 2520decea9972880704baab98bed89d794820d76)
  • Edwin Elia <edwinelia@google.com>

    Edwin Elia <edwinelia@google.com> 916c1eaff2a794ec3d62812ed730d2d74bfeb806 m

    [CDAP-14856] fix minimap interaction in pipeline detail view

    • cdap-ui/app/directives/dag-minimap/dag-minimap-ctrl.js (version 916c1eaff2a794ec3d62812ed730d2d74bfeb806)
  • albertshau <ashau@google.com>

    albertshau <ashau@google.com> 9770e847d05ca29b6022f4f10092a6c26051ed2b m

    CDAP-14855 add healthcheck endpoints to pipeline and report apps

    • cdap-app-templates/cdap-etl/cdap-data-pipeline/src/main/java/co/cask/cdap/datapipeline/service/ValidationHandler.java (version 9770e847d05ca29b6022f4f10092a6c26051ed2b)
    • cdap-app-templates/cdap-program-report/src/main/java/co/cask/cdap/report/ReportGenerationSpark.java (version 9770e847d05ca29b6022f4f10092a6c26051ed2b)