org.apache.hadoop.yarn.client.api.impl

Class YarnClientImpl

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.YarnClient

org.apache.hadoop.yarn.client.api.impl.YarnClientImpl

All Implemented Interfaces:

Closeable, AutoCloseable, Service

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class YarnClientImpl
extends YarnClient

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected AHSClient	historyClient
protected ApplicationClientProtocol	rmClient
protected long	submitPollIntervalMillis
protected TimelineClient	timelineClient
protected boolean	timelineServiceBestEffort
protected boolean	timelineServiceEnabled

Constructor Summary

Constructors

Constructor and Description
YarnClientImpl()

Method Summary

Modifier and Type	Method and Description
YarnClientApplication	createApplication() Obtain a YarnClientApplication for a new application, which in turn contains the ApplicationSubmissionContext and GetNewApplicationResponse objects.
ReservationDeleteResponse	deleteReservation(ReservationDeleteRequest request) The interface used by clients to remove an existing Reservation.
List<QueueInfo>	getAllQueues() Get information (QueueInfo) about all queues, recursively if there is a hierarchy
org.apache.hadoop.security.token.Token<AMRMTokenIdentifier>	getAMRMToken(ApplicationId appId) Get the AMRM token of the application.
ApplicationAttemptReport	getApplicationAttemptReport(ApplicationAttemptId appAttemptId) Get a report of the given ApplicationAttempt.
List<ApplicationAttemptReport>	getApplicationAttempts(ApplicationId appId) Get a report of all (ApplicationAttempts) of Application in the cluster.
ApplicationReport	getApplicationReport(ApplicationId appId) Get a report of the given Application.
List<ApplicationReport>	getApplications() Get a report (ApplicationReport) of all Applications in the cluster.
List<ApplicationReport>	getApplications(EnumSet<YarnApplicationState> applicationStates) Get a report (ApplicationReport) of Applications matching the given application states in the cluster.
List<ApplicationReport>	getApplications(Set<String> applicationTypes) Get a report (ApplicationReport) of Applications matching the given application types in the cluster.
List<ApplicationReport>	getApplications(Set<String> applicationTypes, EnumSet<YarnApplicationState> applicationStates) Get a report (ApplicationReport) of Applications matching the given application types and application states in the cluster.
List<QueueInfo>	getChildQueueInfos(String parent) Get information (QueueInfo) about all the immediate children queues of the given queue
Set<String>	getClusterNodeLabels() The interface used by client to get node labels in the cluster
ContainerReport	getContainerReport(ContainerId containerId) Get a report of the given Container.
List<ContainerReport>	getContainers(ApplicationAttemptId applicationAttemptId) Get a report of all (Containers) of ApplicationAttempt in the cluster.
Map<String,Set<NodeId>>	getLabelsToNodes() The interface used by client to get labels to nodes mapping in existing cluster
Map<String,Set<NodeId>>	getLabelsToNodes(Set<String> labels) The interface used by client to get labels to nodes mapping for specified labels in existing cluster
List<NodeReport>	getNodeReports(NodeState... states) Get a report of nodes (NodeReport) in the cluster.
Map<NodeId,Set<String>>	getNodeToLabels() The interface used by client to get node to labels mappings in existing cluster
List<QueueUserACLInfo>	getQueueAclsInfo() Get information about acls for current user on all the existing queues.
QueueInfo	getQueueInfo(String queueName) Get information (QueueInfo) about a given queue.
Token	getRMDelegationToken(Text renewer) Get a delegation token so as to be able to talk to YARN using those tokens.
List<QueueInfo>	getRootQueueInfos() Get information (QueueInfo) about top level queues.
YarnClusterMetrics	getYarnClusterMetrics() Get metrics (YarnClusterMetrics) about the cluster.
protected boolean	isSecurityEnabled()
void	killApplication(ApplicationId applicationId) Kill an application identified by given ID.
void	moveApplicationAcrossQueues(ApplicationId appId, String queue) Attempts to move the given application to the given queue.
protected void	serviceInit(Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	setRMClient(ApplicationClientProtocol rmClient)
ApplicationId	submitApplication(ApplicationSubmissionContext appContext) Submit a new application to YARN. It is a blocking call - it will not return ApplicationId until the submitted application is submitted successfully and accepted by the ResourceManager.
ReservationSubmissionResponse	submitReservation(ReservationSubmissionRequest request) The interface used by clients to submit a new reservation to the ResourceManager.
ReservationUpdateResponse	updateReservation(ReservationUpdateRequest request) The interface used by clients to update an existing Reservation.

Methods inherited from class org.apache.hadoop.yarn.client.api.YarnClient

createYarnClient

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

rmClient

protected ApplicationClientProtocol rmClient

submitPollIntervalMillis

protected long submitPollIntervalMillis

historyClient

protected AHSClient historyClient

timelineClient

protected TimelineClient timelineClient

timelineServiceEnabled

protected boolean timelineServiceEnabled

timelineServiceBestEffort

protected boolean timelineServiceBestEffort

Constructor Detail

YarnClientImpl

public YarnClientImpl()

Method Detail

serviceInit

protected void serviceInit(Configuration conf)
                    throws Exception

Overrides:

serviceInit in class AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class AbstractService

Throws:

Exception

createApplication

public YarnClientApplication createApplication()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Obtain a YarnClientApplication for a new application, which in turn contains the ApplicationSubmissionContext and GetNewApplicationResponse objects.

Specified by:

createApplication in class YarnClient

Returns:

YarnClientApplication built for a new application

Throws:

YarnException

IOException

submitApplication

public ApplicationId submitApplication(ApplicationSubmissionContext appContext)
                                throws YarnException,
                                       IOException

Description copied from class: YarnClient

Submit a new application to YARN. It is a blocking call - it will not return ApplicationId until the submitted application is submitted successfully and accepted by the ResourceManager.

Users should provide an ApplicationId as part of the parameter ApplicationSubmissionContext when submitting a new application, otherwise it will throw the ApplicationIdNotProvidedException.

This internally calls (SubmitApplicationRequest), and after that, it internally invokes (GetApplicationReportRequest) and waits till it can make sure that the application gets properly submitted. If RM fails over or RM restart happens before ResourceManager saves the application's state, #getApplicationReport(GetApplicationReportRequest) will throw the ApplicationNotFoundException. This API automatically resubmits the application with the same ApplicationSubmissionContext when it catches the ApplicationNotFoundException

Specified by:

submitApplication in class YarnClient

Parameters:

appContext - ApplicationSubmissionContext containing all the details needed to submit a new application

Returns:

ApplicationId of the accepted application

Throws:

YarnException

IOException

See Also:

YarnClient.createApplication()

isSecurityEnabled

@InterfaceAudience.Private
protected boolean isSecurityEnabled()

killApplication

public void killApplication(ApplicationId applicationId)
                     throws YarnException,
                            IOException

Description copied from class: YarnClient

Kill an application identified by given ID.

Specified by:

killApplication in class YarnClient

Parameters:

applicationId - ApplicationId of the application that needs to be killed

Throws:

YarnException - in case of errors or if YARN rejects the request due to access-control restrictions.

IOException

See Also:

YarnClient.getQueueAclsInfo()

getApplicationReport

public ApplicationReport getApplicationReport(ApplicationId appId)
                                       throws YarnException,
                                              IOException

Description copied from class: YarnClient

Get a report of the given Application.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

If the user does not have VIEW_APP access then the following fields in the report will be set to stubbed values:

• host - set to "N/A"
• RPC port - set to -1
• client token - set to "N/A"
• diagnostics - set to "N/A"
• tracking URL - set to "N/A"
• original tracking URL - set to "N/A"
• resource usage report - all values are -1

Specified by:

getApplicationReport in class YarnClient

Parameters:

appId - ApplicationId of the application that needs a report

Returns:

application report

Throws:

YarnException

IOException

getAMRMToken

public org.apache.hadoop.security.token.Token<AMRMTokenIdentifier> getAMRMToken(ApplicationId appId)
                                                                         throws YarnException,
                                                                                IOException

Description copied from class: YarnClient

Get the AMRM token of the application.

The AMRM token is required for AM to RM scheduling operations. For managed Application Masters Yarn takes care of injecting it. For unmanaged Applications Masters, the token must be obtained via this method and set in the UserGroupInformation of the current user.

The AMRM token will be returned only if all the following conditions are met:

• the requester is the owner of the ApplicationMaster
• the application master is an unmanaged ApplicationMaster
• the application master is in ACCEPTED state

Else this method returns NULL.

Specified by:

getAMRMToken in class YarnClient

Parameters:

appId - ApplicationId of the application to get the AMRM token

Returns:

the AMRM token if available

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of all Applications in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of all running applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(Set<String> applicationTypes)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application types in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(EnumSet<YarnApplicationState> applicationStates)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application states in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(Set<String> applicationTypes,
                                               EnumSet<YarnApplicationState> applicationStates)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application types and application states in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getYarnClusterMetrics

public YarnClusterMetrics getYarnClusterMetrics()
                                         throws YarnException,
                                                IOException

Description copied from class: YarnClient

Get metrics (YarnClusterMetrics) about the cluster.

Specified by:

getYarnClusterMetrics in class YarnClient

Returns:

cluster metrics

Throws:

YarnException

IOException

getNodeReports

public List<NodeReport> getNodeReports(NodeState... states)
                                throws YarnException,
                                       IOException

Description copied from class: YarnClient

Get a report of nodes (NodeReport) in the cluster.

Specified by:

getNodeReports in class YarnClient

Parameters:

states - The NodeStates to filter on. If no filter states are given, nodes in all states will be returned.

Returns:

A list of node reports

Throws:

YarnException

IOException

getRMDelegationToken

public Token getRMDelegationToken(Text renewer)
                           throws YarnException,
                                  IOException

Description copied from class: YarnClient

Get a delegation token so as to be able to talk to YARN using those tokens.

Specified by:

getRMDelegationToken in class YarnClient

Parameters:

renewer - Address of the renewer who can renew these tokens when needed by securely talking to YARN.

Returns:

a delegation token (Token) that can be used to talk to YARN

Throws:

YarnException

IOException

getQueueInfo

public QueueInfo getQueueInfo(String queueName)
                       throws YarnException,
                              IOException

Description copied from class: YarnClient

Get information (QueueInfo) about a given queue.

Specified by:

getQueueInfo in class YarnClient

Parameters:

queueName - Name of the queue whose information is needed

Returns:

queue information

Throws:

YarnException - in case of errors or if YARN rejects the request due to access-control restrictions.

IOException

getQueueAclsInfo

public List<QueueUserACLInfo> getQueueAclsInfo()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get information about acls for current user on all the existing queues.

Specified by:

getQueueAclsInfo in class YarnClient

Returns:

a list of queue acls (QueueUserACLInfo) for current user

Throws:

YarnException

IOException

getAllQueues

public List<QueueInfo> getAllQueues()
                             throws YarnException,
                                    IOException

Description copied from class: YarnClient

Get information (QueueInfo) about all queues, recursively if there is a hierarchy

Specified by:

getAllQueues in class YarnClient

Returns:

a list of queue-information for all queues

Throws:

YarnException

IOException

getRootQueueInfos

public List<QueueInfo> getRootQueueInfos()
                                  throws YarnException,
                                         IOException

Description copied from class: YarnClient

Get information (QueueInfo) about top level queues.

Specified by:

getRootQueueInfos in class YarnClient

Returns:

a list of queue-information for all the top-level queues

Throws:

YarnException

IOException

getChildQueueInfos

public List<QueueInfo> getChildQueueInfos(String parent)
                                   throws YarnException,
                                          IOException

Description copied from class: YarnClient

Get information (QueueInfo) about all the immediate children queues of the given queue

Specified by:

getChildQueueInfos in class YarnClient

Parameters:

parent - Name of the queue whose child-queues' information is needed

Returns:

a list of queue-information for all queues who are direct children of the given parent queue.

Throws:

YarnException

IOException

setRMClient

@InterfaceAudience.Private
public void setRMClient(ApplicationClientProtocol rmClient)

getApplicationAttemptReport

public ApplicationAttemptReport getApplicationAttemptReport(ApplicationAttemptId appAttemptId)
                                                     throws YarnException,
                                                            IOException

Description copied from class: YarnClient

Get a report of the given ApplicationAttempt.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

Specified by:

getApplicationAttemptReport in class YarnClient

Parameters:

appAttemptId - ApplicationAttemptId of the application attempt that needs a report

Returns:

application attempt report

Throws:

YarnException

ApplicationAttemptNotFoundException - if application attempt not found

IOException

getApplicationAttempts

public List<ApplicationAttemptReport> getApplicationAttempts(ApplicationId appId)
                                                      throws YarnException,
                                                             IOException

Description copied from class: YarnClient

Get a report of all (ApplicationAttempts) of Application in the cluster.

Specified by:

getApplicationAttempts in class YarnClient

Returns:

a list of reports for all application attempts for specified application.

Throws:

YarnException

IOException

getContainerReport

public ContainerReport getContainerReport(ContainerId containerId)
                                   throws YarnException,
                                          IOException

Description copied from class: YarnClient

Get a report of the given Container.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

Specified by:

getContainerReport in class YarnClient

Parameters:

containerId - ContainerId of the container that needs a report

Returns:

container report

Throws:

YarnException

ContainerNotFoundException - if container not found.

IOException

getContainers

public List<ContainerReport> getContainers(ApplicationAttemptId applicationAttemptId)
                                    throws YarnException,
                                           IOException

Description copied from class: YarnClient

Get a report of all (Containers) of ApplicationAttempt in the cluster.

Specified by:

getContainers in class YarnClient

Returns:

a list of reports of all containers for specified application attempts

Throws:

YarnException

IOException

moveApplicationAcrossQueues

public void moveApplicationAcrossQueues(ApplicationId appId,
                                        String queue)
                                 throws YarnException,
                                        IOException

Description copied from class: YarnClient

Attempts to move the given application to the given queue.

Specified by:

moveApplicationAcrossQueues in class YarnClient

Parameters:

appId - Application to move.

queue - Queue to place it in to.

Throws:

YarnException

IOException

submitReservation

public ReservationSubmissionResponse submitReservation(ReservationSubmissionRequest request)
                                                throws YarnException,
                                                       IOException

Description copied from class: YarnClient

The interface used by clients to submit a new reservation to the ResourceManager.

The client packages all details of its request in a ReservationSubmissionRequest object. This contains information about the amount of capacity, temporal constraints, and gang needs. Furthermore, the reservation might be composed of multiple stages, with ordering dependencies among them.

In order to respond, a new admission control component in the ResourceManager performs an analysis of the resources that have been committed over the period of time the user is requesting, verify that the user requests can be fulfilled, and that it respect a sharing policy (e.g., CapacityOverTimePolicy). Once it has positively determined that the ReservationRequest is satisfiable the ResourceManager answers with a ReservationSubmissionResponse that includes a ReservationId. Upon failure to find a valid allocation the response is an exception with the message detailing the reason of failure.

The semantics guarantees that the ReservationId returned, corresponds to a valid reservation existing in the time-range request by the user. The amount of capacity dedicated to such reservation can vary overtime, depending of the allocation that has been determined. But it is guaranteed to satisfy all the constraint expressed by the user in the ReservationDefinition

Specified by:

submitReservation in class YarnClient

Parameters:

request - request to submit a new Reservation

Returns:

response contains the ReservationId on accepting the submission

Throws:

YarnException - if the reservation cannot be created successfully

IOException

updateReservation

public ReservationUpdateResponse updateReservation(ReservationUpdateRequest request)
                                            throws YarnException,
                                                   IOException

Description copied from class: YarnClient

The interface used by clients to update an existing Reservation. This is referred to as a re-negotiation process, in which a user that has previously submitted a Reservation.

The allocation is attempted by virtually substituting all previous allocations related to this Reservation with new ones, that satisfy the new ReservationDefinition. Upon success the previous allocation is atomically substituted by the new one, and on failure (i.e., if the system cannot find a valid allocation for the updated request), the previous allocation remains valid.

Specified by:

updateReservation in class YarnClient

Parameters:

request - to update an existing Reservation (the ReservationUpdateRequest should refer to an existing valid ReservationId)

Returns:

response empty on successfully updating the existing reservation

Throws:

YarnException - if the request is invalid or reservation cannot be updated successfully

IOException

deleteReservation

public ReservationDeleteResponse deleteReservation(ReservationDeleteRequest request)
                                            throws YarnException,
                                                   IOException

Description copied from class: YarnClient

The interface used by clients to remove an existing Reservation.

Specified by:

deleteReservation in class YarnClient

Parameters:

request - to remove an existing Reservation (the ReservationDeleteRequest should refer to an existing valid ReservationId)

Returns:

response empty on successfully deleting the existing reservation

Throws:

YarnException - if the request is invalid or reservation cannot be deleted successfully

IOException

getNodeToLabels

public Map<NodeId,Set<String>> getNodeToLabels()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

The interface used by client to get node to labels mappings in existing cluster

Specified by:

getNodeToLabels in class YarnClient

Returns:

node to labels mappings

Throws:

YarnException

IOException

getLabelsToNodes

public Map<String,Set<NodeId>> getLabelsToNodes()
                                         throws YarnException,
                                                IOException

Description copied from class: YarnClient

The interface used by client to get labels to nodes mapping in existing cluster

Specified by:

getLabelsToNodes in class YarnClient

Returns:

node to labels mappings

Throws:

YarnException

IOException

getLabelsToNodes

public Map<String,Set<NodeId>> getLabelsToNodes(Set<String> labels)
                                         throws YarnException,
                                                IOException

Description copied from class: YarnClient

The interface used by client to get labels to nodes mapping for specified labels in existing cluster

Specified by:

getLabelsToNodes in class YarnClient

Parameters:

labels - labels for which labels to nodes mapping has to be retrieved

Returns:

labels to nodes mappings for specific labels

Throws:

YarnException

IOException

getClusterNodeLabels

public Set<String> getClusterNodeLabels()
                                 throws YarnException,
                                        IOException

Description copied from class: YarnClient

The interface used by client to get node labels in the cluster

Specified by:

getClusterNodeLabels in class YarnClient

Returns:

cluster node labels collection

Throws:

YarnException

IOException