yarn-application mode

Application mode is a new execution mode which is introduced after Flink 1.11. The purpose of application mode is to mitigate the resource pressure on the client side. yarn-application mode would launch a Flink cluster on YARN, where the main() method of the application jar gets executed on the JobManager in YARN.

Architecture

The is the original yarn mode where Flink Interpreter would run in the same host with Zeppelin server host.

This is yarn-application mode where Flink Interpreter would run in JobManager.

Configuration

It is very to easy to configure yarn-application mode, very similar as yarn mode

  • Configure FLINK_HOME

  • Configure flink.execution.mode as yarn-application

  • Make sure hadoop client installed, and put hadoop in PATH. Because flink itself doesn't ship hadoop jars, 为Flink本身不带hadoop相关依赖（你可以把flink-shaded-hadoop 拷贝到flink lib下），when zeppelin run flink, it would calll command hadoop classpath，then put all the jars under CLASSPATH

Community

Join Zeppelin community to discuss with others

Previousyarn modeNextLanguages

Last updated