nifi操作文档-官方.pdf

发布时间：2022-06-03 发布人：admin 分类：说明书资料大小：3.43M 资料格式：pdf 举报版权申诉

dch215810-10647664-4744302542978561784.pdf-第1页.png

第1页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第2页.png

第2页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第3页.png

第3页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第4页.png

第4页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第5页.png

第5页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第6页.png

第6页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第7页.png

第7页 / 共92页

dch215810-10647664-4744302542978561784.pdf-第8页.png

第8页 / 共92页

Hortonworks DataFlow

Table of Contents

1. NiFi System Administrator's Guide

1.1. System Requirements

1.2. How to install and start NiFi

1.3. Configuration Best Practices

1.4. Security Configuration

1.4.1. TLS Generation Toolkit

1.4.1.1. Standalone

1.4.1.2. Client/Server

1.4.1.2.1. Server

1.4.1.2.2. Client

1.5. User Authentication

1.5.1. Lightweight Directory Access Protocol (LDAP)

1.5.2. Kerberos

1.6. Multi-Tenant Authorization

1.6.1. Authorizer Configuration

1.6.2. Authorizers.xml Setup

1.6.2.1. Initial Admin Identity (New NiFi Instance)

1.6.2.2. Legacy Authorized Users (NiFi Instance Upgrade)

1.6.2.2.1. Global Access Policies

1.6.2.2.2. Component Access Policies on the Root Process Group

1.6.2.3. Cluster Node Identities

1.6.3. Configuring Users & Access Policies

1.6.3.1. Creating Users and Groups

1.6.3.2. Access Policies

1.6.3.2.1. Global Access Policies

1.6.3.2.2. Component Level Access Policies

1.6.3.2.3. Access Policy Inheritance

1.6.3.3. Viewing Policies on Users

1.6.3.4. Access Policy Configuration Examples

1.6.3.4.1. Moving a Processor

1.6.3.4.2. Editing a Processor

1.6.3.4.3. Creating a Connection

1.6.3.4.4. Editing a Connection

1.7. Encryption Configuration

1.7.1. Key Derivation Functions

1.7.1.1. Additional Resources

1.7.2. Salt and IV Encoding

1.7.2.1. NiFi Legacy

1.7.2.2. OpenSSL PKCS#5 v1.5 EVP_BytesToKey

1.7.2.3. Bcrypt, Scrypt, PBKDF2

1.7.3. Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies

1.7.4. Allow Insecure Cryptographic Modes

1.8. Encrypted Passwords in Configuration Files

1.8.1. Encrypt-Config Tool

1.8.2. Sensitive Property Key Migration

1.8.3. Existing Flow Migration

1.8.4. Password Key Derivation

1.8.5. Secure Prompt

1.9. Administrative Tools

1.9.1. Prerequisites for Running Admin Toolkit in a Secure Environment

1.9.2. Notify

1.9.3. Node Manager

1.9.3.1. Expected behavior

1.10. Clustering Configuration

1.11. State Management

1.11.1. Configuring State Providers

1.11.2. Embedded ZooKeeper Server

1.11.3. ZooKeeper Access Control

1.11.4. Securing ZooKeeper

1.11.4.1. Kerberizing Embedded ZooKeeper Server

1.11.4.2. Kerberizing NiFi's ZooKeeper Client

1.11.4.3. Troubleshooting Kerberos Configuration

1.11.5. ZooKeeper Migrator

1.11.5.1. zk-migrator.sh Command Line Parameters

1.11.5.2. Migrating Between Source and Destination ZooKeepers

1.11.5.2.1. ZooKeeper Migration Steps

1.12. Bootstrap Properties

1.13. Notification Services

1.13.1. Email Notification Service

1.13.2. HTTP Notification Service

1.14. Kerberos Service

1.14.1. Notes

1.15. System Properties

1.15.1. Core Properties

1.15.2. State Management

1.15.3. H2 Settings

1.15.4. FlowFile Repository

1.15.5. Swap Management

1.15.6. Content Repository

1.15.7. File System Content Repository Properties

1.15.8. Volatile Content Repository Properties

1.15.9. Provenance Repository

1.15.10. Persistent Provenance Repository Properties

1.15.11. Volatile Provenance Repository Properties

1.15.12. Write Ahead Provenance Repository Properties

1.15.13. Encrypted Write Ahead Provenance Repository Properties

1.15.14. Component Status Repository

1.15.15. Site to Site Properties

1.15.16. Web Properties

1.15.17. Security Properties

1.15.18. Identity Mapping Properties

1.15.19. Cluster Common Properties

1.15.20. Cluster Node Properties

1.15.21. Claim Management

1.15.22. ZooKeeper Properties

1.15.23. Kerberos Properties

1.15.24. Custom Properties

Hortonworks DataFlow Administration (August 9, 2017) docs.hortonworks.com

Hortonworks DataFlow August 9, 2017 Hortonworks DataFlow: Administration Copyright © 2012-2017 Hortonworks, Inc. Some rights reserved. Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License. http://creativecommons.org/licenses/by-sa/4.0/legalcode ii

Hortonworks DataFlow August 9, 2017 Table of Contents 1. NiFi System Administrator's Guide ................................................................................ 1 1.1. System Requirements ........................................................................................ 1 1.2. How to install and start NiFi ............................................................................. 2 1.3. Configuration Best Practices .............................................................................. 3 1.4. Security Configuration ....................................................................................... 4 1.4.1. TLS Generation Toolkit ........................................................................... 5 1.5. User Authentication .......................................................................................... 7 1.5.1. Lightweight Directory Access Protocol (LDAP) ........................................ 8 1.5.2. Kerberos ................................................................................................ 9 1.6. Multi-Tenant Authorization ............................................................................. 10 1.6.1. Authorizer Configuration ...................................................................... 10 1.6.2. Authorizers.xml Setup .......................................................................... 10 1.6.3. Configuring Users & Access Policies ....................................................... 14 1.7. Encryption Configuration ................................................................................ 33 1.7.1. Key Derivation Functions ...................................................................... 33 1.7.2. Salt and IV Encoding ............................................................................ 36 1.7.3. Java Cryptography Extension (JCE) Limited Strength Jurisdiction Policies ........................................................................................................... 39 1.7.4. Allow Insecure Cryptographic Modes .................................................... 40 1.8. Encrypted Passwords in Configuration Files ..................................................... 41 1.8.1. Encrypt-Config Tool .............................................................................. 41 1.8.2. Sensitive Property Key Migration .......................................................... 44 1.8.3. Existing Flow Migration ........................................................................ 44 1.8.4. Password Key Derivation ...................................................................... 45 1.8.5. Secure Prompt ...................................................................................... 45 1.9. Administrative Tools ........................................................................................ 45 1.9.1. Prerequisites for Running Admin Toolkit in a Secure Environment .......... 46 1.9.2. Notify ................................................................................................... 46 1.9.3. Node Manager ..................................................................................... 47 1.10. Clustering Configuration ................................................................................ 49 1.11. State Management ....................................................................................... 53 1.11.1. Configuring State Providers ................................................................ 54 1.11.2. Embedded ZooKeeper Server ............................................................. 55 1.11.3. ZooKeeper Access Control .................................................................. 56 1.11.4. Securing ZooKeeper ........................................................................... 57 1.11.5. ZooKeeper Migrator ........................................................................... 61 1.12. Bootstrap Properties ...................................................................................... 64 1.13. Notification Services ...................................................................................... 65 1.13.1. Email Notification Service ................................................................... 66 1.13.2. HTTP Notification Service .................................................................... 67 1.14. Kerberos Service ............................................................................................ 68 1.14.1. Notes .................................................................................................. 68 1.15. System Properties .......................................................................................... 69 1.15.1. Core Properties ................................................................................... 70 1.15.2. State Management ............................................................................. 72 1.15.3. H2 Settings ......................................................................................... 72 1.15.4. FlowFile Repository ............................................................................. 72 1.15.5. Swap Management ............................................................................ 73 iii

Hortonworks DataFlow August 9, 2017 1.15.6. Content Repository ............................................................................. 73 1.15.7. File System Content Repository Properties .......................................... 73 1.15.8. Volatile Content Repository Properties ................................................ 74 1.15.9. Provenance Repository ....................................................................... 75 1.15.10. Persistent Provenance Repository Properties ..................................... 76 1.15.11. Volatile Provenance Repository Properties ......................................... 77 1.15.12. Write Ahead Provenance Repository Properties ................................. 77 1.15.13. Encrypted Write Ahead Provenance Repository Properties ................. 79 1.15.14. Component Status Repository ........................................................... 80 1.15.15. Site to Site Properties ....................................................................... 81 1.15.16. Web Properties ................................................................................. 81 1.15.17. Security Properties ............................................................................ 82 1.15.18. Identity Mapping Properties ............................................................. 83 1.15.19. Cluster Common Properties .............................................................. 84 1.15.20. Cluster Node Properties .................................................................... 84 1.15.21. Claim Management .......................................................................... 85 1.15.22. ZooKeeper Properties ....................................................................... 85 1.15.23. Kerberos Properties .......................................................................... 86 1.15.24. Custom Properties ............................................................................ 86 iv

Hortonworks DataFlow August 9, 2017 List of Tables 1.1. Table 1. Maximum Password Length on Limited Cryptographic Strength JVM ........... 39 v

Hortonworks DataFlow August 9, 2017 1. NiFi System Administrator's Guide • System Requirements • How to install and start NiFi [2] • Configuration Best Practices [3] • Security Configuration [4] • User Authentication [7] • Multi-Tenant Authorization [10] • Encryption Configuration [33] • Encrypted Passwords in Configuration Files [41] • Administrative Tools [45] • Clustering Configuration [49] • State Management [53] • Bootstrap Properties [64] • Notification Services [65] • Kerberos Service [68] • System Properties [69] 1.1. System Requirements Apache NiFi can run on something as simple as a laptop, but it can also be clustered across many enterprise-class servers. Therefore, the amount of hardware and memory needed will depend on the size and nature of the dataflow involved. The data is stored on disk while NiFi is processing it. So NiFi needs to have sufficient disk space allocated for its various repositories, particularly the content repository, flowfile repository, and provenance repository (see the System Properties section for more information about these repositories). NiFi has the following minimum system requirements: • Requires Java 8 or newer • Supported Operating Systems: • Linux • Unix • Windows • Mac OS X 1

Hortonworks DataFlow August 9, 2017 • Supported Web Browsers: • Microsoft Edge: Current & (Current - 1) • Mozilla FireFox: Current & (Current - 1) • Google Chrome: Current & (Current - 1) • Safari: Current & (Current - 1) Note Under sustained and extremely high throughput the CodeCache settings may need to be tuned to avoid sudden performance loss. See the Bootstrap Properties section for more information. 1.2. How to install and start NiFi • Linux/Unix/OS X • Decompress and untar into desired installation directory • Make any desired edits in files found under /conf • At a minimum, we recommend editing the nifi.properties file and entering a password for the nifi.sensitive.props.key (see System Properties below) • From the /bin directory, execute the following commands by typing ./nifi.sh : • start: starts NiFi in the background • stop: stops NiFi that is running in the background • status: provides the current status of NiFi • run: runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi • install: installs NiFi as a service that can then be controlled via • service nifi start • service nifi stop • service nifi status • Windows • Decompress into the desired installation directory • Make any desired edits in the files found under /conf • At a minimum, we recommend editing the nifi.properties file and entering a password for the nifi.sensitive.props.key (see System Properties below) • Navigate to the /bin directory 2

Hortonworks DataFlow August 9, 2017 • Double-click run-nifi.bat. This runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi • To see the current status of NiFi, double-click status-nifi.bat When NiFi first starts up, the following files and directories are created: • content_repository • database_repository • flowfile_repository • provenance_repository • work directory • logs directory • Within the conf directory, the flow.xml.gz file and the templates directory are created See the System Properties section of this guide for more information about configuring NiFi repositories and configuration files. 1.3. Configuration Best Practices If you are running on Linux, consider these best practices. Typical Linux defaults are not necessarily well tuned for the needs of an IO intensive application like NiFi. For all of these areas, your distribution's requirements may vary. Use these sections as advice, but consult your distribution-specific documentation for how best to achieve these recommendations. Maximum File Handles NiFi will at any one time potentially have a very large number of file handles open. Increase the limits by editing /etc/ security/limits.conf to add something like * hard nofile 50000 * soft nofile 50000 Maximum Forked Processes * hard nproc 10000 * soft nproc 10000 NiFi may be configured to generate a significant number of threads. To increase the allowable number edit /etc/security/limits.conf And your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding * soft nproc 10000 Increase the number of TCP socket ports available This is particularly important if your flow will be setting up and tearing down a large number of sockets in small period of time. 3

分享到：

赞收藏

资料库

nifi操作文档-官方.pdf

相关推荐

大数据

热门标签

最新资料