GBase database (GBase数据库) provides robust capabilities for modern data infrastructure. This guide details the steps for installing GCDW (GBase Cloud Data Warehouse) and configuring HDFS-related settings, ensuring seamless integration and optimal performance.
Resource Limit Configuration
1. Edit /etc/security/limits.conf
Add the following configurations:
gbase soft sugpending unlimited
gbase hard sugpending unlimited
gbase soft nproc unlimited
gbase hard nproc unlimited
2. Distribute the file across all nodes
cpush all: /etc/security/limits.conf /etc/security/limits.conf
3. Edit /etc/hosts
Add the IP addresses and hostnames for:
- Primary and secondary Namenodes in the Hadoop cluster.
- All Datanodes.
- KDC domain and its corresponding hostname.
4. Distribute the file across all nodes
cpush all: /etc/hosts /etc/hosts
5. Configure Trust for gbase User
Refer to the official documentation for configuring trust among cluster nodes.
6. Extract the Installation Package
tar -xf /opt/tools/gcdw-NoLicense-9.8.0.7.6-redhat8-x86_64.tar.bz2 -C /opt/gcdw/
7. Set Permissions for the GCDW Directory
chown -R gbase.gbase /opt/gcdw/
8. Distribute and Execute SetSysEnv.py
cpush all: /opt/gcdw/gcinstall/SetSysEnv.py /opt/gcdw/SetSysEnv.py
cexec all: "python /opt/gcdw/SetSysEnv.py --dbaUser=gbase --installPrefix=/opt/gcdw --cgroup"
9. Configure demo.options for Kerberos Authentication
- Example parameters:
gcluster_instance_name=instance_name
instance_root_name=root_user
instance_root_password=root_password
gcdw_STORAGE_STYLE=hdfs
gcdw_HDFS_URI=hdfs://user@namenode_ip:rpc_port/path/
gcdw_HDFS_AUTH_MODE=kerberos
gcdw_HDFS_PRINCIPAL=hdfs/gcdw@HADOOP.COM
gcdw_HDFS_KEYTAB=/opt/hdfs.keytab
gcdw_HDFS_KERBEROS_CONFIG=/etc/krb5.conf
10. Execute the Deployment Script
cd /opt/gcdw/gcinstall
./gcinstall.py --silent=demo.options
11. Validate Installation
After installation, validate with the following command:
account --show
12. Update gcluster Configuration
On all nodes, edit /opt/:
[gbased]
gcdw_hdfs_client_timeout=600
_t_gcluster_support_cte=1
table_definition_cache=5120
table_open_cache=1280
gcluster_random_insert=1
gcluster_send_client_data_timeout=1800
group_concat_max_len=10240
gbase_hdfs_auth_mode=kerberos
gbase_hdfs_protocol=rpc
gbase_hdfs_keytab=/etc/user.keytab
gbase_hdfs_principal=hadoop/namenode@HADOOP.COM
gcdw_hdfs_namenodes=ns01_primary_ip,ns01_secondary_ip|ns01_primary_hostname,ns01_secondary_hostname
After editing, restart the gcluster service.
13. Update gnode Configuration
Edit /opt/:
[gbased]
gbase_loader_parallel_degree=4
gbase_parallel_degree=4
gbase_parallel_max_thread_in_pool=512
gbase_loader_read_timeout=5000
gbase_loader_max_line_length=32M
thread_stack=524288
gbase_hdfs_auth_mode=kerberos
gbase_hdfs_protocol=rpc
gbase_hdfs_keytab=/etc/user.keytab
gbase_hdfs_principal=hadoop/namenode@HADOOP.COM
gcdw_hdfs_namenodes=ns01_primary_ip,ns01_secondary_ip|ns01_primary_hostname,ns01_secondary_hostname
Restart the gbase service after modifications.
14. Enable Auto-Startup on Boot
cexec wh: 'echo "su - gbase -c "gcluster_services all start"" >> /etc/rc.d/rc.local'
cexec coor: 'echo "su - gbase -c "gcware_services all start"" >> /etc/rc.d/rc.local'
cexec wh: 'chmod +x /etc/rc.d/rc.local'
15. Create Warehouse
1) Use gcadmin to create a warehouse:
gcadmin createwh e wh.xml
2) Edit wh.xml to add node IPs and modify the warehouse name and comment.
3) Execute the warehouse creation command:
gcadmin createwh wh.xml
Conclusion
The GCDW installation and HDFS configuration process within the GBase database (GBase数据库) environment provides a flexible and efficient data management solution. With robust Kerberos authentication and fine-tuned HDFS settings, GBase enhances both security and performance, ensuring your data infrastructure is ready for enterprise-level demands.
