Introduction:

Spectrum Scale 4.1.1 is a software defined open system where the end user has the choice to host /configure protocols (Object/NFS/SMB) on GPFS nodes (CES enabled) also called as protocol nodes

As it is an open system, the customer has root access to the system just like any Linux server administrator.

Firewalls associated with open systems are very specific to deployments and vary from each other and are very sensitive plus niche.

Moreover firewall management varies from platform OS to platform OS.

Purpose of this write up is to help understanding Firewall recommendations from three different perspectives of communications required internally (between nodes of cluster) and externally (out side cluster) when using the spectrum scale protocols.

  1. List of services which are required to exposed

  2. Node’s network interfaces, TCP/IP IP address and ports which are required to opened

  3. Sample rules: using firewall-cmd for RHEL 7.0 or RHEL 7.1 based systems

IMPORTANT Note: This article provides a sample example of firewall setup. Deployment of the firewall will vary based on scenarios and network topology.

Also many deployments use external 3rd party firewall appliance ahead of its storage system network for more secure access, in which case this content can act as a deference.

Firewall Ports-Based Rules Table

Some services in spectrum scale uses dynamic ports. It is highly recommend to convert all the dynamic ports to static ports.

Unless the firewall services are supporting service based bindings for dynamic ports handling

Example: Make following ports static before setting up rules (after GPFS is installed and Protocols are Deployed)

#NFSv3 # /usr/lpp/mmfs/bin/mmnfs configuration change NFS_PORT=2049:MNT_PORT=32767:NLM_PORT=32769:RQUOTA_PORT=32768:STATD_PORT=32765

^ GPFS # /usr/lpp/mmfs/bin/mmchconfig tscCmdPortRange=60000-61000

The below tabular format indicates what ports belonging to which service needs to be accessed from what kind of service. This helps in understanding how the firewall needs to be set.

Open from Following

Ports For GPFS Cluster Nodes

GPFS Server Nodes

Protocol (CES) Nodes

NSD Server GPFS client GPFS Server NFSv3 Protocol NFSv4 Protocol SMB Protocol Object Protocol
Installer Node 1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

1191/TCP22/TCP

8889/TCP
10080/TCP

Inter- gpfs nodes (Intra cluster, including loopback) 1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

Inter-protocol nodes (including loopback) 1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

1191/TCP22/TCP

^60000-61000/TCP

#2049/TCP

#2049/UDP

111 /TCP

111/UDP

1191/TCP22/TCP

^60000-61000/TCP

#2049/TCP

#2049/UDP

111 /TCP

111/UDP

1191/TCP22/TCP

^60000-61000/TCP

4379/TCP

1191/TCP22/TCP

^60000-61000/TCP

6200 /TCP

6201 /TCP

6202 /TCP

11211 /TCP

11211 /UDP

5432/TCP

5432/UDP

Performance Monitoring None 4379/TCP 4379/UDP

8123/TCP

8124/TCP

8125/TCP

8126/TCP

8127/TCP

4379/TCP 4379/UDP

8123/TCP

8124/TCP

8125/TCP

8126/TCP

8127/TCP

4379/TCP 4379/UDP

8123/TCP

8124/TCP

8125/TCP

8126/TCP

8127/TCP

4379/TCP 4379/UDP

8123/TCP

8124/TCP

8125/TCP

8126/TCP

8127/TCP

GPFS Admin client 22/TCP 22/TCP 22/TCP 22/TCP 22/TCP 22/TCP 22/TCP
Keystone Admin clients None None None None None None 5000/TCP35357/TCP
NFSv3 Clients None None None #2049/TCP#2049/UDP

111 /TCP

111/UDP

*32767/TCP

*32767/UDP

*32769/TCP

*32769/UDP

*32768/TCP

*32768/UDP

*32765/TCP

*32765/UDP

#2049/TCP#2049/UDP

111 /TCP

111/UDP

None None
NFSv4 Clients None None None 1191/TCP#2049/TCP

#2049/UDP

111 /TCP

111/UDP

1191/TCP#2049/TCP

#2049/UDP

111 /TCP

111/UDP

None None
SMB Clients None None None None None 445/TCP None
Object Clients None None None None None None 5000/TCP35357/TCP

8080/TCP

Firewall Services-Based Rules Table

The below table indicates the services , its internal/external port dependencies and the nodes it is applicable to.

Function Dependent Network Service(s) name(s) External Ports Used for File/Object Access by client Internal Ports used for Inter cluster communication UDP / TCP Rules applicable to Nodes
Installer chef N/A 8889 (chef)10080 (repo) TCP gpfs-server nsd-server. Protocol-servers, nodes
GPFS gpfs N/A 1191 (GPFS)60000-61000 for tscCmdPortRange

22 (ssh)

TCP & UDP22 TCP only gpfs-server nsd-server. protocol-server, nodes
SMB gpfs-smb.servicegpfs-ctdb.service 445 4379 (CTDB) TCP All protocol-server Nodes Only
NFS ganesha.nfsdrpcbind

rpc.statd

2049 (NFS_PORT)111 (RPC)

32767 (MNT_PORT)

32679 (NLM_PORT)

32768 (RQUOTA_PORT)

32765 (STATD_PORT)

Note: Make the dynamic ports as static with command

mmnfs configuration change NFS_PORT=2049:MNT_PORT=32767:NLM_PORT=32769:\

RQUOTA_PORT=32768:STATD_PORT=32765)

TCP & UDP All protocol-server Nodes Only
Object swift-proxy-serverkeystone-all

postgresql

8080 (proxy server)35357 (keystone) 6200 (local account-server)6201 (local container-server)

6202 (local object-server)

11211 (local memchached)

5432 (postgres)

TCP All protocol-server Nodes Onlyand Rules specific to local host for all local servers

 Diagrammatic Representation of Recommended Firewall Layout using ports or services

The below diagram shows a sample GPFS cluster with mixed nodes and the color coding indicates the ports and the services/nodes they are associated with . It also illustrates the ports that need to be accessible for external client system as well as ports and which are required for intra-cluster communication. The network layout used here is very basic for the purpose of illustration.

Firewall

Recommended Firewall Rules (RHEL 7.0, RHEL 7.1) firewalld deamon + service

Following are sample examples on how one can set the firewalls on the Spectrum Scale 4.1.1 cluster over RHEL systems. For your system you need to understand your requirements

and likewise set the firewall rules. Refer to Redhat administration guide for more information on setting up firewalls on RHEL system

Example:

Let us assume that Node1IP, Node2IP, Node3IP represent the internal IP of protocol nodes while CESIP1, CESIP2,CESIP3 indicates the external Ces IP of the protocol nodes.

Node1IP=10.0.0.1

Node2IP= 10.0.0.2

Node2IP= 10.0.0.3

CesIP1=192.168.122.10

CesIP2=192.168.122.11

CesIP3=192.168.122.12

firewall-cmd –get-zones

firewall-cmd –get-active-zones

firewall-cmd –get-default-zone

firewall-cmd –zone=internal –list-all

firewall-cmd –zone=public –list-all

# Do this if using same network interface for Internal and external communication on all nodes

firewall-cmd –set-default-zone=public

firewall-cmd –permanent –zone=public –change-interface=eth1 # Interface Where all Node external IPs will be bound

# Do this if using different network interface for Internal and external communication on all nodes

firewall-cmd –permanent –zone=internal –change-interface=eth0 # Interface Where all Node internal IPs will be bound

# —————————————————

# Set INTERNAL and PUBLIC Zone Sources

# —————————————————

# Set INTERNAL sources on all nodes

firewall-cmd –permanent –zone=internal –add-source=${Node1IP1} # Node internal IP

firewall-cmd –permanent –zone=internal –add-source=${Node2IP} # Node internal IP

firewall-cmd –permanent –zone=internal –add-source=${Node3IP} # Node internal IP

# Repeat above for ALL Nodes internal IP

firewall-cmd –permanent –zone=internal –add-source=127.0.0.1

# Optionaly you can also add only services

# Set INTERNAL ssh Firewall Rules on all nodes (this is generally enabled by default)

firewall-cmd –permanent –zone=internal –add-service=ssh

firewall-cmd –permanent –zone=public –add-service=ssh

# —————————————————

# Set INTERNAL Rules

# —————————————————

# Set INTERNAL Installer Firewall Rules on all nodes

firewall-cmd –permanent –zone=internal –add-port=8889/tcp

firewall-cmd –permanent –zone=internal –add-port=10080/tcp

# Convert GPFS ports from dynamic to static (after GPFS is installed)

#/usr/lpp/mmfs/bin/mmchconfig tscCmdPortRange=60000-61000

# Set INTERNAL GPFS Firewall Rules on all nodes

firewall-cmd –permanent –zone=internal –add-port=60000-61000/tcp

firewall-cmd –permanent –zone=internal –add-port=60000-61000/udp

firewall-cmd –permanent –zone=internal –add-port=1191/tcp

firewall-cmd –permanent –zone=internal –add-port=1191/udp

# Set INTERNAL Object Firewall Rules, only on CES nodes

firewall-cmd –permanent –zone=internal –add-port=8080/tcp

firewall-cmd –permanent –zone=internal –add-port=35357/tcp

firewall-cmd –permanent –zone=internal –add-port=5432/tcp

firewall-cmd –permanent –zone=internal –add-port=5000/tcp

# Set INTERNAL SMB Firewall Rules, only on CES nodes

firewall-cmd –permanent –zone=internal –add-port=4379/tcp

# Set INTERNAL Performance Monitor Firewall Rules, only on CES nodes

firewall-cmd –permanent –zone=internal –add-port=8080/tcp

firewall-cmd –permanent –zone=internal –add-port=35357/tcp

firewall-cmd –permanent –zone=internal –add-port=5432/tcp

firewall-cmd –permanent –zone=internal –add-port=5000/tcp

# —————————————————

# Set PUBLIC Rules

# —————————————————

# Set PUBLIC OBJECT Firewall Rules only on CES Nodes

firewall-cmd –permanent –zone=public –add-port=5000/tcp

firewall-cmd –permanent –zone=public –add-port=8080/tcp

# Convert NFS ports from dynamic to static (after Protocols are installed)

#/usr/lpp/mmfs/bin/mmnfs configuration change NFS_PORT=2049:MNT_PORT=32767:NLM_PORT=32769:RQUOTA_PORT=32768:STATD_PORT=32765

# Set PUBLIC NFS4 Firewall Rules only on CES Nodes

firewall-cmd –permanent –zone=public –add-port=2049/tcp

firewall-cmd –permanent –zone=public –add-port=2049/udp

firewall-cmd –permanent –zone=public –add-port=111/tcp

firewall-cmd –permanent –zone=public –add-port=111/udp

# Set PUBLIC NFS3 Firewall Rules only on CES Nodes

firewall-cmd –permanent –zone=public –add-port=32765/tcp

firewall-cmd –permanent –zone=public –add-port=32765/udp

firewall-cmd –permanent –zone=public –add-port=32767/tcp

firewall-cmd –permanent –zone=public –add-port=32767/udp

firewall-cmd –permanent –zone=public –add-port=32768/tcp

firewall-cmd –permanent –zone=public –add-port=32768/udp

firewall-cmd –permanent –zone=public –add-port=32769/tcp

firewall-cmd –permanent –zone=public –add-port=32769/udp

# Set PUBLIC SMB Firewall Rules only on CES Nodes

firewall-cmd –permanent –zone=public –add-port=445/tcp

firewall-cmd –reload

firewall-cmd –get-zones

firewall-cmd –get-active-zones

firewall-cmd –get-default-zone

firewall-cmd –zone=internal –list-all

firewall-cmd –zone=public –list-all

References

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Using_Firewalls.html

http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_firewallforinstall.htm

Advertisements