How to Implement Client-Server Encryption on OpenText Open Enterprise Server

Securing your OpenText Open Enterprise Server (OES) environment requires more than just installing the software—it demands proper configuration of encryption for all client-server communications. This guide walks you through implementing comprehensive encryption to protect your enterprise data in transit.

Prerequisites

Before beginning implementation, ensure you have:

• Administrative access to your OES server(s)
• Compatible client software (Novell Client or Open Enterprise Client)
• Valid SSL/TLS certificates (self-signed for testing, CA-signed for production)
• Network access to test connectivity
• Backup of current server configuration

Step 1: Enable NCP Encryption on the Server

The NetWare Core Protocol (NCP) handles core file and print services. To encrypt NCP traffic:

For OES on Linux:

1. Open a terminal session with root privileges
2. Edit the NCP configuration file:

   nano /etc/opt/novell/ncpserv.conf
   

3. Locate or add the encryption parameter:

   NCP_PACKET_SIGNATURE_OPTION=3
   

   Values explained:

   • 0 = Disabled
   • 1 = Enabled if requested
   • 2 = Preferred
   • 3 = Required (recommended for security)
4. Save the file and restart the NCP service:

   rcnovell-ncpserv restart
   

Verify NCP Encryption Status:

Check that encryption is active:

ncp-info | grep -i encrypt

Step 2: Configure SSL/TLS Certificates

Proper certificate management is essential for secure LDAP and HTTPS communication.

Generate or Import Certificates:

1. Access iManager (https://your-server:8443/nps/iManager.html)
2. Navigate to Novell Certificate Server > Create Server Certificate
3. Complete the certificate request with your server’s FQDN
4. For production environments, obtain a certificate from a trusted Certificate Authority

Configure LDAP over SSL:

1. In iManager, go to LDAP > LDAP Group Configuration
2. Select your server
3. Enable SSL/TLS on port 636
4. Specify the certificate to use for LDAP SSL
5. Set Require TLS to enforce encrypted LDAP connections

Restart LDAP Services:

rcnovell-nldap restart

Step 3: Configure Client-Side Encryption

Clients must be configured to use encryption when connecting to OES servers.

For Novell Client (Windows):

1. Open Novell Client Properties
2. Navigate to the Advanced Settings tab
3. Locate Packet Signature and set it to Required (option 3)
4. Under Protocol Preferences, ensure Packet Encryption is enabled
5. Click Apply and restart the workstation

For Open Enterprise Client (Linux):

1. Edit the client configuration:

   sudo nano /etc/novell/nwclient.conf
   

2. Add or modify:

   SIGNATURE_LEVEL=3
   ENCRYPTION_REQUIRED=ON
   

3. Restart the client service:

   sudo systemctl restart novell-xregd
   

Step 4: Enable HTTPS for Management Interfaces

Secure web-based administration tools through HTTPS enforcement.

Configure Apache for HTTPS:

1. Enable SSL module:

   a2enmod ssl
   

2. Edit the Apache SSL configuration:

   nano /etc/apache2/vhosts.d/vhost-ssl.conf
   

3. Ensure proper certificate paths:

   SSLCertificateFile /etc/ssl/certs/server.crt
   SSLCertificateKeyFile /etc/ssl/private/server.key
   

4. Force HTTPS redirection by editing the non-SSL vhost:

   <VirtualHost *:80>
       Redirect permanent / https://your-server.domain.com/
   </VirtualHost>
   

5. Restart Apache:

   rcapache2 restart
   

Step 5: Configure SSH for Remote Administration

Secure remote server management through SSH hardening.

Harden SSH Configuration:

1. Edit the SSH daemon configuration:

   nano /etc/ssh/sshd_config
   

2. Implement security best practices:

   Protocol 2
   PermitRootLogin no
   PasswordAuthentication no
   PubkeyAuthentication yes
   Ciphers [email protected],[email protected]
   MACs hmac-sha2-512,hmac-sha2-256
   

3. Restart SSH service:

   systemctl restart sshd
   

Step 6: Set Encryption Policies via eDirectory

Enforce encryption across your entire tree through eDirectory policies.

Create a Login Policy:

1. In iManager, navigate to Policies > Create Policy Package
2. Select Novell Client Configuration Policy
3. Configure encryption settings:
   • Force Packet Signatures: Enabled
   • Minimum Packet Signature Level: Required
   • Enable Encryption: On
4. Associate the policy with appropriate containers or users
5. Set policy priority to ensure enforcement

Step 7: Testing and Verification

Thorough testing ensures encryption is working correctly without breaking functionality.

Test Client Connectivity:

1. Attempt to log in from a client workstation
2. Verify successful authentication and resource access
3. Check client logs for encryption status

Monitor Network Traffic:

Use packet capture tools to verify encryption:

tcpdump -i eth0 -n host your-server-ip

Encrypted traffic should appear as unintelligible data rather than readable text.

Review Server Logs:

Check for encryption-related errors:

tail -f /var/opt/novell/log/ncpserv.log

Step 8: Handle Legacy Client Compatibility

Some older clients may not support modern encryption standards.

Options for Legacy Systems:

Option 1 – Gradual Migration: Set encryption to “preferred” (option 2) temporarily while upgrading clients, then move to “required” (option 3).

Option 2 – Segmented Network: Place non-compliant systems on a separate VLAN with additional network-level encryption (IPsec VPN).

Option 3 – Client Upgrade: Deploy updated Novell Client or Open Enterprise Client versions that support current encryption standards.

Troubleshooting Common Issues

Clients Cannot Connect After Enabling Encryption:

• Verify client encryption settings match server requirements
• Check firewall rules allow NCP traffic (port 524)
• Ensure client software version supports required encryption level

Certificate Errors on LDAPS Connections:

• Verify certificate CN matches server FQDN
• Import CA certificate to client trust stores
• Check certificate expiration dates

Performance Degradation:

• Monitor CPU utilization—encryption is processor-intensive
• Consider hardware with AES-NI acceleration
• Evaluate whether all traffic requires maximum encryption strength

Best Practices for Production Environments

Use CA-Signed Certificates: Self-signed certificates work for testing but create management overhead and security warnings in production.

Implement Certificate Monitoring: Set up alerts for certificates approaching expiration to prevent service disruptions.

Document Your Configuration: Maintain detailed records of encryption settings, certificate locations, and policy configurations.

Regular Security Audits: Periodically review encryption configurations and update cipher suites as cryptographic standards evolve.

Test Before Enforcing: Always test encryption settings on non-production systems or with a pilot user group before enterprise-wide rollout.

Compliance Considerations

Properly implemented OES encryption helps satisfy various regulatory requirements:

• HIPAA: Protects electronic protected health information (ePHI) in transit
• PCI-DSS: Secures cardholder data during transmission
• GDPR: Demonstrates appropriate technical measures for data protection
• SOX: Provides controls for financial data integrity

Document your encryption implementation as part of compliance evidence.

Conclusion

Implementing comprehensive encryption on OpenText Open Enterprise Server transforms your file and print infrastructure from a potential vulnerability into a secure foundation for enterprise operations. While the configuration requires careful planning and testing, the result is end-to-end protection for your organization’s most sensitive data.

By following this implementation guide, you’ve established multiple layers of encryption covering NCP file services, LDAP directory access, web management interfaces, and remote administration—creating a defense-in-depth security posture appropriate for modern threat environments.

Next Steps: Consider implementing additional security measures such as multi-factor authentication, intrusion detection systems, and regular security assessments to complement your encrypted OES infrastructure.

Have you encountered challenges implementing OES encryption? Share your experiences and solutions in the comments below.