Basic Methodologies

Required Parameters of Server Management Services

A.1.                     Patch Management

According to the Gartner Group, "Over 90% of security exploits are carried out through vulnerabilities for which there are known patches”. The judicious and timely application of software patches is critical to both performance and security considerations.  At the same time, a mindless application of security patches without testing and cross-referencing interdependencies is a recipe for disaster.

Patch Acquisition - proactively download patches from major technology vendors into a secure repository.

Patch Testing and Research -  i609 maintains a test bed of Linux and Windows servers   which allows us to check for proper functionality before introducing the patch into a client’s environment.  Additionally a thorough search for potential compatibility issues is undertaken before the application.

Threat Assessment - determine the severity of the issue(s) addressed by the patch.

Patch Deployment - schedule a deployment of the required patches to production computers .  All patch applications are logged into the change log for the machine, and all files modified are tracked.  In a standard client environment, the server layout is mixed to the point where only base OS patches could be efficiently applied as packages, most application patches can more efficiently be manually applied and documented.

Patch Maintenance - continuously monitor to ensure patches are not corrupted

On average the time to life cycle on a patch from first announcement to application in production is 4 days.  In an emergency situation, like the Code Red and SQL Slammer viral outbreaks, elapsed time from initial release to actual application of the patch can be as low as two hours.  We utilize the Altiris suite of products for patch control.

While consultation with the business client is usually not part of our patch management plan, in cases where it is desired we can set up a process of client notification and signoff on patch introduction.

A.2.                     Host Security

The Network security layer is described in a separate whitepaper.  Host security itself is performed in three layers.  Following i609 policies, these entail:

·         Configuring policies for file level access and following accepted password policies (A.4.)

·         Integrated patch control (A.1.)

·         Antiviral software (where required)

Additionally, we see the host management team as needing to integrate tightly with the networking team (item 13) (and direct them as needed) in order to facilitate business and developer functional needs without compromising exploitable security levels.

A host should only have installed and running on it the packages necessary for its function, whether the host be used as a production server, test server, or development server.  Therefore a minimal set of applications will be maintained on the server, which serves to both maximize the performance of the host and to minimize both maintenance needs and security risks. 

On a server it is usually counterproductive to have the overhead of an antiviral package resident in memory, our policy is to keep patches up to date and activity carefully controlled to remove the risk of viral infection from a host, except in very exceptional circumstances.

A.3.                     Restart of Services

Through proactive service and daemon monitoring i609 can provide for alarm-based manual restart of services or for automated restart of services, depending on parameters requested by UNDGO developers and application managers and on our own practices based on a sevice by service analysis.  Additionally scheduled and requested restarts will be implemented as required.  Our monitoring package allows for clear reporting of alarm configurations for documentation purposes and policy analysis, and any manual restarts will be tracked (as will any manual work done on the machine) in the changelogs (section A.11.) for each server.  Very few events require a physical server restart, but this of course can be facilitated as needed.  Any restart will require notification of UNDGO responsible parties and the monitoring provider(s), if that role is not fully assumed by i609.  Affecting our approach on this is section (A.14.), as redundant systems will be in place unless either i609 assumes all monitoring responsibilities or the current provider of monitoring services maintains full control of monitoring.  Our opinions on this aspect will be covered in more depth under that heading.

A.4.                     User Provisioning

Users will be added, removed, and modified by i609 ONLY as directed by the list of responsible UNDP parties.  Standard password policies enforced by i609: Minimum password length ( 8)  and Password complexity requirements (three of the following alpha lower, alpha upper, integers, non-alphanumeric) .  We have found that other requirements (such as aging and password history) can serve to decrease security in many situations, as these policies tend to increase occurrences of crib sheets and post-its containing passwords.  If more secure methods are requested by a vendor then hardware key schemes can be set up and provisioned by i609.

At time of account creation a user profile form should be filled out (made available to responsible UNDGO persons through our support site), stating (in addition to labeling information) any security considerations, such as group membership, which hosts a user should be created on, and file systems read/write permissions required.  In order to ease administrative burden, a ‘clone existing user’ concept can be requested as long as i609 has access to the template account, allowing for simple role-based user creation.  An initial password generated by i609 will be given directly to the newly added user, with the requirement that the password be changed at first login.  Passwords on Linux and Unix hosts can be cloned for a user, and a user will have consistent UIDs thoughout the environment (if the number of hosts were to increase in the future).  While this is not an option in the Windows environment, a consistent approach to user naming and initial password setup will be utilized.

In the case of forgotten passwords, i609 can reset passwords but will not sniff out any existing passwords.

A.5.                     Configuration of OS Parameters

Configuration of system parameters, such as memory allocation, search path order, kernel configuration (where not provided by hosting provider), and disk layouts will be implemented per a combination of historically generated best practices, anecdotal information provided by UNDGO development staff, and advice by application specialists  All pertinent parameters will be captured upon start of contract during runbook creation, and a strict changelog (section A.11.) maintained of any configuration changes (including an indication of metrics for quantifying performance reward where appropriate.)  

A.6.                     Performance Analysis & Capacity Planning

i609 provides recommendations for appropriate technology scaling to clients as a matter of course based on years of engineering experience across a wide array of technologies and configurations.  Additionally i609 can work with either existing testing applications under license to their clients or can introduce new packages as requested to quantify baseline performance and desired gains, and produce a project plan to implement those goals.  The caveat to this is the ratio of risk to reward, where risk can be defined as cost of action in addition to standard use of ‘risk’ referring to potential degradation of functionality or performance during a transitional period.  A report on both the risk and reward factors will be provided along with any recommendations to help the UNDGO responsible entities make fully informed decisions.

A.7.                     File System Management

File system changes, ranging from repartioning to formatting to creation of links and mount points can be facilitated as long as adequate alternate space is available within the local environment to hold data placed at risk during file maintenance.  A change request form (section A.11.) will be made available to responsible UNDGO persons through our support site.  All actions will of course be tracked in the system changelog.

Important to note is that the server management vendor apparently will not have physical access to the servers, so any changes requiring access below those available at the OS level will have to be handled as requests to the hosting provider.  i609 can facilitate these requests upon the direction of the UNDGO staff.  UNDGO will be solely responsible for any charges incurred by these requests of the hosting provider in these situations. 

A.8.                     Log File Monitoring

Active and continual monitoring of system logs and application logs for alarming and interesting events can prevent the vast majority of critical emergencies.  i609 has its own software for monitoring and alerting engineering to interesting events, but we can certainly leverage preexisting software in license to the UNDGO to provide the same function as required. 

Being aware of the events in logs is only the first part in the process of dealing with the information in the logfiles in order to keep a system functioning to its fullest and most reliable capacity.  Patterns of events, related events, and functions being executed at the time of events all work together to give our engineers the information they need to undertake informed actions.  A notation of relevant knowledge articles will be included with the changelog for any actions undertaken as a result of event logs. 

Rotation schemes for app logs can be implemented to interact with any statistical or data mining packages as necessary.

A.9.                     Software Installation

i609 has experience working with the software tools mentioned, utilizing cygwin as a standard component for windows server toolkit for 7 years and experience configuring both ColdFusion MX and 6.1.   Additionally any other software packages can be installed and configured by our engineering team upon receipt of a change request from the responsible UNDGO staff.  The change request ticket should include with it any custom options necessary.  As with any other change to the system, the changelog will be updated by i609 indicating file system changes and system parametric changes necessary for or due to the installation of a software package.

A.10.                Standard Software Maintenance

Our engineering team has a great deal of experience maintaining a wide array of web servers from IIS to Iplanet to Apache, most of us coming from the engineering teams at a startup dot com.  Additionally, we have specific experience installing and maintaining Cold Fusion, Resin, Web Logic, PHP and Tomcat application servers.  On software where i609 currently has no specialists on the team (SiteMinder Policy Server, VerityK2, sunOne Directory) i609 will either add to its staff specialists in these platforms or will train existing staff on these platforms before the start of the contract.  We have added specialties rapidly and professionally with other clients in the past to provide them a seamless experience and can easily do the same for you to more fully meet the needs outlined in the TOR.

A.11.                Change Management and Quality Control

i609’s policy on change management:

A.12.                Shared Root Access

i609 utilizes the concept of root2/admin2 for root access to systems.  Additionally sudo is a great tool to assist developers and engineers in being able to rapidly make necessary scripts function without the risk of actually functioning in a shell as root.  Unfortunately sudo has not been ported successfully yet to the windows environment, but there are several workarounds to this that we utilize – one is the use of power user groups, and another is the temporary activation of an administrator level account for a fixed period of time for a developer to use ‘run as…’  It is understood that other entities outside of the control of i609 will have root access to the systems covered in this document and that these entities must function together in a responsible manner.

A.13.                Cooperation with Other Third-Party Service Providers

i609, as a service vendor, is very comfortable forming ad hoc teams with other vendors.  As long as the duties of each provider are clearly defined (refer to section 14) and instruction is provided for each interface and communication channel it is a standard part of our services to interact with related vendors.  Many of our staff members have functioned with hosting providers, outsourced DBA services, and outsourced SA and NA services.  The question of chain of command must be clearly outlined by the responsible parties in the UNDGO for the best use of the outsourced providers and to minimize the effort required by the UNDGO staff for managing the task requests.  For maximum efficiency the providers should be left a great deal of autonomy in their respective specialties (of course with the provision of clear reporting to UNDGO), but this requires a very clearly stated flow chart for responsibility overlap and interaction between the vendors, otherwise any cross cutting project can require considerable UNDGO management overhead.

A.14.                System Monitoring

The biggest hurdle to be dealt with in this arrangement is to define the relationship with the monitoring provider if the decision is made to continue current monitoring services rather than rolling the monitoring services into the general server maintenance tasks.  The interdependencies of monitoring with servicing are such that the actions of a servicing agent can cause the partner agent to appear to provide less than professional services.  We recommend strongly that every effort be made to integrate monitoring and management services into the same provider, to provide for a straightforward line of accountability and for the most efficient task flow.

In terms of methodology we’ve elaborated on our techniques and technologies above, see sections 2, 3, and 8.