Assume : domain to be accessed is accessible via the url domain.com
Varnish caching engine will run infront of Apache in port 80, and internally forward the requests to Apache.

Installation of Varnish:

1. Download Varnish source file from sourceforge.net
2. Install libraries below via yum, as well as any other dependencies.

yum install logrotate libgomp gcc cpp binutils kernel-headers glibc-headers glibc-devel pcre-devel

3. Do ./configure, make, make install to install varnish
4. Configure Apache to listen on another port 8080 and restart Apache for changes to reflect.
5. Start Varnish as below, before that you can configure the default config file /usr/local/etc/varnish/default.vcl as mentioned in (6)
If varnish available under services, configure the file below with the parameters below:

Code:

DAEMON_OPTS=”-a :80 \
-T localhost:6082 \
-f /usr/local/etc/varnish/default.vcl\
-S /etc/varnish/secret \
-s malloc,200M \

And do

Code:

/etc/rc.d/init.d/varnish start

If its not available under services, start it as below:

Code:

varnishd -f /usr/local/etc/varnish/default.vcl -T 127.0.0.1:6082 -a 0.0.0.0:80 -s malloc,200M

If you want to connec to the commandline interface use

varnishd -f /usr/local/etc/varnish/default.vcl -T 127.0.0.1:6082 -a 0.0.0.0:80 -s malloc,200M -d

6. A sample config file for Apache is as below. It can be tweaked as per customization required as outlined in http://www.varnish-cache.org/trac/wiki/Introduction

Code:

backend default {
.host = “69.50.217.14″;
.port = “8080″;
}

acl purge {
“localhost”;
“69.50.217.14″;
}

sub vcl_recv {

// Strip cookies for static files:
if (req.url ~ “\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$”) {
unset req.http.Cookie;
return(lookup);
}

// Remove has_js and Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie, “(^|;\s*)(__[a-z]+|has_js)=[^;]*”, “”);

// Remove a “;” prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, “^;\s*”, “”);

// Remove empty cookies.
if (req.http.Cookie ~ “^\s*$”) {
unset req.http.Cookie;
}

if (req.request == “PURGE”) {
if (!client.ip ~ purge) {
error 405 “Not allowed.”;
}
purge(“req.url ~ ” req.url ” && req.http.host == ” req.http.host);
error 200 “Purged.”;
}
}

sub vcl_hash {
if (req.http.Cookie) {
set req.hash += req.http.Cookie;
}
}

sub vcl_fetch {

// Strip cookies for static files:
if (req.url ~ “\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$”) {
unset beresp.http.set-cookie;
}

// Varnish determined the object was not cacheable
if (!beresp.cacheable) {
set beresp.http.X-Cacheable = “NO:Not Cacheable”;
}

// You don’t wish to cache content for logged in users
elsif(req.http.Cookie ~”(UserID|_session)”) {
set beresp.http.X-Cacheable = “NO:Got Session”;
return(pass);
}

// You are respecting the Cache-Control=private header from the backend
elsif ( beresp.http.Cache-Control ~ “private”) {
set beresp.http.X-Cacheable = “NO:Cache-Control=private”;
return(pass);
}

// You are extending the lifetime of the object artificially
elsif ( beresp.ttl < 1s ) {
set beresp.ttl = 300s;
set beresp.grace = 300s;
set beresp.http.X-Cacheable = “YES:Forced”;
}

// Varnish determined the object was cacheable
else {
set beresp.http.X-Cacheable = “YES”;
}

return(deliver);
}

7. Varnish Cache log can be started using the command below:

Code:

/usr/local/bin/varnishlog -w /tmp/vlog &

And tailing /tmp/vlog to see the dump of the requests handled by Varnish.

8. Any other problems with Apache can be checked from Apache error log, for instance http://domain.com shows a blank page.
9. You can also benchmark the difference in performance before and after using Varnish using the Apache bench marking tool ab and the commandline below:

Code:

ab -n1000 -c100 http://domain.com/

This is one of the most useful plug in that can be used in your application when you require to upload a file as attachment to your application, its quite easy to use and with the help of paperclip scaffolding you can achieve file upload feature very easily, below i have explained the steps to create a simple application using scaffolding that enables the file upload feature with the help of paperclip plugin.

Steps:
Install the required gems to run the paperclip plugin from the below mentioned command,

$ gem sources -a http://gemcutter.org
http://gemcutter.org added to sources
$ sudo gem install view_mapper
Successfully installed view_mapper-0.2.0
1 gem installed Installing ri documentation for view_mapper-0.2.0…
Installing RDoc documentation for view_mapper-0.2.0…

1. First you need to create a rails app and change the database.yml setting and set things to install the plug in $rails student -d mysql

2 Then you have to just install the paperclip plugin into your application from the below mentioned command,

$ ./script/plugin install git://github.com/thoughtbot/paperclip.git

3. Once the plugin is installed, let me show you to create a simple scaffold with paperclip file attachment feature in it.
Just create a simple scaffold and check it out,

$ ./script/generate scaffold_for_view Student name:string branch:string comment:string –view paperclip:photo

Then you need to enter $ rake db:create and $ rake db:migrate
This will create the paperclip file attachment feature, just run $ script/server and check it out. The display looks just like the image i have mentioned at the beginning.

Install Sphinx

Just run the following three commands on your server or dev machine to install Sphinx:

./configure make sudo make install

That will setup Sphinx with default for use with MySQL. If you want to use it with PostgreSQL, then run configure with the following flag:

./configure –with-pgsql

Note: you can download the sphinx from http://sphinxsearch.com/downloads/beta/

Install Thinking Sphinx

Even though there are a couple of Sphinx plugins for Rails, I chose to go with Thinking Sphinx, as it seems to be the most popular and feature complete.
So you can install it as a Rails plugin using script/plugin install command:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

Writing code to use sphinx search:

We now need to index our models. This consists of adding a few small lines of code into each model that you want to be able to search. So lets say we have a Blog app (doesn’t everyone!), which has a Post model. And that Post model contains the usual title and description fields. We therefore add the following bit of code beneath our association declarations in

app/models/post.rb:
define_index do indexes title, description end

Those very short three lines will tell Thinking Sphinx to index the title and description fields of the Post model, and allow us to search through all our posts. Now we just need to index and start Sphinx. And Thinking Sphinx makes this very easy with it’s handy Rake tasks.
Just run this:

rake ts:rebuild

That will stop (if it is started), index and start Sphinx for you. Now we need to create a quick search form. This will eventually be a global site search, and not just a Post search. So we will create a new controller:

script/generate controller search

Then create a view at app/views/search/index.html.erb and place the form within it:
Now in your new Search controller, create a new create action:

def create @posts = Post.search params[:search] end

Create your create view at app/views/search/create.html.erb with a bit of code to display your @posts in the usual way.

Note: We can even paginate our results using the WillPaginate plugin:

def create
@posts = Post.search params[:search], :page => params[:page], :per_page => 10 end

Start by removing any old versions previously installed via apt-get:

sudo apt-get remove imagemagick

Then update apt-get and install some supporting packages:

sudo apt-get update

*(sudo apt-get install
libperl-dev gcc libjpeg62-dev libbz2-dev
libtiff4-dev libwmf-dev libz-dev libpng12-dev
libx11-dev libxt-dev libxext-dev libxml2-dev
libfreetype6-dev liblcms1-dev libexif-dev perl
libjasper-dev libltdl3-dev graphviz gs-gpl pkg-config)*

*its a single line command.

get the Imagemagick package from this link:

http://www.imagemagick.org/script/install-source.php#unix

Once the source is downloaded, uncompress it:

tar -xzf ImageMagick.tar.gz

Now configure and make:

cd ImageMagick-6.5.0-0
./configure
sudo make
sudo make install

Add the following line to ~/.bashrc:

export LD_LIBRARY_PATH=/usr/local/lib

Update: If you still get an error like the one above, try running ldconfig:

sudo ldconfig

You can confirm the install and available formats with:

identify -list format

Once Image magick is installed install rmagick gem by typing the command:

$sudo gem install rmagick

Filed under: cloud, cloud computing, Cloud Computing Attacks, Data Security Lifecycle, Network Security, Operating System Security, Operational Security, security

There are a lot of concerns in the minds of experts while moving to cloud computing. The first one is information or data security. While cost and ease of use are two great benefits of cloud computing, there are significant security concerns that need to be addressed when considering moving critical applications and sensitive data to public and shared cloud environments. Prior to any move to the cloud, it’s simple common sense and good practice to ensure that the necessary security procedures, standards, guidelines and processes are already in place at home to ensure the technical security of data.

In a cloud, data can be moved through data centers in different international locations in response to varying levels of demand. Data located in any country may be governed by the laws of that country and there are sophisticated regulatory demands on businesses to ensure the security of the information they hold and generate. Furthermore, organizations have legal obligations to preserve data and ensure that it is available for any legal proceedings—called
electronic discovery—even if the customer is not in direct possession or control of that data. Therefore, cloud customers must seek contractual assurance regarding the geographical storage and transit of their cloud-based data, to ensure compliance with existing as well as the inevitable introduction of new laws governing Therefore, cloud customers must seek contractual assurance regarding the geographical storage and transit of their cloud-based data, to ensure compliance with existing as well as the inevitable introduction of new laws governing.

Physical Security

The cloud provider is responsible for providing platform and infrastructure security. Physical access should be strictly controlled by professional security staff utilizing video surveillance, state of the art intrusion detection systems, and other electronic means. As a result, an organization is responsible for managing:

• The physical location of the data center (affecting which country’s law applies);
• The security of the data center
• The trustworthiness of system administrators; and
• The documented information security program that protects the confidentiality, integrity, and availability of data and systems, including, but not limited to, configuration, patching, incident response, and business continuity management.

Environmental Controls

Cloud computing clusters architecture should be designed with resiliency and redundancy in mind,
helping minimize single points of failure and the impact of common equipment failures and
environmental risks. Dual circuits, switches, networks, and other necessary devices are utilized
to provide redundancy. Facilities infrastructure at the data centers has been designed to be
robust, fault tolerant, and concurrently maintainable.

Operational Security

Network Security: Most of the cloud computing providers employs multiple layers of defense to help protect the network perimeter from external attacks. Enforcement of network segregation using industry standard firewall and ACL technology can prevent network attacks.
Operating System Security: You have to make sure that a standard hardened operating system (OS), and security fixes are uniformly deployed to the entire infrastructure.
Access Control: Authentication controls and Authorization controls are very much required in host levels and application levels to prevent unauthorized access.

Data Security Lifecycle

The Data Security Lifecycle consists of six phases as shown in the diagram given below:
Data security: Confidentiality, Integrity, Availability, Authenticity, Authorization, Authentication, and Non-Repudiation.
Location of the data: There must be assurance that the data, including all of its copies and backups is stored only in geographic locations permitted by contract, SLA, and/or regulation. For instance, use of “compliant storage” as mandated by the European Union for storing electronic health records can be an added challenge to the data owner and cloud service provider.
Data remanance or persistence: Data must be effectively and completely removed to be deemed ‘destroyed.’ Therefore, techniques for completely and effectively locating data in the cloud, erasing/destroying data, and assuring the data has been completely removed or rendered unrecoverable must be available and used when required.
In addition, the destruction of information is more difficult to guarantee in a shared environment. Encryption, and the provision of different encryption keys for each customer in a public cloud may go some way to mitigate this, but not all cloud providers can offer this.
It is wise to ensure that data isn’t held in a proprietary format and is readily transferable to other systems as required. The format of information entrusted to a provider should be agreed and be documented in the contract.

Cloud Computing Attacks

Denial of Service attacks: Some security professionals have argued that the cloud is more vulnerable to DoS attacks, because it is shared by many users, which makes DoS attacks much more damaging. Twitter suffered a devastating DoS attack during 2009.
Side Channel attacks: An attacker could attempt to compromise the cloud by placing a malicious virtual machine in close proximity to a target cloud server and then launching a side channel attack.
Authentication attacks: Authentication is a weak point in hosted and virtual services and is frequently targeted. There are many different ways to authenticate users; for example, based on what a person knows, has, or is. The mechanisms used to secure the authentication process and the methods used are a frequent target of attackers.
Man-in-the-middle cryptographic attacks: This attack is carried out when an attacker places himself between two users. Anytime attackers can place themselves in the communication’s path, there is the possibility that they can intercept and modify communications.

Cloud computing therefore offers real benefits to companies seeking a competitive edge in today’s fast growing economy. A lot of providers are moving into this area, and the competition is driving prices even lower. Attractive pricing, the ability to free up staff for other duties, and the ability to pay for “as needed” services will continue to drive more businesses to consider cloud computing. The decision to move to cloud-based services should fit into the organization’s overall corporate objectives. A cloud provider must be able to deliver the security, availability, audit and reporting configurations you require. It is therefore essential that enterprises seeking the benefits of cloud computing ensure that their cloud providers can deliver the levels of security and control that they need and should expect.

Vmware is the leader in the infrastructure Virtualization space with industry’s first bare-metal X86 architecture that was launched in 2001, and now into the fourth generation of Vsphere infrastructure suite. Vsphere is now a fully mature production ready enterprise class Virtualization solution with a customer base that includes all 100 of the Fortune 100 companies. Their solution stands out in terms of reliability, scalability, flexibility and efficiency.

We’ll now see what are the key differentiators that gives Vmware its competetive edge over other virtualization solution providers like Xen or Hyper-V.

• Vmware’s greatest advantage is its bare-metal Hypervisor that lays the foundation for its Virtualization platform. The hypervisor with a small disk footprint of 70Mb and its non-reliance on any general purpose OS makes its less error prone, coupled with the direct driver model that gives it higher performance and greater virtual machine density.

• Its a highly reliable platform that gives maximum uptime advantages by making use of technologies such as distributed resource scheduling, logical resource pools, Vmotion, hot-add resources and Storage motion, HA , Fault tolerance and NIC teaming. Many competetors such as Xen or Hyper-V do not come with the complete capabilities or offers limited features compared to whats offered by Vmware.

• Vmware’s management tool or the vCenter is a comprehensive and completely SLA-driven virtualization management platform catering to all requirements of managing the datacenter such as cluster and resource pool management, BCDR setups, DRS, HA and Fault Tolerance, Vmotion rules, patch management, vmSafe and vShield zones and distributed virtual switch and NIC teaming rules.

• Vmware also offers the largest hardware compatibility list compared to other competitors supporting more than 1000 servers, 500 HBAs, 300 network I/O cards and more than 53 guest operating systems, the highest in the industry. It also has partnership alliances with more than 400 enterprise software vendors giving it the broadest support for applications.

• Though Vmware’s per license costs may look to be more expensive than Xen or Hyper-V solutions, on the long run Vmware offers unparalleled Return On Investment (ROI), lowered capital expenditures (Capex) and reduced operating expenses (Opex) in the data center and above all, lower cost per application by utilizing features such as CPU and memory Over-Commit,transparent page sharing, thin provisioning of virtual disks giving Vmware a very high VM density ,better performance and consolidation ratios.

Vmware is by far the most trusted and reliable Virtualization platform available now for production systems and reported by Garnet as the leader in this industry. The stability of Vmware was rightly said by Redmondmag.com that the least stable part of ESX is usually the administrator. The code is virtually bomb-proof. Citrix Xen and Hyper-V are catching up with their virtualization features but still comes with a lot of limitations that needs to be taken care of to bridge the gap with Vmware.

Preparing Network:

Install “bridge-utils”

# sudo apt-get install bridge-utils

Now edit /etc/network/interfaces and setup a bridge using a static, private ip-address.

#cat /etc/network/interfaces

auto lo
iface lo inet loopback
auto br0
iface br0 inet static
address 192.168.3.251
netmask 255.255.255.0
gateway 192.168.3.1
bridge_ports eth0
bridge_fd 0
bridge_hello 2
bridge_maxage 12
bridge_stp off

#sudo /etc/init.d/networking restart

#brctl show

bridge name bridge id STP enabled interfaces
br0 8000.002215be747a no eth0

Now setup the static ip-address (192.168.3.251) and hostname in /etc/hosts. Please make sure that the hostname (cloud) does not appear in the line starting with 127.0.0.1.

#cat /etc/hosts

127.0.0.1 localhost
192.168.88.3 cloud cloud

# The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Preparing Storage:

Install lvm2, nfs-kernel-server, iscsi-target and vblade.

#apt-get install lvm2 nfs-kernel-server iscsitarget vblade

Now prepare the dedicated partition to be used with lvm, then create a logical volume group “vol”.

#pvcreate /dev/sda3

#pvs

PV VG Fmt Attr PSize PFree
/dev/sda3 lvm2 — 186.23g 186.23g

#vgcreate vol /dev/sda3

Volume group “vol” successfully created

#vgs

VG #PV #LV #SN Attr VSize VFree
vol 1 0 0 wz–n- 186.22g 186.22g

Edit /etc/default/iscsitarget and set the iscsitarget to be started on boot-up.

#cat /etc/default/iscsitarget

ISCSITARGET_ENABLE=true

Then start the iscsitarget and nfs-kernel-server services.

#/etc/init.d/iscsitarget start

#/etc/init.d/nfs-kernel-server start

Prepare Storage:

For the openQRM Server Database backend we install “mysql-server”.

#apt-get install -y mysql-server

For sake of simplicity we have left the myslq-password empty.

Prepare KVM:

Install the “kvm” package.

#apt-get install -y kvm

Reading package lists… Done
Building dependency tree
…..<snip>
Setting up qemu-kvm (0.12.3+noroms-0ubuntu9) …
qemu-kvm start/running
Setting up kvm (1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9) …
Processing triggers for libc-bin …
ldconfig deferred processing now taking place

Install openQRM:

We build openQRM from the sources which are available in the openQRM Projects subversion repository.

#sudo apt-get install -y subversion make

Now checkout the openQRM sources from the svn repository.

#svn co https://openqrm.svn.sourceforge.net/svnroot/openqrm openqrm

#cd openqrm/trunk/src/

“make” requires a working internet connection. If no internet is available on this system you

can download the build-cache which will avoid any downloads using:

http://sourceforge.net/projects/openqrm/files/openQRM-4.6/source/openqrm-thirdparty-cache.tgz/download

#make

All compilation results are cached by the openQRM build-system. To ensure all components are build correctly simply run “make” again. The second (and every further “make” run) will just takes a few seconds.

Then run “make install”.

#make install

And finally initialize and start openQRM by “make start”.

#make start

Once initialization is complete, please configure your openQRM Server at: http://[server-ip-address]/openqrm/

-> User: openqrm -> Password: openqrm

“make start” triggers a check for the openQRM runtime-dependencies which will install all additional required packages automatically. At first start the openQRM Server is initialized.

Configure openQRM:

Login to your openQRM Server at http://localhost/openqrm [http://localhost/openqrm] . User and password is “openqrm”.

• First select the bridge-interface as the openQRM management network-device.
• Then select “myslq” as the Database to use as the openQRM backend.
• And then configure the Database connection credentials.

The openQRM is now fully configured and will forward to the Datacenter Dashboard.

OpenQRM(open Qlusters Resource Manager) is an Open Source Data Center Management and Cloud Computing Platform. It is a fully community driven, Open-Source project backed up by main Sponsor openQRM Enterprise. It offers full automation of data centers and their scalable management.

Datacenters are custom and complex environments due to the number of involved subsystems like physical servers, virtual machines, different operating systems, network components, network configuration and network services like DNS and dhcpd plus a system- and service monitoring etc. and the complexity of each subsystem. OpenQRM aims at offering a centralized management of all these within a single administration console.

openQRM simplifies the complex tasks by using the concept of ‘breaking’ the datacenters into manageable subsystems. Each subsystem is separately implemented via. A plugin which provides the functionality to manage it. Then open QRM creates automated and generic interfaces between the different components via its completely plug-able software architecture. Actually the base server of openQRM is designed to just have a single function, to manage plugins.

openQRM has a unique architecture that unifies physical and virtual machine deployment within a single management console. openQRM integrates with all mainstream virtualization technologies and supports transparent P-to-V, V-to-P and also V-to-V migrations. The “plug-ability” of openQRM helps combine open-source and commercial third-party components for data center administration.

Working:

A server(Linux) consists of the following components :

-> (Linux) Kernel

-> kernel modules

-> Initial Ramdisk (initrd)

-> Root-filesystem

It is to be noted that all these components are files. Hence openQRM treats the systems as files. Hence it is recommended to use Modern Storage Systems.

In general, a small, initial ramdisk is used by Linux systems to pre-setup the init procedure by running /sbin/init. The initrd is responsible for finding and mounting the root-filesystem. openQRM uses this generic Linux mechanism in an enhanced way by making the process of finding and mounting the root-filesystem completely plug-able. In openQRM the storage-plugin helps the booting system within the initrd stage in mounting the root-filesystem. Since storage-types are plug-able in openQRM, any kind of external, remote storge devices can be used.

The following is the boot process of openQRM:

• Bios is configured to do a network-boot (PXE)
• System sends a PXE request and asks for an automated network-configuration via dhcp
• openQRM’s dhcpd-server answers the request and provides an ip-address
• System activates its network-configuration and reads its PXE-configuration from the openQRM server
• System downloads its operating system kernel and initial ramdisk (initrd)
• System executes the Operating System kernel and loads the initial ramdisk
• Within the ramdisk network-hardware is being automatically detected and initialized
• Having full network connection the system now downloads its full set of kernel modules
• Additional hardware-detection with all available kernel modules
• System gets configuration parameters from the openQRM server

By now hardware is detected and the network is fully initialized. The system can continue in two different ways now:

1. As idle, free server resource

OR 2. As an active, assigned resource acting as a service provider

In case the system is active assigned to a started appliance, the following steps are executed:

• The deployment type check and method of the server-image assigned to its appliance
• The image-deployment hook provided by specific storage-plugin is downloaded
• System executes “mount_root” function provided in the image-deployment hook
• The image-deployment hook mounts the server-image-location from the remote storage
• Kernel- and kernel-modules files are transferred to the root-filesystem mounted rw
• The openQRM-client is being installed on to the mounted root-filesystem
• The image-deployment hook re-mounts the server-image-location read-only
• System continues with regular init (running /sbin/init on the root-filesystem)
• During futher init procedure of the System, the openQRM-client is started
• Further plugin services are started

openQRM’s hardware detection uses “pcimodules”. The “pcimodules” command is available as a patch for the “pcituils” package and simply lists all needed kernel modules according the pci ids of the detected hardware. “pcimodules” hardware detection method on physical and virtual systems is found to detect much more hardware than the “hwsetup” utility.

Resource

A resource in openQRM is “everything which has a CPU and some memory”. Resources in openQRM have different types such as Physical Systems, Virtualization Host or Virtual Machine. According its type openQRM interfaces and communicates with the specific resource.

Kernel

Kernels in openQRM are Linux Operating System kernels which can be assigned to resources. This happens automatically through the appliance model through openQRM’s integrated, centralized network-boot-manager PXELINUX from the Syslinux project.

Image

An image (server-image) is located on a network-attached storage device (NAS or SAN) and contains a root-filesystem of an Operating system supported by openQRM. The image is completely self-contained and, via the appliance model it and in combination with a kernel can be started on any available (idle) resource. The server-image root-filesystem may be a minimal Operating system installation which is then further leveraged (e.g.via the Puppet integration) or it also can be a full installed completely pre-configured set of applications. An image can even be a snapshot of an existing server or a clone or copy of an an existing server-image.

Appliance

It represents one(or more) of the actual services which should be provided by the Datacenter. An appliance is the combination of a kernel, an (server-) image, a resource and service requirements plus service-level-agreements(SLA). With those informations openQRM then fully automates the management of the specific services running on the appliance’s server-image.

Storage

A storage component in openQRM consists of an integrated resource containing some kind of network-attachable storage (NAS or SAN). Storages in openQRM are providing the image-locations, meaning the place where server-images are stored and directly attached to resource as required. By creating a storage server openQRM then exactly knows how to interface with its specific storage technology and further allows automated management of the available storage space and volumes.

Migrations:

openQRM supports transparent P-to-V, V-to-P and also V-to-V migrations.

P-to-V Migration

Since resources are decoupled from their root-files-systems migration appliances from physical server to virtual machines is complete transparent and easy. The steps below outline how to exchange an appliance physical resource with a virtual one :

• stop the appliance

• edit the applinace

• change the “resource-type” from “Physical System” to a “Virtulization-type VM”

• select a new resource from the type “Virtulization-type”

• save the appliance

• start the appliance

The appliance will now start using the new, virtual resource as defined in the appliance configuration.

V-to-P Migration

This is similar to the P2V migration. The steps to migrate an appliance from a virtual machine to a physical system :

• stop the appliance

• edit the appliance

• change the “resource-type” to “Physical System”

• select a new resource from the type “Physical System”

• save the appliance

• start the appliance

The appliance now runs on a physical system.

V-to-V Migration

Using the same method appliances can be moved from one Virtualization type to another. Here the steps necessary for the migration :

• stop the appliance (Virtualization type A)

• edit the appliance

• change the appliance resource type from “Virtualization-VM” (type A) to “Virtualization-VM” (type B)

• select a new resource from “Virtualization-VM” (type B)

• save the apliance

• start the appliance

The appliance will now start using the new, virtual resource(Virtualization type B) as defined in the appliance configuration.

Requirements:

• Simple Proof-of-Concept Setup
• 1 Physical System dedicated for the openQRM Server
• VT CPU Extension (full Virtualization Support)
• A free partition dedicated for the server-image store (at least 20 GB)
• 1 GB Memory
• Basic Setup
• 3 Physical Systems (openQRM Server, Storage and Virtualization Host)
• VT CPU Extension (full Virtualization Support)
• A free partition dedicated for the server-image store (at least 100GB) on the system dedicated for the Storage Production Setup
• 4+N Physical Systems (2 for openQRM Server HA, N Storage and N Virtualization Hosts)
• VT CPU Extension (full Virtualization Support)

There are various solutions available for achieving high availability and scalability of MySQL data/service. ‘MySQL Replication’ and ‘MySQL Cluster’ the solutions offered and supported by MySQL. Third-party solutions such as DRBD (Distributed Replicated Block Device) and Heartbeat can also be used or one could implement a combination of these technologies.

The two solutions offered by MySQL are discussed here. Of these, MySQL Cluster is considered a feasible and reliable solution and is our subject of interest.

MySQL Replication:

It enables data from one MySQL server instance to be replicated to another MySQL server instance. It allows replication of data from a single master server to any number of slaves using a simple setup. However, this is an asynchronous replication solution, so the synchronization does not take place in real time, and data replication among all slaves is can not guaranteed always.

• Advantages
  • This can be implemented on any platform (which MySQL supports) and isn’t OS specific.
  • Replication is asynchronous and can be stopped and restarted at any time, and is easily manageable during network issues.
  • Data can be replicated from one master to any number of slaves. This is suitable for environments with heavy reads, but light writes, by spreading the load across multiple slaves.
• Disadvantages
  • Data can only be written to the master.
  • There is no guarantee that data on master and slaves will be consistent at a given point in time. Due to asynchronous replication of data, there may be a small delay between data being written to the master and it being available on the slaves.
• Recommended uses
  • Scale-out solutions that require a large number of reads but fewer writes (for example, web serving).
  • Logging/data analysis of live data. Queries can be performed on the slave without affecting the operation of the master.
  • Online backup (high availability – when used in combination with heartbeat). However, since asynchronous replication is used, the data may be incomplete.
  • Offline backup – Snapshot of existing data at a given point of time can be taken by replicating the data to a slave.

MySQL Cluster:

It is a is a high-availability, high-redundancy synchronous solution that enables multiple MySQL instances to share database information. Data in a cluster can be read from or written to any node within the cluster, and information will be distributed to the other nodes.

This technology enables clustering of in-memory databases in a shared-nothing system. Shared-nothing means each component is allocated its own memory and disk and such an architecture allows the system to work with very inexpensive hardware, and with a minimum of specific requirements for hardware or software. Use of shared storage mechanisms such as SAN, NFS etc is not supported.

• Advantages
  • Offers multiple read and write nodes for data storage.
  • Provides automatic failover between nodes. Only transaction information for the active node being used is lost in the event of a failure.
  • Data on nodes is instantaneously distributed to the other data nodes.
• Disadvantages
  • Available on a limited range of platforms.

Only a development version is available for Microsoft Windows and Apple Mac OSX. It is not supported on SuSE Eneterprise Linux 11 as of this writing. Please check the following link for more details: http://www.mysql.com/support/supportedplatforms/cluster.html

• • Nodes within a cluster should be connected via a LAN; geographically separate nodes are not supported. However, you can replicate from one cluster to another using MySQL Replication, although the replication in this case is still asynchronous.
• Recommended uses
  • Applications that need very high availability, such as telecoms and banking.
  • Applications that require an equal or higher number of writes compared to reads.

Preparing Network:

Install “bridge-utils”

# sudo apt-get install bridge-utils

Now edit /etc/network/interfaces and setup a bridge using a static, private ip-address.

#cat /etc/network/interfaces

auto lo

iface lo inet loopback

auto br0

iface br0 inet static

address 192.168.3.251

netmask 255.255.255.0

gateway 192.168.3.1

bridge_ports eth0

bridge_fd 0

bridge_hello 2

bridge_maxage 12

bridge_stp off

#sudo /etc/init.d/networking restart

#brctl show

bridge name bridge id STP enabled interfaces

br0 8000.002215be747a no eth0

Now setup the static ip-address (192.168.3.251) and hostname in /etc/hosts. Please make sure that the hostname (cloud) does not appear in the line starting with 127.0.0.1.

#cat /etc/hosts

127.0.0.1 localhost

192.168.88.3 cloud cloud

# The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

Preparing Storage:

Install lvm2, nfs-kernel-server, iscsi-target and vblade.

#apt-get install lvm2 nfs-kernel-server iscsitarget vblade

Now prepare the dedicated partition to be used with lvm, then create a logical volume group “vol”.

#pvcreate /dev/sda3

#pvs

PV VG Fmt Attr PSize PFree

/dev/sda3 lvm2 — 186.23g 186.23g

#vgcreate vol /dev/sda3

Volume group “vol” successfully created

#vgs

VG #PV #LV #SN Attr VSize VFree

vol 1 0 0 wz–n- 186.22g 186.22g

Edit /etc/default/iscsitarget and set the iscsitarget to be started on boot-up.

#cat /etc/default/iscsitarget

ISCSITARGET_ENABLE=true

Then start the iscsitarget and nfs-kernel-server services.

#/etc/init.d/iscsitarget start

#/etc/init.d/nfs-kernel-server start

Prepare Storage:

For the openQRM Server Database backend we install “mysql-server”.

#apt-get install -y mysql-server

For sake of simplicity we have left the myslq-password empty.

Prepare KVM:

Install the “kvm” package.

#apt-get install -y kvm

Reading package lists… Done

Building dependency tree

…..<snip>

Setting up qemu-kvm (0.12.3+noroms-0ubuntu9) …

qemu-kvm start/running

Setting up kvm (1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9) …

Processing triggers for libc-bin …

ldconfig deferred processing now taking place

Install openQRM:

We build openQRM from the sources which are available in the openQRM Projects subversion repository.

#sudo apt-get install -y subversion make

Now checkout the openQRM sources from the svn repository.

#svn co https://openqrm.svn.sourceforge.net/svnroot/openqrm openqrm

#cd openqrm/trunk/src/

“make” requires a working internet connection. If no internet is available on this system you

can download the build-cache which will avoid any downloads using:

http://sourceforge.net/projects/openqrm/files/openQRM-4.6/source/openqrm-thirdparty-cache.tgz/download

#make

All compilation results are cached by the openQRM build-system. To ensure all components are build correctly simply run “make” again. The second (and every further “make” run) will just takes a few seconds.

Then run “make install”.

#make install

And finally initialize and start openQRM by “make start”.

#make start

Once initialization is complete, please configure your openQRM Server at: http://[server-ip-address]/openqrm/

-> User: openqrm -> Password: openqrm

“make start” triggers a check for the openQRM runtime-dependencies which will install all additional required packages automatically. At first start the openQRM Server is initialized.

Configure openQRM:

Login to your openQRM Server at http://localhost/openqrm [http://localhost/openqrm] . User and password is “openqrm”.

• First select the bridge-interface as the openQRM management network-device.
• Then select “myslq” as the Database to use as the openQRM backend.
• And then configure the Database connection credentials.

The openQRM is now fully configured and will forward to the Datacenter Dashboard.

There are a various solutions available for achieving high availability and scalability of MySQL data/service. ‘MySQL Replication’ and ‘MySQL Cluster’ the solutions offered and supported by MySQL. Third-party solutions such as DRBD (Distributed Replicated Block Device) and Heartbeat can also be used or one could implement a combination of these technologies.

The two solutions offered by MySQL are discussed here. Of these, MySQL Cluster is considered a feasible and reliable solution and is our subject of interest.

MySQL Replication:

It enables data from one MySQL server instance to be replicated to another MySQL server instance. It allows replication of data from a single master server to any number of slaves using a simple setup. However, this is an asynchronous replication solution, so the synchronization does not take place in real time, and data replication among all slaves is can not guaranteed always.

• Advantages
  • This can be implemented on any platform (which MySQL supports) and isn’t OS specific.
  • Replication is asynchronous and can be stopped and restarted at any time, and is easily manageable during network issues.
  • Data can be replicated from one master to any number of slaves. This is suitable for environments with heavy reads, but light writes, by spreading the load across multiple slaves.
• Disadvantages
  • Data can only be written to the master.
  • There is no guarantee that data on master and slaves will be consistent at a given point in time. Due to asynchronous replication of data, there may be a small delay between data being written to the master and it being available on the slaves.
• Recommended uses
  • Scale-out solutions that require a large number of reads but fewer writes (for example, web serving).
  • Logging/data analysis of live data. Queries can be performed on the slave without affecting the operation of the master.
  • Online backup (high availability – when used in combination with heartbeat). However, since asynchronous replication is used, the data may be incomplete.
  • Offline backup – Snapshot of existing data at a given point of time can be taken by replicating the data to a slave.

MySQL Cluster:

It is a is a high-availability, high-redundancy synchronous solution that enables multiple MySQL instances to share database information. Data in a cluster can be read from or written to any node within the cluster, and information will be distributed to the other nodes.

This technology enables clustering of in-memory databases in a shared-nothing system. Shared-nothing means each component is allocated its own memory and disk and such an architecture allows the system to work with very inexpensive hardware, and with a minimum of specific requirements for hardware or software. Use of shared storage mechanisms such as SAN, NFS etc is not supported.

• Advantages
  • Offers multiple read and write nodes for data storage.
  • Provides automatic failover between nodes. Only transaction information for the active node being used is lost in the event of a failure.
  • Data on nodes is instantaneously distributed to the other data nodes.
• Disadvantages
  • Available on a limited range of platforms.

Only a development version is available for Microsoft Windows and Apple Mac OSX. It is not supported on SuSE Eneterprise Linux 11 as of this writing. Please check the following link for more details: http://www.mysql.com/support/supportedplatforms/cluster.html

• • Nodes within a cluster should be connected via a LAN; geographically separate nodes are not supported. However, you can replicate from one cluster to another using MySQL Replication, although the replication in this case is still asynchronous.
• Recommended uses
  • Applications that need very high availability, such as telecoms and banking.
  • Applications that require an equal or higher number of writes compared to reads.

An Active Directory is a directory structure used on Microsoft Windows based computers and servers to store information and data about networks and domains. Here are some of the important parameters used in Active Directory.

Object: An object is anything that can be part of the directory. e.g: user, group, shared folder, printer etc.

Organizational Unit: It is a Active Directory container into which you can place users, groups, computers, and other organizational units in a hierarchial manner for ease of administration or management.

Domain: A domain is a collection of objects. The benefits of a domain are Centralized Administration – Management of the entire domain can be done with access to one database.

Single Logon Process – Access to network resources can be granted through a single logon.

Scalability – Very large networks can be created.

Organizational Unit: An organizational unit is a container with objects.

Tree: Trees are a collection of one or more Domains.

Forest: A forest is a collection of Trees.

Schema: The structure of Active Directory Domain Service database is the schema. A schema is a set of attributes used to describe a particular object in Active Directory.

Trust: A trust is a relationship established between two or more domains to allow users in one domain to be authenticated by a domain controller in another domain.

StarWind(Based on iSCSI) is used to create a shared storage area on a Windows Server for use with clustered environment.

Download the software from http://www.starwindsoftware.com and complete the installation steps. Then,

1. Start > All Programs > StarWind Software > StarWind
2. Click on Connect, provide the username and password.
3. Right click on Targets, you will get options to Add a new target.
4. Enable the correct device type.
5. Select the method to add a device (In case the device is already created, use: Mount Existing Device) else, create A Virtual Device.
6. Specify the disk location and size.
7. Specify Image File device parameters (Make sure you enable Allow Multiple Concurrent iSCSI Connections for a cluster environment).
8. Choose a target name and complete the installation.

iSCSI: Internet Small Computer Interface for linking data storage facilities

Uses port : 860 and 3260

The Microsoft iSCSI Initiator Service has to be started by running services.msc, where you’ll get the services listed over. To access a virtual disk,

1. Start > Administrative tools > iSCSI Initiator
2. iSCSI Initiator properties will open, then go to Discovery > Click on Discovery Portal > Add the IP or the domain and check the port (default port is 3260)
3. Once IP/domain is added, goto Favourite Targets and refresh to get the list of targets.
4. Under the Targets option you can connect/disconnect a particular virtual disk.

Once this is done, the virtual disks will be available under Server

Manager > Storage > Disk Management.

Hardware requirements

• Front-end Web server and application server computers: a dual-processor computer with processor clock speeds of 2.5-gigahertz (GHz) or higher and a minimum of 2 gigabytes (GB) of RAM.
• Back-end database server: a dual-processor computer with processor clock speeds of 2.0 GHz or higher and a minimum of 2 GB of RAM.

Software requirements Web and Application Server

• Microsoft Windows Server 2003 (Standard, Enterprise, Datacenter, or Web Edition) with Service Pack 1 (SP1)
• Microsoft .Net Framework 3.5
• The Web server and application server computers must be configured as Web servers running Microsoft Internet Information Services (IIS) in IIS 6.0 / IIS 7.0

Back-End Database Server

The back-end database server computer must be running Microsoft SQL Server 2008 or running Microsoft SQL Server 2005 or Microsoft SQL Server 2000 with Service Pack 3 (SP3) or later

Note : Install standalone sql server or clustered sql server 2008/2005 on a separate machine. It is required during the sharepoint installation.

Security accounts required for the Sharepoint installation

To install Office SharePoint Server 2007 in a server farm environment, at-least 2 accounts are required:

A user account that you can use to install Office SharePoint Server 2007 and run the SharePoint Products and Technologies Configuration Wizard. This account must be:

• A domain user account.
• A member of the Administrators group on each of your front-end servers.
• A member of the SQL Server Logins, which grants login access to your SQL Server instance.
• A member of the SQL Server Database Creator server role, which grants permission to create and alter databases.
• A member of the SQL Server Security Administrators server role, which grants permission to manage server logins.

Install and configure IIS

IIS is not installed or enabled by default in Windows Server 2008. To make your server a Web server, you must install and enable IIS.

• Click Start, point to All Programs, point to Administrative Tools, and then click Configure Your Server Wizard.
• On the Welcome to the Configure Your Server Wizard page, click Next.
• On the Preliminary Steps page, click Next.
• On the Server Role page, click Application server (IIS, ASP.NET), and then click Next.
• On the Application Server Options page, click Next.
• On the Summary of Selections page, click Next.
• Click Finish.

Install and configure 2007 Office SharePoint Server

It is recommended that you install and configure Office SharePoint Server 2007 on all of your front-end servers before you configure Office SharePoint Server 2007 services and create sites. If you want to build a minimal server farm configuration, and incrementally add front-end servers to expand the farm, you can install and configure Office SharePoint Server 2007 on a single front-end server and configure the front-end server as both a Web server and an application server. Regardless how many front-end servers you have in your server farm, you must have SQL Server running on at least one back-end database server before you install Office SharePoint Server 2007 on your front-end servers.

Run 2007 Office SharePoint Server Setup

1. Run Officeserver.exe, on one of your Web server computers.
2. On the Enter your Product Key page, enter your product key and click Continue.
3. On the Read the Microsoft Software License Terms page, review the terms, select the I accept the terms of this agreement check box, and then click Continue.
4. On the Choose the installation you want page, click Advanced.
5. click Complete, and then click Install Now.( that will act as an application server and Web server)
6. When Setup finishes, a dialog box appears telling you that you must complete the configuration of your server. Make sure that the Run the SharePoint Products and Technologies Configuration Wizard now check box is selected.

Run the SharePoint Products and Technologies Configuration Wizard

After Setup finishes, you can use the SharePoint Products and Technologies Configuration Wizard to configure Office SharePoint Server 2007. The SharePoint Products and Technologies Configuration Wizard automates several configuration tasks, including: installing and configuring the configuration database, installing Office SharePoint Server 2007 services, and installing SharePoint Central Administration. Use the following instructions to run the SharePoint Products and Technologies Configuration Wizard.

1. On the Welcome to SharePoint Products and Technologies page, click Next.
2. Click Yes in the warning dialog box that appears notifying you that some services might need to be restarted during configuration.
3. On the Connect to a server farm page, select the following option
4. If this is the first front-end server that you are configuring in your server farm, click No, I want to create a new server farm, and then click Next.
5. On the Specify Configuration Database Settings dialog box, in Database server, type the name of the computer that is running SQL Server, select the following option
6. If this is the first server that you are configuring in your server farm, type a name for your configuration database in Database name, or use the default database name.
7. In User name , type the user name of the account used to connect to the computer running SQL Server (be sure to type the user name in the format DOMAIN\username).
8. In Password, type the user’s password, and click Next.
9. On the Configure SharePoint Central Administration Web Application page, select the Specify port number check box and type a port number if you want the SharePoint Central Administration Web application to use a specific port, or leave the Specify port number check box unchecked if you do not care which port number the SharePoint Central Administration Web application uses.
10. On the Configure SharePoint Central Administration Web Application dialog box, select NTLM authentication (the default), click Next.
11. On the Completing the SharePoint Products and Technologies Wizard page, click Next.
12. On the Configuration Successful page, click Finish.

Adding nodes to the existing server farm

It is very similar to our base installation. we should install share point on the machines which you want to join to the server farm.

1. On the Welcome to SharePoint Products and Technologies page, click Next.
2. Click Yes in the warning dialog box that appears notifying you that some services might need to be restarted during configuration.
3. On the Connect to a server farm page, select the following option
4. If you have already configured your first server in your server farm, click Yes, I want to connect to an existing server farm, and then click Next.
5. On the Specify Configuration Database Settings dialog box, in Database server, type the name of the computer that is running SQL Server, select the following option
6. If this is the first server that you are configuring in your server farm, type a name for your configuration database in Database name, or use the default database name.
7. In User name , type the user name of the account used to connect to the computer running SQL Server (be sure to type the user name in the format DOMAIN\username).
8. In Password, type the user’s password, and click Next.
9. On the Configure SharePoint Central Administration Web Application page, select the Specify port number check box and type a port number if you want the SharePoint Central Administration Web application to use a specific port, or leave the Specify port number check box unchecked if you do not care which port number the SharePoint Central Administration Web application uses.
10. On the Configure SharePoint Central Administration Web Application dialog box, select NTLM authentication (the default), click Next.
11. On the Completing the SharePoint Products and Technologies Wizard page, click Next.
12. On the Configuration Successful page, click Finish.

If we can get the Ubuntu 10.04 latest server edition, this can be done easily. As it comes with an option to select whether we need to setup a normal server or Ubuntu Cluster. On the main cluster controller we can select the cluster option and also need to specify the IP range needed for the nodes. So, once the cluster server is setup, while installing on nodes, it automatically detects the node and necessary ssh-key authentication also will be done. We just need to give which IP from the range has to be used for the node.

Before starting would like to mention that, Eucalyptus has 3 major packages.

Cluster Controller:  provides support for the virtual network overlay

Cloud Controller: includes the front-end services and the Walrus storage system.

Node Controller: that interacts with XEN to manage individual VMs.

While doing the setup from package installation:

Cloud/Cluster/Storage/Walrus Front End Server(s)

1. Need to install Ubuntu server 10.04.

2. Necessary packages has to be installed

sudo apt-get install eucalyptus-cloud eucalyptus-cc eucalyptus-walrus eucalyptus-sc

While installting these packages we would be getting options to define the IP range and to specify the server’s hostname.

3. On the nodes also we need to install the server and update the archives.

4. Necessary packages has to be installed.

sudo apt-get install eucalyptus-nc

5. Next we need to set the primary interface of the node as a bridge. While creating virtual network interfaces, the node controller will add it to this bridge. Usually a bridge can be set by making below given modifications in the /etc/network/interfaces file.

auto br0
iface br0 inet dhcp
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

Once this is set need to restart the networking daemon.

6. In the /etc/eucalyptus/eucalyptus.conf VNET_BRIDGE has to be changed to $bridge. Then restart the node controller using

/etc/init.d/eucalyptus-nc restart

7. Now, the cloud controller and the all other controller like node, storage and warlus controller should be able to communicate each other via ssh keys. This has to be enabled. While installing cluster using 10.04 UEC cd, ssh-key authentication is done automatically.

To enable ssh keys, on the target controller (node, storage or warlus)

-> temporarily set a password for the eucalyptus user

sudo passwd eucalyptus

-> Then from the cloud controller

sudo -u eucalyptus ssh-copy-id -i ~eucalyptus/.ssh/id_rsa.pub eucalyptus@<IP_OF_NODE>

-> Then we can remove the temporary password.

sudo passwd -d eucalyptus

8. Next step is to define controller’s IP Addresses in the eucalyptus configuration file and then register

In /etc/eucalyptus/eucalyptus.conf file on the cluster controller

Define the shell variable CC_NAME in /etc/eucalyptus/eucalyptus-cc.conf

Define the shell variable CC_IP_ADDR in /etc/eucalyptus/eucalyptus-ipaddr.conf, as a space separated list of one or more IP addresses.

Define the shell variable WALRUS_IP_ADDR in /etc/eucalyptus/eucalyptus-ipaddr.conf, as a single IP address.

Define the shell variable SC_IP_ADDR in /etc/eucalyptus/eucalyptus-ipaddr.conf, as a space separated list of one or more IP addresses.

9. Now for each controller, we need to start the publication

sudo start eucalyptus-walrus-publication (Warlus controller)

sudo start eucalyptus-sc-publication (Storage controller)

sudo start eucalyptus-cc-publication (Cluster controller)

sudo start eucalyptus-nc-publication (Node controller)

10. Need to start the listeners on the cluster and cloud controllers

sudo start uec-component-listener

The regitration process can be verified by checking the log file /var/log/eucalyptus/registration.log

11. Now since the cloud is setup and the controllers have started communicating, the users of the cloud would need to donwload thier login credentials.

From your web browser access the URL: https://<cloud-controller-ip-address>:8443/

For the first time, username ‘admin’ and password ‘admin’ can be used(you will be prompted to change your password).

Then follow the on-screen instructions to update the admin password and email address.

Once the first time configuration process is completed, click the ‘credentials’ tab located in the top-left portion of the screen.

Click the ‘Download Credentials’ button to get your certificates

Save them to ~/.euca

Unzip the downloaded zipfile into a safe location (~/.euca) unzip -d ~/.euca mycreds.zip

12. Next need to setup EC2 API and AMI tools on your server using X.509 certificates.

For this on the cloud controller sudo apt-get install euca2ools

13. Next step is to install necessary images on the nodes. There will be few default stored images, we can either use it or else usually an image according to the needs would be created and installed.

The cloud is setup and applications are deployed to it, we would need to make changes accordingly.

The main advantage of cloud computing is that both small and medium sized businesses can instantly obtain the benefits of the enormous infrastructure without having to implement and administer it directly. This also permits accessibility to multiple data centers anywhere on the globe. It also means that as the need for resources increases, companies can add additional service as and when needed from the cloud computing vendor.