Importing VMDK to AWS using EC2 CLI

Yesterday I showed how we can import OVA file to AWS as an AMI image, today I'm going to demonstrate how we can import actual VMDK file that is exported from any virtualize software as an instance in EC2.

Before we started, there are prerequisites to be done first.

1. Setting up EC2 CLI to work with your terminal. Test it with simple command:

> ec2-describe-regions

2. Create S3 bucket, name it anything you want. Mine is "instance-manifest". This bucket will save our VMDK's manifest and small VMDK parts then convert them to EC2 instance.

3. Now use this command to start importing VMDK:

> ec2-import-instance -o <Access Key> -w <Secret Key> -t <Instance Type> -a x86_64 -f <File Type> -p <Platform> -b <Bucket name> -s <EBS disk size> -z <Availability Zone> "\path\to\file.vmdk"

Change to red highlight to appropriated value to suit your environment.

Example Command:

> ec2-import-instance -o Your_Access_Key -w Your_Secret_Key -t t2.micro -a x86_64 -f VMDK -p Linux -b instance-manifest -s 10 -z ap-southeast-1a "D:\Downloads\ubuntu-disk1-streamed.vmdk"

4. Let's check to status of importing with the following command:

> ec2-describe-conversion-tasks --region <region_name>

or

> ec2-describe-conversion-tasks <task_id>

5. When the importing is done. Go to AWS console, EC2 then you'll see new instance from conversion.

Importing OVA to AWS as AMI using AWS CLI (Part 2 of 2)

Okay, Now we are going to import our OVA files using AWS CLI tool. I've already created S3 bucket in my AWS account and uploaded OVA files also, put it in bucket named "exported-vmdk"

Let's open our terminal/cmd console or whatever console you're using with AWS CLI and type the following command to import OVA file and convert it into AMI image.

Here's my example of the command:

>aws ec2 import-image --cli-input-json "{  \"Description\": \"Ubuntu OVA\", \"DiskContainers\": [ { \"Description\": \"First CLI task\", \"UserBucket\": { \"S3Bucket\": \"exported-vmdk\", \"S3Key\" : \"ubuntu.ova\" } } ]}"

****S3Bucket is the bucket name on S3, S3Key is the name of the OVA file you're going to import
****Don't forget to change the command to suit with your environment.


Now If it successes, you can track the importing status by using the given image ID with this command:

>aws ec2 describe-import-image-tasks –import-task-ids <image-ID>

Once everything is done, login to AWS console, go to EC2 and look for AMI section. There will be an AMI image wait for you to create a new instance.

Importing OVA to AWS as AMI using AWS CLI (Part 1 of 2)

It's been awhile since my last post again. Now I'm busy with AWS and today I'm going to show you how we can upload our OVA file (exported from any virtualize vendor) to our S3 bucket and convert it to AMI which is an image format using by AWS.

Before we started, there are some prerequisites that needed to be done.

1. Download and install AWS CLI tool from AWS website, click here

(Don't forget to set your AWSCLI_HOME)

2. Now we need "VM Import Service Role" this will uses a role in your AWS account to perform certain operations. The role must be created with the name "vmimport" with the following policy and trust entities. Create a file name "trust-policy.json" with following policies:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"",
         "Effect":"Allow",
         "Principal":{
            "Service":"vmie.amazonaws.com"
         },
         "Action":"sts:AssumeRole",
         "Condition":{
            "StringEquals":{
               "sts:ExternalId":"vmimport"
            }
         }
      }
   ]
} 

Then use this command to create service role:

> aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json

3. Then create a file named "role-policy.json" to use as policy for this service role: 

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":[
            "arn:aws:s3:::<disk-image-file-bucket>"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObject"
         ],
         "Resource":[
            "arn:aws:s3:::<disk-image-file-bucket>/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "ec2:ModifySnapshotAttribute",
            "ec2:CopySnapshot",
            "ec2:RegisterImage",
            "ec2:Describe*"
         ],
         "Resource":"*"
      }
   ]
}

Then create policy and replace <disk-image-file-bucket> with the appropriate Amazon S3 bucket where the disk files are stored. Run this command to attach the policy to the role created above:

> aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

4. If you're logged on as an IAM user, you'll need the following permissions in your IAM policy to import or export a VM:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:DeleteObject",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": ["arn:aws:s3:::mys3bucket","arn:aws:s3:::mys3bucket/*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CancelConversionTask",
        "ec2:CancelExportTask",
        "ec2:CreateImage",
        "ec2:CreateInstanceExportTask",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DescribeConversionTasks",
        "ec2:DescribeExportTasks",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeInstances",
        "ec2:DescribeTags",
        "ec2:ImportInstance",
        "ec2:ImportVolume",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances",
        "ec2:ImportImage",
        "ec2:ImportSnapshot",
        "ec2:DescribeImportImageTasks",
        "ec2:DescribeImportSnapshotTasks",
        "ec2:CancelImportTask"
      ],
      "Resource": "*"
    }
  ]
}

Now we're ready to import our OVA files using AWS CLI tool. Let's continue on part 2...

Nested KVM in ESXi5.5

Since I tested ESXi 5.5 for awhile, I wanted to learn something new like KVM-QEMU (which i heard they have best performance and less overhead) So, I installed them inside ubuntu guest OS, but before doing that, ESXi and guest needed to be configure to run KVM.

Here are the pre-step I took
1. Go to your guest and upgrade hardware version to at least 9 (Well, It's on ESXi 5.5 so I upgraded my guest's hardware version to 10.
2. Since hardware version 10 not allow you to edit the setting with vSphere Client (that's bad), you need vSphere web client to do so.
3. Before edit guest's configuration, ssh to your ESXi host and add this line to /etc/vmware/config

vhv.enable = "TRUE"

4. Edit the guest's setting with vSphere web client, go to "Virtual Hardware" > under CPU option

5. After all this done, check the kvm compatible by install cpu-checker inside ubuntu

#sudo apt-get install cpu-checker

#sudo /usr/sbin/kvm-ok

If there's an output saying something like KVM acceleration can be used, then we're good to go.

Install/Update drivers on esxi 5.1

Well, I've been update many drivers lately. Not sure why after we updated the esxi 5.1 build patch from 7xxxxx to 1xxxxxx version, some of the drivers remain the same. Anyway, manual update is the best solution for some system that use variety of hardware's components.

Usually, you need to transfer a driver to the host first. For me, I'd unzip the downloaded file first (to get xxx-offlinebundle.zip) and then just throw everything inside the decompressed folder to the host.

The command to install/update the driver.
#esxcli software vib install --depot=/path/to/offlinebundle.zip
or you can use
#esxcli software vib update --depot=/path/to/offlinebundle.zip

To make sure the driver installed successfully, please reboot the host and check with this command.
#esxcli software vib list |grep <The driver's name>

To remove the driver
#esxcli software vib remove --vibname=<name of your driver, you can find with the previous command>

Don't forget to reboot the server every time you make a change to the driver.
And please enter maintenance mode before taking any action.

Moving ESX host from 4.1 cluster to 5.1 cluster with EVC enabled.

Because we have an updated plan for our entire virtual system (From 4.1 to 5.1) I created new vCenter and Clusters enable with the same version of EVC. If you need to move the hosts without downtime, you need to do these step carefully.

1. Make sure both cluster (4.1 and 5.1) have the same version of EVC. Just created a new empty cluster within vCenter 5.1 and enable EVC feature.
2. Remove hosts from old cluster.
3. Add the hosts to vCenter 5.1. Under Data Center not the cluster
4. Drag the hosts into your cluster.

That's it !
Seems easy, but if you have difference version of EVC, you might need to follow the step from KB which may requires downtime if you don't have enough resource to hold your virtual machines.

Another way to install VMWare Server 2 on Ubuntu 10.04

The first way I've provided is not the only way that you could follow in order to install VMWare Server 2.
 Now Let's consider this method of installation.

1. Download VMware Server (VMware-server-2.0.2-203138.i386.tar.gz).
2. Download the VMware Server 2 update patch #2

$ wget -N http://risesecurity.org/~rcvalle/VMware-server-2.0.2-203138-update-2.patch

3. Extract VMWare Server

$ tar -xzf VMware-server-2.0.2-203138.i386.tar.gz

4. Extract VMWare Server Modules

$cd vmware-server-distrib/lib/modules/source

$tar -xf vmci.tar

$tar -xf vmmon.tar

$tar -xf vmnet.tar

$tar -xf vsock.tar

5. Apply the patch

$cd vmware-server-distrib/

$patch -p1 < ../VMware-server-2.0.2-203138-update-2.patch

6. Archive VMWare Server Modules again

$cd vmware-server-distrib/lib/modules/source

$rm -f vmci.tar

$rm -f vmmon.tar

$rm -f vmnet.tar

$rm -f vsock.tar

$tar -cf vmci.tar vmci-only/

$tar -cf vmmon.tar vmmon-only/

$tar -cf vmnet.tar vmnet-only/

$tar -cf vsock.tar vsock-only/

7. Run the installer script as root

$cd vmware-server-distrib/

$sudo ./vmware-install.pl

Note :
1. Don't forget to add a new administrative user when it asked for.
2. The VMWare Remote Console Plug-in doesn't work properly on Ubuntu 10.04. You can set the environment variable  VMWARE_USE_SHIPPED_GTK before running the VMWare Remote Console Plug-in. You can do this by add the following line to your ~/.profile

export VMWARE_USE_SHIPPED_GTK=yes

Credits : Rise Security

Where is my datastore!? (VMWare Server 2 on Ubuntu Lucid)

reJust put your own existing virtual machine in to this place and added it into your datastore with vmware-web console.

/var/lib/vmware/Virtual Machines

That's all :)

when 503 Service Unavailable comes up!! (cause from vmware server2)

Ok after last night I've recently finished my vmware server 2 installation.
I came back this evening and found that I couldn't connect to my vmware web console.
And it keeps telling me that 503 service unavailable thing. so I google and found some interesting solution.

Let's start with stopping all of its services.

$/etc/init.d/vmware stop

If it resulted in something like this. (Something just resulted in "Failed")

Stopping VMware services:
Virtual machine monitor failed
Bridged networking on /dev/vmnet0 done
DHCP server on /dev/vmnet1 done
Host-only networking on /dev/vmnet1 done
DHCP server on /dev/vmnet8 done
NAT service on /dev/vmnet8 done
Host-only networking on /dev/vmnet8 done
Virtual ethernet failed

or when you try to start your vmware and you resulted in

$sudo /etc/init.d/vmware start
vmware is installed, but it has not been (correctly) configured
for this system. To (re-)configure it, invoke the following command:
/usr/bin/vmware-config.pl.

It's telling you to run vmware-config.pl again. So I try to do as it said. It resulted in this.

The following VMware kernel modules have been found on your system that were

not installed by the VMware Installer.  Please remove them then run this

installer again.

vmmon

vmci

vmnet

I.e. - 'rm /lib/modules/2.6.28-11-generic/misc/<ModuleName>.{o,ko}'

Execution aborted.

Then you need to remove the 3 mods that it mentioned with rmmod.
For me "rmmod" doesn't work. So I use hard remove (just delete it where they are.)
How to find it? easy!

$locate [THINGS YOU WANT TO FIND]

mine is in the /lib/modules/2.6.32-24-server/misc/
So I removed all of them

$rm /lib/modules/2.6.32-24-server/misc/*

There you go! now you can stop vmware services and reconfigure your vmware-config.pl

$/etc/init.d/vmware stop

$/usr/bin/vmware-config.pl

 Credits : Ubuntu forum 

Installing VMWare Server 2 on Ubuntu Lucid Lynx (10.04)

After conversion those files, now it's time for a new clean install ubuntu and vmware server 2.
I'll skip the ubuntu installation, it's the thing you gotta do by your own, not so hard.

Let's take a look in the first step before install vmware server 2 after clean installed of ubuntu.

1. Don't forget to update and upgrade all the necessary packages.

sudo apt-get update 

sudo apt-get upgrade 

2. Build essential packages

sudo apt-get install buil-essential

3. If you don't have gcc, you might want to install this package before the installation.

sudo apt-get install gcc 

 4. Download the vmware package and put in somewhere you want to.
 5. Because the ubuntu 10.04 has update its kernel to 2.6.32 so you need to patch you vmware package before apply the installation. Go download script from here
 6. Run the script with super user rights either in the same folder where you have downloaded the server archive (mine is /home/mazaruz/VMware-server-2.0.2-203138.x86_64.tar.gz.gz So I put the scripts in /home/mazaruz/ )

chmod +x vmware-server.2.0.1_x64-modules-2.6.30.4-fix.sh

sudo ./vmware-server.2.0.1_x64-modules-2.6.30.4-fix.sh [PATH_TO_VMWARE_ARCHIVE]

 7. Then it'll lead you to the installation of vmware itself. just do as it said. Don't forget to add your new user for administration your vmware server. 
 8. the license key can be found in vmware website when you registered there. just copy it.
 9. All is finish!!

Credits : Radu Thank you for a nice script :)

Migration back from Esxi4 to Server2

It looks like my server can't handle this esxi anymore.
All of the virtual machines slow down for weeks.
So, I've been trying to do migration for 3 days.
And Today I've found a way to do it easily.

First you gotta have a VMware vCenter Converter Standalone Product
you can download it for free at vmware.com

Now you got installed, It will be like this.

Then just select Convert Machine on the left side of the panel.
It will ask you whether type you want to convert. I choose the first one
"VMware Infrastructure virtual machine"
Don't forget to enter your esxi's address and username with password.

Then choose the virtual machine you'd like to convert.

 Wait until it completed converting. Then it's all done!!