Archives are the heart and history of any blog. While the recent blog posts appear on the blog's home page, older posts are harder to find. Thanks to the archiving feature, so that the older posts can be found online at any time in the future...


Archive


In WEB based applications RESTful APIs are used to manipulate data between the browser and the server. User may enter some data which is formatted to a valid JSON object (Java Script Object Notation) before sending it to the server for processing. However at the server side this data may be stored in an XML format, which means we need to now convert this data from JSON to XML and vice-versa while sending data from server to the browser.

Below is a screenshot showing a sample XML on the left side and the generated JSON on the right side:


Conversion of JSON to XML isn't straight forward since they don't share common aspects about their structure. One may write a parser to read JSON elements one by one and then generate an XML but its a tedious job. Thanks to the power of Java Architecture for XML Binding (JAXB) which can easily parsing of XML (based on its dtd - document type definition) ie: convert an XML string into a java object and vice-versa. However pure jaxb implemetation isin't sufficient to solve our problem. We need a third party library named eclipselink.jar (download from here) which has the jaxb implementation to convert XML to JSON and vice versa. Alternatively we can use 3 sub jars from the eclipselink.jar - org.eclipse.persistence.core-2.5.1.jar, org.eclipse.persistence.antlr-2.5.1.jar, org.eclipse.persistence.moxy-2.5.1.jar

Steps:
1. Generate java classes using the dtd file of the XML
/usr/java/jdk1.7.0_60/bin/xjc
-dtd /home/cubicrace/workspace/json2xml/samples/DatabaseInventory.dtd
-d /home/cubicrace/workspace/json2xml/src/
-p com.utilities.jaxb

parsing a schema...
compiling a schema...
com/utilities/jaxb/Administrator.java
com/utilities/jaxb/DatabaseAttributes.java
com/utilities/jaxb/DatabaseInventory.java
com/utilities/jaxb/DatabaseName.java
com/utilities/jaxb/ObjectFactory.java


2. Create a jaxb.properties file with the below content:
javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory

Ensure that this file exists at the same location as the .class files of the above generated java files.

3. Add the eclipselink.jar to the classpath or project's java build path

4. Write a simple class to perform the conversion.
Below is a working standlone class:


package com.utilities;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;

import org.eclipse.persistence.jaxb.MarshallerProperties;
import org.eclipse.persistence.jaxb.UnmarshallerProperties;

import com.utilities.jaxb.DatabaseInventory;

public class JsonXmlConverter {

 public static final String base = "/home/cubicrace/workspace";
 
 static String srcJsonFile = base + "/json2xml/samples/dbInventory.json";
 static String srcXmlFile = base +  "/json2xml/samples/dbInventory.xml";    
 static String convertedJsonFile = base + "/json2xml/samples/convertedFile.json";
 static String convertedXmlFile = base + "/json2xml/samples/convertedFile.xml";
 
 public boolean createFile(String filePath) {
  File file = new File(filePath);
  file.delete();
  try {
   return file.createNewFile();
  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
   return false;
  }
 }
 
 private void jsonToXml(String jsonFile) throws Exception{
  File jsonObj = new File(jsonFile);
  JAXBContext jc = JAXBContext.newInstance(DatabaseInventory.class);
  Unmarshaller unmarshaller = jc.createUnmarshaller();
  unmarshaller.setProperty(UnmarshallerProperties.MEDIA_TYPE, "application/json");
  DatabaseInventory activity = (DatabaseInventory) unmarshaller.unmarshal(jsonObj);
  
  Marshaller xmlM = jc.createMarshaller();
  xmlM.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
  FileOutputStream fos = new FileOutputStream(new File(convertedXmlFile));
  xmlM.marshal(activity, fos);
  fos.close();
 }
 
 private void xmlToJson(String xmlFile) throws Exception{
  File xml = new File(xmlFile);
  JAXBContext jc = JAXBContext.newInstance(DatabaseInventory.class);
  Unmarshaller unmarshaller = jc.createUnmarshaller();
  DatabaseInventory activity = (DatabaseInventory) unmarshaller.unmarshal(xml);

  Marshaller marshaller = jc.createMarshaller();
  marshaller.setProperty( Marshaller.JAXB_FORMATTED_OUTPUT, true);
  marshaller.setProperty( MarshallerProperties.MEDIA_TYPE, "application/json");
  marshaller.setProperty( MarshallerProperties.JSON_INCLUDE_ROOT, true);
  FileOutputStream fos = new FileOutputStream(new File(convertedJsonFile));
  marshaller.marshal(activity, fos);
  fos.close();
 }
 
 public static void main(String[] args) {

  JsonXmlConverter jxc = new JsonXmlConverter();
    
  try {
   jxc.createFile(convertedJsonFile);
   System.out.println("Source XML: " +  srcXmlFile);
   jxc.xmlToJson(srcXmlFile);
   System.out.println("Converted Output JSON: " + convertedJsonFile);
   
   jxc.createFile(convertedXmlFile);
   System.out.println("Source JSON: " + srcJsonFile);
   jxc.jsonToXml(srcJsonFile);
   System.out.println("Converted Output XML: " + convertedXmlFile);
  } catch (Exception e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
  
 }

}


Note: Ensure to update the workspace variable in the above java class before executing in your local environment.

If you see the below error, it means the jaxb.properties file (step2 above) is missing at the expected location or has incorrect property value pair.
javax.xml.bind.PropertyException: name: eclipselink.media-type value: application/json
at javax.xml.bind.helpers.AbstractMarshallerImpl.setProperty(AbstractMarshallerImpl.java:337)
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.setProperty(MarshallerImpl.java:539)

Download the dtd, sample files, eclipselink.jar and source code as single zip: here

How to convert a given XML snippet to an equivalent JSON format and vice-versa

The RBI will soon be issuing ₹ 2000 denomination banknotes in the Mahatma Gandhi (New) Series, without the inset letter, bearing signature of Dr. Urjit R. Patel, Governor, Reserve Bank of India, and the year of printing '2016' printed on the reverse of the banknote.

The new denomination has picture of Mangalayan on the reverse, depicting the country’s first venture into the interplanetary space. The base colour of the note is magenta. The note has other designs like geometric patterns aligning with the overall colour scheme, both at the obverse and reverse.

RBI's website doesn't mention anything about the NGC chip embedded in the currency ... Seems like a rumour/fake news.

Certain rumours about the 2000 currency note:
  1. Apparently, the new notes will have nano-GPS chips installed on them, for tracking. These chips will reportedly allow satellites to track the currency, even at 120 metres below ground level.
  2. Further, rumours say that the nano-GPS chips work without any power source, which is quite interesting if true.
  3. Moreover, the satellites will reportedly track large amounts of money stored at any place, automatically, and pass on the information to the Income Tax department.
  4. The department can then track such currency and recover it, if it’s found to be illegal.
  5. Removing the nano-GPS will reportedly render the currency unusable in the first place. 

Tracking currency may not be true, WHY ? 

1. Real Fact:
World’s smallest fully integrated GPS receiver has been developed by OriginGPS Nano Spider, which measures 4 x 4 x 2.1 mm. Comparably, this device is really small (but not smaller than a currency note), and can open a new avenue of wearable devices – now this chip can track clothes, watches, electronic appliances and more. However thickness of 2.1mm is big to fit in a currency note which is 0.1-0.2 mm thick.

2. Common PHYSICS:
GPS satellites are located in the medium earth orbit which starts at 20,200 Kilometers from Earth's surface, so deflecting a signal that far without any energy is impossible (https://en.wikipedia.org/wiki/GPS_satellite_blocks). If this was an available technology, it would have been used in mobile phones and there was no need to use the A-GPS ( which uses your mobile's data connection and GPS sensor to detect your mobile's location )

3. Link to RBI's site for details about the ₹2000 note:
 https://rbidocs.rbi.org.in/rdocs/PressRelease/PDFs/PR1144EFECD860ED0D479D88AB8D5CA036FC35.PDF

4. A chip without any power source will be able to embed the currency series number to the received GPS signal and deflect it back ???  this is beyond imagination and against science laws ... 


Front Features (As seen on RBI website)
  1. See through register with denominational numeral 2000
  2. Latent image with denominational numeral 2000
  3. Denominational numeral २००० in Devnagari
  4. Portrait of Mahatma Gandhi at the centre
  5. Micro letters ‘RBI’ and ‘2000’ on the left side of the banknote
  6. Windowed security thread with inscriptions ‘भारत’, RBI and 2000 on banknotes with colour shift. Colour of the thread changes from green to blue when the note is tilted
  7. Guarantee Clause, Governor’s signature with Promise Clause and RBI emblem towards right
  8. Denominational numeral with Rupee Symbol, ₹2000 in colour changing ink (green to blue) on bottom right 
  9. Ashoka Pillar emblem on the right
  10. Mahatma Gandhi portrait and electrotype (2000) watermarks
  11. Number panel with numerals growing from small to big on the top left side and bottom right side
    For visually impaired
    Intaglio or raised printing of Mahatma Gandhi portrait, Ashoka Pillar emblem, bleed lines and identity mark
  12. Horizontal rectangle with ₹2000 in raised print on the right
    Seven angular bleed lines on left and right side in raised print 
Reverse Features (As seen on RBI website)
  1. Year of printing of the note on the left
  2. Swachh Bharat logo with slogan
  3. Language panel towards the centre
  4. Motif of Mangalayan
  5. Denominational numeral २००० in Devnagari

India - Nano GPS chip or NGC in currency


When you read the word snapshot what is the first thing that comes to your mind is , "A Photograph which preserves the best moments of your life". Technically snapshot is very much the same with the difference that it preserves the state of some digital resource. In VMware a disk "snapshot" is a copy of the VM's disk file (.vmdk) captured at a certain point in time. This snapshot preserves the disk file system and the files stored on it which can be of any type (including all the operating system files). So if something goes wrong with your virtual machine, you can restore it to a snapshot which was working previously.

One can also create snapshots for different versions/service-packs on an OS.
Hence snapshots can also be looked upon as version controlling mechanism at OS level. So if your computer was shut down abruptly or gets infected by virus, just revert to a snapshot.

So how do snapshots really work ? There's just one Thumb Rule to VMware's snapshot technology : "Snapshots only store the differences between the current state and the original state". It follows the copy-on-write scheme with every subsequent disk access. Lets try to understand what that means ...

Consider that you have a text file with the word "COMPUTING" stored in it.
This file is a sparse in nature : which means it spans across multiple blocks on the disk. Step 1 below demonstrates this scenario. The black lines indicate the links on to the stored data. For demonstration purpose lets consider that each block on disk has only one character.


Note : The blocks shown above contain only one character and is purely for example purpose. In real the block size could be of say 1MB or a sector on disk.
 
Now when you take a snapshot another file named Snapshot1.vmdk will be created. When you create a snapshot, any changes made on the original virtual disk image are not made on the original disk, but they are written to a new (snapshot) disk file. This action is very fast as there is no need to copy whole virtual disk image.

Thumb Rule : "While saving changed data blocks in a snapshot, all modified block will be saved first , followed by blocks which were deleted as compared to base disk blocks." As seen in Step 2 , blue block is linked at the end of the snapshot1.vmdk

Lets suppose that you take a snapshot after you have saved the word "COMPUTING" in the file. After the snapshot you modify the file by changing its last two blocks (letters N and G circled above) and clear the letter I. The new changed word is "COMPUTER" as show in the Step 2. The blue block above is nothing but an empty block created by deleting letter I. The blocks in RED represents the new snapshot1.vmdk disk which contains only the changed characters.

Thumb Rule : "While reading any file in current state read only the data accessible by first level links, irrespective of number of snapshots and original data of the file."

Reading the first links of the "Current State" disk from Step 2 (Green blocks) , word "COMPUTER" is retrieved and the size of snapshot1.vmdk is 3 blocks (2 filled and one empty). However since its a differential snapshot file its size is much less than original base disk (9 blocks).  Snapshot image size grows as you continue to change more and more data from your original virtual disk image (which remains untouched from the moment you took the snapshot).

Thumb Rule : "Size of a snapshot will always be less than the base disk, but in worst case it will be exactly the same size if all blocks were to be changed."


As seen in Step 3 , we now take another snapshot after saving the word "COMPUTER" in the file. On making more changes after snapshot2 , similar process is followed to create snapshot2.vmdk file. The new changed word in the file is  "CONTAINER". As a result now neither snapshot 1 nor the base file are written two, but are still referenced. Snapshot 2 will store the new changes as compared to snapshot 1. If you read the first level links of the green blocks from top to bottom , the word "CONTAINER" is read with the fact that only 5 letters are stored in snapshot2.

Below are the list of changes made to the file
--------------------------------------------------------------------------
Step1 - Base Disk  = COMPUTING
Step2 - Snapshot1 = COMPUTER
Step3 - Snapshot2 = CONTAINER

From the above scenarios its clear that taking snapshots in VMware involves only writing the differences in files changed from the time of the snapshot, not the complete virtual machine disk. This mechanism is similar to taking diff and patch in Unix, but in a more sophisticated way that diffs on a binary level with the knowledge of how a VMFS ( Virtual Machine File System ) is structured.

Now we have a clear idea about the Copy-On-Write Protocol - Every time a block from the base disk or previous snapshot is changed , copy it to the current delta or snapshot file, which implies that when you perform a snapshot restoration, it only has to rewrite the sectors that were modified since you took the snapshot. As a result snapshot revert is also super fast.

But the Question is, What happens when you revert to an older snapshot? The VM software throws away the contents of snapshot2.vmdk and starts over with  reading contents from snapshot1.vmdk. During this all the Blue links in Step3 are replaced with Green Links. ( this is a similar strategy followed in deleting a node from a linklist ... Pure programming stuff ). Note : Links to Snapshot1 and Base are not changed.

There are two more important aspects of Snapshot management : Discarding a snapshot and Merging a snapshot (reverting to a non-immediate parent snapshot).

A KB article from VMware on Snapshot management.
http://www.youtube.com/watch?feature=player_embedded&v=rj8ugLrmU-M

How do Virtual Machine Snapshots work in VMware

Steven Paul "Steve" Jobs (February 24, 1955 – October 5, 2011) was an American information technology entrepreneur and inventor. He was the co-founder, chairman, and chief executive officer (CEO) of Apple Inc.


























More about him: https://en.wikipedia.org/wiki/Steve_Jobs

12 Rules of Success by Steve Jobs


You would fire a HTTP or HTTPS request to a Remote Server (HTTP) to send data. Sending JSON data / XML data is different than sending file data.
A HTTP multipart request is a HTTP request that HTTP clients construct to send files and data over to a HTTP Server. It is commonly used by browsers and HTTP clients to upload files to the server.

Let me explain how a browser encodes an html form before sending to server.

When you a select a file to be uploaded via a browser, it creates a multipart (Multiple Parts) request which consists of file content and other supporting information like filesize, boundary string, content type, additional parameters etc. Instead of URL encoding the form parameters, the form parameters (including the file data) are sent as sections in a multipart document in the body of the request.

Here's what a Multipart Request looks like (Firebug View):



Transfer-Encoding chunked:
Chunked transfer encoding is a data transfer mechanism in which data is sent in a series of "chunks".

Content-Type     multipart/form-data; boundary=---------------------------204472851627279345251768965
The Content-Type "multipart/form-data" specifies how the form data set is encoded. Form-data is a sequence of "name / value" pairs defined inside 'form' section. Boundary is just random unique string used to mark separation between different form-data sequences. Let me show you what I mean:

Post Data:

For example in the above form, data set is:
name=filename
value=color-scheme-6656-main.png

name=filePath
value=lib/

name=uploadedfiles[]
value=binary encode filecontent, which is a image file

The encoding "multipart/form-data" says, message contains a series of parts each part contains a name-value pair in the "form data set". Each part is separated using the "boundary" string and is specified in the below format:

// New line
// New line
--<value-of-boundary>
Content-Disposition: form-data; name="<control-name>" [filename="name-of-the-file-if-control-is-file"]
[Content-Type: <The-content-type-of-the-uploded-file-if-control-is-file>]
// New line
Actual file content either as clear text of binary data depending on file type
// New line
// New line

After all parts have been added to the request body, we need to indicate the end of all parts with a string that is formed by appending and prep-ending the boundary attribute by two dashes (--). For example it is ---------------------------204472851627279345251768965-- as show in the last red box of the above screenshot.
Once the request is ready & fired with all the above data, the receiver (server) will analyse all the information from the request and should be able to save the file will all the content intact.

Securely upload files to a Remote Server using JAVA client code - Part 1



Where Good Ideas come from ? ... by Steven Jhonson.
A Fantastic video filled with loads of real life examples and inspirational thoughts....


Good Ideas by Smarter people for a Smarter planet


Frisby is a REST API testing framework built on node.js and Jasmine that makes testing API endpoints easy, fast, and fun. Its as simple as tossing a frisby. Lets quickly setup Frisby on a Linux system.

Install NodeJS and Jasmine node:

curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash -
yum install -y nodejs
npm install -g jasmine-node

Install the required modules:

cd /usr/lib/node_modules/jasmine-node

#Form-Data for uploading files using REST API  
npm install --save form-data

#For sending and receiving WEB requests
npm install --save request

#Express is a minimal and flexible Node.js web application framework that
#provides a robust set of features for web and mobile applications.
npm install --save express

#For reading and validate the raw body of a readable stream 
npm install --save raw-body

#Our core framework for playing with REST APIs
npm install --save-dev frisby

#Used for sequencing REST calls or rather for sending synchronous 
#REST calls to the server
npm install --save sequenty

#Used to create a readable report from all the generated XML junit reports
npm install junit-viewer -g  

Done. Frisby is ready to be tossed. 
Lets quickly write a simple frisby test case and toss it.


//Load the frisby module
var frisby = require('frisby');

//Define our URL for making a REST request
var URL = 
'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q="setup frisby on linux"';

// globalSetup is for ALL requests
frisby.globalSetup({ 
 request: {
   'Accept': 'application/json', 
   'Content-Type': 'application/json',
  },
timeout: (60 * 1000)
});

frisby.create("GET_Google_Search") // Create a frisby and name it 
    .get(URL)                      // Make a GET http request
    .afterJSON(function(json) {    // Response converted JSON
                                   // Post processing, after the request
    })
    .expectStatus(200)             // Request success criteria
    .inspectBody()                 // Print the response to console
.toss();                           // Just throw the frisby


Project Setup:
Copy the above code into a new file and name it helloFrisby_spec.js
Since firsby.js is using jasmine, so the name of the test case file must end with spec.js
You need to create a link to the modules that were installed using npm.
This step is required only once and make sure that all your spec files are stored in that directory for execution.
mkdir -p /cubicrace/jasmine/rester
cd /cubicrace/jasmine/rester
npm link express form-data raw-body frisby request

Setup Directory Structure:
[cubicrace@textbox rester]$ ls -al
-rw-rw-r-- 1 cubicrace cubicrace   907 Apr 21 20:47 helloFrisby_spec.js
drwxr-xr-x 9 cubicrace cubicrace  4096 Apr  7 13:06 node_modules
 

Execution:
[cubicrace@testbox rester]$ which jasmine-node
/usr/bin/jasmine-node

[cubicrace@testbox rester]$ jasmine-node helloFrisby_spec.js

{"responseData": {"results":[{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.cubicrace.com/","url":"http://www.cubicrace.com/","visibleUrl":"www.cubicrace.com","cacheUrl":"http://www.google.com/search?q\u003dcache:KW0LoXuB_rUJ:www.cubicrace.com","title":"cubicrace","titleNoFormatting":"cubicrace","content":"How to \u003cb\u003eSetup Frisby on Linux\u003c/b\u003e. Apr 21, 2016. Short video about IBM past, present \nand future. Welcome to the cognitive era! This video reminds us as to why IBM ..."}],"cursor":{"resultCount":"1","pages":[{"start":"0","label":1}],"estimatedResultCount":"1","currentPageIndex":0,"moreResultsUrl":"http://www.google.com/search?oe\u003dutf8\u0026ie\u003dutf8\u0026source\u003duds\u0026start\u003d0\u0026hl\u003den\u0026q\u003d%22setup+frisby+on+linux%22","searchResultTime":"0.13"}}, "responseDetails": null, "responseStatus": 200}
.

Finished in 0.992 seconds
1 test, 1 assertion, 0 failures, 0 skipped

The JSON data in the output is a result of the .inspectBody() statement in the above code.
1 test and 1 assertion, is because of the .expectStatus(200) statement in the above code.

If while executing the frisby script, you get an error like "Cannot find module '<module_name>'", then follow the below resolution step:

npm install --save module_name

For example, if the error is: { [Error: Cannot find module '/root/.npm/form-data'] code: 'MODULE_NOT_FOUND' } then you can resolve this issue by executing the command:

npm install --save form-data.
Link to my stackoverflow resolution steps: MODULE_NOT_FOUND

How to Setup Frisby on Linux

+