Monday, 28 November 2011

How to share data among your .NET applications

Introduction

Suppose you have several different web applications that require users to login. It would be nice that users login only once and be able to move across different applications without logging in again. It would also be nice that once users make some choices in one application, they would apply to other applications as well. In other words, users would not know that they were using different web applications. Google is a good example: if you login to Google Voice, you can navigate to GMail without logging in again. There are probably many ways to accomplish this, one of them is to share session data among your applications. After you login to one application, your session will contain your information including the fact that you have successfully logged in. When going to a different application (by clicking a link in the current application, for example), the second application will fetch your current session data and the data will be used to determine whether you have already logged in. Besides user sessions, there are other types of data that you may want to share among your applications, such as common application settings, user preferences, user browsing history, the products that have been viewed or selected, etc.
Let's focus on sessions now. How do you share user session data among different web applications? With built-in session management in ASP.NET, there are three different modes you can choose from: in-proc (in memory), state server, and SQL Server. All of them do not allow you to share session data among different applications. It is possible to hack the ASP.NET session database to achieve the goal of sharing session data, this is not what I want to discuss in this article, however.
I wrote and published an in-memory session management tool on CodeProject a long time ago, it allowed sharing of session data. In the following sections, I am going to introduce a new database based solution that can be used to share data (including session data, of course) among different .NET applications. As you can see, it is simple and requires minimal coding to use, and it is not limited to web applications.

A New Database Session Tool

Here is what you need to do to install and use this solution (all files are included in the zip download):
  1. Create a new empty SQL Server database called SharedSessions.
  2. Run the SQL script file SharedSessions.sql in the new database to create tables and Stored Procedures.
  3. Create a user (login) for this new database, to be used by your applications, and grant it privileges to execute all Stored Procedures.
  4. Add a reference to SessionService.dll in your .NET applications and use it to create / delete session and store / retrieve session data.
First, let's see what is session in this article. Session is a collection of .NET objects, called data items, that you want to group together and store in the session database, it does not have to be related to any user session. Each session is uniquely identified by a session ID string. Each data item within a session is uniquely identified by an item key string. There is no restriction in my solution on session ID string and item key string besides database field length: for session ID, the maximal length is 100, for item key, the maximal length is 300. A session has an integer timeout value which is the number of minutes after which the session will expire: if the timeout value for a session is 30 and your applications do not read or write any data in this session for over 30 minutes, then the session is expired and data in the session can no longer be accessed.
By looking at the database created in the above using SQL Server Management Studio, you can see that there are only two tables: Session and SessionData.
The Session table stores information about session, there are four fields: SessionID, Timeout, DateTimeCreated, and LastAccess. The SessionData table stores session data items, there are three fields: SessionID, ItemKey, and ItemText. ItemKey is a text field that holds the actual data value, a Base64 encoding of .NET object.
Besides the two tables, there are seven Stored Procedures in the SharedSessions database.

  1. BrowseSessions returns database records that match the given session ID, item key, start date, and end date.
  2. CreateSession will create a new session in the database with the given session ID; if a session with the given ID already exists, then the session will be cleaned up (all data items in the existing session will be deleted) and the timeout value reset.
  3. GetSessionData returns the session data item with the given session ID and item key.
  4. RemoveSession will delete the session with the given session ID. If no session ID is provided, it will delete all expired sessions.
  5. RemoveSessionData will delete the session data item with the given session ID and item key.
  6. SetSessionData will insert / update a session data item in the database. If a session with the given session ID does not exist, it will be created first using the default timeout value (30 minutes).
  7. SetSessionTimeout will reset the timeout value for a session. If a session with the given session ID does not exist, it will be created.
SessionService.dll is a .NET library that can be used in any .NET application (not limited to web applications) to read from / write to the SharedSessions database. It contains one class, Session, with the following public static methods; all of them access the SharedSessions database by calling the Stored Procedures listed above:
  • StartCleanupProcess: This method starts a background thread to periodically delete expired sessions from the SharedSessions database.
  • CreateSession: It creates a new session.
  • SetSessionTimeout: It resets the timeout value for a given session. It will create a new session if there is none with the given session ID.
  • SetSessionData: It is used to insert / update a session data item. It will create a new session if there is none with the given session ID.
  • GetSessionData: It retrieves a session data item. If the data item does exist or the session has expired, it will return null.
  • RemoveSession: It deletes the session with the given session ID or deletes all expired sessions if no session ID is given.
  • RemoveSessionData: It deletes the session data item with the given session ID and item key.
  • SetAppSetting: To be explained later.
  • GetAppSetting: To be explained later.
  • ExecuteSQL: Returns a data set that is the output of a SQL query on the SharedSessions database.
In order to use SessionService.dll, your .NET application needs to reference it and have a database connection string for SharedSessions defined in the AppSetting "SessionDBConnection" in the application configuration file, or web.config file if it is a web application. Note that three methods can cause a new session to be created: CreateSession, SetSessionTimeout, and SetSessionData.

Types of session data items

So what kind of data can be saved in the SharedSessions database? Any .NET data object that is serializable can be saved into and retrieved from the database. In particular, you can save an object array as a session data item as long as each object in the array is a serializable object or an array of serializable objects. This gives us a lot of possibilities. See the following code example.
string SessionID = "my session id 123";
string ItemKey = "my item key 456";
object[] ItemValue = new object[3];
ItemValue[0] = "Hello, world";
ItemValue[1] = DateTime.Now;
string[] StateList = {"California", "Alaska", 
         "Washington", "Virginia"};
ItemValue[2] = StateList;
// store ItemValue into session database
SessionService.Session.SetSessionData(SessionID, ItemKey, ItemValue);
// retrieve ItemValue from session database
object[] ItemValueOutput = 
  (object[])SessionService.Session.GetSessionData(SessionID, ItemKey);
// print value of ItemValueOutput
System.Console.Out.WriteLine(ItemValueOutput[0].ToString());
System.Console.Out.WriteLine(ItemValueOutput[1].ToString);
string[] StateListOutput = (string[])ItemValueOutput[2];
for(int i=0; i<StateListOutput.Length; i++)
    System.Console.Out.WriteLine(StateListOutput[i]);
In the above code, ItemValue is an object array of length 3. The first item in the array is the string "Hello, world", the second item is the current date time, the third item is a string array of four state names. Such an object is stored into the SharedSessions database with only one call to SetSessionData and retrieved from the database by calling GetSessionData. Internally, the ItemValue object is serialized into a byte array and then encoded as ASCII text using the Base64 scheme. The Base64 text is what is stored as the ItemText field in the SessionData table. Here is the code in SessionService.dll that encodes an (serializable) object into text and decodes the text back to an object:
private static string ObjectToBase64(object obj)
{
    if (obj == null) return null;
    BinaryFormatter formatter = new BinaryFormatter();
    MemoryStream stream = new MemoryStream(64*1024);
    formatter.Serialize(stream, obj);
    byte[] output = stream.ToArray();
    return Convert.ToBase64String(output);
}

private static object Base64ToObject(string input)
{
    if (input == null || input == "") return null;
    byte[] data = Convert.FromBase64String(input);
    if (data == null) return null;
    BinaryFormatter formatter = new BinaryFormatter();
    MemoryStream stream = new MemoryStream(data);
    return formatter.Deserialize(stream);
}

Going beyond sessions

As I mentioned before, this solution is not limited to user sessions. It can be used to share other kinds of data among .NET applications. For example, you have a common dropdown list in several of your applications. The data items in the dropdown list can be stored in each application's configuration file, a better way is to store the data items in the SharedSessions database.
// store list items into database once
string AppSettingCategory = "All lists in my applications";
string AppSettingName = "How to kill myself option list";
object[] ItemList = new object[3];
ItemList[0] = new string[2]{"0", "Shoot myself with a gun"};
ItemList[1] = new string[2]{"1", "Listen to a boring project presentation"};
ItemList[2] = new string[2]{"2", "Use Lotus Notes"};
SessionService.Session.SetAppSetting(AppSettingCategory, AppSettingName, ItemList);

...
// retrieve and use list items from database in multiple applications
object[] ItemListOutput = 
  (object[])SessionService.Session.GeAppSetting(
   AppSettingCategory, AppSettingName);
for(int i=0; i<ItemListOutput.Length; i++)
{
    string[] Item = (string[])ItemListOutput[i];
    string ItemValue = Item[0];
    string ItemText = Item[1];
    System.Console.Out.WriteLine("Value: " + ItemValue + ", Text: " + ItemText);
}
Note that we use SetAppSetting and GetAppSetting instead of SetSessionData and GetSessionData. The difference is, with SetAppSetting, an internal session is created to store application data (the internal session will always have the string "__@@##" as the prefix of its session ID) and the timeout value of the internal session is 30 years instead of 30 minutes, so it will "never" expire!
This is great. The SharedSessions database is not just used for user sessions any more, you can save all kinds of useful application data and use the saved data in all of your .NET applications! But there is a flaw, a serious one. Let's say you save a list of items for a lot of common dropdown lists into SharedSessions, three months form now, how do you know what data has been saved for each dropdown list? Remember, the ItemText field in the SessionData table is a Base64 representation of a serializable .NET object, it will never be obvious what data object it represents. What if you want to make modifications to the data saved in SharedSessions? Do you need to write code to deserialize the data first and then print everything out to the console, if you forget what has been saved? Don't worry, I have another tool to solve this problem. It will be discussed in the next section.

SessionBrowser.exe, a tool to view and modify your data

SessionBrowser.exe is a simple Windows Forms application that uses SessionService.dll to access the SharedSessions database. It provides an easy way for you to view and modify the data you saved into the database.
How do you find your data using SessionBrowser.exe? If you know the session ID, type it into the "Session ID" text box and click the "Select Sessions" button. All sessions matching your session ID (you do not need to type the full session ID, by the way) will be displayed into the list view "Session List". The list view "Session List" will display up to 200 most active sessions that were created between "Start Date" and "End Date" and these sessions will be arranged in descending order of last access time. If your data is not found, then maybe your session was not created in the date range; in that case, you need to change the "Start Date" and/or "End Date" and click the "Select Sessions" button again.
When you select a session in the list view "Session List", up to 200 session data items for that selected session will be retrieved and displayed in the list view "Session Item List". If you select a session item from "Session Item List", an XML representation of the selected session data item will be displayed in the text box "Session Item Selected". You can modify the XML text and click the "Update Selected Item" button, and the selected session data item will be updated in the database!
Now let's look at each UI control on the application window.
  1. "Database" text box: This is where the "SessionDBConnection" setting in the configuration file is displayed. You can modify it to access a different database.
  2. "Session ID" text box: This is used when finding existing sessions and creating a new session and creating a new session item.
  3. "Item Key" text box: This is used when finding existing session items and creating a new session item.
  4. "Start Date" and "End Date" text boxes: These are used to limit sessions displayed in the list view "Session List" when the "Select Sessions" button is clicked.
  5. "Select Sessions" button: It uses the text boxes "Session ID", "Start Date", and "End Date" to find sessions in the database and display them in list view "Session List".
  6. "Select Session Items" button: It uses text boxes "Session ID", "Item Key" to find session items in the database and display them in the list view "Session Item List".
  7. "Clear All" button: It clears everything on the screen except the "Database" text box.
  8. "Session List" list view: When a session in this list view is selected, the data items in this session will be displayed in the list view "Session Item List".
  9. "Session Item List" list view: When a session item in this list view is selected, the session item data will be displayed in the text box "Session Item Selected" in XML format.
  10. "Session Item Selected" text box: It displays the XML representation of the selected session data item from the list view "Session Item List". You can use this text box to modify the selected session data item.
  11. "Timeout" text box: This is the timeout value for the new session when you click the "Create Session" button.
  12. "Create Session" button: Clicking this button will create a new session with the session ID in the "Session ID" text box and timeout value in the "Timeout" text box.
  13. "Delete Session" button: This button will delete the selected session from the list view "Session List". To prevent you from deleting a session by accident, you must also enter the session ID into the "Session ID" text box when deleting the selected session.
  14. "Create Session Item" button: This will create a new session item using the session ID in the "Session ID" text box, item key in the "Item Key" text box, and using an empty string as data value.
  15. "Update Selected Item" button: Click this button to update the selected session item after you have modified the XML content in the "Session Item Selected" text box.

XML representation of .NET object in SessionBrowser.exe

Here is an example of the XML representation of a selected session item:
<?xml version="1.0" encoding="UTF-8"?>
<SessionItemRoot>
    <SessionID>MySessionID</SessionID>
    <ItemKey>MyItemKey</ItemKey>
    <SessionItem>
        <System.String.Array>
            <System.String>Hello, world</System.String>
            <System.String>Test</System.String>
            <System.String>This is a test</System.String>
        </System.String.Array>
    </SessionItem>
</SessionItemRoot>
The above session item is obviously an array of System.String. If we modify the XML text in text box "Session Item Selected" as follows and click the "Update Session Item" button, the selected session item will be changed into an object array, with the first item a string "Yes, the end is now!", the second item a date time value "12/31/2012 23:59:59", and third item a boolean value "true":
<?xml version="1.0" encoding="UTF-8"?>
<SessionItemRoot>
    <SessionID>MySessionID</SessionID>
    <ItemKey>MyItemKey</ItemKey>
    <SessionItem>
        <System.Object.Array>
            <System.String>Yes, the end is now!</System.String>
            <System.DateTime>12/21/2012 23:59:59</System.DateTime>
            <System.Boolean>true</System.Boolean>
        </System.Object.Array>
    </SessionItem>
</SessionItemRoot>
If you want to know how to turn the above .NET object into XML or turn XML text into a .NET object, see code for the BuildXML method and the BuildObject method in SessionBrowser.exe.
Note: Unfortunately, not all serializable types can be handled by SessionBrowser.exe. Only basic types and arrays of basic types, and object arrays (each object in the object array can be any basic type, array of basic types, or object array, etc.) can be viewed and modified by SessionBrowser.exe. As I said, you can save any serializable .NET object into SharedSesssions, the catch is, you may not be able to view or update some of them using SessionBrowser.exe.
As with any other tool, this one can be used badly, causing endless nightmares. Yes, that is definitely possible if your applications are messing up each other's data in the SharedSessions database. In addition, it is harmful when swallowed and may cause birth defects in certain populations, ... well, I will stop here.

Silverlight: How to Send Message to Desktop Application

Summary

This is a very simple example showing how to send a message from Silverlight application to standalone desktop application.

Introduction

The example below shows how to use Eneter Messaging Framework to send text messages from Silverlight application to standalone desktop application using TCP connection.
(Full, not limited and for non-commercial usage free version of the framework can be downloaded from http://www.eneter.net/. The online help for developers can be found at http://www.eneter.net/OnlineHelp/EneterMessagingFramework/Index.html).
When we want to use the TCP connection in Silverlight, we must be aware of following specifics:
  • The Silverlight framework requires the policy server.
  • The Silverlight framework allows only ports of range 4502 - 4532.

Policy Server

The Policy Server is a special service listening on the port 943. The service receives '<policy-file-request/>' and responses the policy file that says who is allowed to communicate.
Silverlight automatically uses this service when it creates the TCP connection. It sends the request on the port 943 and expects the policy file. If the policy server is not there or the content of the policy file does not allow the communication, the TCP connection is not created.
Eneter Messaging Framework wraps the low-level socket communication and provides convenient API for the TCP communication. It also provides the Policy Server for you. Therefore the implementation is very simple. See the code below.

Desktop Application

The desktop application is responsible for starting the policy server and receiving messages.
The whole implementation is here:
using System;
using Eneter.Messaging.EndPoints.StringMessages;
using Eneter.Messaging.MessagingSystems.MessagingSystemBase;
using Eneter.Messaging.MessagingSystems.TcpMessagingSystem;

namespace Receiver
{
    class Program
    {
        static void Main(string[] args)
        {
            // Start policy server enabling Silverlight security to communicate via TCP.
            TcpPolicyServer aPolicyServer = new TcpPolicyServer();
            aPolicyServer.StartPolicyServer();

            // Create receiver of string messages.
            IStringMessagesFactory aStringMessageReceiverFactory = 
     new StringMessagesFactory();
            IStringMessageReceiver aStringMessageReceiver = 
  aStringMessageReceiverFactory.CreateStringMessageReceiver();
            aStringMessageReceiver.MessageReceived += StringMessageReceived;

            // Create TCP listening channel.
            // Note: Silverlight supports only ports of range 4502 - 4532
            IMessagingSystemFactory aTcpMessagingFactory = 
     new TcpMessagingSystemFactory();
            IInputChannel anInputChannel = 
  aTcpMessagingFactory.CreateInputChannel("127.0.0.1:4502");

            // Attach the input channel to the string message receiver 
            // and start listening.
            Console.WriteLine("Receiver is listening.");
            aStringMessageReceiver.AttachInputChannel(anInputChannel);
        }

        static void StringMessageReceived(object sender, StringMessageEventArgs e)
        {
            // Process incoming message.
            Console.WriteLine(e.Message);
        }
    }
}

Silverlight Application

The Silverlight application is responsible for sending text messages. (The communication with the policy server is invoked automatically by Silverlight before the connection is established.)
The whole implementation is very simple:
using System.Windows;
using System.Windows.Controls;
using Eneter.Messaging.EndPoints.StringMessages;
using Eneter.Messaging.MessagingSystems.MessagingSystemBase;
using Eneter.Messaging.MessagingSystems.TcpMessagingSystem;

namespace SilverlightApplication
{
    public partial class MainPage : UserControl
    {
        public MainPage()
        {
            InitializeComponent();

            // Create sender of string messages.
            IStringMessagesFactory aStringMessageReceiverFactory = 
      new StringMessagesFactory();
            myStringMessageSender = 
   aStringMessageReceiverFactory.CreateStringMessageSender();

            // Create output channel sending via TCP.
            // Note1: Silverlight supports only ports of range 4502 - 4532
            IMessagingSystemFactory aTcpMessagingFactory = 
      new TcpMessagingSystemFactory();
            IOutputChannel anOutputChannel = 
   aTcpMessagingFactory.CreateOutputChannel("127.0.0.1:4502");

            // Attach the output channel to the string message sender
            // to be able to send messages via TCP.
            myStringMessageSender.AttachOutputChannel(anOutputChannel);
        }

        // Send message when the button is clicked.
        private void button1_Click(object sender, RoutedEventArgs e)
        {
            string aMessage = textBox1.Text;
            myStringMessageSender.SendMessage(aMessage);
        }

        private IStringMessageSender myStringMessageSender;
    }
}
I hope you found the article useful. If you have any comments or questions, feel free to ask.

Magic of JQuery in ASP.NET, Simplifying AJAX

jQuery is a powerful tool that can be used to enhance our ASP.NET Application.
In the time of development we often face problem like “need to show certain data based on a button click” or “need to implement tab in pages where each tab contains costly DB operation”.
There is lots of way to to this using traditional AJAX like ASP.NET AJAX. But if we consider the hassle related with UpdatePanel then we can find a few people who will go for it.
Lets see how we can achieve this short of Magic in our application easily with JQuery!
For simplicity lets assume that we have one aspx page with one div and one button. After clicking the button one extensive database operation will be held and the div’s inner html will be updated without page refresh.
Its to be noted that in order to leverage this facility of accessing Page Method through client script we need to add script manger and need to enable page method.
<asp:ScriptManager ID="ScriptManager1" runat="server"
EnablePageMethods="true"></asp:ScriptManager>


<asp:panel ID="divExtensiveData" runat="server"></asp:panel>


<asp:Button ID="btnFetchData" Text="Fetch Data" runat="server" OnClientClick="fetchData(); return false;" />
Now lets write one method in code behind i.e. in aspx.cs file that will do the actual db operatio

[WebMethod]
public static string fetchData(string someParameter)
{
string result = string.Empty;
/// Do extensive db operation
/// Assign value to result
return result;
}
Now we will get back to our aspx page and will write one method to access this page method.
<script type="text/javascript">
function fetchData(parameter)
 { 
  PageMethods.fetchData(parameter,dataFetched,dataNotFetched);
 }
function dataFetched(result)
{
 var panelID = '<%=divExtensiveData.ClientID %>';
 $("#"+panelID).html(result);
}
function dataNotFetched()
{
 var panelID = '<%=divExtensiveData.ClientID %>';
 $("#"+panelID).html("<h1>Error Occured.</h1>");
}
</script>
Now explaining the page method call from script. In our call to page method  I have given three parameters whereas the page method takes only one. The second one is just  a reference of the method that will be executed after successfully page method call and the third one is the reference of the method that will be called if the page method encounter some error.
Just use this concept and be a magician of JQuery in ASP.NET!!!

Library for scripting SQL Server database objects with examples

Introduction

Couple months before I have written article about comparing databases. In my previous article I have described approach using SMO. Using SMO for script generation is very easy but there are some performance issues. I my case it took nearly 4 minutes to compare 2 databases with approximately 2500 objects respectively. After publication of this article I made decision to create my own scripting library and after 3 months I have almost complete version of this library. In this article I will describe library's principle and I show you how to use it on two examples. First example is about comparison of database schemas and second is documentation generating tool.

Background

This library uses dynamic management views for getting information about database objects. This approach is really fast. By this information the collections of objects are populated. When you want to use this library, the first step is to create an ObjectDb object which accepts connection string as parameter. Next you must create an ScriptingOptions object which specify what kind of objects do you want to script and the final step is to call FetchObjects method which takes ScriptingOptions parameter. When fetching of objects is completed, you can access scripted objects by accessing collections of ObjectDb object. Following list represents supported database objects:
  • Tables 
  • Indexes
  • Ddl triggers
  • Dml triggers
  • Clr triggers
  • Stored procedures
  • View
  • Application roles
  • Database roles
  • Users
  • Assemblies
  • Aggregates
  • Defaults
  • Synonyms
  • Xml schema collections
  • Message types
  • Contracts
  • Partition functions
  • Service queues
  • Full text catalogs
  • Full text stop lists
  • Full text indexes
  • Services
  • Broker priorities
  • Partition schemes
  • Remote service bindings
  • Rules
  • Routes
  • Schemas
  • Sql user defined functions
  • Clr user defined functions
  • User defined data types 
  • User defined types
  • User defined table types
Project ObjectHelper contains main classes which are located directly in project folder, database object classes which are located in DBObjectType folder and sql statements important for script generation located in SQL folder.
Note: This library can be used only for databases with compatibility level 90 (MS Sql Server 2005) or 100 (MS Sql Server 2008).  

Using the code

As I mentioned in previous chapter, ObjectHelper project contains several main classes. First of them is BaseDbObject class which is base class for all database object classes.
public class BaseDbObject
    {
        public string Name{get;set;}
        public long ObjectId{get;set;}
        public string Description { get; set; }
        public DateTime CreateDate { get; set; }
        public DateTime ModifyDate { get; set; }
    }
Next main class is FetchEventArgs class which serves as event class when object is scripted.
public class FetchEventArgs : EventArgs
    {
        public BaseDbObject DbObject;

        public FetchEventArgs(BaseDbObject obj)
        {
            DbObject = obj;
        }
    }
Most important is ObjectDB class. This class fetches all information about database objects and creates scripts according to ScriptingOptions class. This class has one constructor which accepts connection string parameter. This connection string is used for connection to database. It also has one event called ObjectFetched which fires when object is fetched (when data are collected and database object class is populated using this data).
public delegate void ObjectFetchedEventHandler(object sender, FetchEventArgs e);

    public class ObjectDb
    {
        readonly string _connString;
        private SqlDatabase _sqlDatabase;
        Hashtable _hsResultSets = new Hashtable();

        public ObjectDb(string connString)
        {
            _connString = connString;
            _tables = new List();
        }

        public event ObjectFetchedEventHandler ObjectFetched;

        protected virtual void OnObjectFetched(FetchEventArgs e)
        {
            if (ObjectFetched != null)
            {
                ObjectFetched(this,e);
            }
        }

        ...

    }
This class contains generic Lists of database objects.
public List<fulltextindex> FullTextIndexes {get { return _fullTextIndexes;}}
public List<dependency> Dependencies {get{return _dependencies;}}
public List<assembly> Assemblies {get{return _assemblies;}}

...

public List<userdefinedtype> UserDefinedTypes {get{return _userDefinedTypes;}}
public List<userdefinedtabletype> UserDefinedTableTypes {get{return _userDefinedTableTypes;}}
</userdefinedtabletype></userdefinedtype></assembly></dependency></fulltextindex>
FetchObjects method of this class serves for fetching objects. It takes ScriptingOptions parameter. According to ScriptingOptions sql statement for retrieving of information about database objects is generated using ScriptGenerator class. Here is an example demonstrating how scripts for table object are prepared:
if (so.Tables)
{
    sql.Append("SELECT COUNT(*) FROM sys.tables;");
    sql.AppendLine();
    ResultSets.Add("TableCount", resultSetCount++);
    sql.Append(GetResourceScript("ObjectHelper.SQL.Tables_" + so.ServerMajorVersion + ".sql"));
    sql.AppendLine();
    ResultSets.Add("TableCollection", resultSetCount++);

    if (so.DataCompression)
    {
        if (so.ServerMajorVersion >= 10)
        {
            sql.Append(GetResourceScript("ObjectHelper.SQL.TableDataCompression_" + so.ServerMajorVersion + ".sql"));
            sql.AppendLine();
            ResultSets.Add("TableDataCompressionCollection", resultSetCount++);
        }
        else
        {
            so.DataCompression = false;
        }
    }
    if (so.DefaultConstraints)
    {
        sql.Append(GetResourceScript("ObjectHelper.SQL.DefaultConstraints.sql"));
        sql.AppendLine();
        ResultSets.Add("DefaultConstraintCollection", resultSetCount++);
    }
    if (so.CheckConstraints)
    {
        sql.Append(GetResourceScript("ObjectHelper.SQL.CheckConstraints.sql"));
        sql.AppendLine();
        ResultSets.Add("CheckConstraintCollection", resultSetCount++);
    }
    if (so.ForeignKeys)
    {
        sql.Append(GetResourceScript("ObjectHelper.SQL.ForeignKeys.sql"));
        sql.AppendLine();
        ResultSets.Add("ForeignKeyCollection", resultSetCount++);
        ResultSets.Add("ForeignKeyColumnCollection", resultSetCount++);
    }
    sql.Append(GetResourceScript("ObjectHelper.SQL.Columns_" + so.ServerMajorVersion + ".sql"));
    sql.AppendLine();
    ResultSets.Add("ColumnCollection", resultSetCount++);
}
In this case ScriptingOptions are represented by object called so . If Tables property of so object is true then sql statements for tables are appended to sql StringBuilder variable. Scripts for tables are stored in Tables_X.sql files which are stored in SQL folder as embedded resource. The X in file name represents version of SQL Server. Now two versions are supported (90 for SQL Server 2005 and 100 for SQL Server 2010). You can see that sql statement is dynamic generated because only scripts for objects specified by ScriptingOptions are executed. After that, this dynamic sql statement is executed and data are retrieved and objects are fetched. Every time object is fetched, ObjectFetched event is fired.
if (so.Tables)
{
    DataTable dtTables = ds.Tables[int.Parse(_hsResultSets["TableCollection"].ToString())];
    foreach (DataRow drTable in dtTables.Rows)
    {
        var table = new Table();
        table.AnsiNullsStatus = bool.Parse(drTable["AnsiNullsStatus"].ToString());
        table.ChangeTrackingEnabled = bool.Parse(drTable["ChangeTrackingEnabled"].ToString());
        table.Description = drTable["Description"].ToString();
        …
        OnObjectFetched( new FetchEventArgs(table));
    }
}

Example

Now let’s take a look at the basic example. In following example I will show you how to use this library. The first step is to create ObjectDb object and pass connection string as parameter. Next you must create ScriptingOptions object and specify what kind of objects do you want to script. You can also specify other options like whether to script collation, identity ets. Keep in mind, that you must set ServerMajorVersion. This is important because according to this version scripts for objects are generated. You can set event handler for ObjectFetched event to monitor currently fetched object. The last step is to call FetchObject method which takes ScriptingOptions parameter. If you want to get scripts for tables, just loop through Tables property of ObjectDb object and call Script method of object.
var objDb = new ObjectDb("server='ANANAS\\ANANAS2009';Trusted_Connection=true;multipleactiveresultsets=false; Initial Catalog='AdventureWorks2008R2'");
var so = new ScriptingOptions { Tables = true, ServerMajorVersion = 10 };
objDb.ObjectFetched += ObjectFetched;
objDb.FetchObjects(so);
foreach (var table in objDb.Tables)
{
    Console.WriteLine("--------------------[" + table.Name + "]--------------------");
    Console.WriteLine(table.Script(so));
}

static void ObjectFetched(object sender, FetchEventArgs e)
{
    Console.WriteLine("Fetched: " + e.DbObject.Name);
}

Database comparison tool

main.jpg Next example which demonstrates how to use this library is database comparison tool. Sometimes when developers work on big systems some inconsistencies in database objects occur. I have faced this problem many times and I decided to create tool which allows me to compare database objects. In my previous article I have created tool based on SMO but this approach was very slow. I have modify this project by using my own scripting library and the performance of this tool rapidly increased. DBCompare project consists of 5 screens: Login, MDIMain, ObjectCompare, ObjectFetch and ScriptView. It also uses external component called DiffereceEngine which is used as a base class for script comparison. More about that class can be find here.

Login screen

login_screen.jpg The login screen is used for creating a connection to the databases. It has two tabs. First tab serves for entering connection information like server, authentication and database name.
scriptingoptions_screen.jpg In second tab you can specify scripting options. Here you can choose what kind of object do you want to script and other options like whether to script identities, collations etc. Here is a list of supported options:
Indexes
Clustered Indexes Gets or sets a Boolean property value that specifies whether statements that define clustered indexes are included in the generated script.
Full Text Indexes Gets or sets the Boolean property value that specifies whether full text indexes are included in the generated script.
Non-Clustered Indexes Gets or sets the Boolean property value that specifies whether non-clustered indexes are included in the generated script.
Misc
Aggregates Gets or sets the Boolean property value that specifies whether aggregates are included in the list of scripted objects.
Script ANSI nulls Gets or sets the Boolean property value that specifies whether to script ANSI nulls.
Script Dependencies Gets or sets the Boolean property value that specifies whether dependencies are included.
Script Quoted identifiers Gets or sets the Boolean property value that specifies whether to script Quoted identifiers
Synonyms Gets or sets the Boolean property value that specifies whether synonyms are included in the list of scripted objects.
XML Schema Collections Gets or sets the Boolean property value that specifies whether xml schema collections are included in the list of scripted objects.
Programmability
Assemblies Gets or sets the Boolean property value that specifies whether assemblies are included in the list of scripted objects.
CLR User Defined Functions Gets or sets the Boolean property value that specifies whether clr user defined functions are included in the list of scripted objects.
Defaults Gets or sets the Boolean property value that specifies whether defaults are included in the list of scripted objects.
Rules Gets or sets the Boolean property value that specifies whether rules are included in the list of scripted objects.
SQL User Defined Functions Gets or sets the Boolean property value that specifies whether sql user defined functions are included in the list of scripted objects.
Stored Procedures Gets or sets the Boolean property value that specifies whether stored procedures are included in the list of scripted objects.
Views Gets or sets the Boolean property value that specifies whether vies are included in the list of scripted objects.
Security
Application Roles Gets or sets the Boolean property value that specifies whether application roles are included in the list of scripted objects.
Schemas Gets or sets the Boolean property value that specifies whether schemas are included in the list of scripted objects.
Users Gets or sets the Boolean property value that specifies whether users are included in the list of scripted objects.
Service Broker
Broker Priority Gets or sets the Boolean property value that specifies whether broker priorities are included in the list of scripted objects.
Message Types Gets or sets the Boolean property value that specifies whether message types are included in the list of scripted objects.
Remote Service Bindings Gets or sets the Boolean property value that specifies whether remote service binding are included in the list of scripted objects.
Routes Gets or sets the Boolean property value that specifies whether routes are included in the list of scripted objects.
Service Queues Gets or sets the Boolean property value that specifies whether service queues are included in the list of scripted objects.
Services Gets or sets the Boolean property value that specifies whether services are included in the list of scripted objects.
Contracts Gets or sets the Boolean property value that specifies whether contracts are included in the list of scripted objects.
Storage
Database roles Gets or sets the Boolean property value that specifies whether database roles are included in the list of scripted objects.
Full Text Catalog Path Gets or sets the Boolean property value that specifies whether full text catalog paths are included in the list of scripted objects.
Full Text Catalogs Gets or sets the Boolean property value that specifies whether full text catalogs are included in the list of scripted objects.
Full Text Stop Lists Gets or sets the Boolean property value that specifies whether full text stop lists are included in the list of scripted objects.
Partition Functions Gets or sets the Boolean property value that specifies whether partition functions are included in the list of scripted objects.
Partition Schemes Gets or sets the Boolean property value that specifies whether partition schemes are included in the list of scripted objects.
Tables
Check Constraints Gets or sets the Boolean property value that specifies whether check constraints are included in the list of table's constraints
Collation Gets or sets the Boolean property value that specifies whether to include the Collation clause in the generated script.
Data Compression Gets or sets the Boolean property value that specifies whether to include the DATA_COMRESSION clause in the generated script.
Default Constraints Gets or sets the Boolean property value that specifies whether default constraints are included in the list of table's constraints
Foreign Keys Gets or sets the Boolean property value that specifies whether dependency relationships defined in foreign keys with enforced declarative referential integrity are included in the script.
No File Stream Gets or sets an object that specifies whether to include the FILESTREAM_ON clause when you create VarBinaryMax columns in the generated script.
No Identities Gets or sets the Boolean property value that specifies whether definitions of identity property seed and increment are included in the generated script.
Primary Keys Gets or sets the Boolean property value that specifies whether primary key constraints are included in the list of table's constraints
Script Tables Gets or sets the Boolean property value that specifies whether tables are included in the list of scripted objects.
Unique Constraints Gets or sets the Boolean property value that specifies whether unique constraints are included in the list of table's constraints
Triggers
CLR Triggers Gets or sets the Boolean property value that specifies whether clr triggers are included in the list of scripted objects.
DDL Triggers Gets or sets the Boolean property value that specifies whether ddl triggers are included in the list of scripted objects.
DML Triggers Gets or sets the Boolean property value that specifies whether dml triggers are included in the list of scripted objects.
Types
User Defined Data Types Gets or sets the Boolean property value that specifies whether user defined data types are included in the list of scripted objects.
User Defined Table Types Gets or sets the Boolean property value that specifies whether user defined table types are included in the list of scripted objects.
User Defined Types Gets or sets the Boolean property value that specifies whether user defined types are included in the list of scripted objects.

ObjectFetch screen

objectfetch_screen.jpg This screen is the second most important part of application. Here all of the objects are fetched, scripted and then compared. Every object when is fetched is stored in ScriptedObject object:
public class ScriptedObject
{
    public string Name="";
    public string Schema="";
    public string Type="";
    public DateTime DateLastModified;
    public string ObjectDefinition="";
    public Urn Urn; 
}
When all object are fetched and scripted they are compared and next stored in DataTable object which is passed to the ObjectCompare screen.

ObjectCompare screen

This screen is the main part of project. Here you can compare database objects and see differences between them. This screen has three parts. In first part (left panel) you can select what kind of database objects do you want to compare. In second part (list of database objects) are objects devided into 4 parts:
  • Objects that exist only in DB1
  • Objects that exist only in DB2
  • Objects that exist in both databases and are different
  • Objects that exist in both databases and are identical
object_list.jpg
In the third part you can see the differences between database objects. When you click on one of the items in the list of objects, scripts of the selected objects are compared and displayed.
script_compare.jpg
All objects are stored in the DataTable dbObjects which has six columns:
  • ResultSet - it specifies the group into which the object belogns (objects that exist only in DB1 [1], objects that exist only in DB2 [2], ...).
  • Name - name of the database object.
  • Type - type of the database object.
  • Schema - schema into which the database object belongs (not all objects belong to a database schema).
  • ObjectDefinition1 - if two databases have objects with the same name and type but with different definition, ObjectDefinition1 stores the definition of the database object of the source database.
  • ObjectDefinition2 - if two databases have objects with the same name and type but with different definition, ObjectDefinition2 stores the definition of the database object of the target database. If the object exists only in one of the databases, this property is blank and the definition of the object is stored in ObjectDefinition1.

Database documentation tool

dbdoc_main.jpg Here is an another example how to use my scripting library. This little tool allows you to create html base documentation of your database. For the moment this tool generate documentation for following objects but in the future I will add rest of the objects:
  • Defaults
  • Message types
  • Contracts
  • Service queues
  • Services
  • Broker priorities
  • Indexes
  • Aggregates
  • Assemblies
  • Sql user defined functions
  • Clr user defined functions
  • Stored procedures
  • Views
  • User defined types
  • User defined table types
  • User defined data types
  • Triggers
  • Tables
  • Full text indexes
  • Full text catalogs
  • Full text stop lists
This project called DBDocumentation consists of one screen called Main. This screen has two tabs. In fists tab you can set connection information such as server, credentials and database.
dbdoc_scriptingoptions.jpg In second tab you can choose what kind of objects do you want to document and you can also set what kind of information do you want to have in documentation. Before generation of documentation you must specify the output directory where html documents are going to be saved. This tool can retrieve the list of dependencies for every object and make cross references to them.
Here you can find working example.
documentaion.jpg

Future development

In future I will improve performance of collecting of information used for scripting of database objects. Next I will add the rest of objects to database documentation tool and i will adapt this scripting library for new version of SQL Server.

Migration of Relational Data structure to Cassandra (No SQL) Data structure

Introduction

With the uninterrupted growth of data volumes ever since the primitive ages of computing, storage of information, support and maintenance has been the biggest challenge. Also, the cloud computing technology has positioned a new dimension ("pay-as-you-go" model) to the information storage with efficient use of computer resources. Even the matured relational database products in the current market fall behind to scaling the applications according to the incoming traffic at a conventional cost. The demands of huge data and elastic scaling with desired performance has led to concept of No SQL databases. No SQL - undoubtedly is the hottest stint in today’s database technology and moving the data from the existing data structures to No SQL would be the potential area of interest for the customers.

What are No SQL Databases

No SQL databases are the data stores that are non-relational (without any fixed schemas & joins), distributed, horizontally scalable and often don’t adhere to the principles (ACID: atomicity, consistency, isolation, durability) of traditional relational databases.
NoSQL.jpg
Based on type of data storage, No SQL databases are broadly classified as:
  • Column family (Wide column) stores
  • Graph databases
  • Key value/tulip store
  • Document stores
There are more than 100+ No SQL databases in current market. The information on some of the most popular No SQL databases is depicted in the below list (For detailed information on No SQL databases, please refer to: http://nosql-databases.org):
Product
Details
Cassandra
Latest Version: V 0.8.6
License: Apache
Protocol: Custom, binary (Thrift)
API: Java/ Any Writer
HBase
Latest Version: V 0.90.3
License: Apache
Protocol: HTTP/REST
API: Thrift
MongoDB
Latest Version: V 2.0.0
License: Affero General Public License (AGPL)
Protocol: Custom, Binary
API: BSON
CouchDB
Latest Version: V 1.1.0
License: Apache
Protocol: HTTP/REST
API: JSON
Neo4J
Latest Version: V 1.5.M01
License: AGPL/commercial
Downloaded At:  http://neo4j.org/download/
Protocol: HTTP/REST
API: Java Embedded
Riak
Latest Version: V 1.0.0
License: Apache
Protocol: HTTP/REST or custom binary
API: JSON
Redis
Latest Version: V 2.2.14
License: Berkeley Software Distribution (BSD)
Downloaded At: http://redis.io/download
Protocol: Telnet-Like
API: Wide Spread Languages
Membase
Latest Version: V 1.7
License: Apache
Protocol: Memcached REST interface for cluster conf + management
API: JSON

What is JSON

JSON (stands for JavaScript Object Notation) is a lightweight and highly portable data-interchange format. JSON is intuitive to the web as well as the browser. Interoperability with any/all platforms in the current market can be easily achieved using JSON message format. 
According to JSON.org (www.json.org):
“JSON is built on two structures:

·         A collection of name/value pairs. In various languages, this is realized as an object, record, dictionary, structure, keyed list, hash table or associative array.
·         An ordered list of values. In most languages, this is realized as an array, list, vector, or sequence.  

These are universal data structures. Virtually all modern programming languages support them in one form or another. It makes sense that a data format that is interchangeable with programming languages also be based on these structures.”

A typical JSON syntax is as follows:


  • Data is represented in the form of name-value pairs.
  • A name value pair is comprised of a “Member Name” in double quotes, followed by colon “:” and the value in double quotes
  • Each data member (Name-Value pair) is separated by comma
  • Objects are held with-in curly (“{ }”) brackets.
  • Arrays are held with-in square (“[ ]”) brackets.
JSON Example:
{"Author": 
      {
    "First Name": "Phani Krishna",
    "Last Name":  "Kollapur Gandla",
    "Contact": {
        "URL": “in.linkedin.com/in/phanikrishnakollapurgandla",
        "Mail ID": "<a href="mailto:phanikrishna_gandla@xyz.com">phanikrishna_gandla@xyz.com</a>"}
       }
}
JSON and XML  
JSON is significantly like XML:
•  JSON is plain text data format
•  JSON is human readable and self-describing
•  JSON is categorized (contains values within values)
•  JSON can be parsed by scripting languages like Java script
•  JSON data is supported and transported using AJAX
JSON vs. XML
Though JSON & XML are both data formats, JSON has the upper hand over XML because of the following reasons:
•  JSON is lighter compared to XML (No unnecessary/additional tags in JSON)
•  JSON is easier to read and understand by humans.
•  JSON is easier to parse and generate for machines.
•  For AJAX related applications, JSON is quite faster compared to XML
Several No SQL products have provided built-in capabilities/readily available tools for loading data from JSON file format.  Below is the list of import/export utilities for some of the widely held No SQL products

Product
Import/Export Utilities for JSON
Cassandra
json2sstable -> JSON to Cassandra data structure
sstable2JSON -> Cassandra data structure to JSON
MongoDB
mongoimport -> JSON/CSV/TSV to MongoDB data structure
mongoexport -> MongoDB data structure to JSON/CSV
CouchDB
tools/load.py-> JSON to CouchDB data structure
tools/dump.py -> CouchDB data structure to JSON
Riak
bucket_importer:import_data-> JSON to Riak data structure
bucket_exporter:EXport_data -> Riak data structure to JSON






Cassandra Data Model

Understanding Cassandra data model
The Cassandra data model is premeditated for highly distributed and large scale data. It trades off the customary database guidelines (ACID compliant) for important benefits in operational manageability, performance and availability.
An illustration of how a Cassandra data model would like is as below:
Cassandra.jpg
The basic elements of the Cassandra data model are as follows:
•  Column
•  Super Column
•  Column Family
•  Keyspace
•  Cluster
Column: A column is the basic unit of Cassandra data model. A column comprises of name, value and a time stamp (by default). An example of column in JSON format is as follows:

{ // Example of Column
  "name": "EmployeeID",
  "value": "01234",
  "timestamp": 123456789
}

Super Column: A Super column is a dictionary of boundless number of columns, identified by the column name. An example of super column in JSON format is as follows:

{ // Example of Super Column
  "name": "designation",
  "value": {
"role" : {"name": "role", "value": "Architect", "timestamp": 123456789},
"band" : {"name": "band", "value": "6A", "timestamp": 123456789}
} 


The major differences between a column and a super column are:
•  Column’s value is a string but the super column’s value is a record of columns
•  A super column doesn’t include any time stamp (only terms name & value).
Note: Cassandra does not index sub columns, so when a super column is loaded into memory; all of its columns are loaded as well.
Column Family (CF):  A column family resembles an RDBMS table closely and is an assembly of ordered collection of rows which in-turn are ordered collection of columns.  A column family can be a “standard” or a “super” column family.

A row in a standard column family contains collections of name/value pairs whereas the row in a super column family(SCF) holds collections of super columns (group of sub columns).  An example for a column family is described below (in JSON):
Employee = { // Employee Column Family
   "01234" : {   // Row key  for Employee ID - 01234
        // Collection of name value pairs
        "EmpName" : "Jack",
        "mail" : "<a href="mailto:Jack@xyz.com">Jack@xyz.com</a>",
        "phone" : "9999900000"
        //There can be N number of columns
          }, 
   "01235" : {   // Row key  for Employee ID - 01235
        // Collection of name value pairs
        "EmpName" : "Jill",
        "mail" : "<a href="mailto:Jill@xyz.com">Jill@xyz.com</a>",
        "phone" : "9090909090"
        "VOIP" : "0404787829022",
        "OnsiteMail" : "<a href="mailto:jackandjill@abcdef.com">jackandjill@abcdef.com</a>"
    },
}


Note:  Each column would contain “Time Stamp” by default. For easier narration, time stamp is not included here.
The address of a value in a regular column family is a row key pointing to a column name pointing to a value, while the address of a value in a column family of type “super” is a row key pointing to a column name pointing to a sub column name pointing to a value. An example for Super column in JSON format is as follows:
ProjectsExecuted = { // Super column family
    "01234" :  {    // Row key  for Employee ID - 01235
  //Projects executed by the employee with ID - 01234
       "project1" : {"projcode" : "proj1", "start": "01012011", "end": "03082011", "location": "hyderabad"},
       "project2" : {"projcode" : "proj2", "start": "01042010", "end": "12122010", "location": "chennai"},
       "project3" : {"projcode" : "proj3", "start": "06062009", "end": "01012010", "location": "singapore"}
 
      //There can be N number of super columns

     }, 
   "01235" :  {    // Row key  for Employee ID - 01235
    //Projects executed by the employee with ID - 01235
       "projXYZ" : {"projcode" : "Cod1", "start": "01012011", "end": "03082011", "location": "bangalore"},
       "proj123" : {"projcode" : "Cod2", "start": "01042010", "end": "12122010", "location": "mumbai"},
     }, 
 }

Columns are always organized as per the Column‘s name within their rows. The data would be sorted as soon as it is inserted into the data model.


Keyspace:  A keyspace is the outmost grouping for data in Cassandra, closely resembling an RDBMS database. Similar to the relational database, a keyspace has title and properties that describe the keyspace demeanor. The keyspace is a container for a list of one or more column families (without any enforced association between them).


Cluster: Cluster is the outermost structure in Cassandra (also called as ring). Cassandra database is specially designed to be spread across several machines functioning together that act as a single occurrence to the end user. Cassandra allocates data to nodes in the cluster by arranging them in a ring.

Relational data model vs. Cassandra data model
Relational Data Model
Cassandra data model (Standard)
Cassandra data model (Super)
         Server
Cluster
         Database
Key space
         Table
Column Family
         Primary Key
Key
Column Value
Column Name
Super Column Name
Column Value
Column Name
Column Value
Unlike the traditional RDBMS, Cassandra doesn’t support
  • Query language like SQL (T-SQL, PL/SQL etc.). Cassandra provides an API called thrift through which the data could be accessed.
  • Referential Integrity (operations like cascading deletes are not available)
Designing Cassandra data structures
1. Entities – Point of Interest
The finest way to model a Cassandra data structure is to identify the entities on which most queries would be attentive and creating the entire structure around the entity. The activities performed (generally the use cases) by the user applications, how the data is retrieved and displayed would be the areas of interest for designing the Cassandra column families.
For example, a simple employee data model (in any RDMBS) would contain:
·         Employee
·         Employee contact details
·         Employee financial information
·         Employee role information
·         Employee attendance information
·         Employee projects
….
And so on…
Here “Employee” is the entity for point of interest and any application using this design would frame the queries relating to the employee.
EmpDataModel.jpg
2. De-normalization
Normalization is the set of rules established to aid in the design of tables and their relation-ships in any RDBMS. The benefits of normalizing would be:
•   Avoiding repetitive entries
•   Reduction of storage space required
•   Prevention of schema restructuring for future needs.
•   Improved speed and flexibility of SQL queries, joins, sorts, and search results.
Achieving the similar kind of performance for the growing data volume is a challenge in traditional relational data models and the companies could compromise on de-normalization to achieve performance. Cassandra does not support foreign key relationships like a relational database and the better way is to de-normalize the data model. The important fact is that instead of modeling the data first and framing the queries, with Cassandra the queries would be modeled and the data be framed around them.
3. Planning for Concurrent Writes
In Cassandra, every row within a column family is identified by the unique row key (generally a string of unlimited length). Unlike the traditional RDBMS primary key (which enforces uniqueness), Cassandra doesn’t impose uniqueness (Duplicate row key insertion might disturb the existing column structure). So the care must be taken to create the rows with unique row keys. Some of the ways for creating unique row keys is as follows:
•   Surrogate/ UUID type of row keys
•   Natural row keys

Data Migration approach (Using ETL)

MigrationApp.png
There are various ways of porting the data from relational data structures to Cassandra structures, but the migrations involving complex transformations and business validations might accommodate a data processing layer comprising ETL utilities.
In case of using in-built data loaders, the processed data can be extracted to flat files (in JSON format) and then uploaded to the Cassandra data structure’s using these loaders. Custom loaders could be fabricated in case of additional dispensation rules, which could either deal the data from the processed store or the JSON files.
The overall migration approach would be as follows:
  1. Data preparation as per the JSON file format.
  2. Data extractions into flat files as per the JSON file format or extraction of data from the processed data store using custom data loaders.
  3. Data loading using in-built or custom loaders into Cassandra data structure (s).

The various activities for all the different stages in migration are further discussed in detail in below sections.
Data Preparation and Extraction
  • ETL is the standard process for data extraction, transformation and loading
  • At the end of the ETL process, reconciliation forms an important part. This comprises validation of data with the business processes.
  • The ETL process also involves the validation and enrichment of the data before loading into staging tables.
DataPreparation.png
Data Preparation Activities:
The following activities will be executed during data preparation: 
  1.  Creation of database objects
    • Necessary staging tables are to be created as per the requirements based on which will resemble standard open interface / base table structure.
  2. Validate & Transform data before Load from the given source (Dumps/Flat files).
    •  Data Cleansing
      • Filter incorrect data as per the JSON file layout specifications.
      • Filter redundant data as per the JSON file layout specifications.
      • Eliminate obsolete data as per the JSON file layout specifications.
  3. Load data into staging area
  4. Data Enrichment
    • Default incomplete data
    • Derive missing data based on mapping or lookups
    • Differently structured data (1 record in as-is = multiple records in to-be)
Data Extraction Activities (into JSON files):
The following activities will be executed during data extraction into JSON file formats:
  1. Data Selection as per the JSON file layout
  2. Creation of SQL programs based on as the JSON file layout
    • Scripts or PLSQL programs are created based on the data mapping requirements and the ETL processes. These programs shall serve various purposes including the loading of data into staging tables and standard open interface tables.
  3. Data Transformation before extract as per the JSON files layout specification and mapping documents.
  4. Flat files in form of JSON format for data loading
Data Loading
Cassandra data structures can be accessed using different programing languages like (.net, Java, Python, Ruby etc.). Data can be directly loaded from the relational databases (like Access, SQL Server, Oracle, MySQL, IBM DB2, etc.) using these programing languages. Custom loaders could be used to load data into Cassandra data structure(s) based on the enactment rules, customization level and the kind of data processing.

References

  1. No SQL Databases (http://nosql-database.org/ )
  2. What is JSON (www.json.org )
  3. Getting started with Cassandra (http://wiki.apache.org/cassandra/GettingStarted )