Writing an OPOS Service Object using ATL

What is OPOS?
An architecture that allows Point-of-Sale (POS) devices to be integrated to Microsoft Windows family of operating systems. The devices are divided into categories called device classes such as PIN Pads, Cash Drawers etc.

OPOS model
OPOS Control is software abstraction layer for a device class and it consists of a Control Object and a Service Object. A device is characterized by its Properties, Methods and Events. A POS application accesses a device through the OPOS Control APIs that expose these properties, methods and events. The layers are shown in the figure below taken from the UnifiedPOS specification document[1].

Control Object (CO) 

• exposes a set of properties, methods and events
• is an ActiveX
• can be downloaded from the web

  
Service Object (SO)

• called by the CO
• implements OPOS functionality for a specific device class
• usually provided by the device manufacturer or third parties

  
How to use an OPOS Control?
The application usually calls the following Control Object methods:

• add event handlers to get event data
• call Open to open the device
• call ClaimDevice to gain exclusive access to the device
• set DeviceEnabled property to TRUE
• call any other methods to perform required operations
• call ReleaseDevice to release the device from exclusive access mode
• call Close to release the device and associated resources when the application is finished using the device

Relationship between CO and SO methods
The table below shows which Service  Object method is called for the given Control Object method. For example, when the application calls CO's Open method, the CO calls the SO's OpenService method.

Category    Control Object                  Service Object

method      Open                            OpenService

            Close                           CloseService

            other method                    corresponding other method

            

property    get String property             GetPropertyString

            set String property             SetPropertyString

            get Numeric property            GetPropertyNumber

            set Numeric property            SetPropertyNumber

            get/set other property type     corresponding GetProperty/SetProperty

event                                       SO calls a CO event request method:

                                            SOData, SODirectIO, SOError, 

                                            SOOutputComplete and SOStatusUpdate

Control Object Open method
In order to write a Service Object, it's important to know how the Control Object's Open method works. If the Service Object does not satisfy certain requirements, the CO Open method will fail and the application won't be able to use the OPOS control. CO Open method as shown below takes a DeviceName as its argument.

long Open(BSTR DeviceName);

The CO Open method will

• query the registry to find the device class and device name
• load the service object for the device
• check if at least the methods defined in the initial OPOS version for the device class are supported by the SO
• call SO's OpenService method
• call SO's GetPropertyNumber(PIDX_ServiceObjectVersion) to get the service object version, then check if the major version number matches that of the CO's
• check if all methods that must be supported by the particular SO version are available

Service Object version
Three version levels are specified for SO version.
Major: The millions place
Minor: The thousands place
Build: The units place provided by the SO developer specifying a build number

For example, the version number 1002038 is interpreted as 1.2.38.
An SO will work with a CO as long as their major version numbers match.

Service Object for a simple virtual MSR
Now that we know the basics, we'll create a minimal Service Object for a virtual MSR (Magnetic Stripe Reader) supporting following functionality:
open, claim, enable, enable data events, get a data event notification, release and close.

The figure below shows a list of MSR CO methods and the OPOS versions they were first introduced.

Not all MSR CO methods are shown in the above list. For our minimal virtual MSR SO, we are only interested in version 1.0 (i.e. initial version) equivalents.
 
First we should download the latest OPOS Control Objects and header files from here. Download the CCO Runtime zip file, then register the MSR CO. Then follow the steps given below to create the SO.

Steps
1. Launch Visual Studio and create an ATL Project. Name it MyVirtualMsr. Select DLL as the Application Type and check Allow merging proxy/stub code in the ATL Project Wizard.

Create MyVirtualMsr ATL project

Selecting Application type and merging proxy/stub code

2. In the class view, right click the project, then select Add Class to add a Simple ATL Object. Enter MyVirtualMsrSO under Short name and set the ProgID value to VIRTUAL.OPOS.MSR.so. Set the Options as shown and click finish. This will create a CMyVirtualMsrSO class and a IMyVirtualMsrSO interface.

Setting ProgID

Options

3. Add the following methods to the IMyVirtualMsrSO interface (Right click the IMyVirtualMsrSO interface in class view and select add method from the menu. Make sure you set the parameter attributes out and retval for pRC).

HRESULT OpenService(BSTR DeviceClass, BSTR DeviceName, IDispatch* pDispatch, [out, retval] long* pRC);

HRESULT CheckHealth(long Level, [out, retval] long* pRC);

HRESULT ClaimDevice(long ClaimTimeout, [out, retval] long* pRC);

HRESULT ClearInput([out, retval] long* pRC);

HRESULT CloseService([out, retval] long* pRC);

HRESULT COFreezeEvents(VARIANT_BOOL Freeze, [out, retval] long* pRC);

HRESULT DirectIO(long Command, [in, out] long* pData, [in, out] BSTR* pString, [out, retval] long* pRC);

HRESULT ReleaseDevice([out, retval] long* pRC);

HRESULT GetPropertyNumber(long PropIndex, [out, retval] long* pNumber);

HRESULT GetPropertyString(long PropIndex, [out, retval] BSTR* pString);

HRESULT SetPropertyNumber(long PropIndex, long Number);

HRESULT SetPropertyString(long PropIndex, BSTR PropString);

4. Add the following variable to CMyVirtualMsrSO class. We are using this instance for event handling.

CComDispatchDriver m_pDriver;

5. Include OposMsr.hi from the Include directory of the CCO Runtime that you downloaded earlier. This includes the definitions of various OPOS constants such as ResultCodes and property IDs.

#include "OposMsr.hi"

6. Modify the OpenService methos as shown below:

STDMETHODIMP CMyVirtualMsrSO::OpenService(BSTR DeviceClass, BSTR DeviceName, IDispatch* pDispatch, LONG* pRC)

{

    m_pDriver = pDispatch;

    *pRC = OPOS_SUCCESS;

    return S_OK;

}

7. Set return code value pRC of the methods to OPOS_SUCCESS

*pRC = OPOS_SUCCESS;

8. Modify the GetPropertyNumber method as shown below to return a valid version number

STDMETHODIMP CMyVirtualMsrSO::GetPropertyNumber(LONG PropIndex, LONG* pNumber)

{

    if (PIDX_ServiceObjectVersion == PropIndex)

    {

        *pNumber = 1000111;

    }

    return S_OK;

}

9. Create a registry key under OLEforRetail for our device. We'll call our virtual MSR 'MyVirtualDevice' and this is the name that we'll be passing to CO Open method. The default value of the device name key MyVirtualDevice must be the value we entered for the ProgID while we were creating the ATL object, which in our case is the value VIRTUAL.OPOS.MSR.so.

If you are on a 64bit machine:

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\OLEforRetail\ServiceOPOS\MSR\MyVirtualDevice]

@="VIRTUAL.OPOS.MSR.so"

If you are on a 32bit machine:

[HKEY_LOCAL_MACHINE\SOFTWARE\OLEforRetail\ServiceOPOS\MSR\MyVirtualDevice]

@="VIRTUAL.OPOS.MSR.so"

10. Now build the project. By default, it is set to register the dll after build. If you didn't launch Visual Studio with admin privileges, it'll give you an error. You can either launch Visual Studio with admin privileges, or disable registering by setting Register Output option in the Linker settings to No, then manually register the dll. To manually register,

if you are on a 64bit machine:

c:\windows\syswow64\regsvr32 MyVirtualMsr.dll

if you are on a 32bit machine:

c:\windows\system32\regsvr32 MyVirtualMsr.dll

Now we have satisfied all conditions for a minimal valid MSR Service Object by adding registry entries and methods, then returning a valid version number. This means Control Object Open(DeviceName) method will return success. Now, let's add some code to fire a simple data event to the Control Object.

11. Add a FireDataEvent method to CMyVirtualMsrSO.

void CMyVirtualMsrSO::FireDataEvent(void)

{

    VARIANT v; v.vt = VT_I4; v.lVal = 10;

    m_pDriver.Invoke1(L"SOData", &v);

}

12. We'll let a data event fire 500ms after the application enables Data Events. 
Please note that the code below is just for demo purpose of data events. If you call CO Close method before this 500ms, it'll destroy the SO, and this in turn will crash the application because the pVirtualMsr is no longer a valid object.

DWORD WINAPI MyThreadFunction( LPVOID lpParam ) 

{

    CMyVirtualMsrSO* pVirtualMsr = (CMyVirtualMsrSO*)lpParam;

    Sleep(500);

    if (NULL != pVirtualMsr)

    {

        pVirtualMsr->FireDataEvent();

    }

    return 0;

}

STDMETHODIMP CMyVirtualMsrSO::SetPropertyNumber(LONG PropIndex, LONG Number)

{

    switch (PropIndex)

    {

    case PIDX_DeviceEnabled:

        break;

    case PIDX_DataEventEnabled:

        CreateThread( 

                NULL,                   // default security attributes

                0,                      // use default stack size  

                MyThreadFunction,       // thread function name

                this,                   // argument to thread function 

                0,                      // use default creation flags 

                NULL);

        break;

    default:

        break;

    }

    return S_OK;

}

Rebulid the project and register the dll. Now we can test our SO easily with an MFC or a VB test application.

References
[1] UnifiedPOS 1.12 documentation (link)

Decorrelation Stretching

Decorrelation stretch is used to enhance color differences in images with high interchannel correlation. Therefore it allows us to see details that are otherwise not so obvious or invisible to the human eye.

I first came across decorrelation stretch in a Matlab webinar and was curious to find out how it works. From Matlab online help [1] and some other online resources [2, 3] I learned the theory and implemented it in C++ using OpenCV. You can find a very nice application of decorrelation stretching to uncover rock art here.

The theory is well explained in [2, 3], so I won't go into details but provide a brief explanation.

To remove interchannel correlation in input data, we project it to a space that removes this correlation. We can find such space by performing PCA on input data. We'll then scale the projected data in the new eigenspace and project it back to the original space and stretch it. The process is illustrated in the figure below for a 2-D dataset.

The operation can be expressed as a pointwise linear transformation:

y = S_t R^T S_c R (x - μ) + target_μ

x : n x 1 input data point.
y : n x 1 output data point.
μ : n x 1 mean of the input data.
R : n x n rotation matrix formed by the eigenvectors found from PCA. R (x - μ) projects zero mean input data to the new eigenspace.
S_c : n x n scaling matrix which is a diagonal matrix. Elements in the main diagonal are of the form 1/σ_i, where σ is the standard deviation of the corresponding variable in the new eigenspace. Multiplication by S_c makes variables in eigenspace have unit variance.
R^T : R^T projects data back to original space (R^T = R^-1, since rotation matrix satisfies R^T R = I). Reprojected data will still have unit variance.
S_t : n x n stretching matrix which is a diagonal matrix. Elements in the main diagonal are of the form target_σ_i, where target_σ is the desired standard deviation for the corresponding variable in the input data space. Multiplication by S_t sets the variance of variables in input data space to desired variance.
target_μ : backprojected data is still zero mean. Adding target_μ shifts mean of the variables to the desired mean.

The dstretch function below takes as input a multi-channel image and two optional column vectors specifying the desired mean and standard deviation for each channel of the output decorrelation stretched image.
 

#include <opencv2/opencv.hpp>

/*
input       : p x q x n multi-channel image.
targetMean  : n x 1 vector containing desired mean for each channel of the dstretched image. If empty, mean of the input data is used.
targetSigma : n x 1 vector containing desired sigma for each channel of the dstretched image. If empty, sigma of the input data is used.

returns floating point dstretched image
*/

Mat dstretch(Mat& input, Mat& targetMean = Mat(), Mat& targetSigma = Mat())
{
   CV_Assert(input.channels() > 1);
 
   Mat dataMu, dataSigma, eigDataSigma, scale, stretch;

   Mat data = input.reshape(1, input.rows*input.cols);
   /*
   data stored as rows. 
   if x(i) = [xi1 xi2 .. xik]' is vector representing an input point, 
   data is now an N x k matrix:
   data = [x(1)' ; x(2)' ; .. ; x(N)']
   */

   // take the mean and standard deviation of input data
   meanStdDev(input, dataMu, dataSigma);

   /*
   perform PCA that gives us an eigenspace.
   eigenvectors matrix (R) lets us project input data into the new eigenspace.
   square root of eigenvalues gives us the standard deviation of projected data.
   */
   PCA pca(data, Mat(), CV_PCA_DATA_AS_ROW);

   /*
   prepare scaling (Sc) and strecthing (St) matrices.
   we use the relation var(a.X) = a^2.var(X) for a random variable X and 
   set
   scaling factor a = 1/(sigma of X) for diagonal entries of scaling matrix.
   stretching factor a = desired_sigma for diagonal entries of stretching matrix.
   */

   // scaling matrix (Sc)
   sqrt(pca.eigenvalues, eigDataSigma);
   scale = Mat::diag(1/eigDataSigma);

   // stretching matrix (St)
   // if targetSigma is empty, set sigma of transformed data equal to that of original data
   if (targetSigma.empty())
   {
      stretch = Mat::diag(dataSigma);
   }
   else
   {
      CV_Assert((1 == targetSigma.cols) &&  (1 == targetSigma.channels()) && 
         (input.channels() == targetSigma.rows));

      stretch = Mat::diag(targetSigma);
   }
   // convert to 32F
   stretch.convertTo(stretch, CV_32F);
 
   // subtract the mean from input data
   Mat zmudata;
   Mat repMu = repeat(dataMu.t(), data.rows, 1);
   subtract(data, repMu, zmudata, Mat(), CV_32F);

   // if targetMean is empty, set mean of transformed data equal to that of original data
   if (!targetMean.empty())
   {
      CV_Assert((1 == targetMean.cols) && (1 == targetMean.channels()) && 
         (input.channels() == targetMean.rows));

      repMu = repeat(targetMean.t(), data.rows, 1);
   }

   /*
   project zero mean data to the eigenspace, normalize the variance and reproject,
   then stretch it so that is has the desired sigma: StR'ScR(x(i) - mu), R'R = I.
   since the x(i)s are organized as rows in data, take the transpose of the above
   expression: (x(i)' - mu')R'Sc'(R')'St' = (x(i)' - mu')R'ScRSt,
   then add the desired mean:
   (x(i)' - mu')R'ScRSt + mu_desired
   */
   Mat transformed = zmudata*(pca.eigenvectors.t()*scale*pca.eigenvectors*stretch);
   add(transformed, repMu, transformed, Mat(), CV_32F);

   // reshape transformed data
   Mat dstr32f = transformed.reshape(input.channels(), input.rows);

   return dstr32f;
}  

The demo program below shows how dstretch was used to highlight different regions of a Landsat image.

int _tmain(int argc, _TCHAR* argv[])
{
   // rgb image
   Mat bgr = imread("14_L.jpg");
   // target mean and sigma
   Mat mean = Mat::ones(3, 1, CV_32F) * 120;
   Mat sigma = Mat::ones(3, 1, CV_32F) * 50;
   // convert to CIE L*a*b* color space
   Mat lab;
   cvtColor(bgr, lab, CV_BGR2Lab);
   // dstretch Lab data. dstretch outputs a floating point matrix
   Mat dstrlab32f = dstretch(lab, mean, sigma);
   // convert to uchar
   Mat dstrlab8u;
   dstrlab32f.convertTo(dstrlab8u, CV_8UC3);
   // convert the stretched Lab image to rgb color space
   Mat dstrlab2bgr;
   cvtColor(dstrlab8u, dstrlab2bgr, CV_Lab2BGR);
   // dstretch RGB data
   Mat dstrbgr32f = dstretch(bgr, mean, sigma);
   // convert to uchar
   Mat dstrbgr8u;
   dstrbgr32f.convertTo(dstrbgr8u, CV_8UC3);

   imshow("RGB", bgr);
   imshow("dstretched CIE L*a*b* converted to RGB", dstrlab2bgr);
   imshow("dstretched RGB", dstrbgr8u);
   waitKey();

   return 0;
} 

Input image: Philadelphia, Pennsylvania, USA image courtesy of the U.S. Geological Survey

dstretched CIE L*a*b* image converted to RGB

dstretched RGB image

Data distribution of the images

(Data distribution plotted using Matplotlib mplot3D toolkit.)

References
[1]Matlab decorrstretch documentation http://in.mathworks.com/help/images/ref/decorrstretch.html
[2]http://www.dstretch.com/DecorrelationStretch.pdf
[3]A. R. Gillespie, A. B. Kahle, and R. E. Walker, “Color enhancement of highly correlated images. I. Decorrelation and HSI contrast stretches,” Remote Sensing of Environment, vol. 20, no. 3, pp. 209–235, Dec. 1986.
 

Object localization using color histograms

This post shows how to use histogram backprojection to find the location of a known object within an image. Histogram backprojection for object localization was first proposed in the paper 'Color Indexing' by Swain and Ballard [1]. Following the terminology in the paper, the known object is referred to as the 'model' and the image within which we are searching the model is referred to as the 'image' below. I'm using OpenCV and C++ to demonstrate the techniques presented in the paper. 

Histogram backprojection tells us where in an image the colors of a given model histogram occur. Here for localization, we are backprojecting what is called the ratio histogram onto the image. Ratio histogram is defined as

R[i] = max(M[i]/I[i], 1) for all bins
where M, I and R are Model, Image and Ratio histograms respectively.

This backprojected image is then convolved with a circular mask having the same area as our model. The peak of the convolution should hopefully give us the location of the model within the image if the model appears in the image. 

Following program shows how to apply the above technique to locate a model within an image. It uses RGB color space with 32 bins for each channel. The model here was extracted from a different image than the test image and there were no significant lighting changes.

#include <opencv2/opencv.hpp>

using namespace cv;

int main(int argc, char* argv[])   
{ 
   Mat model = imread("model.jpg");
   Mat image = imread("image.jpg");

   int bins = 32;
   double d, max;
   Point maxPt;

   MatND histModel, histImage, histRatio;
   Mat model32fc3, image32fc3, backprj, kernel, conv;

   Mat temp, color;

   const int channels[] = {0, 1, 2};
   const int histSize[] = {bins, bins, bins};
   const float rgbRange[] = {0, 256};
   const float* ranges[] = {rgbRange, rgbRange, rgbRange};

   // model histogram
   model.convertTo(model32fc3, CV_32FC3);
   calcHist(&model32fc3, 1, channels, Mat(), histModel, 3, histSize, ranges, true, false);
   // image histogram
   image.convertTo(image32fc3, CV_32FC3);
   calcHist(&image32fc3, 1, channels, Mat(), histImage, 3, histSize, ranges, true, false);
   // ratio histogram
   divide(histModel, histImage, histRatio, 1.0, CV_32F);
   cv::min(histRatio, 1.0, histRatio);
   // backproject ratio histogram onto the image
   calcBackProject(&image32fc3, 1, channels, histRatio, backprj, ranges);
   // obtain a circular kernel having the same area as the model
   d = sqrt(4*model.rows*model.cols/CV_PI);
   kernel = getStructuringElement(MORPH_ELLIPSE, Size((int)d, (int)d));
   // convolve the kernel with the backprojected image
   filter2D(backprj, conv, CV_32F, kernel);
   // find the peak
   minMaxLoc(conv, NULL, &max, NULL, &maxPt);

   // display with color map
   // ratio histogram backprojected image
   backprj.convertTo(temp, CV_8U, 255);
   applyColorMap(temp, color, COLORMAP_JET);
   imshow("ratio histogram backprojection onto the image", color);
   // convolution result
   conv.convertTo(temp, CV_8U, 255/max);
   applyColorMap(temp, color, COLORMAP_JET);
   imshow("convolution with circular mask", color);
   // location of the peak
   image.copyTo(color);
   circle(color, Point(maxPt.x, maxPt.y), (int)(d/2), Scalar(0, 0, 255), 3);
   imshow("location of the peak", color);
   waitKey();

   return 0;   
}  

It's worth mentioning few points here:

• The significance of backprojecting the ratio histogram rather than the normalized model histogram is that, if a certain model color is distributed over large areas in the image (i.e. M[i] << I[i] for some bin i), those areas in the backprojected image will get a low value, hence less likely to distract the localization mechanism.
• Convolution kernel needn't be a circular mask. It is used here to handle the general case where the orientation of the model within the image is unknown.
• The case where I[i] = 0 for any M[i]: Since this color is absent in the image, the corresponding R[i] won't be backprojected onto the image.
• Color histograms are very sensitive to illumination. If the lighting conditions vary between models and images, this technique will not perform well. Models and Images should be preprocessed using a color constancy method in this case.
• This technique is robust to viewpoint changes and occlusion.

Below are the inputs and results of the program:

Model:

Location of the model within the image:

Ratio histogram backprojection onto the image:

Result of convolution with circular mask:

References
[1]M. J. Swain and D. H. Ballard, “Color indexing,” Int J Comput Vision, vol. 7, no. 1, pp. 11–32, Nov. 1991.