3D Point Cloud Processing with PCL

"What's better than 2D?"

Three dimensional (3D) point cloud can be understood as a set of points in the Euclidean coordinate system, with every point has X-, Y-, and Z-coordinates. Combined with the corresponding normal vectors or normals, this set forms oriented points - providing 6 degrees of freedom (DOF). This article aims to briefly introduce readers to the 3D point cloud processing with an open source library called Point Cloud Library (PCL). Two examples of using PCL are given and visualized.

Follow @HafizVonPJ

A 3D point cloud can be acquired using range sensors such as Microsoft Kinect, ASUS Xtion, or laser scanners. These devices apply either triangulation or time-of-flight (TOF). Apart from that, using the autofocus system on conventional industrial cameras can also deliver 3D point clouds.

To process and visualize 3D point clouds, PCL offers various modules, e.g., filtering, segmentation, I/O, data structures, and visualization. The author has been tasked to implement several of these features in the open source application for image processing called MILAN. The implementations are available within one particular plug-in of MILAN. The following images are examples of using MILAN with PCL.

Figure 1 - The features of PCL in MILAN are accessible via a toolbox docked to the right side as seen above. This is also where the Kinect sensor can be used and controlled.

Figure 2: PCL Downsampling Filter, to reduce the size of an input cloud (left - 63792 vertices) to an output (right - 20462 vertices).

Figure 3: PCL Euclidean Cluster Segmentation, to segmentize spatially isolated point clouds.

Before using MILAN, the author has used the PCL tutorials and implemented them in separate Microsoft Visual Studio (MSVS) 2008 projects. The Service Packet 1 (SP 1) of this IDE needs to be installed, otherwise the developers can only code in Release mode.

Another alternative to MSVS is the Qt Creator. The author has covered this in one post on the app FingKinector 2.0. It was developed with Qt 4 along OpenCV 2.4.5. Without writing any .PRO file for the Qt project, the author used a CMakeLists.txt entirely to build this Qt project.

Hands-on Learning OpenCV with Qt Creator in Ubuntu

"One App to learn them all,

One App to find them,

One App to apply them all and in the Creator bind them."

In this entry the writer presents his results from putting some lessons from the book OpenCV 2 Computer Vision Application Programming Cookbook by Robert Laganière into practice by incorporating the implementations into just one application. The IDE chosen for this is Qt Creator on the OS Ubuntu Gnome (a Linux variant). Initially the application is intended to comply with the principle of K.I.S.S. - Keep It Simple and Stupid. The designated application is supposed to process an input image according to the lessons by respective chapter of the book. The application should consist only of buttons and space for output images. After a certain point, the writer halted and decided to revisit some fundamentals in software architecture.

The cookbook mentioned above is de facto the standard book to learn OpenCV 2 for beginners. Consisting of 10 chapters, the book begins by introducing the steps to install OpenCV, then moves to the creating of OpenCV projects in MS Visual Studio and Qt. Various algorithms for image processing are elaborated in the next chapters - filtering, morphological operations, etc.

Follow @HafizVonPJ

Previously the writer had successfully installed OpenCV in Windows XP and Windows 7. Taking the next level, the writer challenges himself to try OpenCV in a Linux system, which in this case refers to Ubuntu. By using Qt in Ubuntu, the usage of a CMakeLists is replaced with the editing the .pro file for the Qt project. Designing and implementing the GUI file for the application happen within the same platform - Qt Creator.

The result is an application called FingsCV, seen here in the Fig. 0 below.

Fig. 0 - Overview of the application FingsCV (No copyright infringement is intended for the Transformers' Decepticon logo. For academic and non-commercial purpose only).

The Simple and Stupid Design

Fig. 1 below shows the design of the GUI in the designer section of Qt Creator. The GUI consists only of one button to trigger a dialog window to load an input image. The main part of the GUI is only a tab widget that has several tabs that signify the respective chapters.

Fig. 1 - The design of the GUI in Qt Creator.

Results

Fig. 2 below displays the input image used for the operations in the tab Ch. 1 and Ch. 2.

Fig. 2 - Test image (Photo courtesy of a popular cosplayer's Instagram page - no copyright infringement is intended. For academic purpose only).

Chapter 1 of the cookbook is basically about one of the simplest operations in image processing and editing - reflecting or flipping of an image. The result from flipping the test image is shown in Fig. 3 below.

Fig. 3 - The output of flipping the test image from Fig. 2.

Then in Chapter 2, more operations are introduced  - putting salt and pepper (white and black pixels) effect, color reducing, and image sharpening. Fig. 4 below shows the after-effect of random and synthetic white pixels (the 'salt') on the test image.

Fig. 4 - Test image with added random salt - marked by the tiny white pixels.

Discussion

In the beginning, the writer used OpenCV's HighGUI for visualization of images. This had a conflict with Qt, while the latter has its own built-in implementations for image displays. Albeit no error and just warnings, the writer switched to just use Qt's features such as QImage, QPixmap, etc for the visualizations.

The writer decided to stop this development until Chapter 2, because the Chapter 3 requires more interactions via the GUI - input parameters, signal buttons, etc. This ought to be done in a separate application. To implement the next lessons, the application should be more interactive and pragmatic. 

Future Prospects

The predicaments from Chapter 3 led the writer to refer a book from his college's library - Software Architecture - Foundations, Theory, and Practice by Richard N. Taylor.

From this book by R. Taylor, it looks like the process of software development shouldn't be taken lightly. In simple words, the development of a software can be summed up in these steps:

• Listing of the requirements
• Designing
• Implementing - the coding
• Analysis and testing
• Evolution and maintenance

Thus, the K.I.S.S. should be modified to 'Keep It Stupid and Simple - enough'. The last word 'enough' implies that a certain degree of requirements must be firstly met.

At the moment, the writer considers that every version of this app should be dedicated to only one chapter of the cookbook. Therefore separate Qt projects for all the chapters of the cookbook.

FingKinector 2.0 - on Qt Creator with PCL and OpenCV

How can one build a Qt project that implements both OpenCV and Point Cloud Library (PCL)? The writer aims to have the best of both worlds available on the same platform. This article briefs on the writer's attempt to achieve that objective with one CMakeLists file and the development within the Qt Creator. The project in this article is FingKinector 2.0, the writer's updated version of a simple application for Microsoft Kinect sensor.

Follow @HafizVonPJ

Mainly the writer deals with 3-dimensional point clouds, which is why the PCL is used for most of the writer's C++ code. But when it comes to situations that require linear algebra calculations especially matrix operations (e.g. transformation), the writer finds OpenCV library to be convenient. The OpenCV library enables one to deal with n-dimensional images, which one usually treats them as matrices.

For personal/independent projects, the writer aims to develop one simple application on the platform of Qt Creator and uses both PCL and OpenCV. Previously the writer has developed the 'stupid and simple' app: FingKinector. This first version only had access to the joint motor functions of the Kinect sensor. Furthermore it was developed on Microsoft Visual Studio. For the sake of variety, the writer thought this time the upgrading of this app should be done on Qt Creator.

CMake Build Process

The following images show the step by step in the build process of the software for FingKinector 2.0, within the Qt Creator, instead of the CMake GUI.

Step 1: Open file CMakeLists.txt (Fig. 1)

Figure 1: Open File - CMakeLists.txt

Step 2: Specify the directory for the build (Fig. 2)

Figure 2: Specify the directory for the build process

Step 3: Run CMake (Figs. 3.1 and 3.2)

Figure 3.1: Run CMake

Figure 3.2: Finish the build process

Step 4: Done - start coding! (Fig. 4)

Figure 4: Start coding on Qt Creator

Source Codes

The writer finds it prudent to divide the codes for Kinect motor functions and its image processing in different 2 separate files. The figure (Fig. 5) shows the source files for this application:

Figure 5: Source codes and other files for the project

1) main: contains the function int main()

2) FingKinectorWindow: manages the actions/signals on the GUI

3) KinectFunkts: manages Kinect initialization and frame grabber

4) PCLOpenCV: manages the processing of 3D-point clouds and its visualization.

The writer has decided not to make these source codes available for others. However the writer welcomes suggestions and questions from all readers.

New Features

In the first version of FingKinector, one could only toy with the joint motor feature of the Kinect sensor. For this new version, FingKinector offers two options for point cloud acquisition:

i) the fundamental single mode / one shot, and

ii) convoluted triple shots - at 3 different angles: 27°, 0° and -27°.

The following figures 6.1 until 6.3 show the feature of every tab on the app FingKinector.

Fig. 6.1: FingKinector - Initialization

Figure 6.2: FingKinector - Motor functions

Figure 6.3: FingKinector - Visualization

The latter mode is actually redundant, but the writer was curious about the outcome of such acquisition.

Future prospects

The writer would add two more features for this app in the future - live view of 3D-point cloud and display of range image.

Demonstrations

FingKinector - Interacting with the Kinect Motor

Follow @HafizVonPJ

The pivot which connects the Kinect sensor to its base is motorized. This post is about an app called 'FingKinector', which is developed to utilize that motorized pivot without having to rotate the Kinect sensor manually. This is tested only on the Kinect sensor for XBOX 360 and developed for Windows 7.

"K.I.S.S. = Keep It Stupid, Simple"

Image 1: Kinect Sensor for XBOX 360, equipped with a wide-angle lens.

To be able to control the Kinect sensor from a PC, you will need an AC-adapter for your Kinect. It costs around € 8.00 on the Amazon. It has 2 outputs: the orange plug connector is used to supply the Kinect with 12 V at 1 A, whereas the second output is a regular USB connector [1].

If your Kinect is connected to your PC for the first time, by default Windows will attempt to automatically download and install the driver for that Kinect, which comes from PrimeSense. PrimeSense is the developer of the range camera in the Kinect sensor and it is an Israeli 3D sensing technology. As of this time - PrimeSense has become a subsidiary of Apple Inc [2].

At the bottom of this post, the writer puts a link for you to download FingKinector.

FingKinector works in 2 options:

1. Single run
2. Sequential run

The following image shows the first mode (single run). It is known from [3] that the motor is allowed to be tilted to an angle between -27° and 27°.

Image 2: Single Run mode on FingKinector

The second mode, as seen in Image 3 below - the sequential run, uses a factor to divide the range [-27°,27°] up to 10 sectors. By default this mode will first turn the Kinect to the position of 0°, then starts the sequential movement from either top (at 27°) or bottom (at -27°).

Image 3: Sequential mode on FingKinector

The writer also took a liberty to record an amateur video to demonstrate the sequential mode. The Kinect here is seen with an extra hardware called Kinect Zoom Wide-Angle Lens by Nyko [4]. Its cost is around € 14.00.

Sources:

[1]: Kinect Adapter in Amazon.de

[2]: PrimeSense (Wikipedia)

[3]: Kinect Sensor - Microsoft Developer Network

[4]: Kinect Zoom Wide-Angle Lense by Nyko (Amazon.de)

REMINDER / WARNING:

1. For Windows only. (Linux version will be made available in the next development)
2. The writer is NOT liable for any damage.
3. Use at your own risk.

Have fun!

Download link:
Download link for FingKinector (Windows)

A Stupid and Simple App: FingCryptor

Follow @HafizVonPJ

Julius Caesar used to apply a simple trick in sending his protected messages of military significance. The trick, coined as 'Caesar cipher', is one of the simplest and widely known encryption techniques. The writer took the liberty to apply this technique in his program called 'FingCryptor'.

"K.I.S.S. = Keep It Stupid, Simple"

In this spirit, the writer has developed this application in order to assist him in his passwords management. The Wikipedia article on Caesar Cipher [1] is the main source as the guide to develop this application. The GUI - graphical user interface, is designed on Qt 4, whereas the codes for this application are written in C++. The writer is also planning to expand this application and making it more complex albeit low security in the future.

The following image shows how to understand Caesar cipher at its fundamental principle - i.e. shifting any letter three steps backwards.

Image 1: Caesar cipher left shift of 3 [1]

The main advantage of FingCryptor: the writer extended the use of this operation on all ASCII characters e.g. '@', '#', '?' etc, including numbers from 0 to 9.

Images 2 and 3 show the usage of FingCryptor.

Image 2: The ciphering on FingCryptor

Image 2: The deciphering on FingCryptor

The first version of this application is available for download at the bottom of this post. 

Source(s):

[1] Caesar cipher (Wikipedia): http://en.wikipedia.org/wiki/Caesar_cipher

REMINDER / WARNING:

1. For Windows only. (Linux version will be made available in the next development)
2. This application is NOT intended for any high security objective.
3. The writer is NOT liable for any security breach - i.e. deciphering by a third party etc.
4. Use at your own risk.

Have fun!

Download link:

Download FingCryptor (zipped) for Windows.