A simple depth-key-based image composition is proposed, which uses two still images with depth information, background and foreground object. The proposed method can place the object at various locations in the background considering the depth in the 3D world coordinate system. The main feature is that a simple algorithm is provided, which enables us to achieve the depthward movement within the camera plane, without being aware of the 3D world coordinate system. Two algorithms are proposed (P-OMDD and O-OMDD), which are based on the pin-hole camera model. As an advantage, camera calibration is not required before applying the algorithm in these methods. Since a single image is used for the object representation, each of the proposed methods has its limitations in terms of fidelity of the composite image. P-OMDD faithfully reproduces the angle at which the object is seen, but the pixels of the hidden surface are missing. On the contrary, O-OMDD can avoid the hidden surface problem, but the angle of the object is fixed, wherever it moves. It is verified through several experiments that, when using O-OMDD, subjectively natural composite images can be obtained under any object movement, in terms of size and position in the camera plane. Future tasks include improving the change in illumination due to positional changes and the partial loss of objects due to noise in depth images.

Multiscale retinex is one of the most popular image enhancement methods. However, its control parameters, such as Gaussian kernel sizes, gain, and offset, should be tuned carefully according to the image contents. In this letter, we propose a new method that optimizes the parameters using practical swarm optimization and multi-objective function. The method iteratively verifies the visual quality (i.e. brightness, contrast, and colorfulness) of the enhanced image using a multi-objective function while subtly adjusting the parameters. Experimental results shows that the proposed method achieves better image quality qualitatively and quantitatively compared with other image enhancement methods.

Computer aided design (CAD) technology is widely used for architectural design, but current CAD tools still require high-level design specifications from human. It would be significant to construct an intelligent CAD system allowing automatic architectural layout parsing (AutoALP), which generates candidate designs or predicts architectural attributes without much user intervention. To tackle these problems, many learning-based methods were proposed, and benchmark dataset become one of the essential elements for the data-driven AutoALP. This paper proposes a new dataset called SCUT-AutoALP for multi-paradigm applications. It contains two subsets: 1) Subset-I is for floor plan design containing 300 residential floor plan images with layout, boundary and attribute labels; 2) Subset-II is for urban plan design containing 302 campus plan images with layout, boundary and attribute labels. We analyzed the samples and labels statistically, and evaluated SCUT-AutoALP for different layout parsing tasks of floor plan/urban plan based on conditional generative adversarial networks (cGAN) models. The results verify the effectiveness and indicate the potential applications of SCUT-AutoALP. The dataset is available at https://github.com/designfuturelab702/SCUT-AutoALP-Database-Release.

A multi-task convolutional neural network leading to high performance and interpretability via attribute estimation is presented in this letter. Our method can provide interpretation of the classification results of CNNs by outputting attributes that explain elements of objects as a judgement reason of CNNs in the middle layer. Furthermore, the proposed network uses the estimated attributes for the following prediction of classes. Consequently, construction of a novel multi-task CNN with improvements in both of the interpretability and classification performance is realized.

In this paper, we improve upon the automatic building extraction method, which uses a variational inference Gaussian mixture model for performing color clustering, by accelerating its computational speed. The improved method decreases the computational time using an image with reduced resolution upon applying color clustering. According to our experiment, in which we used 106 scenery images, the improved method could extract buildings at a rate 86.54% faster than that of the conventional methods. Furthermore, the improved method significantly increased the extraction accuracy by 1.8% or more by preventing over-clustering using the reduced image, which also had a reduced number of the colors.

High dynamic range (HDR) imaging refers to digital image processing that modifies the range of color and contrast to enhance image visibility. To create an HDR image, two or more images that include various information are needed. In order to convert low dynamic range (LDR) images to HDR images, we consider the possibility of using a generative adversarial network (GAN) as an appropriate deep neural network. Deep learning requires a great deal of data in order to build a module, but once the module is created, it is convenient to use. In this paper, we propose a weight map for local luminance based on learning to reconstruct locally tone-mapped images.

Troika is a recently proposed sponge-based hash function for IOTA's ternary architecture and platform, which is developed by CYBERCRYPT and is now used in IOTA's blockchain. In this paper, we introduce the preimage attack on 2/3 rounds of Troika with a divide-and-conquer approach. Firstly, we propose the equivalent conditions to determine whether a message is the preimage with an algebraic method. As a result, for the preimage attack on two-round Troika, we can search the preimage only in a valid smaller space and efficiently enumerate the messages which can satisfy most of the equivalent conditions with a guess-and-determine technique. Our experiments show that the time complexity of the preimage attack on 2-round Troika can be improved to 379 from 3243. For the preimage attack on 3-round Troika, the MILP-based method is applied to achieve the optimal time complexity, which is 327 times faster than brute force.

In this letter, we propose a hierarchical segmentation (HS) method for color images, which can not only maintain the segmentation accuracy, but also ensure a good speed. In our method, HS adopts the fuzzy simple linear iterative clustering (Fuzzy SLIC) to obtain an over-segmentation result. Then, HS uses the fast fuzzy C-means clustering (FFCM) to produce the rough segmentation result based on superpixels. Finally, HS takes the non-iterative K-means clustering using priority queue (KPQ) to refine the segmentation result. In the validation experiments, we tested our method and compared it with state-of-the-art image segmentation methods on the Berkeley (BSD500) benchmark under different types of noise. The experiment results show that our method outperforms state-of-the-art techniques in terms of accuracy, speed and robustness.

The task of image annotation is becoming enormously important for efficient image retrieval from the web and other large databases. However, huge semantic information and complex dependency of labels on an image make the task challenging. Hence determining the semantic similarity between multiple labels on an image is useful to understand any incomplete label assignment for image retrieval. This work proposes a novel method to solve the problem of multi-label image annotation by unifying two different types of Laplacian regularization terms in deep convolutional neural network (CNN) for robust annotation performance. The unified Laplacian regularization model is implemented to address the missing labels efficiently by generating the contextual similarity between labels both internally and externally through their semantic similarities, which is the main contribution of this study. Specifically, we generate similarity matrices between labels internally by using Hayashi's quantification method-type III and externally by using the word2vec method. The generated similarity matrices from the two different methods are then combined as a Laplacian regularization term, which is used as the new objective function of the deep CNN. The Regularization term implemented in this study is able to address the multi-label annotation problem, enabling a more effectively trained neural network. Experimental results on public benchmark datasets reveal that the proposed unified regularization model with deep CNN produces significantly better results than the baseline CNN without regularization and other state-of-the-art methods for predicting missing labels.

We aim at HDR imaging with simple processing while preventing spatial resolution degradation in multiple-exposure-time image sensor where the exposure time is controlled for each pixel. The contributions are the proposal of image interpolation by motion area detection and pixel adaptive weighting method by overexposure and motion blur detection.

Although the classical vector median filter (VMF) has been widely used to suppress the impulse noise in the color image, many thin color curve pixels aligned in arbitrary directions are usually removed out as impulse noise. This serious problem can be solved by the proposed method that can protect the thin curves in arbitrary direction in color image and remove out the impulse noise at the same time. Firstly, samples in the 3x3 filter window are considered to preliminarily detect whether the center pixel is corrupted by impulse noise or not. Then, samples outside a 5x5 filter window are conditionally and partly considered to accurately distinguish the impulse noise and the noise-free pixel. At last, based on the previous outputs, samples on the processed positions in a 3x3 filter window are chosen as the samples of VMF operation to suppress the impulse noise. Extensive experimental results indicate that the proposed algorithm can be used to remove the impulse noise of color image while protecting the thin curves in arbitrary directions.

Finding the correspondence between two images of the same object or scene is an active research field in computer vision. This paper develops a rapid and effective Content-based Superpixel Image matching and Stitching (CSIS) scheme, which utilizes the content of superpixel through multi-features fusion technique. Unlike popular keypoint-based matching method, our approach proposes a superpixel internal feature-based scheme to implement image matching. In the beginning, we make use of a novel superpixel generation algorithm based on content-based feature representation, named Content-based Superpixel Segmentation (CSS) algorithm. Superpixels are generated in terms of a new distance metric using color, spatial, and gradient feature information. It is developed to balance the compactness and the boundary adherence of resulted superpixels. Then, we calculate the entropy of each superpixel for separating some superpixels with significant characteristics. Next, for each selected superpixel, its multi-features descriptor is generated by extracting and fusing local features of the selected superpixel itself. Finally, we compare the matching features of candidate superpixels and their own neighborhoods to estimate the correspondence between two images. We evaluated superpixel matching and image stitching on complex and deformable surfaces using our superpixel region descriptors, and the results show that new method is effective in matching accuracy and execution speed.

Embedding efficiency is an important issue in steganography methods. Matrix embedding (1, n, h) steganography was proposed by Crandall to achieve high embedding efficiency for palette images. This paper proposes a steganography method based on multilayer matrix embedding for palette images. First, a parity assignment is provided to increase the image quality. Then, a multilayer matrix embedding (k, 1, n, h) is presented to achieve high embedding efficiency and capacity. Without modifying the color palette, hk secret bits can be embedded into n pixels by changing at most k pixels. Under the same capacity, the embedding efficiency of the proposed method is compared with that of pixel-based steganography methods. The comparison indicates that the proposed method has higher embedding efficiency than pixel-based steganography methods. The experimental results also suggest that the proposed method provides higher image quality than some existing methods under the same embedding efficiency and capacity.

Pedestrian detection is a critical problem in computer vision with significant impact on many real-world applications. In this paper, we introduce an fast dual-task pedestrian detector with integrated segmentation context (DTISC) which predicts pedestrian location as well as its pixel-wise segmentation. The proposed network has three branches where two main branches can independently complete their tasks while useful representations from each task are shared between two branches via the integration branch. Each branch is based on fully convolutional network and is proven effective in its own task. We optimize the detection and segmentation branch on separate ground truths. With reasonable connections, the shared features introduce additional supervision and clues into each branch. Consequently, the two branches are infused at feature spaces increasing their robustness and comprehensiveness. Extensive experiments on pedestrian detection and segmentation benchmarks demonstrate that our joint model improves the performance of detection and segmentation against state-of-the-art algorithms.

This Letter proposes a autoencoder model supervised by semantic similarity for zero-shot learning. With the help of semantic similarity vectors of seen and unseen classes and the classification branch, our experimental results on two datasets are 7.3% and 4% better than the state-of-the-art on conventional zero-shot learning in terms of the averaged top-1 accuracy.

High frame rate and ultra-low delay matching system plays an increasingly important role in human-machine interactions, because it guarantees high-quality experiences for users. Existing image matching algorithms always generate mismatches which heavily weaken the performance the human-machine-interactive systems. Although many mismatch removal algorithms have been proposed, few of them achieve real-time speed with high frame rate and low delay, because of complicated arithmetic operations and iterations. This paper proposes a temporal constraints and block weighting judgement based high frame rate and ultra-low delay mismatch removal system. The proposed method is based on two temporal constraints (proposal #1 and proposal #2) to firstly find some true matches, and uses these true matches to generate block weighting (proposal #3). Proposal #1 finds out some correct matches through checking a triangle route formed by three adjacent frames. Proposal #2 further reduces mismatch risk by adding one more time of matching with opposite matching direction. Finally, proposal #3 distinguishes the unverified matches to be correct or incorrect through weighting of each block. Software experiments show that the proposed mismatch removal system achieves state-of-the-art accuracy in mismatch removal. Hardware experiments indicate that the designed image processing core successfully achieves real-time processing of 784fps VGA (640×480 pixels/frame) video on field programmable gate array (FPGA), with a delay of 0.858 ms/frame.

This paper proposes a method to create various training images for instance segmentation in a semi-supervised manner. In our proposed learning scheme, a few 3D CG models of target objects and a large number of images retrieved by keywords from the Internet are employed for initial model training and model update, respectively. Instance segmentation requires pixel-level annotations as well as object class labels in all training images. A possible solution to reduce a huge annotation cost is to use synthesized images as training images. While image synthesis using a 3D CG simulator can generate the annotations automatically, it is difficult to prepare a variety of 3D object models for the simulator. One more possible solution is semi-supervised learning. Semi-supervised learning such as self-training uses a small set of supervised data and a huge number of unsupervised data. The supervised images are given by the 3D CG simulator in our method. From the unsupervised images, we have to select only correctly-detected annotations. For selecting the correctly-detected annotations, we propose to quantify the reliability of each detected annotation based on its silhouette as well as its textures. Experimental results demonstrate that the proposed method can generate more various images for improving instance segmentation.

In this paper, we present the effectiveness of image compression based on a convolutional auto encoder (CAE) with region of interest (ROI) for quality control. We propose a method that adapts image quality for prioritized parts and non-prioritized parts for CAE-based compression. The proposed method uses annotation information for the distortion weights of the MS-SSIM-based loss function. We show experimental results using a road damage image dataset that is used to check damaged parts and an image dataset with segmentation data (ADE20K). The experimental results reveals that the proposed weighted loss function with CAE-based compression from F. Mentzer et al. learns some characteristics and preferred bit allocations of the prioritized parts by end-to-end training. In the case of using road damage image dataset, our method reduces bpp by 31% compared to the original method while meeting quality requirements that an average weighted MS-SSIM for the road damaged parts be larger than 0.97 and an average weighted MS-SSIM for the other parts be larger than 0.95.

We propose an effective 2d image based end-to-end deep learning model for malware detection by introducing a black & white embedding to reserve bit information and adapting the convolution architecture. Experimental results show that our proposed scheme can achieve superior performance in both of training and testing data sets compared to well-known image recognition deep learning models (VGG and ResNet).

Due to the superior performance, deep learning has been widely applied to various applications, including image classification, bioinformatics, and cybersecurity. Nevertheless, the research investigations on deep learning in the adversarial environment are still on their preliminary stage. The emerging adversarial learning methods, e.g., generative adversarial networks, have introduced two vital questions: to what degree the security of deep learning with the presence of adversarial examples is; how to evaluate the performance of deep learning models in adversarial environment, thus, to raise security advice such that the selected application system based on deep learning is resistant to adversarial examples. To see the answers, we leverage image classification as an example application scenario to propose a framework of Evaluating Deep Learning for Image Classification (EDLIC) to conduct comprehensively quantitative analysis. Moreover, we introduce a set of evaluating metrics to measure the performance of different attacking and defensive techniques. After that, we conduct extensive experiments towards the performance of deep learning for image classification under different adversarial environments to validate the scalability of EDLIC. Finally, we give some advice about the selection of deep learning models for image classification based on these comparative results.