Problem Statement:
In this project, we aim to develop a Deep learning and image processing solution that enables the detection of nails in images for the purpose of applying various colors and designs. The solution will not only facilitate aesthetic enhancements by varying color schemes and decorative patterns on the nails but will also allow for modifications in the size of the nails as per user preferences. These capabilities require sophisticated image analysis and modification techniques, which will be explored and discussed in detail throughout this article. The primary challenge lies in accurately detecting nail boundaries and implementing transformations that maintain the realism and quality of the original image while introducing desired alterations.
Solution:
In the initial phase of our project, we curated a specialized dataset of nails which we subsequently annotated for precise boundary recognition. Utilizing the advanced capabilities of Detectron2, we trained our model to accurately detect nail boundaries and apply segmentation masks. This foundational setup enabled us to color the nails effectively and overlay various decorative stickers based on user selection.
As we transitioned to phase two of the project, our focus shifted to nail enlargement—a feature that required careful consideration to maintain realism in nail appearance. The primary challenge was to ensure that while enlarging the nails, the lower part remained fixed, and enlargement occurred from the upper portion of the nail only. This task was further complicated by the diverse shapes of nails a user might select, which necessitated a flexible yet precise approach to handling each unique nail form.
To address these complexities, our algorithm was designed to identify the orientation and positional hierarchy of each finger, beginning with the thumb, which is typically the bottommost predicted item. By determining the correct angle of enlargement for the thumb, we could then apply similar transformations to the other four fingers, ensuring uniformity and accuracy across all nails. This strategy also involved managing the randomness in nail detection by Detectron2, by associating each detected nail with its corresponding finger, allowing for targeted and appropriate image processing enhancements.
Through this meticulous approach, we successfully implemented a solution that adapts to various nail shapes and sizes, providing a dynamic and user-friendly tool for nail image enhancements.
Implementation:
# Find the bottom-most coordinates among the box centers
bottommost_coords = max(box_centers, key=lambda coord: (coord[1], coord[0]))
# Find the index of the box with the bottom-most coordinates
bottommost_index = next(i for i, coord in enumerate(box_centers) if coord == bottommost_coords)
# Iterate through predicted masks and apply rotation only to the mask with the bottom-most point
for i, (pred_mask, bbox) in enumerate(zip(pred_masks, bboxes)):
mask = pred_mask.cpu().numpy().astype('uint8')
contour, = cv2.findContours(mask, cv2.RETRLIST, cv2.CHAIN_APPROX_NONE)
# Apply rotation only if it's the mask with the bottom-most point
if i == bottommost_index:
# Color the nail shape using the specified color
abc = nail_shape_coloring(nail_of_shape, color)
# Get the shape of the image to be rotated
rows, cols, _ = abc.shape
#rows2, cols2, = styleimg.shape
angle = 55 # Rotate in the opposite direction for the bottom-most point
center = (cols // 2, rows // 2)
shift_amount = 2
center = (center[0] + shift_amount, center[1])
M = cv2.getRotationMatrix2D(center, angle, 1)
# Rotate the sticker image
rotate_image = cv2.warpAffine(abc, M, (cols, rows))
# starting for Sticker imageeeee
rows2, cols2, = styleimg.shape
angle = 55 # Rotate in the opposite direction for the bottom-most point
# Calculate the center of the image for rotation
center = (cols2 // 2, rows2 // 2)
# Get the rotation matrix with adjusted size
M = cv2.getRotationMatrix2D(center, angle, 1.0)
# Get the bounding rectangle of the rotated image
cos = np.abs(M[0, 0])
sin = np.abs(M[0, 1])
new_width = int(rows2 sin + cols2 cos)
new_height = int(rows2 cos + cols2 sin)
# Adjust the rotation matrix to take into account translation
M[0, 2] += (new_width - rows2) // 2
M[1, 2] += (new_height - cols2) // 2
# Rotate the sticker image with the adjusted matrix
rotate_sticker = cv2.warpAffine(style_img, M, (new_width, new_height))
# Apply style to the rotated image at the bottom-most point
img = apply_nail_style2(img, rotate_sticker, cv2.minAreaRect(contour[0]), nail_shape_string, rotate_image)
# Draw bounding box around the rotated thumb
box = bbox.cpu().numpy().astype(int)
x1, y1, x2, y2 = box # Unpack the coordinates
height = y2 - y1 # Calculate the height of the bounding box
y1_new = y1 - height # Increase the height by moving y1 upwards
#cv2.rectangle(img, (x1, y1_new), (x2, y2), (0, 255, 0), 2) # Green rectangle
else:
# Apply style without rotation for other masks
abc = nail_shape_coloring(nail_of_shape, color)
img = apply_nail_style2(img, style_img, cv2.minAreaRect(contour[0]), nail_shape_string, abc)
Results:
The left and right hand codes were separately written and to achieve the results with less resources and small dataset was a big achievement in itself. These are some pictures of the results.
Issues while solving this problem:
Here are the key issues encountered during the project, framed in a structured manner:
Order of Detection Challenges:
The model does not consistently identify fingers in a specific order, which complicates the accurate detection and enhancement of the thumb, often crucial for defining the orientation for image processing tasks.
Incorporating AI Outputs into Image Processing:
Integrating AI-generated results with traditional image processing techniques poses difficulties, especially when these outputs (such as nail boundaries or enhancement effects) need seamless application.
Incomplete Finger Detection:
At times, the model fails to detect the thumb. This results in having only four fingers to process, which impacts the effectiveness of the applied transformations. This suggests a need to expand the training dataset to cover more scenarios and improve detection robustness.
Misidentification of Hands:
The model occasionally misclassified other objects as hands, necessitating additional validation steps to ensure that only valid hand images are processed, thereby maintaining the integrity and accuracy of enhancements.
Dimensionality Constraints:
Nails naturally possess a curved, three-dimensional shape, but the images captured and processed are two-dimensional. This discrepancy poses a challenge in accurately rendering enhancements that need to reflect the 3D nature of nails.
Boundary Control for Style Images:
It is crucial to devise an efficient algorithm to confine the various sizes and designs of style images strictly within the nail boundaries. This ensures that enhancements do not exceed the nail area, preserving the natural look of the nail enhancements.
Great