top of page

Abstract

 

Recent years have witnessed a growing trend of fashion compatibility modeling, which scores the matching degree of the given outfit and then provides people with some dressing advice. Existing methods have primarily solved this problem by analyzing the discrete interaction among complementary items. However, the fashion items would present certain occlusion and deformation when they are tried on. Therefore, the discrete item interaction cannot capture the fashion compatibility in a combined manner due to the neglect of a crucial factor: the overall try-on appearance. In light of this, we propose a multi-model try-on-guided compatibility modeling scheme to jointly characterize the discrete interaction and try-on appearance. In particular, we first propose a multi-modal try-on template generator to automatically generate a try-on template of the outfit, depicting the overall look of its composing fashion items. Then, we introduce a new compatibility modeling scheme which integrates outfit try-on appearance into the discrete item interaction modeling. To fulfill the proposal, we construct a large-scale real-world dataset from SSENSE, named FOTOS, consisting of 11,000 well-matched outfits and their corresponding realistic try-on images.  

Figure 2: Illustration of the proposed TryOn-CM, which could analyze the fashion compatibility from both the discrete item interaction and try-on appearance.

微信截图_20200709134321.png
微信截图_20200709134342.png

Framework

Figure 3: Structure of the multi-modal try-on template generator, comprising a visual and textual generator.

FOTOS Dataset

微信截图_20200709140954.png
微信截图_20200710095002.png

Table 1: Statistics of the FOTOS dataset.

Figure 3: Examples in FOTOS dataset, which consists of 11,000 well-matched outfit composed by 20,384 fashion items from the online website SSENSE.

Resources

                         [Baidu Drive: 74bs] (A new edition)

Results of Multi-modal Template Generator

微信截图_20200710095516.png

Figure 4: Examples of the multi-modal try-on template generation. 𝑃𝑣 and 𝑃𝑡 are the generated templates from the visual and textual modality, respectively. 𝑃 is the ground truth try-on image.

Results of Complementary Item Retrieval

微信截图_20200710095537.png

Figure 5: Ranking results of the discrete compatibility modeling (DCM) and our multi-modal try-on-guided compatibility modeling. The positive complementary item of the query is circled in the green box and we visualize the try-on template of the outfit generated by the try-on template generator in the last line.

Copyright (C) <2019>  Shandong University

This program is licensed under the GNU General Public License 3.0 (https://www.gnu.org/licenses/gpl-3.0.html). Any derivative work obtained under this license must be licensed under the GNU General Public License as published by the Free Software Foundation, either Version 3 of the License, or (at your option) any later version, if this derivative work is distributed to a third party.

The copyright for the program is owned by Shandong University. For commercial projects that require the ability to distribute the code of this program as part of a program that cannot be distributed under the GNU General Public License, please contact <dongxue.sdu@gmail.com> to purchase a commercial license.

bottom of page