Abstract PDF BibTeX

In creative workflows, designers compile a collection (or “moodboard”) of inspirational assets for ideation. They may use this as reference for finding additional assets. In this work, we aim to stimulate creative ideation by making suggestions that take cues from the designer’s moodboard but also carefully diverge from it. A collection of images may have rich information along different axes (e.g. color, composition, or style) – these axes or channels can be used to model relevance or divergence of any image from a query collection. So, we develop a self-supervised model that can extract channel-specific representations from collections of images. We propose a search algorithm that uses these representations to obtain results that satisfy the collection’s intent along some channels but diverge from the query along others. We show that this allows for effective exploration of the creative space of possibilities. Finally, we demonstrate a mix-and-match visual querying mechanism that allows us to combine channels from different collections of inspirational content, thus facilitating ease in creative expression.

@article{jaininspiration,
  title={Inspiration Retrieval for Visual Exploration},
  author={Jain, Nihal and Vaddamanu, Praneetha and Maheshwari, Paridhi and Vinay, Vishwa and Kulkarni, Kuldeep}
  booktitle={NeurIPS Workshop on Machine Learning for Creativity and Design},
  year={2021}
}}

Abstract PDF BibTeX

We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition - the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Frechet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks.

@inproceedings{maheshwari2021generating,
  title={Generating Compositional Color Representations from Text},
  author={Maheshwari, Paridhi and Jain, Nihal and Vaddamanu, Praneetha and Raut, Dhananjay and Vaishay, Shraiysh and Vinay, Vishwa},
  booktitle={Proceedings of the 30th ACM International Conference on Information \& Knowledge Management},
  pages={1222--1231},
  year={2021}
}

Abstract PDF Talk BibTeX

Scene graphs are a powerful structured representation of the underlying content of images, and embeddings derived from them have been shown to be useful in multiple downstream tasks. In this work, we employ a graph convolutional network to exploit structure in scene graphs and produce image embeddings useful for semantic image retrieval. Different from classification-centric supervision traditionally available for learning image representations, we address the task of learning from relative similarity labels in a ranking context. Rooted within the contrastive learning paradigm, we propose a novel loss function that operates on pairs of similar and dissimilar images and imposes relative ordering between them in embedding space. We demonstrate that this Ranking loss, coupled with an intuitive triple sampling strategy, leads to robust representations that outperform well-known contrastive losses on the retrieval task. In addition, we provide qualitative evidence of how retrieved results that utilize structured scene information capture the global context of the scene, different from visual similarity search.

@inproceedings{maheshwari2021scene,
  title={Scene Graph Embeddings Using Relative Similarity Supervision},
  author={Maheshwari, Paridhi and Chaudhry, Ritwick and Vinay, Vishwa},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={35},
  number={3},
  pages={2328--2336},
  year={2021}
}

Abstract PDF BibTeX

Image search engines rely on appropriately designed ranking features that capture various aspects of the content semantics as well as the historic popularity. In this work, we consider the role of colour in this relevance matching process. Our work is motivated by the observation that a significant fraction of user queries have an inherent colour associated with them. While some queries contain explicit colour mentions (such as 'black car' and 'yellow daisies'), other queries have implicit notions of colour (such as 'sky' and 'grass'). Furthermore, grounding queries in colour is not a mapping to a single colour, but a distribution in colour space. For instance, a search for 'trees' tends to have a bimodal distribution around the colours green and brown. We leverage historical clickthrough data to produce a colour representation for search queries and propose a recurrent neural network architecture to encode unseen queries into colour space. We also show how this embedding can be learnt alongside a cross-modal relevance ranker from impression logs where a subset of the result images were clicked. We demonstrate that the use of a query-image colour distance feature leads to an improvement in the ranker performance as measured by users' preferences of clicked versus skipped images.

@inproceedings{maheshwari2020learning,
  title={Learning Colour Representations of Search Queries},
  author={Maheshwari, Paridhi and Ghuhan, Manoj and Vinay, Vishwa},
  booktitle={Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages={1389--1398},
  year={2020}
}

Abstract PDF BibTeX

Navigating a collection of documents can be facilitated by obtaining a human-understandable concept hierarchy with links to the content. This is a non-trivial task for two reasons. First, defining concepts that are understandable by an average consumer and yet meaningful for a large variety of corpora is hard. Second, creating semantically meaningful yet intuitive hierarchical representation is hard, and can be task dependent. We present out system Navigation.ai which automatically processes a document collection, induces a concept hierarchy using Wikipedia and presents an interactive interface that helps user navigate to individual paragraphs using concepts.

@inproceedings{modani2020exploratory,
  title={Exploratory Navigation and Selective Reading},
  author={Modani, Natwar and Maheshwari, Paridhi and Deshpande, Harsh and Sirpurkar, Saurab and Aditya, Somak and others},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={34},
  number={09},
  pages={13632--13633},
  year={2020}
}

Abstract PDF Poster Demo BibTeX

Augmented Reality (AR) is rapidly gaining popularity, enhancing human perception of the real world by augmenting digital experiences. Existing tools for authoring AR scenes are either template based or require domain knowledge from experts, and are therefore restrictive. ARComposer is a novel interface that enables easy authoring of AR experiences from free-form text describing the scene. Our proposed interface allows creators to compose varied scenes comprising of multiple objects with diverse relationships to each other as well as human models with animations, starting merely with a textual description. A qualitative evaluation shows that ARComposer provides a good flow experience to its users. Furthermore, a crowd-sourced experiment evaluating various aspects of the rendered AR scenes indicates the viability of the proposed approach.

@inproceedings{shekhar2019arcomposer,
  title={ARComposer: Authoring Augmented Reality Experiences through Text},
  author={Shekhar, Sumit and Maheshwari, Paridhi and Singhal, Amrit and Singh, Kush Kumar and Krishna, Kundan},
  booktitle={The Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology},
  pages={63--65},
  year={2019}
}

Abstract PDF Slides Talk BibTeX

Banners are present in several forms and a person might be inspired by one or more of these. However, designing banners is a non-trivial task, especially for novices. Starting from a blank canvas can often be overwhelming, and exploring alternatives is time-consuming. In this paper, we propose an automatic approach to transfer a novice user's content into an example banner. Our algorithm begins with extracting the template of the example banner via a semantic segmentation approach. This is followed by an energy-based optimization framework to combine multiple design elements and arrive at an optimal layout. A crowd-sourced experiment comparing our automatic results against banners designed by creative professionals indicates the viability of the proposed work.

@inproceedings{maheshwari2019exemplar,
  title={Exemplar based experience transfer},
  author={Maheshwari, Paridhi and Bansal, Nitish and Dwivedi, Surya and Kumar, Rohan and Manerikar, Pranav and Srinivasan, Balaji Vasan},
  booktitle={Proceedings of the 24th International Conference on Intelligent User Interfaces},
  pages={673--680},
  year={2019}
}

Abstract PDF Slides BibTeX

Recent efforts towards digitization of cultural heritage artifacts have resulted in a surge of information around these artifacts. However, the organization of these artifacts falls short with respect to accessing the facts across these entities. In this paper, we present a method to harvest the knowledge and form a knowledge graph from the digitized artifacts in the Museums of India repository via distant supervision to enable better accessibility of the facts and ability to extract new insights around the artifacts. Triples extracted from an open information extractor are first canonicalized to a standard taxonomy based on a metricbased scoring. Since a standard taxonomy is insufficient to capture all the relationships, we propose a sequential clustering based approach to add artifact specific relationships to the taxonomy (and to the knowledge graph). The graph is enriched by inferring missing facts based on a probabilistic soft logic approach seeded from a frequent item set framework. Human evaluation of the final knowledge graph showed an accuracy of 75% on par with knowledge bases like DBpedia.

@inproceedings{sancheti2018harvesting,
  title={Harvesting Knowledge from Cultural Heritage Artifacts in Museums of India},
  author={Sancheti, Abhilasha and Maheshwari, Paridhi and Chaturvedi, Rajat and Monsy, Anish V and Goyal, Tanya and Srinivasan, Balaji Vasan},
  booktitle={Pacific-Asia Conference on Knowledge Discovery and Data Mining},
  pages={312--324},
  year={2018},
  organization={Springer}
}

Abstract Document BibTeX

A framework is provided for constructing enterprise-specific knowledge bases from enterprise-specific data that includes structured and unstructured data. Relationships between entities that match known relationships are identified for each of a plurality of tuples included in the structured data. Where possible, relationships between entities that match known relationships also are identified for tuples included in the unstructured data. If matching relationships between entities that cannot be identified for tuples in the unstructured data, extracted relationships are sequentially clustered to similar relationships and a relationship is assigned to the clustered tuples. An enterprise-specific knowledge graph is constructed from the structured-data-tuples and their identified relationships, the unstructured-data-tuples where the relationships could be mapped to a known relationship and their identified relationships, and the unstructured-data-tuples that could not be mapped to a known relationship and their assigned relationships. The knowledge graph is enriched with any information determined to be missing therefrom.

@misc{srinivasan2021constructing,
  title={Constructing enterprise-specific knowledge graphs},
  author={Srinivasan, Balaji Vasan and Chaturvedi, Rajat and Goyal, Tanya and Maheshwari, Paridhi and Monsy, Anish Valliyath and Sancheti, Abhilasha},
  year={2021},
  month=feb # "~9",
  publisher={Google Patents},
  note={US Patent 10,915,577}
}

Abstract Document BibTeX

A natural language scene description is converted into a scene that is rendered in three dimensions by an augmented reality (AR) display device. Text-to-AR scene conversion allows a user to create an AR scene visualization through natural language text inputs that are easily created and well-understood by the user. The user can, for instance, select a pre-defined natural language description of a scene or manually enter a custom natural language description. The user can also select a physical real-world surface on which the AR scene is to be rendered. The AR scene is then rendered using the augmented reality display device according to its natural language description using 3D models of objects and humanoid characters with associated animations of those characters, as well as from extensive language-to-visual datasets. Using the display device, the user can move around the real-world environment and experience the AR scene from different angles.

@misc{shekhar2020visualizing,
  title={Visualizing natural language through 3D scenes in augmented reality},
  author={Shekhar, Sumit and Maheshwari, Paridhi and Monisha, J and Krishna, Kundan and Singhal, Amrit and Singh, Kush Kumar},
  year={2020},
  month=may # "~26",
  publisher={Google Patents},
  note={US Patent 10,665,030}
}

Document

Document

Document

Document

Document