New HCI group + upcoming papers and demos at UIST and ISS 2018

Fabrice Matulic

2018-10-15 09:11:17

Creation of HCI group

At PFN, we aspire to create next-generation “intelligent” systems and services, powered by cutting-edge AI technology, but we also recognise that humans will remain essential actors in the design and usage of such systems and therefore it is paramount to think about how the dialogue occurs. Human-Computer Interaction (HCI) approaches, which focus on bridging the gap between people and machines, can considerably contribute to improving intricate machine-learning processes requiring human intervention. With the creation of a dedicated HCI group at PFN, we aim to advance user-centred design for AI and machines and make sure the “humans in the loop” are supported with powerful tools when working with such systems.

Broadly, there are three main lines of research that the team would like to pursue:

  • HCI for machine learning: Utilise HCI methods to facilitate complex or tedious machine-learning processes in which people are involved (such as data gathering, labelling, pre-processing, augmentation; neural network engineering, deployment, and management)
  • Machine-learning for HCI: Use deep learning to enhance existing or enable new interaction techniques (e.g. advanced gesture recognition, activity recognition, multimodal input, sensor fusion, embodied interaction, collaboration between AI, robots and humans, generative model to create interactive content etc.)
  • Human-Robot Interaction (HRI): Make communication and interaction between smart robots and their users natural, intuitive and hopefully even fun!

Of course, HCI does not necessarily involve machine learning or robots and we are also generally interested in creating novel and exciting interactive experiences.

The HCI group will benefit from the expertise of Prof. Takeo Igarashi, of The University of Tokyo, who has been hired as an external consultant. In addition to his wide experience in HCI and HRI, Prof. Igarashi has recently started a JST CREST project on “HCI for machine learning” at his lab, which very much aligns with our research interests. We look forward to a long and fruitful collaboration.

Papers and demos at UIST and ISS 2018

Although the group was just officially created, we have been active in HCI research for the past months already and we will present two papers on recent work, respectively at UIST this week and ISS next month.

The first project, which was started at the University of Waterloo with Drini Cami and Prof. Dan Vogel, proposes to use different ways of holding a stylus pen while writing on a tablet to trigger different UI actions. The technique uses machine learning on the raw touch input data to detect these different pen grips when the user contacts the surface with the hand. The advantage of our technique is that it allows to rapidly switch between various pen modes using the same hand that writes and without resorting to cumbersome UI widgets.

In addition to the paper presentation, Drini will also be showing the technique at UIST’s popular demo session.

Our second contribution is the interactive projection mapping system for PaintsChainer that we showed at the Winter Comiket last year. For those of you who missed it, ColourAIze (which is how we call it in the paper) works directly with drawings and art on paper. Specifically, it projects colour fills determined by PaintsChainer directly onto the paper drawing with the colouring superimposed on the line art. Like with the web version of PaintsChainer, the ability to specify local colour hints to influence the colourisation is supported through simple (digital) pen strokes.

As with the pen-posture project above, we will both present our paper and do a demo of the system at the conference. If you’d like to try the fun experience of having your paper sketches, drawings and mangas coloured by AI, come and see us at ISS in Tokyo in November!

Last but not least, we are looking for talented HCI researchers to join our team, so if you think you can contribute in the areas mentioned above, please check the details of the position on our jobs page and apply!

 

We have released ChainerUI, a training visualizer and manager for Chainer

ofk

2017-12-20 10:58:31

We have released ChainerUI, to help visualize training results and manage training jobs.

Among Chainer users, there are demands to watch the progress of training jobs or to compare jobs by plotting the training loss, accuracy, and other logs of multiple runs. These tasks tend to be intricate because there were no suitable applications available. ChainerUI offers functions listed below in order to support your DNN training routine.

  • Visualizing training logs: plot values like loss and accuracy
  • Managing histories of training jobs with experimental conditions
  • Operating the training jobs: take a snapshot, modify hyper parameter like within training

ChainerUI consists of a web application and an extension module of Trainer in Chainer, which enables easy training. If you have already used the LogReport extension, you can watch training logs on web browser without any change. If you add other extensions of ChainerUI, more experimental conditions will be displayed on the table and the training job can be managed from ChainerUI.

Visualizing training logs


ChainerUI monitors training log file and plots values such as loss, accuracy, and so on. Users can choose which variables to plot on the chart.

Managing training jobs


ChainerUI’s web application shows the list of multiple training jobs on the result table with experimental conditions. In addition, you can take actions such as taking snapshot or modifying hyperparameters from the job control panel.

How to use

To install, please use the pip module, and then setup the ChainerUI database.

pip install chainerui
chainerui db create
chainerui db upgrade

Secondly, register a “project” and run server. The “project” is the repository or the directory that contains chainer based scripts.

chainerui project create -d PROJECT_DIR [-n PROJECT_NAME]
chainerui server

You can also call chainerui project create while the server is running.

Finally, open http://localhost:5000/ on web browser, and you are ready!!

Visualizing training logs

The standard LogReport extension included in Chainer exports “log” file. ChainerUI watches that “log” file and plots a chart automatically. The following code is an example to run MNIST example to show plotting with ChainerUI.

chainerui project create -d path/to/result -n mnist
python train_mnist.py -o path/to/result/1

The “…result/1” result is added in “mnist” project. ChainerUI watches the “log” file updated in “path/to/result/1” continuously and plots values written in the file.

Managing training jobs

ChainerUI monitors the “args” file that is located in the same directory with the  “log” file, and shows its information on the results table as experimental conditions. The “args” file has key-value pairs in JSON style.
The below sample code shows how to save “args” file using ChainerUI’s utility function.

# [ChainerUI] import chainerui util function
from chainerui.utils import save_args

def main():
parser.add_argument('--out', '-o', default='result',
help='Directory to output the result')
args = parser.parse_args()

# [ChainerUI] save 'args' to show experimental conditions
save_args(args, args.out)

To operate training jobs, set CommandsExtension in the training script. This extension supports taking a snapshot and changing hyperparameters such as learning rate while running the training job.

# [ChainerUI] import CommandsExtension
from chainerui.extensions import CommandsExtension

def main():
trainer = training.Trainer(updater, (args.epoch, 'epoch'), out=args.out)

# [ChainerUI] enable to send commands from ChainerUI
trainer.extend(CommandsExtension())

To see whole code, examples/train_mnist.py.

Background

ChainerUI was mainly developed by Inagaki-san and Kobayashi-san who participated in summer internship at Preferred Networks this year.

During the two months of their internship program, they specified user requirements and implemented a prototype. They have continued to contribute after the internship as part-time workers. They are proud to release their work as “ChainerUI.”

Future plan

ChainerUI is being developed under the Chainer organization. The future plan includes the following functions.

  • Output chart as image file
  • Add other extensions to operate training script, etc.

We are also hiring front-end engineers to work on such! We are looking forward to receiving your applications.

Release Chainer Chemistry: A library for Deep Learning in Biology and Chemistry

Kosuke Nakago

2017-12-18 11:40:20

 

* Japanese blog is also written here.

We released Chainer Chemistry, a Chainer [1] extension to train and run neural networks for tasks in biology and chemistry.

The library helps you to easily apply deep learning on molecular structures.

For example, you can apply machine learning on toxicity classification tasks or HOMO (highest occupied molecular orbital) level regression task with compound input.

The library was developed during the PFN 2017 summer internship, and part of the library has been implemented by an internship student, Hirotaka Akita at Kyoto University.

 

Supported features

Graph Convolutional Neural Network implementation

The recently proposed Graph Convolutional Network (Refer below for detail) opened the door to apply deep learning on “graph structure” input, and the Graph Convolution Networks are currently an active area of research. We implemented several Graph Convolution Network architectures, including the network introduced in this year’s paper.

The following models are implemented:

  • NFP: Neural Fingerprint [2, 3]
  • GGNN: Gated-Graph Neural Network [4, 3]
  • WeaveNet: Molecular Graph Convolutions [5, 3]
  • SchNet: A continuous-filter convolutional Neural Network [6]

 

Common data preprocessing/research dataset support

Various datasets can be used with a common interface with this library. Also, some research datasets can be downloaded automatically and preprocessed.

The following datasets are supported:

  • QM9 [7, 8]: dataset of organic molecular structures with up to nine C/O/N/F atoms and their computed physical property values. The values include HOMO/LUMO level and internal energy. The computation is B3LYP/6-31G level of quantum chemistry.
  • Tox21 [9]: dataset of toxicity measurements on 12 biological targets

 

Train/inference example code is available

We provide example code for training models and inference. You can easily try training/inference of the models implemented in this library for quick start.

 

Background

In the new material discovery/drug discovery field, simulation of molecule behavior is important. When we need to take quantum effects into account with high precision, DFT (density functional theory) is widely used. However it requires a lot of computational resources especially for big molecules. It is difficult to apply simulation on many molecule structures.

There is a different approach from the machine learning field: learn the data measured/calculated in previous experiments, and predict the unexperimented molecule’s chemical property. The neural network may calculate the prediction faster than the quantum simulation.

 

Cited from “Neural Message Passing for Quantum Chemistry”, Justin et al. https://arxiv.org/pdf/1704.01212.pdf

 

An important question is how to deal with the input/output of compounds in order to apply deep learning. The main problem is that all molecular structures have variable numbers of atoms, represented as different graph structures, while conventional deep learning methods deal with a fixed size/structured input.

However “Graph Convolutional Neural Network” is proposed to deal with graph structure for input.

 

What is a Graph Convolutional Neural Network

Convolutional Neural Networks introduce “convolutional” layers which applies a kernel on local information in an image. It shows promising results on many image tasks, including classification, detection, segmentation, and even image generation tasks.

Graph Convolutional Neural Networks introduce a “graph convolution” operation which applies a kernel among the neighboring nodes on the graph, to deal with graph structure.

 

How graph convolutions work

CNN deals with an image as input, whereas Graph CNN can deal with a graph structure (molecule structure etc) as input.

Its application is not limited to molecule structure. “Graph structures” can appear in many other fields, including social networks, transportation etc, and the research of graph convolutional neural network applications is an interesting topic. For example, [10] applied graph convolution on image, [11] applied it on knowledge base, [12] applied it on traffic forecasting.

 

Target users

  1. Deep learning researchers
    This library provides latest Graph Convolutional Neural Network implementation
    Graph Convolution application is not limited to Biology & Chemistry, but various kinds of fields. We would like many people to use this library.
  2. Material/drug discovery researchers
    The library enables the user to build their own model to predict various kinds of chemical properties of a molecule.

 

Future plan

This library is still a beta version, and in active development. We would like to support the following features:

  • Provide pre-trained models for inference
  • Add more datasets
  • Implement more networks

We prepared a Tutorial to get started with this library, please try and let us know if you have any feedback.

 

Reference

[1] Tokui, S., Oono, K., Hido, S., & Clayton, J. (2015). Chainer: a next-generation open source framework for deep learning. In Proceedings of workshop on machine learning systems (LearningSys) in the twenty-ninth annual conference on neural information processing systems (NIPS) (Vol. 5).

[2] Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., & Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems (pp. 2224-2232).

[3] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.

[4] Li, Y., Tarlow, D., Brockschmidt, M., & Zemel, R. (2015). Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493.

[5] Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016). Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30(8), 595-608.

[6] Kristof T. Schütt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, Klaus-Robert Müller (2017). SchNet: A continuous-filter convolutional neural network for modeling quantum interactions. arXiv preprint arXiv:1706.08566

[7] L. Ruddigkeit, R. van Deursen, L. C. Blum, J.-L. Reymond, Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17, J. Chem. Inf. Model. 52, 2864–2875, 2012.

[8] R. Ramakrishnan, P. O. Dral, M. Rupp, O. A. von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules, Scientific Data 1, 140022, 2014.

[9] Huang R, Xia M, Nguyen D-T, Zhao T, Sakamuru S, Zhao J, Shahane SA, Rossoshek A and Simeonov A (2016) Tox21 Challenge to Build Predictive Models of Nuclear Receptor and Stress Response Pathways as Mediated by Exposure to Environmental Chemicals and Drugs. Front. Environ. Sci. 3:85. doi: 10.3389/fenvs.2015.00085

[10] Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst (2016), Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering, NIPS 2016.

[11] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling (2017) Modeling Relational Data with Graph Convolutional Networks. arXiv preprint arXiv: 1703.06103

[12] Yaguang Li, Rose Yu, Cyrus Shahabi, Yan Liu (2017) Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. arXiv preprint arXiv: 1707.01926

 

ChainerCV Release

Yusuke Niitani

2017-08-14 11:07:10

We released ChainerCV: a utility library for computer vision in deep learning. This library aims at making the process of training and applying deep learning models for computer vision easier using Chainer. It contains high quality implementations of computer vision models, and tools that are necessary to conduct research in this field.

GitHub page: https://github.com/chainer/chainercv
Documentation: http://chainercv.readthedocs.io/en/stable/

more »

Pieter Abbeel becomes Technical Advisor at Preferred Networks

hido

2017-07-24 12:00:56

Today, Preferred Networks is pleased to announce that Professor Pieter Abbeel, UC Berkeley and OpenAI, will be appointed as a technical advisor. PFN researchers and engineers are very excited about this news and looking forward to working with him.

Prof. Pieter Abbeel

 

Though he does not require any introduction, let me share the story behind it in the following.

more »

Chainer-GAN-lib Release

matsumoto

2017-07-11 10:30:40

We released chainer-GAN-lib: the collection of Chainer implementation of recent GAN variants. This library is targeted to those who think “the progress of GAN is too fast and hard to follow”, “Experiments in GAN articles can not be reproduced at all”, “How can I implement the gradient penalty with Chainer?”

https://github.com/pfnet-research/chainer-gan-lib

more »