AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We find that Data Boost improves the performance of classification tasks, is classifier-agnostic, and that it surpasses several prior augmentation methods in three diverse classification tasks

Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation

EMNLP 2020, pp.9031-9041, (2020)

Cited by: 2|Views114
Full Text
Bibtex
Weibo

Abstract

Data augmentation is proven to be effective in many NLU tasks, especially for those suffering from data scarcity. In this paper, we present a powerful and easy to deploy text augmentation framework, Data Boost, which augments data through reinforcement learning guided conditional generation. We evaluate Data Boost on three diverse text cl...More

Code:

Data:

0
Introduction
  • Data augmentation is a widely-used technique in classification tasks. In the field of computer vision (CV), data is augmented by flipping, cropping, tilting, and altering RGB channels of the original images (Krizhevsky et al, 2012; Chatfield et al, 2014; Szegedy et al, 2015); similar intuitive and simple strategies do not obtain equal success in NLP tasks.
  • Naive methods imitate pixel manipulation in CV, augmenting sentences by adding spelling errors (Xie et al, 2017), or randomly deleting and swapping tokens (Wei and Zou, 2019).
  • The output of such augmentation methods are often illegible since the word order is disrupted (e.g., “is The baby very!”); even worse, Original
Highlights
  • Data augmentation is a widely-used technique in classification tasks
  • We evaluated and compared Data Boost with several state-of-the-art text augmentation methods on the following three tasks: Offense Detection3 ICWSM 20’ Data Challenge dataset (N = 99, 603) for offensive language detection on tweets
  • Since we used BERT as our classifier, which is already pre-trained on a large corpus, our results confirm that Data Boost can even improve the performance of large-scale language models (LM) based classifiers
  • We find Data Boost is effective for relatively simple classifiers (e.g., CNN), and beneficial to complex LM-based classifiers (e.g., BERT and XLNet), which are already trained on a large corpus and generally used as very strong baselines for text classification tasks
  • We have proposed a powerful and easy to deploy approach to augment text data through conditional generation
  • We find that Data Boost improves the performance of classification tasks, is classifier-agnostic, and that it surpasses several prior augmentation methods in three diverse classification tasks
Methods
  • Offense Detection Sentiment Analysis Irony Classification 10% 40% PPL 10% 40% PPL 10% 40% PPL

    Naive Aug. (Coulombe, 2018)

    Word Replace Aug. (Niu and Bansal, 2018)

    EDA (Wei and Zou, 2019)

    Word2Vec Aug. (Wang and Yang, 2015)

    Contextual Word Embs Aug. (Kobayashi, 2018)

    Back-Translation Aug. (Yu et al, 2018) (Eng. → Fr. → Eng. as aug. text)

    Ours: Data Boost (RL-guided conditional generation) sifiers? In other words, is Data Boost a classifieragnostic augmentation method? To answer this question, the authors ran experiments on four other mainstream classifiers, including the plain CNN classifier (Kim, 2014), the Bi-LSTM with attention mechanism (Zhou et al, 2016), the self-attention based Transformer network (Vaswani et al, 2017), and another LM-based classifier XLNet (Yang et al, 2019) for comparison.
  • In other words, is Data Boost a classifieragnostic augmentation method?
  • The authors find Data Boost is effective for relatively simple classifiers (e.g., CNN), and beneficial to complex LM-based classifiers (e.g., BERT and XLNet), which are already trained on a large corpus and generally used as very strong baselines for text classification tasks.
  • Table 4 compares the performance of Data Boost with six prior text augmentation methods on all three tasks and using a BERT classifier.
  • Data Boost outperforms the other methods in the majority of the experiments (Table 4)
Results
  • 6.3.1 Label Agreement

    The authors conducted paired sample t-tests to examine how much participants agreed with the assigned labels.
  • Compared to the vanilla GPT-2, Data Boost samples received higher label agreement scores in eight out of nine classes.
  • Boosted data, except for the spam and normal class in Offense Detection (p = .02 and p = .03).
  • This result further confirms that Data Boost samples look very similar to the original samples and that Data Boost generates higher quality samples than the vanilla GPT-2
Conclusion
  • The authors have proposed a powerful and easy to deploy approach to augment text data through conditional generation.
  • The authors find that Data Boost improves the performance of classification tasks, is classifier-agnostic, and that it surpasses several prior augmentation methods in three diverse classification tasks.
  • The authors plan to implement a more sophisticated guidance for the augmentation by adding syntactic and position features to the reward function, to enable augmentation of more diverse types of text data.
Summary
  • Introduction:

    Data augmentation is a widely-used technique in classification tasks. In the field of computer vision (CV), data is augmented by flipping, cropping, tilting, and altering RGB channels of the original images (Krizhevsky et al, 2012; Chatfield et al, 2014; Szegedy et al, 2015); similar intuitive and simple strategies do not obtain equal success in NLP tasks.
  • Naive methods imitate pixel manipulation in CV, augmenting sentences by adding spelling errors (Xie et al, 2017), or randomly deleting and swapping tokens (Wei and Zou, 2019).
  • The output of such augmentation methods are often illegible since the word order is disrupted (e.g., “is The baby very!”); even worse, Original
  • Methods:

    Offense Detection Sentiment Analysis Irony Classification 10% 40% PPL 10% 40% PPL 10% 40% PPL

    Naive Aug. (Coulombe, 2018)

    Word Replace Aug. (Niu and Bansal, 2018)

    EDA (Wei and Zou, 2019)

    Word2Vec Aug. (Wang and Yang, 2015)

    Contextual Word Embs Aug. (Kobayashi, 2018)

    Back-Translation Aug. (Yu et al, 2018) (Eng. → Fr. → Eng. as aug. text)

    Ours: Data Boost (RL-guided conditional generation) sifiers? In other words, is Data Boost a classifieragnostic augmentation method? To answer this question, the authors ran experiments on four other mainstream classifiers, including the plain CNN classifier (Kim, 2014), the Bi-LSTM with attention mechanism (Zhou et al, 2016), the self-attention based Transformer network (Vaswani et al, 2017), and another LM-based classifier XLNet (Yang et al, 2019) for comparison.
  • In other words, is Data Boost a classifieragnostic augmentation method?
  • The authors find Data Boost is effective for relatively simple classifiers (e.g., CNN), and beneficial to complex LM-based classifiers (e.g., BERT and XLNet), which are already trained on a large corpus and generally used as very strong baselines for text classification tasks.
  • Table 4 compares the performance of Data Boost with six prior text augmentation methods on all three tasks and using a BERT classifier.
  • Data Boost outperforms the other methods in the majority of the experiments (Table 4)
  • Results:

    6.3.1 Label Agreement

    The authors conducted paired sample t-tests to examine how much participants agreed with the assigned labels.
  • Compared to the vanilla GPT-2, Data Boost samples received higher label agreement scores in eight out of nine classes.
  • Boosted data, except for the spam and normal class in Offense Detection (p = .02 and p = .03).
  • This result further confirms that Data Boost samples look very similar to the original samples and that Data Boost generates higher quality samples than the vanilla GPT-2
  • Conclusion:

    The authors have proposed a powerful and easy to deploy approach to augment text data through conditional generation.
  • The authors find that Data Boost improves the performance of classification tasks, is classifier-agnostic, and that it surpasses several prior augmentation methods in three diverse classification tasks.
  • The authors plan to implement a more sophisticated guidance for the augmentation by adding syntactic and position features to the reward function, to enable augmentation of more diverse types of text data.
Tables
  • Table1: A simple demo of existing text data augmentation methods on positive sentiment label
  • Table2: The classifier-agnostic experiments for five main-stream classifiers. We show the results before and after we apply Data Boost on two settings of training data: 20% original + 20% boosting data, and 40% original + 40% boosting data. We also list the performance of 80% as training data (full) as reference
  • Table3: Evaluation of the generation quality in terms of F1 deterioration and perplexity (PPL) increase. We keep the training data size the same, but control the ratio of original/boosting. The first results column corresponds to no boosting
  • Table4: Performance comparison with other text augmentation methods. 10%: 10% original data + 30% augmented data; 40%: 40% original data + 40% augmented data. We report the F1 score of the BERT classifier over five times repeat experiments. We also report the perplexity score (PPL) of the augmented data (10,000 randomly sampled) from different methods scored by kenLM language models trained on the training data of each task
  • Table5: Sample generation of Data Boost for all classes from three tasks. Salient words are underlined
  • Table6: Human evaluation results on readability. p-value describes the significance of difference. (* corresponds to p < 0.05, ** to p < 0.01 and *** to p < 0.001.)
Download tables as Excel
Funding
  • This research was supported in part by the Dartmouth Burke Research Initiation Award and the Amazon Research Award
Study subjects and analysis
Participants: 178
We conducted human evaluation on Amazon Mechanical Turk (MTurk) in May 2020. Participants (N = 178) were randomly assigned to evaluate one of the three tasks, respectively Irony Classification (n = 60), Sentiment Analysis (n = 58), and Offense Detection (n = 60). Participants were all from the United States and above 18 years old

data: 178
We also compare Data Boost with six prior text augmentation methods. Through human evaluations (N =178), we confirm that Data Boost augmentation has comparable quality as the original data with respect to readability and class consistency. Offense Detection Sentiment Analysis Irony Classification 10% 40% PPL 10% 40% PPL 10% 40% PPL

Naive Aug. (Coulombe, 2018) (keyboard / OCR / spelling error)

Word Replace Aug. (Niu and Bansal, 2018) (synonyms + antonym from WordNet)

EDA (Wei and Zou, 2019) (randomly delete, swap, etc.)

Word2Vec Aug. (Wang and Yang, 2015) (insert, replace using Word2Vec)

Contextual Word Embs Aug. (Kobayashi, 2018) (insert, replace using Bi-RNN LM)

Back-Translation Aug. (Yu et al, 2018) (Eng. → Fr. → Eng. as aug. text)

Ours: Data Boost (RL-guided conditional generation) sifiers? In other words, is Data Boost a classifieragnostic augmentation method? To answer this question, we ran experiments on four other mainstream classifiers, including the plain CNN classifier (Kim, 2014), the Bi-LSTM with attention mechanism (Zhou et al, 2016), the self-attention based Transformer network (Vaswani et al, 2017), and another LM-based classifier XLNet (Yang et al, 2019) for comparison

data: 178
We also compare Data Boost with six prior text augmentation methods. Through human evaluations (N =178), we confirm that Data Boost augmentation has comparable quality as the original data with respect to readability and class consistency. Data augmentation is a widely-used technique in classification tasks

data: 99
for t = 0, 1, 2, . . . do Generate (at|st) by unconditional policy πθ as trajectories; Estimate reward R(xct ) using Eq 2; Compute policy update using Eq 6 by taking k steps of SGD (via Adam); if KL(θ||θc) ≥ 2σ then βt+1 = 2βt; else if KL(θ||θc) ≤ σ/2 then βt+1 = βt / 2; end Return the conditional policy πθc; end. We evaluated and compared Data Boost with several state-of-the-art text augmentation methods on the following three tasks: Offense Detection3 ICWSM 20’ Data Challenge dataset (N = 99, 603) for offensive language detection on tweets. The dataset consists of four classes: {normal, spam, abusive and hateful} with ratio {53.9%, 27.1%, 14.1%, 4.9%} respectively

data: 20
The dataset consists of four classes: {normal, spam, abusive and hateful} with ratio {53.9%, 27.1%, 14.1%, 4.9%} respectively. Sentiment Analysis4 SemEval 2017 Task 4A dataset (N = 20, 631) for sentiment analysis in tweets. There are three classes in the dataset: {positive, neutral and negative} with ratio {34.7%, 49.8%, 15.5%}

data: 3
There are three classes in the dataset: {positive, neutral and negative} with ratio {34.7%, 49.8%, 15.5%}. Irony Classification5 SemEval 2018 Task 3A dataset (N = 3, 817) for irony detection in tweets. It has binary classes: {ironic, non-ironic}, with ratio {50.2%, 49.8%}

data: 3
We run both normal training and boosted training over the following training set fractions (%): {1%, 5%, 20%, 40%, 60%, 80%} of the total data for both Offense Detection and Sentiment Analysis. Since the dataset for Irony Classification is small (N = 3, 810), we use the following fractions: {10%, 20%, 30%, 40%, 60%, 80%}. Note that for boosted training we add augmentation samples to training data until the training data size reaches 80% of the total size (same as fully-loaded size), to make sure that the size of the training set does not influence the results

Reference
  • Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 747–754.
    Google ScholarLocate open access versionFindings
  • Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531.
    Findings
  • Mathieu Cliche. 2017. Bb twtr at semeval-2017 task 4: Twitter sentiment analysis with cnns and lstms. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 573– 580.
    Google ScholarLocate open access versionFindings
  • Claude Coulombe. 2018. Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint arXiv:1812.04718.
    Findings
  • Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
    Google ScholarLocate open access versionFindings
  • Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13042–13054.
    Google ScholarLocate open access versionFindings
  • Jakob Elming, Barbara Plank, and Dirk Hovy. 2014. Robust cross-domain sentiment analysis for lowresource languages. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2–7.
    Google ScholarLocate open access versionFindings
  • Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567– 573.
    Google ScholarLocate open access versionFindings
  • Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482–491.
    Google ScholarLocate open access versionFindings
  • Andreas Graefe, Mario Haim, Bastian Haarmann, and Hans-Bernd Brosius. 2018. Readers’ perception of computer-generated news: Credibility, expertise, and readability. Journalism, 19(5):595–610.
    Google ScholarLocate open access versionFindings
  • Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
    Findings
  • Yoon Kim. 20Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751.
    Google ScholarLocate open access versionFindings
  • Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452–457.
    Google ScholarLocate open access versionFindings
  • Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105.
    Google ScholarLocate open access versionFindings
  • Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
    Findings
  • Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
    Google ScholarLocate open access versionFindings
  • Michael Mohler, Mary Brunson, Bryan Rink, and Marc Tomlinson. 2016. Introducing the lcc metaphor datasets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4221–4227.
    Google ScholarLocate open access versionFindings
  • Remi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. 2016. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054–1062.
    Google ScholarLocate open access versionFindings
  • Tong Niu and Mohit Bansal. 2018. Adversarial oversensitivity and over-stability strategies for dialogue models. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 486–496.
    Google ScholarLocate open access versionFindings
  • Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
    Google ScholarLocate open access versionFindings
  • Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
    Findings
  • Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129–140.
    Google ScholarLocate open access versionFindings
  • Nils Reimers and Iryna Gurevych. 2019. Sentencebert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973–3983.
    Google ScholarLocate open access versionFindings
  • John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
    Findings
  • Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96.
    Google ScholarLocate open access versionFindings
  • Aliaksei Severyn and Alessandro Moschitti. 2015. Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 959–962.
    Google ScholarLocate open access versionFindings
  • Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 90–99.
    Google ScholarLocate open access versionFindings
  • William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557–2563, Lisbon, Portugal. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138– 142, Austin, Texas. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 6383–6389.
    Google ScholarLocate open access versionFindings
  • Ziang Xie, Sida I Wang, Jiwei Li, Daniel Levy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data noising as smoothing in neural network language models. arXiv preprint arXiv:1703.02573.
    Findings
  • Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. arXiv, pages arXiv–2004.
    Google ScholarFindings
  • Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764.
    Google ScholarLocate open access versionFindings
  • Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
    Google ScholarLocate open access versionFindings
  • Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657.
    Google ScholarLocate open access versionFindings
  • Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
  • Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: Short papers), pages 207– 212.
    Google ScholarLocate open access versionFindings
Author
Guangxuan Xu
Guangxuan Xu
Chenyan Jia
Chenyan Jia
Weicheng Ma
Weicheng Ma
Lili Wang
Lili Wang
Soroush Vosoughi
Soroush Vosoughi
Your rating :
0

 

Tags
Comments
小科