Using deep neural network with small dataset to predict material defects

Shuo Feng

Huiyu Zhou

[0]

Hongbiao Dong (董洪标)

[0]

Materials & Design, pp. 300-310, 2019.

Cited by: 48|Bibtex|Views61|DOI:https://doi.org/10.1016/j.matdes.2018.11.060

Other Links: academic.microsoft.com

Keywords:

Pre-trainingsolidification cracking susceptibilityneural networkPrincipal component analysisrestricted Boltzmann machineMore(19+)

Weibo:

Study in this paper demonstrates that small/narrow deep neural network with small dataset and special training methods has huge potential for extensive applications in material study, especially for those multivariable nonlinear problems

Abstract:

Abstract Deep neural network (DNN) exhibits state-of-the-art performance in many fields including microstructure recognition where big dataset is used in training. However, DNN trained by conventional methods with small datasets commonly shows worse performance than traditional machine learning methods, e.g. shallow neural network and s...More

Code:

Data:

Insight

Full Text

PPT (Upload PPT)

Similar

Reference

Introduction

Research papers [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18] and Reviews [19,20] about SNN applications in material science and engineering have been widely reported.

Highlights

The success of the previous generation neural network is limited to shallow neural network (SNN) of 1 or 2 hidden layers, because training deep neural network (DNN) is not easy and the resultant accuracy is usually worse than that of SNN [17]
DNN can be perceived as an upgrade of SNN
Matsuda [75] reported that Mn addition to SUS310S with 0.002 wt% P and 0.004 wt% S exerts a harmful effect on cracking resistance under 4% applied-strain conditions using transverse varestraint test
The derived mathematical expression can be used in the material defects prediction, new alloys development, and comparison with experimental results
Study in this paper demonstrates that small/narrow DNN with small dataset and special training methods has huge potential for extensive applications in material study, especially for those multivariable nonlinear problems

Methods

Training algorithm and Bayesian regularization.
The performance function mse and regularized performance function msereg of a neural network are defined as follows: mse 1⁄4 1 N XN ðeiÞ2 i1⁄41.
XN i1⁄41 ðt i −ai Þ2.
Ð1Þ msereg 1⁄4 γmse þ ð1−γÞmsw ð2Þ msw 1 n Xn W 2 j ð3Þ j1⁄41

Results

Matsuda [75] reported that Mn addition to SUS310S with 0.002 wt% P and 0.004 wt% S exerts a harmful effect on cracking resistance under 4% applied-strain conditions using transverse varestraint test.

Conclusion

Connected DNN which consists of 3 or more hidden layers shows its advantage over shallow neural network and support vector machine in that it can achieve higher prediction accuracy and better generalization performance.

Through DNN regression, vast scattered experimental data in the literature can produce simple quantitative expression of specific material property as function of chemistry composition and processing parameters, etc.
Deep and narrow neural network are suitable for material problems, such as solidification cracking susceptibility prediction in this paper, and pretraining using stacked auto-encoder is effective and necessary in the DNN regression of numerical small datasets.
Study in this paper demonstrates that small/narrow DNN with small dataset and special training methods has huge potential for extensive applications in material study, especially for those multivariable nonlinear problems

Summary

Introduction:
Research papers [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18] and Reviews [19,20] about SNN applications in material science and engineering have been widely reported.
Methods:
Training algorithm and Bayesian regularization.
The performance function mse and regularized performance function msereg of a neural network are defined as follows: mse 1⁄4 1 N XN ðeiÞ2 i1⁄41.
XN i1⁄41 ðt i −ai Þ2.
Ð1Þ msereg 1⁄4 γmse þ ð1−γÞmsw ð2Þ msw 1 n Xn W 2 j ð3Þ j1⁄41
Results:
Matsuda [75] reported that Mn addition to SUS310S with 0.002 wt% P and 0.004 wt% S exerts a harmful effect on cracking resistance under 4% applied-strain conditions using transverse varestraint test.
Conclusion:
Connected DNN which consists of 3 or more hidden layers shows its advantage over shallow neural network and support vector machine in that it can achieve higher prediction accuracy and better generalization performance.

Through DNN regression, vast scattered experimental data in the literature can produce simple quantitative expression of specific material property as function of chemistry composition and processing parameters, etc.
Deep and narrow neural network are suitable for material problems, such as solidification cracking susceptibility prediction in this paper, and pretraining using stacked auto-encoder is effective and necessary in the DNN regression of numerical small datasets.
Study in this paper demonstrates that small/narrow DNN with small dataset and special training methods has huge potential for extensive applications in material study, especially for those multivariable nonlinear problems

Tables

Table1: The accuracies of SVM, SNN and DNN based on dataset without PCA pre-process and datasets after PCA pre-process
Table2: The specifications (wt%) of 304 and 310S stainless steel and the testing parameters used for prediction
Table3: Compositions (wt%) of stainless steels used in prediction (varied P and S)

Download tables as Excel

Funding

Shuo Feng wishes to acknowledge EPSRC CDT (Grant No: EP/ L016206/1) in Innovative Metal Processing for providing a PhD studentship for this study

Study subjects and analysis

raw input datasets: 21

Principal component analysis (PCA) is known to reduce data dimensions and noise in the input data [71]. For this study, 21 raw input datasets were pre-proceeded by PCA method that resulted in two datasets, one with 15 principle components (percentage of the total variance N99%) and the other with 10 principle components (percentage of the total variance N95%). These two pre-proceeded datasets as well as raw dataset were used as input in the training and testing of SVM, SNN and DNN

Reference

L. Guo, H. Roelofs, M.I. Lembke, H.K.D.H. Bhadeshia, Modelling of transition from upper to lower bainite in multi-component system, Mater. Sci. Technol. 33 (4) (2017) 430–437.
M. Vasudevan, A.K. Bhaduri, B. Raj, K.P. Rao, Artificial neural network modelling of solidification mode in austenitic stainless steel welds, Mater. Sci. Technol. 23 (4) (2007) 451–459.
D.R. Cassar, A.C.P.L.F. de Carvalho, E.D. Zanotto, Predicting glass transition temperatures using neural networks, Acta Mater. 159 (2018) 249–256.
R.C. Dimitriu, H.K.D.H. Bhadeshia, C. Fillon, C. Poloni, Strength of ferritic steels: neural networks and genetic programming, Mater. Manuf. Process. 24 (1) (2008) 10–15.
S. Malinov, W. Sha, Application of artificial neural networks for modelling correlations in titanium alloys, Mater. Sci. Eng. A 365 (1) (2004) 202–211.
W. Solano-Alvarez, M.J. Peet, E.J. Pickering, J. Jaiswal, A. Bevan, H.K.D.H. Bhadeshia, Synchrotron and neural network analysis of the influence of composition and heat treatment on the rolling contact fatigue of hypereutectoid pearlitic steels, Mater. Sci. Eng. A 707 (2017) 259–269.
H.K.D.H. Bhadeshia, D.J.C. MacKay, L.E. Svensson, Impact toughness of C–Mn steel arc welds – Bayesian neural network analysis, Mater. Sci. Technol. 11 (10) (1995) 1046–1051.
T. Sourmail, H.K.D.H. Bhadeshia, D.J.C. MacKay, Neural network model of creep strength of austenitic stainless steels, Mater. Sci. Technol. 18 (6) (2002) 655–663.
G. Allegri, Modelling fatigue delamination growth in fibre-reinforced composites: power-law equations or artificial neural networks? Mater. Des. 155 (2018) 59–70.
Z. Anjum, F. Qayyum, S. Khushnood, S. Ahmed, M. Shah, Prediction of nonpropagating fretting fatigue cracks in Ti6Al4V sheet tested under pin-in-dovetail configuration: experimentation and numerical simulation, Mater. Des. 87 (2015) 750–758.
B.D. Conduit, N.G. Jones, H.J. Stone, G.J. Conduit, Design of a nickel-base superalloy using a neural network, Mater. Des. 131 (2017) 358–365.
M.W. Dewan, D.J. Huggett, T. Warren Liao, M.A. Wahab, A.M. Okeil, Prediction of tensile strength of friction stir weld joints with adaptive neuro-fuzzy inference system (ANFIS) and neural network, Mater. Des. 92 (2016) 288–299.
G. Liu, L. Jia, B. Kong, K. Guan, H. Zhang, Artificial neural network application to study quantitative relationship between silicide and fracture toughness of Nb-Si alloys, Mater. Des. 129 (2017) 210–218.
O. Palavar, D. Özyürek, A. Kalyon, Artificial neural network prediction of aging effects on the wear behavior of IN706 superalloy, Mater. Des. 82 (2015) 164–172.
X. Xia, J.F. Nie, C.H.J. Davies, W.N. Tang, W. Xu, N. Birbilis, An artificial neural network for predicting corrosion rate and hardness of magnesium alloys, Mater. Des. 90 (2016) 1034–1043.
L. Aijun, L. Hejun, L. Kezhi, G. Zhengbing, Applications of neural networks and genetic algorithms to CVI processes in carbon/carbon composites, Acta Mater. 52 (2) (2004) 299–305.
S. Malinov, W. Sha, J.J. McKeown, Modelling the correlation between processing parameters and properties in titanium alloys using artificial neural network, Comput. Mater. Sci. 21 (3) (2001) 375–394.
K. Ichikawa, H.K.D.H. Bhadeshia, D.J.C. MacKay, Model for solidification cracking in low alloy steel weld metals, Sci. Technol. Weld. Join. 1 (1) (1996) 43–50.
H.K.D.H. Bhadeshia, Neural networks in materials science, ISIJ Int. 39 (10) (1999) 966–979.
H.K.D.H. Bhadeshia, R.C. Dimitriu, S. Forsik, J.H. Pak, J.H. Ryu, Performance of neural networks in materials science, Mater. Sci. Technol. 25 (4) (2009) 504–510.
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT press, Cambridge, 2016.
J. Schmidhuber, Deep learning in neural networks: an overview, Neural Netw. 61 (2015) 85–117.
G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (5786) (2006) 504–507.
Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436.
G.E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (7) (2006) 1527–1554.
Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, Greedy layer-wise training of deep networks, Adv. Neural Inf. Proces. Syst. (2007) 153–160.
D. Kotecki, J. Lippold, Welding Metallurgy and Weldability of Stainless Steels, John Wiley & Sons, 2005.
S. Kou, Welding Metallurgy, John Wiley & Sons, 2003.
J. Campbell, Chapter 8 - Cracks and tears, Complete Casting Handbook, Butterworth-Heinemann, Oxford, 2011 465–497.
J.A. Dantzig, M. Rappaz, Solidification, EPFL press, 2009.
S. Li, D. Apelian, Hot tearing of aluminum alloys, Int. J. Met. 5 (1) (2011) 23–40.
W.J. Sames, F.A. List, S. Pannala, R.R. Dehoff, S.S. Babu, The metallurgy and processing science of metal additive manufacturing, Int. Mater. Rev. 61 (5) (2016) 315–360.
D.D. Gu, W. Meiners, K. Wissenbach, R. Poprawe, Laser additive manufacturing of metallic components: materials, processes and mechanisms, Int. Mater. Rev. 57 (3) (2012) 133–164.
L. Aucott, D. Huang, H.B. Dong, S.W. Wen, J. Marsden, A. Rack, A.C.F. Cocks, A threestage mechanistic model for solidification cracking during welding of steel, Metall. Mater. Trans. A 49 (5) (2018) 1674–1682.
L. Aucott, D. Huang, H. Dong, S. Wen, J. Marsden, A. Rack, A. Cocks, Initiation and growth kinetics of solidification cracking during welding of steel, Sci. Rep. 7 (2017), 40255.. [36] D.G. Eskin, L. Katgerman Suyitno, Mechanical properties in the semi-solid state and hot tearing of aluminium alloys, Mater. Sci. Eng. R 49 (5) (2004) 629–711.
[37] T. Soysal, S. Kou, A simple test for assessing solidification cracking susceptibility and checking validity of susceptibility prediction, Acta Mater. 143 (2018) 181–197.
[38] S. Kou, A criterion for cracking during solidification, Acta Mater. 88 (2015) 366–374.
[39] M. Rappaz, J.M. Drezet, M. Gremaud, A new hot-tearing criterion, Metall. Mater. Trans. A 30 (2) (1999) 449–455.
[40] C. Bordreuil, A. Niel, Modelling of hot cracking in welding with a cellular automaton combined with an intergranular fluid flow model, Comput. Mater. Sci. 82 (2014) 442–450.
[41] A. Niel, C. Bordreuil, F. Deschaux-Beaume, G. Fras, Modelling hot cracking in 6061 aluminium alloy weld metal with microstructure based criterion, Sci. Technol. Weld. Join. 18 (2) (2013) 154–160.
[42] B.G. Thomas, Modeling of hot tearing and other defects in casting processes, ASM Handbook 22 (2009) 362–374.
[43] A. Cecen, H. Dai, Y.C. Yabansu, S.R. Kalidindi, L. Song, Material structure-property linkages using three-dimensional convolutional neural networks, Acta Mater. 146 (2018) 76–84.
[44] B.L. Decost, T. Francis, E.A. Holm, Exploring the microstructure manifold: image texture representations applied to ultrahigh carbon steel microstructures, Acta Mater. 133 (2017) 30–40.
[45] R. Kondo, S. Yamakawa, Y. Masuoka, S. Tajima, R. Asahi, Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics, Acta Mater. 141 (2017) 29–38.
[46] E. Kim, K. Huang, S. Jegelka, E. Olivetti, Virtual screening of inorganic materials synthesis parameters with deep learning, npj Comp. Mater. 3 (1) (2017) 53.
[47] L. Ward, A. Agrawal, A. Choudhary, C. Wolverton, A general-purpose machine learning framework for predicting properties of inorganic materials, npj Comp. Mater. 2 (2016), 16028..
[48] L. Ward, S.C. O'Keeffe, J. Stevick, G.R. Jelbert, M. Aykol, C. Wolverton, A machine learning approach for engineering bulk metallic glass alloys, Acta Mater. 159 (2018) 102–111.
[49] Y.T. Sun, H. Bai, M.-Z. Li, W. Wang, Machine learning approach for prediction and understanding of glass-forming ability, J. Phys. Chem. Lett. 8 (14) (2017) 3434–3439.
[50] H.B. Demuth, M.H. Beale, O. De Jess, M.T. Hagan, Neural Network design, Martin Hagan, 2014.
[51] D.J. MacKay, A practical Bayesian framework for backpropagation networks, Neural Comput. 4 (3) (1992) 448–472.
[52] D.J. MacKay, Bayesian interpolation, Neural Comput. 4 (3) (1992) 415–447.
[53] Y. Arata, F. Matsuda, S. Saruwatari, Varestraint test for solidification crack susceptibility in weld metal of austenitic stainless steels, Trans. JWRI 3 (1) (1974) 79–88.
[54] J.A. Brooks, W.M. Garrison, Weld microstructure development and properties of precipitation-strengthened martensitic stainless steels, Weld. J. 78 (8) (1999) 280S–291S. [55] M.J. Cieslak, W.F. Savage, Weldability and solidification phenomena of cast stainlesssteel, Weld. J. 59 (5) (1980) S136–S146.
[56] D.H. Kah, D.W. Dickinson, Weldability of ferritic stainless-steels, Weld. J. 60 (8) (1981) S135–S142.
[57] L. Li, R.W. Messler, Effects of phosphorus and sulfur on susceptibility to weld hot cracking in austenitic stainless steels, WRC Bul. 480 (2003).
[58] J.C. Lippold, An investigation of weld cracking in alloy 800, Weld. J. 63 (3) (1984) S91–S103.
[59] C. Lundin, C. Qiao, Y. Kikuchi, C. Shi, T. Gill, Investigation of Joining Techniques for Advanced Austenitic Alloys, Oak Ridge National Lab, TN (United States); Tennessee Univ., Knoxville, TN (United States), 1991.
[60] C.D. Lundin, C.H. Lee, C.Y.P. Qiao, Evaluation of backfilled solidification cracks in austenitic stainless welds in relationship to evaluation of hot cracking, Weld. J. 72 (7) (1993) S321–S328.
[61] C.D. Lundin, C. Qiao, C. Lee, G. Batten, Weldability and hot ductility behavior of nuclear grade austenitic stainless steels, WRC Bul. 509 (2006).
[62] C.D. Lundin, C.Y.P. Qiao, T.P.S. Gill, G.M. Goodwin, Hot ductility and hot cracking behavior of modified 316 stainless-steels designed for high-temperature service, Weld. J. 72 (5) (1993) S189–S200.
[63] H. Madarame, T. Sukegawa, H. Inoue, Cracking susceptibility of stainless-steel subjected to plasma disruption, Fusion Eng. Des. 27 (1995) 499–506.
[64] T. Ogawa, T. Koseki, Weldability of newly developed austenitic alloys for cryogenic service.
2. High-nitrogen stainless-steel weld metal, Weld. J. 67 (1) (1988) S8–S17.
[65] T. Ogawa, E. Tsunetomi, Hot cracking susceptibility of austenitic stainless-steels, Weld. J. 61 (3) (1982) S82–S93.
[66] A.M. Ritter, W.F. Savage, Solidification and solidification cracking in nitrogenstrengthened austenitic stainless-steels, Metall. Trans. A. 17 (4) (1986) 727–737.
[67] C.V. Robino, J.R. Michael, M.C. Maguire, The solidification and welding metallurgy of galling-resistant stainless steels, Weld. J. 77 (11) (1998) 446S–457S.
[68] S. Shono, O. Matsumoto, S. Kawaguchi, T. Urabe, Weldability of 18-percent chromium ferritic stainless-steel with titanium addition, Trans. Iron Steel Inst. Jpn. 24 (7) (1984) 557–565.
[69] I. Varol, W.A. Baeslack, J.C. Lippold, Characterization of weld solidification cracking in a duplex stainless-steel, Metallography 23 (1) (1989) 1–19.
[70] M. Zaghloul, A. Sadek, A. El-Batahgy, M. Hanafy, Effect of welding parameters on hot cracking susceptibility of alloy 800, Yosetsu Gakkai Ronbunshu 12 (3) (1994) 335–341.
[71] R. May, G. Dandy, H. Maier, Review of input variable selection methods for artificial neural networks, Artificial Neural Networks-methodological Advances and Biomedical Applications, InTech, 2011.
[72] Statistics and Machine Learning ToolboxTM User's Guide., The MathWorks, Inc., 2018
[73] A.K. Bhaduri, S.K. Albert, B. Raj, 2 - robust welding technologies for ferrous alloys, in: R. Singh (Ed.), Weld Cracking in Ferrous Alloys, Woodhead Publishing 2009, pp. 34–95.
[74] Y. Arata, F. Matsuda, H. Nakagawa, S. Katayama, Solidification crack susceptibility in weld metals of fully austenitic stainless steels (report IV): effect of decreasing P and Son solidification crack susceptibility of SUS 310S austenitic stainless steel weld metals, Trans. JWRI 7 (2) (1978) 169–172.
[75] F. Matsuda, H. Nakagawa, S. Katayama, Y. Arata, Solidification crack susceptibility in weld metals of fully austenitic stainless steels (report VII): effect of Mn and N on solidification crack resistance, Trans. JWRI 11 (2) (1982) 79–85.
[76] F. Matsuda, S. Katayama, Y. Arata, Solidification crack susceptibility in weld metals of fully austenitic stainless steels (report IX): effect of titanium on solidification crack resistance, Trans. JWRI 12 (2) (1983) 247–252.
[77] F. Matsuda, H. Nakagawa, S. Katayama, Y. Arata, Solidification crack susceptibility in weld metals of fully austenitic stainless steels (report VIII): effect of nitrogen on cracking in SUS 304 weld metal, Trans. JWRI 12 (6) (1983) 89–95.
[78] Y. Arata, F. Matsuda, S. Katayama, Solidification crack susceptibility in weld metals of fully austenitic stainless steels (report II): effect of ferrite, P, S, C, Si and Mn on ductility properties of solidification brittleness, Trans. JWRI 6 (1) (1977) 105–116.

Full Text

Your rating :

Using deep neural network with small dataset to predict material defects

Introduction:

Methods:

Results:

Conclusion: