Application of fractal theory to quantify the roughness of rock fractures has been well developed, and various methods have been presented to estimate the fractal dimension of roughness profiles, D. However, determination of D is still a challenging issue, and researchers have attributed different values to fractal dimensions of the standard profiles of rock fracture roughness. Two kinds of errors affect the estimated fractal dimension: stochastic errors and systematic errors. Both errors cannot be modeled by any explicit function, because of the complexity and the uncertainty in the relationship between the fractal dimension and the measurable variables. In this paper, a new method is presented to overcome these deficiencies. First, a large number of fractional Brownian profiles with different values of fractal parameters (D and standard deviation sigma) were generated and their statistical features were extracted. Then, a hybrid algorithm consisting of two kinds of algorithms, particle swarm optimization (PSO) algorithm and multi-layer perceptron (MLP) neural network, was developed. The inputs of the system were the features of profile namely root mean square of profile and its first derivatives. In each iteration of the hybrid system, the PSO and MLP algorithms exchanged information with each other to optimize values of the fractal parameters. Finally, the system provided the best set of fractal parameters for a given profile. The effectiveness of this method was shown by estimating the fractal parameters of a testing set of profiles. Finally, the values of D and sigma were estimated for Barton's standard roughness profiles and for the profiles extracted from digitized surfaces of natural rock fractures. (C) 2012 Elsevier Ltd. All rights reserved.