You called a math function and the result is not defined. for example, the log of 0 is not defined. Or, the result is too large/small to be represented on your machine.
This error number is also returned for example in gethostbyname_r (the re-entrant version of gethostbyname(3)) if the buffer given to the function is too small to store "auxiliary data".
Compare with EDOM
