A cost function compares the output of the function you are trying to optimize with the real data.
You have a function f(x)=prediction. Cost function=C tells you the difference between prediction and real data.
In addition, if the cost function is differentiable, you can calculate gradients for function f parameters and calculate how to change those parameters to reduce cost.
Here I have mentioned the other important part of Cost Functions “Parameters”. You can think of Parameters as sort of configuration for f(), telling it how to behave. As is common in math notation, in further discussion lets name parameters as $\theta$
So developing the above claim bit further
You have a function f(x, $\theta$ ) =prediction. Cost function=C tells you the difference between prediction and real data.
The most simple example of a cost function is Mean Squared Error MSE = $\sum(f(x)-y)^2$. Deeper dive on MSE here
If we start to try to calculate the derivative of above MSE, we see that for that we need to know what