The position of A and B can be random.

If the position of A and B are xa and xb, then the center of mass of the system is xc=(ma*xa+mb*xb)/(ma+mb);

You can get similar equation for ya and yb.

i.e. yc=(ma*ya+mb*yb)/(ma+mb);

For the particle to be in circular motion, the velocity need to be perpendicular to the radial direction.

And ma*va+mb*vb=0; to prevent drift motion

the centripetal force f=m*v^2/r

You have all the equations you need now.

You are welcomed to check out http://www.phy.ntnu.edu.tw/ntnujava/index.php?topic=2184.0