To which extent can we increase the speed of a motor by VFD from its rated RPM
It depends on the application when you increase the operating frequency over the rated frequency you start losing torque so it depends on the load some vsds let you operate till you reach 200 hz and others more than that.
Max permitted motor speed by the motor supplier is the top safe speed limit. But you should know the torque max of the motor at that speed. As you enter the field weakening region (beyond the rated motor speed) motor torque falls ( torque proportional to V/f). It means you have to calculate load torque check suitability with the motor. By torque … not just the running torque required. Also, the acceleration and deceleration torques as required by the application. Many branded VFDs have power optimization programs built in the drive control software.