Let's take a simple brushed DC series wound motor for this example. It's rated as 1700 max rmp @ 48V. We overvolt it to 144V. It's max revs are now 5,100 rpm ( 3x 1700 - in theory).
I think you might be confusing max ratings with an expected result. This is a fictional motor I presume, so it probably wouldn't be rated this way.
For almost any other motor type, you can easily predict the unloaded RPM of a motor in wide circumstances. For example, synchronous AC motors will rotate based on however fast the coils get pulsed, asynchronous will always be behind a bit but fairly close. Permanent Magnet DC motors will turn a certain speed based on the voltage you give them, and you can pretty much linearly plot that (double the voltage, double the RPM). You might sometimes hear the term "KV" applied to a motor which means how many RPMs it will spin to, per volt you feed it (so a KV of 100 means if you give it 1 volt it will spin 100 RPM, 12 volts it will spin 1200 RPM, etc. They're telling you the ratio of RPM:Volts). SepEx and Shunt (parallel) wound DC motors will have similar, predictable speeds.
But DC Series motors you can't tell. With any voltage and no load, they accelerate forever, very quickly. The only thing that slows them down is having a load on them, and their speed/current curve isn't linear. So for example, they're not a great choice for any drive system that has a chain, because when the chain inevitably snaps or hops off a sprocket, the motor will just about instantly accelerate until it rips itself apart.
So in your example, if the motor is "rated" for 1700 RPM, that probably means that is its mechanical ceiling before its physical structural integrity can't be relied on anymore (as per whatever conservative degree of its rating and circumstances). Or, perhaps that in that specific application with that load, at 48V it would be expected to spin 1700 RPM. But it would have to be a very specifically known load.
For example, if you give an unloaded DC Series motor 5v, it will reach 12,000 RPM and then maybe explode. If you give it 50v, it will reach 12,000 RPM in much less time, but then maybe explode. It doesn't really matter what voltage you give it, without some drag, it spins up and then explodes. 😛
By tripling the voltage, you don't physically change anything about the motor. It becomes no more strong at holding itself together. Much like if a chain is rated for 1700 lbs, you can't make it be rated for 5100 lbs. It just is what it is.
Bigger motors will tend to be rated for a lower max RPM than smaller motors because the outside edge of the rotor is moving faster on a larger diameter.
TL;DR -
1 - Your (fictional?) motor is probably not rated for a max of only 1700 RPM. That's pretty slow. But if it was, max means mechanically the max before it blows up, nothing to do with voltage, and,
2 - Your (fictional) motor is probably not rated for 1700 RPM @ 48volts, because you say it's a DC Series motor and that's nonsensical since there isn't a "no load" speed for that type of motor.
What will happen when the DC motor at 144V, tries to reach 5100 rpm, but can't, because it hasn't got enough torque? While it struggles to reach 5100 rpm and fails to, will it overheat?
In short yes. Demand too much of anything and it will fail. Almost all electronics have a thermal limit by which they fail (in addition to the RPM limit mentioned earlier which you probably won't reach).
That's probably not the easiest way to think about it, for any motor type.
For example, think about going down a steep downhill. You might barely have to push at all to make the motor spin at 5100 RPM. So it's not how fast you spin the motor, it's how hard it is to spin the motor that fast.
I would kind of abstract the whole "target speed" thing and not think about it that way.
Think about a normal gas car driving down a highway. You have the accelerator pushed down some amount. Let's say it's flat and not windy so nothing is changing. Because wind resistance increases as a cube of speed (it's increasingly hard to go faster), you will eventually come to a steady speed based on how far you're pushing the pedal.
You don't have to know the math or quantities behind how much gas is pouring into the engine every second, exploding and propelling the car. You know that if you back off the gas, the car will slow down and reach a slower steady speed. If you step on the gas, the car will speed up and reach a faster steady speed. You don't have a pedal position that dictates "speed" and half way down is half speed, all the way down is full speed, etc. It depends on wind, whether you're climbing, whether you're towing, whether you're accelerating, etc. More is more and less is less.
Same thing for electric motors. Tell the throttle to give you "more", it will demand more. Tell the throttle for "less", it will give less.
If you redline a gas engine, suppose you have a small engine and you're trying to tow a heavy trailer up a hill, or, suppose you're trying to go 200mph on a crappy car... you will be demanding More and More and More. At some point you will only be able to put so much gas into the engine and it can't give you move (pedal to the metal), but more troublesome is that with you redlining the engine it's going to be getting hotter than you can keep cool, temperatures will rise, and something will blow up.
Ditto for electric motors. If you have too small of a motor and you're trying to do too much to it... accelerate too fast, reach too high a speed, drag too much weight up a hill... it's going to overheat and something will blow up. Unlike with gas engines though, motors have a bigger swing in what you can demand from them. You can only shove so much fuel and oxygen into a piston, and chemistry can only happen so quick. But electric motors you could abuse to a wider rate, several whole multiples of their rated steady power for very short periods, hitting only magnetic limits in the core I suppose.
I would think more in terms of power.
How much power is required from your motor, and can it sustain that?
You can know about the power it takes to travel a certain speed, or accelerate a weight at a certain rate, or travel up a hill a certain speed from that calculator I linked earlier. Those are physical facts. It will take X amount of power to get a certain car of a certain weight and shape, to travel a certain speed up a certain incline in certain weather. That amount of power has to come from the motor.
When you press the accelerator, you're in the end asking for more power. The actual sequence will be something like:
1 - Pedal position tells a circuit what duty cycle to pulse the motor on/off at since changing the actual voltage is impractical.
2 - The power transistors do as instructed, and pulse the voltage to the motor with those proportions of on
ff time.
3 - This averages a net voltage on the motor (i.e. 50% on/off time for 100 volts, appears like 50 volts and the motor would act the same if you gave it 50v steady).
4 - Voltage causes current (amps) to flow.
5 - Current makes the motor spin.
6 - The harder it is to spin, (not speed directly, but all the things that make a car difficult to move like wind, hills, and accelerating) the more current flows for a given voltage.
7 - The more current the more heat.
8 - Too much heat and something melts, either power transistors or motor brushes or motor wires or, something.
So you're not really telling it "Go this fast", you're telling it "Push this much harder". And pushing hard is what makes heat.
Think of perhaps, a toy electric car that you put your foot on. What happens? The motor can't pull you, it tries as hard as it can, and it overheats and melts. Ditto for an undersized motor in a car. It tries, it overheats, and melts.
By whatever method you're asking for "more", if it's more than the motor can provide, it dies.
